We describe best practices in reporting reliability estimates in counseling research with consideration to precision, generalization, and diverse populations. We provide a historical context to reporting reliability estimates, the limitations of past practices, and new methods to address reliability generalization. We highlight best practices…

We present the R package and web app "statcheck" to automatically detect statistical reporting inconsistencies in primary studies and meta-analyses. Previous research has shown a high prevalence of reported p-values that are inconsistent--meaning a re-calculated p-value, based on the reported test statistic and degrees of freedom, does…

The cloud-based research synthesis platform metaBUS is presented as a tool for human resource development research. A general overview of the meta-analysis tool metaBUS is provided. In addition, this article shows how metaBUS can be utilized for other research purposes such as a starting point for identifying keywords when conducting literature…

Network meta-analysis enables the simultaneous synthesis of a network of clinical trials comparing any number of treatments. Potential inconsistencies between estimates of relative treatment effects are an important concern, and several methods to detect inconsistency have been proposed. This paper is concerned with the node-splitting approach,…

In this article I provide an overview of the methodological issues involved in measuring constructs relevant to character development and education. I begin with a nontechnical overview of the 3 fundamental psychometric properties of measurement: reliability, validity, and equivalence. Developing and evaluating measures to ensure evidence of all 3…

States and school districts are exploring alternatives to state tests for measuring teachers' contributions to student learning. One approach applies statistical value-added methods to alternative student assessments such as commercially available tests and end-of course tests. The evidence suggests that these methods can reliably distinguish…

The authors have over the years tried to revise meta-analysis because it's basic premise is to add apples and oranges together and analyze. In other words, various data on the same subject are chosen using different samples, research strategies, and number properties. The findings are then homogenized and a statistical analysis is used (Snell, J.…

Clinical judgment is foundational to psychological practice. Accurate judgment forms the basis for establishing reasonable goals and selecting appropriate treatments, which in turn are essential in achieving positive therapeutic outcomes. Therefore, Spengler and colleagues' meta-analytic finding--clinical judgment accuracy improves marginally with…

Reviews issues, raised in part because of "Educational and Psychological Measurement" (EPM) policies, regarding "test reliability," which is psychometric terminology, and "score reliability," score-centric terminology. Discusses datametrics and provides a critique of T. Vacha-Haase's proposed meta-analytic reliability generalization via…

Responds to criticisms of some "Educational and Psychological Measurement" policies and the reliability generalization meta-analytic methods of T. Vacha-Haase. Explores consequences of misunderstanding score reliability. (SLD)

Discusses reliability issues in light of recent studies and debates focused on psychometrics versus datametrics terminology and reliabilities generalization. Discusses the way multiple perspectives on score reliability may affect research practice, editorial policies, and reliability generalization across studies. (SLD)

This paper discusses the fact that reliability is about scores and not tests and how reliability limits effect sizes. The paper also explores the classical reliability coefficients of stability, equivalence, and internal consistency. Stability is concerned with how stable test scores will be over time, while equivalence addresses the relationship…

The validity generalization model of meta analysis is described and modified for application to concurrent validity studies of motor behavior. An example is presented. (Author/MT)

Researchers consistently fail to report reliability estimates for data used in their studies. This lack of reporting hinders appropriate evaluation and interpretation of data and may lead to inappropriate conclusions. Because reliability is inured to scores obtained from a test, and not the test itself, it is important to report score reliability…

Guidelines are offered that make it more likely that high-quality information will be extracted and coded from primary research reports in meta-analyses. It is also noted that the methodology of meta-analysis results in pressure to change the type of information that appears in primary research reports. (SLD)