The replication crisis in social and behavioral sciences has raised concerns about the reliability and validity of empirical studies. While research in the literature has explored contributing factors to this crisis, the issues related to analytical tools have received less attention. This study focuses on a widely used analytical tool -…

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

Procedures are developed for testing the hypothesis that Cronbach's alpha reliability coefficient is equal for two tests given to the same subjects. (Author/JKS)

The objective of this paper is to examine the relationship between the unreliability of difference scores and the power of tests of significance in an attempt to determine the validity of the paradox for the measurement of change presented by Overall and Woodward: that the power of tests of significance is maximum when the reliability of the…

Some of the methods originally used to find relationships between reliability and power associated with a single measurement are extended to difference scores. Results, based on explicit power calculations, show that augmenting the reliability of measurement by reducing error score variance can make significance tests of difference more powerful.…

Two articles discuss the controversy about the relationship between reliability and the power of significance tests in response to the discussion of Donald W. Zimmerman, Richard H. Williams, and Bruno D. Zumbo. Lloyd G. Humphreys emphasizes the differences between what statisticians can do and constraints on researchers. Zimmerman, Williams, and…

Livingston's work is a careful analysis of what occurs when one pools two populations with different means, but similar variances and reliability coefficients. However, his work fails to advance reliability theory for the special case of criterion-referenced testing. See ED 042 802 for Livingston's paper. (MS)

The implications of a mathematical model of test scores are explored where the data are limited to a random sample of items without replacement from an indefinitely large population or item domain in which items are scored either zero or one. The purpose is to obtain an unbiased estimate of a student's proportion of items correct in the item…

Eight different methods are reviewed for determining whether two or more tests are equivalent measures. These methods vary in restrictiveness from the Wilks-Votaw test of compound symmetry (which requires that all means, variances, and covariances are equal), to Joreskog's theory of congeneric tests (which requires only that the tests are measures…

Three tables are provided to aid in the clinical interpretation of factor scores for the Wechsler Adult Intelligence Scale-Revised (WAIS-R; 1981). The factor structure of the WAIS-R has proven to be robust across samples, tests, time, statistical analyses, measurement scales, and distortions of the distribution. Information necessary to make…

The regression-discontinuity approach to evaluating educational programs is reviewed, and regression-discontinuity post-program mean differences under various conditions are discussed. The regression-discontinuity design is used to determine whether post-program differences exist between an experimental program and a control group. The difference…

The four main approaches to measuring treatment effects in schools; raw gain, residual gain, covariance, and true scores; were compared. A simulation study showed true score analysis produced a large number of Type-I errors. When corrected for this error, this method showed the least power of the four. This outcome was clearly the result of the…

The statistical properties of two methods of estimating gain scores for groups in quasi-experiments are compared: (1) gains in scores standardized separately for each group; and (2) analysis of covariance with estimated true pretest scores. The fan spread hypothesis is assumed for groups but not necessarily assumed for members of the groups.…