One common score reported from diagnostic classification assessments is the vector of posterior means of the skill mastery indicators. As with any assessment, it is important to derive and report estimates of the reliability of the reported scores. After reviewing a reliability measure suggested by Templin and Bradshaw, this article suggests three…

Meta-analyses of clinical trials typically focus on one outcome at a time. However, treatment decision-making depends on an overall assessment of outcomes balancing benefit in various domains and potential risks. This calls for meta-analysis methods for combined outcomes that encompass information from different domains. When individual patient…

Correlation analysis has been used widely by researchers and analysts when analyzing large-scale assessment data. Limit research provided reliable methods to estimate various correlations and their standard errors with the complex sampling design and multiple plausible values taken into account. This report introduces the methodology used by the…

Longitudinal data analysis has received widespread interest throughout educational, behavioral, and social science research, with latent growth curve modeling currently being one of the most popular methods of analysis. Despite the popularity of latent growth curve modeling, limited attention has been directed toward understanding the issues of…

In the lead article, Davenport, Davison, Liou, & Love demonstrate the relationship among homogeneity, internal consistency, and coefficient alpha, and also distinguish among them. These distinctions are important because too often coefficient alpha--a reliability coefficient--is interpreted as an index of homogeneity or internal consistency.…

Two questions about certainty in mathematics are asked. First, is mathematical knowledge known with certainty? Second, why is the belief in the certainty of mathematical knowledge so widespread and where does it come from? This question is little addressed in the literature. In explaining the reasons for these beliefs, both cultural-historical and…

As a global measure of precision, item response theory (IRT) estimated reliability is derived for four coefficients (Cronbach's a, Feldt-Raju, stratified a, and marginal reliability). Models with different underlying assumptions concerning test-part similarity are discussed. A detailed computational example is presented for the targeted…

Outcome studies in biomedical research usually focus on testing mean changes across samples of subjects and, in so doing, often obscure changes in individuals. These changes, however, may be very informative in studies in which large or homogeneous samples are unavailable and mechanisms of action are still under scrutiny, as is often the case for…

Previous assessments of the reliability of test scores for testlet-composed tests have indicated that item-based estimation methods overestimate reliability. This study was designed to address issues related to the extent to which item-based estimation methods overestimate the reliability of test scores composed of testlets and to compare several…

The focus of this paper is assessing the impact of measurement errors on the prediction error of an observed-score regression. Measures are presented and described for decomposing the linear regression's prediction error variance into parts attributable to the true score variance and the error variances of the dependent variable and the predictor…

The problem of factor score indeterminacy implies that the factor and the error scores cannot be completely disentangled in the factor model. It is therefore proposed to compute Harman's factor score predictor that contains an additive combination of factor and error variance. This additive combination is discussed in the framework of classical…

The conventional fixed-effects (FE) and random-effects (RE) confidence intervals that are used to assess the average alpha reliability across multiple studies have serious limitations. The FE method, which is based on a constant coefficient model, assumes equal reliability coefficients across studies and breaks down under minor violations of this…

In this study, the authors aimed to examine 8 of the different methods for computing confidence intervals around alpha that have been proposed to determine which of these, if any, is the most accurate and precise. Monte Carlo methods were used to simulate samples under known and controlled population conditions wherein the underlying item…

The maximal split-half coefficient is computed by calculating all possible split-half reliability estimates for a scale and then choosing the maximal value as the reliability estimate. Osburn compared the maximal split-half coefficient with 10 other internal consistency estimates of reliability and concluded that it yielded the most consistently…

Two Monte Carlo simulations were performed to compare methods for estimating and testing hypotheses of quadratic effects in latent variable regression models. The methods considered in the current study were (a) a 2-stage moderated regression approach using latent variable scores, (b) an unconstrained product indicator approach, (c) a latent…