An approach is presented that can predict internal consistency of cognitively complex assessments on two dimensions, those of adding tasks with similar or different solution strategies and adding test takers with different solution strategies. Data from the 1992 National Assessment of Educational Progress mathematics assessment are used to…

Describes a Statistical Analysis System (SAS) MACRO for computing various indices of interrater agreement, including a new generalizability coefficient, for categorical data in a single-facet, crossed design. (Author/SLD)

Proposes true score models that can account for correlated errors and their effect on coefficient alpha. These models allow random measurement errors on earlier items to affect directly or indirectly the scores on later items. Conditions under which coefficient alpha may yield spuriously high estimates or reliability are discussed. (SLD)

Standard-setting techniques concerning testing, comparisons of these techniques, and methods for assessing interrater and intrarater reliability are described. These techniques are discussed in relation to counselor education programs. (Author/MKA)

Reviews issues, raised in part because of "Educational and Psychological Measurement" (EPM) policies, regarding "test reliability," which is psychometric terminology, and "score reliability," score-centric terminology. Discusses datametrics and provides a critique of T. Vacha-Haase's proposed meta-analytic reliability generalization via…

Responds to criticisms of some "Educational and Psychological Measurement" policies and the reliability generalization meta-analytic methods of T. Vacha-Haase. Explores consequences of misunderstanding score reliability. (SLD)

B. Thompson and T. Vacha-Haase have examined the statement "the reliability of the test" with emphasis on the following three words: (1) the first "the"; (2) "test"; and (3) the second "the." This discussion focuses instead on the word "reliability." (Author)

Discusses the fundamental qualities that must be used to judge the merit of performance standards: validity, reliability, and genuineness. The Rasch-based objective model has demonstrated these qualities, but other models should also be evaluated for these characteristics. (SLD)

Presents frequently encountered measurement misconceptions and various measurement "rules." Origins of the misconceptions and rules are described, along with the reasons why they are problematic. Alternate approaches or considerations are given. Misconceptions discussed pertain to the estimation of internal consistency reliability and item…

An initial evaluation of the psychometric properties of the Central Relationship Questionnaire (CRQ) is presented. Central relationship patterns refer to people's characteristic ways of relating to others. Results indicate that the CRQ components could be differentiated into meaningful subscales. Validity, reliability, and specific suggestions for…

Provides a general mathematical framework is provided that can be specialized to four different reliability coefficients. Consideration of this general framework makes it easier to convey to students the individual character of the formulations of reliability and the extent of their underlying similarity. (SLD)

This paper explains how different factors affect classical reliability estimates, such as test-retest, interrater, internal consistency, and equivalent forms coefficients. The limitations of classical test theory are explored, and the advantages of generalizability theory are discussed. Concrete examples are used. (SLD)

Computer simulation indicated that reliability increased with the degree of similarity between underlying and observed distributions when the observed categorical distribution was deliberately constructed to match the shape of the underlying distribution of the trait being measured. Reliability also increased with correlation among variables and…

Examined the frequency of use of various types of reliability coefficients for a sample of 696 tests in a directory published by the American Psychological Association. Coefficient alpha was the overwhelming favorite, and several measures treated almost universally in textbooks were rarely or never used. Identifies problems encountered in the…

Evaluates a training Web site for professional development of health and safety enforcement officers in the United Kingdom and reviews the use of knowledge elicitation in work-based training in accident investigations. Results showed that the training was realistic and that the imbedded discussion facility had the potential to improve enforcement…