Used reliability generalization to explore the variance of scores on 10 Minnesota Multiphasic Personality Inventory (MMPI) clinical scales drawing on 1,972 articles in the literature on the MMPI. Results highlight the premise that scores, not tests, are reliable or unreliable, and they show that study characteristics do influence scores on the…

The rater agreement literature is complicated by the fact that it must accommodate at least two different properties of rating data: the number of raters (two versus more than two) and the rating scale level (nominal versus metric). While kappa statistics are most widely used for nominal scales, intraclass correlation coefficients have been…

In the present study, the authors investigated the initial development of the Early Expository Comprehension Assessment (EECA) by examining its reliability. The EECA consists of a compare/contrast passage, manipulatives to represent the information in the paragraph, and three response tasks ("Retelling, Mapping, and Comparing"). The authors…

Sample size recommendations in confirmatory factor analysis (CFA) have recently shifted away from observations per variable or per parameter toward consideration of model quality. Extending research by Marsh, Hau, Balla, and Grayson (1998), simulations were conducted to determine the extent to which CFA model convergence and parameter estimation…

A sample of 917 children, aged 7 to 17 years, their parents, and their teachers each completed the appropriate version of the Strengths and Difficulties Questionnaire (SDQ), and 120 from each group did so again 2 weeks later. The results indicate that the SDQ demonstrates sound interinformant and test-retest reliability. Younger children, whose…

Marking reliability is purported to be produced by having an effective community of practice. No experimental research has been identified which attempts to verify empirically the aspects of a community of practice that have been observed to produce marking reliability. This research outlines what that community of practice might entail and…

The commentaries include a wealth of insightful and interesting observations and suggestions, and I appreciate each author taking the time to comment on my efforts. In responding to their suggestions, I am inclined to develop a few general points raised in the commentaries a bit further.

Two experiments examined whether listening to multiple presentations of recorded speech stimuli influences the reliability and accuracy of judgments of children's speech production accuracy. In Experiment 1, 10 listeners phonetically transcribed words produced by children with phonological impairments after a single presentation and after the word…

This investigation assesses interobserver agreement on conversation analytic (CA) transcription. Four professional CA transcribers spent a maximum of 3 hours transcribing 2.5 minutes of a previously unknown, naturally occurring, mundane telephone call. Researchers unitized transcripts into words, sounds, silences, inbreaths, outbreaths, and laugh…

Using survey results compiled from an extensive study into the ethical decision making of school principals, this article analyses the nature and consistency of principals' ethical decision-making processes. Based on the findings, the article argues that even though principals on the whole have well-meaning intentions, by and large, they do not…

This article offers an alternative methodology for practitioners and researchers to use in establishing interrater reliability for testing purposes. The majority of studies on interrater reliability use a traditional methodology where by two raters are compared using a Pearson product-moment correlation. This traditional method of estimating…

Because nominal-scale judgments cannot directly be aggregated into meaningful composites, the addition of a second rater is usually motivated by a desire to estimate the quality of a single rater's classifications rather than to improve reliability. When raters agree, the aggregation problem does not arise. Nevertheless, a proportion of this…

Purpose: We designed this study to examine the reliability of the nursing home survey process in the state of Kansas using regular and simultaneous survey teams. In particular, the study examined how two survey teams exposed to the same information at the same time differed in their interpretations. Design and Methods: The protocol for…

Though group interventions are widely used in community-based and clinical settings, there are few brief instruments for assessing the group environment. Two studies on the development of a brief measure to assess intervention group environments are described, and psychometric properties of the new scale are presented. The new measure is based on…

One of the factors that have contributed to the explosive popularity of the World Wide Web (WWW) is the ease with which almost anyone can become a web publisher. It has been estimated that 30 million new websites are created every day on the WWW, with a vast majority of these being health related. It is further asserted that there is a healthy…