The phenomenon of "halo" effects in subjective grading was investigated. Two groups of three raters evaluated 20 term papers in introductory psychology. Term paper grades correlated significantly with course grades when information about previous academic performance was made available. When this information was not available, the…

Stuttering self-judgments from 15 adults who stutter, judgments of each others' stuttering, and the judgments of a panel of 10 stuttering researchers were compared. Results found substantial differences in stuttering judgments across speakers, judges, and judgment conditions, but across-task comparisons were complicated by low self-agreement among…

The Interrater/Test Reliability System (ITRS) is described. The ITRS is a comprehensive computer tool used to address questions of interrater reliability that computes several different indices of interrater reliability and the generalizability coefficient over raters and topics. The system is available in IBM compatible or Macintosh format. (SLD)

Data collected at two law enforcement agencies were used to address three specific issues concerning frame-of-reference rater training: (1) prototype-anchored rating system; (2) sensitivity and threshold analyses to identify idiosyncratic raters; and (3) areas of performance where supervisors and subordinates were likely to disagree on frame of…

An extension of the Rasch model is used to obtain objective measurements for examinations graded by judges. The model calibrates elements of each facet of the examination on a common log-linear scale. Real examination data illustrate the way correcting for judge severity improves fairness of examinee measures. (SLD)

Four experienced stuttering researchers viewed videodisks of spontaneous speech from chronic stutterers and attempted to locate the precise onset and offset of individual stuttering events. Results showed interjudge disagreements that challenge the reliability and validity of onset and offset judgments. Highly agreed stuttering events were…

In a study involving eight judges, analysis with the FACETS model provides evidence that judges grade differently, whether or not scores correlate well. This outcome suggests that adjustments for differences among judges should be made before student measures are estimated to produce reproducible decisions. (SLD)

This study investigated interparental agreement of the Eyberg Child Behavior Inventory for 44 clinic-referred families. Mothers rated their children's disruptive behavior as more frequent and more problematic than did fathers. However, strong evidence for cross-informant reliability was obtained. For maternal vs. paternal reports, classification…

Study examined the impact of method on speech ratings of classroom teachers by concurrently testing reliability and validity for the analytic and holistic methods, holding the variables of raters and speeches constant. Student speeches were videotaped and scored. Both methods produced similar and acceptable levels of reliability and concurrent…

The study calculated indices of interjudge reliability and interjudge and intrajudge agreement on ratings made by 15 experienced speech clinicians on 5 deviant speech dimensions of 15 speakers with ataxic dysarthria and a wide range of speech intelligibility. Judges were reliable in tracking imprecise consonants, excess and equal stress, and harsh…

Using data from an individually administered interview schedule (the Consumer Satisfaction Inventory), reliability among nine interviewers was estimated with several statistical methods, including simple percentages of agreement, kappa and weighted kappa, Pearson correlations, t tests on interviewers' means, and generalizability theory techniques.…

Four indexes of agreement between ratings of a person that correct for chance and are interpretable as intraclass correlation coefficients for different analysis of variance models are investigated. Relationships among the estimators are established for finite samples, and the equivalence of these estimators in large samples is demonstrated. (SLD)

The relationship between self- and peer report on the Coping Inventory for Stressful Situations (CISS) (N. Endler and J. Parker, 1990) was studied with 163 pairs of friends (college students). Positive but modest correlations were found between peer and self-report for three types of coping. Findings that attenuate these correlations are…

The Correct Information Unity analysis for measuring the communicative information and efficiency of connected speech was applied to the naturally occurring conversation of a person with moderate aphasia. Results indicated low intrarater and interrater reliability although reliability of word counts was good. Most rater disagreements resulted from…

Studied four forms of score resolution used by testing agencies and investigated the effect that each has on the interrater reliability associated with the resulting operational scores. Results, based on 120 essays from the Georgia High School Writing Test, show some forms of resolution to be associated with higher reliability and some associated…