NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
ERIC Number: ED306254
Record Type: Non-Journal
Publication Date: 1989
Pages: 26
Abstractor: N/A
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Rater Stringency Error in Performance Rating: A Contrast of Three Models.
Cason, Gerald J.; Cason, Carolyn L.
The use of three remedies for errors in the measurement of ability that arise from differences in rater stringency is discussed. Models contrasted are: (1) Conventional; (2) Handicap; and (3) deterministic Rater Response Theory (RRT). General model requirements, power, bias of measures, computing cost, and complexity are contrasted. Contrasts are illustrated by application of the models to small and large sets of clinical performance ratings of junior-year medical students in an internal medicine clerkship in 1982. The small sample consisted of the ratings for a cohort of 24 students by 42 raters; 129 ratings were used. The large set included ratings for all of the students of that year; 744 ratings by 93 raters on 163 students were used. The RRT was the most powerful model, with Handicap a close second, although biased. The RRT obtained any specified interrater reliability with only a third of the independent ratings needed by the Conventional model. Pass/fail decisions are illustrated for the 163 students and a hypothetical individual student whose fate is shown to reflect his ability, the measurement model, the passing score, and the stringency and number of raters. The choice of measurement procedures should reflect balance in improving the accuracy of the measures and the costs of faulty decisions based on the measures. Sixteen tables illustrate the data. (SLD)
Publication Type: Reports - Evaluative
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A