NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1359319
Record Type: Journal
Publication Date: 2022
Pages: 16
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1536-6367
EISSN: EISSN-1536-6359
Available Date: N/A
Rater Connections and the Detection of Bias in Performance Assessment
Measurement: Interdisciplinary Research and Perspectives, v20 n2 p91-106 2022
In many performance assessments, one or two raters from the complete rater pool scores each performance, resulting in a sparse rating design, where there are limited observations of each rater relative to the complete sample of students. Although sparse rating designs can be constructed to facilitate estimation of student achievement, the relatively limited observations of each rater can pose challenges for identifying raters who may exhibit scoring idiosyncrasies specific to individual or subgroups of examinees, such as differential rater functioning (DRF; i.e., rater bias). In particular, when raters who exhibit DRF are directly connected to other raters who exhibit the same type of DRF, there is limited information with which to detect this effect. On the other hand, if raters who exhibit DRF are connected to raters who do not exhibit DRF, this effect may be more readily detected. In this study, a simulation is used to systematically examine the degree to which the nature of connections among raters who exhibit common DRF patterns in sparse rating designs impacts the sensitivity of DRF indices. The use of additional "monitoring ratings" and variable rater assignment to student performances are considered as strategies to improve DRF detection in sparse designs. The results indicate that the nature of connections among DRF raters has a substantial impact on the sensitivity of DRF indices, and that monitoring ratings and variable rater assignment to student performances can improve DRF detection.
Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A