NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1463704
Record Type: Journal
Publication Date: 2025-Mar
Pages: 23
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0022-0655
EISSN: EISSN-1745-3984
Available Date: 2025-02-20
The Vulnerability of AI-Based Scoring Systems to Gaming Strategies: A Case Study
Peter Baldwin1; Victoria Yaneva1; Kai North2; Le An Ha3; Yiyun Zhou1; Alex J. Mechaber1; Brian E. Clauser1
Journal of Educational Measurement, v62 n1 p172-194 2025
Recent developments in the use of large-language models have led to substantial improvements in the accuracy of content-based automated scoring of free-text responses. The reported accuracy levels suggest that automated systems could have widespread applicability in assessment. However, before they are used in operational testing, other aspects of their performance warrant examination. In this study, we explore the potential for examinees to inflate their scores by gaming the ACTA automated scoring system. We explore a range of strategies including responding with words selected from the item stem and responding with multiple answers. These responses would be easily identified as incorrect by a human rater but may result in false-positive classifications from an automated system. Our results show that the rate at which these strategies produce responses that are scored as correct varied across items and across strategies but that several vulnerabilities exist.
Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www-wiley-com.bibliotheek.ehb.be/en-us
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: 1National Board of Medical Examiners, Philadelphia, MA, United States; 2George Mason University; 3Ho Chi Minh City University of Foreign Languages and Information Technology (HUFLIT)