ERIC Number: EJ1491342
Record Type: Journal
Publication Date: 2025-Dec
Pages: 24
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0022-0655
EISSN: EISSN-1745-3984
Available Date: 2025-10-31
Finding Words Associated with DIF: Predicting Differential Item Functioning Using LLMs and Explainable AI
Journal of Educational Measurement, v62 n4 p883-906 2025
We fine-tuned and compared several encoder-based Transformer large language models (LLM) to predict differential item functioning (DIF) from the item text. We then applied explainable artificial intelligence (XAI) methods to identify specific words associated with the DIF prediction. The data included 42,180 items designed for English language arts and mathematics summative state assessments among students in grades 3 to 11. Prediction R[superscript 2] ranged from 0.04 to 0.32 among eight focal and reference group pairs. Our findings suggest that many words associated with DIF reflect minor subdomains included in the test blueprint by design, rather than construct-irrelevant content that may need to be removed from assessments. This may explain why qualitative reviews of DIF items often yield inconclusive results. Our approach can be used to (1) screen words associated with DIF during the item-writing process for immediate revision to reduce preventable adverse DIF, (2) assist traditional DIF item reviews by highlighting key words, or (3) use DIF prediction as an alternative when obtaining sufficient sample size for traditional DIF analyses is impossible. Extensions of this research can enhance the assessment fairness, especially programs that lack resources to build high-quality items, and among smaller subpopulations with insufficient sample sizes for traditional DIF analyses.
Descriptors: Artificial Intelligence, Prediction, Test Bias, Test Items, Vocabulary, Language Arts, Mathematics, Summative Evaluation, Elementary Secondary Education, Test Content
Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www-wiley-com.bibliotheek.ehb.be/en-us
Publication Type: Journal Articles; Reports - Research
Education Level: Elementary Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: 1Smarter Balanced, University of California—Santa Cruz, Santa Cruz, CA, USA; 2Department of Educational Psychology, University of Minnesota—Twin Cities, Minneapolis, MN, USA

Peer reviewed
Direct link
