ERIC Number: EJ1464974
Record Type: Journal
Publication Date: 2025-Dec
Pages: 27
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-2196-0739
Available Date: 2025-03-28
Reassessing Weights in Large-Scale Assessments and Multilevel Models
Umut Atasever1,2; Francis L. Huang3; Leslie Rutkowski4
Large-scale Assessments in Education, v13 Article 9 2025
When analyzing large-scale assessments (LSAs) that use complex sampling designs, it is important to account for probability sampling using weights. However, the use of these weights in multilevel models has been widely debated, particularly regarding their application at different levels of the model. Yet, no consensus has been reached on the best method to apply weights. To address this, we conducted a Monte Carlo simulation, modeling a typical LSA population with known true values for the variables of interest. Using repeated sampling from this population, we generated weights using a stratified two-stage cluster design, where clusters (schools) were selected using probability proportional to size (PPS) sampling from designated explicit strata. We examined both class-level and student-level sampling structures and applied a nonresponse model at both the school and student levels. For each sample drawn, we assessed bias and coverage rates across models that applied weights at two levels, only at level 2, only at level 1, and without weights. Our findings show that applying only level-2 weights produced the most precise estimates, while models with no weights or only rescaled level-1 weights led to the highest bias. Using both level-1 and level-2 weights together was acceptable, although variance components were slightly underestimated. However, scaling level-1 weights would mirror using only the level-2 weights in datasets where there is no variation of weights within clusters. An applied example using TIMSS data supports these findings. This study contributes to the literature by explaining the least biased weight methods with complex sampling scenarios and offering practical guidance on using weights in multilevel models. We provide the R syntax for both the simulation and the applied example for reproducibility.
Descriptors: Mathematics Tests, International Assessment, Elementary Secondary Education, Foreign Countries, Mathematics Achievement, Science Tests, Science Achievement, Achievement Tests, Secondary School Students, Sampling, Probability, Statistical Bias, Scaling, Hierarchical Linear Modeling, Monte Carlo Methods
Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link-springer-com.bibliotheek.ehb.be/
Publication Type: Journal Articles; Reports - Research
Education Level: Elementary Secondary Education; Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Trends in International Mathematics and Science Study; Program for International Student Assessment
Grant or Contract Numbers: N/A
Author Affiliations: 1International Association for the Evaluation of Educational Achievement, Hamburg, Germany; 2TU Dortmund University, Dortmund, Germany; 3University of Missouri, Columbia, USA; 4Indiana University Bloomington, Bloomington, USA