This study presents several multivariate Generalizability theory designs for analyzing automatic item-generated (AIG) based test forms. The study used real data to illustrate the analysis procedure and discuss practical considerations. We collected the data from two groups of students, each group receiving a different form generated by AIG. A…

Random Equating (RE) and Heuristic Approach (HA) are two linking procedures that may be used to compare the scores of individuals in two tests that measure the same latent trait, in conditions where there are no common items or individuals. In this study, RE--that may only be used when the individuals taking the two tests come from the same…

Clustered data structures are common in many areas of educational and psychological research (e.g., students clustered in schools, patients clustered by clinician). In the course of conducting research, questions are often administered to obtain scores reflecting latent constructs. Multilevel measurement models (MLMMs) allow for modeling…

Measuring opportunities to learn (OTL) is crucial for evaluating education quality and equity, but obtaining accurate and comprehensive OTL data at a large scale remains challenging. We attempt to address this issue by investigating measurement concerns in data collection and sampling. With the primary goal of estimating group-level OTLs for large…

Data visualization is a core tenet of communicating measurement research and outcomes. Measurement professionals utilize data visualization in various phases of research, including exploration and communication. However, data visualization has not received enough attention in the measurement field. While it is true that many measurement graphics…

The COVID-19 pandemic negatively affected the quality of data from educational testing programs. These data were previously used for many important purposes ranging from placing students in instructional programs to school accountability. In this article, we draw from the research design literature to point out the limitations inherent in…

In this paper, we describe two empirical studies that demonstrate the application and modeling of keystroke logs in writing assessments. We illustrate two different approaches of modeling differences in writing processes: analysis of mean differences in handcrafted theory-driven features and use of large language models to identify stable personal…

Automatic item generation may supply many items instantly and efficiently to assessment and learning environments. Yet, the evaluation of item quality persists to be a bottleneck for deploying generated items in learning and assessment settings. In this study, we investigated the utility of using large-language models, specifically Llama 3-8B, for…

Country differences in response styles (RS) may jeopardize cross-country comparability of Likert-type scales. When adjusting for rather than investigating RS is the primary goal, it seems advantageous to impose minimal assumptions on RS structures and leverage information from multiple scales for RS measurement. Using PISA 2015 background…

Technical documentation for educational tests focuses primarily on properties of individual scores at single points in time. Reliability, standard errors of measurement, item parameter estimates, fit statistics, and linking constants are standard technical features that external stakeholders use to evaluate items and individual scale scores.…

Many educational summative and formative assessments have been transferred to a remote online setting because of the pandemic. Educational professionals and stakeholders have shown interest in learning how this change in the test mode influenced test takers; that is, whether test-taking experiences in a remote test setting were different from…

I study student characteristics and academic performance at the University of California, where consideration of an applicant's ethnicity has been banned since 1996 and SAT scores were used in admitting students to the university until fall 2021. I show the following: (1) SAT scores were more important than high school grades in predicting…