Educational assessment data are often collected from a set of test centers across various geographic regions, and therefore the data samples contain clusters. Such cluster-based data may result in clustering effects in variance estimation. However, in many grouped jackknife variance estimation applications, jackknife groups are often formed by a…

This study was conducted to analyze potential item parameter drift (IPD) impact on person ability estimates and classification accuracy when drift affects an examinee subgroup. Using a series of simulations, three factors were manipulated: (a) percentage of IPD items in the CAT exam, (b) percentage of examinees affected by IPD, and (c) item pool…

In this article, we present the results of three question design experiments on inter-item correlations, which tested a grid design against a single-item design. The first and second experiments examined the inter-item correlations of a set with five and seven items, respectively, and the third experiment examined the impact of the question design…

Though panel data are increasingly used in the social sciences, the question whether repeatedly participating in a panel survey affects respondents' attitudes and (response) behaviour is still largely unsolved. Drawing on a model of associative networks that is extended by assumptions on survey satisficing, we present a theoretical framework that…

This paper serves as a practical guide to mediation design and analysis by evaluating the ability of mediation models to detect a significant mediation effect using limited data. The cross-sectional mediation model, which has been shown to be biased when the mediation is happening over time, is compared to longitudinal mediation models:…

Given the longitudinal nature of L2 grit, the use of conventional research methodologies with cross-sectional data to examine the validity of L2 grit scale seems inadequate. The present research was an attempt to extend the domain-specific phase of research on L2 grit, with the pursuit of long-term goals at its core, into a dynamic one. Thus, we…

To plan cluster-randomized trials with sufficient statistical power to detect intervention effects on student achievement, researchers need multilevel design parameters, including measures of between-classroom and between-school differences and the amounts of variance explained by covariates at the student, classroom, and school level. Previous…

Students' perceptions of the learning environment play an important role in their academic achievement and social lives. While most measures of school environment have been developed for middle- and high-school students, they also have been used for younger students, such as 4th and 5th graders. What is unclear is whether these measures are…

The linguistic complexity of many text-based tests can be a source of construct-irrelevant variance, as test-takers' performance may be affected by factors that are beyond the focus of the assessment itself, such as reading comprehension skills. This experimental study examined the extent to which the use of animated videos, as opposed to written…

Purpose: Some confounders are nonlinearly associated with dependent variables, but they are often adjusted using a linear term. The purpose of this study was to examine the error of mis-specifying the nonlinear confounding effect. Methods: We carried out a simulation study to investigate the effect of adjusting for a nonlinear confounder in the…

By aggregating residual gain scores (the differences between each student's current score and a predicted score based on prior performance) for a school or a teacher, value-added models (VAMs) can be used to generate estimates of school or teacher effects. It is known that random errors in the prior scores will introduce bias into predictions of…

In a pioneering research article, Wollack and colleagues suggested the "erasure detection index" (EDI) to detect test tampering. The EDI can be used with or without a continuity correction and is assumed to follow the standard normal distribution under the null hypothesis of no test tampering. When used without a continuity correction,…

Standard errors computed according to the operational practices of international large-scale assessment studies such as the Programme for International Student Assessment's (PISA) or the Trends in International Mathematics and Science Study (TIMSS) may be biased when cross-national differential item functioning (DIF) and item parameter drift are…

Randomized controlled trials (RCTs) admit unconfounded design-based inference--randomization largely justifies the assumptions underlying statistical effect estimates--but often have limited sample sizes. However, researchers may have access to big observational data on covariates and outcomes from RCT non-participants. For example, data from A/B…

Although a wealth of literature exists studying the effect of sponsor characteristics on self-reports of mental health, little work assesses a related but potentially powerful effect: a context comprehension effect, that is, a change in the respondent's interpretation of a survey question, given the concept elicited by the interviewer. Further,…