When randomized control trials are not possible, quasi-experimental methods often represent the gold standard. One quasi-experimental method is difference-in-difference (DiD), which compares changes in outcomes before and after treatment across groups to estimate a causal effect. DiD researchers often use fairly exhaustive robustness checks to…

This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross…

The aim of this study was to investigate and empirically derive parameters commonly used for statistical power and sample size calculations to better inform future trial design. Towards achieving this aim, the research project leveraged the richness of the National Pupil Database (NPD) and the Education Endowment Foundation (EEF) Archive to: (1)…

We present an aggregation scheme that increases power in randomized controlled trials and quasi-experiments when the intervention possesses a robust and well-articulated theory of change. Intervention studies using longitudinal data often include multiple observations on individuals, some of which may be more likely to manifest a treatment effect…

Applied researchers are increasingly interested in whether and how treatment effects vary in randomized evaluations, especially variation not explained by observed covariates. We propose a model-free approach for testing for the presence of such unexplained variation. To use this randomization-based approach, we must address the fact that the…

This article reports on a Monte Carlo simulation study, evaluating two approaches for testing the intervention effect in replicated randomized AB designs: two-level hierarchical linear modeling (HLM) and using the additive method to combine randomization test "p" values (RTcombiP). Four factors were manipulated: mean intervention effect,…

Testing has been found to facilitate students' long-term retention of information. However, the learning performance of highly test-anxious students can be impaired by tests. Thus, these students may learn ineffectively in a testing context. By contrast, summary writing may not trigger test anxiety and is therefore another learning strategy to…

More than 100 million people globally are estimated to be exposed to arsenic in drinking water that exceeds the World Health Organization guideline of 10 µg/L. In an effort to develop and test a low-cost sustainable approach for water arsenic testing in Bangladesh, we conducted a randomized controlled trial which found arsenic educational…

Item response time data were used in investigating the differences in student test-taking behavior between two device conditions: computer and tablet. Analyses were conducted to address the questions of whether or not the device condition had a differential impact on rapid guessing and solution behaviors (with response time effort used as an…

The objective of the present study was to evaluate the extent to which students who took a computer adaptive test of reading comprehension accounting for testlet effects were administered fewer passages and had a more precise estimate of their reading comprehension ability compared to students in the control condition. A randomized controlled…