Most randomized field experiments experience some attrition. Moreover, the extent of attrition may differ by treatment condition in systematic, non-random ways, biasing estimates of treatment effects and contributing to invalid inferences. We address concerns about non-random attrition by quantifying the conditions necessary in the attritted data…

Multisite trials that randomize individuals (e.g., students) within sites (e.g., schools) or clusters (e.g., teachers/classrooms) within sites (e.g., schools) are commonly used for program evaluation because they provide opportunities to learn about treatment effects as well as their heterogeneity across sites and subgroups (defined by moderating…

Combining observational and experimental data for causal inference can improve treatment effect estimation. However, many observational data sets cannot be released due to data privacy considerations, so one researcher may not have access to both experimental and observational data. Nonetheless, a small amount of risk of disclosing sensitive…

Background: Despite many adolescents experiencing mental health concerns, a substantial portion lack access to evidence-based treatments (EBTs) for psychopathology; this issue is magnified for adolescents belonging to communities considered marginalized. One way to ameliorate this is by adapting existent EBTs--typically delivered in research…

For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by trained human raters. This process, the current standard, is both time-consuming and limiting: even the largest human coding efforts are typically constrained to…

Background: Schools need to implement universal student supports that prevent social, emotional, and behavioral difficulties; minimize associated risks; and promote social, emotional, and behavioral competencies. The purpose of this study is to examine the efficacy of the Helping Educational Leaders Mobilize Evidence (HELM) implementation strategy…

In cluster randomized evaluations, a treatment or intervention is randomly assigned to a set of clusters each with constituent individual units of observations (e.g., student units that attend schools, which are assigned to treatment). One consideration of these designs is how many units are needed per cluster to achieve adequate statistical…

Randomized controlled trials (RCTs) are an increasingly common research design for evaluating the effectiveness of community college (CC) interventions. However, when planning an RCT evaluation of a CC intervention, there is limited empirical information about what sized effects an intervention might reasonably achieve, which can lead to under- or…

As reporting of effect sizes in evaluation studies has proliferated, researchers and consumers of research need tools for interpreting or benchmarking the magnitude of those effect sizes that are relevant to the intervention, target population, and outcome measure being considered. Similarly, researchers planning education studies with social and…

The last two decades have seen a dramatic increase in randomized controlled trials (RCTs) conducted in community colleges. Yet, there is limited empirical information on the design parameters necessary to plan the sample size for RCTs in this context. We provide empirical estimates of key design parameters, discussing lessons based on the pattern…

Noncompliance to treatment assignment is widespread in randomized trials and presents challenges in causal inference. In the presence of noncompliance, the most commonly estimated effect of treatment assignment, also known as the intent-to-treat (ITT) effect, is biased. Of interest in this setting is the complier average causal effect (CACE), the…

Emergent technologies present platforms for educational researchers to conduct randomized controlled trials (RCTs) and collect rich data on study students' performance, behavior, learning processes, and outcomes in authentic learning environments. As educational research increasingly uses methods and data collection from such platforms, it is…

Online credit recovery will likely expand in the coming years as school districts try to address increased course failure rates brought on by the coronavirus pandemic. Some researchers and policymakers, however, raise concerns over how much students learn in online courses, and there is limited evidence about the effectiveness of online credit…

Retention of early career teachers is a critical issue in education, with burnout and self-efficacy serving as important precursors to teachers leaving the field. An integration of the PAX Good Behavior Game (GBG; Barrish et al., 1969) and MyTeachingPartner (MTP; Allen et al., 2015) was tested in a randomized controlled trial (RCT) to investigate…

In the complex world of educational policy, causal inferences will be debated. As we review non-experimental designs in educational policy, we focus on how to clarify and focus the terms of debate. We begin by presenting the potential outcomes/counterfactual framework and then describe approximations to the counterfactual generated from the…