I. Purpose of the Study: Detecting whether interventions work or not (through main effect analysis) can provide empirical evidence regarding the causal linkage between malleable factors (e.g., interventions) and learner outcomes. In complement, moderation analyses help delineate for whom and under what conditions intervention effects are most…

Practitioners and policymakers often want estimates of the effect of an intervention for their local community, e.g., region, state, county. In the ideal, these multiple population average treatment effect (ATE) estimates will be considered in the design of a single randomized trial. Methods for sample selection for generalizing the sample ATE to…

Optimal design of multisite randomized trials leverages sampling costs to optimize sampling ratios and ultimately identify more efficient and powerful designs. Past implementations of the optimal design framework have assumed that costs of sampling units are equal across treatment conditions. In this study, we developed a more flexible optimal…

The purpose of this dissertation is to develop principles and strategies for and identify limitations of multisite cluster randomized trials in the context of partially and fully nested designs. In the first study, I develop principles of estimation, sampling variability, and inference for studies that leverage multisite designs within the context…

Optimal sampling frameworks attempt to identify the most efficient sampling plans to achieve an adequate statistical power. Although such calculations are theoretical in nature, they are critical to the judicious and wise use of funding because they serve as important starting points that guide practical discussions around sampling tradeoffs and…

Sample size determination in multilevel randomized trials (MRTs) and multilevel regression discontinuity designs (MRDDs) can be complicated due to multilevel structure, monetary restrictions, differing marginal costs per treatment and control units, and range restrictions in sample size at one or more levels. These issues have sparked a set of…

When well-implemented, mediation analyses play a critical role in probing theories of action because their results help lay the ground work for the critical development of a treatment and the iterative advancement of theories that are foundational to a discipline. Despite strong interest in designs that incorporate mediation, few studies have…

Conventional optimal design frameworks consider a narrow range of sampling cost structures that thereby constrict their capacity to identify the most powerful and efficient designs. We relax several constraints of previous optimal design frameworks by allowing for variable sampling costs in cluster-randomized trials. The proposed framework…

With many pretreatment covariates and treatment factors, the classical factorial experiment often fails to balance covariates across multiple factorial effects simultaneously. Therefore, it is intuitive to restrict the randomization of the treatment factors to satisfy certain covariate balance criteria, possibly conforming to the tiers of…

We initiated and structured a single program that supervised teachers, some with neuroscience or psychology degrees, to collaborate and explore the effects of science of learning-translated pedagogy. This article reports on the 34 findings from teacher-led randomized controlled trials (RCTs) and replications. Teachers designed trials, looking at…

Recent studies show cluster randomized trials may be well powered to detect mediation or indirect effects in multilevel settings. However, literature has rarely provided guidance on designing cluster-randomized trials aim to assess indirect effects. In this study, we developed closed-form expression to estimate the variance of and the statistical…

Multilevel experiments have been widely used in education and social sciences to evaluate causal effects of interventions. Two key considerations in designing experimental studies are statistical power and the minimal use of resources. Optimal design framework simultaneously addresses both considerations. This dissertation extends previous optimal…

Underlying all What Works Clearinghouse (WWC) products are WWC Study Review Guides, which are intended for use by WWC certified reviewers to assess studies against the WWC evidence standards. As part of an ongoing effort to increase transparency, promote collaboration, and encourage widespread use of the WWC standards, the Institute of Education…

Randomized experiments are generally considered to provide the strongest basis for causal inferences about cause and effect. Consequently randomized field trials have been increasingly used to evaluate the effects of education interventions, products, and services. Populations of interest in education are often hierarchically structured (such as…

This document provides step-by-step instructions on how to complete the Study Review Guide (SRG, Version S3, V2) for single-case designs (SCDs). Reviewers will complete an SRG for every What Works Clearinghouse (WWC) review. A completed SRG should be a reviewer's independent assessment of the study, relative to the criteria specified in the review…