Context: Assessments of baseline equivalency of intervention and control groups, "balance," play a critical role in evaluating educational interventions. The highest What Works Clearinghouse (WWC) of the Institute of Educational Studies (IES) standard for educational studies, "Meets WWC Design Standards Without Reservations,"…

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies (Freedman, 2008; Lin, 2013; Imbens and Rubin, 2015; Schochet, 2013, 2016; Yang and Tsiatis, 2001). The non-parametric estimators are derived using the building blocks of experimental designs with minimal…

This paper examines the "ecological validity" of a randomized controlled trial (RCT) conducted in 42 high schools to measure the impact of a content literacy program. This study of comparable schools implementing the same program in different contexts highlights characteristics that are often not attended to in rigorous effectiveness…

The Comparative Short Interrupted Time Series (C-SITS) design is a frequently employed quasi-experimental method, in which the pre- and post-intervention changes observed in the outcome levels of a treatment group is compared with those of a comparison group where the difference between the former and the latter is attributed to the treatment. The…

A consistent finding across early childhood interventions targeting mathematics and other achievement-related skills that is that initial treatment effects fade over time, with children not receiving the intervention catching up to children who did (Clements et al., 2013; Leak et al., 2010; Puma et al., 2010). One popular explanation for this is…

The purpose of this study is to study the performance of different methods (inverse probability weighting and estimation of informative bounds) to control for differential attrition by comparing the results of different methods using two datasets: an original dataset from Portland Public Schools (PPS) subject to high rates of differential…

When randomized experiments cannot be conducted in practice, propensity score (PS) techniques for matching treated and control units are frequently used for estimating causal treatment effects from observational data. Despite the popularity of PS techniques, they are not yet well studied for matching multilevel data where selection into treatment…

The Knowledge Is Power Program (KIPP) is a national network of public charter schools whose stated mission is to help underserved students enroll in and graduate from college. As of 2014-2015 the network included 162 elementary, middle, and high schools serving 59,495 students. Questions remain, however, about the ability of charter school…

This article introduces an alternative impact parameter for group-based RCTs with student mobility--the survivor average causal effect ("SACE")--that pertains to the subpopulation of original cohort students who would remain in their baseline study schools in either the treatment or control condition. The "SACE" parameter has a clear…

The Fidelity of Implementation (FOI) study that is the focus of this report was conducted as a component of a scale-up effectiveness trial of the SRA/McGraw-Hill Open Court Reading program. The overall purpose of the FOI study was to support and provide context for findings from the larger experimental impact study of Open Court Reading (OCR). To…

Meta-analyses are syntheses of effect-size estimates obtained from a collection of studies to summarize a particular field or topic (Hedges, 1992; Lipsey & Wilson, 2001). These reviews are used to integrate knowledge that can inform both scientific inquiry and public policy, therefore it is important to ensure that the estimates of the effect…

Out-of-school factors can significantly impact students' readiness to learn and thrive in school. Research confirms that larger social structures and contexts beyond the school are critical, accounting for up to two-thirds of the variance in student achievement (Coleman et al., 1966; Rothstein, 2010; Phillips, Brooks-Gunn, Duncan, Klebanov, &…

Student achievement in literacy has been a focal concern in the United States for many years. Improving teachers' knowledge and skill that leads to improved student achievement, particularly in the early grades, can place children on an improved trajectory that can have long-term impacts on life outcomes. Over the past decade, a large body of…

North Carolina's Early College model is the subject of an IES-funded eleven-year longitudinal experimental study that utilized a lottery process to assign early college applicants to either treatment or control groups. This paper presents findings related to high school outcomes. The primary goal of the early college model is to increase the…

This study focuses on how matching, a method of preprocessing data prior to estimation and analysis, can be used to reduce imbalance between treatment and control group in regression discontinuity design. To examine the effects of academic probation on student outcomes, researchers replicate and expand upon research conducted by Lindo, Sanders,…