Background: Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs)…

Purpose: Key considerations in designing multilevel experimental studies are to efficiently use resources and to determine the sample size allocation such that designs have adequate statistical power. The utility of optimal sampling and power analysis results depends on accurate information about design parameters, such as intraclass correlations…

Current practice for conducting power analyses in hierarchical trials using survey based ICC and effect size estimates may be misestimating power because ICCs are not being adjusted to account for treatment effect heterogeneity. Results presented in Table 1 show that the necessary adjustments can be quite large or quite small. Furthermore, power…

Randomized experiments are generally considered to provide the strongest basis for causal inferences about cause and effect. Consequently randomized field trials have been increasingly used to evaluate the effects of education interventions, products, and services. Populations of interest in education are often hierarchically structured (such as…

Cluster randomized experiments are now widely used to examine intervention effects in prevention science. It is meaningful to use empirical benchmarks for interpreting effect size in prevention science. The effect size (i.e., the standardized mean difference, calculated by the difference of the means between the treatment and control groups,…

This paper presents evidence on the amount of classroom variation in gains across various domains of achievement within the Tennessee Voluntary Pre-Kindergarten (TN-VPK) program; the data are taken from the randomized control trial (RCT) design of the TN-VPK evaluation. This study explores areas where those differences are largest, and offers…

Of all the approaches to collecting data in the social sciences, the administration of questionnaires to respondents is among the most prevalent. Despite their popularity, there is broad consensus among survey design experts that using these items introduces excessive error into respondents' ratings. The authors attempt to answer the following…

A cluster-randomized trial (CRT) relies on random assignment of intact clusters to treatment conditions, such as classrooms or schools (Raudenbush & Bryk, 2002). One specific type of CRT, a multi-site CRT (MSCRT), is commonly employed in educational research and evaluation studies (Spybrook & Raudenbush, 2009; Spybrook, 2014; Bloom,…

The current study examines classroom-level peer behavioral regulation skills and their implications for children's school readiness outcomes. Specifically, this study will answer the following research questions: (1) Is the average level of peers' behavioral regulation skills in a classroom in the fall associated with growth in children's school…

As part of the 2010 economic stimulus, a $55 million "Investing in Innovation" (i3) grant from the US Department of Education was awarded to scale up Reading Recovery across the nation. This paper presents the final round of results from the large-scale, mixed methods randomized evaluation of the implementation and impacts of Reading…

The capacity of the field to conduct power analyses for group randomized trials (GRTs) of educational interventions has improved over the past decade (Authors, 2009). However, a power analysis depends on estimates of design parameters. Hence it is critical to build the empirical base of design parameters for GRTs across a variety of outcomes and…

As a result of the 1998 reauthorization of Head Start, the Department of Health and Human Services conducted a national evaluation of the Head Start program. The goal of Head Start is to improve the school readiness skills of low-income children in the United States. There is a substantial body of experimental and correlational research that has…

The gap among ethnicities and gender in mathematics achievement is a well-known problem. While the gap has been shrinking over the past three decades, it has not completely diminished (Jencks & Phillips, 1998; McGraw, Lubienski, & Strutchens, 2006). The ALEKS, Assessment and LEarning in Knowledge Spaces, tutoring system is one promising…

Collaborative Strategic Reading (CSR) is a multicomponent reading intervention aimed at improving students' text comprehension. Two 1-year randomized controlled trials were conducted to determine the efficacy of CSR with seventh and eighth grade students. The Year 2 replication study was identical to the original Year 1 study except that the…

Head Start is the largest publicly funded preschool program in the U.S., and one of its primary goals is to improve the school readiness of low-income children. As has been widely reported, the first randomized trial of Head Start in the program's history found some evidence that it is achieving this goal. Receiving one year of Head Start had…