A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower "proof-of-concept" evidence of whether the…

In multisite trials, learning about treatment effect variation across sites is critical for understanding where and for whom a program works. Unadjusted comparisons, however, capture "compositional" differences in the distributions of unit-level features as well as "contextual" differences in site-level features, including…

New Hampshire's Performance Assessment of Competency Education (PACE) innovative assessment system uses student scores from classroom performance assessments as well as other classroom tests for school accountability purposes. One concern is that not having annual state testing may incentivize schools and teachers away from teaching the breadth of…

School discipline disproportionality has long been documented in educational research, primarily impacting Black/African American and non-White Hispanic/Latinx students. In response, federal policymakers have encouraged educators to change their disciplinary practice, emphasizing that more proactive support is critical to promoting students'…

This tutorial uses publicly available data from drug makers and the Food and Drug Administration to guide learners to estimate the confidence intervals of COVID-19 vaccine efficacy rates with a Bayesian framework. Under the classical approach, there is no probability associated with a parameter, and the meaning of confidence intervals can be…

Policymakers and institutional leaders in higher education too often make decisions based on descriptive data analyses or even anecdote when better analysis options could produce more nuanced and more valuable results. Employing the setting of higher education program evaluation at a midwestern regional public university, for this study we…

While doctoral leadership programs have widely adopted Improvement Science (IS) as a signature pedagogy, few studies have examined how to best equip doctoral students with the knowledge and research skills they need to utilize IS in practice. More specifically, research is needed to determine the most effective and meaningful pedagogy for…

This paper reviews the use of multiple trials, defined as multiple sites or multiple arms in a single evaluation and replications, in evaluating social programs. After defining key terms, the paper discusses the rationales for conducting multiple trials, which include increasing sample size to increase statistical power; identifying the most…

Evidence-based health interventions are frequently translated into real-world settings where practical needs drive changes to intervention protocols. Due to logistical and resource constraints, these naturally arising adaptations are rarely assessed for comparative effectiveness using a randomized trial. Nevertheless, when observational data are…

Objective: After the discontinuation of School Health Policies and Practices Study (SHPPS) in 2016, the US Centers for Disease Control and Prevention (CDC) began exploring innovative ways to gather school health information using existing surveillance systems. School Health Profiles (Profiles) is a school-based system of surveys that monitors…

Evaluators are often called upon to assess the effects of programs. To assess a program effect, evaluators need a clear understanding of how a program effect is defined. Arguably, the most widely used definition of a program effect is the counterfactual one. According to the counterfactual definition, a program effect is the difference between…

The present simulation study examined the statistical properties (namely, Type I error and statistical power) of various novel randomized single-case multiple-baseline designs and associated randomized-test analyses for comparing the A- to B-phase immediate abrupt outcome changes in two independent intervention conditions. It was found that with…

As the use of data in courses that incorporate statistical methods has become more prevalent, so has the need for tools for working with such data, including those for data creation and adjustment. While numerous tools exist that support faculty who teach statistical methods, many are focused on data analysis or theoretical concepts, and there…

This article describes a modified importance-performance framework for use in evaluation of recreation-based experiential learning programs. Importance-performance analysis (IPA) provides an effective and readily applicable means of evaluating many programs, but the near universal satisfaction associated with recreation inhibits the use of IPA in…

Using research focused on best practices, focus group information, and data analytics, the Title V: Focus on Student Success (FOSS) Grant created a model for the development, implementation, and evaluation of a summer bridge program. Results included increased academic performance indicators in first-year Hispanic college students. Validation for…