Practitioners are interested in not only the average causal effect of a treatment on the outcome but also the underlying causal mechanism in the presence of an intermediate variable between the treatment and outcome. However, in many cases we cannot randomize the intermediate variable, resulting in sample selection problems even in randomized…

In educational assessments and achievement tests, test developers and administrators commonly assume that test-takers attempt all test items with full effort and leave no blank responses with unplanned missing values. However, aberrant response behavior--such as performance decline, dropping out beyond a certain point, and skipping certain items…

This paper reviews the use of multiple trials, defined as multiple sites or multiple arms in a single evaluation and replications, in evaluating social programs. After defining key terms, the paper discusses the rationales for conducting multiple trials, which include increasing sample size to increase statistical power; identifying the most…

We designed a sequence of courses for the DataCamp online learning platform that approximates the content of a typical introductory statistics course. We discuss the design and implementation of these courses and illustrate how they can be successfully integrated into a brick-and-mortar class. We reflect on the process of creating content for…

Scholars increasingly rely on indirect questioning techniques to reduce social desirability bias and item nonresponse for sensitive survey questions. The major drawback of these approaches, however, is their inefficiency relative to direct questioning. We show how to improve the statistical analysis of the list experiment, randomized response…

Universities around the world seek to increase their students' learning abroad in forms like international co-op and exchange. The authors build on findings in a 2016 publication by Behrisch in this journal to focus on the correlation of perceived risk with students' completion of a learning abroad experience. Using binary logistic regression…

Maintaining and cultivating student engagement is critical for learning. Understanding factors affecting student engagement can help in designing better courses and improving student retention. The large number of participants in massive open online courses (MOOCs) and data collected from their interactions on the MOOC open up avenues for studying…

Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would…

At least sixteen US states have taken steps toward holding teacher preparation programs (TPPs) accountable for teacher value-added to student test scores. Yet it is unclear whether teacher quality differences between TPPs are large enough to make an accountability system worthwhile. Several statistical practices can make differences between TPPs…

Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would…

Network meta-analysis compares multiple treatments in terms of their efficacy and harm by including evidence from randomized controlled trials. Most clinical trials use parallel design, where patients are randomly allocated to different treatments and receive only 1 treatment. However, some trials use within person designs such as split-body,…

Meta-analyses often include only a small number of studies ([less than or equal to]5). Estimating between-study heterogeneity is difficult in this situation. An inaccurate estimation of heterogeneity can result in biased effect estimates and too narrow confidence intervals. The beta-binominal model has shown good statistical properties for…

Through a sensitivity analysis, the analyst attempts to determine whether a conclusion of causal inference could be easily reversed by a plausible violation of an identification assumption. Analytic conclusions that are harder to alter by such a violation are expected to add a higher value to scientific knowledge about causality. This article…

This paper is part two of the research paper entitled, "AMOVA" ["Accumulative Manifold Validation Analysis"]: An Advanced Statistical Methodology designed to measure and test the Validity, Reliability, and Overall Efficacy of Inquiry--Based Psychometric Instruments" published in the 2015 Journal of Educational Technology.…

A new visual representation for insight problems permits 22 new quantitative measures; which leads to a detailed diagnosis of a person's (or team's) creative weaknesses; which then leads to prescribing targeted, effective counter-techniques for each weakness. Currently, only two measures are consistently used for insight problem solving: the…