Psychologists are emphasizing the importance of predictive conclusions. Machine learning methods, such as supervised neural networks, have been used in psychological studies as they naturally fit prediction tasks. However, we are concerned about whether neural networks fitted with random datasets (i.e., datasets where there is no relationship…

Background: For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require each document first be manually coded for constructs of interest by trained human raters. These hand-coded scores are then used as a measured outcome for an impact analysis, with the average scores of the treatment group…

Background: Multilevel randomized controlled trials (MRCTs) have been widely used to evaluate the causal effects of educational interventions. Traditionally, educational researchers and policymakers focused on the average treatment effects (ATE) of the intervention. Recently there has been an increasing interest in evaluating the heterogeneity of…

For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by trained human raters. This process, the current standard, is both time-consuming and limiting: even the largest human coding efforts are typically constrained to…

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

Background: Randomized controlled trials (RCTs) give unbiased estimates of average effects. However, positive effects for the majority of students may mask harmful effects for smaller subgroups, and RCTs often have too small a sample to estimate these subgroup effects. In many RCTs, covariate and outcome data are drawn from a larger database. For…

This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross…

Randomized A/B tests within online learning platforms enable us to draw unbiased causal estimators. However, precise estimates of treatment effects can be challenging due to minimal participation, resulting in underpowered A/B tests. Recent advancements indicate that leveraging auxiliary information from detailed logs and employing design-based…

We perform a large-scale randomized controlled trial to evaluate the potential of machine learning-based instruction sequencing to improve memorization while allowing the learners the freedom to choose their review times. After controlling for the length and frequency of study, we find that learners for whom a machine learning algorithm determines…

Objective: Our study aimed to compare the efficacy of artificial intelligence (AI)-based immersive training with human-led workshops to improve the English language skills of non-English early career researchers (NEECRs) in a Nigerian public university. Methods: Our study employed a randomized pretest/posttest control group design. A total of 124…

Although there is increasing acceptance that personalization improves learning outcomes, there is still limited experimental evidence supporting this claim. The aim of this study was to implement and evaluate the effectiveness of an adaptive recommendation system for Singapore primary and secondary education. The system leverages users trace data…

Researchers often include covariates when they analyze the results of randomized controlled trials (RCTs), valuing the increased precision of the estimates over the potential of inducing small-sample bias when doing so. In this paper, we develop a sufficient condition which ensures that the inclusion of covariates does not cause small-sample bias…

Researchers often include covariates when they analyze the results of randomized controlled trials (RCTs), valuing the increased precision of the estimates over the potential of inducing small-sample bias when doing so. In this paper, we develop a sufficient condition which ensures that the inclusion of covariates does not induce small-sample bias…

Randomized A/B tests in educational software are not run in a vacuum: often, reams of historical data are available alongside the data from a randomized trial. This paper proposes a method to use this historical data--often highdimensional and longitudinal--to improve causal estimates from A/B tests. The method proceeds in two steps: first, fit a…

The concept of precision medicine aims to provide additional context to patient data for healthcare providers. Precision medicine overlays three additional layers of patient data on top of standard patient information: environmental exposure, personal lifestyle and behavior patterns, and information gleaned from their genome. While precision…