Psychologists are emphasizing the importance of predictive conclusions. Machine learning methods, such as supervised neural networks, have been used in psychological studies as they naturally fit prediction tasks. However, we are concerned about whether neural networks fitted with random datasets (i.e., datasets where there is no relationship…

Background: For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require each document first be manually coded for constructs of interest by trained human raters. These hand-coded scores are then used as a measured outcome for an impact analysis, with the average scores of the treatment group…

Background: Multilevel randomized controlled trials (MRCTs) have been widely used to evaluate the causal effects of educational interventions. Traditionally, educational researchers and policymakers focused on the average treatment effects (ATE) of the intervention. Recently there has been an increasing interest in evaluating the heterogeneity of…

For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by trained human raters. This process, the current standard, is both time-consuming and limiting: even the largest human coding efforts are typically constrained to…

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

Background: Randomized controlled trials (RCTs) give unbiased estimates of average effects. However, positive effects for the majority of students may mask harmful effects for smaller subgroups, and RCTs often have too small a sample to estimate these subgroup effects. In many RCTs, covariate and outcome data are drawn from a larger database. For…

This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross…

Randomized A/B tests within online learning platforms enable us to draw unbiased causal estimators. However, precise estimates of treatment effects can be challenging due to minimal participation, resulting in underpowered A/B tests. Recent advancements indicate that leveraging auxiliary information from detailed logs and employing design-based…

Objective: Our study aimed to compare the efficacy of artificial intelligence (AI)-based immersive training with human-led workshops to improve the English language skills of non-English early career researchers (NEECRs) in a Nigerian public university. Methods: Our study employed a randomized pretest/posttest control group design. A total of 124…

Although there is increasing acceptance that personalization improves learning outcomes, there is still limited experimental evidence supporting this claim. The aim of this study was to implement and evaluate the effectiveness of an adaptive recommendation system for Singapore primary and secondary education. The system leverages users trace data…