NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED677747
Record Type: Non-Journal
Publication Date: 2025-Oct-9
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
A Moment Estimator for the Survivor Average Treatment Effect, with Applications for RCTs with Attrition
Adam C. Sales; Lora Dufresne; Anzhe Tao; Sean Sullivan
Society for Research on Educational Effectiveness
Background: One of the most vexing threats to education field trials is attrition--when some subjects drop out before it is complete. Since outcomes are not available for subjects who attrit, they are typically dropped from any analysis estimating average effects on the outcome. However, since attrition may itself have been affected by treatment assignment, dropping attrittors from the analysis sample may introduce confounding or collider bias--while treatment is independent of baseline covariates in the randomization sample, it may not be independent in the analysis sample. One promising approach to estimating effects in the presence of attrition is based on principal stratification [Zhang and Rubin, 2003, PS; ]. In PS, the goal is to estimate the "survivor average treatment effect" (SATE), the average effect for the (unknown) subset of subjects who would stay in the trial regardless of their treatment assignment; within this subset, which is established (though unknown) at baseline, treatment assignment remains ignorable. However, most available PS methods require either assumptions about the adequacy of available covariates or about the adequacy of parametric models, along with other strong assumptions. Purpose: This talk will present and illustrate a new moment-based estimator for the SATE in RCTs with attrition. The estimator builds on the identification results of Ding et al. [2011] and the PS estimator of Sales et al. [2022]. We will present results from a small preliminary simulation study, and illustrate its use in a new analysis of a randomized trial with roughly 50% attrition. Data: Decker-Woodrow et al. [Forthcoming] described a large-scale field trial contrasting four computer-based prealgebra programs that featured individual-level randomization. The RCT took place during the COVID-19 pandemic, when many students attended school remotely, leading to very high attrition rates: although 3,591 students were randomized between the four conditions, only 1,850 ([approximately equal to] 51.5%) took the posttest. Moreover, there is some evidence that attrition was higher for one of the conditions, a proprietary computer application that some students struggled to install on their computer, leading to dropout. Methodology: In a randomized experiment with two conditions, let Z[subscript i] = 1 if subject i, i = 1, . . . , n is assigned to the treatment condition, and let Z[subscript i] = 0 otherwise. If interest is in the effect of Z on an outcome Y , under the "stable unit treatment value assumption" [Angrist et al., 1996] of no interference between units and no hidden versions of the treatment, define potential outcomes [Splawa-Neyman et al., 1990] Y[subscript Ti] as the value of Y that i would exhibit if Z[subscript i] = 1, and let Y[subscript Ci] be the value if Z[subscript i] = 0. Then the treatment effect of Z on Y for subject i may be defined as [tau subscript i] = Y[subscript Ti] - Y[subscript Ci]. Let x[subscript i] be a vector of covariates for subject i, and let D[subscript i] = 1 if student i dropped out of the study before taking the posttest, with D[subscript i] = 0 otherwise. Since D may also be affected by treatment assignment, let D[subscript i] = Z[subscript i]D[subscript Ti] + (1 - Z[subscript i])D[subscript Ci], where D[subscript Ti] and D[subscript Ci] are indicators of whether i would attrit if assigned to treatment or if assigned to control, respectively. Then the goal of estimation is the local average treatment effect [tau subscript NA] = E[Y[subscript T] - Y[subscript C][vertical bar]D[subscript T] = D[subscript C] = 0], the average effect for never attritors, NA, often called the "survivor average treatment effect" or SATE. Since D[subscript T] and D[subscript C] are never observed simultaneously, the usual approaches to estimating subgroup treatment effects do not apply, and additional assumptions are necessary. First, assume "monotonicity" [Angrist et al., 1996, e.g.], that D[subscript T] [greater than or equal to] D[subscript C]; in other words, there are no subjects who would drop out of the study if assigned to control, but would stay in the study if assigned to treatment. Under monotonicity, in a simple randomized assignment E[Y[subscript T][vertical bar]D[subscript T] = D[subscript C] = 0] = E[Y[vertical bar]Z = 1, D = 0] since everyone assigned to treatment who remained in the study is a never-attritor. Estimating E[Y[subscript C][vertical bar]D[subscript T] = D[subscript C] = 0] is tricker, since the subset with Z = 0 and D = 0 is composed of both never attritors and subjects who would attrit under treatment. To estimate E[Y[subscript C][vertical bar]D[subscript T] = D[subscript C] = 0], we first estimate the "principal score," e(X) = Pr(D[subscript T] = D[subscript C] = 0[vertical bar]X) by adapting an E-M algorithm described in Ding and Lu [2017]. Then, following logic described in Sales et al. [2022], we estimated [tau subscript NA] with the least-squares estimate [tau with circumflex subscript NA] from the model: Y[subscript i] = x[prime subscript i beta] + [gamma](1 - e(x[subscript i])) + [tau subscript NA]Z[subscript i] + [epsilon subscript i]. We estimated standard errors with the bootstrap. Preliminary Simulation: For each of 500 replications, we simulated n = 1, 000 outcomes Y as linear in three standard normal covariates, with no treatment effect, assigned half to treatment Z = 1 or control Z = 0, and simulated D from a multinomial model with three categories, D[subscript T] = D[subscript C] = 0, D[subscript T] = D[subscript C] = 1 and D[subscript T] = 0, D[subscript C] = 1. The third covariate was an unobserved confounder, not made available to the analysis models. There were two conditions: in the "No confounding" condition, only the first two covariates contributed to the model for D, and in the "Confounding" condition all three covariates did. We estimated treatment effects in each dataset with three methods: a t-test ignoring attrition, ordinary least squares (OLS) regression of Y on Z and the two observed covariates, and using our SATE estimator. The results are in Table 1 and Figure 1. The t-test estimator is always biased; when there is no unobserved confounder both the OLS and SATE estimators are approximately unbiased, but when there is the SATE estimator is markedly less biased than OLS. However, the SATE estimator is noisier. Applied Results: Comparing two of the treatment conditions, the proprietary software versus one of the open-source programs, resulted in a treatment effect estimate of 0.185 questions correct on a 10-item posttest. The bootstrap standard error was 0.247. For comparison, the same regressions without incorporating principal effects--i.e. assuming that D[subscript T] = D[subscript C]--gave an estimate of 0.169 with a standard error of 0.144. Conclusions: PS is a principled approach to analyzing data from RCTs with high attrition, as long as the monotonicity assumption is plausible. However, in our example it led to a substantially larger standard error than the usual regression estimator. Future research will include simulation results exploring the performance of the PS estimator more generally, and hopefully a heteroscedasticity-consistent standard error estimator other than the bootstrap.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A