NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
ERIC Number: ED628289
Record Type: Non-Journal
Publication Date: 2022-Dec-15
Pages: 46
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation
Benjamin Lu; Eli Ben-Michael; Avi Feller; Luke Miratrix
Grantee Submission
In multisite trials, learning about treatment effect variation across sites is critical for understanding where and for whom a program works. Unadjusted comparisons, however, capture "compositional" differences in the distributions of unit-level features as well as "contextual" differences in site-level features, including possible differences in program implementation. Our goal in this article is to adjust site-level estimates for differences in the distribution of observed unit-level features: If we can reweight (or "transport") each site to have a common distribution of observed unit-level covariates, the remaining treatment effect variation captures contextual and unobserved compositional differences across sites. This allows us to make apples-to-apples comparisons across sites, parceling out the amount of cross-site effect variation explained by systematic differences in populations served. In this article, we develop a framework for transporting effects using approximate balancing weights, where the weights are chosen to directly optimize unit-level covariate balance between each site and the common target distribution. We first develop our approach for the general setting of transporting the effect of a single-site trial. We then extend our method to multisite trials, assess its performance via simulation, and use it to analyze a series of multisite trials of adult education and vocational training programs. In our application, we find that distributional differences are potentially masking cross-site variation. Our method is available in the balancer R package. [This is the online version of an article published in "Journal of Educational and Behavioral Statistics."]
Related Records: EJ1386905
Publication Type: Reports - Evaluative
Education Level: Adult Education
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED); National Science Foundation (NSF)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R305D200010; 1745640
Author Affiliations: N/A