ERIC Number: ED677762
Record Type: Non-Journal
Publication Date: 2025-Oct-11
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Robust Variance Estimation for Matching Methods with Overlapping Controls: Theory and Applications in Educational Research
Xiang Meng; Luke Miratrix; Natesh Pillai; Aaron Smith
Society for Research on Educational Effectiveness
Matching methods are widely used in educational research to estimate causal effects when randomization is not feasible. These techniques pair treated units (such as schools receiving an intervention) with similar control units based on observable characteristics. However, current statistical inference procedures for these methods can produce misleading confidence intervals when control units are reused across multiple matches--a common scenario in educational settings where the treatment group is often small relative to the control group. We address this challenge through two main contributions. First, we analyze a practical variance estimator that demonstrates robust performance when existing bootstrap methods fail. While this estimator has been used in empirical education research, its comparative advantage over state-of-the-art approaches has not been systematically established. Our analysis shows that this estimator maintains validity even with extensive control unit reuse, providing education researchers with reliable tools for inference in common research scenarios. Second, we develop a theoretical framework that justifies the estimator's validity by establishing its consistency and asymptotic normality under more flexible conditions than previously required. Our work introduces two novel technical conditions that make the theory more applicable to educational research contexts. First, our "derivative control" condition relaxes traditional requirements about outcome functions, allowing for more complex relationships between covariates and outcomes that are common in education data. Second, our "shrinking clusters" assumption accommodates various matching methods beyond the specific nearest-neighbor approach in earlier work, including radius matching and propensity score techniques frequently used in educational program evaluation. Through simulation studies designed to mimic educational research settings, we demonstrate that our estimator maintains proper coverage rates while the current state-of-the-art wild bootstrap method fails in scenarios with substantial overlap. Additionally, we show how our framework extends to weighting estimators, another important class of methods in educational research. This work provides education researchers with both theoretical guarantees and practical tools for conducting valid inference across diverse study designs. Our methods are particularly valuable for studies examining targeted interventions with limited treatment groups, such as pilot programs in a small number of schools or specialized interventions for specific student populations, where ensuring accurate confidence intervals is crucial for making sound policy recommendations.
Descriptors: Educational Research, Computation, Robustness (Statistics), Statistical Analysis, Statistical Inference
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A

Peer reviewed
Direct link
