ERIC Number: ED658547
Record Type: Non-Journal
Publication Date: 2022-Sep-22
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Findings from a Bayesian Meta-Regression Analysis of the What Works Clearinghouse with Non-Normal Random Effects: Important Implications for Evaluation Design and Decision Making in the Field of Education
John Deke; Mariel Finucane; Dan Thal
Society for Research on Educational Effectiveness
Background/Context: Methodological background: Meta-analysis typically depends on the assumption that true effects follow the normal distribution. While assuming normality of effect "estimates" is often supported by a central limit theorem, normality for the distribution of interventions' "true" effects is a computational convenience. Noma et al (2022) highlight two examples of systematic reviews of medical studies in which the distribution of effects is not well represented by the normal distribution. They argue that: "Conclusions obtained from meta-analyses are widely applied to public health, clinical practice, health technology assessments, and policy-making. If misleading results have been produced by inadequate methods, the impact might be enormous." Education research and policy background: When evidence informs decision-making, misperceptions of evidence can lead to bad decisions. Perceptions of prior evidence also affect which research questions to ask in future studies. In education research, perceptions of evidence are influenced by the What Works Clearinghouse (WWC). Though perceptions of the WWC have evolved, the early perception that 'nothing works' has been hard to shake. If our perception of the evidence in the WWC is systematically too pessimistic, it could lead to underinvestment in education interventions. It could also lead to underinvestment in new research examining how intervention effects vary across contexts. It would also be problematic if the WWC systematically overstated the effects of interventions. If perceptions were too optimistic, we might waste resources implementing ineffective interventions when we should be investing in the development of new approaches. Purpose/Objective/Research Question: We hypothesize that intervention effects (relative to a 'business as usual' control condition) is not symmetric, but rather skewed towards more favorable effects. Setting/Population/Participants/Subjects/Intervention/Program/Practice: This research involves a meta-analysis of all estimates included in the WWC database that meet evidence standards. Evidence cataloged by the WWC pertains to the effects of educational interventions in the United States. Research Design: We use Bayesian meta-regression analysis to estimate the distribution of effects in the WWC. The model accounts for a form of publication bias in which researchers calculate multiple impact estimates and report only the most favorable. Data Collection and Analysis: Using all evidence that met standards as of July 2020, we model the findings from the WWC database as follows: In this equation, is the reported impact estimate and is the reported standard error, in effect size units, of finding. We model as coming from a normal sampling distribution. The mean of this distribution has two components: is the true impact that seeks to estimate, and is an adjustment for possible publication bias. The adjustment for publication bias is motivated by the idea that the reported impact estimates are maximum order statistics. The variance of the distribution, is the variance of the maximum order statistic obtained by taking d draws from the distribution of the impact estimate. We model the true impacts () as: (1) is the intercept; (2) captures the effect of the finding pertaining to multiple grades; (3) captures the effect of the finding pertaining to one of the four achievement domains; (4) is a set of random effects of grade span studied in finding; (5) is a set of random effects of the outcome domain of the finding; (6) is a set of random effects for the interaction of grade span and outcome domain; (7) is a set of random effects capturing the idiosyncratic effect specific to each finding; and (8) is a set of random effects capturing the overall effect associated with each publication. The publication and finding random effects follow the skewed t-distribution, with degrees of freedom and skew parameter estimated from data. This model was estimated using Markov Chain Monte Carlo as implemented in the software Stan. Findings/Results: In Exhibit 1 shows that the distribution of effects in the WWC is not normal -- it is skewed to the right with a high peak. In Exhibit 2, we report the percentage of effects that are greater than specified values for both distributions. Conclusions: Several conclusions/caveats: The 'nothing works' interpretation of the WWC is clearly wrong. Over 70 percent of effects are positive, and over 1/3 are greater than 0.20 standard deviations. "Key Caveat": Evidence in the WWC does not necessarily reflect what intervention effects will be if implemented in new contexts or at scale. Effects reported by the WWC do not follow the normal distribution. Two implications: Normality overstates the fraction of large, harmful effects (effect sizes less than -0.20). Under normality, 9 percent of effects appear less than -0.20; under skewed t, 4 percent. Normality overstates the fraction of large favorable effects. Under normality, 49 percent of effects appear larger than 0.20; under skewed-t 37 percent. Implications for researchers: On average, business-as-usual control groups will have worse outcomes than treatment group. This is an important ethical concern. One solution is to offer compensation to control group members. Since most interventions are more likely to work than not, comparing interventions to business-as-usual control conditions may be less useful than comparing interventions that are equally resourced but differ in logic model. This might also address ethical concerns. Implications for decision makers: Keeping in mind the "Key Caveat," these findings suggest that interventions included in the WWC are more likely than not to have favorable effects. When considering implementing an intervention, keep in mind that there are likely other interventions that are also more likely than not to have favorable effects, if well implemented and appropriate for the context. "Another key caveat": the risk of an interventions doing harm is not small enough to ignore. If intervention effects are favorable 70 percent of the time, that means they are unfavorable 30 percent of the time.
Descriptors: Bayesian Statistics, Meta Analysis, Regression (Statistics), Research Design, Evaluation Methods, Decision Making, Educational Research, Evidence, Standards
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Information Analyses; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A