NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED599157
Record Type: Non-Journal
Publication Date: 2017-Apr-28
Pages: 32
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-
EISSN: N/A
Available Date: N/A
Bias and Precision of within- and between- Series Effect Estimates in the Meta-Analysis of Multiple Baseline Studies
Joo, Seang-hwane; Wang, Yan; Ferron, John M.
AERA Online Paper Repository, Paper presented at the Annual Meeting of the American Educational Research Association (San Antonio, TX, Apr 27-May 1, 2017)
Multiple-baseline studies provide meta-analysts the opportunity to compute effect sizes based on either within-series comparisons of treatment phase to baseline phase observations, or time specific between-series comparisons of observations from those that have started treatment to observations of those that are still in baseline. The advantage of the within-series approach is that because all data are used the treatment effects are expected to be estimated with greater precision, whereas the advantage of the between-series approach is that it does not rely on assumptions that time trends are correctly specified and can be extrapolated -- assumptions that are questionable in contexts where shifts in the time series may result from effects other than the treatment. Regardless of whether within- or between-series comparisons form the basis for the effect size estimates, meta-analysts also need to decide whether to analyze the individual participant data or to analyze effect sizes obtained from the primary studies, and whether or not to standardize the treatment effect with small sample size adjustments to meta-analyze outcomes that are scaled differently across studies. The purpose of this study was to compare eight multilevel modeling based approaches to meta-analyzing multiple-baseline studies. The approaches were defined by the crossing of three analytic decisions: 1) whether to use a within- or between-series estimate of effect, 2) whether to analyze individual participant data or effect sizes, and 3) whether to standardize the effect with sample size adjustments or not. The performance of these multilevel modeling approaches are evaluated based on bias and root mean square error of the estimated treatment effect for a variety of data conditions. A Monte Carlo simulation study is currently being run that examines the eight meta-analytic approaches for data conditions that vary in the number of studies (10, 30), the number of participants per study (4, 8), the number of observations per participant (20, 40), the treatment effect size (0, 1), the treatment effect variance across participants and across studies (0, 1), and the size of randomly placed other than treatment event effects (0, 0.5). For each of the 64 conditions obtained by crossing the design factors, 3000 meta-analyses were simulated, where the generated data and effect estimation were based on an immediate shift in level model and where the within case error variance was set to 1.0. Initial results suggest that within- and between- series effect estimates were comparably well-estimated with lower bias and RMSE when individual participant data were meta-analyzed. Based on initial run times for the simulation program, the full simulation study is expected to be finished in October. Graphical displays will be used to compare the distribution of bias and RMSE results for the eight meta-analytic approaches as a function of the manipulated design factors. Discussion will focus on practical advice for those selecting among multilevel modeling approaches for meta-analyzing multiple-baseline studies, while also highlighting the limitations that arise from the design of the simulation study.
AERA Online Paper Repository. Available from: American Educational Research Association. 1430 K Street NW Suite 1200, Washington, DC 20005. Tel: 202-238-3200; Fax: 202-238-3250; e-mail: subscriptions@aera.net; Web site: http://www.aera.net
Publication Type: Speeches/Meeting Papers; Information Analyses
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A