ERIC Number: ED663053
Record Type: Non-Journal
Publication Date: 2024-Sep-20
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Designing Multisite Randomized Trials to Detect (Moderated) Mediation Effects
Fangxing Bai; Ben Kelcey; Amota Ataneka; Yanli Xie; Kyle Cox; Nianbo Dong
Society for Research on Educational Effectiveness
Purpose: Multisite mediation studies are a cornerstone in mapping out developmental processes because they probe the mechanisms of a treatment while creating key opportunities to learn from and about variation in those mechanisms across sites. Despite the prevalence of multisite studies, a significant gap in the literature is how to plan such studies to detect mediation effects. Complementing main effect power analyses with mediation effect power analyses is an important step in understanding the capacity of different designs as well as balancing priorities, key tradeoffs and resource allocations. Here, we develop formulas and an R ShinyApp to predict power and guide the design of multisite mediation studies. Background: Multisite experiments are essential in research for their flexibility and robustness in drawing inferences across various contexts. Major funding agencies and professional bodies have recognized their significance and established infrastructures to support multisite research endeavors (e.g., Nunes, 2011; NIH, 2016; Rog, 2015; Shirley et al., 2021). Despite their widespread use, a notable gap exists in methodological guidance for multisite mediation studies. Unlike other estimands, such as main or moderation effects, established methods for determining sample sizes required to detect mediation effects in multisite experiments are lacking. To address this gap, we developed expressions to predict statistical power and optimal sampling plans for detecting mediation in multisite experiments. To address this methodological gap, we developed expressions to predict statistical power for detecting mediation in multisite experiments. Our results are organized around two prevalent approaches in multisite studies: (1) Individual-level mediation (only): Focuses on settings where only individual-level effects are plausible (e.g., contextual influences are immaterial) and the proportion treated in a site is constant across sites; and (2) Both individual and site-level mediation: Considers settings where both individual and contextual effects are plausible and/or the proportion treated in a site varies across sites. Methods: We present the approach capturing both individual and site-level mediation effects for simplicity and will include both approaches in our fully developed paper. Previous research stresses the need to separate individual- and site-level mediation effects. Individual-level effects examine how treatment and mediator changes affect individuals within the same site, while site-level effects analyze changes in the average treatment and mediator within a site. Ignoring factors like the proportion treated or average mediator value within a site can distort mediation effect estimates. Furthermore, it is essential to examine different proportions treated across sites to understand how treatment exposures shape site context and impact outcomes. We draw a working example to explicate principles of estimation and design when the proportion treated or the average mediator value within a site influences the mediator or outcome. For instance, we evaluate how a teacher professional development program (T) enhances the quality of classroom instruction (Y) by improving teacher knowledge (M) by designing a multisite experiment where teachers within each school are randomly assigned to either participate in the program or a control condition. It is conceivable that the proportion treated ([T-bar]) or the average mediator value ([M-bar]) within a site could influence the mediation effect beyond the modulation of an individual's treatment assignment or mediator value. A high proportion of teachers exposed to the professional development program within a school might foster a more conducive learning community compared to lower exposure proportions (e.g., Phelps et al. 2016). Similarly, higher levels of average teacher knowledge at a school may enhance instructional quality by fostering better collaboration and dissemination among colleagues (e.g., Carlisle et al., 2013). To address both individual- and site-level mediation processes, we decompose mediation effects into components by group-mean centering individual-level predictors and introducing their means at the site-level. This approach disentangles how individual-level changes in treatment status influence outcomes through changes in the mediator, and how changes in the proportion treated at a site affect the site-level mediator to influence outcomes. We introduce site-level means of the treatment and mediator in the random slope equations to account for moderated mediation. The model becomes Mediator Model: [equations omitted] and Outcome Model: [equations omitted] We use and as the mediator and outcome values for individual i in site j, and as the overall intercepts, as the individual-level treatment value, as the average individual-level effects with average effects, are average site-level effects, as individual-level covariates with coefficients and , as the site-specific random intercepts or deviations from the average path values, and and as the individual-level error terms, and as group-mean centering individual-level predictors. Mediation effect. Individual-level mediation: Under the formulation, the mediation effect for a particular site is simply a b . However, because the a and b path coefficients can co-vary, the expected or average mediation effect is The resulting error variance of the expected value of the indirect effect is Here we use and as the maximum likelihood estimates of those paths, as the covariance between those paths with an error variance of , and and as the error variances. Site-level mediation: With consideration of site-level aggregates of the mediator, treatment and covariates, the expected site-level mediation effect is The expected error variance of the site-level average mediation effect can be traced as Illustration: Illustrating on our working example, suppose we have parameter values based on pilot studies as follows: . As in Figure 1 and Figure 2, a school-level sample size of approximately 125 schools will provide an 80% chance of detecting an individual-level mediation effect while sampling 154 schools will be required to achieve the same power to detect the site-level mediation effect (AB). Summary: Mediation studies play a pivotal role in understanding program effects and advancing core theories. However, without robust statistical guidance, the inferences drawn from such studies may be limited. This can impede progress in evaluation science by constraining the breadth and reliability of evidence. In our study, we developed expressions to predict statistical power for detecting mediation in multisite evaluations, addressing this critical gap.
Descriptors: Randomized Controlled Trials, Mediation Theory, Statistical Analysis, Robustness (Statistics), Statistical Inference, Reliability, Evidence
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A

Peer reviewed
Direct link
