NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED658587
Record Type: Non-Journal
Publication Date: 2022-Sep-21
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
When Can Contextual Multilevel Models Be Trusted? A Discussion from a Graphical Models Perspective
Yi Feng; Peter M. Steiner
Society for Research on Educational Effectiveness
Research Context: In educational research, "context effects" are often of inferential interest to researchers as well as of evaluative interest to policymakers. While student education outcomes likely depend on individual-level influences like individual academic achievement, school contexts may also make a difference. Such questions are conventionally addressed using "contextual" or "compositional" multilevel models (MLM; Enders, 2013; Enders & Tofighi, 2007; Kreft et al., 1995; Paccagnella, 2006). The conventional contextual MLM, however, does not explicitly take the data generating process (DGP) into consideration. Specifically, the conventional perspective does not differentiate between "compositional effects" (Leyland & Groenewegen, 2020) and "interference effects" (Ogburn & VanderWeele, 2014). When crude measures of cluster averages are formed and contextual effects are assessed simply as the direct effect of cluster averages on the individual-level outcome (i.e., compositional effect), it implies a highly restricted DGP where students do not directly influence one another within schools, but rather affect peers only via the corresponding school-level mean. This highly restricted DGP model is rarely plausible in real school settings considering all the direct daily interactions among individual students. Objectives: Using causal graphs, the current study differentiates and articulates different types of context effects and their corresponding DGP. More specifically, the causal identification of context effects and level-1 treatment effects will be examined under different DGPs, via analytical derivation, graphical analyses, as well as simulation. The main research questions include: 1) How to clearly define and distinguish between different types of context effects using causal graphical models? 2) Under what circumstances can conventional contextual MLM be utilized to causally identify compositional effects, interference effects, and level-1 treatment effects? 3) When the conventional contextual MLM is not aligned with the true DGP, can the causal effects of interest be recovered by additional computational adjustments? Theoretical Framework: In practice, the terms "context effects," "contextual effects," and "compositional effects" often are used interchangeably, mainly only referring to the impact from aggregate-level characteristics. As mentioned earlier, such ambiguity neglects the causal mechanisms through which students' immediate context shapes individual outcomes. Therefore, we specifically distinguish different types of context effects from a causal perspective. More specifically, a "compositional effect" is defined as the direct causal effect of group-level aggregates on individual outcomes; "higher-level context effect" refers to the direct causal effect of a group-level construct on individual outcomes; "direct interference effect" is the direct causal effect of one individual's treatment (or characteristic, more general speaking) on another individual's outcome (Ogburn & VanderWeele, 2014; Rosenbaum, 2007); and "contagion interference effect" is the direct causal effect of one individual's outcome (or intermediate outcome) on another individual's outcome (Ogburn & VanderWeele, 2014). Methods: Causal graphs have become an important framework for causal inference (Pearl et al., 2016; Pearl & Mackenzie, 2018). Importantly, "causal interference graphs" can be employed to elucidate the DGP for context effects (Ogburn & VanderWeele, 2014). In Figures 1-7, causal graphs are presented to illustrate seven possible DGPs relevant to context effects: 1) no context effects, 2) compositional effects only, 3) direct interference effects only, 4) level-2 confounding context effects, 5) both compositional and direct interference effects, 6) cross-level confounding, and 7) contagion interference effects. In practice, the dominating analytical approach for modeling contextual effects is the compositional MLM (insert equation 1 here) , where the "contextual effect" is estimated as (insert math symbol). To investigate the performance of this conventional approach, the expected value of (insert math symbol) are each derived analytically under 11 different DGPs, assuming linearity and normal random errors. First, we consider the seven causal graphs (Figures 1-7) assuming homogeneous interference effects within and across clusters. Further, graphs with direct interference effects (Figures 3 and 5) are extended to accommodate more complex conditions, allowing heterogeneous interference effects within clusters: 1) some individuals have positive impact on peers while some have negative influence on peers (i.e., "offsetting interference"), and 2) some individuals have direct impact on peers while some do not influence others (i.e., "selective interference"). Following the graphical analyses and analytical derivation, Monte Carlo simulation studies are carried out to empirically assess the causal identification of contextual effects and level-1 direct effects. Results: Table 1 provides a summary of the analytic results across the different scenarios. As shown in Table 1, out of the 11 possible DGPs, the conventional compositional MLM Eq. is only able to causally identify the level-1 direct effect (A[subscript ik]->Y[subscript ik]) when the interference effects are absent or perfectly cancel out within groups; it is only able to causally identify the compositional contextual effect (insert math symbols) when the interference effects are absent or perfectly cancel out within groups, and when there is no omitted level-2 or cross-level confounding. The simulation results are summarized in Table 2, along with the corresponding analytic results and the true population parameter values. As shown in the table, the simulation results are consistent with the analytical derivations, with minimal deviations due to Monte Carlo simulation error. The results also suggest that under certain circumstances, interference effects can be recovered through MLM by simple algebraic manipulations, even when the analytical model is misspecified. For Figure 3 with homogeneous interference effects, the direct interference effect (insert math symbol) can be estimated as (insert equation 2 here). Applying the same logic, for Figure 7 with contagion interference effects, the indirect interference effect of A through S can be estimated with (insert equation 3 here). Conclusions: Relevant policy and interventions may be implemented based on research findings about contextual effects. It is thus of great importance that the nature of contextual effects is understood properly as a causal effect and that the analytical approach utilized is able to identify such causal effect and yield interpretable results. The current study suggests that conventional compositional models can fall short of correctly identifying both contextual effects and level-1 direct effects when peers affect individual student outcomes via interference. It is our hope that the current study can help shed some light on the ambiguity in interpreting and estimating contextual effects in educational research.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A