ERIC Number: ED663435
Record Type: Non-Journal
Publication Date: 2024-Sep-20
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Experimental Design and Analysis of Multiple-Group Individually-Randomized Group Trials
Ben Kelcey; Fangxing Bai; Amota Ataneka; Yanli Xie; Kyle Cox
Society for Research on Educational Effectiveness
We consider a class of multiple-group individually-randomized group trials (IRGTs) that introduces a (partially) cross-classified structure in the treatment condition (only). The novel feature of this design is that the nature of the treatment induces a clustering structure that involves two or more non-nested groups among individuals in the treatment condition that does not exist in the control condition. Although review of prior literature suggests that multiple-group IRGTs are present in theory and practice, the appropriate analysis of this structure has largely gone undeveloped. In this study, we develop the statistical theory that motivates this design, map out its use in practice, and develop estimation methods and closed-form expressions to track treatment effects, their sampling variability and the statistical power to detect the treatment effect. The results provide the core tools to effectively and efficiently design and analyze studies drawing on multiple-group IRGTs. Background: Literature has sought to develop and extend the principles and practices of experimental design to a broad array of contexts, novel programmatic features, and structural considerations. One type of design that has received growing attention in recent years is IRGTs. In IRGTs, individuals are the unit of assignment but the treatment delivery occurs within groups. This combination of individual assignment with group implementation often results in disparate (hierarchical) structures between the treatment and control conditions. In many intervention studies the treatment condition induces a form of nesting or clustering that does not naturally exist in the control condition. For example, many interventions operate outside of the purview of customary school hours, calendars and structures (e.g., summer programs, after-school interventions) and participation in such programs frequently introduces new clustering structures that are not present within a control condition. Prior research has widely documented the prevalence and scope of such designs. However, research has suggested that extant methods have not been well-adapted for the types of complex nesting structures routinely encountered in contemporary interventions. Prior research has largely designed and analyzed data from IRGTs by either ignoring the clustered structure in the treatment arm, or artificially imposing a clustered structure in the control arm. However, both analytic approaches are incongruent with the nature of the design because the former inappropriately assumes the clustered individuals in the treatment arm are independent while the latter inappropriately assumes individuals in the control arm share dependence. In this study, we develop principles and strategies for multiple-group IRGTs. We focus on four developments. First, we describe the nature of cross-classified IRGTs, how they arise in practice, and then detail the corresponding statistical models. Second, we develop principles of estimation and inference for this design. Third, we develop expressions to track the statistical power with which the design can detect effects and follow with design implications and strategies that emerge from these results. Last, we compare our results with those from a commonplace misspecified model that omits the partially nested structure. We end with an illustration and a discussion. Methods: Consider a simple cross-classified individually-randomized group trial where students are assigned to a participate in a summer school program or continue unschooled throughout summer (T). This summer school intervention is a full press program that involves two complementary components: (a) teacher led group instruction with classmates in a conventional summer school classroom and (b) one on one supplemental tutoring at a community center by a tutor (e.g., Mozolic & Shuster, 2015). Assume students are individually assigned to participate in the program or to control condition that involves no summer instruction. Further assume that we have sampled n[subscript 1][superscript (t)] and n[subscript 1][superscript (c)] total students in the treatment and control conditions and n[subscript 2a][superscript (t)] teachers and n[subscript 2b][superscript (t)] tutors in the treatment condition. For brevity, we omit consideration of covariates, interactions, higher-order terms, and more complex aggregate constructs. However, we note that our framework incorporates these and other considerations and features. We draw on a multiple-arm partial nesting (MA-PN) approach that allows differing models across study arms. We use superscript t to indicate the treatment condition, superscript c to indicate the control condition. The treatment-arm model is Treatment-arm: Within-level: [equation omitted] Between-level: [equation omitted]. For the treatment arm on within-level, [pi][subscript 0jk][superscript (t)] is the intercept for student i served by teacher j and tutor k and [epsilon][subscript ijk][superscript (t)] is the student-specific error term. For the between-level, [gamma][subscript 0][superscript (t)] is the grand mean of all students under the treatment condition, r[subscript 00j][superscript (t)] is the teacher-specific random effect, and u[subscript 00k][superscript (t)] is the tutor-specific random effect. Control arm students continue unschooled through summer. As a result, we can adopt a simple single-level model such that Control-arm Within-level: [equation omitted] with [gamma][subscript 0][superscript (c)] as the overall intercept under the control condition and [epsilon][subscript i][superscript (c)] is the student-specific error term. The main effect ([delta]) of the treatment is the difference between the overall intercept for the treatment condition and the intercept for the control condition [equation omitted]. To draw inferences regarding the main treatment effect under maximum likelihood, the sampling variability of the estimated main effect is [equation omitted]. To test whether the main effect is equal to zero in the population we can specify the standard hypothesis test as [equation omitted]. A resulting t-test can then be formed as the ratio of the estimated effect to its standard error [equation omitted]. The resulting test has a t-distribution with approximate degrees of freedom determined by the Kenward-Roger adjustment. The power of this test can be estimated using [equation omitted] where t([delta]) is the test statistic outlined above, [phi] represents the cumulative t density function, and as [subscript critical] the critical value. Simulation Assessment: To provide an initial assessment of the performance of our analytical results, we drew on the above models to conduct a Monte Carlo simulation. Collectively, the results suggested that our analytic results closely tracked the treatment effect, its sampling variability and the statistical power for a given sample size (Table 1). Significance: The results facilitate the design and analysis of multiple-group IRGTs. This work addresses current gaps and critically undergirds the scope and quality of evidence produced by these types of studies.
Descriptors: Randomized Controlled Trials, Research Design, Statistical Analysis, Error of Measurement, Statistical Inference
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A