ERIC Number: ED658713
Record Type: Non-Journal
Publication Date: 2022-Sep-22
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Power Analysis in Three-Level Meta-Analysis
Bixi Zhang; Spyros Konstantopoulos
Society for Research on Educational Effectiveness
Background: Meta-analysis refers to the statistical methods employed to combine results of several empirical studies in a topic of interest (Hedges & Olkin, 1985). Meta-analysis is often included in literature review studies to quantitatively analyze data from a collection of studies (Valentine et al., 2010). The statistical power of a statistical test is the probability of detecting an effect when it exists (Cohen, 1988). Power computations in primary studies are not too different from power computations in the meta-analysis (Hedges & Pigott, 2001). Two commonly used meta-analytic models are the fixed and the random effects. Fixed effects meta-analysis assumes a single underlying population effect size parameter and the inference is about the particular collection of studies and their effect sizes. Random effects meta-analysis follows a two-level structure (effect sizes are nested within studies) and introduces a between-study variance component. Hedges and Pigott (2001) developed methods to calculate statistical power for the average effect size in random effects meta-analysis. More recent research also delineated methods for power analysis in fixed and random effects models (Valentine et al., 2010). Sometimes the nested structure of meta-analytic data is more complicated with studies connected to certain groups of researchers. Whenever studies are nested within groups of researchers, this nesting effect needs to be considered both in the power and statistical analyses stages. Ignoring potential systematic variability between groups of researchers may produce smaller standard errors of the overall mean effect size estimates and may increase the Type I error probability. Thus, with more complicated data structures, a three-level model that estimates the between group of researchers variance at the third level may be preferable to a two-level model (Konstantopoulos, 2011). Purpose: Although previous research has argued that statistical tests in meta-analysis have typically greater statistical power than tests in primary studies (Borenstein et al., 2021; Cohn & Becker, 2003), it is still prudent to conduct a priori power analysis in meta-analysis before investing money, time and resources to perform the meta-analytic review. This study provides closed form formulas for power analysis for three-level univariate meta-analysis. Additionally, a simulation study provides guidance about the number of studies needed to achieve acceptable power (e.g., > 0.80, suggested by Cohen (1988)) in the presented model. Methodology: A univariate unconditional three-level model for study i in group g is expressed as Level-1: T_ig=?_ig+?_ig,Ê?_ig~N(0,v_i), (1) Level-2: ?_ig=?_0g+?_ig ?, ??_ig~N(0,?_((2))^2), (2) Level-3: ? _0g=?_00+?_0g, ?_0g~N(0,?_((3))^2), (3) where T_ig is the effect size estimate in study i in group g, ?_00 is the average effect size parameter across all studies and groups that needs to be estimated, ?_ig is the within-study, within-group level-1 residual with a known variance v_i, ?_ig is the between-study within-group level-2 random effect with variance ?_((2))^2, and ?_0g is the between-group level-3 random effect with variance ?_((3))^2. The variance-covariance matrix of group g at the third level following Konstantopoulos (2011) is written as where V_((2,g) ) is the variance-covariance matrix of a two-level model, 1_((3,g)) is a vector of ones, T_3 is the matrix of random effects at the third (group) level and the subscripts indicate hierarchy level and group of researchers. Equation (4) can also be expressed as where n_g indicates the number of studies in group g, I_(n_g ) is an n_g?n_g identity matrix, 1_((n_g)) is a vector of n_g ones and ? is the Kronecker product. The whole variance-covariance matrix for a three-level model with m groups of researchers is a block diagonal matrix with m matrices in the diagonal, namely The inverse of this block diagonal matrix equals to the inverse of each block in the matrix and is written as When the alternative hypothesis is true the Z statistic used to determine the statistical significance of the mean effect size follows a non-central normal distribution with a mean equal to a non-centrality parameter (NCP) ?_3^* and a variance of 1. In prospective power analysis the mean effect size and its variance can be extracted or hypothesized using findings from prior and current relevant research. The NCP is defined as the quotient of the theorized average effect size TÊ?_.^ and the square root of its variance, namely where 1 is a vector of ones, W_3=(V_3 )^(-1) is the weight matrix. The NCP can be also written as: where k_g is the number of studies in the gth group, m is the number of groups, and W_(3(st,g)) indicates the element at the sth row and tth column in the gth group of the weight matrix. The NCP is a scalar because both the numerator and denominator are scalars. The NCP is a direct function of the power of the Z test, that is, as the NCP increases power increases. To calculate the power of a two-tailed Z statistic of the anticipated average effect size when the type I error rate is set to 0.05 we use the formula: where ?(x) indicates the standard normal cumulative distribution function. Simulation The simulation considers several conditions (see Table 1). For each condition, 2000 replications were done to compute power. Table 2 shows the number of studies in each group needed to achieve adequate power (> 0.80). Suppose the expected mean effect size is 0.20, the within-study variance is 0.10 and the level-2 and level-3 ICCs are 0.20 and 0.05 respectively (condition 6). Assuming six groups, six studies are needed in each group to achieve adequate power (36 studies altogether). Assuming 10 groups, three studies are needed in each group to achieve adequate power (30 studies in total). Conclusion: The present study developed methods for power analysis in three-level meta-analysis extending previous research in two-level meta-analysis. Overall, other things being equal increasing the number of groups has a larger impact on power than increasing the number of studies per group. Also, increases in the mean effect sizes increases power other things being constant. Larger values of level-2 and level-3 ICCs decrease power other things being constant. A larger third-level variance could strongly influence power.
Descriptors: Meta Analysis, Probability, Effect Size, Research Methodology, Error of Measurement, Simulation
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A