NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED677792
Record Type: Non-Journal
Publication Date: 2025-Oct-10
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Statistical Power for Incremental Cost-Effectiveness Ratios: A Monte Carlo Confidence Interval Approach
Nianbo Dong; Wei Li; Ben Kelcey
Society for Research on Educational Effectiveness
Background: Effectiveness studies that ignore costs can lead to misguided policies. For instance, reducing class sizes gained traction due to positive outcomes on student achievement (Konstantopoulos & Li, 2012; Krueger, 1999), yet cost-effectiveness studies revealed the high-cost relative to its modest benefits (Brewer et al., 1999). More recently, cost-effective alternatives like web-based tutoring have shown comparable effectiveness to in-person programs but at significantly lower costs, influencing adoption across sectors including education, economics, and psychology (Boggs et al., 2022; Fisherman et al., 2013; Huang et al., 2021; Lay et al., 2020). As a result, agencies such as the Institute of Education Sciences (IES) and the Department of Labor now require economic analyses in program evaluations (Bowden, 2017). This trend has pushed more researchers to incorporate cost-effectiveness analysis (CEA) in the design and evaluation of interventions (e.g., Ma et al., 2023; Nicklas et al., 2022). A key component in designing CEAs is statistical power analysis to determine the required sample sizes for detecting cost-effectiveness. Unlike traditional power analysis for effectiveness alone, power analysis for CEA must consider both cost and effectiveness outcomes--and their covariance. A commonly used metric in CEA is the Incremental Cost-Effectiveness Ratio (ICER), defined as [delta]C/[delta]E, where [delta]C is the incremental cost and [delta]E is the incremental effect (Bowden et al., 2017; Levin & Belfield, 2015). However, estimating the standard error of this ratio is non-trivial due to the complex distribution of ratio statistics and lack of unbiased estimators (O'Brien et al., 1994; Wakker & Klaassen, 1995). Health researchers have developed power analysis methods using the Incremental Net Monetary Benefit (INMB = [kappa][delta]E - [delta]C), which simplifies inference by converting the ratio into a linear form (Willan & Briggs, 2006; Li et al., 2020, 2022). Yet despite the widespread use of ICERs across disciplines, no known studies have provided validated power analysis methods for ICER itself. Purpose: This paper aims to (1) develop and validate power analysis methods for ICER using Monte Carlo confidence intervals, and (2) create practical tools in Excel, SAS, R, and R Shiny for planning cost-effectiveness studies in education and other fields. Methods: Methods We base our approach on the Monte Carlo Confidence Interval (MCCI) method, which has shown strong performance in mediation analysis settings (Kelcey et al., 2017, 2020; Preacher & Selig, 2012) and was recently demonstrated to perform well in constructing confidence intervals for ICER (Dong et al., 2024). A key advantage of this method is that it can be applied during the design phase using summary statistics--unlike resampling-based bootstraps that require raw data. Procedure for Monte Carlo Power Analysis: Outer Loop--Assuming multivariate normality, we simulate estimates of [delta]C* and [delta]E* from the joint distribution: [equation omitted]. [overarc][delta]C equals the desired incremental cost and [overarc][delta]E equals the desired incremental effectiveness to be tested for power analysis. The formulas for the variances (or standard errors) of the incremental cost and the incremental effectiveness estimates have been derived for various study designs. Table 1 summarizes the study designs, statistical models, formulas for the standardized standard error of the treatment effect estimate for RCTs and MIRTs (Bloom, 2006; Hedges & Rhoads, 2010; Schochet, 2008), and the note about design parameters. The statistical models in Table 1 have included covariates. The estimate of the covariance (Cov([overarc][delta]C, [overarc][delta]E)) or correlation (r[subscript [overarc][delta]C, [overarc][delta]E] = Cov([overarc][delta]C, [overarc][delta]E)/([overarc][delta]C, [overarc][delta]E) of the incremental cost and the incremental effectiveness equals the desired covariance or correlation to be tested for power analysis. We can generate data of [delta]C* and [delta]E* based on the multivariate normal distribution with a sample size of J (e.g., J = 3,000) for any design given design parameters (e.g., sample size, ICCs, R[superscript 2]). It is crucial to recognize that when cost data are collected at higher levels than effectiveness data, or when the sample sizes for cost data differ from those for effectiveness outcomes, appropriate formulas from Table 1 should be selected. These formulas correspond to varying levels and sample sizes and are used to estimate the standard error of the cost data. Furthermore, this study will explore the use of multivariate t-distribution for [delta]C* and [delta]E*, potentially offering more precise power estimates in scenarios involving small sample sizes. This consideration is an integral part of our methodology, aiming to enhance the accuracy and reliability of cost-effectiveness analyses across different data collection and sampling scenarios. Inner Loop: For each outer sample j, we simulate I inner samples (e.g., I = 200,000) to estimate ICER = [delta]C**/[delta]E**. [equation omitted] From these, we construct 100(1 - [alpha])% confidence intervals (e.g., 95%) using percentile methods. The null hypothesis is rejected (i.e., ICER is deemed significantly less than k) if the CI excludes the threshold k. Statistical power is calculated as the proportion of simulations for which the null is rejected. Evaluation via Monte Carlo Experiments: To assess method performance, we conduct simulation studies under a wide range of design conditions--(1) Sample sizes: n, J, K ranging from 10 to 100; (2) Effect sizes: 0, 0.1, 0.5 SD; (3) Cost-effectiveness correlations: -0.5 to 0.5; (4) ICCs: 0.05 to 0.3; (5) Covariate R[superscript 2] values: 0 to 0.8; (6) Cost distributions: normal and lognormal; and (7) Replications: M = 3,000. For each scenario, we calculate true power using empirical bootstrap and compare it to power estimated via our MCCI method. We report absolute and percent bias to assess accuracy. Preliminary Results: We conducted a pilot study using a single-level RCT with N = 1000 participants, where [delta]E = 0.5 SD, [delta]C = $499, and cost-effectiveness correlation = 0.33. Under a threshold k = $1,500, the resulting ICER was $998. Across 3,000 replications, estimated power was 0.78 using bootstrapping and 0.75 using MCCI, yielding a bias of -0.03 (-3.8% bias), indicating acceptable accuracy (Table 2). Scholarly Significance: This study introduces the first validated method for conducting power analysis for ICER using Monte Carlo intervals--filling a critical gap in the CEA literature. The approach supports more rigorous study design and informed funding decisions. The proposed software tools will help researchers across disciplines efficiently plan cost-effectiveness studies that meet funder expectations and maximize policy relevance.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A