NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
ERIC Number: ED599082
Record Type: Non-Journal
Publication Date: 2017
Pages: 61
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-
EISSN: N/A
Available Date: N/A
Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers
Porter, Kristin E.
Grantee Submission
Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs) are statistical procedures that counteract this problem by adjusting p-values for effect estimates upward. While MTPs are increasingly used in impact evaluations in education and other areas, an important consequence of their use is a change in statistical power that can be substantial. Unfortunately, researchers frequently ignore the power implications of MTPs when designing studies. Consequently, in some cases, sample sizes may be too small, and studies may be underpowered to detect effects as small as a desired size. In other cases, sample sizes may be larger than needed, or studies may be powered to detect smaller effects than anticipated. This paper presents methods for estimating statistical power, for multiple definitions of statistical power and presents empirical findings on how power is affected by the use of MTPs. [This is the online version of an article published in "Journal of Research on Educational Effectiveness." For the final published version of this article, see EJ1175213. For the full-text non-journal submission, see ED567783.]
Publication Type: Guides - Non-Classroom
Education Level: N/A
Audience: Researchers
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R305D140024
Author Affiliations: N/A