ERIC Number: EJ1374477
Record Type: Journal
Publication Date: 2023-Mar
Pages: 15
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1098-2140
EISSN: EISSN-1557-0878
Available Date: N/A
Embedding a Proof-of-Concept Test in an At-Scale National Policy Experiment: Greater Policy Learning but at What Cost to Statistical Power? The Social Security Administration's Benefit Offset National Demonstration (BOND)
Bell, Stephen H.; Stapleton, David C.; Wood, Michelle; Gubits, Daniel
American Journal of Evaluation, v44 n1 p118-132 Mar 2023
A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower "proof-of-concept" evidence of whether the policy can work for motivated individuals. Both forms of learning carry value, yet evaluations rarely combine the two designs. The U.S. Social Security Administration conducted an exception, the Benefit Offset National Demonstration (BOND). This article uses BOND to examine the statistical power implications and potential gains in policy learning--relative to costs--from combining volunteer and population-representative experiments. It finds that minimum detectable effects of volunteer experiments rise little when one adds a population-representative experiment, but those of a population-representative experiment double or quadruple with the addition of a volunteer experiment.
Descriptors: Public Policy, Policy Formation, Federal Programs, Social Services, Disabilities, Insurance, Research Design, Statistical Analysis, Costs, Volunteers, Program Effectiveness, Evidence, Program Evaluation
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub-com.bibliotheek.ehb.be
Publication Type: Journal Articles; Reports - Descriptive
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A