ERIC Number: ED562897
Record Type: Non-Journal
Publication Date: 2014
Pages: 19
Abstractor: ERIC
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
The Cost of Agree-Disagree: Satisficing and Sacrificing Reliability
McIntyre, Joe; Gehlbach, Hunter
Society for Research on Educational Effectiveness
Of all the approaches to collecting data in the social sciences, the administration of questionnaires to respondents is among the most prevalent. Despite their popularity, there is broad consensus among survey design experts that using these items introduces excessive error into respondents' ratings. The authors attempt to answer the following question: what are the specific impacts on survey data quality if one uses statements and A-D response anchors as opposed to questions and construct-specific (C-S) response anchors? Data were collected from a nationally representative SurveyMonkey panel of parents/primary caregivers for school-aged children. In return for their participation, participants received the chance to win money for a charity of their choosing. The authors found strong evidence that web-pages with only AD items elicited more straight-lining than corresponding pages with only C-S items. To their surprise, it was found that A-D scale scores were frequently less reliable than C-S scale scores even before removing straight-liners. While the authors consider these findings to be conclusive, they believe that more work remains to be done, both theoretical and practical. Tables and scales from the two forms of the survey are appended.
Descriptors: Questionnaires, Surveys, Likert Scales, Test Items, Test Format, Online Surveys, Parents, Child Caregivers, Randomized Controlled Trials, Test Reliability, Statistical Analysis, Correlation
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org
Publication Type: Reports - Research; Tests/Questionnaires
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A