ERIC Number: ED670562
Record Type: Non-Journal
Publication Date: 2024
Pages: 148
Abstractor: As Provided
ISBN: 979-8-3021-6822-1
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
The Effect of Removing Counselors from the "U.S. News & World Report" Expert Opinion Survey on Institutions' Peer Assessment Scores: Evidence from a Comparative Interrupted Time Series Analysis
Anthony S. Archer
ProQuest LLC, Ph.D. Dissertation, The Florida State University
The peer assessment scores published annually by "U.S. News & World Report" ("USNWR") have fast become the de facto measure of institutional reputation in higher education. While higher education policymakers often use this score for benchmarking and performance-based funding purposes, relatively little is known about this subjective, opinion-based metric. Prior research reveals that perceptions of institutional reputation can differ across stakeholder groups, but information on the magnitude of that difference remains limited. Therefore, the purpose of this study was to compare changes in institutional reputation scores before and after "USNWR's" 2020 organizational decision to exclude high school counselors from their expert opinion survey. Significant differences were expected because respondents to "USNWR's" expert opinion survey are susceptible to a number of cognitive biases that can positively or negatively influence their perceptions of peer institutions. The bias known as the severity effect was used to hypothesize that institutions subject to the policy change would experience significant declines in peer assessment scores. Six other biases (the anchoring effect, the Matthew effect, the echo effect, the halo effect, the size effect, and territorial bias) were hypothesized to explain that the aforementioned reputational score declines would be smaller following the policy change for "USNWR's" Top 50 institutions, R1 institutions, older institutions, institutions with three high ranking undergraduate programs, larger institutions, and institutions in areas of low unemployment. Employing a rigorous quasi-experimental comparative interrupted time series (CITS) research design, this study examined all 1,599 unique four-year bachelor's degree-granting institutions in the U.S. ranked by "USNWR" as a national or regional institution between 2014 and 2024. Data were extracted from purchased editions of "USNWR's Best Colleges" guidebooks and were combined with publicly available data sets from the National Center for Education Statistics, the U.S. Bureau of Labor and Statistics, and the Carnegie Classification of Institutions of Higher Education. This study benefitted from the addition of a nonequivalent no-intervention comparison group. Only national institutions were subjected to "USNWR's" 2020 policy change as they were the only institutions that ever had high school counselors (in addition to HELs) participate in their expert opinion survey. However, regional institutions also received an annual reputational score from HELs, but never had high school counselors participate in their expert opinion survey. Using regional institutions as a comparison group reduced the threat that a rival historical event affected reputational scores at the time of the policy change because any treatment-correlated historical event effects would have been present in the regional comparison group as well. This study revealed that removing counselors from the "USNWR" peer assessment survey triggered significant reputational score declines across all national institutions. However, the magnitude of the effect was larger for national universities compared to national liberal arts colleges--empirically supporting prior anecdotal evidence that national universities game the college rankings. Further analysis revealed heterogeneity in policy impacts by institutional characteristics, with different findings for national universities and national liberal arts colleges. For example, significantly smaller post-policy reputational score declines were identified at national universities with high enrollments, an R1 status, front page status, or three high ranking undergraduate programs. National liberal arts colleges, on the other hand, experienced significantly smaller post-policy reputational score declines based only on high enrollments and a low county-level unemployment rate. Three major conclusions emerged from the interpretation of the results. First, seemingly simple "USNWR" methodological changes can have dramatic effects on university rankings. Second, national university HELs unequivocally game the "USNWR" rankings. And third, not unlike HELs, high school counselors in this study appear to have their own unique biases. These findings seemingly justify repeated calls for "USNWR" to completely eliminate the peer assessment survey given concerns over the instrument's legitimacy, validity, and reliability. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml.]
Descriptors: Surveys, High Schools, School Counselors, Counselor Attitudes, Higher Education, Reputation, Institutional Evaluation, Peer Evaluation, Opinions, Scores, Bias, Publications, Expertise
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: High Schools; Secondary Education; Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A