Education policymakers view transparency and accountability as critical to the success of schools. To support these goals, the District of Columbia Office of the State Superintendent of Education (OSSE) has developed an online school report card for communicating information about the characteristics and performance of schools. To support OSSE's interest in making report cards more usable, this study assessed the effect of different designs on how easy the report cards are to use and understand, how easy it is to find information in them, and whether users would recommend the site to others. The study found that moving the link to details of the district's School Transparency and Reporting (STAR) framework from the top of the page to beneath the STAR score improved the site's usability and that reporting the number of points possible for each metric led to a better understanding of how the score is calculated. The combination of design features that produced the best performance on all measures included these two design changes. Other designs had mixed effects. In particular, making year-over-year change in school performance salient made it easier to identify which schools had improved the most, but participants disliked this feature (demonstrated by lower ratings for usability and satisfaction). In general, participants who accessed the site with mobile devices had more difficulty using it. This study illustrates how policymakers and practitioners in other states can efficiently test school report card design changes at scale. [For the Study Snapshot, see ED613807. For the appendixes, see ED613808.]