This paper introduces the Website Acceptability Tiered Checklist (WATCH), a preliminary version of a multi-trait scale that could be used by instructors and students to assess the quality of websites chosen as source materials in students' research papers in a Humanities program. The scale includes bands for assessing: (i) the authority and reputation of the author(s); (ii) the accuracy and objectivity of the contents; (iii) evidence of academic rigor in the text; and (iv) the currency and maintenance of the site as well as the clarity of bibliographical information. The scale was used to evaluate the suitability of 84 Web-based sources selected by first-year English for Academic Purposes (EAP) students (N = 36) to support their points in a research-based argumentative essay. Each Web source was scored individually by two trained raters; inter-rater reliability (using the Cronbach-alpha test) was found to be 0.89 for the overall scores and between 0.77 and 0.87 for the individual bands. The paper discusses the value of using an instrument such as WATCH to assess students' choices of websites, as well as potential difficulties faced by instructors when evaluating these sources.