The English Language Proficiency Assessment (ELPA) is used in the state of Michigan in the United States to fulfill government-mandated No Child Left Behind (NCLB) requirements. The test is used to promote and monitor achievement in English language learning in schools that receive federal NCLB funding. The goal of the project discussed here was to evaluate the perceived effectiveness of the ELPA and to see if those perceptions could meaningfully contribute to a broad concept of the test's validity. This was done by asking teachers and test administrators their views on the ELPA directly after its administration. Two hundred and sixty-seven administrators took a survey with closed and open-ended questions that aimed to tap into the consequential dimensions of test validity. An exploratory factor analysis identified five factors relating to the participants' perceptions of the ELPA. Analysis of variance results revealed that educators at schools with lower concentrations of English language learners reported significantly more problems in administering the ELPA. Three themes (the test's appropriateness, impacts, and administration) emerged from an analysis of qualitative data. This article discusses these results not only as a means of better understanding the ELPA, but also to contribute to larger-scale discussions about consequential validity and standardized tests of English language proficiency. It recommends that broadly defined validity data be used to improve large-scale assessment programs such as those mandated by NCLB. (Contains 5 figures, 5 tables, and 5 footnotes.)