In this study, retrospective interviews were used to investigate reliability (and thus validity) threats to a computerized ESL listening comprehension test administered at a university in the US. The participants in the investigation, six international graduate students, were asked to respond to semi- and open-ended questions during individual interviews. Their responses were used to confirm a model of potential sources of systematic and random error variance and to generate ideas about unanticipated sources of systematic and random error variance for further investigation. The results suggested that the specific retrospective interview methodology proposed in the report was useful, dependable, and credible in generating broad based, yet focused information specific to the use of a particular test. Based on the results, further areas of research into the inescapable and complex issues of test-taker attributes as they interact with test method and conditions were suggested. (Contains 6 tables.)