Reliability and validity of the Ilyin Oral Interview (IOI) are examined with respect to interscorer agreement. This test is a structured questionnaire based on a sequence of pictures depicting common events in the daily life of a student. Interviews of 11 students from an English as a second language (ESL) class at the University of New Mexico were taped and later scored by 20 native speakers of English. All scorers were either practicing ESL teachers or ESL teachers in training. Interscorer agreement in the IOI scores showed a 79 per cent variance overlap across the 12 most consistent judges and a 45 per cent variance overlap across scores and external validity criteria. The latter included ratings of the IOI interviews by two judges on the five Foreign Service Institute oral interview scales and an independent ranking of the 11 students interviewed by their regular ESL teacher. Item analysis included a questionnaire assessing the pragmatic appropriateness of the questions in the IOI. Interscorer agreement shows the IOI to be a dependable measure of oral proficiency even in the case of relatively homogeneous ability levels and with minimal instructions to scorers. Items which the scorers felt were more natural were generally better discriminators. (Author/AMH)