The National Education Monitoring Project (NEMP) is responsible in New Zealand for the national assessment of primary school children's achievement in the essential learning areas, one of which is social studies. Individual interviews are one of the approaches used to assess students' understanding. The assessors are registered teachers, selected by NEMP, who attend a training week where they learn how to conduct the standardised assessment tasks and associated interviews. This study examines the reliability of the assessment interviews used in the 2005 round of social studies monitoring, in particular the variations between teacher administrators (TAs) in their use of prompts and probes. The extent of variation observed in the use of three kinds of prompt was sufficient to raise questions about the reliability of the assessment process. A surprising outcome was the consistent failure of TAs to clarify and elucidate students' social studies understandings through the judicious use of probes. The prompt-related variations between TAs and their failure fully to "tap into" understandings assessed by interview-based tasks are serious threats to the validity of claims regarding students' achievement in social studies. This is particularly concerning as NEMP data are used as the basis for identifying and reporting national patterns and trends in educational performance and making recommendations to policy-makers, curriculum planners and educators. (Contains 2 tables and 1 note.)