In this study, we describe a framework for monitoring rater performance over time. We present several statistical indices to identify raters whose standards drift and explain how to use those indices operationally. To illustrate the use of the framework, we analyzed rating data from the 2002 Advanced Placement English Literature and Composition…

Although the Angoff procedure is among the most widely used standard setting procedures for tests comprising multiple-choice items, research has shown that subject matter experts have considerable difficulty accurately making the required judgments in the absence of examinee performance data. Some authors have viewed the need to provide…

In this article, we introduce a person-fit statistic called the hierarchy consistency index (HCI) to help detect misfitting item response vectors for tests developed and analyzed based on a cognitive model. The HCI ranges from -1.0 to 1.0, with values close to -1.0 indicating that students respond unexpectedly or differently from the responses…

Three questions are discussed concerning this test: for what population is it appropriate; what is its purpose; and what are the qualifications required for proper administration and interpretation? It is concluded that it is a useful test, but caution is appropriate in answering the three questions. (CTM)

Cut-off scores from two approaches for setting standards are examined. Standards determined from judgments about groups and from inspection of test content are compared. Results indicate that there was neither consistency nor pattern to cut-off scores set from the two procedures. (Author/RD)

The nature and quality of teacher-developed tests was studied in a national sample of 228 teachers representing four grade levels and several subjects. Teachers described their patterns of test use, concerns about assessment, and use of performance testing. Teacher-developed performance assessment was heavily used. (Author/GDC)