The modified caution index's use in identifying judges whose patterns of item judgment appear aberrant when compared with the pattern produced by the entire group (N=158) of judges was studied. Effects on test standards and passing rates of removing test standards of these judges were also assessed. (TJH)

Five authors representing diverse perspectives comment on the revised "Program Evaluation Standards" approved by the American National Standards Institute (ANSI). Standards are considered in light of their development; measurement issues; program evaluation; evaluation in the local education agency; and the context of evaluation…

Development of a multistage dominant profile method for setting standards on complex performance assessments is detailed. The method grew from experiences with a judgmental policy-capturing procedure and an extended Angoff method. The design of an early adolescence English language arts assessment illustrates the complexity of decisions panelists…

A statistical technique is presented for developing performance standards based on benchmark groups. The benchmark groups are selected using a multivariate technique that relies on a squared Euclidean distance method. For each observation unit (a school district in the example), a unique comparison group is selected. (SLD)

Examines the standards process used for developing the Escrowed Encryption Standard (EES) and its possible impact on national communication and information policies. Discusses the balance between national security and law enforcement concerns versus privacy rights and economic competitiveness in the area of foreign trade and export controls. (67…

Possible sources of intrajudge inconsistency in standard setting are reviewed, and approaches are presented to improve the accuracy of rating. Procedures for providing judges with feedback through discussion or computerized communication are discussed. Monitoring and maintaining judges' consistency throughout the rating process are essential. (SLD)

Two frame-of-reference rater training approaches were compared for effects on reliability and accuracy of cutoff scores generated by 21 raters using Angoff methods on tests taken by 155 undergraduates. Both approaches result in higher interrater reliability and more accuracy than does a non-frame-of-reference method. (SLD)

This study is the National Board of Medical Examiners exploration of content-based techniques (standard-setting techniques in which pass/fail decisions are based upon the performance of examinees in relation to test content). Two content-based techniques (Angoff and Ebel) and three methods of evaluating examinee performance were studied. (MLW)

Applied the analytical judgment standard setting method to 90 papers from the 1996 Grade 8 National Assessment of Educational Progress science assessment. Compared sorting versus direct classification, long and short versions of the classification scale, and effects of discussion on cutscores. Results from 17 Georgia teachers and 8 Michigan…

Sixth-grade teachers (n=26) estimated item performance for their students (724 total students) on a 50-item district-wide science test. Teachers were more accurate in estimating performance of the total group than of the borderline group, but in neither case was their accuracy high. Estimating proportion-correct values using the Angoff standard…

Several initiatives hold great promise to reform teaching: redesigning initial teacher preparation; rethinking professional development; and involving teachers in research, collaborative inquiry, and professional standard-setting. U.S. teachers need to emulate their European and Asian counterparts, who are better paid, prepared, and supported and…

The American Federation of Teachers has developed a series of documents to help educators develop appropriate achievement standards or to describe the standards of other countries. Ordering information is given for these documents and kits for setting standards in particular disciplines. (SLD)

Policymakers often complain that teacher education programs do not have to answer for the quality of their graduates. Over the past five years, as a result of new accreditation rules, hundreds of those institutions have been quietly revamping how they collect and use data about their students. This article reports on the use of performance…

Selection of educational leaders in the United States typically involves four decision points controlled by three types of institutions: admission to and graduation from a principal preparation program controlled by an institution of higher education, certification controlled by the state, and employment controlled by a local school district.…

The purpose of this article is to promote an increased awareness of the processes for setting cut-scores for complex performance assessments by (a) describing the Analytic Judgment Method (AJM) for setting cut-scores, and (b) critically evaluating the technical adequacy and practicability of the AJM by focusing on one investigation where the AJM…