The ways in which standard-setting procedures are carried out in the medical specialty certification area are described. Three experiments that have been conducted with alternatives to currently used normative standards are examined, and reasons why these experiments have had limited success are suggested in this speech. The defensibility of…

A new approach based on Rasch measurement theory is described for examining the quality of ratings from standard-setting judges. Ratings of nine judges for 213 items on a nursing examination show that judges vary in their views of the essential items for nursing certification, with statistically significant variability in the judged essentiality…

Standard setting on licensure and certification tests is difficult both to execute and to defend. There may, however, be certain minimum standards for standard setting on which most will be able to agree. One such standard is logical consistency. M. T. Kane (1984, 1986) has suggested an approach to evaluating the logical consistency of one widely…

Specific steps for applying the Angoff method are described. In the Angoff method, judges are asked to estimate the probabilities of minimally competent candidates' answering multiple choice test items correctly. Initial information must be obtained for designing the standard-setting process, beginning with the purpose of the examination and any…

Three studies assessed the psychometric characteristics of the Direct Standard Setting Method (DSSM). The Angoff technique was also used in each study. The DSSM requires judges to consider an examination 10 items at a time and determine the minimum items in that set a candidate should answer correctly to receive the credential. Nine judges set a…

Discrepancies among the standards produced by different criterion-referenced standard-setting techniques may be the result of a failure to adequately define the minimally competent candidate. Current research in this area is reviewed in terms of three categories: studies in which no formal assistance in conceptualization is given to judges,…

This report investigates an area of uncertainty in using the Angoff method for setting standards, namely whether or not a judge's conceptualizations of borderline group performance are realistic. Ratings are usually made with reference to the performance of this hypothetical group, therefore the Angoff method's success is dependent on this point.…

Credentialing programs were surveyed to determine the procedures they use to set performance standards on multiple-choice and open-ended assessments. Implications of the various standard-setting approaches for the National Assessment of Educational Progress are discussed, and it is asserted that generalizing from standard-setting in professional…

To address the issues associated with testing standards, sets of context-setting variables and technical matters associated with standard-setting are presented to assist groups or committees desiring to set standards in a systematic way. (Author/CM)

The major forms of evidence that support a standard's credibility are reviewed, and what can be done over time and for different forms of an examination to enhance its comparability in a credentialing setting is outlined. Pass-fail decisions must be consistent to ensure a standard's credibility. (SLD)

Scores from four samples each of 250 and 1,000 physicians demonstrated that a linear procedure for equating scores and rescaling judges' standards for a certification test could be applied to individual item data gathered through the Angoff standard-setting method. Equated and rescaled values were close to those actually assigned. (SLD)

An agenda for research on standards for professional licensure and certification examinations is outlined. The relationship between scores and standards is explored, and major classes of standards are described. Topics for future research are discussed in the areas of judgment basis, process efficiency, group effects, and content and expertise…

Setting performance standards is an area that different constituencies see quite differently. The choices of elements for a particular standard depend to a large extent on the purposes the standard is intended to serve. Standards can be used in certification, as predictors, as descriptors, and as motivators. While performance standards indicate…

A historic change is taking place in American education with the development of voluntary national standards that will identify what all students should know and be able to do in order to live and work in the twenty-first century. Content standards will define what students should know and be able to do, and performance standards will identify the…

Two new indices of stability of content-referenced standard-setting results are presented, relating variability of judges' decisions to the variability of candidate scores and to the reliability of the test. These indices are used to indicate whether scores resulting from a standard-setting study are of sufficient precision. (Author/LMO)