Multiple matrix sampling is applied to a variation of Angoff's standard setting method. Thirty-six experts (internists) and 190 items were divided into five groups, and borderline examinee performance was estimated. There was some variability in the cutting scores produced by the individual groups, but various components were well estimated. (SLD)

Specific steps for applying the Angoff method are described. In the Angoff method, judges are asked to estimate the probabilities of minimally competent candidates' answering multiple choice test items correctly. Initial information must be obtained for designing the standard-setting process, beginning with the purpose of the examination and any…

Three studies assessed the psychometric characteristics of the Direct Standard Setting Method (DSSM). The Angoff technique was also used in each study. The DSSM requires judges to consider an examination 10 items at a time and determine the minimum items in that set a candidate should answer correctly to receive the credential. Nine judges set a…

An ignored aspect of standard setting, namely the possibility that Angoff or Nedelsky judges specify inconsistent probabilities (e.g., low probabilities for easy items but large probabilities for hard items) is explored. A latent trait method is proposed to estimate such misspecifications, and an index of consistency is defined. (Author/PN)

This paper explores common concerns about competency testing as they relate to the certification of high school graduates seeking a diploma in the United States. Competency testing is widespread in the United States, with 40 states engaged in competency testing in at least one grade. In general, and particularly for graduation requirements, the…

The implicit definition of competence and the inferential chain that links the standard-setting process to the decision outcomes of the method are considered for two classes of standard-setting procedures: those involving data-free judgments of items and those involving data-based judgment of items. The major underlying assumptions of competence…

The influence of judges' knowledge on standard setting for competency tests was studied with 17 judges who took an economics teacher certification test while setting competency standards using the Angoff procedure. Judges tended to set higher standards for items they answered correctly and lower standards for items they answered incorrectly. (SLD)

Six measurement-related problems which face educators attempting to implement minimum competency-oriented instruction (MC-OI) are isolated. They are: (1) choosing appropriate competencies to measure; (2) acquiring appropriate tests; (3) setting standards for MC-OI; (4) selecting suitable sanctions; (5) providing equidifficult examination…

The modified caution index's use in identifying judges whose patterns of item judgment appear aberrant when compared with the pattern produced by the entire group (N=158) of judges was studied. Effects on test standards and passing rates of removing test standards of these judges were also assessed. (TJH)

In order to evaluate standard setting procedures, apart from the more commonly applied approach of simply comparing the derived standards or failure rates across various techniques, this study investigated the errors of classification associated with the contrasting groups procedures. Monte Carlo simulations were employed to produce…

Standards, passing scores, or cut scores have been seen as an element of criterion-referenced tests since their introduction. This paper discusses at least two issues surrounding the establishment of cut scores which appear to need clarification: (1) the theoretical definition of a cut score; and (2) decisions which must be made in selecting a…

Studies of the Angoff method of standard setting suggest that judges agree in their estimates of the relative difficulties of test questions for minimally competent examinees and that each judge's estimates correlate well with the observed item difficulties for examinees whose total test scores are near the judge's personal standard (G. E.…

This study examined the contribution of five categories of variables to the standard-setting recommendations of a representative sample of 241 judges recommending standards for the General Knowledge and Communication Skills Tests of the National Teacher Examinations (NTE) for the state of North Carolina. It was based on the assumption that…

Teacher competency tests are being used more frequently to assess the knowledge and skills of prospective teachers. Educational policymakers face the dilemma of setting passing standards for these tests which will satisfy the desire for meaningful quality standards for teachers while meeting the necessity for making available a reasonable number…

The impact of overall test length and difficulty on the expert judgments of item performance by the Nedelsky method were studied. Five university-level instructors predicting the performance of minimally competent candidates on a mathematics examination were fairly consistent in their assessments regardless of length or difficulty of the test.…