Purpose: Development Agents (DAs) are employed by agricultural departments to provide capacity development for farmers. In this contribution, the adjustment of a competence profile originally developed for the Province of Esfahan [Karbasioun, M., M. Mulder, and H. J. A. Biemans. 2007. "Towards a Job Competency Profile for Agricultural…

Establishing cut scores using the Angoff method requires panelists to evaluate every item on a test and make a probability judgment. This can be time-consuming when there are large numbers of items on the test. Previous research using resampling studies suggest that it is possible to recommend stable Angoff-based cut score estimates using a…

Purpose: The purpose of the research was to develop a competency-based performance appraisal resource for evaluating school nutrition (SN) managers and assistants/technicians. Methods: A two-phased process was used to develop the competency-based performance appraisal resource for SN managers and assistants/technicians. In Phase I, draft…

Multiple matrix sampling is applied to a variation of Angoff's standard setting method. Thirty-six experts (internists) and 190 items were divided into five groups, and borderline examinee performance was estimated. There was some variability in the cutting scores produced by the individual groups, but various components were well estimated. (SLD)

Setting standards for the purpose of certification is frequently performed using judgmental techniques such as the Angoff method. This study was performed to identify types of items that judges find hard to rate accurately, that is, types of items on which examinees perform differently than predicted by the judges. Once identified these item types…

Alternative group judgment approaches to setting minimum competency standards were compared. Replication of results was possible for eight different tests (reading and mathematics, across four grade levels). The Kansas Competency Based Tests in reading and mathematics were administered statewide to students in grades two, four, six, and eight.…

Three studies assessed the psychometric characteristics of the Direct Standard Setting Method (DSSM). The Angoff technique was also used in each study. The DSSM requires judges to consider an examination 10 items at a time and determine the minimum items in that set a candidate should answer correctly to receive the credential. Nine judges set a…

An ignored aspect of standard setting, namely the possibility that Angoff or Nedelsky judges specify inconsistent probabilities (e.g., low probabilities for easy items but large probabilities for hard items) is explored. A latent trait method is proposed to estimate such misspecifications, and an index of consistency is defined. (Author/PN)

The sources of variability in the Angoff standard-setting procedure, when applied to the Alabama High School Graduation Examination (AHSGE), were examined. The sources of variability examined are judges, rounds (replications), competencies (items), and interactions among these three sources. After training, the judges were given a statement of a…

The influence of judges' knowledge on standard setting for competency tests was studied with 17 judges who took an economics teacher certification test while setting competency standards using the Angoff procedure. Judges tended to set higher standards for items they answered correctly and lower standards for items they answered incorrectly. (SLD)

Eleven judges representing 11 different geographic regions in the United States participated in a standard-setting session designed to determine the possibility of obtaining interjudge consensus and intrajudge consistency simultaneously. Each judge had experience in the field for which standards were being set. The judges rated 65 multiple-choice…

Three questions were addressed by this study: (1) Do the Ebel, Nedelsky, and Angoff methods of setting minimum competency standards produce different cutting scores for the same test? (2) When utilized by raters of different competencies, do these methods result in different Standards? and (3) Are the standards set by the different methods the…

The modified caution index's use in identifying judges whose patterns of item judgment appear aberrant when compared with the pattern produced by the entire group (N=158) of judges was studied. Effects on test standards and passing rates of removing test standards of these judges were also assessed. (TJH)

The present study examines the influence of judges' item-related knowledge on setting standards for competency tests. Seventeen judges from different professions took a 122-item teacher-certification test in economics while setting competency standards for the test using the Angoff procedure. Judges tended to set higher standards for items they…

These five papers describe the methods, results, and implications of a multimethod approach to the establishment of standards for the three subject areas of the Alabama High School Graduation Examination (AHSGE). James E. McLean introduces the problems and methods involved by identifying three general approaches to scoring procedures found in the…