Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing…

The institutions in education use various assessment methods to decide on the proficiency levels of students in a particular construct. This study investigated whether the decisions differed based on the type of assessment: norm-and criterion-referenced assessment. An achievement test with 20 multiple-choice items was administered to 107 students…

Subscore reporting under item response theory models has always been a challenge partly because the test length of each subdomain is limited for precisely locating individuals on multiple continua. Diagnostic classification models (DCMs), providing a pass/fail decision and associated probability of pass on each subdomain, are promising…

Purpose: This study aimed to compare the cut scores obtained by the Extended Angoff and Contrasting Groups methods for an achievement test consisting of constructed-response items. Research Methods: This study was based on survey research design. In the collection of data, the study group of the research consisted of eight mathematics teachers for…

This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM-ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom…

The aim of this study is to standardize Autism Diagnostic Observation Schedule (ADOS) scores within a large sample to approximate an autism severity metric. Using a dataset of 1,415 individuals aged 2-16 years with autism spectrum disorders (ASD) or nonspectrum diagnoses, a subset of 1,807 assessments from 1,118 individuals with ASD were divided…

This article reports on two related studies carried out to link the State examination of Dutch as a second language to the Common European Framework of Reference for languages (CEFR). In the first study, key persons from institutions for higher education were asked to determine the minimally required language level of beginning students. In the…

Some of the different approaches to standard setting are discussed. Brief comments and references are offered concerning strategies that rely primarily on the use of expert judgment. Controversy surrounds methods that use expert judges, as well as those using test groups to set scores empirically. A minimax procedure developed by H. Huynh, an…

A solution is offered to problems associated with the inequality in the manipulability of probabilities of classification errors of masters versus nonmasters, based on competency test results. Eschewing the typical arbitrary establishment of observed-score standards below 100 percent, the solution incorporates a self-correction of wrong answers.…

In 1997, the Education Department of Western Australia, in conjunction with the teachers' union, initiated a trial project to establish a career path for teachers. Standards are to be set for the career path for three stages: (1) entry to Level 1; (2) transition from Level 1 to 2; and (3) transition from Level 2 to 3. This paper discusses the…

In 1999 a validation study of the 1998 Civics Assessment of the National Assessment of Educational Progress (NAEP) was conducted, combining a similarity classification study and a booklet classification study. The rationale was that if the teachers who had participated in the 5-day achievement levels-setting process could not use the descriptions…

This paper describes the feedback included in operational achievement level setting (ALS) procedures for the National Assessment of Educational Progress (NAEP). It does not describe the feedback form field trials, pilot studies, or other research studies related to the NAEP. The NAEP ALS process includes three rounds of item-by-item ratings, and…

Applied the analytical judgment standard setting method to 90 papers from the 1996 Grade 8 National Assessment of Educational Progress science assessment. Compared sorting versus direct classification, long and short versions of the classification scale, and effects of discussion on cutscores. Results from 17 Georgia teachers and 8 Michigan…

This paper reports on a standard-setting method designed for complex performance assessments with multiple performance categories. The method studied, the Analytical Judgment Method, involves panelists' making analytical classification decisions for each of the test's components individually. It also allows for discussion and reconsideration of…

The results of classifying test items on the basis of their Mantel-Haenszel (MH) alpha estimates were compared to the results of classifying these items using an item response theory (IRT) based procedure involving the comparison of item difficulties in the interest of identifying the alpha value that maximized the decision concordance between the…