Embedded Standard Setting (ESS; Lewis & Cook, 2020) transforms standard setting from a standalone workshop to an active part of the assessment development lifecycle. ESS purports to lower costs by eliminating the standard-setting workshop and enhance the validity argument by maintaining a consistent focus on the evidentiary relationship…

Standard setting is the process of identifying the point(s) on a scale that serve to differentiate between individuals of distinct proficiency levels. While standard setting is ultimately a policy decision, most of the process is carried out by subject matter experts who are tasked with reconciling item-level or examinee-level information (e.g.…

Changes to the American Board of Internal Medicine's Maintenance of Certification Program: Preserving Core Assessment Values As Maintenance of Certification (MoC) for practicing physicians has evolved into more continuous programs, controversy has arisen over how physicians should self-regulate their profession and serve the public trust in doing…

In this paper we describe content focused, practical procedures to establish benchmarked performance standards for educational and other tests. To implement our procedures, we identify one or more external benchmark (e.g., NAEP or PISA performance levels, college and career readiness benchmarks); link to the external benchmarks to locate those…

This article describes a hybrid standard setting method that combines characteristics of the Angoff (1971) and Bookmark (Mitzel, Lewis, Patz & Green, 2001) methods. The proposed approach utilizes strengths of each method while addressing weaknesses. An ordered item booklet, with items sorted based on item difficulty, is used in combination…

Web-based standard setting holds promise for reducing the travel and logistical inconveniences of traditional, face-to-face standard setting meetings. However, because there are few published reports of setting standards via remote meeting technology, little is known about the practical potential of the approach, including technical feasibility of…

This study examined the relationships between the evaluations obtained from standard setting panelists and changes in ratings between different rounds of a standard setting study that involved setting standards on different language versions of an exam. We investigated panelists' evaluations to determine if their perceptions of the standard…

Psychometricians continue to introduce new approaches to setting cut scores for educational assessments in an attempt to improve on current methods. In this paper we describe the Item-Descriptor (ID) Matching method, a method based on IRT item mapping. In ID Matching, test content area experts match items (i.e., their judgments about the knowledge…