The credibility of standard-setting cut scores depends in part on two sources of consistency evidence: intrajudge and interjudge consistency. Although intrajudge consistency feedback has often been provided to Angoff judges in practice, more evidence is needed to determine whether it achieves its intended effect. In this randomized experiment with…

Purpose: The study of the problems hindering improvement of the system of training through standardization of qualification requirements is relevant because, in a globalized system of teaching staff and high rates of migration, not only national but also international requirements for employees should be considered. This increases the…

This experimental study is aimed at examining the effect of involving pupils in primary education in setting assessment criteria and standards on their appraisal of a peer's work and their peer feedback style. It is expected that our intervention will lead to (1) a criteria-referenced appraisal and (2) more final vocabulary, which is associated…

The discussion takes up the common theme across the seven contributions of this special issue, namely the CEFR's suitability as a basis for setting university entrance standards. The special issue allows insights into this theme from a multitude of contexts, languages and perspectives, including data from stakeholders, students, tests and their…

The past decade has seen an emerging interest in mapping (aligning or linking) test scores to language proficiency levels of external performance scales or frameworks, such as the Common European Framework of Reference (CEFR), as well as locally developed frameworks, such as China's Standards of English Language Ability (CSE). Such alignment is…

Institutions of higher education commonly employ a conjunctive standard setting strategy, which requires students to resit failed examinations until they pass all tests. An alternative strategy allows students to compensate a failing grade with other test results. This paper uses regression discontinuity design to compare the effect of first-year…

When the safety of the public is at stake, it is particularly relevant for licensing and credentialing exam agencies to use defensible standard setting methods to categorize candidates into competence categories (e.g., pass/fail). The aim of this study was to gather evidence to support change to the Comprehensive Osteopathic Medical Licensing-USA…

This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM-ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom…

This article presents the results of an interpretive policy analysis of China's Ministry of Education Standards (2013) for the professional practice of principals. In addition to revealing the evolution of the evaluation of principals in China and the processes by which this policy is formulated, a comparative analysis was conducted to compare it…

The ways in which examination standards are conceptualised and operationalised differently across nations has not been given sufficient attention. The international literature on standard-setting has been dominated by the psychometrics tradition. Broader conceptualisations of examination standards have been discussed in the literature in England,…

Innovation has become an increasingly important concept for Australian businesses in enabling them to continue to compete in an international market, but where does vocational education and training (VET) sit in this? To date, VET has been largely overlooked by the National Innovation and Science Agenda as a potential contributor in this field.…

This article introduces the Diagnostic Profiles (DP) standard setting method for setting a performance standard on a test developed from a cognitive diagnostic model (CDM), the outcome of which is a profile of mastered and not-mastered skills or attributes rather than a single test score. In the DP method, the key judgment task for panelists is a…

Cut-score setting processes are used to establish the passing standards for all kinds of tests in education and for credentialing. While experts use their best efforts to guide cut-score setting processes to generate valid and reliable results, cut-score participants often have a difficult time understanding the standard at which the cut score is…

In language programs, it is crucial to place incoming students into appropriate levels to ensure that course curriculum and materials are well targeted to their learning needs. Deciding how and where to set cutscores on placement tests is thus of central importance to programs, but previous studies in educational measurement disagree as to which…

The decision to pass or fail a medical student is a "high stakes" one. The aim of this study is to introduce and demonstrate the feasibility and practicality of a new objective standard-setting method for determining the pass/fail cut-off score from borderline grades. Three methods for setting up pass/fail cut-off scores were compared: the…