This paper introduces an application of cross-classified item response theory (IRT) modeling to an assessment utilizing the embedded standard setting (ESS) method (Lewis & Cook). The cross-classified IRT model is used to treat both item and person effects as random, where the item effects are regressed on the target performance levels (target…

This study evaluated the efficacy of two proposed methods in an operational standard-setting study conducted for a high-stakes language proficiency test of the U.S. government. The goal was to seek low-cost modifications to the existing Yes/No Angoff method to increase the validity and reliability of the recommended cut scores using a convergent…

A common belief is that the Bookmark method is a cognitively simpler standard-setting method than the modified Angoff method. However, a limited amount of research has investigated panelist's ability to perform well the Bookmark method, and whether some of the challenges panelists face with the Angoff method may also be present in the Bookmark…

The objective structured practical examination (OSPE) is a reliable assessment of practical skills in anatomy teaching. It is often administered as low-stake assessments to track progress at multiple time points in anatomy curricula. Standard-setting OSPEs to derive a pass mark and to ensure assessment quality and rigor is a complex task. This…

An Angoff standard setting study generally yields judgments on a number of items by a number of judges (who may or may not be nested in panels). Variability associated with judges (and possibly panels) contributes error to the resulting cut score. The variability associated with items plays a more complicated role. To the extent that the mean item…

In this study, the efficiency of various random sampling methods to reduce the number of items rated by judges in an Angoff standard-setting study was examined and the methods were compared with each other. Firstly, the full-length test was formed by combining Placement Test 2012 and 2013 mathematics subsets. After then, simple random sampling…

The aim of this study was to compare, by simulation, the accuracy of mapping a cut-score from one test to another by expert judgement (using the Angoff method) versus the accuracy with a small-sample equating method (chained linear equating). As expected, the standard-setting method resulted in more accurate equating when we assumed a higher level…

The purpose of the current examination was to preliminarily suggest severity standards for the recently revised Cognitive Test Anxiety Scale-Second Edition (CTAS-2). Participants responded to the CTAS-2, Motivated Strategies for Learning Questionnaire (MSLQ), and FRIEDBEN Test Anxiety Scale. Using both latent class and cluster analyses, we were…

Purpose: University placement test is an important admission policy priority in Nigeria, because it serves as a university-based selection criterion for placement of students into undergraduate programs in Nigeria. Although recently attention have been shifted on the call to develop a standard content and standardize the test, yet attention has…

This article discusses regression effects that are commonly observed in Angoff ratings where panelists tend to think that hard items are easier than they are and easy items are more difficult than they are in comparison to estimated item difficulties. Analyses of data from two credentialing exams illustrate these regression effects and the…

Within initial teacher education there is increasing pressure to enhance the use of assessment data to support students to improve their knowledge and skills, and to determine what standards they meet upon graduation. For such data to be useful, both programme designers and students require meaningful and comprehensive assessment reports on…

Measurement and evaluation of students' achievement are an important aspect to make sure that students really understand the course content and monitor students' achievement level. Performance is not only reflected from the numbers of high achievers of the students, but also on quality of the grade obtained; does the grade "A" truly…

The determination of Pass/Fail decisions over Borderline grades, (i.e., grades which do not clearly distinguish between the competent and incompetent examinees) has been an ongoing challenge for academic institutions. This study utilises the Objective Borderline Method (OBM) to determine examinee ability and item difficulty, and from that…

Cut-score setting processes are used to establish the passing standards for all kinds of tests in education and for credentialing. While experts use their best efforts to guide cut-score setting processes to generate valid and reliable results, cut-score participants often have a difficult time understanding the standard at which the cut score is…

(Purpose) The purpose of our study was to assess the need for an external searchable resource to be used in conjunction with the American Board of Family Medicine's (ABFM) Maintenance of Certification for Family Physicians (MC-FP) Examination, discuss the philosophical question of whether an ESR should be allowed on the examination, and outline…