Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing…

This paper addresses issues raised in two recent papers published in this journal about the UK "Association of Business Schools' Journal Quality Guide (ABS Guide)". While much of the debate about journal rankings in general, and the "ABS Guide" in particular, has focused on the construction, power and (mis)use of these…

College rankings create much talk and discussion in the higher education arena. This love/hate relationship has not necessarily resulted in better rankings, but rather, more rankings. This paper looks at some of the measures and pitfalls of the current rankings systems, and proposes areas for improvement through a better focus on teaching and…

In many large-scale assessment programs, achievement level descriptors (ALDs) provide a critical role in communicating what scores on the assessment mean and in interpreting what examinees know and are able to do based on their test performance. Based on their test performance, examinees are often classified into performance categories. The…

While the ranking of higher education institutions (HEIs) has become more and more popular, there are increasing concerns about the quality of such ranking. In response to such legitimate expectations, in May 2006, the International Ranking Expert Group (IREG) developed and endorsed a guideline document--the Berlin Principles on Ranking of Higher…

"Classification" and "ranking" are tools for comparing colleges and universities, albeit for different purposes. Although "classification" focuses on identifying similarities rather than ordering institutions, it is closely intertwined with "ranking", whether directly through its content, structure, and…

Some of the different approaches to standard setting are discussed. Brief comments and references are offered concerning strategies that rely primarily on the use of expert judgment. Controversy surrounds methods that use expert judges, as well as those using test groups to set scores empirically. A minimax procedure developed by H. Huynh, an…

A solution is offered to problems associated with the inequality in the manipulability of probabilities of classification errors of masters versus nonmasters, based on competency test results. Eschewing the typical arbitrary establishment of observed-score standards below 100 percent, the solution incorporates a self-correction of wrong answers.…

We look at some of the theoretical and methodological issues underlying international university ranking systems and, in particular, their conceptual connection with the idea of excellence. We then turn to a critical examination of the two best-known international university ranking systems--the "Times Higher Education Supplement (THES)" World…

This paper reports on a standard-setting method designed for complex performance assessments with multiple performance categories. The method studied, the Analytical Judgment Method, involves panelists' making analytical classification decisions for each of the test's components individually. It also allows for discussion and reconsideration of…

The new e-learning generation depends on Semantic Web technology to produce learning objects. As the production of these components is very costly, they should be produced and registered once, and reused and adapted in the same context or in other contexts as often as possible. To produce those components, developers should use learning standards…

This paper presents and evaluates a new procedure to classify examinees into meaningful categories based on their responses to (performance on) items comprising a test instrument. The proposed procedure uses the data analytic technique of cluster analysis to partition the examinee population into homogeneous groupings. This procedure was applied…

This paper considers issues relating to the measurement of skill for national statistical purposes. It draws upon the work program and research underlying the revision of the national occupational classification for the United Kingdom (UK), SOC90 (Standard Occupational Classification introduced in 1990). The report's introduction states the…

Commissioned by the Institute of Museum and Library Services (IMLS) to assess the status of data collection activities in the museum community, this report reinforces recommendations made by a similar 1998 study, addressing the need for systematic data collection to inform public policy and museum management decision-making. The report examines a…

Methods that might be used to establish standards of performance that will permit the National Board for Professional Teaching Standards to classify candidate teachers as highly accomplished (worthy of National Board certification) or less than highly accomplished (not worthy of certification) are contrasted. Plans for research on these issues are…