Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing…

A solution is offered to problems associated with the inequality in the manipulability of probabilities of classification errors of masters versus nonmasters, based on competency test results. Eschewing the typical arbitrary establishment of observed-score standards below 100 percent, the solution incorporates a self-correction of wrong answers.…

The problem of standard setting on language proficiency tests is often approached by the use of norms derived from the group being tested, a process commonly known as "grading on the curve." One particular problem with this ad hoc method of standard setting is that it will usually result in a fluctuating standard dependent on the particular group…

Characterizing this special "Kappan" section on test-based accountability as a cautionary plea, this article sees five themes: tests seldom measure what matters; standardization gets confused with standards; tests constitute a limited reform lever; test-based accountability plans often misplace trust and protection; and the…

This paper reports on the promises and pitfalls of the recently legislated basic skills testing requirement for teaching or service credential applicants in California. The California Basic Education Skills Test (CBEST), developed by Educational Testing Service from specifications set by Advisory Board to the State Department of Education, tests…

If annual performance assessments are to yield results that can be compared from year to year, many technical problems must be addressed. It is essential that tests to be equated measure the same construct. Methods of equating performance assessment scores, ways of equating system assessments, and standard setting are discussed. (SLD)

Ways to use standard-setting data to adjust cutoff scores on examinations are reviewed. Ten sources of information to be used in determining standards are listed. The decision to modify passing scores should be based on these types of information and consideration of adverse impact or rating process irregularities. (SLD)

Confusion in the terminology used in criterion-referenced measurement specifications and development and standard setting and the attendant role of cut-off scores are shown to need practical clarification through psychometric research on test applications and consequences. (CM)

Possible sources of intrajudge inconsistency in standard setting are reviewed, and approaches are presented to improve the accuracy of rating. Procedures for providing judges with feedback through discussion or computerized communication are discussed. Monitoring and maintaining judges' consistency throughout the rating process are essential. (SLD)

Issues concerning the selection of judges for standard setting are discussed. Determining the consistency of judges' recommendations, or their congruity with other expert recommendations, would help in selection. Enough judges must be chosen to allow estimation of recommendations by an entire population of judges. (SLD)

Examines the appropriateness of a cut-off score derived from the Angoff procedure for a reading test in the United Kingdom. Shows that the recommended cut-off score is too low. Suggests ways that standard setting might draw on a range of information to produce appropriate and rationally defensible cut-off scores. (DSK)

Issues related to the establishment of formal standards for student achievement on large-scale assessments are discussed, and two approaches to standard setting that have been used in Saskatchewan (Canada) are described. In spite of the general success of these programs, numerous issues to consider in future evaluations are raised. (SLD)

A scoring rubric transition study is described. It was designed to evaluate possible drift in scoring the Maryland Writing Test from year to year (when using a modified holistic scoring method), to evaluate strategies for revising swing rubrics from narrative and explanatory writing while maintaining original scoring standards, and to establish…

A standard-setting procedure was developed for the Georgia Teacher Certification Testing Program as tests in 30 teaching fields were revised. A list of important characteristics of a standard-setting procedure was derived, drawing on the work of R. A. Berk (1986). The best method was found to be a highly formalized judgmental, empirical Angoff…

Claims that national tests will improve student learning are explored, asking whether national examinations will ensure high-quality instruction and greater student learning and whether tests developed to meet urgent political deadlines will retain essential features of authentic curriculum-driven assessments. Part I presents research evidence on…