A new standard setting approach is introduced, called the cognitive components approach. Like the Angoff method, the cognitive components method generates minimum pass levels (MPLs) for each item. In both approaches, the item MPLs are summed for each judge, then averaged across judges to yield the standard. In the cognitive components approach,…

This paper discusses recent developments in language testing. It begins with a review of the traditional criteria that are applied to all measurement and outlines recent emphases that derive from the expanding range of stakeholders. Drawing on Alderson's seminal work, criteria are presented for evaluating communicative language tests. Developments…

Suggestions are provided for future research in item bias detection, reduction of essay-reader variation in setting cut-score levels, and limitations of equating theory. (TJH)

Differences in the Texas Teacher Appraisal System scores of teacher subgroups over 2 years were examined for 2,366 teachers for scores on individual domains, sums of scores of the 1st 4 domains, and overall summary performance scores, as well as appraiser differences. Implications for teacher evaluation are discussed. (SLD)

Whether adequate levels of interrater reliability could be obtained on a national, standardized examination using one examiner per observation was studied with 101 paired candidate observations on an examination for optometry. Results indicate that psychometrically sound judgments can be obtained with one examiner. (SLD)

Multifaceted Rasch analysis was used to determine whether bias was evident in the way a group of raters graded two different versions of an oral interaction test, undertaken by the same candidates. Results indicate that certain raters consistently rated the tape version of the test more harshly while others rated the live one more harshly. (10…

Issues concerning the selection of judges for standard setting are discussed. Determining the consistency of judges' recommendations, or their congruity with other expert recommendations, would help in selection. Enough judges must be chosen to allow estimation of recommendations by an entire population of judges. (SLD)

Training judges to generate item ratings in standard setting once the reference group has been defined is discussed. It is proposed that sensitivity to the factors that determine difficulty can be improved through training. Three criteria for determining when training is sufficient are offered. (SLD)

Medical school admission interview ratings for four entering classes (n=356 students) were compared with preadmission academic variables (admission test scores, undergraduate grades), student characteristics (age, gender, residence), and interviewer characteristics (gender, professional background, admission committee membership). Recommendations…

A study investigated the consistency of faculty and peer evaluations of the oral communication skills of 44 fourth-year pharmacology students. Substantial agreement between faculty and students was found. Peer evaluations were independent of their own communication skills. In addition, a significant correlation between oral and written…

Examined the generalizability of written-response scores on the English 30 diploma examination administered to Alberta 12th-grade students. Student scores differed as a function of rater, but this variance component was small across two tasks and two administrations; score generalizability was high using a two-rater system; and scale variability…

Two studies involving 368 elementary and high school students and 29 readers were conducted to investigate reader consistency, score reliability, and reader time requirements of three hands-on science performance tasks. Holistic scores were as reliable as analytic scores, and there was a high correlation between them after they were disattenuated…

Presents a novel application of analysis of variance (ANOVA) techniques to compare the reliability of multiple peer ratings with single teacher ratings. Uses rating data from two different courses, both involving multiple peer and individual teacher ratings that were used to assess student contributions to group process work. Discusses…

Each January since 1997, "Education Week," the K-12 industry's newspaper of record, has issued its "Quality Counts" report, ranking states by, among other things, the "equity" of their school finances. On the other hand, every fall since 2001, the "Education Trust," a national organization devoted to closing the achievement gap in public schools,…

We examined the interrater reliability of the "Child and Adolescent Needs and Strengths-Mental Health" (CANS-MH) scale among researchers and between researchers and clinicians. All children presenting to a treatment facility for either protective or mental health needs were eligible to be included in the study. As part of standard assessment…