A method of estimating Rasch-model difficulty calibrations from judges' ratings of item difficulty is described. The ability of judges to estimate item difficulty was assessed by correlating estimated and empirical calibrations on each of four examinations offered by the American Association of State Social Work Boards. Thirteen members of the…

The L statistic of E. B. Page (1963) tests the agreement of a single group of judges with an a priori ordering of alternative treatments. This paper extends the two group test of D. W. Leitner and C. M. Dayton (1976), an extension of the L test, to analyze difference in consensus between two unequally sized groups of judges. Exact critical values…

Several issues relating to agreement estimates for different types of data from performance evaluations are considered. New indices of agreement are presented for ordinal level items and for summative scores produced by nominal or ordinal level items. Two sets of empirical data illustrate the performance of the two formulas derived to estimate…

This annotated bibliography was developed upon noting a deficiency of information in the literature regarding the training of raters for establishing agreement. The ERIC descriptor, "Interrater Reliability", was used to locate journal articles. Some of the 33 resulting articles focus on mathematical concepts and present formulas for computing…

Comparing the manner in which contestants' scores were tabulated at both the 1985 American Forensic Association National Individual Events Tournament (AFA-NIET) and National Forensic Association Individual Events Nationals (NFA-IEN), a study (1) examined whether a correlation exists between contestants placing in the quarterfinals with five…

Decision making about an observational recording system for family interaction research is crucial. Alternative coding-recording methods and combinations thereof are discussed, including: (1) paper-and-pencil on-site method; (2) video-tapes; (3) paper-and-pencil and mechanical coding devices; (4) transcripts; and (5) transcripts combined with…

Differences between the academic (American Council on the Teaching of Foreign Languages) and government (Foreign Service Institute) versions of the oral proficiency interview test are examined, and data from two studies of interrater reliability are presented and discussed. (MSE)

Differences in agreement between observers and analysts of naturalistic narrative data cause problems in observation research. This paper discusses the advantages and disadvantages of several possible solutions. (Author/GDC)

Five sets of paired comparison judgments were made concerning test item difficulty, in order to identify the most probable source of intrasensitivity in the data. The paired comparisons method was useful in providing information about sensitivity to stimulus differences, but less useful for assessing dimensionality of judgment criteria.…

Compared three methods of rating thoughts: self-rating by subjects, rating by experts with thoughts presented randomly, and rating by experts with thoughts presented in context among 107 students who listed their thoughts prior to giving a speech. Results indicated all three methods were equal in predictions of speech anxiety and performance.…

Presents an in-class activity which uses videotaped television shows to teach time sampling, event sampling, and trait rating techniques. Students responded favorably to this activity, and many reported that it increased their understanding of the different observation techniques. (Author/JDH)

Investigates the relative and interactive effects of rater-, and ratee-, relationship-, situational-, and group-level contingencies on peer assessments of open communication. Results suggest that, given certain procedural conditions, peer assessments are highly reliable and valid. Rater bias accounted for a relatively small amount of rating…

Compares peer ratings, supervisor ratings, and self-ratings of counseling performance. Earlier studies of the relationship of performance ratings from different sources have indicated some comparability across source of rating, particularly late in the training process. These results indicated considerable variability across sources of ratings…

Reports that coders were able to judge adequately the difference between immediate reward and delayed reward in news stories but not the difference between subcategories. (FL)

Used the intraclass correlation coefficient to estimate the interrater reliability of judgments of clinician and novice raters of depressed females (N=20) who took the Hamilton Rating Scale for Depression (HRSD). Expert and student raters both made reliable ratings on the HRSD. Criterion validity for student raters was also satisfactory.…