The traditional kappa statistic in assessing interrater agreement is not adequate when multiraters and multiattributes are involved. In this article, latent trait models are proposed to assess the multirater multiattribute (MRMA) agreement. Data from the Third International Mathematics and Science Studies (TIMSS) are used to illustrate the…

This paper suggests that in addressing the issue of parent-professional congruence in child assessment, researchers should avoid focusing on the conventional aspects of interrater reliability and rater interchangeability, but rather should focus on the reliability of the pooled assessment information from parents and professionals. A…

This paper dealt with the use of classroom observation data for formative evaluation purposes, and with a research project in which scores based on observed performance of teachers in secondary school algebra and English classes were compared with efficiency scores based on an input-output model. The model, using Data Envelopment Analysis (DEA)…

This study describes a computer based, performance rating information processing system, performance rating theory, and programs for the application of the theory to obtain ratings free from the effects of reviewer stringency in reviewing abstracts of conference papers. Originally, the Performance Rating (PR) System was used to evaluate the…

A total of 1,071 rating sheets were completed by individual reviewers evaluating abstracts submitted to Sigma Theta Tau International Honor Society for the International Research Congress in Edinburgh, Scotland. Only 972 sheets contained usable data. Reviewers indicated a total of 12 ratings with possible comments for each abstract on machine…

Five common play criteria were applied by subjects to a videotape of nursery school children's behavior, rated separately for occurrence of play. The idea that play is best predicted by a combination of criteria was supported by the finding that, when more criteria occurred simultaneously, the more certainly a judgment of play was implied.…

In circumstances where judges are required to make ratings of performance, it is usually required to have two or more raters who are trained to agree on independent ratings of the same performance. It is suggested that such a requirement may produce the paradox of attenuation associated with item analysis, in which too high a correlation between…

Cason and Cason's model of performance rating was used to determine the extent to which variation in reviewer standards affected the reliability and validity of the program review process used to select papers for inclusion in the annual program. Data analyzed were the overall recommendation for acceptance and ratings on seven quality criteria…

The evaluation team of the Lincoln Public Schools (Nebraska) used the multi-attribution utility technology (MAUT) approach to prioritize potential evaluation projects. The priorities were used to allocate resources to the district's most important projects, and to eliminate or scale down less important projects. The problem was caused initially…

This study examined the difficulty level of essay topics used in the large-scale assessment of writing in relation to five different scoring models, and sought to determine what effects the scoring models would have on passing rates. In model one, examinee's score is the direct result of a score assigned by the reader or the sum of scores assigned…

This paper reports the results from an investigation of a method for systematic development of a procedure for assessing the administrator and instructional leader roles in implementation of an innovative school program. The purpose of the study was to design and field-test a feasible and useful approach to examining the degree of…