Examined the effects of order of stimulus presentation on observer ratings of counseling performance. Results revealed a statistically significant interaction between quality of performance and the order in which the performances were rated. (Author/ABB)

Analysis of gymnastics judges scores of their own and other countries' gymnasts' performance during the 1984 Olympic Games indicated that the judges were biased in favor of their own country's gymnasts. (Author/CB)

Three experienced neuropsychologists rated brain damaged and control subjects for brain damage using the Halstead-Reitan Battery and the Luria-Nebraska Neuropsychological Battery. Using either battery, raters were accurate in judging the presence of brain damage. There was a high degree of consistency between raters and test batteries when both…

This program computes multiple judge reliability levels under the following conditions. (1) different sets of judges perform the ratings; (2) the number of judges is a constant; and (3) the scale of measurement is nominal. (Author)

Investigated the interscorer reliability between a novice and a professional psychologist for the Minnesota Percepto-Diagnostic Test-Revised (MPDT-R), using a sample of 30 individuals. Results indicated that for three of the four MPDT-R scores there was a significant positive correlation between expert and novice scoring criteria. (JAC)

Fleiss' popular multirater kappa is known to be influenced by prevalence and bias, which can lead to the paradox of high agreement but low kappa. It also assumes that raters are restricted in how they can distribute cases across categories, which is not a typical feature of many agreement studies. In this article, a free-marginal, multirater…

Used Monte Carlo simulation to compare the properties of a goodness-of-fit (GOF) procedure and a test statistic developed by E. Bartfay and A. Donner (2001) to the likelihood ratio test in assessing the existence of extra variation. Results show the GOF procedure possess satisfactory Type I error rate and power. (SLD)

Generalizability Theory can be used to assess reliability in the presence of multiple sources and different types of error. It provides a flexible alternative to Classical Theory and can handle estimation of interrater reliability with any number of raters. (SK)

Developed method for aggregating psychodynamic formulations of independent clinicians. Panels of clinicians observed videotaped interviews of patients and wrote individual formulations which were combined into consensual formulation. Other clinical raters read each consensual formulation and judged whether each problem was apt to be distressing…

To test model of general managerial effectiveness, superiors (N=271), subordinates (N=605), and peers (N=469) rated 344 managers. Study designed to test three specific hypotheses on criterion type and criterion weights found consensus in effectiveness models of superiors, subordinates, and peers. Consensus among different raters was high on both…

New coefficients of agreement are suggested for the measure of intraclass consistency between observations on two variables. The coefficients are derived from a general coefficient for measuring intraclass dependence in a bivariate analysis context. Various coefficients for the univariate agreement analysis are shown to be cases of the suggested…

Examined interrater reliability of diagnoses made on basis of structured interview for psychiatric patients with and without psychoactive substance use disorders (PSUDs). Results from 47 pairs of ratings by 9 clinical interviewers revealed that interrater reliability for non-PSUD psychiatric diagnoses was quite high when patient had no diagnosable…

A coding method for measuring reflective thinking in student journals was tested twice, demonstrating acceptable reliability among evaluators and supporting the precision of the guidelines for coding. Coding categories were as follows: habitual action, introspection, thoughtful action, content reflection, process reflection, content and process…

To examine the interrater reliability of the Ruff Figural Fluency Test (RFFT) (R. Ruff, 1988), 124 college students completed the measure and scored RFFT test protocols. Results indicated substantial interscorer reliability on the RFFT, particularly for number of unique designs. Reliability was lower for scoring perseverative errors and error…

Three commercial job evaluation firms rated the same set of 27 jobs. Statistical analysis indicated that evaluators differed in which job traits they used to evaluate inherent job worth. Comparable worth may thus be sensitive to the choice of evaluator. (Contains 24 references.) (Author/SK)