Agreement between two independent reviews of each of 278 manuscripts was compared on an overall recommendation and on specific rating items. Agreement between reviewers on separate dimensions, the unweighted sum of the dimensions, and various weighted sums was no better than that for the overall recommendation itself. (SLD)

The subjective judgment of observers was used to assess verbal creativity. Students, aged 5-10, told a story to accompany a picture series. Teachers rated the stories relative to one another. Interjudge reliability of the creativity measure was highly satisfactory. Two subsequent studies affirmed the results, with slightly lower interjudge…

Reviews several scales used to evaluate competency of psychotherapists. Discusses concerns about interrater reliability and predictive validity of scales. Considers competency a state-like variable, with therapists demonstrating higher competence when they skillfully treat patients across range of difficulty levels. Contends that development of…

The Triple Jump, a procedure for assessing students' problem-based learning, is applied to assessment of inquiry-based learning in a graduate course. Results suggest the need for more research into interrater reliability and other characteristics of the exercise. Some simple strategies for making the instrument cost effective are offered. (MSE)

A performance-standard setting procedure termed judgmental policy capturing (JPC) and its application are described. A study involving 12 panelists demonstrated the feasibility of the JPC method for setting performance standards for classroom teachers seeking certification from the National Board for Professional Teaching Standards. (SLD)

Professionals (n=201) working with child sexual abuse victims rated the normalcy of various behaviors with anatomical dolls for children ages two to five. Respondents agreed that overtly sexual behaviors were abnormal for nonabused children, but ratings of ambiguous behaviors varied depending on respondent's profession, gender, and years of…

This study tested the interrater reliability of the Modified Ashworth Scale in measuring upper and lower limb spasticity in 34 hemiplegic adult patients examined by 2 physiotherapists and 2 doctors. Findings indicated satisfactory reliability for upper limb spasticity but less satisfactory results for lower limb spasticity. (DB)

To investigate the validity of anesthesiologist certification, 146 anesthesiology program directors were asked whether they would permit each of their graduating residents to complete 3 increasingly complex anesthetic regimens to the directors themselves and rate residents on specific skills. Director responses generally correspond to…

Seventy physicians and two nurse practitioners rated colposcopic photographs. Results showed that leaders in the field of child sexual abuse assessment made significantly more accurate assessments than pediatricians, pediatric and family practice residents, and intern physicians. Predictors of agreement with standard assessments, although weak,…

A modification of a test of the equality of nonindependent alpha reliability coefficients is proposed. It avoids the limitation that the product of the number of test parts times the number of subjects be quite large. Monte Carlo studies indicate that this test can be used in comparing interrater reliabilities. (SLD)

This study assessed the psychometric properties of two subscales of the Adelaide Behaviour Disorder Scale that have been hypothesized to describe conduct problems and emotional problems of adults with intellectual disability. Criterion scores for identifying individuals needing clinical intervention were established and validated against…

Business undergraduates (n=261) participating in an assessment center simulation were evaluated by graduate students and faculty. Assessor-peer and assessor-self ratings lacked convergent and divergent validity, but self-peer ratings had both. (SK)

Using a recruitment test for Korean teachers, the use of the Rasch measurement model to control the effects of judge variable on the grading of essay-type items is examined. Ways of minimizing the variation of grading due to judge severity and reducing the number of judges without threatening objectivity of ability measurements are presented.…

This study assessed the psychometric properties of the Environmental Rating Scale, a measure specifically designed to assess residential treatment programs for individuals with autism. The measure's reliability was demonstrated by assessments of the internal consistency, stability, and interrater reliability. Preliminary analysis of validity…

Generalizability of performance-based assessment scores across raters and tasks is examined, focusing on implications of generalizability analyses for specific uses and interpretations of assessment results. Although it seems probable that assessment conditions, task characteristics, and interactions with instructional experiences affect the…