A previous paper proposed a generalized family of coefficients of relational agreement for multiple judges. It focused on the concept of empirically meaningful relationships. This paper presents an ordinal coefficient of relational agreement as a special case of the generalized family. The proposed ordinal coefficient encompasses other ordinal…

Three studies investigating the existence of sex bias in the grading of undergraduate students, by examining interrater reliability for blind and non-blind grading, are reported. Negative evidence found in the results and the confusing picture presented by previous research indicate little firm evidence of sex bias in grading. (Author/MSE)

The study, with 79 rehabilitation therapists evaluating 21 single-subject graphs, found that the low interrater agreement often associated with visual analysis of single-subject data may be improved by simple supplements (such as trend lines) to visually inspected charts. (Author/DB)

Investigated referral question bias on school psychologists' classification decisions across different types of cases. Findings from 175 school psychologists who classified 5 case studies on basis of scores from intelligence, achievement, and behavioral measures revealed lack of congruence among respondents' classification decisions that was more…

Advanced-level English and chemistry examinations in Britain were graded in several conditions: with or without examinees' names; with male or female names; and with "male" or "female" handwriting. No consistent evidence of gender bias was found in the marking. (SK)

The Delaware Performance Appraisal Process includes indicators that contain subjective terminology and are open to various interpretations, leading to concerns about interrater reliability and professional staff morale. The Lake Forest School District developed a two-pronged approach that retains the three-observation method while eliminating the…

Lecturers at a British university participated in two workshops to examine the consistency of assessments of undergraduates' work. Use of both analytical and global quality measures, when clearly understood by the raters, improved assessment practices. Ongoing discussion of evaluation criteria was recommended. (SK)

Evaluated Angoff ratings from eight different occupational licensing examinations through generalizability theory to estimate the optimal number of raters. Results indicate that approximately 10 to 15 raters is an optimal target range. (SLD)

Ratings of child involvement in manual-based cognitive-behavioral treatment for anxiety were associated with the absence of primary anxiety diagnosis and reductions in impairment ratings at posttreatment for 59 children with anxiety (ages 8-14 years). Good-to-excellent interrater reliability was established for the independent ratings of 237…

This article presents a formula for weighted kappa in terms of rater means, rater variances, and the rater covariance that is particularly helpful in emphasizing that weighted kappa is an absolute agreement measure in the sense that it is sensitive to differences in rater's marginal distributions. Specifically, rater mean differences will decrease…

This study examined the sexual messages depicted in music videos aired on MTV, MTV2, BET, and GAC from August 2, 2004 to August 15, 2004. One-hour segments of music videos were taped daily for two weeks. Depictions of sexual attire and sexual behavior were analyzed via a four-page coding sheet (interrater-reliability = 0.93). Results indicated…

Since the 1920s, textbook critics have maintained that textbooks should offer a homogenous editorial approach, including an acknowledgment of a mix of author opinion and scholarly research. Several researchers indicated that some textbooks are not homogenous. The purpose of this quantitative content analysis study was to examine whether…

Behavioral aspects of self-regulation, including controlling and directing actions, paying attention, and remembering instructions, are critical for successful functioning in preschool and elementary school. In recent years, several direct assessments of these skills have appeared, but few studies provide complete psychometric data and many are…

Background: Many speech and language therapists work in a multilingual environment, making cross-linguistic studies of speech disorders clinically and theoretically important. Aims: To investigate the effect of listeners' linguistic background on their perceptual ratings of hypernasality and the reliability of the ratings. Methods &…

The standards of the National Council for Accreditation of Teacher Education require that teacher education programs document teacher candidates' effectiveness across different domains. Most challenging for our program is the directive to document the impact that candidates have on preK-12 student learning. To gather information on the effects…