This study describes the four-phase process used in developing the "Employer Openness Survey" (EOS). The EOS is an 18-item instrument designed to measure the openness of employers to hiring, accommodating, and promoting workers with disabilities. During the first phase, the authors generated potential questions and pilot-tested them with…

The issue of arriving at agreement over outcomes in summative assessment of portfolios has been a major concern, given the complexity of the assessment task, the educational and political context, and the widespread and growing use of portfolios in higher education. This article examines research findings in this area. The discussion takes place…

When raters assign different scores to a performance task, a method for resolving rating differences is required to report a single score to the examinee. Recent studies indicate that decisions about examinees, such as pass/fail decisions, differ across resolution methods. Previous studies also investigated the interrater reliability of…

This article describes the development of assessment criteria for specific aspects of examination answers and coursework essays in psychology. The criteria specified the standards expected for seven aspects of students' work: addressing the question, covering the area, understanding the material, evaluating the material, developing arguments,…

Many studies have indicated that at least 2 raters should score writing assessments to improve interrater reliability. However, even for assessments that characteristically demonstrate high levels of rater agreement, 2 raters of the same essay can occasionally report different, or discrepant, scores. If a single score, typically referred to as an…

Two studies assessed the reliability and utility of the Stimulus Preference Coding System (SPCS) to measure approach, avoidance, and happy and unhappy behaviors in persons with developmental disorders. Study 1 took place in an institutional setting. The nine participants were all adults with mental retardation and multiple associated disabilities.…

Play assessment is gaining attention as a measure of the developing skills of young children. The procedures and methods of coding child behaviours vary considerably across researchers and practitioners. Because of this, definitive statements about the use of play assessment cannot be made without further research. The present study is an attempt…

Objective: Given the present emphasis on accountability and maintaining quality, the objective of this study was to develop, apply, and assess the reliability of a fidelity rating instrument for consumer-operated services--a promising model, but one for which fidelity criteria are not yet established. Method: Based on observations, documents, and…

The piece that follows had its first hearing as part of a symposium at the 1990 annual meeting of the American Educational Research Association in Boston. Sigrun Gudmundsdottir and Grace Grant were at the podium, presenting their two narrative accounts of one schoolteacher's practice. The schoolteacher was present as well, through the medium of a…

The application range of Cohen's Kappa is extended to the field of sequential observation data, where omission mistakes of an observer may often occur. It is shown how the omission mistakes can be incorporated into the calculation of the Kappa coefficient without violating the statistic it is based on. The enhanced coefficient is termed Kappa…

One of the ways to improve the reliability in perceptual voice quality rating is to provide listeners with external anchors. A paired comparison matching paradigm using synthesized Cantonese voice stimuli that covered a range of rough and breathy qualities were used to investigate the rating reliability. Twenty-five speech pathology students rated…

The purpose of this study was to determine what trends exist in the identification and description of participants used in data-based studies published in "Adapted Physical Activity Quarterly" and the "Journal of Teaching in Physical Education". Data were analyzed using frequency counts for journals and time periods from the 1980s to 2005 with…

Educators often have to choose among different types of rating scales to assess second-language (L2) writing performance. There is little research, however, on how different rating scales affect rater performance. This study employed a mixed-method approach to investigate the effects of two different rating scales on EFL essay scores, rating…

This study examined the impact of treatment adherence and therapist competence on treatment outcome in a controlled trial of individual cognitive-behavioral therapy (CBT) and multidimensional family therapy (MDFT) for adolescent substance use and related behavior problems. Participants included 136 adolescents (62 CBT, 74 MDFT) assessed at intake,…

The training of raters for writing assessment through web-based programmes is emerging as an attractive and flexible alternative to the conventional method of face-to-face training sessions. Although some online training programmes have been developed, there is little published research on them. The current study aims to compare the effectiveness…