Observational ratings of student clinical performance are influenced by factors other than the quality of the performance. Individual raters may be more stringent or lenient than their colleagues. In this medical school setting, multiple raters evaluated each student. To reduce the influence of "error" due to differences among raters, each rater…

Two aspects of rater accuracy in performance assessment were analyzed: rater stringency/leniency, and rater consistency. Data were obtained from three administrations of an oral certification examination in a health profession. The examination consists of clinical cases in four content areas or subspecialities. A total of 364 candidates were…

An instrument for to assessing the extent to which postsecondary vocational education programs are competency-based consisted of 159 items grouped into 8 subscales corresponding to 4 standards: (1) identification and validation of competencies; (2) procedures for assessing student competencies; (3) instructional procedures; and (4) administrative…

The Center for the Study of Evaluation's (CSE) expository and narrative rating scales have been developed to meet the need for instructionally relevant methods for assessing students' writing competence. Research indicates that large numbers of raters can be trained in the use of these scales and that, during training and independent rating, they…

The purpose of this paper is to compare the precision of direct and indirect measures of writing assessment using the test information functions from a graded response Item Response Theory (IRT) model. Subjects were 192 sophomore English students from a parochial high school in Wisconsin. Both direct and indirect measures of writing ability were…

This study examines the general hypothesis that surveys can be used to elicit reliable and valid student ratings of the effectiveness of most aspects of the delivery of the academic programs in which they are enrolled, and reports data on test-retest reliability for upper division majors as well as on the reliability with which ratings…

The Wechsler Intelligence Scale for Children-Revised (WISC-R), one of the most commonly used tests of cognitive ability, is difficult to administer accurately. The purpose of this study was primarily to assess interrater agreement on the WISC-R Administration Observational Checklist (WAOC), a new observational instrument that can be used by an…

Classroom researchers are frequently urged to provide evidence of the reliability of their data. In the case of observational data, three approaches to this have emerged: observer agreement, generalizability theory, and measurement error. Generalizability theory provides the most powerful approach given an adequate data collection design, but…

The graded response (GR) model of Samejima (1969) and the partial credit model (PC) of Masters (1982) were fitted to identical writing samples that were holistically scored. The performance and relative benefits of each model were then evaluated. Writing samples were both expository and narrative. Data were from statewide assessments of secondary…

A total of 1,071 rating sheets were completed by individual reviewers evaluating abstracts submitted to Sigma Theta Tau International Honor Society for the International Research Congress in Edinburgh, Scotland. Only 972 sheets contained usable data. Reviewers indicated a total of 12 ratings with possible comments for each abstract on machine…

This report describes the procedures followed in scoring the October 1978 Florida Minimal Writing Production Skills Assessment and reports the results of that assessment. The assessment was conducted on a sample of Florida public school students in grades 3, 5, 8, and 11. Sections include descriptions of the rating scale and scorer's guide as well…

This study was designed to determine the feasibility of using the Teacher Performance Assessment Instruments (TPAI) to evaluate student teacher performance in northern Nevada by a more accurate and mutually acceptable method. Developed and validated by the University of Georgia, passing the TPAI is required for teacher certification in Georgia.…

Examined whether differences in counselors' retention of information about a female and a male client in a previous study would also be obtained with a noncounselor sample of 218 college students. In findings similar to those of the previous study, participants retained less information under the female client conditions, indicating a possible…

Second-year medical students (N=187) evaluated their own videotaped performances during physical assessment examinations. Peers and expert evaluators also rated the performances. Significant differences emerged across types of raters. Results show that these students do not accurately assess their own or peers' skills performances. (TJH)

This article describes procedures to be followed in developing a system for observation of teachers in the classroom and use of the observation to evaluate teachers. A list of criteria is presented, including various types of validity, measurement characteristics, and practicality characteristics for observation systems. (Author/LMO)