The purpose of this study was to examine rater, domain, and gender influences on the assessed quality of student writing using weighted and unweighted scores. Twenty rates were randomly selected from a group of 87 operational raters contracted to rate essays as part of the 1993 field test of the Georgia High School Writing Test. All of the raters…

The purpose of this study was to describe the Rasch model for measurement and apply the model to examine the relationship between raters, domains of written compositions, and student writing ability. Twenty raters were randomly selected from a group of 87 operational raters contracted to rate essays as part of the 1993 field test of the Georgia…

The Michigan accountability system, based on statewide objectives-referenced testing at two grade levels generated considerable controversy, including this response to criticisms of an evaluation of the program by a panel of three persons. The contract under which the evaluation was performed is quoted to show that the authors of the report were…

A study was conducted to evaluate the effect of different modes (modalities) of assigning raters to test items. The impact on total constructed response (c.r.) score, and subsequently on total test score, of assigning a single versus multiple raters to an examination reading of a student's set of c.r. responses was evaluated for several mixed-item…

Differences in the Texas Teacher Appraisal System scores of teacher subgroups over 2 years were examined for 2,366 teachers for scores on individual domains, sums of scores of the 1st 4 domains, and overall summary performance scores, as well as appraiser differences. Implications for teacher evaluation are discussed. (SLD)

The necessity of controlling the order in which trained essay raters for a statewide writing assessment program receive student essays was studied. The underlying theoretical question concerns possible rater bias caused by raters reading long strings of essays of homogeneous quality; this problem is usually referred to as context effect or…

This document presents information to trainers already trained in the Registered Holistic scoring method. It has been developed to assist these trainers in training others in the procedures and the criteria of the new scoring method. The Registered Holistic scoring method was designed and field tested for the ninth grade writing sample component…

The Maryland School Performance Assessment Program (MSPAP) is a relatively new, statewide performance assessment of students in grades 3, 5, and 8. When first administered in May of 1991, the MSPAP included a battery of performance assessment tasks designed to generate written or drawn responses to reading texts. This study evaluated selected…

A standard-setting procedure was developed for the Georgia Teacher Certification Testing Program as tests in 30 teaching fields were revised. A list of important characteristics of a standard-setting procedure was derived, drawing on the work of R. A. Berk (1986). The best method was found to be a highly formalized judgmental, empirical Angoff…

This report presents the results of an initial, iterative performance standards-setting (SS) task of a comprehensive on-the-job statewide teacher assessment system--the System for Teaching and Learning Assessment and Review (STAR). The 1990-91 STAR assesses and makes inferences about the quality of teaching and learning on sets of assessment…

This paper presents the results of ongoing analyses of the reliability of the System for Teaching and Learning Assessment and Review (STAR) as a comprehensive measure of classroom teaching and learning for making teacher certification decisions. Focus was on the effects of high stakes assessment conditions for certification on the generalizability…

Analyses of 10 raters' reasoning during think-aloud interviews provided evidence to support a model of the fundamental processes involved in rating standards-based, nonprescriptive portfolios. This process model provides a framework within which to conceptualize sound-rater reasoning and to identify reasoning that distorts the meaning of scores.…

The New Standards Project is a joint effort of the Learning Research and Development Center (LRDC) at the University of Pittsburgh (Pennsylvania) and the National Center on Education and the Economy toward creation of a national examination system based on performance assessments. This study explored the feasibility of comparing performance on…

This paper extends the factor analytic research of C. W. Swartz and others (1990) on the North Carolina Teaching Performance Appraisal Instrument (NCTPAI), which contains 28 teaching practices grouped into 5 functions (management of instructional time, management of student behavior, instructional presentation, instructional monitoring, and…

A formative evaluation of a pilot review and appeal process for the Maryland Writing Test (MWT) is described. The MWT is a large-scale direct assessment of writing. A three-year pilot phase was to culminate in the implementation of an operational procedure for the review and appeal of scores impacting a pass/fail decision for examinees. The MWT…