Raters judged essays on two occasions making it possible to address the question of whether multiple ratings, however obtained, represent the same true scores. Multiple ratings of a given rater did represent the same true scores, but ratings of different raters did not. Reliability, validity, and invalidity coefficients were computed. (Author/DWH)

Judgment analysis (JAN) and JAN-paired comparison (JAN-PC) were compared as a statistical policy-capturing strategy. The readability of 15 selected passages used in grades 1-6 was evaluated. More consistent results for each judge (five Arabic teachers of reading) and fewer policies were obtained with JAN-PC than with JAN. (TJH)

Results of an administration of the Language Proficiency Measure indicated that the interrater reliability was adequate, internal-consistency reliability estimates were high, concurrent validity coefficients were adequate, and the classification validity was acceptable. (Author/LMO)

Job-naive raters provided with job descriptive information made Position Analysis Questionnaire (PAQ) ratings which were validated against ratings of job analysts who were also job content experts. None of the reduced job descriptive information conditions enabled job-naive raters to obtain either acceptable levels of convergent validity with…

Examines the tendency of observers to make less reliable recordings of behavorial events when a calibrating observer is absent. Using four different multicategory systems, 26 experienced observers coded 200 hours of videotaped family interactions. Concludes that observers lapse into a less attentive "set" prior to coding without a…

The detrimental effects of nonresponse bias are particularly significant given the widespread use of the survey data collection method in educational surveys. Current methods for remediating nonresponse bias in educational surveys are explored and critiqued. (Author/LMO)

Using development of a State Event Observation System as an example, the decision rules and procedures for the constructing of standardized multiple behavior observational systems that provide accurate, reliable data for school-based assessment, intervention, and research are described. Reliability and validity data from the SECOS are provided.…

A study investigated whether examiners' personal familiarity and professional experience with examinees affects handicapped children's test performance. Professionally experienced and inexperienced examiners were used in test conditions that varied by degree of personal familiarity of examiners and children. Results are discussed. (Author/DF)

Shows that synonymic scoring of cloze tests is highly variable--that the score seems to appear simply on who grades the test. (FL)

Regression analyses indicated that the Coopersmith Self-Esteem Inventory has convergent validity with regard to the Piers-Harris Children's Self-Concept Scale and the Coopersmith Behavioral Academic Assessment Scale, has discriminant validity with regard to the Children's Social Desirability Scale, is sensitive to differences in achievement level,…

The purposes of this study were to investigate the role of benchmark writing samples in direct assessment of writing and to examine the consequences of differential benchmark selection with a common writing rubric. The influences of discourse and grade level were also examined within the context of differential benchmark selection. Raters scored…

Examined couples' and observers' perspectives of marital interaction. Nondistressed (N=10) and distressed (N=6) couples and objective observers (N=10) evaluated couples' interactional behaviors. Spouses' ratings of their partners' behavior were not consistent with observers' ratings of partners' behaviors but were consistent with observers'…

The phenomenon of "halo" effects in subjective grading was investigated. Two groups of three raters evaluated 20 term papers in introductory psychology. Term paper grades correlated significantly with course grades when information about previous academic performance was made available. When this information was not available, the…

Stuttering self-judgments from 15 adults who stutter, judgments of each others' stuttering, and the judgments of a panel of 10 stuttering researchers were compared. Results found substantial differences in stuttering judgments across speakers, judges, and judgment conditions, but across-task comparisons were complicated by low self-agreement among…

The Interrater/Test Reliability System (ITRS) is described. The ITRS is a comprehensive computer tool used to address questions of interrater reliability that computes several different indices of interrater reliability and the generalizability coefficient over raters and topics. The system is available in IBM compatible or Macintosh format. (SLD)