The relationship between job skills and specific tasks is difficult to analyze, since any one task may require multiple skills. In this study, a sequence of statistical evaluations was conducted to examine the reliability of a subject matter expert panel's association of tasks and job skills and the factor structure of the task by job skill…

An examination of the extent to which scores on the Stuttering Severity Instrument (SSI) for Children and Adults, Third Edition, accurately reflect 10 judges' observations of stuttering behaviors found that SSI scores obscured the wide range of judges' raw counts and did not accurately reflect the observational data from which they were derived.…

This study, involving 10 children (ages 7-11) attending an English school for children with moderate learning difficulties and their parents, revealed that parents' and teachers' estimates of attainments did not deviate significantly from the scores actually obtained by the children. There was considerable variation between teachers and parents…

Several extensions to the Angoff method of standard setting are described that can accommodate characteristics of performance-based assessment. A study involving 12 panelists supported the effectiveness of the new approach but suggested that panelists preferred an approach that was at least partially conjunctive. (SLD)

Behavioral ratings by mothers and teachers of 94 children with Down's Syndrome (between 8 and 14 years of age) indicated general support for the amiable personality stereotype, but ratings of low persistence were associated with maternal impressions of difficulty. There was little agreement between mothers and teachers regarding individual child…

The behavior of each of 120 students in grades 9-12 was rated by 2 of the student's teachers using the Revised Behavior Problem Checklist. Results indicated a generally low to moderate degree of relationship among teacher ratings. It is recommended that clinicians collect behavioral ratings from many raters before reaching diagnostic conclusions.…

This study investigated whether 13 young children presented with anatomically correct dolls would exhibit behaviors that professionals (n=14) could agree on to determine the child's abuse status. The study concluded that experienced professionals agree poorly with each other about a child's abuse status and that sexually anatomically correct dolls…

Using results for 1,487 West German eleventh graders from the International Study of Achievement in Written Composition, problems related to interrater and intrarater reliability and generalizability of composition ratings were assessed. Results indicate that numerous writing assignments are required to assure reliability. (TJH)

This study examined judgments of document relevance made by library science graduate students who were not the originators of the queries for which the documents were retrieved. Although the secondary judgments compared well with those of the original users, it was found that secondary judges used document record fields differently and had a…

Forty-seven young children, 15 with disabilities, were observed 4 times for types and levels of engagement. Results indicated that engagement is difficult to measure through molecular data collection techniques because of error in dependability measures. The number of observed sessions could be increased to achieve generalizability, but increases…

Reports on a study of the use of musical scores and rating scales by 80 pianists who listened to 21 trials of solo piano music. Found that the use of musical scores and rating scales did not improve interrater reliability. Discovered that the subjects were less consistent when evaluating slow musical pieces than faster pieces. (CFR)

Parents and teachers of 83 young children diagnosed with a pervasive developmental disorder rated each child on the Autism Behavior Checklist (ABC) and the Vineland Adaptive Behavior Scales (VABS). Although there was good agreement between informants on the VABS, there was virtually no agreement on the ABC, with parents reporting more autistic and…

A quantitative ethogram was developed for the zebra finch, using one-zero focal animal sampling on an ethologically comprehensive checklist of 52 behavioral items, and it was assessed for interobserver reliability and construct validity. Applying the quantitative methods of psychometrics allows verification of ethological theory and testing of…

Four approaches to estimating interrater reliability in early childhood special education research are illustrated and compared: correlation, comparison of means, percentage of agreement, and generalizability theory techniques. Generalizability theory techniques are proposed as a method for estimating the amount of variance attributable to…

Describes holistic scoring as one of the biggest breakthroughs in writing assessment. Suggests that the technique's high interrater reliability coefficients partly explain holistic scoring's popularity. Argues that validity has been largely neglected. Concludes that more must be learned about the uses and effects of holistic scoring. (SG)