Two factors which may affect the ratings assigned to an essay test are investigated: (1) context effects; and (2) score level effects. Context effects exist in essay scoring if an essay is rated higher when preceded by poor quality essays than when preceded by high quality essays. A score level effect is defined as a change in the score (value)…

A total of 1,487 eleventh grade students from the Hamburg (West Germany) school system were asked to complete four writing assignments used in an International Association for the Evaluation of Educational Achievement (IEA) study of writing assessment. In analyzing the writing samples, the study focused on: (1) between-rater effects; (2)…

This report gives some simple examples of stability for one factor and 2 x 2 factorial analysis of variance, reliability and correlations. The findings are very different: from superstability (no transformation whatsoever can change the result) to almost total instability. This is followed by a discussion of applications to multivariate analysis,…

Today's psychological measurement depends almost exclusively on the "standardized test." A certain amount of non-standardization, however, exists in the administration of any standardized test, with the amount unknown for any given test score. Time limits on tests pose a bigger problem since another variable is introduced, pressure. Test taking…

Livingston's work is a careful analysis of what occurs when one pools two populations with different means, but similar variances and reliability coefficients. However, his work fails to advance reliability theory for the special case of criterion-referenced testing. See ED 042 802 for Livingston's paper. (MS)

This paper examines the individual consistency of eleventh grade respondents in normal test-retest situations as well as situations involving changed response methods. It postulates the personality-trait "discriminator" (high vs low) and confirms the trait's influence on individual consistency within and across Biology Cognitive…

Observer disagreement is important because it limits the reliabilities of observational measures. To avoid this limitation, observers should be trained, and criterion-related and intraobserver agreement measures should be used both before and during a study. Several agreement coefficients are examined for their applicability. (Author/BW)

This research aimed to investigate the effect of computers on the test and inter-rater reliability of writing test scores of ESL learners. Writing samples of 20 pen-paper and 20 computer group students were scored in analytic scoring method by two scorers, and then the scores were analyzed in Alpha (Cronbach) model. The results showed that the…

Writing fits well within the realm of outcomes suitable for observation by performance assessments. Studies of the reliability of performance assessments have suggested that interrater reliability can be consistently high. Scoring consistency, however, is only one aspect of quality in decisions based on assessment results. Another is…

This paper suggests that in addressing the issue of parent-professional congruence in child assessment, researchers should avoid focusing on the conventional aspects of interrater reliability and rater interchangeability, but rather should focus on the reliability of the pooled assessment information from parents and professionals. A…

Teaching professionals (n=29) who shared the same classroom for a minimum of one hour per day provided independent ratings of the same child (ages 7-17) on the Adjustment Scales for Children and Adolescents (ASCA). Results indicated that statistically significant interrater agreement was achieved across all 22 syndromic profile classification…

This study tested the interrater reliability of the Modified Ashworth Scale in measuring upper and lower limb spasticity in 34 hemiplegic adult patients examined by 2 physiotherapists and 2 doctors. Findings indicated satisfactory reliability for upper limb spasticity but less satisfactory results for lower limb spasticity. (DB)

Standardization data based on responses of 616 Puerto Ricans to the Spanish version of the Wechsler Adult Intelligence Scale (D. Wechlser, 1981) reveal reliability data and base rates to assist in evaluating the clinical significance of differences between Performance Intelligence Quotient (PIQ) and Verbal Intelligence Quotient (VIQ).…

This study assessed the psychometric properties of two subscales of the Adelaide Behaviour Disorder Scale that have been hypothesized to describe conduct problems and emotional problems of adults with intellectual disability. Criterion scores for identifying individuals needing clinical intervention were established and validated against…

This study assessed the psychometric properties of the Environmental Rating Scale, a measure specifically designed to assess residential treatment programs for individuals with autism. The measure's reliability was demonstrated by assessments of the internal consistency, stability, and interrater reliability. Preliminary analysis of validity…