H. Bosch, F. Steinkamp, and E. Boller's (see record 2006-08436-001) meta-analysis, which demonstrated (a) a small but highly significant overall effect, (b) a small-study effect, and (c) extreme heterogeneity, has provoked widely differing responses. After considering D. B. Wilson and W. R. Shadish's (see record 2006-08436-002) and D. Radin, R.…

The third edition of the "Boston Diagnostic Aphasia Examination" (Goodglass, Kaplan, and Barresi) introduced standardized procedures for coding discourse samples elicited using the well known Cookie Theft illustration. To evaluate the reliability of this discourse coding procedure, a transcribed sample was coded by 14 novice examiners…

Technological innovation undoubtedly offers many potential benefits for education and the assessment of learning, which have been acknowledged elsewhere. One area that is relatively under-researched relates to the practice of how assessors interact with longer texts that are presented on screen. This is an important area of study because there…

Education programs have fostered advanced levels of graphic representation ability in young children but have not detailed the specific mechanisms responsible for the accelerated growth. Research suggests that between 6 and 8 years of age children begin to observe more carefully before drawing and that observation prompts aid children's…

One of the ways to improve the reliability in perceptual voice quality rating is to provide listeners with external anchors. A paired comparison matching paradigm using synthesized Cantonese voice stimuli that covered a range of rough and breathy qualities were used to investigate the rating reliability. Twenty-five speech pathology students rated…

The purpose of this study was to determine what trends exist in the identification and description of participants used in data-based studies published in "Adapted Physical Activity Quarterly" and the "Journal of Teaching in Physical Education". Data were analyzed using frequency counts for journals and time periods from the 1980s to 2005 with…

Educators often have to choose among different types of rating scales to assess second-language (L2) writing performance. There is little research, however, on how different rating scales affect rater performance. This study employed a mixed-method approach to investigate the effects of two different rating scales on EFL essay scores, rating…

It is incorrect to say "the test is reliable" because reliability is a function not only of the test itself, but of many factors. The present paper explains how different factors affect classical reliability estimates such as test-retest, interrater, internal consistency, and equivalent forms coefficients. Furthermore, the limits of classical test…

A discussion of the differences between the Test of English as a Foreign Language (TOEFL), an American test battery, and the Cambridge English Examinations (Cambridge), a British battery, focuses on the different approaches to language test development embodied in the tests as the source of difficulty in translating between them for individual…

Most currently used measures of inter-rater agreement for the nominal case incorporate a correction for "chance agreement." The definition of chance agreement is not the same for all coefficients, however. Three chance-corrected coefficients are Cohen's Kappa; Scott's Pi; and the S index of Bennett, Goldstein, and Alpert, which has…

If content analysis is to satisfy the requirement of objectivity, measures and procedures must be reliable. Reliability is usually measured by the proportion of agreement of all categories identically coded by different coders. For such data to be empirically meaningful, a high degree of inter-coder reliability must be demonstrated. Researchers in…

The clinical usefulness of various empirically derived suicide potential rating scales has been questioned by several suicidologists. This study used actual case histories in an attempt to anchor suicide risk ratings. Thirty-three brief case histories of suicidal patients were given to 19 experienced crisis workers for seven-point ratings of…

Quality of performance data yielded by subjective judgment is of major concern to researchers in performance appraisal. However, some confusion exists in the analysis of quality on ratings obtained from different rating scale formats and from different raters. To clarify this confusion, a study was conducted to assess the quality of judgmental…

Of the many short forms of the Minnesota Multiphasic Personality Inventory (MMPI) that have been developed, the MMPI-168 is among the most promising. To determine whether clinical judgments based on the MMPI-168 are comparable to judgments based on the standard MMPI, 30 clinical psychologists participated in a randomized block, repeated treatment…

Efforts to expand the generic language proficiency guidelines of the American Council on the Teaching of Foreign Languages (ACTFL) to the less commonly taught languages (LCTLs) began when developers realized that the ACTFL guidelines were too Eurocentric; the guidelines included grammatical categories specific to Western European languages and…