The Client Verbal Response Category System classifies client responses into Temporal, Directional and Experiential categories. The categories with their subcategories are defined, interjudge reliability data is presented, and the instrument's utility in psychotherapy process research is demonstrated. Initial results indicate that the instrument is…

Examined whether less experienced judges could also produce accurate estimates of the validity of cognitive tests. Shows that the estimates of less experienced judges contain less information than those of experts, but also that average of estimates of several less experienced judges are as accurate as those obtained from small-sample empirical…

The present article reviews the procedures that have been developed for measuring the reliability of human observers' judgments when making direct observations of behavior. These include the percentage of agreement, Cohen's Kappa, phi, and univariate and multivariate agreement measures that are based on quasi-equiprobability and quasi-independence…

Questions repeatedly arise about whether evaluations of research papers are systematically influenced by factors unrelated to quality, such as gender or prestige of the authors or reviewers. This study examined the reliability of reviews of proposals submitted for inclusion in the 1984 American Educational Research Association annual meeting…

Due to grade inflation, transcripts reveal little more than courses a student has completed. Recommendation letters have become an important criterion in teacher candidate selection. Suggestions for writing a subjective recommendation are offered. (DF)

Concludes that the Cloze Reading Inventory and its coding form can be reliably employed by a variety of teachers for a variety of grade levels and passages. (FL)

Assessed the adequacy of reporting the use of observers in counseling research. Each sampled article was classified according to the use made of the observer, the type and length of training given the observer, and the assessments made of the observers' reliability and validity. (Author/JAC)

This report describes 4 years of research by the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) on developing indicators of classroom practice that have the potential to be used in large-scale settings and that draw attention to important aspects of standards-based learning and instruction. CRESSTs method was…

A procedure was developed to analyze data with missing observations by extracting data from a sparsely filled data matrix into analyzable smaller subsets of data. This subdividing method, based on the conceptual framework of meta-analysis, was accomplished by creating data sets that exhibit structural designs and then pooling variance components…

Against an historical backdrop, this paper summarizes four uses of intraclass correlation of importance to contemporary researchers in the behavioral sciences. First, it shows how the intraclass correlation coefficient can be used to adjust confidence intervals for statistical significance testing when data are intracorrelated and the independence…

Studied whether pubertal timing by self-report (SR), parent report (PR), or physical examination predicted the same aspects of adjustment and behavior problems. Findings for 52 girls, 56 boys, and their parents show that pubertal timing by SR and PR did not always provide the same level of prediction as did physical examination. (SLD)

To identify a procedure for establishing cut scores for National Occupational Competency Testing Institute examinations in Pennsylvania, an expert panel assessed written and performance test items for minimally competent workers. Recommendations about the number, type, and training of judges used were made. (Contains 18 references.) (SK)

Describes a FORTRAN software program that calculates the probability of an observed difference between agreement measures obtained from two independent sets of raters. An example illustrates the use of the DIFFER program in evaluating undergraduate essays. (Author/SLD)

Investigates how coder characteristics such as language skills, political knowledge, coding experience, and coding certainty affected inter-coder and coder-training reliability. Shows that language skills influenced both reliability types. Suggests that cross-national researchers should pay more attention to cross-national assessments of…

Eighteen adult listeners assessed whether stuttering samples were authentic or simulated. Results support the concepts that the production of stuttered and nonstuttered speech disruptions are experienced as being qualitatively different; only stutterers can validly recognize the difference, and only when it occurs; stuttering is a…