This article addresses the perennial issue of the criteria by which qualitative research should be evaluated. At the present time, there is a sharp conflict between demands for explicit criteria, for example in order to serve systematic reviewing and evidence-based practice, and arguments on the part of some qualitative researchers that such…

As its title suggests, this is an account of "the Second E-Forum on Competency-based Approaches" and summarises the opinions and experiences expressed by the participants. The forum was based on a discussion paper prepared by the Canadian Observatory of Educational Reforms and a series of questions raised by Philippe Jonnaert were used to…

This study examined the impact of rater agreement on decisions concerning the alignment between the "Golden State Examination in High School Mathematics" (California Department of Education, 2001a) and the University of California (UC) "Statement on Competencies in Mathematics Expected of Entering College Students" (Academic…

This retrospective longitudinal study was designed to show grading leniency patterns of judges within and across clinical examination administrations. Data from 17 different administrations of the histology examination of the American Society of Clinical Pathologists over 10 years were studied. Over the 10 years there were 4,683 candidates and 57…

Metaphor has recently become a topic of considerable interest and empirical investigation. Cognitive theories view metaphor as an aid to cognitive restructuring. Beyond the heuristic and theoretical concepts, the results of empirical research have suggested that metaphor is a potentially valuable tool for enhancing client involvement in counseling…

Many empirical studies have examined factors that influence ratings of performance. This study examined the rating variable performance of a single individual. Serial position of a single poor or good performance in a series of otherwise good or poor performances was manipulated to examine its effects on both ratings and recommended actions toward…

A study of interrater reliability among 17 faculty members assessing medical student case reports revealed marked disparities in the criteria raters felt to be important and an unacceptable spread in the ratings given. A standardized assessment instrument is recommended instead. (MSE)

Problems in studying teacher effectiveness through the use of classroom observation are discussed. Four assumptions in the observation of classroom process are offered and ways in which these assumptions can be dealt with in designing an observation study are suggested. (DF)

Two studies concerned with agreement in ratings of temperament are reported. Ratings of the mothers of toddlers versus daycare workers were compared on the Toddler Temperament Scale (Study 1), and on ratings of a videotape of a 2-year-old child for responses relevant to six dimensions of temperament (Study 2). (Author/BN)

Proposes a highly structured six-step employment interviewing technique which includes asking the same questions, consistently administering the process to all candidates, and having an interview panel. Results of a field study of 243 job applicants using this technique demonstrated interrater reliability, predictive validity, test fairness for…

A performance-based student teacher evaluation process was investigated to see if halo and leniency errors could be eliminated. Results are presented. (MT)

A study revealing wide variation in the grading of electronics engineering test items by different evaluators has implications for evaluator and test item selection, analysis and manipulation of grades, and the use of numerical methods of assessment. (MSE)

Type and comprehensiveness of training were experimentally manipulated (N=128) to study their effects on the reliability and validity of rated counselor empathy. Implications for observer training are discussed. (Author)

Nearly all of the research on standard setting focuses on different standard setting methods rather than the interaction of group members and the instructions given to group members. This study explored the effect of deliberation style and the requirement to reach consensus on the passing score, on rater satisfaction, and on postdecision…

This paper illustrates a method to study rater severity across exam administrations. A multi-facet Rasch model defined the ratings as being dominated by four facets: examinee ability, rater severity, project difficulty, and task difficulty. Ten years of data from administrations of a histotechnology performance assessment were pooled and analyzed…