Many large-scale performance assessments include score resolution procedures for resolving discrepancies in rater judgments. The goal of score resolution is conceptually similar to person fit analyses: To identify students for whom observed scores may not accurately reflect their achievement. Previously, researchers have observed that…

Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting "idiosyncratic rater variance" is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical…

Sparse-rated data are common in operational performance-based language tests, as an inevitable result of assigning examinee responses to a fraction of available raters. The current study investigates the precision of two generalizability-theory methods (i.e., the rating method and the subdividing method) specifically designed to accommodate the…

Background: Many in education argue for the importance of incorporating teacher judgements in the assessment and reporting of student performance. Advocates of such an approach are cognisant, though, that obtaining a satisfactory level of consistency in teacher judgements poses a challenge. Purpose: This study investigates the extent to which the…

This review of current writing assessment practices focuses upon the adult population, an area significantly underrepresented within psychoeducational literature. As compared to other populations, such as K-12 students, there are few options for the practitioner wishing to evaluate adult writers by means of standardized assessment instruments.…

The current study examines how using a multidimensional battery of predictors (high-school grade point average (GPA), SAT/ACT, and biodata), and weighting the predictors based on the different values institutions place on various student performance dimensions (college GPA, organizational citizenship behaviors (OCBs), and behaviorally anchored…

A table is provided giving observed difference frequencies for caregiver versus teacher ratings of children on the Child Behavior Checklist and Teacher's Report Form Internalizing, Externalizing, and Total Problems scales per the original normative samples. The table permits accurate evaluation of the empirical rarity of specific cross-informant…

This study describes how multiple-measure teacher evaluations were put into practice in a set of ten volunteering local education agencies (LEAs) in Arizona. After a key shift in state policy, five "pilot" LEAs implemented the new Arizona Department of Education teacher evaluation model in the 2012/13 school year, while five other…

Critics of standardized testing have recommended replacing standardized tests with more authentic assessment measures, such as classroom assignments, projects, or portfolios rated by a panel of raters using common rubrics. Little research has examined the consistency of scores across multiple authentic assignments or the implications of this…

What is effective teaching? It is not enough to say "I know it when I see it"--not when we are expecting so much more from students and teachers than in the past. To help teachers achieve greater success with their students we need new and better ways to identify and develop effective teaching. The Measures of Effective Teaching (MET)…

In response to the diverse requirements of 21st-century police work and the increasing emphasis on community-policing philosophy, the Los Angeles Police Department has implemented changes within its academy curricula and methods of instruction, including the use of adult-learning concepts, a community policing problem-solving model known as…

Competency-based assessment and an emphasis on obtaining higher-level outcomes that reflect physicians' ability to demonstrate their skills has created a need for more advanced assessment practices. Simulation-based assessments provide medical education planners with tools to better evaluate the 6 Accreditation Council for Graduate Medical…

This study compared the creative outcomes in student work resulting from two pedagogical approaches to creative problem solving activities. A secondary goal was to validate the Consensual Assessment Technique (CAT) as a means of assessing creativity. Linear models for problem solving and design processes serve as the current paradigm in classroom…

Western Governors University (WGU) is an online competency-based university in which students demonstrate content competence through a series of assessments. Assessments most often are performance-based or objective assessments that are developed in accordance with specific content objectives. Objective assessments generally assess lower-level…

Background: Coaches and physical educators must obtain valid data relating to the contribution of each of their players in order to assess their level of performance in team sport competition. This information must also be collected and used in real game situations to be more valid. Developed initially for a physical education class context, the…