This article addresses the capacity of high stakes tests to measure the most significant kinds of learning. It begins by examining a set of philosophical arguments pertaining to construct validity and alleged conceptual obstacles to attributing specific knowledge and skills to learners. The arguments invoke philosophical doctrines of holism and…

Networked learning aims to foster students' knowledge construction processes as well as the quality of knowledge construction. In this respect, it is crucial to be able to analyse both aspects of networked learning. Based on theories on networked learning and the empirical work of relevant authors in this domain, two coding schemes are presented…

The Online Academy (HO29K73002) was funded by the Office of Special Education Programs (OSEP) to develop research-based online instructional modules in the content areas of reading, positive behavior support and technology across the curriculum. Targeted to preservice teacher education programs in Institutions of Higher Education (IHE), but also…

The increasing availability and performance of computer-based testing has prompted more research on the automatic assessment of language and speaking proficiency. In this investigation, we evaluated the feasibility of using an off-the-shelf speech-recognition system for scoring speaking prompts from the LanguEdge field test of 2002. We first…

Establishing and refining a framework for performance assessment is especially difficult in large-scale settings that can involve hundreds of judges and thousands of students. This presentation advocates the interactive use of two complementary analytic perspectives and illustrates the approach in the context of the College Entrance Examination…

Setting standards on performance measures is discussed in the context of the State Collaborative on Assessment and Student Standards (SCASS) initiative supported by the Council of Chief State School Offices. The usual item-based methods for standard setting, the methods developed by Nedelsky (1954), Angoff (1971), and Ebel (1972), were developed…

This study, part of a 10-year longitudinal study assessing the teacher education program at Iowa State University, examined the performance of student teachers and their teaching potential as judged by multiple raters. University supervisors, cooperating teachers, and student teachers (N=260) rated items on a questionnaire addressing professional…

Although the standard-setting method of W. H. Angoff (1971) has broad-based support in the research literature, inconsistencies in the resulting standards do occur. Sources of these inconsistencies are examined in a study of judges, competencies (items), rounds (replications), and the interactions among them. A modified Angoff approach was used to…

Performance rating systems frequently use multiple raters in order to improve the reliability of ratings. However, unless all candidates are rated by the same raters, some candidates will be at an unfair advantage or disadvantage solely because they were rated by more stringent or lenient raters. To obtain fair and accurate evaluations of…

The relationships among independent observer ratings of a child care program on the Early Childhood Environment Rating Scale (ECERS), state department personnel ratings of program quality using the Child Development Program Evaluation Scale (CDPES), and self-evaluation ratings using the self-assessment instrument designed for the Early Childhood…

The Management Assessment Center (MAC) of the Dade County (Florida) Public Schools is a unique project, employing multiple techniques to evaluate behavior for school-level administrator personnel selection. This final report of an evaluation deals primarily with issues of the validity and the utility of the MAC. To ascertain validity of the MAC,…

This study sought to determine the degree of variance found existing among three different groups of evaluators relative to their assessments of the teaching competencies demonstrated by 60 preservice teacher education students. Data were obtained relative to the undergraduates' self-evaluations regarding the ability to demonstrate 25 teaching…

The methods used by the General Educational Development Testing Service (GEDTS) to establish and maintain score stability and reading reliability on its direct assessment of writing are described. Using the 1988 site certification and monitoring results of several scoring sites, the focus is on describing how the score scale was established and…

This report presents two experiments that compared the performance of map experts to that of novices. Subjects (from the areas of geography, educational psychology, and sociology) were 13 university faculty members in experiment one and 12 undergraduate students in experiment two. Following a practice trial, the learning of a logical map and a…

This paper isolates and examines separately three distinct sources of error in essay scores: lack of agreement between raters; inconsistencies in performance within mode of discourse, and inconsistencies in performance between modes of discourse. Essay prompts in the Iowa Tests of Basic Skills (ITBS) Writing Supplement were designed to assess…