The central purpose of this paper is to examine validity problems arising from the multiple-choice items and technical passages in the Test of English as a Foreign Language Internet-based Test (TOEFL iBT) reading section, primarily concentrating on construct-irrelevant variance (Messick, 1989). My personal TOEFL iBT experience, along with my…

The purpose of this article of reflection is to raise awareness of how poor design of language assessments may have detrimental effects, if crucial qualities and technicalities of test design are not met. The article first discusses these central qualities for useful language assessments. Then, guidelines for creating listening assessments, as an…

This article introduces the development and validation of the spatial thinking ability test (STAT). The STAT consists of sixteen multiple-choice questions of eight types. The STAT was validated by administering it to a sample of 532 junior high, high school, and university students. Factor analysis using principal components extraction was applied…

This study explores measurement of a construct called knowledge integration in science using multiple-choice and explanation items. We use construct and instructional validity evidence to examine the role multiple-choice and explanation items plays in measuring students' knowledge integration ability. For construct validity, we analyze item…

Describes methodology for using verbal reports of thinking to develop and validate multiple-choice tests of critical thinking. Procedure includes devising normative models of thinking for each item; collecting verbal reports of thinking from samples of subjects; using normative models to rate quality of thinking; comparing quality of thinking to…

In contrast to multiple choice, alternative modes of assessment afford varying degrees of openness in the allowable responses. Prominent among the alternatives is the assessment of performance, sometimes in its own right where the issue is the quality of the particular performance per se, but more often as a vehicle for the assessment of…

Contrasts between constructed response items and stem-equivalent multiple-choice counterparts typically have involved averaging item characteristics, and this aggregation has masked differences in statistical properties at the item level. Moreover, even aggregated format differences have not been explained in terms of differential cognitive…

The purpose of this study was to examine the construct validity of a performance assessment program, the Maryland School Performance Assessment Program (MSPAP). Based on analyses of the longitudinal associations of Grade 5 MSPAP data in 1996 with Grade 3 MSPAP data in 1994, the following hypothesis was examined: the unattentuated correlation or…

The construct validity of biographical items in the Individual Achievement Record (IAR) was assessed in an attempt to develop a confirmatory factor model. This first phase of a two-phase study focused on the IAR's use as a predictor of job performance. The IAR is an empirically keyed biodata inventory containing 112 multiple-choice items. Item…

Four major problem areas inhibit the standardized assessment of critical thinking (CT): (1) content validity; (2) construct validity; (3) technical jargon; and (4) background knowledge. Practical examples of framing multiple-choice items for assessment are suggested. In the area of content validity, new agreement about the definition of CT now…

This article addresses the capacity of high stakes tests to measure the most significant kinds of learning. It begins by examining a set of philosophical arguments pertaining to construct validity and alleged conceptual obstacles to attributing specific knowledge and skills to learners. The arguments invoke philosophical doctrines of holism and…

The proliferation of minimum competency testing in recent years has not resulted in a critical analysis of the construct validity of such tests. The present study examined the dimensionality and construct validity of an urban life skills competency test (the 1987 eighth-grade Chicago Minimum Proficiency Skills Test (MPST) via linear analysis. The…

Despite increasing interest in interlanguage pragmatics research, research on assessment of this crucial area of second language competence still lags behind assessment of other aspects of learners' developing second language (L2) competence. This study describes the development and validation of a 36-item web-based test of ESL pragmalinguistics,…

Standardized tests often produce results that are inaccurate, inconsistent, and biased against minority, female, and low-income students. Such tests shift control and authority into the hands of the unregulated testing industry and can undermine school achievement by narrowing the curriculum, frustrating teachers, and driving students out of…