Long-running surveys need a systematic way to reflect social change and to keep items relevant to respondents, especially when they ask about controversial subjects, or they threaten the items' validity. We propose a protocol for updating measures that preserves content and construct validity. First, substantive experts articulate the current and…

In the fall of 2014, the University at Buffalo created a new universal Student Evaluation of Teaching (SET). The purpose of the present study was to establish the construct validity of SET items. Rasch analyses of data from 7 semesters (N=203,194 students) revealed problems with item fit indices and threshold distances. Changes to items and…

The aim of this mixed methods study is to identify the underlying structure of the construct of Foreign Language Anxiety in integrated listening-to-speak tasks. First, the analysis of the qualitative interviews with six postsecondary ESL learners reveals that anxiety for integrated speaking stems from four different factors: "listening,"…

Many Flynn effect (FE) studies compare scores across different editions of Wechsler's IQ tests. When construct changes are introduced by the test developers in the new edition, however, the presumed generational effects are difficult to untangle from changes due to test content. To remove this confound, we use the same edition of Wechsler…

The Test Of Astronomy STandards (TOAST) is a comprehensive assessment instrument designed to measure students' general astronomy content knowledge. Built upon the research embedded within a generation of astronomy assessments designed to measure single concepts, the TOAST is appropriate to measure across an entire astronomy course. The TOAST's…

This study aimed to develop an instrument, named the inquiry-based laboratory classroom environment instrument (ILEI), for assessing senior high-school science students' preferred and perceived laboratory environment. A total of 262 second-year students, from a senior-high school in Taiwan, were recruited for this study. Four stages were included…

This chapter introduces theory that undergirds the role of testing adaptations in assessment, provides examples of item modifications and testing accommodations, reviews research relevant to each, and introduces a new paradigm that incorporates opportunity to learn (OTL), academic enablers, testing adaptations, and inferences that can be made from…

The purpose of this study was to develop a Korean College Stress Inventory (KCSI), which is designed to measure Korean college students' experiences and symptoms of career stress. Even though there have been numerous scales related to career issues, few scales measure the career stress construct and its dimensions. Factor structure, internal…

Although classroom assessment is one of the most frequent practices carried out by teachers in all educational programs, limited research has been conducted to investigate the dependability and validity of criterion-referenced tests (CRTs). The main purpose of this study is to develop a criterion-referenced test for first-year Japanese university…

Lissitz and Samuelsen (2007) propose a new framework for validity theory and terminology, emphasizing a shift in theory and practice toward issues of test content rather than constructs. The author of this article argues that several of Lissitz and Samuelsen's critiques of validity theory focus on previously considered, but subsequently discarded,…

The purpose of this paper is to clarify the seemingly discrepant views of test theorists and test developers about terminology related to the evaluation of test content. The origin and evolution of the concept of content validity are traced, and the concept is reformulated in a way that emphasizes the notion that content domain definition,…

A study investigated whether examinees taking the Test of English as a Foreign Language (TOEFL) attended to the text passages in the "minitalks" when answering the multiple-choice items (n=337) testing listening comprehension. Results support the construct validity of the minitalks, and also allow comparison between reading and listening…

A qualitative study investigated the test-taking behaviors, knowledge, and perceptions of 20 urban, adult basic education students reading at third to fifth grade equivalency levels. The entire reading comprehension subtest of the Test of Adult Basic Education, levels E and M, was administered under standardized conditions. A combination of…

An expanded version of the method of content evaluation proposed by S. G. Sireci and K. F. Giesinger (1992) was evaluated with respect to a national licensure examination and a nationally standardized social studies achievement test. Two groups of 15 subject-matter experts rated the similarity and content relevance of the items. (SLD)

This study was designed to address two frequent criticisms of the SAT essay--that essay length is the best predictor of scores, and that there is an advantage in using more "sophisticated" examples as opposed to personal experience. The study was based on 2,820 essays from the first three administrations of the new SAT. Each essay was…