Viable methods for the identification of item misfit or Differential Item Functioning (DIF) are central to scale construction and sound measurement. Many approaches rely on the derivation of a limiting distribution under the assumption that a certain model fits the data perfectly. Typical DIF assumptions such as the monotonicity and population…

One major issue with large-scale assessments is that the respondents might give no responses to many items, resulting in less accurate estimations of both assessed abilities and item parameters. This report studies how the types of items affect the item-level nonresponse rates and how different methods of treating item-level nonresponses have an…

In July 2010, the Massachusetts Board of Elementary and Secondary Education (BESE) voted to adopt Common Core's standards in English language arts (ELA) and mathematics in place of the state's own standards in these two subjects. The vote was based largely on recommendations by Commissioner of Education Mitchell Chester and then Secretary of…

This article describes the results of a simulation study to investigate the impact of missing data on the detection of differential item functioning (DIF). Specifically, it investigates how four methods for dealing with missing data (listwise deletion, zero imputation, two-way imputation, response function imputation) interact with two methods of…

In this article, we introduce a person-fit statistic called the hierarchy consistency index (HCI) to help detect misfitting item response vectors for tests developed and analyzed based on a cognitive model. The HCI ranges from -1.0 to 1.0, with values close to -1.0 indicating that students respond unexpectedly or differently from the responses…

Computer-based assessments are becoming more commonplace, perhaps as a necessity for faculty to cope with large class sizes. These tests often occur in large computer testing venues in which test security may be compromised. In an attempt to limit the likelihood of cheating in such venues, randomised presentation of items is automatically…

The adequacy of a test developed for statewide assessment of basic mathematics skills was investigated. The test, comprised of multiple-choice items reflecting a series of behavioral objectives, was compared with a more extensive criterion measure generated from the same objectives by the application of a strict item sampling model. In many…

In recent years, the controversy surrounding testing has grown, and the charge of bias is the most often cited criticism of testing and assessment. A review of the literature indicates that psychometricians and other researchers speak of bias as a property of the test or of items in the test. Conversely, test critics speak of bias as a quality or…

Problems in accountability assessment are examined from a unique perspective by considering the prices paid as a result of the use of secret tests (tests comprised of items drawn from non-publicly available item banks). This report is a compilation of the following articles: (1) "The Social, Intellectual, and Psychological Prices of…

An alternative to multiple-choice testing is suggested for educational assessment. The use of what is called "multiple-rating items" is proposed. A multiple-rating item calls for the examinee to rate all of a set of things instead of picking one as with a multiple-choice item. The respondent has to provide a specific rating of each…

The idea that validity should be considered a property of inferences, rather than of assessments, has developed slowly over the past century. In early writings about the validity of educational assessments, validity was defined as a property of an assessment. The most common definition was that an assessment was valid to the extent that it…

This report examines the potential of adaptive testing, two-stage testing in particular, for improving the data quality of the National Assessment of Educational Progress (NAEP). Following a discussion of the rationale for adaptive testing in assessment and a review of previous studies of two-stage testing, this report describes a 1993 Ohio field…

Issues in alternative assessment for accountability purposes are discussed. Most new forms of performance assessment are linked in the literature, but all alternative forms of assessment do not have the same attributes in terms of technical and feasibility criteria. Tradeoffs in the validity of inferences that can be drawn from alternative…

The National Assessment of Educational Progress (NAEP) item development procedures, possible improvements or alternatives to these procedures, and methods used to control potential sources of errors of interpretation are described. Current procedures call for the assessment of 9-, 13- and 17-year-olds in subject areas typically taught in schools.…

This paper focuses on the questions which arise as test practitioners monitor score scales derived from latent trait theory. Large scale assessment programs are dynamic and constantly challenge the assumptions and limits of latent trait models. Even though testing programs evolve, test scores must remain reliable indicators of progress.…