Topic models are mathematical and statistical models used to analyze textual data. The objective of topic models is to gain information about the latent semantic space of a set of related textual data. The semantic space of a set of textual data contains the relationship between documents and words and how they are used. Topic models are becoming…

Many organizations utilize some form of automation in the test assembly process; either fully algorithmic or heuristically constructed. However, one issue with heuristic models is that when the test assembly problem changes the entire model may need to be re-conceptualized and recoded. In contrast, mixed-integer programming (MIP) is a mathematical…

The ultimate goal of physics education research (PER) is to develop a theoretical framework to understand and improve the learning process. In this journey of discovery, assessment serves as our headlamp and alpenstock. It sometimes detects signals in student mental structures, and sometimes presents the difference between expert understanding and…

Test data generated according to two different multidimensional item response theory (IRT) models were compared at both the item response level and the test score level to determine whether measurable differences between the models could be detected when the data sets were constrained to be equivalent in terms of item "p"-values. The…

At the Educational Testing Service, the Mantel-Haenszel procedure is used for differential item functioning (DIF) detection, and the standardization procedure is used to describe DIF. This report describes these procedures. First, an important distinction is made between DIF and impact, pointing to the need to compare the comparable. Then, these…

This paper attempts to provide some initial information about the use of a variety of item response theory (IRT) models in the item selection process; its purpose is to compare the information curves derived from the selection of items characterized by several different IRT models and their associated parameter estimation programs. These…

This paper develops and applies three nonparametric comparisons of the shapes of two item characteristic surfaces: (1) proportional latent odds; (2) uniform relative difficulty; and (3) item sensitivity. A method is presented for comparing the relative shapes of two item characteristic curves in two examinee populations who were administered an…

This National Council on Measurement in Education (NCME) instructional module compares classical test theory and item response theory and describes their applications in test development. Related concepts, models, and methods are explored; and advantages and disadvantages of each framework are reviewed. (SLD)

The similarity of item and ability parameter estimations was investigated using two numerical analysis techniques via marginal maximum likelihood estimation (MMLE) with a large simulated data set (n=1,000 examinees) and changing the number of quadrature points. MMLE estimation uses a numerical analysis technique to integrate examinees' abilities…

Single administration (beta-binomial) estimates for the raw agreement index p and the corrected-for-chance kappa index in mastery testing are compared with those based on two test administrations in terms of estimation bias and sampling variability. Bias is about 2.5 percent for p and 10 percent for kappa. (Author/RL)

Three methods of factor analyzing dichotomously scored item performance data were compared using two raw score data sets of 20-item tests, one reflecting normally distributed latent traits and the other reflecting uniformly distributed latent traits. This comparison was accomplished by using phi and tetrachoric correlations among dichotomous data…

The reasons for recent endeavors to evaluate item bias are discussed, and item bias is defined. Some of the literature regarding the most promising methods of detecting item bias is reviewed. Three classes of methods for detecting item bias are discussed using concrete examples and illustrations. These methods are: (1) latent trait; (2)…

To demonstrate that some methods used to assess item fit can be applied to assess person fit and vice versa, performance of a chi-squared item-fit statistic was compared with that of a likelihood-based person-fit statistic for examinees and items under Monte Carlo conditions. (SLD)

The interest in the application of large-scale computerized adaptive testing has served to focus attention on issues that arise when theoretical advances are made operational. Some of these issues stem less from changes in testing conditions and more from changes in testing paradigms. One such issue is that of the order in which questions are…

Computerized adaptive testing procedures (CATPs) based on the graded response method (GRM) of F. Samejima (1969) and the partial credit model (PCM) of G. Masters (1982) were developed and compared. Both programs used maximum likelihood estimation of ability, and item selection was conducted on the basis of information. Two simulated data sets, one…