Two procedures were developed to determine whether examinees in a given test center were affected by a testing irregularity on the Listening Comprehension section of the Test of English as a Foreign Language (TOEFL). One approach employed analysis of covariance (ANCOVA) on Listening Comprehension (Section 1) means using scores on Structure and…

Hierarchical Bayes procedures were compared for estimating item and ability parameters in item response theory. Simulated data sets from the two-parameter logistic model were analyzed using three different hierarchical Bayes procedures: (1) the joint Bayesian with known hyperparameters (JB1); (2) the joint Bayesian with information hyperpriors…

Diagnostic score reports linking assessment outcomes to instructional interventions are one of the most requested features of assessment products. There is a body of interesting work done in the last 20 years including Tatsuoka's rule space method (Tatsuoka, 1983), Haertal and Wiley's binary skills model (Haertal, 1984; Haertal & Wiley, 1993),…

SCORIGHT is a very general computer program for scoring tests. It models tests that are made up of dichotomously or polytomously rated items or any kind of combination of the two through the use of a generalized item response theory (IRT) formulation. The items can be presented independently or grouped into clumps of allied items (testlets) or in…

Large-scale educational assessments such as the National Assessment of Educational Progress (NAEP) sample examinees to whom an exam will be administered. In most situations the sampling design is not a simple random sample and must be accounted for in the estimating model. After reviewing the current operational estimation procedure for NAEP, this…

This paper reviews the literature about item response models for the subject level and aggregated level (group level). Group-level item response models (IRMs) are used in the United States in large-scale assessment programs such as the National Assessment of Educational Progress and the California Assessment Program. In the Netherlands, these…

A variety of locally dependent models are introduced having individual difference parameters that may be interpreted as reflecting effective learning abilities. One version is a univariate extension of the Rasch model with a Markov property: the probability that a given individual will pass an item depends on previous items only through the…

The decision making processes of children in a probabilistic environment were studied within the context of the theory of signal detection (TSD). The relationship between the age of the child and his ability to revise decision criteria was examined, as well as the appropriateness of TSD measures and methodology with children. It was hypothesized…

Standard procedures for estimating item parameters in Item Response Theory models make no use of auxiliary information about test items, such as their format or content, or the skills they require for solution. This paper describes a framework for exploiting this information, thereby enhancing the precision and stability of item parameter…

Tailored testing provides the same information as group-administered standardized tests, but can do so using fewer items because the items administered are selected for the ability of the individual student. Thus, tailored testing offers several advantages over traditional methods. Because individual tailored tests are not timed, anxiety is…

The technical problems involved in obtaining Bayesian model estimates for the regression parameters in m similar groups are studied. The available computer programs, BPREP (BASIC), and BAYREG, both written in FORTRAN, require an amount of computer processing that does not encourage regular use. These programs are analyzed so that the performance…

A large number of seemingly diverse coefficients have been proposed as indices of dependability, or reliability, for domain-referenced and/or mastery tests. In this paper, it is shown that most of these indices are special cases of two generalized indices of agreement: one that is corrected for chance, and one that is not. The special cases of…

The Multi-Attribute Utilities Theory (MAUT), a model for the evaluation of public postsecondary programs, is presented and is applied to a study of WINNERS (Women's Inner-City Educational Resources Service) with an analysis of the effects of the model on the program. The first chapter provides an overview of the MAUT approach, outlining 16 steps…

The assumptions of the Criterion-Referenced Test (CRT) model proposed by Kriewall are compared to those of Emrick and Adam's Mastery-Learning (ML) model. Testing, in the context of instructional management, serves three general purposes: performance evaluation (achievement of objectives), placement (classification of students for instruction), and…

A Markov model for predicting performance on criterion-referenced tests is presented,. The model is expressed mathematically as a function of transition matrix, a current state vector, and a future state vector. The matrix is defined in terms of conditional probabilities, i.e., the probability of making a transition to a specific future…