A measurement approach called generalizability theory (G-theory) is an important alternative to the more familiar classical measurement theory that yields less useful coefficients such as alpha or the KR-20 coefficient. G-theory is a theory about the dependability of behavioral measurements that allows the simultaneous estimation of multiple…

The use of stepwise regression has been criticized for both interpretive misunderstandings and statistical aberrations. A major statistical problem with stepwise regression and other procedures that involve multiple significance tests is the inflation of the Type I error rate. General approaches to control the family-wise error rate such as the…

Given the complexity of classroom reading observations, maintenance of measurement reliability is a concern to researchers. Sources of variation contributing to the unreliability of many measurements may be either (1) lasting and specific, (2) lasting and general, (3) temporary and specific, or (4) temporary and general. Lasting-specific sources…

Use of clustering methods with data from the 1987 High School Transcript (HST) component of the 1986 National Assessment of Educational Progress (NAEP) is discussed in an attempt to persuade data analysts and researchers that they should be interested in problems traditionally turned over to samplers. Nearly all surveys used by the National Center…

In item response theory (IRT), preequating depends upon item parameter estimate invariance. Three separate simulations, all using the unidimensional three-parameter logistic item response model, were conducted to study the impact of the following variables on preequating: (1) mean differences in ability; (2) multidimensionality in the data; and…

This was a replicative study that was initiated with a journeyman level certification instrument for an international union, when industry monitors were observed suggesting to examinees to "go with your first response." The question arose whether this was a researched-based practice. If not, wouldn't this practice inject constant error…

The Test of Standard Written English (TSWE) is a 50-item multiple choice instrument designed to assess the ability of college students to use English. In this study, based upon a sample of 45 students, the TSWE was revalidated with writing samples. The coefficient of 0.54 was most impressive given that the TSWE scores were restricted to those…

LISREL-type structural equation modeling is a powerful statistical technique that seems appropriate for social science variables which are complex and difficult to measure. The literature on the specification, estimation, and testing of such models is voluminous. The greatest proportion of this literature, however, focuses on the technical aspects…

This report presents the rationale, development, and standardization of the Air Force Officer Qualifying Test (AFOQT) Form O. The test is used to select individuals for officer commissioning programs, and candidates for pilot and navigator training. Form O contains 380 items organized in 16 subtests. All items are administered in a single test…

The purpose of this paper is to present two new alternative methods to the current goodness of fit methodology. With the increase use of computerized adaptive test (CAT), the ability to determine the accuracy of calibrated item parameter estimates is paramount. The first method applies a normalizing transformation to the logistic residuals to make…

In order to evaluate standard setting procedures, apart from the more commonly applied approach of simply comparing the derived standards or failure rates across various techniques, this study investigated the errors of classification associated with the contrasting groups procedures. Monte Carlo simulations were employed to produce…

The current state of usage of regression models in analysis of variance (ANOVA) designs is empirically examined, and examples of several statistical errors made in usage are presented. The assumptions of the general linear model are that all predictors are known without error of measurement and are fixed with no replication or sample variation; in…

The results of six major projects are discussed including a comprehensive mathematical and statistical analysis of the problems caused by errors of measurement in linear models for assessing change. In a general matrix representation of the problem, several new analytic results are proved concerning the parameters which affect bias in…

When criterion-referenced tests are used to assign examinees to states reflecting their performance level on a test, the better known methods for determining test length, which consider relationships among domain scores and errors of measurement, have their limitations. The purpose of this paper is to present a computer system named TESTLEN, which…

Multiple matrix sampling of three subscales of the California Psychological Inventory was used to investigate the effects of four variables on error estimates of the mean (EEM) and variance (EEV). The four variables were examinee population size (600, 450, 300, 150, 100, and 75); number of subtests, (2, 3, 4, 5, 6, and 7), hence the number of…