The Standards for Educational and Psychological Testing indicate that validation of assessments should include analyses of participants' response processes. However, such analyses typically are conducted only to supplement quantitative field studies with qualitative data, and seldom are such data connected to quantitative data on student or item…

In recent guidelines for fair educational testing it is advised to check the validity of individual test scores through the use of person-fit statistics. For practitioners it is unclear on the basis of the existing literature which statistic to use. An overview of relatively simple existing nonparametric approaches to identify atypical response…

As with any psychometric models, the validity of inferences from cognitive diagnosis models (CDMs) determines the extent to which these models can be useful. For inferences from CDMs to be valid, it is crucial that the fit of the model to the data is ascertained. Based on a simulation study, this study investigated the sensitivity of various fit…

The use of some very simple loss or utility functions in educational evaluation has recently been advocated by Gross and Su, Petersen and Novick, and Petersen. This paper demonstrates that more realistic utility functions can easily be used and may be preferable in some applications. (Author/CTM)

Relations between standard statistical techniques for analyzing dichotomous data and ANOVA procedures are indicated. The need for usefulness of analyzing transformed data as opposed to direct analysis of dichotomous data are discussed. Required statistical procedures employing transformed data are outlined. (Author/AG)

Despite the numerous individual illustrations in the literature showing how the discrimination index may be used to identify items with faults, its overall effectiveness as a measure of item quality, defined in terms of the presence or absence of faults, is not clear. This study investigates its validity. (Author/RK)

The common approach to studies of predictive bias is analyzed within the context of a conceptual model in which predictors and criterion measures are viewed as fallible indicators of idealized qualifications. (Author/PN)

Ten item discrimination indices, across a variety of item analysis situations, were compared, based on the validities of tests constructed by using each of the indices to select 40 items from a 100-item pool. Item score data were generated by a computer program and included a simulation of guessing. (Author/CTM)

Results of using multimethod factor analysis and exploratory factor analysis for the analysis of three multitrait-multimethod matrices are compared. Results suggest that the two methods can give quite different impressions of discriminant validity. In the examples considered, the former procedure tends to support discrimination while the latter…

This review of issues about correcting for attenuation concludes that the basic difficulty lies in being able to identify and equate sources of variance in estimates of validity and reliability. Recommendations are proposed for cautious use of correction for attenuation. (Author/CM)

Nine methods of smoothing double-entry expectancy tables (tables that relate two predictor variables to probability of attaining success on a criterion) were compared using data for entering students at 85 colleges and universities. The smoothed tables were more accurate than those based on observed relative frequencies. (Author/CTM)

Meta-analysis of research on a topic may exclude reports that do not present their results in a statistical form amendable to summarizing procedures used. This article illustrates how researchers can consider this aspect in computing statistics from their data. An illustration of how that can sometimes be done is presented. (Author/JAZ)

Reviews research literature on comparability issues in computerized adaptive testing (CAT) and synthesizes issues specific to comparability and test security. Develops a framework for evaluating comparability that contains three categories of criteria: (1) validity; (2) psychometric property/reliability; and (3) statistical assumption/test…

Responses to a sample test were simulated for examinees under free-response and multiple-choice formats. Test score sets were correlated with randomly generated sets of unit-normal measures. The extent of superiority of free response tests was sufficiently small so that other considerations might justifiably dictate format choice. (Author/DWH)