There has been much controversy over the null hypothesis significance testing procedure, with much of the criticism centered on the problem of inverse inference. Specifically, p gives the probability of the finding (or one more extreme) given the null hypothesis, whereas the null hypothesis significance testing procedure involves drawing a…

In 1881, Donald MacAlister posed a problem in the "Educational Times" that remains relevant today. The problem centers on the statistical evidence for the effectiveness of a treatment based on a comparison between two proportions. A brief historical sketch is followed by a discussion of two default Bayesian solutions, one based on a…

There are many reasons to believe that open-ended (OE) and multiple-choice (MC) items elicit different cognitive demands of students. However, empirical evidence that supports this view is lacking. In this study, we investigated the reactions of test takers to an interactive assessment with immediate feedback and answer-revision opportunities for…

The present study investigates the statistical consequences of attribute misspecification in the rule space method for cognitively diagnostic measurement. The two types of attribute misspecifications examined in the present study are exclusion of an essential attribute (which affects problem-solving performance) and inclusion of a superfluous…

This article discusses the concept of an invariant item ordering (IIO) for polytomously scored items and proposes methods for investigating an IIO in real test data. Method manifest IIO is proposed for assessing whether item response functions intersect. Coefficient H[superscript T] is defined for polytomously scored items. Given that an IIO…

This article investigates several methods of identifying individual guessers from their response data. Both the posterior probability method and the likelihood ratio method are based on the two-state mixture modeling approach to response times. The accuracy method is based on response accuracy data. Results from the simulation study showed that…

The predictive validity of several mathematics and language variables relative to criteria of grade point average and word problem-solving skills for 60 bilingual Hispanic technical students is investigated. Examination of standardized college admission criteria is urged. Revealed is a disparity in academic performance between bilingual and…

College undergraduates were given Zelniker and Jeffrey's modification of the Matching Familiar Figures Test to assess its psychometric credibility and construct validity for adult problem solvers. The modified test has improved internal consistency and stability over the original. The construct's possible correlation with general problem solving…

A computer program for scoring written simulation tests provides individual scores and basic item analysis data. The program is written in Fortran IV and can accomodate up to thirty-five hundred options and up to ten thousand examinees. (Author/JKS)

Psychometric characteristics of the test scores of 5,663 examinees on six patient management problem tests were studied. The total scores can be divided into subscores based on all options keyed "select" and "omit." Component scores are generally correlated negatively, as reflected in reduced discrimination indices and…

It is shown that the Watson-Glaser Critical Thinking Appraisal (WGCTA) significantly predicted standing in a specialized course at the college level, lending credence to its criterion-related validity. The paper is based on a study of accelerated students pursuing a combined baccalaureate and professional program of medicine. (CE)

The importance of computational skill for answering items in the quantitative thinking subtest of the Iowa Tests of Educational Development was investigated. Results show that computational skill is not a major factor contributing to the examinee's score on the subtest. (Author/GK)

A measure of problem identification skills, entitled the "Clinton Assessment of Problem Identification Skills for Instructional Supervisors" (CAPIS), was developed and validated by using the Cronbach and Muhl test validation paradigm. (PN)

This study describes the use of generalizability theory (GT) and many-facet Rasch measurement (MFRM) to evaluate psychometric properties of responses obtained from an assessment designed to measure complex problem-solving skills. The assessment revolved around the school activity of kickball. The task required of each student was to decide on a…