Background: Increasingly, students are using ChatGPT to assist them in learning and even completing their assessments, raising concerns of academic integrity and loss of critical thinking skills. Many articles suggested educators redesign assessments that are more 'Generative-AI-resistant' and to focus on assessing students on higher order…

Abstract This study proposes a new Bayesian differential item functioning (DIF) detection method using posterior predictive model checking (PPMC). Item fit measures including infit, outfit, observed score distribution (OSD), and Q1 were considered as discrepancy statistics for the PPMC DIF methods. The performance of the PPMC DIF method was…

The residual differential item functioning (RDIF) detection framework was developed recently under a linear testing context. To explore the potential application of this framework to computerized adaptive testing (CAT), the present study investigated the utility of the RDIF[subscript R] statistic both as an index for detecting uniform DIF of…

The Kaufman Test of Educational Achievement, Third Edition (KTEA-3) is a revised and updated comprehensive academic achievement test (Kaufman & Kaufman, 2014). Authored by Drs. Alan and Nadeen Kaufman and published by Pearson, the KTEA-3 remains an individual achievement test normed for individuals of ages 4 through 25 years, or for those in…

The purpose of this study is to determine the extent of scale drift on a test that employs cut scores. It was essential to examine scale drift for this testing program because new forms in this testing program are often put on scale through a series of intermediate equatings (known as equating chains). This process may cause equating error to…

Differential item functioning (DIF) analysis is a statistical technique used for ensuring the equity and fairness of educational assessments. This study formulates a new DIF analysis method using the information similarity index (ISI). ISI compares item information functions when data fits the Rasch model. Through simulations and an international…

This retrospective case study assessed the influence of item non-response error on subsequent response to questionnaire items assessing adolescent alcohol and marijuana use. Post-hoc analyses were conducted on survey results obtained from 4,371 7th to 12th grade students in Ohio in 2005. A skip pattern design in a conventional questionnaire…

The valid provision of subscores from an item response theory-based test implies a multidimensional test structure. Assuming, in the construction of a new test, that the test features required for a valid and reliable total test score have been specified already, this article describes the resulting subscore performance and the resulting…

The Sato Caution Index takes into account the number and difficulty of items gotten wrong by a student within his or her ability, as well as the number and difficulty of items gotten right beyond his or her ability. Sato subtracts the two components to define a single Caution Index. In this study, the components are kept separate, defining a…

This research examined 2 elements of multiple-choice test construction, balancing the key and optimal number of options. In Experiment 1 the 3 conditions included a balanced key, overrepresentation of a and b responses, and overrepresentation of c and d responses. The results showed that error-patterns were independent of the key, reflecting…

Using simulation, the effect that misspecification of elements in the weight matrix has on estimates of basic parameters of the linear logistic test model was studied. Results indicate that, because specifying elements of the weight matrix is a subjective process, it must be done with great care. (SLD)

The National Assessment of Educational Progress (NAEP) collects data in the form of repeated, discrete measures (test items) with hierarchical structure for both measures and subjects, that is complex by any standard. This complexity has been managed through a "divide and conquer" approach of isolating and evaluating sources of…

The main purpose of this study was to illustrate a polytomous IRT-based linking procedure that adjusts for rater variations. Test scores from two administrations of a statewide reading assessment were used. An anchor set of Year 1 students' constructed responses were rescored by Year 2 raters. To adjust for year-to-year rater variation in IRT…

In a classroom, a teacher attempts to convey his or her knowledge to the students, and thus it is important for the teacher to obtain formative feedback about how well students are understanding the new material. By gaining insight into the students' understanding and possible misconceptions, the teacher will be able to adjust the teaching and to…

Uses item analysis techniques to compare two multiple-choice test formats for assessing second language students' competence in English structure in terms of mean scores, item difficulty, and item discrimination. Findings reveal that difficulty and discrimination aspects are item-specific rather than format-dependent and items with moderate…