There is a gap in the current research literature regarding evaluation of TAFE outcomes and it stems from a predominant human capital focus. The existing paradigm of human capital, which values the acquisition of knowledge and skills for their economic value, has been of primary interest and significance, particularly in terms of government policy…

This paper introduces criterion-referenced tests (CRTs), compares them with norm-referenced tests (NRTs), discusses how they can be evaluated and revised, and presents a study of an actual class and textbook test evaluation using CRTs. NRTs have dominated testing methodology since the mid-1970s; an example is the Test of English as a Foreign…

A study was designed to provide recommendations regarding the use of the achievement levels set in 1992 for reporting National Assessment of Educational Progress (NAEP) reading results in 1994 and in future NAEP reading assessments. Two procedures were used: the Item Difficulty Categorization (IDC) procedure involved an evaluation of the…

This paper presents a technique for applying the Rule Space Model of cognitive diagnosis (Tatsuoka, 1983) to assessment in a semantically rich domain. Responses of 122 architects to 22 architecture test items developed to assess a range of architectural knowledge were analyzed using Rule Space. Verbal protocol analysis guided the construction of a…

This technical report describes the development of the College-Level Academic Skills Test (CLAST), an instrument designed to measure Florida college students' achievement of the computation and communication skills expected by the completion of their sophomore year. Section I covers CLAST's background and purpose, the requirement that all students…

A study was undertaken to identify the prerequisite relations (or hierarchies among the items) existing in the item responses of a sample of 86 foreign students who took the Test of English as a Foreign Language (TOEFL) vocabulary and reading comprehension test, Form 3JTF1. The form contains 30 vocabulary items and 30 reading comprehension items.…

In item response theory (IRT), preequating depends upon item parameter estimate invariance. Three separate simulations, all using the unidimensional three-parameter logistic item response model, were conducted to study the impact of the following variables on preequating: (1) mean differences in ability; (2) multidimensionality in the data; and…

The purpose of this study was to determine whether item difficulty is significantly affected by language difficulty and response set convergence. Language difficulty was varied by increasing sentence (stem) length, increasing syntactic complexity, and substituting uncommon words for more familiar terms in the item stem. Item wording ranged from…

Four methods for determining the dimensionality of a set of test items were compared: (1) linear factor analysis; (2) residual analysis; (3) nonlinear factor analysis; and (4) Bejar's method. Five artificial test data sets (for 40 items and 1500 examinees) were generated, consistent with the three-parameter logistic model and the assumption of…

The goal of test design is to devise an instrument that will provide a stable and accurate assessment of student ability in some area. One means of reaching this goal is through the use of latent trait models, which determine the relationship between the unobservable trait or ability and the observable test performance. Three common latent trait…

The purpose of this paper is to present two new alternative methods to the current goodness of fit methodology. With the increase use of computerized adaptive test (CAT), the ability to determine the accuracy of calibrated item parameter estimates is paramount. The first method applies a normalizing transformation to the logistic residuals to make…

The effects of a multidimensional latent trait space on estimation of item and person parameters by the computer program LOGIST are examined. Several item pools were simulated that ranged from truly unidimensional to an inconsequential general latent trait. Item pools with intermediate levels of prepotency of the general latent trait were also…

A review of the implications for the validity of the Scholastic Aptitude Test (SAT) of scientific developments accompanying the revival of cognitive psychology provides insights into the importance of such changes. A distinction can be made between a process-oriented or diagnostic test and an outcomes-oriented test such as the SAT. Since the SAT…

Although item bias statistics are widely recommended for use in test development and test analysis work, problems arise in their interpretation. The purpose of the present research was to evaluate the validity of logistic test models and computer simulation methods for providing a frame of reference for item bias statistic interpretations.…

At Santa Fe Community College, the development of a biology course and departmental final examination was accomplished in the following stages: (1) faculty reached consensus on course content, major objectives and sub-objectives; (2) faculty contributed final examination questions corresponding to the 10 major course topics to a pool for review…