The adequacy of a test developed for statewide assessment of basic mathematics skills was investigated. The test, comprised of multiple-choice items reflecting a series of behavioral objectives, was compared with a more extensive criterion measure generated from the same objectives by the application of a strict item sampling model. In many instances, the two instruments provided different classifications of students regarding mastery of an objective. Many of the discrepancies were attributed to the small number of items per objective and to the multiple-choice format of the original test. Consequently, the use of criterion-referenced tests in situations that severely limit test length and item format options was questioned. In addition, the problems associated with the practice of assuming content validity for criterion-referenced tests were discussed. (Author)