The validity of criterion-referenced interpretations of the proficiency scales of the National Assessment of Educational Progress (NAEP) is discussed. A major goal of the NAEP scales is to describe student achievement in specific content areas from grade 3 (age 9 years) through grade 11 (age 17 years). The numerical values for NAEP scales are intended to be interpreted almost exclusively from a criterion-referenced perspective. The NAEP intends to accomplish the goal of making criterion-referenced interpretations of examinee scores through the use of item response theory methodology. The basis for classifying an item at a proficiency level is the estimated probability of success of examinees at different proficiency levels. The implications for criterion-referenced interpretations are discussed in the context of the Science Proficiency Scale (SPS). Problems with interpreting SPS results are due to the following factors: (1) the science domain is not well defined; (2) it is doubtful that learning in this domain proceeds through an orderly sequence; (3) the number of test items that could be developed is virtually unlimited; and (4) many problems are generated by the interactions between the examinee's past experiences and item content. The NAEP mathematics, reading, and history scales are also reviewed. It is suggested that the NAEP proficiency scales do not yield valid criterion-referenced interpretations. Four figures illustrate the discussion. A 27-item list of references is included. (SLD)