The statute authorizing the National Assessment of Educational Progress (NAEP) calls for the National Assessment Governing Board (NAGB) to set appropriate achievement levels in all areas and grades tested by the NAEP. These levels are intended to establish what students should know, not just what they do know. In 1990, the NAEP posited three achievement levels (advanced, proficient, and basic) at each grade level assessed (grades 4, 8, and 12). This preliminary report examines the reasonableness and technical adequacy of the achievement levels, addressing the fundamental question of whether the levels are adequate for supporting the conclusions or inferences for which they will be used. Analyses indicate that the achievement levels are seriously flawed. Robustness appears to be inadequate in two respects: (1) interrater reliability is inadequate; and (2) the coherence of rating across the three grades has not been established. The available evidence does not address the question of whether the achievement levels were actually criterion-based, although there are suggestions that they were not. It is suggested that practical post hoc adjustments will not remedy these problems. It is recommended that these achievement levels should not be used in any public reporting of state or national NAEP results. Fourteen tables and 22 figures present results of the analyses. (SLD)