Issues in designing valid tests for the National Assessment of Educational Progress (NAEP) are discussed. Test scores are often provided without any information on the nature of the tasks represented. Because test domains are defined by individual item writers, the generalizability between tests and items is suspect. While typical content validation procedures help assure that the included items are important, they still might not represent the full range of knowledge and skills constituting given domains. As a result, the underlying meaning of what is tested is vague, and the specific definition of what is to be tested escapes public scrutiny. This is especially important when matching particular tests and curricula among states. Better specification of test content and task structure is recommended. Elements in good task structure should include: task description; content limits; linguistic features; cognitive complexity; and format. Recent NAEP assessments defined four different types of context for test items: (1) scientific, (2) personal, (3) societal, and (4) technological. Three levels of cognitive complexity items were defined: (1) knows, (2) uses, and (3) integrates. Six categories of subject content were specified. In conclusion, NAEP planners should emphasize content validity; define more specifically what is to be tested; provide better models for item construction; and assure that the entire domain is represented. (GDC)