The multilevel characteristics of test item data are considered as a method for examining the characteristics of standardized norm-referenced tests. A theoretical rationale for examining multilevel characteristics is presented. It can be used as an aid to understand why program and instructional effects on measures constructed from individual-level psychometric data are weak, to improve instructional sensitivity and program relevance of tests, and to indicate what features of a test will increase the sensitivity of the test to instructional program variables. An empirical example form the International Evaluation of Education Achievement study and an analysis of one test from the Beginning Teacher Evaluation Study are examined to better understand how multilevel analysis can lead to more informed use of item data. It was found that between-student analysis fails to take into account the instructional context and its effect on student item response. This failure has two effects. First, the relationship between item response and other variables cannot be explained since it is a conglomerate of two different processes. Second, the between-student analysis may give a distorted view of whether an effect does or does not exist. (Author/PN)