Development of the test instrument in the Third International Mathematics and Science Study (TIMSS) was based on the expertise of many researchers, including "distinguished scholars from 10 countries" who participated on the TIMSS Subject Matter Advisory Committee. However, a close examination of the TIMSS Science items suggests that not all the items measure student science achievement. Thus, cautions are raised from this study to urge researchers and policymakers to take a more mindful stance against potential misinterpretation of TIMSS scores in international comparisons. (Author/WRM)