Data from the Third International Mathematics and Science Study (TIMSS) were examined to determine the extent to which the rank ordering of countries based on pupil test performance was consistent across three different item formats: multiple-choice, short-answer, and extended-response. Findings from the analysis are used to make the case that international comparative studies are very complex and that the data they generate cannot be taken at face value but need close examination before firm conclusions can be drawn about a country's relative performance. The focus was the science performance of Irish second year secondary school students (Grade 8) in TIMSS across different item types, comparing this with the performance of similar cohorts in 11 other countries. Irish student performance was close to the international averages for short-answer and multiple-choice items, but performance on extended-response items was significantly above the international average. An examination of the match of these test items to the Irish curriculum was not good, and the Irish curriculum was judged to encourage higher-order thinking less than in other countries. Both of these factors made the good performance on extended-response items surprising. In many respects, these findings confirm the suspicion of W. Cooley and G. Leinhart (1980) that frequent exposure to test format will make a difference in performance. In Ireland there is a tradition of more open-ended essay type tests, and this may account for students' success with extended-response items. These findings also demonstrate the difficulties involved in making international comparisons of academic performance. An appendix contains a table of science averages for TIMSS participants. (Contains 63 references.) (SLD)