Data from science and mathematics tests administered to 12th graders in Massachusetts were analyzed to compare the use of multiple-choice items with that of open-ended questions. Specifically, the study was designed to determine the statistical equivalence of the two types of test items, the information yielded by the items, and the grounds upon which one should be preferred over the other. Mathematics tests included 380 multiple-choice items and 12 free response questions, while science tests included 361 multiple-choice items and 11 free response questions. Data were taken from test administrations to students attending 27 schools randomly selected from a total population of 297 schools participating in the study. The number of students in each school ranged from 190 to 651, with a median of 15 students completing each form. A total of 100 students completed each open-ended question. Results indicate that students who perform well when presented with multiple-choice questions perform at comparable levels when presented with more open-ended questions; however, they do not perform equally well. Data from use of the open-ended questions provide far more direct evidence about students' understanding than do data from use of the multiple-choice items. Finally, open-ended responses show potential as instructional devices. Six sample test items and seven tables are included. (TJH)