Concept inventories have been determined to be useful assessment tools for evaluating students' knowledge, particularly in the sciences. However, these assessment tools must be validated to reflect as accurately as possible students' understanding of concepts. One possible threat to this validation is what previous literature calls the item order effect: that the ordering of items on an assessment may affect how students perform. This study attempts to test the item order effect using different orderings of pictorial and verbal items on a geoscience concept inventory. Two different orderings of the same concept inventory were created and distributed to students in introductory geoscience courses via an online survey software (n = 432). The students' performance on the pictorial and verbal items was statistically analyzed. Seven interviews were also conducted to probe students about their preferred item format and order. The results revealed no predictable item order effect, though suggestive trends are present that warrant further study. Implications for educators and researchers are discussed.