This paper reports a multi-method approach for examining the cognitive level of multiple-choice items used in a medical pathology course at a large midwestern medical school. Analysis of the standard item analysis data and think-out-loud reports of a sample of students completing a 66 item examination were used to test assumptions related to the differences in cognitive demands pertinent to higher versus lower level multiple-choice items. Items answered by recalling information based exclusively on course content were coded as "knowledge." Items requiring reformulation of course information were coded as "thinking." The validity of the items' cognitive level categorization was assessed by item analyese data (item difficulty, item discrimination, and homogeneity of variance) and categorization of the think aloud responses of 12 students. Results indicated that thinking items were significantly more difficult than knowledge items. Discrimination, when difficulty was held constant, was significantly greater for knowledge items. It was concluded that faculty do write items which assess student ability to reason with what they know; and the method presented can be used by faculty to test their own judgment about the cognitive level of their test items. (BS)