This paper reports on data collected via an audience response system, where a convenience sample of 300 adults aged 17-50 pressed a button to register their answers for twenty multiple choice questions. The responses were then discussed with the respondents at the time. The original dataset includes physics, biology and chemistry questions. The questions were derived from the Assessment of Achievement Programme, Sixth Survey of Science 2003 [AAP 2003] (Scottish Executive Education Dept, 2005). The findings presented in this paper are a subset of the data collected as the paper discusses the designs seen in terms of the style and nature of chemistry questions and the degree of perceived confidence in, and the answers provided by, 300 adult participant responses. The focus of this paper is not on misconceptions or participants' strong or weak understanding of science. Instead the paper analyses the demands made by the questions. The findings suggest that structuring multiple choice chemistry questions is complex. What may appear to be a simple question in terms of the curriculum prescribed levels may be rendered more complex phrasing, familiarity and taxonomy. Furthermore, if international studies rely on multiple choice questions which favour testing recall of information, then it is possible that pedagogy that supports recall rather than comprehension or application may be endorsed. (Contains 1 table.)