Most research in psycholinguistics relies on online measures such as reading time to inform and test theories of language comprehension. However, the value of offline measures such as question-answering performance is sometimes overlooked in sentence processing work. Consequently, psycholinguists do not yet understand how the tasks and measures used in online experiments might reflect the content of the representations that are formed by the subject and how our comprehension tasks might influence subjects' reading strategies. We begin this review by briefly discussing evidence that comprehenders often misinterpret language. We then consider some of the reasons why psycholinguists tend not to use offline measures of comprehension to evaluate competing theories of sentence processing and discuss the role of subject engagement and motivation. Finally, we explore what is currently known about the relationships among task, depth of comprehension, and reading measures.