Background: Extensive second language (L2) reading research has examined the effects of comprehension strategies on comprehension performance. Often ignored, however, is the potential impact of comprehension measures that differ in cognitive complexity, which might cause potential bias when interpreting the (in)effectiveness of comprehension strategies. Method: With a total of 54 Chinese learners of English at college in China, this study examined whether the effect of the questioning strategy (answering "what" and "why" questions while reading) on L2 reading comprehension of expository texts might vary by comprehension measure (short-answer and multiple-choice questions). L2 readers' perception of the questioning strategy was also thematically analysed. Results: Mixed-effects modelling analyses revealed a significant interaction between the questioning strategy and comprehension measure, controlling for individual-level differences in first language (L1) reading ability, L2 vocabulary size and topic familiarity. Using the questioning strategy significantly facilitated comprehension measured by short-answer questions but not multiple-choice questions. Three themes regarding L2 readers' perception of the questioning strategy were identified: memorising text details, understanding main ideas and disrupting reading fluency. Conclusion: Evaluation of comprehension strategies needs to consider comprehension measures differing in response format and cognitive complexity. For positive effect to occur, there needs a match between cognitive processing triggered by a comprehension strategy and cognitive skills demanded to successfully complete a comprehension task.