Classroom response systems (i.e., clickers) have become increasingly popular to facilitate student learning. Unfortunately, the common practice of pausing a lecture to ask questions takes up precious time to cover content. Asking questions "on the fly" without pausing is a possible solution. But can students both attend to lecture and answer questions simultaneously? Is this multitasking detrimental to student learning? In three experiments, we examined the effects of relevant and irrelevant "on-the-fly" questions and note taking on lecture retention. Undergraduates watched a video of a classroom lecture while either taking notes or not and receiving 0, 6, 18, or 36 questions that were either relevant or irrelevant to the lecture and then took a test. Students performed better on the test when receiving relevant rather than irrelevant questions. As for an optimal number of questions or whether note taking should also be allowed, there were no obvious advantages. Thus, when considering using "on the fly" clicker questions during a lecture vs. having no such questions, our evidence indicates no clear interference. Rather, such activities such as clickers may counter lecture boredom by allowing students to multitask with relevant activities.