Educational videos are increasingly used to let students prepare lesson material at home prior to in-class activities in flipped classrooms. The main challenge of this teaching strategy is to stimulate students to watch these videos attentively before going to class. This paper describes the use of questions that pop-up within relatively long educational videos of 16 min on average and designed to enhance students' engagement and understanding when preparing for in-class activities. The effects of such pop-up questions on students' learning performance were studied within a flipped course in molecular biology. Students had access to videos with or without a variable set of pop-up questions. The experimental group with pop-up questions showed significantly higher test results compared to the group without pop-up questions. Interestingly, students that answered pop-up questions on certain concepts did not score better on items testing these specific concepts than the control group. These results suggest that merely the presence of pop-up questions enhances students' learning. Additional data from interviews, surveys, and learning analytics suggest that pop-up questions influence viewing behavior, likely by promoting engagement. It is concluded that pop-up questions stimulate learning when studying videos outside class through an indirect testing effect.