The level of students' engagement during active learning activities conducted in small groups is important to understanding the effectiveness of these activities. The Interactive-Constructive-Active-Passive (ICAP) framework is a way to determine the cognitive engagement of these groups by analyzing the conversations that occur while student groups work on an activity. This study used qualitative content analysis and ICAP to investigate cognitive engagement during group activities in a General Chemistry course at the question level, a finer grain size than previously studied. The analysis determined the expected engagement based on question design and the observed engagement based on group conversations. Comparisons of expected and observed engagement showed cases of mismatch, and further analysis determined that incorrect model use, unfamiliar scientific vocabulary, and difficulty moving between molecular representations were all contributing themes to the observed mismatches. The implications of these findings with regard to teaching and research are discussed.