Teachers in a collaborative learning (CL) environment have the demanding task of monitoring several groups of students at the same time and intervening when needed. This withitness (both the situational awareness and interventions taken in class) of the teacher might be increased with the help of a guiding dashboard alerting the teacher of problems and providing suggestions for interventions. This paper introduces a quasi-experiment carried out in authentic classrooms. We examined how a mirroring and an alerting & guiding dashboard affected the withitness of teachers in a face-to-face learning environment while students discussed and used a collaborative writing tool. Twenty-four teachers were observed, interviewed, and answered surveys in three different conditions altogether: with no extra information about the situation, using a dashboard mirroring low-level data about the collaboration, and additionally an AI assistant indicating problems in pedagogical terms and potential solutions (i.e., a guiding dashboard). The results show that the situational awareness of the teachers increased with the introduction of a mirroring dashboard. The workload of the participating teachers dropped more with the introduction of an alerting & guiding dashboard, helping teachers feel less frustrated and more accomplished.