This paper presents the integration of a real time evaluation method of collaboration quality in a monitoring application that supports teachers in class orchestration. The method is implemented as an automatic rater of collaboration quality and studied in a real time scenario of use. We argue that automatic and semi-automatic methods which provide assessments of collaboration quality or detect disorders and anomalies in collaborative activities should be further used to assist class monitoring. To that end, we study the routine of teachers when they monitor computer-supported collaborative activities with respect to the size of the classroom and the support of an automatic rater. We focus on the way the rater affects their task and how it can be further used to empower the practice of teachers.