Classroom engagement is a critical factor for evaluating students' learning outcomes and teachers' instructional strategies. Traditional methods for detecting classroom engagement, such as coding and questionnaires, are often limited by delays, subjectivity, and external interference. While some neural network models have been proposed to detect engagement using video data, they generally rely on fixed feature combinations, which fail to capture the logical connections and temporal dynamics of engagement. To address these challenges, this article introduces the MSC-Trans Engagement Detecting Network, a temporal multimodal data fusion framework that integrates a convolutional neural network (CNN) and a multilayer encoder-decoder structure. The proposed network includes two key components: first, a multilabel classifier based on ResNet and Transformer, which embeds labels into image features extracted by the CNN for high-precision classification through background inference, second, a temporal feature fusion module, which leverages an encoder-decoder structure to integrate multimodal features over time, enabling stable tracking of classroom engagement. Meanwhile, this open framework allows users to freely select feature combinations for temporal fusion based on specific scenarios and needs. The MSC-Trans Engagement Detecting Network was validated on the DAiSEE dataset, augmented with real classroom data. Experimental results demonstrate that the proposed method achieves state-of-the-art performance in continuous engagement tracking metrics, with flexible and scalable feature selection. This work offers a robust and effective approach for advancing engagement detection in educational settings.