In the field of Science, Technology, Engineering, and Mathematics (STEM) education, which aims to cultivate problem-solving skills, accurately assessing learners' engagement remains a significant challenge. We present a solution to this issue with the Real-time Automated STEM Engagement Detection System (RASEDS). This innovative system capitalizes on the power of artificial intelligence, computer vision, and the Interactive, Constructive, Active, and Passive (ICAP) framework. RASEDS uses You Only Learn One Representation (YOLOR) to detect and map learners' interactions onto the four levels of engagement delineated in the ICAP framework. This process informs the system's recommendation of adaptive learning materials, designed to boost both engagement and self-efficacy in STEM activities. Our study affirms that RASEDS accurately gauges engagement, and that the subsequent use of these adaptive materials significantly enhances both engagement and self-efficacy. Importantly, our research suggests a connection between elevated self-efficacy and increased engagement. As learners become more engaged in their learning process, their confidence is bolstered, thereby augmenting self-efficacy. We underscore the transformative potential of AI in facilitating adaptive learning in STEM education, highlighting the symbiotic relationship between engagement and self-efficacy.