NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
ERIC Number: ED645750
Record Type: Non-Journal
Publication Date: 2023
Pages: 166
Abstractor: As Provided
ISBN: 979-8-3816-9294-5
ISSN: N/A
EISSN: N/A
Available Date: N/A
Multimodal Modeling of Collaborative Learning with Adaptive Data Fusion
Yingbo Ma
ProQuest LLC, Ph.D. Dissertation, University of Florida
Collaborative learning provides learners with significant opportunities to collaborate on solving problems and creating better products. There has been a growing utilization of adaptive and intelligent systems to support productive learning while promoting collaborative practices. One of the core capabilities of these adaptive and intelligent systems is the modeling of the complex dynamics of collaborative interactions based on learners' behavioral cues. These cues can be extracted from multiple modalities, including speech, facial expressions, and gestures. Each of these modalities of data provides rich insights into the collaborative learning process from a different perspective that may not be obtained from other modalities. Decades of research have shown that combining multiple modalities of data, also referred to as multimodal data fusion, can support improved accuracy and more robust predictions. Despite the well-established advantages of multimodal data fusion, there are two main challenges in using this approach for collaborative learning. First, machine learning models trained with traditional data fusion methods are insufficient to learn the correlations and dependencies among different modalities. For example, late fusion, a widely adopted data fusion method, trains separate learning models with unimodal features and then considers the outputs from these models during the final decision-making process. These methods consider each modality as an independent entity, and hence ignore the relationships and dependencies between them. Second, the noise that often accompanies multimodal data poses a great challenge for intelligent systems to analyze and understand learners' dialogues. For example, during a collaborative learning activity, learners' speech can become significantly unclear and difficult to understand when it is disrupted by background noise from the classroom, and facial expressions of learners may appear incomplete if they are not directly oriented toward the cameras. How to build reliable multimodal models in the face of substantial data noise remains an open question. This dissertation presents a novel adaptive framework to address the above-mentioned two challenges in the task of modeling Confusion and Conflict in collaborative learning. Confusion and conflict represent critical moments when learners face cognitive and social challenges and learning support may be needed. To address the aforementioned first challenge of learning correlations and dependencies among modalities, the framework uses crossmodal attention mechanism, in which information from one modality is enriched by searching for relevant information from another modality. Experimental results showed that the crossmodal transformers demonstrated an overall accuracy (87%) improvement compared to a series of machine learning and deep learning baseline models (73%). To address the second challenge of modeling collaborative learning in the face of substantial data noise, the framework uses an additional adaptive fusion mechanism in which multimodal data are fused in a way in which proportional weights are assigned to each modality over time based on data quality. Experimental results show that this adaptive fusion mechanism can further boost model accuracy (89%) compared to models trained with the non-adaptive fusion mechanism (87%). Moreover, the adaptive fusion approach is effective in improving the averaged precision score (0.73) of Confusion and Conflict by a large margin compared to models trained with the non-adaptive fusion approach (0.56). These results provide strong empirical evidence that such an adaptive multimodal fusion framework is effective in addressing the above-mentioned two challenges related to the multimodal modeling of collaborative learning. This research contributes to developing intelligent systems that could significantly improve learners' collaborative learning experience in the future. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml.]
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A