ERIC Number: EJ1346547
Record Type: Journal
Publication Date: 2022-May
Pages: 19
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-1492-3831
Available Date: N/A
Cross Validating a Rubric for Automatic Classification of Cognitive Presence in MOOC Discussions
Hu, Yuanyuan; Donald, Claire; Giacaman, Nasser
International Review of Research in Open and Distributed Learning, v23 n2 p242-260 May 2022
As large-scale, sophisticated open and distance learning environments expand in higher education globally, so does the need to support learning at scale in real time. Valid, reliable rubrics of critical discourse are an essential foundation for developing artificial intelligence tools that automatically analyse learning in educator-student dialogue. This article reports on a validation study where discussion transcripts from a target massive open online course (MOOC) were categorised into phases of cognitive presence to cross validate the use of an adapted rubric with a larger dataset and with more coders involved. Our results indicate that the adapted rubric remains stable for categorising the target MOOC discussion transcripts to some extent. However, the proportion of disagreements between the coders increased compared to the previous experimental study with fewer data and coders. The informal writing styles in MOOC discussions, which are not as prevalent in for-credit courses, caused ambiguities for the coders. We also found most of the disagreements appeared at adjacent phases of cognitive presence, especially in the middle phases. The results suggest additional phases may exist adjacent to current categories of cognitive presence when the educational context changes from traditional, smaller-scale courses to MOOCs. Other researchers can use these findings to build automatic analysis applications to support online teaching and learning for broader educational contexts in open and distance learning. We propose refinements to methods of cognitive presence and suggest adaptations to certain elements of the Community of Inquiry (CoI) framework when it is used in the context of MOOCs.
Descriptors: Validity, Scoring Rubrics, Automation, Classification, Cognitive Processes, MOOCs, Discussion, Content Analysis, Asynchronous Communication
Athabasca University Press. 1200, 10011-109 Street, Edmonton, AB T5J 3S8, Canada. Tel: 780-497-3412; Fax: 780-421-3298; e-mail: irrodl@athabascau.ca; Web site: http://www.irrodl.org
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A