ERIC Number: EJ1465642
Record Type: Journal
Publication Date: 2025-Apr
Pages: 25
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1360-2357
EISSN: EISSN-1573-7608
Available Date: 2024-09-19
Is ChatGPT Like a Nine-Year-Old Child in Theory of Mind? Evidence from Chinese Writing
Siyi Cao1,2; Yizhong Xu3; Tongquan Zhou1; Siruo Zhou4
Education and Information Technologies, v30 n5 p5787-5811 2025
ChatGPT has been demonstrated to possess significant capabilities in generating intricate human-like text, and recent studies have established that its performance in theory of mind (ToM) tasks is strikingly comparable to a nine-year-old child's. However, it remains unknown whether ChatGPT outperforms children of this age group in Chinese writing, a task credibly related to ToM. To justify the claim, this study compared ChatGPT with nine-year-old children in making Chinese compositions (i.e., science-themed and nature-themed narratives), aiming to unveil the relative advantages and disadvantages by human writers and ChatGPT in Chinese writing. Based on the evaluative framework comprising of four indices (i.e., fluency, accuracy, complexity, and cohesion) to test writing quality, this study added an often-overlooked index "emotion" to extend the framework. Afterward, we collected 120 writing samples produced by ChatGPT and children and used the confirmatory factor analysis (CFA) and structural equation modelling (SEM) for data analysis and comparison. The results revealed that this age group of children surpassed ChatGPT in fluency and cohesion while ChatGPT transcended the children in accuracy. With respect to complexity, the children exhibited better skills in science-themed writing, but ChatGPT better in nature-themed writing. Most importantly, this study unlocked the pioneering discovery that children display more potent emotional expressions than ChatGPT in Chinese writing, providing an instance of evidence that ChatGPT is really even poorer than a nine-year-old child in ToM to some extent.
Descriptors: Foreign Countries, Artificial Intelligence, Theory of Mind, Chinese, Writing (Composition), Writing Evaluation, Writing Skills, Emotional Experience, Emotional Response, Preadolescents, Comparative Analysis
Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link-springer-com.bibliotheek.ehb.be/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: China
Grant or Contract Numbers: N/A
Author Affiliations: 1Southeast University, School of Foreign Languages, Nanjing, China; 2The Hong Kong Polytechnic University, Department of Chinese and Bilingual Studies, Hung Hom, Hong Kong; 3Nanjing University of Aeronautics and Astronautics, College of Foreign Languages, Nanjing, China; 4Nanjing University of Posts and Telecommunications, School of Foreign Studies, Nanjing, China