ERIC Number: ED659131
Record Type: Non-Journal
Publication Date: 2024
Pages: 162
Abstractor: As Provided
ISBN: 979-8-3829-5204-8
ISSN: N/A
EISSN: N/A
Available Date: N/A
Continual Learning with Language Models
Zixuan Ke
ProQuest LLC, Ph.D. Dissertation, University of Illinois at Chicago
The essence of human intelligence lies in its ability to learn continuously, accumulating past knowledge to aid in future learning and problem-solving endeavors. In contrast, the current machine learning paradigm often operates in isolation, lacking the capacity for continual learning and adaptation. This deficiency becomes apparent in the face of rapidly evolving artificial intelligence (AI) technologies, particularly large language models (LLMs), where incremental training remains a challenge. Continual learning (CL), also known as lifelong learning, is indispensable for truly intelligent systems, especially in dynamic environments where constant adaptation is necessary. This dissertation explores recent advancements in continual learning algorithms within the framework of language models. We first introduce the settings, challenges, and general approaches of CL. We then delve into our efforts to achieve both catastrophic forgetting (CF) mitigation and knowledge transfer (KT), and how we apply CL to different stages of language model development, including pre-training and end-task adaptation. With the aid of continual learning, the performance of language models is greatly improved. Finally, we will discuss the opportunities for AI autonomy and open-world continual learning. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml.]
Descriptors: Computational Linguistics, Computer Software, Barriers, Artificial Intelligence, Lifelong Learning, Algorithms, Learning Analytics, Models, Transfer of Training
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A