Continual Pretraining
22 papers with code • 3 benchmarks • 3 datasets
Libraries
Use these libraries to find Continual Pretraining models and implementations
AG News
SciERC
ACL ARC
Most implemented papers
Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis
In recent years, pretrained language models have revolutionized the NLP world, while achieving state of the art performance in various downstream tasks.
Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding
leiboliu/hilat
•
•
In this study, we propose a hierarchical label-wise attention Transformer model (HiLAT) for the explainable prediction of ICD codes from clinical documents.
Continual Pre-Training Mitigates Forgetting in Language and Vision
andreacossu/continual-pretraining-nlp-vision
•
•
We formalize and investigate the characteristics of the continual pre-training scenario in both language and vision environments, where a model is continually pre-trained on a stream of incoming data and only later fine-tuned to different downstream tasks.
Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps
meshidenn/cai
•
•
We conducted experiments using our method on datasets with a large vocabulary gap from a source domain.
AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
Continual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model.
CTP:Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation
kevinlight831/ctp
•
•
ICCV 2023
Regarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly.
CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation
Regarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly.
Effective Long-Context Scaling of Foundation Models
openlmlab/leval
•
•
We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessment
plrbear/pecop
•
•
The limited availability of labelled data in Action Quality Assessment (AQA), has forced previous works to fine-tune their models pretrained on large-scale domain-general datasets.
Continual Learning for Large Language Models: A Survey
wang-ml-lab/llm-continual-learning-survey
•
•
Large language models (LLMs) are not amenable to frequent re-training, due to high training costs arising from their massive scale.
