Universal Language Model Fine-tuning for Text Classification

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model... (read more)

PDF Abstract ACL 2018 PDF ACL 2018 Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
RESULT BENCHMARK
Text Classification AG News ULMFiT Error 5.01 # 4
Text Classification DBpedia ULMFiT Error 0.80 # 6
Sentiment Analysis IMDb ULMFiT Accuracy 95.4 # 9
Text Classification TREC-6 ULMFiT Error 3.6 # 3
Sentiment Analysis Yelp Binary classification ULMFiT Error 2.16 # 6
Sentiment Analysis Yelp Fine-grained classification ULMFiT Error 29.98 # 5

Methods used in the Paper


METHOD TYPE
Dropout
Regularization
Adam
Stochastic Optimization
Sigmoid Activation
Activation Functions
Tanh Activation
Activation Functions
Temporal Activation Regularization
Regularization
DropConnect
Regularization
LSTM
Recurrent Neural Networks
Activation Regularization
Regularization
Embedding Dropout
Regularization
Variational Dropout
Regularization
Weight Tying
Parameter Sharing
AWD-LSTM
Recurrent Neural Networks
Discriminative Fine-Tuning
Fine-Tuning
Slanted Triangular Learning Rates
Learning Rate Schedules
ULMFiT
Language Models