MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

Pretrained language models are promising particularly for low-resource languages as they only require unlabelled data. However, training existing models requires huge amounts of compute, while pretrained cross-lingual models often underperform on low-resource languages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune language models efficiently in their own language. In addition, we propose a zero-shot method using an existing pretrained cross-lingual model. We evaluate our methods on two widely used cross-lingual classification datasets where they outperform models pretrained on orders of magnitude more data and compute. We release all models and code.

PDF Abstract IJCNLP 2019 PDF IJCNLP 2019 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to French - Books MultiFiT, pseudo Accuracy 87.84 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to French - DVD MultiFiT, pseudo Accuracy 83.5 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to French - Music MultiFiT, pseudo Accuracy 85.6 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to German - Books MultiFiT, pseudo Accuracy 89.6 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to German - DVD MultiFiT, pseudo Accuracy 81.8 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to German - Music MultiFiT, pseudo Accuracy 84.4 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to Japanese - Books MultiFiT, pseudo Accuracy 80.45 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to Japanese - DVD MultiFiT, pseudo Accuracy 77.65 # 1
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to Japanese - Music MultiFiT, pseudo Accuracy 81.5 # 1
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Chinese MultiFiT, pseudo Accuracy 82.48 # 2
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-French MultiFiT, pseudo Accuracy 89.42 # 2
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-German MultiFiT, pseudo Accuracy 91.62% # 2
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Italian MultiFiT, pseudo Accuracy 76.02 # 1
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Japanese MultiFiT, pseudo Accuracy 69.57 # 1
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Russian MultiFiT, pseudo Accuracy 67.83 # 2
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Spanish MultiFiT, pseudo Accuracy 79.1 # 2

Methods


No methods listed for this paper. Add relevant methods here