XLM

Introduced by Lample et al. in Cross-lingual Language Model Pretraining

XLM is a Transformer based architecture that is pre-trained using one of three language modelling objectives:

  1. Causal Language Modeling - models the probability of a word given the previous words in a sentence.
  2. Masked Language Modeling - the masked language modeling objective of BERT.
  3. Translation Language Modeling - a (new) translation language modeling objective for improving cross-lingual pre-training.

The authors find that both the CLM and MLM approaches provide strong cross-lingual features that can be used for pretraining models.

Source: Cross-lingual Language Model Pretraining

Latest Papers

PAPER DATE
Probing for Multilingual Numerical Understanding in Transformer-Based Language Models
| Devin JohnsonDenise MakDrew BarkerLexi Loessberg-Zahl
2020-10-13
Chatbot Interaction with Artificial Intelligence: Human Data Augmentation with T5 and Language Transformer Ensemble for Text Classification
Jordan J. BirdAnikó EkártDiego R. Faria
2020-10-12
SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task
Zuchao LiHai ZhaoRui WangKehai ChenMasao UtiyamaEiichiro Sumita
2020-10-11
Evaluating Multilingual BERT for Estonian
Claudia KittaskKirill MilintsevichKairit Sirts
2020-10-01
What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
Rochelle ChoenniEkaterina Shutova
2020-09-27
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding
Yuwei FangShuohang WangZhe GanSiqi SunJingjing Liu
2020-09-10
On Learning Universal Representations Across Languages
Xiangpeng WeiYue HuRongxiang WengLuxi XingHeng YuWeihua Luo
2020-07-31
Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining
Ivana Kvapil{\'\i}kov{\'a}Mikel ArtetxeGorka LabakaEneko AgirreOnd{\v{r}}ej Bojar
2020-07-01
Cooking Is All About People: Comment Classification On Cookery Channels Using BERT and Classification Models (Malayalam-English Mix-Code)
Subramaniam KazhuparambilAbhishek Kaushik
2020-06-15
Transfer learning applied to text classification in Spanish radiological reports
Pilar L{\'o}pez {\'U}bedaManuel Carlos D{\'\i}az-GalianoL. Alfonso Urena LopezMaite MartinTeodoro Mart{\'\i}n-NoguerolAntonio Luna
2020-05-01
``A Passage to India'': Pre-trained Word Embeddings for Indian Languages
Saurav KumarSaunack KumarDiptesh KanojiaPushpak Bhattacharyya
2020-05-01
KLEJ: Comprehensive Benchmark for Polish Language Understanding
| Piotr RybakRobert MroczkowskiJanusz TraczIreneusz Gawlik
2020-05-01
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
| Yaobo LiangNan DuanYeyun GongNing WuFenfei GuoWeizhen QiMing GongLinjun ShouDaxin JiangGuihong CaoXiaodong FanRuofei ZhangRahul AgrawalEdward CuiSining WeiTaroon BhartiYing QiaoJiun-Hung ChenWinnie WuShuguang LiuFan YangDaniel CamposRangan MajumderMing Zhou
2020-04-03
XD: Cross-lingual Knowledge Distillation for Polyglot Sentence Embeddings
Anonymous
2020-01-01
Unsupervised Cross-lingual Representation Learning at Scale
| Alexis ConneauKartikay KhandelwalNaman GoyalVishrav ChaudharyGuillaume WenzekFrancisco GuzmánEdouard GraveMyle OttLuke ZettlemoyerVeselin Stoyanov
2019-11-05
Masked Language Model Scoring
| Julian SalazarDavis LiangToan Q. NguyenKatrin Kirchhoff
2019-10-31
Multilingual Question Answering from Formatted Text applied to Conversational Agents
| Wissam SibliniCharlotte PasqualAxel LavielleCyril Cauchois
2019-10-10
Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
Yuxuan WangWanxiang CheJiang GuoYijia LiuTing Liu
2019-09-15
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Haoyang HuangYaobo LiangNan DuanMing GongLinjun ShouDaxin JiangMing Zhou
2019-09-03
Detecting Toxicity in News Articles: Application to Bulgarian
| Yoan DinkovIvan KoychevPreslav Nakov
2019-08-26
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
| Gen LiNan DuanYuejian FangMing GongDaxin JiangMing Zhou
2019-08-16
Unbabel's Participation in the WMT19 Translation Quality Estimation Shared Task
Fabio KeplerJonay TrénousMarcos TrevisoMiguel VeraAntónio GóisM. Amin FarajianAntónio V. LopesAndré F. T. Martins
2019-07-24
Cross-lingual Language Model Pretraining
| Guillaume LampleAlexis Conneau
2019-01-22

Categories