Unsupervised Cross-lingual Representation Learning at Scale

ACL 2020 Alexis ConneauKartikay KhandelwalNaman GoyalVishrav ChaudharyGuillaume WenzekFrancisco GuzmánEdouard GraveMyle OttLuke ZettlemoyerVeselin Stoyanov

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data... (read more)

PDF Abstract

Results from the Paper

  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods used in the Paper