XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We will release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

PDF ICML 2020 PDF

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Zero-Shot Cross-Lingual Transfer XTREME mBERT AVG 59.6 # 1
Sentence-pair Classification 73.7 # 24
Structured Prediction 66.3 # 23
Question Answering 53.8 # 24
Sentence Retrieval 47.7 # 21

Methods


No methods listed for this paper. Add relevant methods here