HuggingFace's Transformers: State-of-the-art Natural Language Processing

9 Oct 20191 code implementation

In this paper, we present HuggingFace's Transformers library, a library for state-of-the-art NLP, making these developments available to the community by gathering state-of-the-art general-purpose pretrained models under a unified API together with an ecosystem of libraries, examples, tutorials and scripts targeting many downstream NLP tasks.

TEXT GENERATION TRANSFER LEARNING

AllenNLP: A Deep Semantic Natural Language Processing Platform

WS 2018 1 code implementation

This paper describes AllenNLP, a platform for research on deep learning methods in natural language understanding.

READING COMPREHENSION SEMANTIC ROLE LABELING

GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing

9 Jul 20192 code implementations

We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating).

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

ACL 2019 10 code implementations

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.

LANGUAGE MODELLING

RoBERTa: A Robustly Optimized BERT Pretraining Approach

26 Jul 20196 code implementations

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.

 SOTA for Question Answering on SQuAD2.0 dev (using extra training data)

LANGUAGE MODELLING LEXICAL SIMPLIFICATION NATURAL LANGUAGE INFERENCE QUESTION ANSWERING READING COMPREHENSION SEMANTIC TEXTUAL SIMILARITY SENTIMENT ANALYSIS

XLNet: Generalized Autoregressive Pretraining for Language Understanding

NeurIPS 2019 11 code implementations

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling.

DOCUMENT RANKING LANGUAGE MODELLING NATURAL LANGUAGE INFERENCE QUESTION ANSWERING READING COMPREHENSION SEMANTIC TEXTUAL SIMILARITY SENTIMENT ANALYSIS TEXT CLASSIFICATION

Cross-lingual Language Model Pretraining

NeurIPS 2019 7 code implementations

On unsupervised machine translation, we obtain 34. 3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU.

LANGUAGE MODELLING UNSUPERVISED MACHINE TRANSLATION

CamemBERT: a Tasty French Language Model

10 Nov 20191 code implementation

We measure the performance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging, dependency parsing, named-entity recognition, and natural language inference.

DEPENDENCY PARSING LANGUAGE MODELLING NAMED ENTITY RECOGNITION NATURAL LANGUAGE INFERENCE PART-OF-SPEECH TAGGING

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

26 Sep 201911 code implementations

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks.

LINGUISTIC ACCEPTABILITY NATURAL LANGUAGE INFERENCE QUESTION ANSWERING SEMANTIC TEXTUAL SIMILARITY