arXiv 2019

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

arXiv 2019 google-research/text-to-text-transfer-transformer

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP).

 Ranked #1 on Semantic Textual Similarity on STS Benchmark (using extra training data)

LINGUISTIC ACCEPTABILITY NATURAL LANGUAGE INFERENCE QUESTION ANSWERING SEMANTIC TEXTUAL SIMILARITY SENTIMENT ANALYSIS TEXT CLASSIFICATION TRANSFER LEARNING

Language Models with Transformers

arXiv 2019 cgraywang/gluon-nlp-1

In this paper, we explore effective Transformer architectures for language model, including adding additional LSTM layers to better capture the sequential context while still keeping the computation efficient.

LANGUAGE MODELLING NEURAL ARCHITECTURE SEARCH

Language Models with Transformers

arXiv 2019 cgraywang/gluon-nlp-1

In this paper, we explore effective Transformer architectures for language model, including adding additional LSTM layers to better capture the sequential context while still keeping the computation efficient.

LANGUAGE MODELLING