Browse SoTA > Methodology > Transfer Learning

Transfer Learning

797 papers with code · Methodology

Transfer learning is a methodology where weights from a model trained on one task are taken and either used (a) to construct a fixed feature extractor, (b) as weight initialization and/or fine-tuning.

( Image credit: Subodh Malgonde )

Benchmarks

Greatest papers with code

Talking-Heads Attention

5 Mar 2020tensorflow/models

We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation. While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-heads attention leads to better perplexities on masked language modeling tasks, aswell as better quality when transfer-learning to language comprehension and question answering tasks.

LANGUAGE MODELLING QUESTION ANSWERING TRANSFER LEARNING

Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data

18 Oct 2016tensorflow/models

The approach combines, in a black-box fashion, multiple models trained with disjoint datasets, such as records from different subsets of users.

TRANSFER LEARNING

Movement Pruning: Adaptive Sparsity by Fine-Tuning

NeurIPS 2020 huggingface/transformers

Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications.

TRANSFER LEARNING

HuggingFace's Transformers: State-of-the-art Natural Language Processing

9 Oct 2019huggingface/transformers

Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks.

TEXT GENERATION TRANSFER LEARNING

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

NeurIPS 2019 huggingface/transformers

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging.

LANGUAGE MODELLING LINGUISTIC ACCEPTABILITY NATURAL LANGUAGE INFERENCE QUESTION ANSWERING SEMANTIC TEXTUAL SIMILARITY SENTIMENT ANALYSIS TRANSFER LEARNING

GoEmotions: A Dataset of Fine-Grained Emotions

ACL 2020 google-research/google-research

Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior.

TRANSFER LEARNING