Unsupervised Pre-training
103 papers with code • 2 benchmarks • 7 datasets
Pre-training a neural network using unsupervised (self-supervised) auxiliary tasks on unlabeled data.
Benchmarks
These leaderboards are used to track progress in Unsupervised Pre-training
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
Libraries
Use these libraries to find Unsupervised Pre-training models and implementations
CARLANE Benchmark
m2caiSeg
GBUSV
TYC Dataset
Icon645
Most implemented papers
TabTransformer: Tabular Data Modeling Using Contextual Embeddings
lucidrains/tab-transformer-pytorch
•
•
We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning.
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
huggingface/transformers
•
•
TACL 2020
Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing.
A Transformer-based Framework for Multivariate Time Series Representation Learning
In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series.
How far can we go without convolution: Improving fully-connected networks
We propose ways to improve the performance of fully connected networks.
wav2vec: Unsupervised Pre-training for Speech Recognition
pytorch/fairseq
•
•
Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36% when only a few hours of transcribed data is available.
Multilingual Constituency Parsing with Self-Attention and Pre-Training
nikitakit/self-attentive-parser
•
•
ACL 2019
We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions.
Spatiotemporal Contrastive Video Representation Learning
tensorflow/models
•
•
CVPR 2021
Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times.
SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning
YihengZhang-CV/SeCo-Sequence-Contrastive-Learning
•
•
In this paper, we compose a trilogy of exploring the basic and generic supervision in the sequence from spatial, spatiotemporal and sequential perspectives.
Self-training and Pre-training are Complementary for Speech Recognition
pytorch/fairseq
•
•
Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data.
