Browse SoTA > Methodology > Unsupervised Pre-training

Unsupervised Pre-training

31 papers with code · Methodology

Pre-training a neural network using unsupervised (self-supervised) auxiliary tasks on unlabeled data.

Benchmarks

No evaluation results yet. Help compare methods by submit evaluation metrics.

Greatest papers with code

wav2vec: Unsupervised Pre-training for Speech Recognition

11 Apr 2019pytorch/fairseq

Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36% when only a few hours of transcribed data is available.

Ranked #5 on Speech Recognition on TIMIT (using extra training data)

SPEECH RECOGNITION UNSUPERVISED PRE-TRAINING

Unsupervised Pre-Training of Image Features on Non-Curated Data

ICCV 2019 facebookresearch/deepcluster

Our goal is to bridge the performance gap between unsupervised methods trained on curated data, which are costly to obtain, and massive raw datasets that are easily available.

CLUSTERING UNSUPERVISED PRE-TRAINING

How far can we go without convolution: Improving fully-connected networks

9 Nov 2015yell/boltzmann-machines

We propose ways to improve the performance of fully connected networks.

UNSUPERVISED PRE-TRAINING

Multilingual Constituency Parsing with Self-Attention and Pre-Training

ACL 2019 nikitakit/self-attentive-parser

We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions.

CONSTITUENCY PARSING UNSUPERVISED PRE-TRAINING

A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition

ICLR 2021 athena-team/athena

In this paper, we conduct a further study on MPC and focus on three important aspects: the effect of pre-training data speaking style, its extension on streaming model, and how to better transfer learned knowledge from pre-training stage to downstream tasks.

SPEECH RECOGNITION TRANSFER LEARNING UNSUPERVISED PRE-TRAINING

Data-dependent Initializations of Convolutional Neural Networks

21 Nov 2015philkr/magic_init

Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable.

IMAGE CLASSIFICATION OBJECT DETECTION UNSUPERVISED PRE-TRAINING

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

20 Dec 2013ducha-aiki/LSUVinit

We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times.

UNSUPERVISED PRE-TRAINING