InfoNCE, where NCE stands for Noise-Contrastive Estimation, is a type of contrastive loss function used for self-supervised learning.

Given a set $X = ${$x_{1}, \dots, x_{N}$} of $N$ random samples containing one positive sample from $p\left(x_{t+k}|c_{t}\right)$ and $N − 1$ negative samples from the 'proposal' distribution $p\left(x_{t+k}\right)$, we optimize:

$$ \mathcal{L}_{N} = - \mathbb{E}_{X}\left[\log\frac{f_{k}\left(x_{t+k}, c_{t}\right)}{\sum_{x_{j}\in{X}}f_{k}\left(x_{j}, c_{t}\right)}\right] $$

Optimizing this loss will result in $f_{k}\left(x_{t+k}, c_{t}\right)$ estimating the density ratio, which is:

$$ f_{k}\left(x_{t+k}, c_{t}\right) \propto \frac{p\left(x_{t+k}|c_{t}\right)}{p\left(x_{t+k}\right)} $$

Source: Representation Learning with Contrastive Predictive Coding

Latest Papers

PAPER DATE
Uncovering the structure of clinical EEG signals with self-supervised learning
Hubert BanvilleOmar ChehabAapo HyvärinenDenis-Alexander EngemannAlexandre Gramfort
2020-07-31
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
Senthil PurushwalkamAbhinav Gupta
2020-07-28
Multi-label Contrastive Predictive Coding
Jiaming SongStefano Ermon
2020-07-20
Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets
María Andrea Cruz BlandónOkko Räsänen
2020-07-08
Parametric Instance Classification for Unsupervised Visual Feature Learning
Yue CaoZhenda XieBin LiuYutong LinZheng ZhangHan Hu
2020-06-25
CoMIR: Contrastive Multimodal Image Representation for Registration
Nicolas PielawskiElisabeth WetzerJohan ÖfverstedtJiahao LuCarolina WählbyJoakim LindbladNataša Sladoje
2020-06-11
What makes instance discrimination good for transfer learning?
Nanxuan ZhaoZhirong WuRynson W. H. LauStephen Lin
2020-06-11
Embedding Task Knowledge into 3D Neural Networks via Self-supervised Learning
Jiuwen ZhuYuexiang LiYifan HuS. Kevin Zhou
2020-06-10
3D Self-Supervised Methods for Medical Imaging
| Aiham TalebWinfried LoetzschNoel DanzJulius SeverinThomas GaertnerBenjamin BergnerChristoph Lippert
2020-06-06
On Mutual Information in Contrastive Learning for Visual Representations
Mike WuChengxu ZhuangMilan MosseDaniel YaminsNoah Goodman
2020-05-27
Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge
| Benjamin van NiekerkLeanne NortjeHerman Kamper
2020-05-19
Prototypical Contrastive Learning of Unsupervised Representations
| Junnan LiPan ZhouCaiming XiongRichard SocherSteven C. H. Hoi
2020-05-11
COBRA: Contrastive Bi-Modal Representation Algorithm
| Vishaal UdandaraoAbhishek MaitiDeepak SrivatsavSuryatej Reddy VyallaYifang YinRajiv Ratn Shah
2020-05-07
Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
| Gaëtan HadjeresLéopold Crestel
2020-04-21
Improved Baselines with Momentum Contrastive Learning
| Xinlei ChenHaoqi FanRoss GirshickKaiming He
2020-03-09
Hallucinative Topological Memory for Zero-Shot Visual Planning
| Kara LiuThanard KurutachChristine TungPieter AbbeelAviv Tamar
2020-02-27
Unsupervised pretraining transfers well across languages
| Morgane RivièreArmand JoulinPierre-Emmanuel MazaréEmmanuel Dupoux
2020-02-07
Learning Speaker Embedding with Momentum Contrast
Ke DingXuanji HeGuanglu Wan
2020-01-07
Unsupervised Learning of Efficient and Robust Speech Representations
Anonymous
2020-01-01
Momentum Contrast for Unsupervised Visual Representation Learning
| Kaiming HeHaoqi FanYuxin WuSaining XieRoss Girshick
2019-11-13
Energy-Inspired Models: Learning with Sampler-Induced Distributions
Dieterich LawsonGeorge TuckerBo DaiRajesh Ranganath
2019-10-31
Weakly Supervised Prostate TMA Classification via Graph Convolutional Networks
Jingwen WangRichard J. ChenMing Y. LuAlexander BarasFaisal Mahmood
2019-10-29
Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding
Ming Y. LuRichard J. ChenJingwen WangDebora DillonFaisal Mahmood
2019-10-23
Contrastive Multiview Coding
| Yonglong TianDilip KrishnanPhillip Isola
2019-06-13
Selfie: Self-supervised Pretraining for Image Embedding
Trieu H. TrinhMinh-Thang LuongQuoc V. Le
2019-06-07
Putting An End to End-to-End: Gradient-Isolated Learning of Representations
| Sindy LöwePeter O'ConnorBastiaan S. Veeling
2019-05-28
Semi-supervised Learning with Contrastive Predicative Coding
Jiaxing WangYin ZhengXiaoshuang ChenJunzhou HuangJian Cheng
2019-05-25
Data-Efficient Image Recognition with Contrastive Predictive Coding
| Olivier J. HénaffAravind SrinivasJeffrey De FauwAli RazaviCarl DoerschS. M. Ali EslamiAaron van den Oord
2019-05-22
Contrastive Predictive Coding Based Feature for Automatic Speaker Verification
| Cheng-I Lai
2019-04-01
Neural Predictive Belief Representations
Zhaohan Daniel GuoMohammad Gheshlaghi AzarBilal PiotBernardo A. PiresRémi Munos
2018-11-15
Representation Learning with Contrastive Predictive Coding
| Aaron van den OordYazhe LiOriol Vinyals
2018-07-10

Components

COMPONENT TYPE
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories