Loss Functions
InfoNCE
Introduced by Oord et al. in Representation Learning with Contrastive Predictive CodingInfoNCE, where NCE stands for Noise-Contrastive Estimation, is a type of contrastive loss function used for self-supervised learning.
Given a set $X = ${$x_{1}, \dots, x_{N}$} of $N$ random samples containing one positive sample from $p\left(x_{t+k}|c_{t}\right)$ and $N − 1$ negative samples from the 'proposal' distribution $p\left(x_{t+k}\right)$, we optimize:
$$ \mathcal{L}_{N} = - \mathbb{E}_{X}\left[\log\frac{f_{k}\left(x_{t+k}, c_{t}\right)}{\sum_{x_{j}\in{X}}f_{k}\left(x_{j}, c_{t}\right)}\right] $$
Optimizing this loss will result in $f_{k}\left(x_{t+k}, c_{t}\right)$ estimating the density ratio, which is:
$$ f_{k}\left(x_{t+k}, c_{t}\right) \propto \frac{p\left(x_{t+k}|c_{t}\right)}{p\left(x_{t+k}\right)} $$
Source: Representation Learning with Contrastive Predictive Coding
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Self-Supervised Learning | 91 | 14.99% |
| Image Classification | 22 | 3.62% |
| Retrieval | 20 | 3.29% |
| Semantic Segmentation | 15 | 2.47% |
| Object Detection | 14 | 2.31% |
| Self-Supervised Image Classification | 14 | 2.31% |
| Classification | 11 | 1.81% |
| General Classification | 11 | 1.81% |
| Speech Recognition | 10 | 1.65% |
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
