Momentum Contrast

Introduced by He et al. in Momentum Contrast for Unsupervised Visual Representation Learning

MoCo, or Momentum Contrast, is a self-supervised learning algorithm with a contrastive loss.

Contrastive loss methods can be thought of as building dynamic dictionaries. The "keys" (tokens) in the dictionary are sampled from data (e.g., images or patches) and are represented by an encoder network. Unsupervised learning trains encoders to perform dictionary look-up: an encoded “query” should be similar to its matching key and dissimilar to others. Learning is formulated as minimizing a contrastive loss.

MoCo can be viewed as a way to build large and consistent dictionaries for unsupervised learning with a contrastive loss. In MoCo, we maintain the dictionary as a queue of data samples: the encoded representations of the current mini-batch are enqueued, and the oldest are dequeued. The queue decouples the dictionary size from the mini-batch size, allowing it to be large. Moreover, as the dictionary keys come from the preceding several mini-batches, a slowly progressing key encoder, implemented as a momentum-based moving average of the query encoder, is proposed to maintain consistency.

Source: Momentum Contrast for Unsupervised Visual Representation Learning

Latest Papers

PAPER DATE
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
Senthil PurushwalkamAbhinav Gupta
2020-07-28
Parametric Instance Classification for Unsupervised Visual Feature Learning
Yue CaoZhenda XieBin LiuYutong LinZheng ZhangHan Hu
2020-06-25
What makes instance discrimination good for transfer learning?
Nanxuan ZhaoZhirong WuRynson W. H. LauStephen Lin
2020-06-11
Improved Baselines with Momentum Contrastive Learning
| Xinlei ChenHaoqi FanRoss GirshickKaiming He
2020-03-09
Learning Speaker Embedding with Momentum Contrast
Ke DingXuanji HeGuanglu Wan
2020-01-07
Momentum Contrast for Unsupervised Visual Representation Learning
| Kaiming HeHaoqi FanYuxin WuSaining XieRoss Girshick
2019-11-13

Categories