Momentum Contrast

Introduced by He et al. in Momentum Contrast for Unsupervised Visual Representation Learning

MoCo, or Momentum Contrast, is a self-supervised learning algorithm with a contrastive loss.

Contrastive loss methods can be thought of as building dynamic dictionaries. The "keys" (tokens) in the dictionary are sampled from data (e.g., images or patches) and are represented by an encoder network. Unsupervised learning trains encoders to perform dictionary look-up: an encoded “query” should be similar to its matching key and dissimilar to others. Learning is formulated as minimizing a contrastive loss.

MoCo can be viewed as a way to build large and consistent dictionaries for unsupervised learning with a contrastive loss. In MoCo, we maintain the dictionary as a queue of data samples: the encoded representations of the current mini-batch are enqueued, and the oldest are dequeued. The queue decouples the dictionary size from the mini-batch size, allowing it to be large. Moreover, as the dictionary keys come from the preceding several mini-batches, a slowly progressing key encoder, implemented as a momentum-based moving average of the query encoder, is proposed to maintain consistency.

Source: Momentum Contrast for Unsupervised Visual Representation Learning

Latest Papers

Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
Senthil PurushwalkamAbhinav Gupta
Parametric Instance Classification for Unsupervised Visual Feature Learning
Yue CaoZhenda XieBin LiuYutong LinZheng ZhangHan Hu
What makes instance discrimination good for transfer learning?
Nanxuan ZhaoZhirong WuRynson W. H. LauStephen Lin
Improved Baselines with Momentum Contrastive Learning
| Xinlei ChenHaoqi FanRoss GirshickKaiming He
Learning Speaker Embedding with Momentum Contrast
Ke DingXuanji HeGuanglu Wan
Momentum Contrast for Unsupervised Visual Representation Learning
| Kaiming HeHaoqi FanYuxin WuSaining XieRoss Girshick