SwAV

facebookresearch / vissl

Last updated on Feb 27, 2021
SwAV ResNet-50 (100 epochs, 2x224+6x96, 4096 bs)
Parameters 26 Million
FLOPs 4 Billion
File Size 216.72 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_in1k_swav_100ep_batch4k
LR 0.3
Epochs 100
Layers 50
Momentum 0.9
Batch Size 4096
Weight Decay 0.0
Width Multiplier 1
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50 (200 epochs, 2x224+6x96, 256 bs)
Parameters 26 Million
FLOPs 4 Billion
File Size 239.67 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_in1k_swav_200ep_batch256
LR 0.3
Epochs 200
Layers 50
Momentum 0.9
Batch Size 256
Weight Decay 0.0
Width Multiplier 1
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50 (200 epochs, 2x224+6x96, 4096 bs)
Parameters 26 Million
FLOPs 4 Billion
File Size 216.72 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_in1k_swav_200ep_batch4k
LR 0.3
Epochs 200
Layers 50
Momentum 0.9
Batch Size 4096
Weight Decay 0.0
Width Multiplier 1
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50 (400 epochs, 2x224, 4096 bs)
Parameters 26 Million
FLOPs 4 Billion
File Size 216.72 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_in1k_swav_2x224_400ep_batch4k
LR 0.3
Epochs 400
Layers 50
Momentum 0.9
Batch Size 4096
Weight Decay 0.0
Width Multiplier 1
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50 (400 epochs, 2x224+6x96, 256 bs)
Parameters 26 Million
FLOPs 4 Billion
File Size 239.67 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_in1k_swav_400ep_batch256
LR 0.3
Epochs 400
Layers 50
Momentum 0.9
Batch Size 256
Weight Decay 0.0
Width Multiplier 1
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50 (400 epochs, 2x224+6x96, 4096 bs)
Parameters 26 Million
FLOPs 4 Billion
File Size 216.72 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_in1k_swav_400ep_batch4k
LR 0.3
Epochs 400
Layers 50
Momentum 0.9
Batch Size 4096
Weight Decay 0.0
Width Multiplier 1
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50 (800 epochs, 2x224+6x96, 4096 bs)
Parameters 26 Million
FLOPs 4 Billion
File Size 216.72 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_in1k_swav_800ep_batch4k
LR 0.3
Epochs 800
Layers 50
Momentum 0.9
Batch Size 4096
Weight Decay 0.0
Width Multiplier 1
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50-w2 (400 epochs, 2x224+6x96, 4096 bs)
Parameters 94 Million
Layers 50
File Size 984.21 MB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_w2_in1k_swav_400ep
LR 0.3
Epochs 400
Layers 50
Momentum 0.9
Batch Size 4096
Weight Decay 0.0
Width Multiplier 2
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
SwAV ResNet-50-w4 (400 epochs, 2x224+6x96, 2560 bs)
Parameters 375 Million
Layers 50
File Size 3.39 GB
Training Data ImageNet
Training Resources 8x NVIDIA V100 GPUs
Training Time
Training Techniques SwAV, Weight Decay, SGD with Momentum
Architecture 1x1 Convolution, Bottleneck Residual Block, Batch Normalization, Convolution, Global Average Pooling, Residual Block, Residual Connection, ReLU, Max Pooling, Softmax
ID rn50_w4_in1k_swav_400ep
LR 0.3
Epochs 400
Layers 50
Momentum 0.9
Batch Size 2560
Weight Decay 0.0
Width Multiplier 4
SWAV Loss Temperature 0.1
SHOW MORE
SHOW LESS
README.md

Summary

SwaV, or Swapping Assignments Between Views, is a self-supervised learning approach that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, it simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or views) of the same image, instead of comparing features directly as in contrastive learning. Simply put, SwaV uses a swapped prediction mechanism where we predict the cluster assignment of a view from the representation of another view.

How do I train this model?

Get started with VISSL by trying one of the Colab tutorial notebooks.

Citation

@misc{caron2021unsupervised,
      title={Unsupervised Learning of Visual Features by Contrasting Cluster Assignments}, 
      author={Mathilde Caron and Ishan Misra and Julien Mairal and Priya Goyal and Piotr Bojanowski and Armand Joulin},
      year={2021},
      eprint={2006.09882},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Results

Image Classification on ImageNet

Image Classification on ImageNet
MODEL TOP 1 ACCURACY
SwAV ResNet-50-w4 (400 epochs, 2x224+6x96, 2560 bs) 77.03%
SwAV ResNet-50-w2 (400 epochs, 2x224+6x96, 4096 bs) 77.01%
SwAV ResNet-50 (800 epochs, 2x224+6x96, 4096 bs) 74.92%
SwAV ResNet-50 (400 epochs, 2x224+6x96, 4096 bs) 74.81%
SwAV ResNet-50 (400 epochs, 2x224+6x96, 256 bs) 74.3%
SwAV ResNet-50 (200 epochs, 2x224+6x96, 4096 bs) 73.85%
SwAV ResNet-50 (200 epochs, 2x224+6x96, 256 bs) 73.07%
SwAV ResNet-50 (100 epochs, 2x224+6x96, 4096 bs) 71.99%
SwAV ResNet-50 (400 epochs, 2x224, 4096 bs) 69.53%
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy