Big Self-Supervised Models are Strong Semi-Supervised Learners

17 Jun 2020Ting ChenSimon KornblithKevin SwerskyMohammad NorouziGeoffrey Hinton

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to most previous approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet... (read more)

PDF Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Self-Supervised Image Classification ImageNet SimCLRv2 (ResNet-50 x2) Top 1 Accuracy 75.6% # 9
Top 5 Accuracy 92.7% # 6
Number of Params 94M # 8
Self-Supervised Image Classification ImageNet SimCLRv2 (ResNet-152 x3, SK) Top 1 Accuracy 79.8% # 1
Top 5 Accuracy 94.9% # 1
Number of Params 795M # 1
Self-Supervised Image Classification ImageNet SimCLRv2 (ResNet-50) Top 1 Accuracy 71.7% # 16
Top 5 Accuracy 90.4% # 10
Number of Params 24M # 11
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLRv2 distilled (ResNet-50 x2, SK) Top 5 Accuracy 95.0% # 2
Top 1 Accuracy 80.2% # 2
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLRv2 distilled (ResNet-50) Top 5 Accuracy 93.4% # 3
Top 1 Accuracy 77.5% # 4
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLRv2 (ResNet-50 x2) Top 5 Accuracy 91.9% # 5
Top 1 Accuracy 73.9% # 5
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLRv2 (ResNet-50) Top 5 Accuracy 89.2% # 10
Top 1 Accuracy 68.4% # 8
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLRv2 (ResNet-152 x3, SK) Top 5 Accuracy 95.0% # 2
Top 1 Accuracy 80.1% # 3
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLRv2 self-distilled (ResNet-152 x3, SK) Top 5 Accuracy 95.5% # 1
Top 1 Accuracy 80.9% # 1
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLRv2 (ResNet-50 2×) Top 5 Accuracy 87.4% # 7
Top 1 Accuracy 66.3% # 7
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLRv2 (ResNet-152 x3, SK) Top 5 Accuracy 92.3% # 3
Top 1 Accuracy 74.9% # 3
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLRv2 (ResNet-50) Top 5 Accuracy 82.5% # 11
Top 1 Accuracy 57.9% # 11
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLRv2 self-distilled (ResNet-152 x3, SK) Top 5 Accuracy 93.4% # 1
Top 1 Accuracy 76.6% # 1
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLRv2 distilled (ResNet-50 x2, SK) Top 5 Accuracy 93.0% # 2
Top 1 Accuracy 75.9% # 2
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLRv2 distilled (ResNet-50) Top 5 Accuracy 91.5% # 4
Top 1 Accuracy 73.9% # 4

Methods used in the Paper