Big Transfer (BiT): General Visual Representation Learning

24 Dec 2019Alexander KolesnikovLucas BeyerXiaohua ZhaiJoan PuigcerverJessica YungSylvain GellyNeil Houlsby

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task... (read more)

PDF Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
RESULT LEADERBOARD
Image Classification CIFAR-10 BiT-L (ResNet) Percentage correct 99.37 # 1
Percentage error F # 31
Image Classification CIFAR-10 BiT-M (ResNet) Percentage correct 98.61 # 5
Percentage error 1.39 # 2
Image Classification CIFAR-100 BiT-M (ResNet) Percentage correct 92.17 # 2
Percentage error 7.83 # 2
Image Classification CIFAR-100 BiT-L (ResNet) Percentage correct 93.51 # 1
Percentage error 6.49 # 1
Image Classification Flowers-102 BiT-L (ResNet) Accuracy 99.63% # 1
Image Classification Flowers-102 BiT-M (ResNet) Accuracy 99.30% # 2
Image Classification ImageNet BiT-M (ResNet) Top 1 Accuracy 85.39% # 13
Top 5 Accuracy 97.69% # 9
Number of params 928M # 2
Image Classification ImageNet BiT-L (ResNet) Top 1 Accuracy 87.54% # 3
Top 5 Accuracy 98.46% # 2
Image Classification ObjectNet BiT-S (ResNet-152x4) Top 5 Accuracy 57.0 # 3
Top-1 Accuracy 36.0 # 3
Image Classification ObjectNet BiT-M (ResNet-152x4) Top 5 Accuracy 69.0 # 2
Top-1 Accuracy 47.0 # 2
Image Classification ObjectNet BiT-L (ResNet-152x4) Top 5 Accuracy 80.0 # 1
Top-1 Accuracy 58.7 # 1
Image Classification ObjectNet (Bounding Box) BiT-L (ResNet) Top 5 Accuracy 85.1 # 1
Image Classification ObjectNet (Bounding Box) BiT-S (ResNet) Top 5 Accuracy 64.4 # 3
Image Classification ObjectNet (Bounding Box) BiT-M (ResNet) Top 5 Accuracy 76.0 # 2
Fine-Grained Image Classification Oxford 102 Flowers BiT-M (ResNet) Top-1 Error Rate 0.70% # 2
Accuracy 99.30% # 2
Fine-Grained Image Classification Oxford 102 Flowers BiT-L (ResNet) Top-1 Error Rate 0.37% # 1
Accuracy 99.63% # 1
Fine-Grained Image Classification Oxford-IIIT Pets BiT-L (ResNet) Top-1 Error Rate 3.38% # 1
Accuracy 96.62% # 1
Fine-Grained Image Classification Oxford-IIIT Pets BiT-M (ResNet) Top-1 Error Rate 5.53% # 3
Accuracy 94.47% # 4
Image Classification VTAB-1k BiT-M Top-1 Accuracy 70.6 # 5
Image Classification VTAB-1k BiT-L Top-1 Accuracy 76.3 # 1
Image Classification VTAB-1k BiT-S Top-1 Accuracy 66.9 # 9