Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating or reducing the need for annotations. We find that self-supervision can benefit robustness in a variety of ways, including robustness to adversarial examples, label corruption, and common input corruptions. Additionally, self-supervision greatly benefits out-of-distribution detection on difficult, near-distribution outliers, so much so that it exceeds the performance of fully supervised methods. These results demonstrate the promise of self-supervision for improving robustness and uncertainty estimation and establish these tasks as new axes of evaluation for future self-supervised learning research.

PDF Abstract NeurIPS 2019 PDF NeurIPS 2019 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Anomaly Detection Anomaly Detection on Anomaly Detection on Unlabeled ImageNet-30 vs Flowers-102 ROT+Trans ROC-AUC 86.3 # 5
Network ResNet-18 # 1
Anomaly Detection Anomaly Detection on Unlabeled ImageNet-30 vs CUB-200 ROT+Trans ROC-AUC 74.5 # 4
Network ResNet-18 # 1
Out-of-Distribution Detection CIFAR-10 WRN 40-2 + Rotation Prediction FPR95 16.0 # 6
AUROC 96.2 # 10
Out-of-Distribution Detection CIFAR-10 vs CIFAR-100 WRN 40-2 + Rotation Prediction AUPR 67.7 # 7
AUROC 90.9 # 11
Anomaly Detection One-class CIFAR-10 SSOOD AUROC 90.1 # 17
Anomaly Detection One-class ImageNet-30 Supervised (OE) AUROC 56.1 # 11
Anomaly Detection One-class ImageNet-30 RotNet AUROC 65.3 # 10
Anomaly Detection One-class ImageNet-30 RotNet + Translation AUROC 77.9 # 9
Anomaly Detection One-class ImageNet-30 RotNet + Self-Attention AUROC 81.6 # 8
Anomaly Detection One-class ImageNet-30 RotNet + Translation + Self-Attention AUROC 84.8 # 7
Anomaly Detection One-class ImageNet-30 RotNet + Translation + Self-Attention + Resize AUROC 85.7 # 6

Methods