Self-supervised Knowledge Distillation for Few-shot Learning

17 Jun 2020  ·  Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Mubarak Shah ·

Real-world contains an overwhelmingly large number of object classes, learning all of which at once is infeasible. Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. Recent works [7, 41] show that simply learning a good feature embedding can outperform more sophisticated meta-learning and metric learning algorithms for few-shot learning. In this paper, we propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. We follow a two-stage learning process: First, we train a neural network to maximize the entropy of the feature embedding, thus creating an optimal output manifold using a self-supervised auxiliary loss. In the second stage, we minimize the entropy on feature embedding by bringing self-supervised twins together, while constraining the manifold with student-teacher distillation. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods, with further gains achieved by our second stage distillation process. Our codes are available at: https://github.com/brjathu/SKD.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Few-Shot Image Classification CIFAR-FS 5-way (1-shot) SKD Accuracy 76.9 # 18
Few-Shot Image Classification CIFAR-FS 5-way (5-shot) SKD Accuracy 88.9 # 16
Few-Shot Image Classification FC100 5-way (1-shot) SKD Accuracy 46.5 # 13
Few-Shot Image Classification FC100 5-way (5-shot) SKD Accuracy 63.1 # 12
Few-Shot Image Classification Mini-Imagenet 5-way (1-shot) SKD Accuracy 67.04 # 46
Few-Shot Image Classification Mini-Imagenet 5-way (5-shot) SKD Accuracy 83.54 # 29
Few-Shot Image Classification Tiered ImageNet 5-way (1-shot) SKD Accuracy 72.03 # 26
Few-Shot Image Classification Tiered ImageNet 5-way (5-shot) SKD Accuracy 86.66 # 23

Methods


No methods listed for this paper. Add relevant methods here