Feature Normalized Knowledge Distillation for Image Classification

1 code implementation • ECCV 2020 • Kunran Xu, Lai Rui, Yishi Li, Lin Gu

From this perspective, we systematically analyze the distillation mechanism and demonstrate that the L2-norm of the feature in penultimate layer would be too large under the influence of label noise, and the temperature T in KD could be regarded as a correction factor for L2-norm to suppress the impact of noise.

Classification General Classification +2