Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation

1 code implementation • 14 May 2023 • Songming Zhang, Yunlong Liang, Shuaibo Wang, Wenjuan Han, Jian Liu, Jinan Xu, Yufeng Chen

In this work, we first unravel this mystery from an empirical perspective and show that the knowledge comes from the top-1 predictions of teachers, which also helps us build a potential connection between word- and sequence-level KD.

Knowledge Distillation Machine Translation +2