Mixed Distillation Helps Smaller Language Model Better Reasoning

no code implementations • 17 Dec 2023 • Chenglin Li, Qianglong Chen, Liangyue Li, Caiyu Wang, Yicheng Li, Zulong Chen, Yin Zhang

While large language models (LLMs) have demonstrated exceptional performance in recent natural language processing (NLP) tasks, their deployment poses substantial challenges due to high computational and memory demands in real-world applications.

Knowledge Distillation Language Modelling