LEMON: Lossless model expansion

no code implementations • 12 Oct 2023 • Yite Wang, Jiahao Su, Hanlin Lu, Cong Xie, Tianyi Liu, Jianbo Yuan, Haibin Lin, Ruoyu Sun, Hongxia Yang

Our empirical results demonstrate that LEMON reduces computational costs by 56. 7% for Vision Transformers and 33. 2% for BERT when compared to training from scratch.

NTK-SAP: Improving neural network pruning by aligning training dynamics

1 code implementation • 6 Apr 2023 • Yite Wang, Dawei Li, Ruoyu Sun

Recent advances in neural tangent kernel (NTK) theory suggest that the training dynamics of large enough neural networks is closely related to the spectrum of the NTK.

Network Pruning

Balanced Training for Sparse GANs

1 code implementation • NeurIPS 2023 • Yite Wang, Jing Wu, Naira Hovakimyan, Ruoyu Sun

We also introduce a new method called balanced dynamic sparse training (ADAPT), which seeks to control the BR during GAN training to achieve a good trade-off between performance and computational cost.

Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning

1 code implementation • CVPR 2022 • Haoxiang Wang, Yite Wang, Ruoyu Sun, Bo Li

We show that the performance of MetaNTK-NAS is comparable or better than the state-of-the-art NAS method designed for few-shot learning while enjoying more than 100x speedup.

Few-Shot Learning Neural Architecture Search