Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment

no code implementations • 6 Dec 2023 • Fei Yang, Shuang Peng, Ning Sun, Fangyu Wang, Ke Tan, Fu Wu, Jiezhong Qiu, Aimin Pan

Large language models (LLMs) such as GPT-3, OPT, and LLaMA have demonstrated remarkable accuracy in a wide range of tasks.

Scheduling