Sharded Data Parallel Methods
 
Edit

General • Data Parallel Methods • 3 methods

Methods
Add a Method

Method Year Papers
ZeRO
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
2019 6
ZeRO-Infinity
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
2021 1
ZeRO-Offload
ZeRO-Offload: Democratizing Billion-Scale Model Training
2021 1
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy