Distributed Methods
ZeRO-Infinity
Introduced by Rajbhandari et al. in ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep LearningZeRO-Infinity is a sharded data parallel system that extends ZeRO with new innovations in heterogeneous memory access called the infinity offload engine. This allows ZeRO-Infinity to support massive model sizes on limited GPU resources by exploiting CPU and NVMe memory simultaneously. In addition, ZeRO-Infinity also introduces a novel GPU memory optimization technique called memory-centric tiling to support extremely large individual layers that would otherwise not fit in GPU memory even one layer at a time.
Source: ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
Papers
| Paper | Code | Results | Date | Stars |
|---|
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
