GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

NeurIPS 2019 Yanping HuangYoulong ChengAnkur BapnaOrhan FiratMia Xu ChenDehao ChenHyoukJoong LeeJiquan NgiamQuoc V. LeYonghui WuZhifeng Chen

Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks. In many cases, increasing model capacity beyond the memory limit of a single accelerator has required developing special algorithms or infrastructure... (read more)

PDF Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
RESULT LEADERBOARD
Fine-Grained Image Classification Birdsnap GPIPE Accuracy 83.6% # 2
Image Classification CIFAR-10 GPIPE + transfer learning Percentage correct 99 # 1
Percentage error 1 # 1
Image Classification CIFAR-100 GPIPE Percentage correct 91.3 # 5
Image Classification ImageNet GPIPE Top 1 Accuracy 84.4% # 21
Top 5 Accuracy 97% # 16
Number of params 557M # 3
Fine-Grained Image Classification Stanford Cars GPipe Accuracy 94.6% # 10

Methods used in the Paper