Pipelined Backpropagation

Pipelined Backpropagation is an asynchronous pipeline parallel training algorithm. It was first introduced by Petrowski et al (1993). It avoids fill and drain overhead by updating the weights without draining the pipeline first. This results in weight inconsistency, the use of different weights on the forward and backward passes for a given micro-batch. The weights used to produce a particular gradient may also have been updated when the gradient is applied, resulting in stale (or delayed) gradients. For these reasons PB resembles Asynchronous SGD and is not equivalent to standard SGD. Finegrained pipelining increases the number of pipeline stages and hence increases the weight inconsistency and delay.