Regularization

ShakeDrop

Introduced by Yamada et al. in ShakeDrop Regularization for Deep Residual Learning

ShakeDrop regularization extends Shake-Shake regularization and can be applied not only to ResNeXt but also ResNet, WideResNet, and PyramidNet. The proposed ShakeDrop is given as

$$G\left(x\right) = x + \left(b_{l} + \alpha − b_{l}\alpha\right)F\left(x\right), \text{ in train-fwd} $$ $$G\left(x\right) = x + \left(b_{l} + \beta − b_{l}\beta\right)F\left(x\right), \text{ in train-bwd} $$ $$G\left(x\right) = x + E\left[b_{l} + \alpha − b_{l}\alpha\right]F\left(x\right), \text{ in test} $$

where $b_{l}$ is a Bernoulli random variable with probability $P\left(b_{l} = 1\right) = E\left[b_{l} \right] = p_{l}$ given by the linear decay rule in each layer, and $\alpha$ and $\beta$ are independent uniform random variables in each element.

The most effective ranges of $\alpha$ and $\beta$ were experimentally found to be different from those of Shake-Shake, and are $\alpha$ = 0, $\beta \in \left[0, 1\right]$ and $\alpha \in \left[−1, 1\right]$, $\beta \in \left[0, 1\right]$.

Source: ShakeDrop Regularization for Deep Residual Learning

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Image Classification 2 40.00%
General Classification 1 20.00%
Domain Generalization 1 20.00%
Object Detection 1 20.00%

Components


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories