Gradient Centralization: A New Optimization Technique for Deep Neural Networks

3 Apr 2020Hongwei YongJianqiang HuangXiansheng HuaLei Zhang

Optimization techniques are of great importance to effectively and efficiently train a deep neural network (DNN). It has been shown that using the first and second order statistics (e.g., mean and variance) to perform Z-score standardization on network activations or weight vectors, such as batch normalization (BN) and weight standardization (WS), can improve the training performance... (read more)

PDF Abstract