Search Results for author:
Found 7 papers, 4 papers with code
Memory Efficient Mixed-Precision Optimizers
Traditional optimization methods rely on the use of single-precision floating point arithmetic, which can be costly in terms of memory size and computing power.
Multiplication-Free Transformer Training via Piecewise Affine Operations
Finally, we show that we can eliminate all multiplications in the entire training process, including operations in the forward pass, backward pass and optimizer update, demonstrating the first successful training of modern neural network architectures in a fully multiplication-free fashion.
Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
This study investigates how weight decay affects the update behavior of individual neurons in deep neural networks through a combination of applied analysis and experimentation.
Ghost Noise for Regularizing Deep Neural Networks
Batch Normalization (BN) is widely used to stabilize the optimization process and improve the test performance of deep neural networks.
Adaptive Braking for Mitigating Gradient Delay
Neural network training is commonly accelerated by using multiple synchronized workers to compute gradient updates in parallel.
Pipelined Backpropagation at Scale: Training Large Models without Batches
New hardware can substantially increase the speed and efficiency of deep neural network training.
Online Normalization for Training Neural Networks
Online Normalization is a new technique for normalizing the hidden activations of a neural network.
