Search Results for author:
Found 6 papers, 3 papers with code
LoTR: Low Tensor Rank Weight Adaptation
In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture.
Federated Privacy-preserving Collaborative Filtering for On-Device Next App Prediction
Since the data about user experience are distributed among devices, the federated learning setup is used to train the proposed sequential matrix factorization model.
NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer
Classical machine learning models such as deep neural networks are usually trained by using Stochastic Gradient Descent-based (SGD) algorithms.
Survey on Large Scale Neural Network Training
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training.
Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction
Every modern neural network model has quite a few pointwise nonlinearities in its architecture, and such operation induces additional memory costs which -- as we show -- can be significantly reduced by quantization of the gradients.
Memory-Efficient Backpropagation through Large Linear Layers
Also, we investigate the variance of the gradient estimate induced by the randomized matrix multiplication.
