Scaling Laws for Forgetting When Fine-Tuning Large Language Models

no code implementations • 11 Jan 2024 • Damjan Kalajdzievski

We study and quantify the problem of forgetting when fine-tuning pre-trained large language models (LLMs) on a downstream task.

A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA

no code implementations • 28 Nov 2023 • Damjan Kalajdzievski

This scaling factor, which divides adapters by a factor of the rank, results in slowed learning and stunted performance for LoRA with higher-rank adapters.

Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer

no code implementations • 29 Nov 2022 • Damjan Kalajdzievski, Ximeng Mao, Pascal Fortier-Poisson, Guillaume Lajoie, Blake Richards

When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream).

Learning to live with Dale's principle: ANNs with separate excitatory and inhibitory units

no code implementations • ICLR 2021 • Jonathan Cornford, Damjan Kalajdzievski, Marco Leite, Amélie Lamarquette, Dimitri Michael Kullmann, Blake Aaron Richards

The units in artificial neural networks (ANNs) can be thought of as abstractions of biological neurons, and ANNs are increasingly used in neuroscience research.