no code implementations • 12 Oct 2023 • Yuhan Helena Liu, Aristide Baratin, Jonathan Cornford, Stefan Mihalas, Eric Shea-Brown, Guillaume Lajoie
Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks.
1 code implementation • 6 Oct 2023 • Edward J. Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, Nikolay Malkin
Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions.
no code implementations • 5 Oct 2023 • Tianhong Li, Sangnie Bhardwaj, Yonglong Tian, Han Zhang, Jarred Barber, Dina Katabi, Guillaume Lajoie, Huiwen Chang, Dilip Krishnan
We demonstrate image generation and captioning performance on par with state-of-the-art text-to-image and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data.
1 code implementation • 3 Oct 2023 • Jean-Pierre Falet, Hae Beom Lee, Nikolay Malkin, Chen Sun, Dragos Secrieru, Thomas Jiralerspong, Dinghuai Zhang, Guillaume Lajoie, Yoshua Bengio
We present a new algorithm for amortized inference in sparse probabilistic graphical models (PGMs), which we call $\Delta$-amortized inference ($\Delta$-AI).
no code implementations • 3 Oct 2023 • Andrew Nam, Eric Elmoznino, Nikolay Malkin, Chen Sun, Yoshua Bengio, Guillaume Lajoie
Compositionality is an important feature of discrete symbolic systems, such as language and programs, as it enables them to have infinite capacity despite a finite symbol set.
1 code implementation • 30 May 2023 • Roman Pogodin, Jonathan Cornford, Arna Ghosh, Gauthier Gidel, Guillaume Lajoie, Blake Richards
Overall, our work shows that the current paradigm in theoretical work on synaptic plasticity that assumes Euclidean synaptic geometry may be misguided and that it should be possible to experimentally determine the true geometry of synaptic plasticity in the brain.
1 code implementation • 24 Feb 2023 • Ezekiel Williams, Colin Bredenberg, Guillaume Lajoie
Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with another.
no code implementations • 22 Feb 2023 • Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang, Guillaume Lajoie, Chen Sun, Phillip Isola, Dilip Krishnan
In this paper, we propose a method of learning representations that are instead equivariant to data augmentations.
no code implementations • 13 Feb 2023 • Xu Ji, Eric Elmoznino, George Deane, Axel Constant, Guillaume Dumas, Guillaume Lajoie, Jonathan Simon, Yoshua Bengio
Conscious states (states that there is something it is like to be in) seem both rich or full of detail, and ineffable or hard to fully describe or recall.
no code implementations • 29 Nov 2022 • Damjan Kalajdzievski, Ximeng Mao, Pascal Fortier-Poisson, Guillaume Lajoie, Blake Richards
When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream).
no code implementations • 28 Oct 2022 • MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, Eugene Belilovsky
Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways.
1 code implementation • 25 Oct 2022 • Sarthak Mittal, Guillaume Lajoie, Stefan Bauer, Arash Mehrjou
Consequently, it is reasonable to ask if there is an intermediate time step at which the preserved information is optimal for a given downstream task.
1 code implementation • 19 Sep 2022 • Thomas George, Guillaume Lajoie, Aristide Baratin
Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network can be well approximated by its linearization around initialization.
1 code implementation • 9 Jun 2022 • Giancarlo Kerg, Sarthak Mittal, David Rolnick, Yoshua Bengio, Blake Richards, Guillaume Lajoie
Recent work has explored how forcing relational representations to remain distinct from sensory representations, as it seems to be the case in the brain, can help artificial systems.
1 code implementation • 6 Jun 2022 • Sarthak Mittal, Yoshua Bengio, Guillaume Lajoie
Inspired from human cognition, machine learning systems are gradually revealing advantages of sparser and more modular architectures.
1 code implementation • 2 Jun 2022 • Yuhan Helena Liu, Arna Ghosh, Blake A. Richards, Eric Shea-Brown, Guillaume Lajoie
We first demonstrate that state-of-the-art biologically-plausible learning rules for training RNNs exhibit worse and more variable generalization performance compared to their machine learning counterparts that follow the true gradient more closely.
no code implementations • ICLR 2022 • Tristan Deleu, David Kanaa, Leo Feng, Giancarlo Kerg, Yoshua Bengio, Guillaume Lajoie, Pierre-Luc Bacon
Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field.
no code implementations • 24 Feb 2022 • Léo Gagnon, Guillaume Lajoie
The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores.
no code implementations • 22 Dec 2021 • Jessie Huang, Erica L. Busch, Tom Wallenstein, Michal Gerasimiuk, Andrew Benz, Guillaume Lajoie, Guy Wolf, Nicholas B. Turk-Browne, Smita Krishnaswamy
In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure.
1 code implementation • 6 Dec 2021 • Mohammad Pezeshki, Amartya Mitra, Yoshua Bengio, Guillaume Lajoie
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters.
3 code implementations • ICLR 2022 • Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish, Yoshua Bengio, Guillaume Lajoie
Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed.
no code implementations • 26 Jul 2021 • Alexander Tong, Guillaume Huguet, Dennis Shung, Amine Natik, Manik Kuchroo, Guillaume Lajoie, Guy Wolf, Smita Krishnaswamy
We propose to compare and organize such datasets of graph signals by using an earth mover's distance (EMD) with a geodesic cost over the underlying graph.
1 code implementation • 2 Jul 2021 • Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, Christopher Pal
A central goal for AI and causality is thus the joint discovery of abstract representations and causal structure.
no code implementations • 28 May 2021 • Christian David Márton, Léo Gagnon, Guillaume Lajoie, Kanaka Rajan
For this reason, a central aspect of human learning is the ability to recycle previously acquired knowledge in a way that allows for faster, less resource-intensive acquisition of new skills.
no code implementations • 31 Jan 2021 • Stefan Horoi, Jessie Huang, Bastian Rieck, Guillaume Lajoie, Guy Wolf, Smita Krishnaswamy
This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training.
no code implementations • NeurIPS 2020 • Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal Alias Parth Goyal, Kyle Goyette, Yoshua Bengio, Guillaume Lajoie
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
2 code implementations • NeurIPS 2021 • Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, Guillaume Lajoie
We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks.
Ranked #1 on Out-of-Distribution Generalization on ImageNet-W
1 code implementation • 26 Oct 2020 • Reyhane Askari Hemmat, Amartya Mitra, Guillaume Lajoie, Ioannis Mitliagkas
A central obstacle in the optimization of such games is the rotational dynamics that hinder their convergence.
1 code implementation • NeurIPS Workshop DL-IG 2020 • Aristide Baratin, Thomas George, César Laurent, R. Devon Hjelm, Guillaume Lajoie, Pascal Vincent, Simon Lacoste-Julien
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint.
1 code implementation • ICML 2020 • Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, Yoshua Bengio
To effectively utilize the wealth of potential top-down information available, and to prevent the cacophony of intermixed signals in a bidirectional architecture, mechanisms are needed to restrict information flow.
no code implementations • 25 Jun 2020 • Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, Guillaume Lajoie
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters.
no code implementations • 22 Jun 2020 • Victor Geadah, Giancarlo Kerg, Stefan Horoi, Guy Wolf, Guillaume Lajoie
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks.
no code implementations • 16 Jun 2020 • Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal, Kyle Goyette, Yoshua Bengio, Guillaume Lajoie
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
no code implementations • 9 Jan 2020 • Stefan Horoi, Guillaume Lajoie, Guy Wolf
The efficiency of recurrent neural networks (RNNs) in dealing with sequential data has long been established.
no code implementations • NeurIPS Workshop Neuro_AI 2019 • Pravish Sainath, Pierre Bellec, Guillaume Lajoie
We train these neural networks to solve the working memory task by training them with a sequence of images in supervised and reinforcement learning settings.
no code implementations • NeurIPS Workshop Neuro_AI 2019 • Matthew Farrell, Stefano Recanatesi, Guillaume Lajoie, Eric Shea-Brown
What determines the dimensionality of activity in neural circuits?
no code implementations • 2 Jun 2019 • Stefano Recanatesi, Matthew Farrell, Madhu Advani, Timothy Moore, Guillaume Lajoie, Eric Shea-Brown
Datasets such as images, text, or movies are embedded in high-dimensional spaces.
1 code implementation • NeurIPS 2019 • Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, Guillaume Lajoie
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary.
no code implementations • 17 May 2019 • Aude Forcione-Lambert, Guy Wolf, Guillaume Lajoie
We investigate the learned dynamical landscape of a recurrent neural network solving a simple task requiring the interaction of two memory mechanisms: long- and short-term.