Untangling tradeoffs between recurrence and self-attention in artificial neural networks

no code implementations • NeurIPS 2020 • Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal Alias Parth Goyal, Kyle Goyette, Yoshua Bengio, Guillaume Lajoie

Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.

Model Optimization

Untangling tradeoffs between recurrence and self-attention in neural networks

no code implementations • 16 Jun 2020 • Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal, Kyle Goyette, Yoshua Bengio, Guillaume Lajoie

Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.

Model Optimization

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

1 code implementation • NeurIPS 2019 • Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, Guillaume Lajoie

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary.

Deep Pepper: Expert Iteration based Chess agent in the Reinforcement Learning Setting

no code implementations • 2 Jun 2018 • Sai Krishna G. V., Kyle Goyette, Ahmad Chamseddine, Breandan Considine

An almost-perfect chess playing agent has been a long standing challenge in the field of Artificial Intelligence.

reinforcement-learning Reinforcement Learning (RL)