Temporal Attention Modules for Memory-Augmented Neural Networks
We introduce two temporal attention modules which can be plugged into traditional memory augmented recurrent neural networks to improve their performance in natural language processing tasks. The temporal attention modules provide new inductive biases allowing the models to compute attention distributions over the different time steps of input sequences. The values of these attention distributions can be inspected to identify the sequence's elements that the model considered relevant during the inference. Using the Entity Network (Henaff et al., 2016) as the model backbone, experiments were made on the dataset bAbI tasks, a set of QA tasks. Due to the addition of the temporal attention modules, the performance metric increased 26% when the temporal attention was supervised, and 13,5% when it wasn't. Moreover, the usage of temporal attention modules proved useful at resolving reasoning tasks that the original model was unable to solve.
PDF Abstract
bAbI
