Methods > General > Working Memory Models

Neural Turing Machine

Introduced by Graves et al. in Neural Turing Machines

A Neural Turing Machine is a working memory neural network model. It couples a neural network architecture with external memory resources. The whole architecture is differentiable end-to-end with gradient descent. The models can infer tasks such as copying, sorting and associative recall.

A Neural Turing Machine (NTM) architecture contains two basic components: a neural network controller and a memory bank. The Figure presents a high-level diagram of the NTM architecture. Like most neural networks, the controller interacts with the external world via input and output vectors. Unlike a standard network, it also interacts with a memory matrix using selective read and write operations. By analogy to the Turing machine we refer to the network outputs that parameterise these operations as “heads.”

Every component of the architecture is differentiable. This is achieved by defining 'blurry' read and write operations that interact to a greater or lesser degree with all the elements in memory (rather than addressing a single element, as in a normal Turing machine or digital computer). The degree of blurriness is determined by an attentional “focus” mechanism that constrains each read and write operation to interact with a small portion of the memory, while ignoring the rest. Because interaction with the memory is highly sparse, the NTM is biased towards storing data without interference. The memory location brought into attentional focus is determined by specialised outputs emitted by the heads. These outputs define a normalised weighting over the rows in the memory matrix (referred to as memory “locations”). Each weighting, one per read or write head, defines the degree to which the head reads or writes at each location. A head can thereby attend sharply to the memory at a single location or weakly to the memory at many locations

Source: Neural Turing Machines

Latest Papers

PAPER DATE
Unsupervised Speaker Adaptation using Attention-based Speaker Memory for End-to-End ASR
Leda SarıNiko MoritzTakaaki HoriJonathan Le Roux
2020-02-14
Memory-Augmented Recurrent Networks for Dialogue Coherence
David DonahueYuanliang MengAnna Rumshisky
2019-10-16
A Neural Turing~Machine for Conditional Transition Graph Modeling
Mehdi Ben LazregMorten GoodwinOle-Christoffer Granmo
2019-07-15
Understanding Memory Modules on Learning Simple Algorithms
Kexin WangYu ZhouShaonan WangJiajun ZhangChengqing Zong
2019-07-01
A review on Neural Turing Machine
Soroor Malekmohammadi FaradonbehFaramarz Safi-Esfahani
2019-04-10
Few-Shot Generalization Across Dialogue Tasks
| Vladimir VlasovAkela Drissner-SchmidAlan Nichol
2018-11-28
Context-Aware Neural Model for Temporal Information Extraction
Yuanliang MengAnna Rumshisky
2018-07-01
A Taxonomy for Neural Memory Networks
Ying MaJose Principe
2018-05-01
Meta-Learning via Feature-Label Memory Network
Dawit MurejaHyunsin ParkChang D. Yoo
2017-10-19
Attention-Set based Metric Learning for Video Face Recognition
Yibo HuXiang WuRan He
2017-04-12
Tracking the World State with Recurrent Entity Networks
| Mikael HenaffJason WestonArthur SzlamAntoine BordesYann LeCun
2016-12-12
Neural Turing Machines: Convergence of Copy Tasks
Janez Aleš
2016-12-07
Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
Caglar GulcehreSarath ChandarKyunghyun ChoYoshua Bengio
2016-06-30
Lie Access Neural Turing Machine
Greg Yang
2016-02-28
Empirical Study on Deep Learning Models for Question Answering
Yang YuWei ZhangChung-Wei HangBing XiangBowen Zhou
2015-10-26
A Deep Memory-based Architecture for Sequence-to-Sequence Learning
Fandong MengZhengdong LuZhaopeng TuHang LiQun Liu
2015-06-22
Reinforcement Learning Neural Turing Machines - Revised
| Wojciech ZarembaIlya Sutskever
2015-05-04
Neural Turing Machines
| Alex GravesGreg WayneIvo Danihelka
2014-10-20

Categories