Additive Attention

Introduced by Bahdanau et al. in Neural Machine Translation by Jointly Learning to Align and Translate

Additive Attention, also known as Bahdanau Attention, uses a one-hidden layer feed-forward network to calculate the attention alignment score:

$$f_{att}\left(\textbf{h}_{i}, \textbf{s}_{j}\right) = v_{a}^{T}\tanh\left(\textbf{W}_{a}\left[\textbf{h}_{i};\textbf{s}_{j}\right]\right)$$

where $\textbf{v}_{a}$ and $\textbf{W}_{a}$ are learned attention parameters. Here $\textbf{h}$ refers to the hidden states for the encoder, and $\textbf{s}$ is the hidden states for the decoder. The function above is thus a type of alignment score function. We can use a matrix of alignment scores to show the correlation between source and target words, as the Figure to the right shows.

Within a neural network, once we have the alignment scores, we calculate the final scores using a softmax function of these alignment scores (ensuring it sums to 1).

Source: Neural Machine Translation by Jointly Learning to Align and Translate

Latest Papers

PAPER DATE
Multitask Pointer Network for Multi-Representational Parsing
Daniel Fernández-GonzálezCarlos Gómez-Rodríguez
2020-09-21
Controllable neural text-to-speech synthesis using intuitive prosodic features
Tuomo RaitioRamya RasipuramDan Castellani
2020-09-14
Corrective feedback, emphatic speech synthesis, visual-speech exaggeration, pronunciation learning
Yaohua BuWeijun LiTianyi MaShengqi ChenJia JiaKun LiXiaobo Lu
2020-09-12
MU-GAN: Facial Attribute Editing based on Multi-attention Mechanism
| Ke ZhangYukun SuXiwang GuoLiang QiZhenbing Zhao
2020-09-09
PNEL: Pointer Network based End-To-End Entity Linking over Knowledge Graphs
Debayan BanerjeeDebanjan ChaudhuriMohnish DubeyJens Lehmann
2020-08-31
Groove2Groove: One-Shot Music Style Transfer with Supervision from Synthetic Data
| Ondřej CífkaUmut ŞimşekliGaël Richard
2020-08-26
Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
Wei ShenXiaonan HeChuheng ZhangQiang NiWanchu DouYan Wang
2020-08-25
Enhancing Speech Intelligibility in Text-To-Speech Synthesis using Speaking Style Conversion
Dipjyoti PaulMuhammed PV ShifasYannis PantazisYannis Stylianou
2020-08-13
Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
Rui LiuBerrak SismanFeilong BaoGuanglai GaoHaizhou Li
2020-08-11
Bilevel Learning Model Towards Industrial Scheduling
Longkang LiHui-Ling ZhenMingxuan YuanJiawen LuXialiangTongJia ZengJun WangDirk Schnieders
2020-08-10
SpeedySpeech: Efficient Neural Speech Synthesis
Jan VainerOndřej Dušek
2020-08-09
One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech
| Tomáš NekvindaOndřej Dušek
2020-08-03
Logic Constrained Pointer Networks for Interpretable Textual Similarity
| Subhadeep MajiRohan KumarManish BansalKalyani RoyPawan Goyal
2020-07-15
Attention-based Joint Detection of Object and Semantic Part
| Keval MorabiaJatin AroraTara Vijaykumar
2020-07-05
Generating Informative Conversational Response using Recurrent Knowledge-Interaction and Knowledge-Copy
Xiexiong LinWeiyu JianJianshan HeTaifeng WangWei Chu
2020-07-01
Don't Eclipse Your Arts Due to Small Discrepancies: Boundary Repositioning with a Pointer Network for Aspect Extraction
Zhenkai WeiYu HongBowei ZouMeng ChengJianmin YAO
2020-07-01
Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation
Tao HeLianli GaoJingkuan SongJianfei CaiYuan-Fang Li
2020-06-13
Investigation of learning abilities on linguistic features in sequence-to-sequence text-to-speech synthesis
Yusuke YasudaXin WangJunichi Yamagishi
2020-05-20
Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model
Da-Rong LiuChunxi LiuFrank ZhangGabriel SynnaeveYatharth SarafGeoffrey Zweig
2020-05-15
End-To-End Speech Synthesis Applied to Brazilian Portuguese
| Edresson CasanovaArnaldo Candido JuniorChristopher ShulbyFrederico Santos de OliveiraJoão Paulo TeixeiraMoacir Antonelli PontiSandra Maria Aluisio
2020-05-11
Context-Sensitive Generation Network for Handing Unknown Slot Values in Dialogue State Tracking
Puhai YangHeyan HuangXian-Ling Mao
2020-05-08
Joint User Pairing and Association for Multicell NOMA: A Pointer Network-based Approach
Manyou MaVincent W. S. Wong
2020-04-15
Code Completion using Neural Attention and Byte Pair Encoding
Youri ArkesteijnNikhil SaldanhaBastijn Kostense
2020-04-14
MODRL/D-AM: Multiobjective Deep Reinforcement Learning Algorithm Using Decomposition and Attention Model for Multiobjective Optimization
Hong WuJiahai WangZizhen Zhang
2020-02-13
Joint Contextual Modeling for ASR Correction and Language Understanding
Yue WengSai Sumanth MiryalaChandra KhatriRunze WangHuaixiu ZhengPiero MolinoMahdi NamazifarAlexandros PapangelisHugh WilliamsFranziska BellGokhan Tur
2020-01-28
High-Level Plan for Behavioral Robot Navigation with Natural Language Directions and R-NET
Amar ShresthaKrittaphat PugdeethosapolHaowen FangQinru Qiu
2020-01-08
Filling Conversation Ellipsis for Better Social Dialog Understanding
Xiyuan ZhangChengxi LiDian YuSamuel DavidsonZhou Yu
2019-11-25
A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis
Junjie PanXiang YinZhiling ZhangShichao LiuYang ZhangZejun MaYuxuan Wang
2019-11-11
Neural News Recommendation with Multi-Head Self-Attention
Chuhan WuFangzhao WuSuyu GeTao QiYongfeng HuangXing Xie
2019-11-01
The Concordia NLG Surface Realizer at SRST 2019
Farhood FarahnakLaya RafieeLeila KosseimThomas Fevens
2019-11-01
Relation Extraction among Multiple Entities Using a Dual Pointer Network with a Multi-Head Attention Mechanism
Seong Sik ParkHarksoo Kim
2019-11-01
Deep Copycat Networks for Text-to-Text Generation
| Julia IvePranava MadhyasthaLucia Specia
2019-11-01
Semi-Supervised Semantic Role Labeling with Cross-View Training
Rui CaiMirella Lapata
2019-11-01
Concept Pointer Network for Abstractive Summarization
| Wang WenboGao YangHuang HeyanZhou Yuxiang
2019-10-18
Abstractive Dialog Summarization with Semantic Scaffolds
Lin YuanZhou Yu
2019-10-02
Speech Recognition with Augmented Synthesized Speech
Andrew RosenbergYu ZhangBhuvana RamabhadranYe JiaPedro MorenoYonghui WuZelin Wu
2019-09-25
Hierarchical Pointer Net Parsing
| Linlin LiuXiang LinShafiq JotySimeng HanLidong Bing
2019-08-30
Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts
| Sagi EppelAlan Aspuru-Guzik
2019-08-24
Sentence Specified Dynamic Video Thumbnail Generation
| Yitian YuanLin MaWenwu Zhu
2019-08-12
Semi-supervised Thai Sentence Segmentation Using Local and Distant Word Representations
Chanatip SaetiaEkapol ChuangsuwanichTawunrat ChalothornPeerapon Vateekul
2019-08-04
Combinatorial Keyword Recommendations for Sponsored Search with Deep Reinforcement Learning
Zhipeng LiJianwei WuLin SunTao Rong
2019-07-18
Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning
| Yu ZhangRon J. WeissHeiga ZenYonghui WuZhifeng ChenRJ Skerry-RyanYe JiaAndrew RosenbergBhuvana Ramabhadran
2019-07-09
Supervised Symbolic Music Style Translation Using Synthetic Data
| Ondřej CífkaUmut ŞimşekliGaël Richard
2019-07-04
Deep Reinforcement Learning for Multi-objective Optimization
Kaiwen LiTao ZhangRui Wang
2019-06-06
Left-to-Right Dependency Parsing with Pointer Networks
Daniel Fern{\'a}ndez-Gonz{\'a}lezCarlos G{\'o}mez-Rodr{\'\i}guez
2019-06-01
Learning to Memorize in Neural Task-Oriented Dialogue Systems
Chien-Sheng Wu
2019-05-19
One-Shot Learning for Text-to-SQL Generation
Dongjun LeeJaesik YoonJongyun SongSanggil LeeSungroh Yoon
2019-04-26
A Hierarchical Decoding Model For Spoken Language Understanding From Unaligned Data
Zijian ZhaoSu ZhuKai Yu
2019-04-09
A New GAN-based End-to-End TTS Training Algorithm
Haohan GuoFrank K. SoongLei HeLei Xie
2019-04-09
Taco-VC: A Single Speaker Tacotron based Voice Conversion with Limited Data
Roee Levy LeshemRaja Giryes
2019-04-06
Neural Networks for Modeling Source Code Edits
Rui ZhaoDavid BieberKevin SwerskyDaniel Tarlow
2019-04-04
Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesis
Yanyao BianChangbin ChenYongguo KangZhenglin Pan
2019-04-04
Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet
Mingyang ZhangXin WangFuming FangHaizhou LiJunichi Yamagishi
2019-03-29
Aggregated Deep Local Features for Remote Sensing Image Retrieval
Raffaele ImbriacoClint SebastianEgor BondarevPeter H. N. de With
2019-03-22
Left-to-Right Dependency Parsing with Pointer Networks
| Daniel Fernández-GonzálezCarlos Gómez-Rodríguez
2019-03-20
Fast Prototyping a Dialogue Comprehension System for Nurse-Patient Conversations on Symptom Monitoring
Zhengyuan LiuHazel LimNur Farah Ain Binte SuhaimiShao Chuen TongSharon OngAngela NgSheldon LeeMichael R. MacdonaldSavitha RamasamyPavitra KrishnaswamyWai Leng ChowNancy F. Chen
2019-03-08
Persona-Aware Tips Generation
Piji LiZihao WangLidong BingWai Lam
2019-03-06
Abstractive Summarization of Spoken and Written Conversation
| Prakhar GaneshSaket Dingliwal
2019-02-05
AlphaStar: An Evolutionary Computation Perspective
| Kai ArulkumaranAntoine CullyJulian Togelius
2019-02-05
Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language
| Yusuke YasudaXin WangShinji TakakiJunichi Yamagishi
2018-10-29
Incorporating Background Knowledge into Video Description Generation
Spencer WhiteheadHeng JiMohit BansalShih-Fu ChangClare Voss
2018-10-01
Deep Attentive Sentence Ordering Network
Baiyun CuiYingming LiMing ChenZhongfei Zhang
2018-10-01
Semi-Supervised Sequence Modeling with Cross-View Training
| Kevin ClarkMinh-Thang LuongChristopher D. ManningQuoc V. Le
2018-09-22
Towards one-shot learning for rare-word translation with external experts
Ngoc-Quan PhamJan NiehuesAlex Waibel
2018-09-10
Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis
Yu-An ChungYuxuan WangWei-Ning HsuYu ZhangRJ Skerry-Ryan
2018-08-30
Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis
Daisy StantonYuxuan WangRJ Skerry-Ryan
2018-08-04
Extractive Summarization with SWAP-NET: Sentences and Words from Alternating Pointer Networks
| Aishwarya JadhavVaibhav Rajan
2018-07-01
Voice Imitating Text-to-Speech Neural Networks
Younggun LeeTaesu KimSoo-Young Lee
2018-06-04
An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking
Puyang XuQi Hu
2018-05-03
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
| RJ Skerry-RyanEric BattenbergYing XiaoYuxuan WangDaisy StantonJoel ShorRon J. WeissRob ClarkRif A. Saurous
2018-03-24
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
| Yuxuan WangDaisy StantonYu ZhangRJ Skerry-RyanEric BattenbergJoel ShorYing XiaoFei RenYe JiaRif A. Saurous
2018-03-23
Attention, Learn to Solve Routing Problems!
| Wouter KoolHerke van HoofMax Welling
2018-03-22
Emotional End-to-End Neural Speech Synthesizer
| Younggun LeeAzam RabieeSoo-Young Lee
2017-11-15
Uncovering Latent Style Factors for Expressive Speech Synthesis
Yuxuan WangRJ Skerry-RyanYing XiaoDaisy StantonJoel ShorEric BattenbergRob ClarkRif A. Saurous
2017-11-01
Solving a New 3D Bin Packing Problem with Deep Reinforcement Learning Method
Haoyuan HuXiaodong ZhangXiaowei YanLongfei WangYinghui Xu
2017-08-20
Deep Voice 2: Multi-Speaker Neural Text-to-Speech
Sercan ArikGregory DiamosAndrew GibianskyJohn MillerKainan PengWei PingJonathan RaimanYanqi Zhou
2017-05-24
Tacotron: Towards End-to-End Speech Synthesis
| Yuxuan WangRJ Skerry-RyanDaisy StantonYonghui WuRon J. WeissNavdeep JaitlyZongheng YangYing XiaoZhifeng ChenSamy BengioQuoc LeYannis AgiomyrgiannakisRob ClarkRif A. Saurous
2017-03-29
Here's My Point: Joint Pointer Architecture for Argument Mining
Peter PotashAlexey RomanovAnna Rumshisky
2016-12-28
Learning Python Code Suggestion with a Sparse Pointer Network
| Avishkar BhoopchandTim RocktäschelEarl BarrSebastian Riedel
2016-11-24
Pointer Sentinel Mixture Models
| Stephen MerityCaiming XiongJames BradburyRichard Socher
2016-09-26
Attention-Based Models for Speech Recognition
| Jan ChorowskiDzmitry BahdanauDmitriy SerdyukKyunghyun ChoYoshua Bengio
2015-06-24
Pointer Networks
| Oriol VinyalsMeire FortunatoNavdeep Jaitly
2015-06-09
Neural Machine Translation by Jointly Learning to Align and Translate
| Dzmitry BahdanauKyunghyun ChoYoshua Bengio
2014-09-01

Categories