BERT-JAM: Boosting BERT-Enhanced Neural Machine Translation with Joint Attention

no code implementations • 9 Nov 2020 • Zhebin Zhang, Sai Wu, Dawei Jiang, Gang Chen

In this work, we propose a novel BERT-enhanced NMT model called BERT-JAM which improves upon existing models from two aspects: 1) BERT-JAM uses joint-attention modules to allow the encoder/decoder layers to dynamically allocate attention between different representations, and 2) BERT-JAM allows the encoder/decoder layers to make use of BERT's intermediate representations by composing them using a gated linear unit (GLU).

Machine Translation NMT +1