Improving Neural Language Modeling via Adversarial Training

10 Jun 2019 Dilin Wang Chengyue Gong Qiang Liu

Recently, substantial progress has been made in language modeling by using deep neural networks. However, in practice, large scale neural language models have been shown to be prone to overfitting... (read more)

PDF Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Machine Translation IWSLT2015 German-English Transformer Base + adversarial MLE BLEU score 35.18 # 1
Language Modelling Penn Treebank (Word Level) adversarial + AWD-LSTM-MoS + dynamic eval Validation perplexity 46.63 # 3
Test perplexity 46.01 # 5
Params 22M # 9
Language Modelling WikiText-2 adversarial + AWD-LSTM-MoS + dynamic eval Validation perplexity 40.27 # 2
Test perplexity 38.65 # 3
Number of params 35M # 6
Machine Translation WMT2014 English-German Transformer Big + adversarial MLE BLEU score 29.52 # 11

Methods used in the Paper


METHOD TYPE
🤖 No Methods Found Help the community by adding them if they're not listed; e.g. Deep Residual Learning for Image Recognition uses ResNet