pith. sign in

arxiv: 1808.07561 · v2 · pith:R3MSXPALnew · submitted 2018-08-22 · 💻 cs.CL · cs.AI· cs.LG

Training Deeper Neural Machine Translation Models with Transparent Attention

Ankur Bapna , Mia Xu Chen , Orhan Firat , Yuan Cao , Yonghui Wu This is my paper
classification 💻 cs.CL cs.AIcs.LG
keywords modelsdeeperattentionmachinetranslationapplicationsarchitecturesattempt
0
0 comments X
read the original abstract

While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Deep Modular Co-Attention Networks for Visual Question Answering

    cs.CV 2019-06 conditional novelty 7.0

    MCAN stacks modular co-attention layers to reach 70.63% accuracy on VQA-v2 test-dev, outperforming prior state-of-the-art models.

  2. Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts

    cs.CL 2019-06 conditional novelty 6.0

    Gated lexical shortcut connections added to the transformer yield 0.9 BLEU average gains on five WMT directions while lowering the lexical content stored in hidden states.