pith. machine review for the scientific record. sign in

arxiv: 1609.08144 · v2 · submitted 2016-09-26 · 💻 cs.CL · cs.AI· cs.LG

Recognition: 2 theorem links

· Lean Theorem

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Alex Rudnick, Apurva Shah, Cliff Young, George Kurian, Greg Corrado, Hideto Kazawa, Jason Riesa, Jason Smith, Jeff Klingner, Jeffrey Dean, Keith Stevens, Klaus Macherey, {\L}ukasz Kaiser, Macduff Hughes, Maxim Krikun, Melvin Johnson, Mike Schuster, Mohammad Norouzi, Nishant Patil, Oriol Vinyals, Qin Gao, Quoc V. Le, Stephan Gouws, Taku Kudo, Wei Wang, Wolfgang Macherey, Xiaobing Liu, Yonghui Wu, Yoshikiyo Kato, Yuan Cao, Zhifeng Chen

Pith reviewed 2026-05-12 15:16 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG
keywords neural machine translationGNMTdeep LSTMwordpiecesattention mechanismcoverage penaltymachine translationrare words
0
0 comments X

The pith

GNMT, a deep LSTM neural machine translation system with wordpieces and coverage penalties, reduces translation errors by an average of 60% compared to phrase-based systems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces GNMT as an end-to-end neural approach to machine translation that addresses the computational expense and rare word problems of earlier NMT systems. It details a model using deep LSTMs with eight layers each in encoder and decoder, residual connections, a parallelized attention mechanism, subword wordpieces for vocabulary, low-precision computation for speed, and beam search augmented with length normalization and coverage penalty. Human evaluations on simple sentences demonstrate a 60% average reduction in translation errors relative to the prior phrase-based production system, while matching state-of-the-art on standard benchmarks. This matters because practical deployment requires both high accuracy and fast operation, which previous neural systems struggled to deliver. If the claim holds, it indicates that neural methods can substantially narrow the quality gap to human translators for at least straightforward text.

Core claim

Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. The attention mechanism connects the bottom layer of the decoder to the top layer of the encoder to improve parallelism. Wordpieces are used for both input and output to handle rare words. Low-precision arithmetic accelerates inference. Beam search incorporates length-normalization and a coverage penalty to encourage complete translations. On WMT'14 benchmarks GNMT achieves competitive results, and human side-by-side evaluation shows it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

What carries the argument

GNMT: deep 8-layer LSTM encoder-decoder with attention from decoder bottom to encoder top, wordpiece tokenization, low-precision inference, and coverage-penalized beam search.

If this is right

  • Improves handling of rare words by breaking them into common sub-word units.
  • Accelerates training through better parallelism in the attention mechanism.
  • Speeds up translation inference using low-precision arithmetic.
  • Encourages more complete output sentences via the coverage penalty in beam search.
  • Achieves competitive performance on English-to-French and English-to-German WMT benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar architectural choices could be applied to other sequence generation tasks beyond translation.
  • The gains might increase further with larger training corpora, but the paper does not test this directly.
  • Results on isolated simple sentences may not fully predict performance on long, context-dependent or technical texts.
  • Adoption in production could shift the default from phrase-based to neural systems if the error reduction holds across domains.

Load-bearing premise

The performance improvements are due to the specific model architecture and training choices rather than differences in the amount or quality of training data or available compute resources.

What would settle it

Re-training the phrase-based system on the same data volume and hardware as GNMT and re-evaluating both on a diverse set of complex sentences would show whether the 60% error reduction is architecture-specific.

read the original abstract

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The paper presents GNMT, an 8-layer residual LSTM encoder-decoder NMT system with bottom-decoder-to-top-encoder attention, wordpiece subword segmentation, low-precision inference, and coverage-penalized length-normalized beam search. It reports competitive BLEU scores on the WMT'14 English-to-French and English-to-German benchmarks and claims a 60% average reduction in translation errors versus Google's production phrase-based MT system, measured by human side-by-side ratings on a set of isolated simple sentences.

Significance. If the human-evaluation result holds under controlled conditions, the work is significant for showing that deep NMT can be deployed at production scale and can measurably outperform a mature phrase-based system on the metric that matters most to users. Credit is due for the practical engineering contributions (wordpieces for rare-word handling, residual connections and attention placement for training speed, low-precision arithmetic for inference latency) and for providing a direct, falsifiable human comparison rather than relying solely on automatic metrics.

major comments (2)
  1. [§5] §5 (human side-by-side evaluation): the central claim of a 60% average error reduction is supported only by ratings on 'isolated simple sentences'; the manuscript provides no count of sentences, no description of sentence selection or domain, no inter-rater agreement statistics, and no p-value or confidence interval, making it impossible to judge whether the reported gap is robust or generalizes beyond the evaluated regime.
  2. [§5] §5 (comparison to production PBMT baseline): the 60% error-reduction figure is presented without any statement that training-data volume, parallel-corpus composition, or total optimization effort were held constant between GNMT and the phrase-based production system. Because the production baseline may differ in data scale or tuning regime, the result does not isolate the contribution of the architectural choices (residual LSTMs, attention placement, wordpieces, coverage penalty) that the paper highlights.
minor comments (3)
  1. [Abstract] The abstract states that GNMT 'achieves competitive results' on WMT'14 but omits the actual BLEU numbers; adding the precise scores (and the corresponding state-of-the-art references) would make the summary self-contained.
  2. [Model Architecture] The description of the attention mechanism (bottom decoder layer attending to top encoder layer) would be clearer if accompanied by a small equation or diagram in the model-architecture section.
  3. [§5] Table captions for the WMT results should explicitly note the training data size and whether any external monolingual data were used, to allow direct comparison with contemporaneous systems.

Simulated Author's Rebuttal

2 responses · 1 unresolved

Thank you for the detailed review of our paper on Google's Neural Machine Translation system. We appreciate the positive assessment of the work's significance and will address the concerns raised regarding the human evaluation section to improve the manuscript.

read point-by-point responses

  1. Referee: §5 (human side-by-side evaluation): the central claim of a 60% average error reduction is supported only by ratings on 'isolated simple sentences'; the manuscript provides no count of sentences, no description of sentence selection or domain, no inter-rater agreement statistics, and no p-value or confidence interval, making it impossible to judge whether the reported gap is robust or generalizes beyond the evaluated regime.

    Authors: We acknowledge that §5 provides limited details on the human evaluation protocol. In the revised manuscript, we will add a description of the sentence selection process (isolated simple sentences sampled from production traffic and test sets) and the approximate number of sentences rated. We will also clarify that the evaluation involved multiple professional raters performing side-by-side comparisons. However, inter-rater agreement statistics and formal statistical significance tests were not part of the original evaluation design; we will note this as a limitation of the reported result rather than claiming robustness beyond what the data supports. revision: partial

  2. Referee: §5 (comparison to production PBMT baseline): the 60% error-reduction figure is presented without any statement that training-data volume, parallel-corpus composition, or total optimization effort were held constant between GNMT and the phrase-based production system. Because the production baseline may differ in data scale or tuning regime, the result does not isolate the contribution of the architectural choices (residual LSTMs, attention placement, wordpieces, coverage penalty) that the paper highlights.

    Authors: The comparison is intentionally to the deployed production phrase-based system, which represents the state-of-the-art performance achievable with that paradigm at Google, including all available data and tuning efforts. The goal is to show the practical improvement offered by GNMT over the existing production baseline. We agree that this does not constitute a controlled experiment isolating individual model components. In the revision, we will explicitly state in §5 that the PBMT baseline is the fully optimized production system and that the reported improvement reflects the end-to-end difference rather than the effect of any single architectural decision. revision: yes

standing simulated objections not resolved
  • The lack of inter-rater agreement and p-value statistics for the human evaluation, as these were not computed in the original study and raw data may not be available for re-analysis.

Circularity Check

0 steps flagged

No circularity: empirical claims rest on independent benchmarks and human evaluation

full rationale

The paper describes GNMT architecture (8-layer residual LSTMs, bottom-decoder attention, wordpieces, coverage penalty) and reports results on WMT'14 benchmarks plus a 60% error reduction from human side-by-side evaluation on held-out simple sentences versus the production PBMT system. These metrics derive from external test sets and separate raters rather than any equation or fit that defines success in terms of the model's own parameters. No self-definitional loops, fitted-input predictions, or load-bearing self-citations appear in the derivation of the central performance claims; the evaluation is presented as an independent measurement of the described system.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 1 invented entities

The central claim depends on empirical training of a large neural network whose capacity and efficiency choices (layer depth, precision, vocabulary construction) are selected by hand rather than derived from first principles.

free parameters (2)
  • encoder and decoder depth = 8
    Set to 8 layers each to increase capacity while preserving training parallelism.
  • wordpiece vocabulary size
    Chosen as a limited set of common sub-word units to balance coverage and efficiency.
axioms (2)
  • domain assumption LSTM layers with residual connections can model long-range dependencies in translation sequences.
    Foundation for the 8-layer encoder-decoder stack.
  • domain assumption Low-precision arithmetic during inference maintains acceptable translation quality.
    Justification for using reduced precision to speed up decoding.
invented entities (1)
  • wordpieces no independent evidence
    purpose: Sub-word units that allow the model to translate rare words without an open vocabulary.
    New tokenization scheme introduced to solve the rare-word problem.

pith-pipeline@v0.9.0 · 5744 in / 1573 out tokens · 69065 ms · 2026-05-12T15:16:34.439988+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 33 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    cs.LG 2017-01 accept novelty 8.0

    A noisy top-k gated mixture-of-experts layer between LSTMs scales neural networks to 137B parameters with sub-linear compute, beating SOTA on language modeling and machine translation.

  2. From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models

    cs.LG 2026-05 unverdicted novelty 7.0

    Chirality emerges in SMILES translation models through an abrupt encoder-centered reorganization of representations after a long plateau, identified via checkpoint analysis and ablation.

  3. Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models

    cs.CL 2026-05 conditional novelty 7.0

    Scratchpad Patching decouples compute from patch size in byte-level language models by inserting entropy-triggered scratchpads to update patch context dynamically.

  4. ReTokSync: Self-Synchronizing Tokenization Disambiguation for Generative Linguistic Steganography

    cs.CR 2026-04 unverdicted novelty 7.0

    ReTokSync resolves tokenization ambiguity in generative linguistic steganography via targeted self-synchronizing resets, achieving over 99.7% extraction accuracy and 100% recovery with an auxiliary channel while match...

  5. ReflectMT: Internalizing Reflection for Efficient and High-Quality Machine Translation

    cs.CL 2026-04 unverdicted novelty 7.0

    ReflectMT internalizes reflection via two-stage RL to enable direct high-quality machine translation that outperforms explicit reasoning models like DeepSeek-R1 on WMT24 while using 94% fewer tokens.

  6. From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution

    cs.CL 2026-04 unverdicted novelty 7.0

    SA-BPE regularizes standard BPE training for code by incorporating source diversity to skip problematic merges, substantially cutting unused tokens without altering inference.

  7. Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

    cs.CV 2023-10 unverdicted novelty 7.0

    A new shared video-image tokenizer enables large language models to surpass diffusion models on standard visual generation benchmarks.

  8. Efficient Memory Management for Large Language Model Serving with PagedAttention

    cs.LG 2023-09 conditional novelty 7.0

    PagedAttention achieves near-zero waste in LLM key-value cache memory and enables 2-4x higher serving throughput than prior systems.

  9. iBOT: Image BERT Pre-Training with Online Tokenizer

    cs.CV 2021-11 unverdicted novelty 7.0

    iBOT achieves 82.3% linear probing accuracy and 87.8% fine-tuning accuracy on ImageNet-1K using masked image modeling with a jointly trained online tokenizer.

  10. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    cs.LG 2019-10 unverdicted novelty 7.0

    T5 casts all NLP tasks as text-to-text generation, systematically explores pre-training choices, and reaches strong performance on summarization, QA, classification and other tasks via large-scale training on the Colo...

  11. Fine-Tuning Language Models from Human Preferences

    cs.CL 2019-09 unverdicted novelty 7.0

    Language models fine-tuned via RL on 5k-60k human preference comparisons produce stylistically better text continuations and human-preferred summaries that sometimes copy input sentences.

  12. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

    cs.CL 2018-08 accept novelty 7.0

    SentencePiece trains subword models directly from raw text to enable language-independent neural text processing.

  13. Mixed Precision Training

    cs.AI 2017-10 accept novelty 7.0

    Mixed precision training uses FP16 for most computations, FP32 master weights for accumulation, and loss scaling to enable accurate training of large DNNs with halved memory usage.

  14. Decaf: Improving Neural Decompilation with Automatic Feedback and Search

    cs.SE 2026-05 unverdicted novelty 6.0

    Decaf uses compiler feedback and search to improve neural decompilation, boosting semantic success rate from 26.0% to 83.9% on ExeBench Real -O2 split.

  15. SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

    cs.CL 2026-05 unverdicted novelty 6.0

    SimCT recovers discarded teacher signal in cross-tokenizer on-policy distillation by enlarging supervision to jointly realizable multi-token continuations, yielding consistent gains on math reasoning and code generati...

  16. Context-Aware Wireless Token Communication via Joint Token Masking and Detection

    eess.SP 2026-05 unverdicted novelty 6.0

    A joint token masking and detection scheme with masked language models improves token reconstruction over noisy wireless channels by up to 1.77x on Europarl and 1.63x on WikiText-103 compared to conventional methods.

  17. Neural Grammatical Error Correction for Romanian

    cs.CL 2026-04 unverdicted novelty 6.0

    A new Romanian GEC corpus of 10k pairs plus pretraining a Transformer on artificial errors generated via POS tagger yields F0.5 of 53.76, beating the 44.38 baseline from training only on the corpus.

  18. PARM: Pipeline-Adapted Reward Model

    cs.AI 2026-04 unverdicted novelty 6.0

    PARM adapts reward models to multi-stage LLM pipelines via pipeline data and direct preference optimization, improving execution rate and solving accuracy on optimization benchmarks and showing transfer to GSM8K.

  19. A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection

    cs.SE 2026-04 unverdicted novelty 6.0

    QTyBERT matches or exceeds BERT-based log anomaly detection effectiveness while reducing embedding generation time to near static word embedding levels.

  20. BloombergGPT: A Large Language Model for Finance

    cs.LG 2023-03 conditional novelty 6.0

    BloombergGPT is a 50B parameter LLM trained on a 708B token mixed financial and general dataset that outperforms prior models on financial benchmarks while preserving general LLM performance.

  21. No Language Left Behind: Scaling Human-Centered Machine Translation

    cs.CL 2022-07 unverdicted novelty 6.0

    A sparsely gated mixture-of-experts model trained on newly mined low-resource data achieves 44% relative BLEU improvement across 200 languages while adding human safety evaluation.

  22. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

    cs.CV 2022-06 unverdicted novelty 6.0

    Scaling an autoregressive Transformer to 20B parameters for text-to-image generation using image token sequences achieves new SOTA zero-shot FID of 7.23 and fine-tuned FID of 3.22 on MS-COCO.

  23. ST-MoE: Designing Stable and Transferable Sparse Expert Models

    cs.CL 2022-02 unverdicted novelty 6.0

    ST-MoE introduces stability techniques for sparse expert models, allowing a 269B-parameter model to achieve state-of-the-art transfer learning results across reasoning, summarization, and QA tasks at the compute cost ...

  24. Unsupervised Dense Information Retrieval with Contrastive Learning

    cs.IR 2021-12 unverdicted novelty 6.0

    Contrastive learning trains unsupervised dense retrievers that beat BM25 on most BEIR datasets and support cross-lingual retrieval across scripts.

  25. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    cs.CL 2020-06 unverdicted novelty 6.0

    GShard supplies automatic sharding and conditional computation support that enabled training a 600-billion-parameter multilingual translation model on thousands of TPUs with superior quality.

  26. CodeBERT: A Pre-Trained Model for Programming and Natural Languages

    cs.CL 2020-02 unverdicted novelty 6.0

    CodeBERT pre-trains a bimodal model on code and text pairs plus unimodal data to achieve state-of-the-art results on natural language code search and code documentation generation.

  27. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

    cs.CV 2017-06 accept novelty 6.0

    Linear learning-rate scaling plus warmup lets minibatch size 8192 train ResNet-50 on ImageNet in one hour at full small-batch accuracy.

  28. Stochasticity in Tokenisation Improves Robustness

    cs.CL 2026-04 unverdicted novelty 5.0

    Stochastic tokenisation during pre-training and fine-tuning improves LLM robustness to perturbations while preserving accuracy.

  29. Attention Is All You Need

    cs.CL 2017-06 unverdicted novelty 5.0

    Pith review generated a malformed one-line summary.

  30. Learning to count small and clustered objects with application to bacterial colonies

    cs.CV 2026-04 unverdicted novelty 4.0

    ACFamNet Pro reaches 9.64% mean normalized absolute error on bacterial colony images under 5-fold cross-validation, beating FamNet by 12.71%.

  31. Video-guided Machine Translation with Global Video Context

    cs.CV 2026-04 unverdicted novelty 4.0

    A globally video-guided multimodal translation framework retrieves semantically related video segments with a vector database and applies attention mechanisms to improve subtitle translation accuracy in long videos.

  32. A Survey of Large Language Models

    cs.CL 2023-03 accept novelty 3.0

    This survey reviews the background, key techniques, and evaluation methods for large language models, emphasizing emergent abilities that appear at large scales.

  33. AI and the Research-Education Environment of Physics

    physics.ed-ph 2026-05 unverdicted novelty 1.0

    A summary of expert opinions on AI's impact on the research-education environment in physics from a KITP discussion session.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · cited by 33 Pith papers · 2 internal anchors

  1. [1]

    G., Steiner, B., Tucker, P., V asudevan, V., W arden, P., Wicke, M., Yu, Y., and Zheng, X

    Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., V asudevan, V., W arden, P., Wicke, M., Yu, Y., and Zheng, X. Tensorflow: A system for large-scale machine learning. Tech. rep., Google Brain, 2016. arXiv preprint

    work page 2016

  2. [2]

    Neural machine translation by jointly learning to align and translate

    Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. InInternational Conference on Learning Representations(2015)

    work page 2015

  3. [3]

    D., Pietra, V

    Brown, P., Cocke, J., Pietra, S. D., Pietra, V. D., Jelinek, F., Mercer, R., and Roossin, P. A statistical approach to language translation. InProceedings of the 12th Conference on Computational Linguistics - Volume 1(Stroudsburg, PA, USA, 1988), COLING ’88, Association for Computational Linguistics, pp. 71–76

    work page 1988

  4. [4]

    F., Cocke, J., Pietra, S

    Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J. D., Mercer, R. L., and Roossin, P. S. A statistical approach to machine translation.Computational linguistics 16, 2 (1990), 79–85

    work page 1990

  5. [5]

    F., Pietra, V

    Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. The mathematics of statistical machine translation: Parameter estimation.Comput. Linguist. 19, 2 (June 1993), 263–311

    work page 1993

  6. [6]

    N-gram counts and language models from the common crawl

    Buck, C., Heafield, K., and V an Ooyen, B. N-gram counts and language models from the common crawl. InLREC (2014), vol. 2, Citeseer, p. 4

    work page 2014

  7. [7]

    Learning phrase representations using RNN encoder-decoder for statistical machine translation

    Cho, K., van Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing(2014)

    work page 2014

  8. [8]

    Learning recursive distributed representations for holistic computation.Connection Science 3, 4 (1991), 345–366

    Chrisman, L. Learning recursive distributed representations for holistic computation.Connection Science 3, 4 (1991), 345–366. 20

    work page 1991

  9. [10]

    A character-level decoder without explicit segmentation for neural machine translation.CoRR abs/1603.06147 (2016)

    Chung, J., Cho, K., and Bengio, Y. A character-level decoder without explicit segmentation for neural machine translation.CoRR abs/1603.06147 (2016)

    work page arXiv 2016

  10. [11]

    R., and Fonollosa, J

    Costa-Jussà, M. R., and Fonollosa, J. A. R. Character-based neural machine translation.CoRR abs/1603.00810 (2016)

    work page arXiv 2016

  11. [12]

    S., Monga, R., Chen, K., Devin, M., Le, Q

    Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., and Ng, A. Y. Large scale distributed deep networks. In NIPS (2012)

    work page 2012

  12. [13]

    M., and Makhoul, J

    Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R. M., and Makhoul, J. Fast and robust neural network joint models for statistical machine translation. InACL (1)(2014), Citeseer, pp. 1370–1380

    work page 2014

  13. [14]

    Multi-task learning for multiple language translation

    Dong, D., Wu, H., He, W., Yu, D., and W ang, H. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (2015), pp. 1723–1732

    work page 2015

  14. [15]

    Edinburgh’s phrase-based machine translation systems for WMT-14

    Durrani, N., Haddow, B., Koehn, P., and Heafield, K. Edinburgh’s phrase-based machine translation systems for WMT-14. InProceedings of the Ninth Workshop on Statistical Machine Translation (2014), Association for Computational Linguistics Baltimore, MD, USA, pp. 97–104

    work page 2014

  15. [16]

    E., and Lebiere, C

    F ahlman, S. E., and Lebiere, C. The cascade-correlation learning architecture. InAdvances in Neural Information Processing Systems 2(1990), Morgan Kaufmann, pp. 524–532

    work page 1990

  16. [17]

    A., Schmidhuber, J., and Cummins, F

    Gers, F. A., Schmidhuber, J., and Cummins, F. Learning to forget: Continual prediction with LSTM. Neural computation 12, 10 (2000), 2451–2471

    work page 2000

  17. [18]

    Pointing the unknown words

    Gülçehre, Ç., Ahn, S., Nallapati, R., Zhou, B., and Bengio, Y. Pointing the unknown words. CoRR abs/1603.08148 (2016)

    work page arXiv 2016

  18. [19]

    Deep learning with limited numerical precision.CoRR abs/1502.02551 (2015)

    Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep learning with limited numerical precision.CoRR abs/1502.02551 (2015)

    work page arXiv 2015

  19. [20]

    Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.CoRR abs/1510.00149 (2015)

    work page internal anchor Pith review arXiv 2015

  20. [21]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. InIEEE Conference on Computer Vision and Pattern Recognition(2015)

    work page 2015

  21. [22]

    Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001

    Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001

    work page 2001

  22. [23]

    Long short-term memory.Neural computation 9, 8 (1997), 1735–1780

    Hochreiter, S., and Schmidhuber, J. Long short-term memory.Neural computation 9, 8 (1997), 1735–1780

    work page 1997

  23. [24]

    Recurrent continuous translation models

    Kalchbrenner, N., and Blunsom, P. Recurrent continuous translation models. InConference on Empirical Methods in Natural Language Processing(2013)

    work page 2013

  24. [25]

    Adam: A Method for Stochastic Optimization

    Kingma, D. P., and Ba, J. Adam: A method for stochastic optimization.CoRR abs/1412.6980 (2014)

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  25. [26]

    J., and Marcu, D

    Koehn, P., Och, F. J., and Marcu, D. Statistical phrase-based translation. InProceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics(2003)

    work page 2003

  26. [27]

    arXiv:1605.04711 , year=

    Li, F., and Liu, B. Ternary weight networks.CoRR abs/1605.04711 (2016). 21

    work page arXiv 2016

  27. [28]

    Luong, M., and Manning, C. D. Achieving open vocabulary neural machine translation with hybrid word-character models.CoRR abs/1604.00788 (2016)

    work page arXiv 2016

  28. [29]

    V., Sutskever, I., Vinyals, O., and Kaiser, L

    Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., and Kaiser, L. Multi-task sequence to sequence learning. InInternational Conference on Learning Representations(2015)

    work page 2015

  29. [30]

    Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. InConference on Empirical Methods in Natural Language Processing(2015)

    work page 2015

  30. [31]

    V., Vinyals, O., and Zaremba, W

    Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., and Zaremba, W. Addressing the rare word problem in neural machine translation. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (2015)

    work page 2015

  31. [32]

    Reward augmented maximum likelihood for neural structured prediction

    Norouzi, M., Bengio, S., Chen, Z., Jaitly, N., Schuster, M., Wu, Y., and Schuurmans, D. Reward augmented maximum likelihood for neural structured prediction. InNeural Information Processing Systems(2016)

    work page 2016

  32. [33]

    Understanding the exploding gradient problem.CoRR abs/1211.5063 (2012)

    Pascanu, R., Mikolov, T., and Bengio, Y. Understanding the exploding gradient problem.CoRR abs/1211.5063 (2012)

    work page arXiv 2012

  33. [34]

    Sequence level training with recurrent neural networks

    Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. InInternational Conference on Learning Representations(2015)

    work page 2015

  34. [35]

    Japanese and Korean voice search.2012 IEEE International Conference on Acoustics, Speech and Signal Processing(2012)

    Schuster, M., and Nakajima, K. Japanese and Korean voice search.2012 IEEE International Conference on Acoustics, Speech and Signal Processing(2012)

    work page 2012

  35. [36]

    Bidirectional recurrent neural networks.IEEE Transactions on Signal Processing 45, 11 (Nov

    Schuster, M., and Paliwal, K. Bidirectional recurrent neural networks.IEEE Transactions on Signal Processing 45, 11 (Nov. 1997), 2673–2681

    work page 1997

  36. [37]

    On using very large target vocabulary for neural machine translation

    Sébastien, J., Kyunghyun, C., Memisevic, R., and Bengio, Y. On using very large target vocabulary for neural machine translation. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (2015)

    work page 2015

  37. [38]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics(2016)

    work page 2016

  38. [39]

    Minimum risk training for neural machine translation

    Shen, S., Cheng, Y., He, Z., He, W., Wu, H., Sun, M., and Liu, Y. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(2016)

    work page 2016

  39. [40]

    Highway Networks.arXiv2015, arXiv:1505.00387

    Srivastava, R. K., Greff, K., and Schmidhuber, J. Highway networks.CoRR abs/1505.00387 (2015)

    work page arXiv 2015

  40. [41]

    Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems(2014), pp. 3104–3112

    work page 2014

  41. [42]

    Coverage-based neural machine translation

    Tu, Z., Lu, Z., Liu, Y., Liu, X., and Li, H. Coverage-based neural machine translation. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics(2016)

    work page 2016

  42. [43]

    Quantized convolutional neural networks for mobile devices

    Wu, J., Leng, C., W ang, Y., Hu, Q., and Cheng, J. Quantized convolutional neural networks for mobile devices. CoRR abs/1512.06473 (2015)

    work page arXiv 2015

  43. [44]

    Recurrent neural network regularization, 2014

    Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization, 2014

    work page 2014

  44. [45]

    NMT before RL

    Zhou, J., Cao, Y., W ang, X., Li, P., and Xu, W. Deep recurrent models with fast-forward connections for neural machine translation.CoRR abs/1606.04199 (2016). 22 Table 11: Some example translations from PBMT [15], our GNMT system (the "NMT before RL", Table 9), and Human. Source and target sentences (human translations) are from the public benchmark WMT ...

    work page arXiv 2016