Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Mogrifier LSTM

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The Mogrifier LSTM shows that mutual gating of input and previous state improves language-modeling generalization by 3–4 perplexity points and sets a new state of the art on all but the largest tested dataset.

desk verdict A solid, well-reported architecture paper whose qualitative gains over a well-tuned LSTM hold up; the stress-test concern about best-run selection is real but minor, affecting precise margins more than the core result. read the letter →

arxiv 1909.01792 v2 pith:Q6HDFR23 submitted 2019-09-04 cs.CL

Gábor Melis , Tomáš Kočiský , Phil Blunsom This is my paper · ORCID
classification cs.CL
keywords MogrifierLSTMlanguagemodelingrecurrentneuralnetworksgatingmechanismscontext-dependenttransitionsperplexitybitspercharacter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a small architectural extension to the Long Short-Term Memory network in which the current input token and the previous hidden state repeatedly gate one another before the ordinary LSTM update runs. The authors argue that this mutual gating makes the LSTM's transition function context-dependent, letting the network reshape input embeddings and recurrent state in light of each other. On word-level language modeling they report a 3–4 perplexity-point improvement over an equally sized LSTM on Penn Treebank and Wikitext-2, and on character-level tasks an improvement of 0.01–0.05 bits per character, establishing a new state of the art on all datasets tested except Enwik8. The central motivation is that a context-free input embedding is a bottleneck for generalization, and the alternating gating scheme is a cheap way to relieve it.

What carries the argument

The Mogrifier's load-bearing object is the alternating mutual-gating loop defined by Eqs. (1)–(2): for an odd round the input is rescaled by a sigmoid of a linear map of the previous state, and for an even round the state is rescaled by a sigmoid of a linear map of the updated input. After $r$ rounds (typically $r=5$ or $6$) the final modulated input and state are fed to an ordinary LSTM. This composition is equivalent to making the LSTM's transition function context-dependent; it can also be read as contextualizing the input embedding. The constant factor 2 and low-rank factorization of the gating matrices are what let the loop start near the identity and stay parameter-efficient.

What would settle it

Run the same hyperparameter tuning and evaluation protocol on PTB and Wikitext-2 many times for both the LSTM and the Mogrifier, with the same number of trials, and compare distributions of validation perplexity rather than best runs. If the Mogrifier's rerun mean advantage over the LSTM is not at least a few perplexity points (or overlaps within one standard deviation), the state-of-the-art claim fails.

Watch

Extended reading notes

Core claim

The central claim is that interleaving two elementwise multiplicative gates between $x$ and $h_{\mathrm{prev}}$ — $x_i = 2\sigma(Q_i h^{i-1}_{\mathrm{prev}}) \odot x_{i-2}$ for odd $i$, and $h^i_{\mathrm{prev}} = 2\sigma(R_i x^{i-1}) \odot h^{i-2}_{\mathrm{prev}}$ for even $i$ — before the standard LSTM update consistently improves language-modeling generalization over the plain LSTM with the same parameter budget. The factor 2 keeps randomly initialized gates near identity, and low-rank factorization of $Q_i, R_i$ keeps the added parameters modest. On two word-level corpora the gain is 3–4 perplexity points; on four character-level corpora it is 0.01–0.05 bits per character, enough to claim a new state of the art everywhere except Enwik8, where it closes most of the gap to a similarly sized Transformer.

Load-bearing premise

The reported margin is measured from the single best hyperparameter-tuning run for each model; if the Mogrifier's best run is more optimistic than the LSTM's (the paper only reports rerun statistics for the LSTM), the 3–4 point gap could shrink or vanish.

Editorial extensions

If this is right

  • With the same parameter budget and tuning protocol, the Mogrifier beats the LSTM on every dataset tested, word- and character-level, with and without dynamic evaluation.
  • It sets a new state of the art on Penn Treebank, Wikitext-2, the Mikolov-preprocessed PTB character task, and both English and Finnish MWC subsets.
  • The ablation shows the two-way alternating zigzag matters: a 'no-zigzag' variant that gates from the original inputs only performs worse, and the full-rank Mogrifier also beats the LSTM, so the gain is not mostly from the low-rank factorization.
  • The reverse copy task indicates the architecture works with much smaller embeddings than an LSTM, suggesting the input–state interaction substitutes for embedding capacity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same alternating-gating wrapper could be bolted onto other recurrent cells (the paper mentions GRU and Elman variants as future work); the experiments here leave open whether the benefit transfers.
  • The reverse-copy result implies the mechanism reduces the embedding size needed; a natural test is whether Mogrifier retains its advantage when embedding and hidden size are fixed equal.
  • The paper does not report a rerun distribution for the Mogrifier comparable to the LSTM's; a small repeated-run study would settle how much of the stated 3–4 point margin is tuning luck.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes the Mogrifier LSTM, an extension in which the input x and the previous hidden state h_prev repeatedly gate each other through low-rank factorized linear maps before being fed to a standard LSTM cell (Eqs. 1-2). The authors evaluate the model on word-level language modelling (Penn Treebank, Wikitext-2) and character-level language modelling (PTB-char, MWC English and Finnish, Enwik8), and on a synthetic reverse-copy task. They report consistent improvements over a heavily tuned LSTM baseline (about 3-4 perplexity points on word tasks and 0.012-0.046 bpc on character tasks), claim new state-of-the-art results on all datasets except Enwik8, and provide an ablation study showing that the alternating 'zigzag' gating and the low-rank factorization contribute to the gains. They also test several hypotheses about the source of the improvement and release code and tuner output.

Significance. If the reported gains hold, the Mogrifier is a simple, low-cost modification to a widely used architecture, with consistent gains across languages, modalities, and dataset sizes. The paper is unusually careful in several respects: both the LSTM and the Mogrifier are tuned with the same black-box tuner, the LSTM rerun variance is reported, ablations are included, and code and tuner output are released. The improvement over the LSTM baseline is consistent across all datasets, which makes the qualitative direction of the claim credible. However, the headline quantitative margins and the state-of-the-art subclaims are weakened by the absence of rerun statistics for the Mogrifier and by reliance on external baselines that were not reproduced under the same protocol.

major comments (2)
  1. [Appendix B; Section 3.4; Table 1] Appendix B reports that repeated runs of the LSTM with the best hyperparameters are on average about 0.7 perplexity points worse than the best tuning run, with a standard deviation of about 0.2, but no equivalent rerun distribution is reported for the Mogrifier. Since the Mogrifier has two additional tuned hyperparameters (r and k, Appendix A), its best-run estimate is at least as likely to be optimistic as the LSTM's. This matters because several headline comparisons are at the scale of this bias, for example the Wikitext-2 dynamic-evaluation test result (Mogrifier 39.0 vs. previous state of the art 39.1, Table 1) and the PTB character-level margin of 0.012 bpc (Table 2). The qualitative improvement over the LSTM is probably real, but the claimed 3-4 point margins and the state-of-the-art subclaim need either Mogrifier rerun statistics or an explicit caveat that the numbers are best-run estimates.
  2. [Abstract; Section 3.4; Tables 1 and 2] The abstract's claim of a new state of the art on all datasets except Enwik8 is supported only by comparisons with previously published numbers obtained under different training and evaluation protocols. The paper itself concedes this for Transformer-XL ('we did not test the Transformer-XL ourselves, so fair comparison is not possible'), and the same caveat applies to the FRAGE, AWD-LSTM-MoS, Trellis, and HCLM results used in Tables 1 and 2. Since the authors' own Section 3.1 warns that evaluations on larger datasets come with larger error bars, the state-of-the-art claim should either be restricted to the controlled LSTM-versus-Mogrifier comparison or be accompanied by an explicit statement that external results are listed without protocol matching.
minor comments (5)
  1. [Section 2, Eqs. (1)-(2)] The sentence 'Multiplication with the constant 2 ensures that randomly initialized Qi, Ri matrices result in transformations close to identity' should specify the initialization distribution; the property holds for small zero-mean weights, not for arbitrary random initialization.
  2. [Appendix B] There is a typo: 'repeated runs with the same parameters may be give slightly worse results' should read 'may give slightly worse results'.
  3. [Section 3.4] The statement that 0.012 bpc on character PTB is equivalent to about 3 perplexity points on word-level PTB needs a derivation or a citation, since the relationship between bpc and perplexity depends on the vocabulary size and the comparison is not immediately obvious.
  4. [Section 3.4, Table 2] The two NaN entries for dynamic-evaluation test runs (MWC EN LSTM and MWC FI Mogrifier) are noted only in the table caption; the main text should state how these missing values affect the reported comparisons and the dynamic-evaluation claims for those rows.
  5. [Section 4.3, Fig. 4] The phrase 'both models can solve the task perfectly' is not directly supported by the cross-entropy curves; please state an accuracy threshold or report exact cross-entropy values for the short-sequence conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the Mogrifier's reported gains are direct held-out measurements, with the disclosed best-run caveat being a robustness limitation rather than a constructional equivalence.

full rationale

The paper makes no first-principles derivation that could reduce to its own inputs. The central claim is empirical: the Mogrifier's mutual-gating mechanism (Eqs. 1--2) is trained and evaluated on held-out test sets (PTB, Wikitext-2, MWC, Enwik8), and the reported improvements of 3--4 perplexity points or 0.01--0.05 bpc are measured test-set numbers, not quantities fitted from those same test sets. Hyperparameters are tuned on validation with Google Vizier and the best validation runs are reported; this is disclosed, and Appendix B explicitly quantifies rerun variance for the LSTM (mean about 0.7 perplexity points off, standard deviation about 0.2) while noting that the tuner exploits stochasticity. That caveat affects the precision of state-of-the-art comparisons but does not make the result follow by definition. Self-citations to Melis et al. (2017, 2018) are used only for experimental protocol (BPTT window size, dropout tuning methodology) and are not load-bearing for the architectural claim. The ablation study (Section 4.1), the comparison to the mLSTM (Section 4.2), and the reverse-copy task (Section 4.3) are independent empirical checks against alternative explanations. No equation, fitted parameter, or cited result is equivalent to the paper's conclusion by construction, so there is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The Mogrifier introduces two new hyperparameters (rounds r, rank k) that are tuned on validation data; they are the only architecture-specific free parameters. The paper assumes the standard LSTM formulation and that the chosen benchmarks and tuning procedure give fair comparisons. No invented entities are introduced.

free parameters (2)
  • mogrifier_rounds (r) = r=5-6 typical; tuned 0-6
    Number of alternating gating rounds; tuned per dataset via Google Vizier (Appendix A). r=0 recovers plain LSTM; the improvement is sensitive to r (Fig. 3).
  • mogrifier_rank (k) = k=40-90 typical; full rank allowed
    Rank of low-rank factorization of Q_i and R_i; tuned per dataset. The authors note full-rank is close, so k is not critical to the central claim.
assumptions (3)
  • standard math Standard LSTM update equations (Sak et al. 2014) as the base model.
    The Mogrifier wraps this LSTM; the paper assumes this formulation is correct.
  • domain assumption The chosen language modeling benchmarks and preprocessing are appropriate measures of generalization.
    The empirical claim is based on PTB, Wikitext-2, Enwik8, and MWC; the paper argues for small-scale evaluation but this is a domain choice.
  • domain assumption The hyperparameter tuning procedure (Google Vizier) gives a fair comparison between architectures.
    Both LSTM and Mogrifier are tuned, but the Mogrifier has extra hyperparameters (r,k). The paper's sensitivity analysis partially supports this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mogrifier LSTM." pith.science (2026). https://pith.science/paper/Q6HDFR23

@misc{pith2026190901792,
  author       = {Pith},
  title        = {Pith review of: Mogrifier LSTM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6HDFR23}},
  note         = {Machine review of arXiv:1909.01792}
}
read the original abstract

Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur. Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language. In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output. This mechanism affords the modelling of a richer space of interactions between inputs and their context. Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent. Experiments demonstrate markedly improved generalization on language modelling in the range of 3-4 perplexity points on Penn Treebank and Wikitext-2, and 0.01-0.05 bpc on four character-based datasets. We establish a new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.

Figures

Figures reproduced from arXiv: 1909.01792 by the authors.

Figure 1
Figure 1. Mogrifier with 5 rounds of updates. The previous state [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. “No-zigzag” Mogrifier for the ablation study. Gating is always based on the original inputs. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Perplexity vs the rounds r in the PTB ablation study [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cross-entropy vs sequence length in the reverse copy task with i.i.d. tokens. Lower is better. The [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Average per-word validation cross-entropies for hyperparameter combinations in the neighbourhood of [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Average per-word validation cross-entropies for hyperparameter combinations in the neighbour [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Good, The Efficient and the Inductive Biases: Exploring Efficiency in Deep Learning Through the Use of Inductive Biases

    cs.LG 2024-11 conditional novelty 3.0 of 10

    A dissertation synthesizing the author's papers on continuous kernel convolutions and symmetry-preserving architectures, claiming these inductive biases improve deep learning efficiency.

Reference graph

Works this paper leans on

53 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  2. [2]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  3. [3]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  4. [4]

    Analyzing the behavior of visual question answering models

    Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. arXiv preprint arXiv:1606.07356, 2016

    arXiv 2016

  5. [5]

    Unitary evolution recurrent neural networks

    Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pages 1120--1128, 2016

    work page 2016

  6. [6]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

  7. [7]

    Trellis networks for sequence modeling

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv preprint arXiv:1810.06682, 2018

    arXiv 2018

  8. [8]

    Reinforcement learning with long short-term memory

    Bram Bakker. Reinforcement learning with long short-term memory. In Advances in neural information processing systems, pages 1475--1482, 2002

    work page 2002

Show all 53 references
  1. [9]

    Synthetic and natural noise both break neural machine translation

    Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. arXiv preprint arXiv:1711.02173, 2017

    2017 arXiv

  2. [10]

    Gated feedback recurrent neural networks

    Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In International Conference on Machine Learning, pages 2067--2075, 2015

    2015

  3. [11]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019

    1901 arXiv

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  5. [13]

    Finding structure in time

    Jeffrey L Elman. Finding structure in time. Cognitive science, 14 0 (2): 0 179--211, 1990

    1990

  6. [14]

    Input switched affine networks: An rnn architecture designed for interpretability

    Jakob N Foerster, Justin Gilmer, Jascha Sohl-Dickstein, Jan Chorowski, and David Sussillo. Input switched affine networks: An rnn architecture designed for interpretability. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1136--1145. JM...

    2017

  7. [15]

    A theoretically grounded application of dropout in recurrent neural networks

    Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1019--1027, 2016

    2016

  8. [16]

    Google vizier: A service for black-box optimization

    Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1487--1495. ACM, 2017

    2017

  9. [17]

    Frage: frequency-agnostic word representation

    Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Frage: frequency-agnostic word representation. In Advances in Neural Information Processing Systems, pages 1334--1345, 2018

    2018

  10. [18]

    Hypernetworks

    David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016

    2016 arXiv

  11. [19]

    Lstm can solve hard long time lag problems

    Sepp Hochreiter and J \"u rgen Schmidhuber. Lstm can solve hard long time lag problems. In Advances in neural information processing systems, pages 473--479, 1997

    1997

  12. [20]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018

    2018 arXiv

  13. [21]

    The human knowledge compression contest

    Marcus Hutter. The human knowledge compression contest. URL http://prize. hutter1. net, 6, 2012

    2012

  14. [22]

    Tying word vectors and word classifiers: A loss framework for language modeling

    Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. CoRR, abs/1611.01462, 2016. URL http://arxiv.org/abs/1611.01462

    2016 arXiv

  15. [23]

    Adversarial example generation with syntactically controlled paraphrase networks

    Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804.06059, 2018

    2018 arXiv

  16. [24]

    Adversarial examples for evaluating reading comprehension systems

    Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328, 2017

    2017 arXiv

  17. [25]

    Learning to create and reuse words in open-vocabulary neural language modeling

    Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to create and reuse words in open-vocabulary neural language modeling. arXiv preprint arXiv:1704.06986, 2017

    2017 arXiv

  18. [26]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  19. [27]

    Multiplicative LSTM for sequence modelling

    Ben Krause, Liang Lu, Iain Murray, and Steve Renals. Multiplicative LSTM for sequence modelling. CoRR, abs/1609.07959, 2016. URL http://arxiv.org/abs/1609.07959

    2016 arXiv

  20. [28]

    Dynamic evaluation of neural sequence models

    Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. arXiv preprint arXiv:1709.07432, 2017

    2017 arXiv

  21. [29]

    Dynamic evaluation of transformer language models

    Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of transformer language models. arXiv preprint arXiv:1904.08378, 2019

    1904 arXiv

  22. [30]

    Lstms can learn syntax-sensitive dependencies well, but modeling structure makes them better

    Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. Lstms can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vo...

    2018

  23. [31]

    Assessing the ability of lstms to learn syntax-sensitive dependencies

    Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4: 0 521--535, 2016

    2016

  24. [32]

    Building a large annotated corpus of english: The Penn treebank

    Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The Penn treebank. Computational linguistics, 19 0 (2): 0 313--330, 1993

    1993

  25. [33]

    A system for robotic heart surgery that learns to tie knots using recurrent neural networks

    Hermann Mayer, Faustino Gomez, Daan Wierstra, Istvan Nagy, Alois Knoll, and J \"u rgen Schmidhuber. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Advanced Robotics, 22 0 (13-14): 0 1521--1537, 2008

    2008

  26. [34]

    On the state of the art of evaluation in neural language models

    G \'a bor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589, 2017

    2017 arXiv

  27. [35]

    Pushing the bounds of dropout

    G \'a bor Melis, Charles Blundell, Tom \'a s Ko c isk \`y , Karl Moritz Hermann, Chris Dyer, and Phil Blunsom. Pushing the bounds of dropout. arXiv preprint arXiv:1805.09208, 2018

    2018 arXiv

  28. [36]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, abs/1609.07843, 2016. URL http://arxiv.org/abs/1609.07843

    2016 arXiv

  29. [37]

    Regularizing and optimizing lstm language models

    Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm language models. arXiv preprint arXiv:1708.02182, 2017

    2017 arXiv

  30. [38]

    An analysis of neural language modeling at multiple scales

    Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv:1803.08240, 2018

    2018 arXiv

  31. [39]

    Recurrent neural network based language model

    Tomas Mikolov, Martin Karafi \'a t, Lukas Burget, Jan Cernock \`y , and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, page 3, 2010

    2010

  32. [40]

    Lexical features in coreference resolution: To be used with caution

    Nafise Sadat Moosavi and Michael Strube. Lexical features in coreference resolution: To be used with caution. arXiv preprint arXiv:1704.06779, 2017

    2017 arXiv

  33. [41]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310--1318, 2013

    2013

  34. [42]

    Deep contextualized word representations

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018

    2018 arXiv

  35. [43]

    Using the output embedding to improve language models

    Ofir Press and Lior Wolf. Using the output embedding to improve language models. CoRR, abs/1608.05859, 2016. URL http://arxiv.org/abs/1608.05859

    2016 arXiv

  36. [44]

    Learning representations by back-propagating errors

    David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning representations by back-propagating errors. Cognitive modeling, 5 0 (3): 0 1, 1988

    1988

  37. [45]

    Senior, and Fran c oise Beaufays

    Hasim Sak, Andrew W. Senior, and Fran c oise Beaufays. Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition. CoRR, abs/1402.1128, 2014. URL http://arxiv.org/abs/1402.1128

    2014 arXiv

  38. [46]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  39. [47]

    Generating text with recurrent neural networks

    Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017--1024, 2011

    2011

  40. [48]

    Sequence to sequence learning with neural networks

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104--3112, 2014

    2014

  41. [49]

    Sequence modeling via segmentations

    Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. Sequence modeling via segmentations. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3674--3683. JMLR. org, 2017

    2017

  42. [50]

    Backpropagation through time: what it does and how to do it

    Paul J Werbos et al. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78 0 (10): 0 1550--1560, 1990

    1990

  43. [51]

    On multiplicative integration with recurrent neural networks

    Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan R Salakhutdinov. On multiplicative integration with recurrent neural networks. In Advances in neural information processing systems, pages 2856--2864, 2016

    2016

  44. [52]

    Breaking the softmax bottleneck: a high-rank rnn language model

    Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: a high-rank rnn language model. arXiv preprint arXiv:1711.03953, 2017

    2017 arXiv

  45. [53]

    Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. CoRR, abs/1611.01578, 2016. URL http://arxiv.org/abs/1611.01578

    2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.