{"id":"9b510495-f2fe-41ef-aebf-47f7e1206fa8","arxiv_id":"1909.01792","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Mogrifier LSTM, which applies repeated mutual gating between the input and previous hidden state, outperforms the LSTM on PTB, Wikitext-2, Enwik8, and MWC language modeling.","lead":"This paper introduces the Mogrifier LSTM, which lets an LSTM's input and previous output repeatedly gate each other before the standard update. On word and character language modeling benchmarks it reports consistent gains over the LSTM, including several new state-of-the-art results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-run selection is not controlled for the Mogrifier: only the LSTM rerun distribution is reported, leaving the precise margin and SOTA claims dependent on an unverified symmetry.","rationale":"The reader's weakest assumption identifies precisely the same load-bearing concern: the reported improvements could be inflated by best-run selection, and the Mogrifier's rerun distribution is not provided. I agree this is the most important soft spot. The paper's own Appendix B quantifies the LSTM's best-run optimism at roughly 0.7 perplexity points, which is about one fifth to one quarter of the claimed 3–4 point margin, so the qualitative conclusion that the Mogrifier improves on the LSTM is not seriously threatened. However, the Mogrifier has two extra tuned hyperparameters and its rerun variance could differ from the LSTM's, so the precise magnitude and the state-of-the-art comparisons on datasets where margins are small are not fully secured. A focused rerun experiment would settle the question directly. Since the paper already discloses the limitation and the central architectural claim is well supported by the ablation, the reverse copy task, and the consistent direction of improvement across many datasets, I would not change the reader's ACCEPT verdict.","tokens_in":12760,"tokens_out":8831,"duration_ms":91810,"concrete_test":"Run at least 5 (ideally 10) full training runs of the Mogrifier with the exact best PTB hyperparameters (including r, k, and all tuned dropout and learning-rate values), and the same number of LSTM runs; report mean and standard deviation of validation and test perplexities for both. If the mean Mogrifier advantage over the mean LSTM remains at least 2 perplexity points and the best-run-to-mean shift is similar across architectures, the best-run concern is resolved. A smaller replication on WT2 or one character dataset would additionally test whether the SOTA margins survive selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's quantitative content—3–4 perplexity points on PTB/WT2, 0.01–0.05 bpc on character data, and new state-of-the-art on all but Enwik8—rests on the best validation run from black-box tuning for each architecture. Appendix B reports that for the LSTM with best hyperparameters, reruns have mean about 0.7 perplexity points worse than the best tuning run, with standard deviation about 0.2. No corresponding rerun distribution is reported for the Mogrifier. The Mogrifier has two additional tuned hyperparameters (rounds r and rank k, Appendix A), so its best-run estimate could be more optimistic than the LSTM's. The raw margin of 3–4 points is larger than the LSTM's measured 0.7-point rerun shift, so even a symmetric shift would not erase the qualitative improvement; however, the exact magnitude and several SOTA comparisons are at or below the scale of uncontrolled best-run selection. The paper assumes symmetry of best-run bias between architectures without evidence; this assumption is load-bearing for the stated precision and for the SOTA subclaim, though not for the qualitative direction of the effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Mogrifier LSTM, an extension in which the input x and the previous hidden state h_prev repeatedly gate each other through low-rank factorized linear maps before being fed to a standard LSTM cell (Eqs. 1-2). The authors evaluate the model on word-level language modelling (Penn Treebank, Wikitext-2) and character-level language modelling (PTB-char, MWC English and Finnish, Enwik8), and on a synthetic reverse-copy task. They report consistent improvements over a heavily tuned LSTM baseline (about 3-4 perplexity points on word tasks and 0.012-0.046 bpc on character tasks), claim new state-of-the-art results on all datasets except Enwik8, and provide an ablation study showing that the alternating 'zigzag' gating and the low-rank factorization contribute to the gains. They also test several hypotheses about the source of the improvement and release code and tuner output.","tokens_in":1368,"tokens_out":1839,"duration_ms":99573,"significance":"If the reported gains hold, the Mogrifier is a simple, low-cost modification to a widely used architecture, with consistent gains across languages, modalities, and dataset sizes. The paper is unusually careful in several respects: both the LSTM and the Mogrifier are tuned with the same black-box tuner, the LSTM rerun variance is reported, ablations are included, and code and tuner output are released. The improvement over the LSTM baseline is consistent across all datasets, which makes the qualitative direction of the claim credible. However, the headline quantitative margins and the state-of-the-art subclaims are weakened by the absence of rerun statistics for the Mogrifier and by reliance on external baselines that were not reproduced under the same protocol.","major_comments":[{"comment":"Appendix B reports that repeated runs of the LSTM with the best hyperparameters are on average about 0.7 perplexity points worse than the best tuning run, with a standard deviation of about 0.2, but no equivalent rerun distribution is reported for the Mogrifier. Since the Mogrifier has two additional tuned hyperparameters (r and k, Appendix A), its best-run estimate is at least as likely to be optimistic as the LSTM's. This matters because several headline comparisons are at the scale of this bias, for example the Wikitext-2 dynamic-evaluation test result (Mogrifier 39.0 vs. previous state of the art 39.1, Table 1) and the PTB character-level margin of 0.012 bpc (Table 2). The qualitative improvement over the LSTM is probably real, but the claimed 3-4 point margins and the state-of-the-art subclaim need either Mogrifier rerun statistics or an explicit caveat that the numbers are best-run estimates.","section":"Appendix B; Section 3.4; Table 1"},{"comment":"The abstract's claim of a new state of the art on all datasets except Enwik8 is supported only by comparisons with previously published numbers obtained under different training and evaluation protocols. The paper itself concedes this for Transformer-XL ('we did not test the Transformer-XL ourselves, so fair comparison is not possible'), and the same caveat applies to the FRAGE, AWD-LSTM-MoS, Trellis, and HCLM results used in Tables 1 and 2. Since the authors' own Section 3.1 warns that evaluations on larger datasets come with larger error bars, the state-of-the-art claim should either be restricted to the controlled LSTM-versus-Mogrifier comparison or be accompanied by an explicit statement that external results are listed without protocol matching.","section":"Abstract; Section 3.4; Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence 'Multiplication with the constant 2 ensures that randomly initialized Qi, Ri matrices result in transformations close to identity' should specify the initialization distribution; the property holds for small zero-mean weights, not for arbitrary random initialization.","section":"Section 2, Eqs. (1)-(2)"},{"comment":"There is a typo: 'repeated runs with the same parameters may be give slightly worse results' should read 'may give slightly worse results'.","section":"Appendix B"},{"comment":"The statement that 0.012 bpc on character PTB is equivalent to about 3 perplexity points on word-level PTB needs a derivation or a citation, since the relationship between bpc and perplexity depends on the vocabulary size and the comparison is not immediately obvious.","section":"Section 3.4"},{"comment":"The two NaN entries for dynamic-evaluation test runs (MWC EN LSTM and MWC FI Mogrifier) are noted only in the table caption; the main text should state how these missing values affect the reported comparisons and the dynamic-evaluation claims for those rows.","section":"Section 3.4, Table 2"},{"comment":"The phrase 'both models can solve the task perfectly' is not directly supported by the cross-entropy curves; please state an accuracy threshold or report exact cross-entropy values for the short-sequence conditions.","section":"Section 4.3, Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript's central improvement direction is credible and the paper is generally well executed. The main risk is that the headline state-of-the-art and margin claims rest on best-run selection without a comparable rerun distribution for the Mogrifier. I would ask for a short addendum with Mogrifier rerun statistics (or a softened claim) and a more explicit caveat about external baselines before accepting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, carefully reported architecture paper, and the qualitative claim holds. The Mogrifier's alternating mutual gating of input and previous hidden state is genuinely new relative to mLSTM, and the ablation shows the zigzag is load-bearing. The consistent 3–4 perplexity point / 0.01–0.05 bpc gains over a well-tuned LSTM across word and character tasks are credible, not a tuning artifact.\n\nWhat's good: the empirical discipline is above the usual bar for this line of work. Both models go through the same black-box tuner; ablations cover rounds, rank, no-zigzag, full-rank, and several alternative explanations; the reverse copy task adds a non-language control; code and tuner output are released. The paper also openly flags its own limitations—dynamic evaluation divergences, Transformer-XL not re-run, and the best-run caveat in Appendix B. The comparison to mLSTM is fair and the citation pattern is unremarkable.\n\nThe soft spot the stress-test points to is real but minor. Appendix B gives the LSTM rerun distribution (mean about 0.7 ppl off the best tuning run, std about 0.2) but no corresponding reruns for the Mogrifier, which has two additional tuned hyperparameters. So the precise margins and the state-of-the-art subclaims carry more uncertainty than the main text suggests. The direction of the effect is not at risk—the gains are 4–5 times the LSTM's measured rerun shift—but the exact magnitude and some SOTA comparisons (especially Enwik8 dynamic eval, which relies on a personal communication) should be read as approximate. This is a request for a small additional experiment or a softer wording, not a rejection.\n\nWho it's for: anyone still using LSTMs for sequence modelling, especially language modelling. It won't change the Transformer conversation, but it's a genuinely useful upgrade to a workhorse architecture. I'd accept this for peer review and, after adding Mogrifier rerun stats or tempering the SOTA language, publish. Bring it to reading group if you work on recurrent architectures.","headline":"A solid, well-reported architecture paper whose qualitative gains over a well-tuned LSTM hold up; the stress-test concern about best-run selection is real but minor, affecting precise margins more than the core result.","tokens_in":13538,"tokens_out":2192,"would_cite":true,"duration_ms":21141,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Mogrifier LSTM shows that mutual gating of input and previous state improves language-modeling generalization by 3–4 perplexity points and sets a new state of the art on all but the largest tested dataset.","keywords":["Mogrifier LSTM","language modeling","recurrent neural networks","gating mechanisms","context-dependent transitions","perplexity","bits per character"],"falsifier":"Run the same hyperparameter tuning and evaluation protocol on PTB and Wikitext-2 many times for both the LSTM and the Mogrifier, with the same number of trials, and compare distributions of validation perplexity rather than best runs. If the Mogrifier's rerun mean advantage over the LSTM is not at least a few perplexity points (or overlaps within one standard deviation), the state-of-the-art claim fails.","tokens_in":12561,"feed_emoji":"🧠","tokens_out":6909,"duration_ms":56031,"temperature":0.7,"pith_summary":"This paper proposes a small architectural extension to the Long Short-Term Memory network in which the current input token and the previous hidden state repeatedly gate one another before the ordinary LSTM update runs. The authors argue that this mutual gating makes the LSTM's transition function context-dependent, letting the network reshape input embeddings and recurrent state in light of each other. On word-level language modeling they report a 3–4 perplexity-point improvement over an equally sized LSTM on Penn Treebank and Wikitext-2, and on character-level tasks an improvement of 0.01–0.05 bits per character, establishing a new state of the art on all datasets tested except Enwik8. The central motivation is that a context-free input embedding is a bottleneck for generalization, and the alternating gating scheme is a cheap way to relieve it.","feed_headline":"Mutual gating lifts LSTM by 3–4 perplexity points","feed_subtitle":"An alternating input-state gate lifts the LSTM to state-of-the-art scores on most language modeling benchmarks.","key_machinery":"The Mogrifier's load-bearing object is the alternating mutual-gating loop defined by Eqs. (1)–(2): for an odd round the input is rescaled by a sigmoid of a linear map of the previous state, and for an even round the state is rescaled by a sigmoid of a linear map of the updated input. After $r$ rounds (typically $r=5$ or $6$) the final modulated input and state are fed to an ordinary LSTM. This composition is equivalent to making the LSTM's transition function context-dependent; it can also be read as contextualizing the input embedding. The constant factor 2 and low-rank factorization of the gating matrices are what let the loop start near the identity and stay parameter-efficient.","core_discovery":"The central claim is that interleaving two elementwise multiplicative gates between $x$ and $h_{\\mathrm{prev}}$ — $x_i = 2\\sigma(Q_i h^{i-1}_{\\mathrm{prev}}) \\odot x_{i-2}$ for odd $i$, and $h^i_{\\mathrm{prev}} = 2\\sigma(R_i x^{i-1}) \\odot h^{i-2}_{\\mathrm{prev}}$ for even $i$ — before the standard LSTM update consistently improves language-modeling generalization over the plain LSTM with the same parameter budget. The factor 2 keeps randomly initialized gates near identity, and low-rank factorization of $Q_i, R_i$ keeps the added parameters modest. On two word-level corpora the gain is 3–4 perplexity points; on four character-level corpora it is 0.01–0.05 bits per character, enough to claim a new state of the art everywhere except Enwik8, where it closes most of the gap to a similarly sized Transformer.","pith_inferences":["The same alternating-gating wrapper could be bolted onto other recurrent cells (the paper mentions GRU and Elman variants as future work); the experiments here leave open whether the benefit transfers.","The reverse-copy result implies the mechanism reduces the embedding size needed; a natural test is whether Mogrifier retains its advantage when embedding and hidden size are fixed equal.","The paper does not report a rerun distribution for the Mogrifier comparable to the LSTM's; a small repeated-run study would settle how much of the stated 3–4 point margin is tuning luck."],"forward_implications":["With the same parameter budget and tuning protocol, the Mogrifier beats the LSTM on every dataset tested, word- and character-level, with and without dynamic evaluation.","It sets a new state of the art on Penn Treebank, Wikitext-2, the Mikolov-preprocessed PTB character task, and both English and Finnish MWC subsets.","The ablation shows the two-way alternating zigzag matters: a 'no-zigzag' variant that gates from the original inputs only performs worse, and the full-rank Mogrifier also beats the LSTM, so the gain is not mostly from the low-rank factorization.","The reverse copy task indicates the architecture works with much smaller embeddings than an LSTM, suggesting the input–state interaction substitutes for embedding capacity."],"supporting_citations":[{"why":"Defines the LSTM cell that the Mogrifier wraps; the base architecture for all comparisons.","marker":"Hochreiter and Schmidhuber 1997"},{"why":"Multiplicative LSTM, the closest prior model; the paper reproduces it and compares directly.","marker":"Krause et al. 2016"},{"why":"AWD-LSTM, the prior state-of-the-art recipe and baseline on PTB, Wikitext-2, and character datasets.","marker":"Merity et al. 2017"},{"why":"Transformer-XL, the strong Enwik8 baseline whose gap the Mogrifier largely closes.","marker":"Dai et al. 2019"},{"why":"Provides the dropout, embedding-ratio, and hyperparameter-tuning protocol used for both models.","marker":"Melis et al. 2018"},{"why":"Supplies the Penn Treebank preprocessing used in the word-level experiments.","marker":"Mikolov et al. 2010"},{"why":"Introduces Wikitext-2, the second word-level benchmark.","marker":"Merity et al. 2016"},{"why":"Provides the English and Finnish character-level corpora used for evaluation.","marker":"Kawakami et al. 2017"},{"why":"Dynamic evaluation, the protocol used for the additional model comparisons.","marker":"Krause et al. 2017"},{"why":"Mixture of Softmaxes baselines on PTB and Wikitext-2 that the Mogrifier outperforms.","marker":"Yang et al. 2017"}],"fun_headline_variants":["Mogrifier gates lift LSTM to SOTA on most language benchmarks","Interleaved input-output gates improve LSTM by 3-4 perplexity","Mutual gating: LSTM's new SOTA on four of five datasets","LSTM with mutual gating: close the gap to Transformers on Enwik8","Gated interplay: LSTM gains 3-4 perplexity points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported margin is measured from the single best hyperparameter-tuning run for each model; if the Mogrifier's best run is more optimistic than the LSTM's (the paper only reports rerun statistics for the LSTM), the 3–4 point gap could shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Mogrifier gates lift LSTM to SOTA on most language benchmarks","Interleaved input-output gates improve LSTM by 3-4 perplexity","Mutual gating: LSTM's new SOTA on four of five datasets","LSTM with mutual gating: close the gap to Transformers on Enwik8","Gated interplay: LSTM gains 3-4 perplexity points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000869,"raw_usage":{"total_tokens":3753,"prompt_tokens":921,"completion_tokens":2832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2729}},"tokens_in":537,"tokens_out":2832,"duration_ms":21525,"temperature":1.0,"reasoning_tokens":2729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:07:33.125278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same hyperparameter tuning and evaluation protocol on PTB and Wikitext-2 many times for both the LSTM and the Mogrifier, with the same number of trials, and compare distributions of validation perplexity rather than best runs. If the Mogrifier's rerun mean advantage over the LSTM is not at least a few perplexity points (or overlaps within one standard deviation), the state-of-the-art claim fails.","supporting_citations":[{"cited_title":"Lstm can solve hard long time lag problems","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM cell that the Mogrifier wraps; the base architecture for all comparisons."},{"cited_title":"Pushing the bounds of dropout","cited_arxiv_id":"1805.09208","evidence_quote":"Provides the dropout, embedding-ratio, and hyperparameter-tuning protocol used for both models."},{"cited_title":"Recurrent neural network based language model","cited_arxiv_id":null,"evidence_quote":"Supplies the Penn Treebank preprocessing used in the word-level experiments."}],"review_version":1}