{"id":"b6ea523a-12ed-4417-85bd-edeafa845b1f","arxiv_id":"1909.00111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"More training data and larger LSTM hidden layers yield diminishing returns on subject-verb agreement accuracy, and GPT and BERT sometimes score below LSTMs trained on far less data.","lead":"This study trains 125 LSTM language models at five sizes and five training corpus sizes, and measures how often they pick the grammatically correct verb in tricky sentences. It finds that adding more data or parameters gives diminishing returns, and that some huge pretrained Transformer models do no better.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transformer comparison rests on Wolf (2019)'s different scoring metric and stimuli; without a matched re-evaluation, the claim that Transformers underperform LSTMs is unproven.","rationale":"The reader's weakest assumption identifies the same load-bearing concern. The central scaling finding—saturation beyond 400 hidden units and inconsistent corpus-size gains—is well supported by 125 models, five seeds, and Bayes-factor analyses. The extrapolation to trillions of tokens is an illustrative projection, but the qualitative conclusion is robust: Tables 3 and 4 show that using any observed doubling rate still gives data requirements orders of magnitude beyond current corpora, so even if the log-linear assumption is imperfect, the claim of impractical data needs is not overturned. The only part of the argument that changes the headline claim is the Transformer comparison, because it is the sole basis for extending the conclusion from RNNs to Transformers and it relies on an external, protocol-mismatched result. Since this concern matches the reader's condition, the appropriate verdict is unchanged: conditional acceptance pending a matched re-evaluation of GPT and BERT or a softening of the comparative claim.","tokens_in":10510,"tokens_out":6682,"duration_ms":108059,"concrete_test":"Evaluate GPT (110M) and BERT Base on the exact high-frequency-verb subset of the Marvin and Linzen (2018) challenge set used for the LSTM models, scoring all models with the same protocol: use total sentence log-probability for GPT and left-context masked verb probability for BERT, and additionally re-score the trained LSTMs with Wolf's rank-of-target-verb metric. Then recompute per-construction accuracies and compare with Figures 3-5. If GPT/BERT no longer underperform the LSTMs in the seven constructions, the comparative claim in the Abstract and Section 4 must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The broad conclusion that reliable and data-efficient learning of syntax requires external supervision or stronger inductive bias than RNNs and Transformers depends in part on the claim that GPT and BERT perform even more poorly than the paper's LSTMs in some constructions (Abstract; Section 4). This comparative claim is not established by the paper's own experiments: Transformer accuracies are taken from Wolf (2019), who scored models by the rank of the two target verbs given the prefix, while the LSTMs were scored by total sentence log-probability, and Wolf's stimuli were not restricted to the high-frequency verb set used for the LSTMs (Section 2, footnote 4). These protocols can disagree: a model may rank the correct verb higher under conditional probability yet lose when total sentence probability includes the period and downstream context, or vice versa. Verb frequency also affects agreement probabilities and interacts with the amount of training data. The seven-of-ten underperformance reported in Figures 3-5 may therefore be an artifact of metric or stimulus mismatch. The LSTM scaling results themselves are credible, but the generalization to Transformers in the title and abstract is not directly supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains 125 two-layer LSTM language models, varying hidden size (100, 200, 400, 800, 1600 units) and training corpus size (2M, 10M, 20M, 40M, 80M tokens from WikiText-103), and evaluates them on the Marvin-Linzen agreement challenge set with high-frequency verb pairs. It reports that agreement accuracy improves with model size up to about 400 hidden units and then saturates; that increasing training data yields inconsistent and diminishing improvements; and that extrapolating from observed error reductions suggests that reaching human-level agreement accuracy would require on the order of 10 billion tokens or more. Using results from Wolf (2019), it also compares GPT and BERT, finding that they perform worse than the LSTM models in seven of ten subject-verb agreement conditions, and concludes that reliable and data-efficient syntactic learning likely requires external supervision or a stronger inductive bias than that provided by RNNs and Transformers.","tokens_in":10631,"tokens_out":9896,"duration_ms":135781,"significance":"If the LSTM scaling results hold, the paper is a valuable negative result: it provides a systematic 125-model grid with five random seeds, Bayes-factor analyses, and evaluation on an independent, externally published challenge set, and the saturation pattern and inconsistent data improvements are credible. The data-requirement extrapolation is transparently stated and the qualitative conclusion that the required data would be unrealistic is robust across a wide range of assumed improvement rates. However, the Transformer comparison is not established by the paper's own experiments because it relies on a different scoring protocol and stimulus set, and this weakens the title-, abstract-, and conclusion-level claims about Transformers. The LSTM-specific findings remain a solid contribution.","major_comments":[{"comment":"The headline comparative claim that GPT and BERT 'perform even more poorly than our LSTMs in some constructions' (Abstract) is not established by the experiments in this paper. The GPT/BERT numbers are taken from Wolf (2019), who scored models by the rank of the two target verbs given the prefix, whereas the LSTM models here are scored by total sentence log-probability, and Wolf's stimuli were not restricted to the high-frequency verbs used in Section 3. These protocol differences can change accuracy: the correct verb can rank higher conditionally while the total sentence probability favors the ungrammatical continuation, and verb frequency interacts with training-data size. The seven-of-ten underperformance reported in Figures 3-5 therefore rests on an unmatched comparison. The authors should either run GPT and BERT under the LSTM scoring protocol on the same high-frequency-verb stimuli, or explicitly restrict the abstract and conclusion to LSTM models and reframe the title accordingly.","section":"Section 2, footnote 4; Section 4, 'Transformers'"},{"comment":"The extrapolated data requirements are presented with point estimates such as 'over 10 billion tokens,' but Tables 3 and 4 show that the estimates vary by many orders of magnitude depending on which data-doubling rate is assumed, and Table 2 reports Bayes factors below 1 for several individual doublings, indicating no measurable improvement. The authors should state explicitly that these are idealized lower-bound projections under a constant-error-reduction assumption, not predictions, and should note that exact token counts (for example, the 317-token estimate in Table 4) are not meaningful. The qualitative conclusion that the data requirements would be unrealistic may survive this caveat, but the current wording overstates the precision of the estimates.","section":"Section 5, Tables 3-4"}],"minor_comments":[{"comment":"The methodological limitation that the Transformer comparison uses a different scoring protocol and different stimuli should be promoted from a footnote to the main text, since it qualifies a result advertised in the abstract.","section":"Section 2, footnote 4"},{"comment":"Please clarify that using the largest observed error reduction (20M to 40M tokens) makes the resulting data requirements a lower bound rather than a central estimate; the word 'conservative' is currently ambiguous.","section":"Section 5"},{"comment":"The caption of Figure 2 should explain more explicitly that the lines are projections and the bars are actual training-token counts; the logarithmic scale with values up to 10^96 makes it difficult to read individual estimates.","section":"Figure 2"},{"comment":"The paper does not mention whether the trained models or evaluation code will be released; given the 125-model grid and the importance of reproducing the scaling curves, a reproducibility statement would be helpful.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The LSTM scaling results are credible and the paper makes a useful contribution, but the unsupported Transformer comparison is a load-bearing weakness because the title, abstract, and conclusion generalize to Transformers. A matched re-evaluation of GPT and BERT, or a substantial softening and rescoping of the claims, is needed before publication. I do not see concerns about citation practice or novelty; the self-citation to Marvin and Linzen is appropriate, and the paper's scope fits the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid, careful empirical study of how LSTM size and training corpus size affect syntactic agreement accuracy, and its main finding—that gains saturate quickly—holds up. But the paper's reach exceeds its grasp when it extends that conclusion to Transformers, because the GPT/BERT numbers are borrowed from a different evaluation protocol.\n\nWhat's actually new: 125 LSTM models, crossing five hidden sizes, five corpus sizes, and five random seeds, evaluated on the Marvin-Linzen challenge set with high-frequency verbs substituted in. That's a systematic sweep nobody had done at this scale for targeted syntax evaluation. The Bayes factor analysis is appropriate, and the error bars from the five seeds support the saturation pattern: beyond 400 hidden units, no real gains; corpus size gives inconsistent, diminishing improvements. The extrapolation to trillions of tokens is clearly flagged as an assumption and is a reasonable way to make the \"unrealistically large\" intuition concrete. The authors also deserve credit for reporting the sensitivity of those projections in the appendix.\n\nThe soft spots are real but contained. The Transformer comparison is the weakest link. The GPT and BERT accuracies come from Wolf (2019), who scored by the rank of the two target verbs given the prefix, while the LSTMs are scored by total sentence log-probability, and Wolf didn't restrict the stimuli to high-frequency verbs. Those protocol differences can change accuracy rankings. So the claim in the abstract that these models \"perform even more poorly than our LSTMs in some constructions\" is not directly supported by the experiments in this paper. It may be true, but it needs a matched re-evaluation or the language needs to be softened to \"reported by previous work.\" The extrapolation also relies on a constant error reduction rate taken from one doubling (20M to 40M); the appendix tables show the projections vary hugely depending on which doubling you use, and the main text should make that variability more prominent. Minor point: the larger corpus subsets can't be fully disjoint given WikiText-103's size, but that's an unavoidable constraint, not a flaw.\n\nWho gets value: anyone working on scaling behavior of neural LMs or targeted syntactic evaluation. The LSTM scaling results are a useful baseline, and the methodological caution about matching evaluation protocols is worth remembering. I'd cite it if I worked in that area.\n\nRecommendation: engage with it. It deserves a serious referee. The core scaling result is credible and new; the comparative claim needs either a matched experiment or careful hedging. That's a revision, not a rejection.","headline":"Systematic LSTM scaling study with a credible saturation result and an honest extrapolation; the Transformer comparison is borrowed and under-matched, so the paper's reach slightly exceeds its grasp.","tokens_in":11213,"tokens_out":2247,"would_cite":true,"duration_ms":21202,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that scaling LSTM language models in size and data yields diminishing returns for syntactic agreement, and that reaching human-level syntax may require impractically large corpora unless models get structural supervision…","keywords":["neural language models","syntactic agreement","LSTM","scaling","data efficiency","transformer language models","targeted syntactic evaluation","Bayes factors"],"falsifier":"Score the LSTM models on the verb-rank metric used for GPT and BERT on the identical high-frequency-verb stimuli; if the LSTM advantage vanishes, the paper's cross-architecture conclusion is not supported.","tokens_in":10231,"feed_emoji":"📉","tokens_out":5869,"duration_ms":50221,"temperature":0.7,"pith_summary":"The paper asks whether simply making neural language models bigger and training them on more text can fix their well-documented failures on grammatical agreement. It trains 125 LSTM models spanning five hidden-layer sizes and five corpus sizes, tests them on a targeted agreement challenge set, and finds performance saturates beyond 400 hidden units and improves inconsistently as data grows from 2 million to 80 million words. Extrapolating the best observed improvement rate, every construction still below human level would require more than 10 billion tokens, and most would need trillions for near-perfect accuracy. GPT and BERT, Transformer models trained on billions of words, do not consistently beat these LSTMs. The conclusion is that reliable, data-efficient syntax learning will likely require external supervision or a stronger inductive bias than current architectures provide.","feed_headline":"Scaling up language models won't buy grammatical syntax","feed_subtitle":"Tests of 125 LSTM models show agreement accuracy saturates; matching humans would take trillions of tokens.","key_machinery":"The machinery is a controlled scaling grid: 125 two-layer LSTM language models, crossing hidden sizes of 100 to 1600 units with training corpora of 2 million to 80 million tokens drawn from WikiText-103, all evaluated on the same targeted syntactic agreement challenge set. Bayes factors quantify the evidence that a given increase in size or data improves accuracy, and the data-requirement estimates extrapolate the 20-million-to-40-million-token error-reduction rate under a constant-error-reduction assumption. This design lets the authors separate the effects of capacity and data on a specific structural ability rather than on average perplexity.","core_discovery":"The paper's central claim is that quantity—in parameters and tokens—does not buy quality in syntactic generalization. Across 125 LSTM language models, agreement accuracy plateaus once hidden layers reach 400 units, and each doubling of training data beyond that point gives uneven, often statistically undetectable gains. Under a conservative assumption that doubling data keeps reducing errors at the best observed rate, human-level accuracy on the hardest remaining constructions would need more than 10 billion tokens, and 99.99 percent accuracy would need trillions—an unrealistic amount for simple agreement phenomena. The paper further reports that GPT and BERT, trained on one to three billion words, perform more poorly than the trained LSTMs on several of the ten agreement conditions, supporting the conclusion that the bottleneck is not scale but architecture or supervision.","pith_inferences":["If the saturation pattern holds beyond this grid, then scaling curves for syntactic structure may be governed by an irreducible error floor, so scaling laws fitted to aggregate perplexity could mask construction-specific ceilings.","A testable extension would be to train on the same grid but with a syntax-augmented objective, such as auxiliary agreement prediction; the paper's account predicts such supervision should shift the saturation point upward at far lower data cost.","The asymmetry between short and long coordinated verb phrases suggests challenge sets should systematically vary constituent length, since models that generalize to longer structures need not generalize to shorter ones.","The cross-architecture comparison could be made exact by re-scoring LSTM outputs under the verb-rank metric used for GPT and BERT; the paper flags the procedural mismatch but does not itself perform the matched comparison."],"forward_implications":["Agreement accuracy will not improve meaningfully by training LSTM language models beyond roughly 400 hidden units.","Doubling training data beyond tens of millions of tokens yields inconsistent gains; several constructions show no measurable improvement.","Matching human-level agreement in the remaining hard constructions would require corpora on the order of 10 billion tokens or more, and near-perfect accuracy would require trillions.","Transformer models trained on orders of magnitude more data fail to consistently outperform LSTMs, so scale alone is not a fix.","Remaining deficits call for external syntactic supervision or architectures with a stronger inductive bias."],"supporting_citations":[{"why":"Supplies the challenge set of syntactic agreement constructions that all models are tested on.","marker":"(Marvin and Linzen, 2018)"},{"why":"Provides the GRNN, a publicly available two-layer 650-unit LSTM baseline whose results anchor the comparison.","marker":"(Gulordava et al., 2018)"},{"why":"Source of the GPT and BERT agreement scores used for the cross-architecture comparison.","marker":"(Wolf, 2019)"},{"why":"Establishes the minimal-pair agreement evaluation paradigm this paper extends.","marker":"(Linzen et al., 2016)"},{"why":"Supplies the WikiText-103 corpus from which the five training subsets at each size are drawn.","marker":"(Merity et al., 2016)"},{"why":"Defines the GloVe vocabulary intersected with the GRNN vocabulary to fix a shared 28,438-word vocabulary.","marker":"(Pennington et al., 2014)"},{"why":"Provides the GPT Transformer model compared against the LSTMs.","marker":"(Radford et al., 2018)"},{"why":"Provides the BERT model compared, with its differently trained objective discussed.","marker":"(Devlin et al., 2019)"}],"fun_headline_variants":["Syntax gains stall as language models scale","Trillions of tokens won't buy grammar","Bigger isn't better for neural syntax","GPT, BERT trail LSTMs on syntax tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim that Transformer models fare worse than LSTMs depends on an external evaluation that scored GPT and BERT by the rank of the two target verbs given the prefix rather than full-sentence log-probability and did not restrict stimuli to high-frequency verbs; if that procedural mismatch changes which model wins, the cross-architecture conclusion collapses.","fun_headline_variants_meta":{"raw":{"variants":["Syntax gains stall as language models scale","Trillions of tokens won't buy grammar","Bigger isn't better for neural syntax","GPT, BERT trail LSTMs on syntax tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1324,"prompt_tokens":823,"completion_tokens":501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":439,"tokens_out":501,"duration_ms":30682,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T06:00:53.905154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the LSTM models on the verb-rank metric used for GPT and BERT on the identical high-frequency-verb stimuli; if the LSTM advantage vanishes, the paper's cross-architecture conclusion is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the challenge set of syntactic agreement constructions that all models are tested on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GRNN, a publicly available two-layer 650-unit LSTM baseline whose results anchor the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the GPT and BERT agreement scores used for the cross-architecture comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the minimal-pair agreement evaluation paradigm this paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WikiText-103 corpus from which the five training subsets at each size are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GloVe vocabulary intersected with the GRNN vocabulary to fix a shared 28,438-word vocabulary."}],"review_version":1}