{"id":"e0293438-d8c9-4be2-a64b-7d20929f539e","arxiv_id":"2506.01172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The inverse scaling of language model surprisal with reading-time fit is not explained by training-data leakage, per overlap analysis and leakage-free retraining.","lead":"This paper asks whether big language models fit human reading times worse because they have memorized the test texts during training. It finds little exact overlap between reading-time corpora and model training data, and shows the same \"inverse scaling\" trend appears even when models are trained on filtered, leakage-free data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conclusion overreaches: leakage is measured and filtered only as exact token n-gram overlap, so paraphrastic or lightly edited leakage remains untested and could still explain the inverse scaling.","rationale":"I agree with the reader's weakest-assumption identification. The study is well-executed for exact overlap and the public release of overlap data is a useful resource, but the conclusion's scope is broader than the measurement. The concrete retraining test would determine whether the filtering criterion is sufficient. The reader's CONDITIONAL verdict remains appropriate: the concern is addressable but currently unresolved.","tokens_in":9110,"tokens_out":4361,"duration_ms":53223,"concrete_test":"Run Study 2's data curation again with a fuzzy near-duplicate detector (e.g., MinHash LSH on word 5-shingles with a Jaccard threshold around 0.8, or SimHash) to compare every reading-time passage against the Pile, and identify additional 'leakage-free' chunks after removing all near-duplicate chunks. Then retrain the three LMs on the filtered data and recompute the delta-log-likelihood curves in Figure 2. If the negative relationship between model size and surprisal fit weakens or disappears, exact n-gram overlap was too narrow a definition of leakage; if the relationship is unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a strong negative: previous inverse-scaling results are 'not driven by the effects of data leakage.' Both studies operationalize leakage exclusively as exact token n-gram overlap. Study 1 measures only exact overlaps; Study 2 defines leakage-free training data by excluding chunks with more than 11 continuous overlapping tokens. This definition cannot detect near-duplicate documents, lightly edited or paraphrased text, or semantic re-use, all of which are known to occur in web-scale corpora and can affect model predictions even when no long exact n-gram is present. The paper's own Limitations section concedes that 'data leakage in this work is diagnosed mainly through token n-gram overlaps, which is insensitive to minor variations in form,' but the abstract and conclusion present the finding without that caveat. The artificial-leakage manipulation also only injects exact copies of the reading-time texts, so it estimates the effect of verbatim duplication, not of the paraphrase leakage that the methodology cannot see. As a result, the evidence does not fully support the claim that leakage, broadly construed, does not drive the effect; it supports only the narrower claim about exact-sequence leakage. Since the conclusion is used to vindicate previous psycholinguistic modeling results, the mismatch between the narrow operationalization and the broad conclusion is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether the inverse scaling effect—larger pre-trained language models providing poorer surprisal fits to human reading times—is an artifact of data leakage. Study 1 uses CDAWGs to measure the longest exact token n-gram overlap between five reading-time corpora and two pre-training datasets (Pile and OpenWebText), finding relatively little overlap in most passages. Study 2 trains Pythia-style transformer LMs of three sizes on Pile chunks filtered to have minimal exact overlap with the reading-time corpora, and reports that the negative relationship between model size and surprisal fit is replicated on all five datasets; additional fine-tuning on the reading-time texts shows that severe exact leakage would inflate this negative relationship. The authors conclude that prior inverse-scaling results using Pythia and GPT-2 are not driven by data leakage.","tokens_in":9289,"tokens_out":5171,"duration_ms":61831,"significance":"If the conclusion is restricted to exact-sequence leakage, the paper makes a useful contribution: it provides a scalable method for quantifying a specific form of leakage, releases the overlapping sequences, and supplies a causal control experiment (leakage-free training plus artificial fine-tuning) that is a clear improvement over purely observational overlap checks. The convergence of five datasets and the direction of the fine-tuning manipulation are persuasive. The main limitation is that the paper measures and manipulates only exact token n-gram overlap, so the broad title-level claim about 'data leakage' is stronger than the evidence supports. The re-trained models are also small relative to the Pythia/GPT-2 models discussed, and the ΔLogLik comparisons lack uncertainty quantification.","major_comments":[{"comment":"The central claim is broader than the operationalization. The abstract and conclusion state that previous results are 'not driven by the effects of data leakage,' but Study 1 measures only exact token n-gram overlap and Study 2 defines leakage-free data by removing exact overlaps of more than 11 continuous tokens. The artificial-leakage manipulation in Section 3.1 also injects only exact copies of the reading-time texts. The Limitations section explicitly concedes that the diagnosis 'is insensitive to minor variations in form,' so the evidence supports the narrower conclusion 'not driven by exact token-sequence leakage.' The title and abstract should be revised to reflect this scope, otherwise the headline claim is unsupported.","section":"Abstract and Conclusion"},{"comment":"The central result—the negative relationship between model size and ΔLogLik—is presented as point estimates with no confidence intervals or significance tests. Figure 2 shows trends across five corpora and three model sizes, but the paper does not quantify uncertainty, and Appendix B states that the remaining ~25% of the data was reserved for significance testing and 'not used in this work.' Given that the claim is about a monotonic decrease in fit, the authors should provide at least bootstrap confidence intervals or a permutation test across subjects or items, or explain why the consistent pattern is sufficient.","section":"Section 3.2 and Appendix B"},{"comment":"The re-trained models are 28M, 70M, and 162M parameters, which are at the small end of or smaller than the Pythia and GPT-2 models used in the prior inverse-scaling results cited. The paper claims in the Conclusion that the results 'generalize to trends observed from larger pre-trained LMs like Pythia and GPT-2,' but no model above 162M is studied. The authors should either temper this generalization or provide evidence that the inverse scaling trend is stable in this small-model range and that extrapolation to larger sizes is justified.","section":"Section 3.1, Table 1"}],"minor_comments":[{"comment":"The statement that chunks with 'no more than 11 continuous tokens of overlap' were selected 'which excludes all overlaps improbable enough to meet our threshold in Figure 1a' is unclear; the relationship between the 11-token cutoff and the chance-level threshold of Figure 1 should be explained explicitly.","section":"Section 3.1"},{"comment":"The red pluses denoting the chance-level threshold in Figure 1b are described in the text but the caption would benefit from a self-contained definition of the threshold, including the log-probability values and the 0.05 criterion.","section":"Figure 1"},{"comment":"The description of the Provo corpus overlaps as 'potentially concerning' before noting their low frequencies is slightly confusing; the two observations should be separated clearly so that the reader can assess the frequency evidence without re-reading.","section":"Section 2.2"},{"comment":"There are a few typographical artifacts in the rendering of author names and references (e.g., 'Ku ˇcera'), likely from the PDF extraction, but these are not substantive.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a clever fine-tuning control, but the abstract and title overstate the scope of the leakage definition. The lack of uncertainty quantification on the main ΔLogLik comparisons is also a concern for a journal-level claim. I would like to see the language narrowed to 'exact token-sequence leakage' and either confidence intervals or a clear justification for why they are omitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about whether pre-trained LM surprisal effects in psycholinguistics survive the leakage critique. The paper earns serious attention.\n\nThe novel part is Study 1: a CDAWG-based overlap audit of five reading-time corpora against the full Pythia training set (~300B tokens) and OpenWebText. This is a real computational achievement, and the result — short, low-frequency exact overlaps in almost every passage — directly addresses the concern for the actual Pythia/GPT-2 models used in prior work. They also release the overlap data, which is a practical resource. Study 2 is a reasonable causal follow-up: train small Pythia-like models on Pile chunks with fewer than 12 consecutive overlapping tokens, replicate the inverse scaling in surprisal fit, then fine-tune on the reading-time texts and show that artificial leakage inflates the effect. That fine-tuning control is a good internal diagnostic: leakage makes the inverse slope steeper rather than explaining it away.\n\nThe main soft spot is one the authors themselves flag in the Limitations section: leakage is operationalized as exact token n-gram overlap. The abstract and conclusion say 'not driven by the effects of data leakage,' but the method cannot detect paraphrases, near-duplicates, or lightly edited texts, which are known to exist in web corpora. The stress-test criticism on this point is fair and load-bearing if you read the claim broadly; the evidence supports 'not driven by exact-sequence leakage.' The authors should either soften the conclusion or add a sentence-level similarity audit. Minor issues: Study 2's models are 28M-162M parameters, well below the Pythia/GPT-2 sizes whose trend is at issue — although Study 1 covers the large models directly, so this is a gap in the causal part, not in the measurement. The delta-log-likelihood comparisons in Figure 2 have no confidence intervals or significance tests; the pattern is consistent across five datasets, so this is a presentation problem more than a fatal one. A matched unfiltered training control would also strengthen the causal claim, since the comparison is against the original Pythia models rather than a same-corpus baseline.\n\nOverall, this is a well-executed empirical study with a useful public resource. The central point — that exact-sequence leakage is unlikely to explain inverse scaling — holds; the broader 'data leakage' claim overreaches but is fixable in revision. I would send it for peer review and recommend the authors tighten the claim and add uncertainty quantification.","headline":"Solid, scale-appropriate test of the leakage hypothesis, but the abstract's 'data leakage' overstates what is measured: exact n-gram overlap only.","tokens_in":9879,"tokens_out":3213,"would_cite":true,"duration_ms":35353,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the inverse scaling effect of pre-trained language model surprisal is a real phenomenon, not an artifact of data leakage.","keywords":["inverse scaling","surprisal","reading times","data leakage","pretrained language models","psycholinguistics","CDAWG","n-gram overlap"],"falsifier":"Look for a reading-time passage, especially from Provo or Natural Stories, whose near-duplicate (paraphrase or lightly edited form) occurs in the Pile or OpenWebText, and show that a large model trained on that data gives that passage anomalously low surprisal. Alternatively, deduplicate the pretraining corpora against the reading-time corpora using fuzzy match measures and observe whether the inverse scaling slope between model size and surprisal fit collapses to zero; if the slope disappears entirely, the paper's central claim would be falsified.","tokens_in":8850,"feed_emoji":"🧠","tokens_out":6914,"duration_ms":56274,"temperature":0.7,"pith_summary":"The paper defends the inverse scaling effect of pre-trained language model surprisal: the finding that larger models' next-word probabilities fit human reading times worse than smaller models' do. The worry had been that this trend is a data-leakage artifact, because the reading-time texts are publicly available and may appear inside pretraining corpora. The paper tests that worry directly: first by measuring exact token overlaps between five reading-time corpora and two large pretraining datasets, finding the overlaps short and rare; then by training Pythia-style models on a filtered, leakage-free subset and replicating the inverse scaling. It concludes that the trend is not an artifact of memorization, though severe leakage would make the inverse relationship look stronger than it is.","feed_headline":"Inverse scaling of LM surprisal survives leakage-free training","feed_subtitle":"Reading-time corpora barely overlap pretraining data, and models trained without that overlap keep the same trend.","key_machinery":"The machinery is a pair of leakage-detection tools. First, a CDAWG (compact directed acyclic word graph) indexes each pretraining corpus, letting any reading-time passage be queried to find the longest contiguous subword-token sequence it shares and how often that sequence occurs in the pretraining data. Second, the same index is used as a filter to construct leakage-free training data: Pile chunks sharing 11 or fewer token overlaps with the reading-time corpora are selected, and three Pythia-like transformers are trained on that data. Fit is then quantified as the log-likelihood gain of linear mixed-effects reading-time regressions when surprisal is added to a baseline model.","core_discovery":"The central claim is that previously reported inverse scaling of surprisal's fit to reading times is not caused by data leakage. The authors build compact directed acyclic word graphs (CDAWGs) over the Pythia training subset of the Pile and over OpenWebText, and query them with the passages of the Dundee, Brown, GECO, Provo, and Natural Stories corpora. They report that almost no passage appears entirely in either pretraining corpus, and the longest overlapping token sequences are typically short and occur with low frequency. They then retrain Pythia-like transformers of three sizes on Pile chunks that contain no 12-token-or-longer overlap with the reading-time passages, and the negative relationship between model size and held-out log-likelihood gain from linear mixed-effects regression appears on all five datasets. Artificially fine-tuning these models on the reading-time corpora makes the negative relationship steeper, which the authors interpret as showing that genuine severe leakage would inflate, not create, the inverse scaling.","pith_inferences":["The authors' operationalization of leakage as exact n-gram overlap leaves open the possibility of paraphrase-level leakage; a natural next experiment would be to deduplicate pretraining data with fuzzy near-duplicate detection (e.g., MinHash on embedded documents) and check whether the inverse scaling slope changes.","Since fine-tuning on reading-time data overestimates the negative relationship, the paper implies that original inverse-scaling slopes measured on leaky corpora may be inflated; one could predict that fully deduplicated web-scale models show the same trend but with a shallower slope.","The ≤11-token-overlap filtering threshold is a reusable, principled criterion for building leakage-free evaluation sets for psycholinguistics, and could generalize to other naturalistic stimulus materials such as speech transcripts or self-paced reading items."],"forward_implications":["Researchers can continue using Pythia- and GPT-2-style pretrained models for psycholinguistic surprisal studies without attributing the inverse scaling trend to exact-text memorization.","The released overlap annotations give a concrete resource for screening future reading-time corpora and pretraining data.","Because severe leakage is shown to steepen the inverse relationship, corpora that do contain long exact overlaps should be handled with extra caution in surprisal-fitting studies.","The finding supports the view that larger models' worse reading-time fit reflects learned distributional associations rather than verbatim memory."],"supporting_citations":[{"why":"Supplies the CDAWG construction and query implementation used to find longest overlapping token sequences.","marker":"Merrill et al. (2024)"},{"why":"Defines the Pythia training data and model recipe that Study 2 imitates with smaller models.","marker":"Biderman et al. (2023)"},{"why":"Provides the Pile, the pretraining corpus analyzed in Study 1 and the source of leakage-free chunks in Study 2.","marker":"Gao et al. (2020)"},{"why":"Provides OpenWebText, the second pretraining corpus whose overlap with reading-time corpora is measured.","marker":"Gokaslan and Cohen (2019)"},{"why":"Reports the inverse scaling effect that motivates the leakage concern and is replicated under leakage-free training.","marker":"Oh and Schuler (2023b)"},{"why":"Provides large-scale reading-time evidence that also exhibits the inverse scaling trend this paper defends.","marker":"Shain et al. (2024)"},{"why":"Raises the data-leakage explanation for inverse scaling that the two studies directly test and reject.","marker":"Wilcox et al. (2023a)"},{"why":"The Natural Stories corpus, one of the five reading-time datasets used for overlap queries and fine-tuning.","marker":"Futrell et al. (2021)"},{"why":"The Provo corpus, whose passages show the longest overlaps in Study 1 and thus the strongest leakage signal examined.","marker":"Luke and Christianson (2018)"}],"fun_headline_variants":["No leak: bigger LM surprisal still worsens reading-time fit","Leakage-free models still show inverse surprisal scaling","Data leakage doesn't explain inverse scaling of surprisal","Inverse surprisal scaling persists when leaks are removed","Inverse scaling of surprisal is not leak-driven"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that data leakage can be fully detected as exact repeated sequences of subword tokens; if leakage arrives through paraphrases, reordered sentences, or light edits, the leakage-free training data may still contain the reading-time texts and the conclusion would not follow.","fun_headline_variants_meta":{"raw":{"variants":["No leak: bigger LM surprisal still worsens reading-time fit","Leakage-free models still show inverse surprisal scaling","Data leakage doesn't explain inverse scaling of surprisal","Inverse surprisal scaling persists when leaks are removed","Inverse scaling of surprisal is not leak-driven"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001379,"raw_usage":{"total_tokens":5563,"prompt_tokens":900,"completion_tokens":4663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":4578}},"tokens_in":516,"tokens_out":4663,"duration_ms":31918,"temperature":1.0,"reasoning_tokens":4578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:50:16.492884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look for a reading-time passage, especially from Provo or Natural Stories, whose near-duplicate (paraphrase or lightly edited form) occurs in the Pile or OpenWebText, and show that a large model trained on that data gives that passage anomalously low surprisal. Alternatively, deduplicate the pretraining corpora against the reading-time corpora using fuzzy match measures and observe whether the inverse scaling slope between model size and surprisal fit collapses to zero; if the slope disappears entirely, the paper's central claim would be falsified.","supporting_citations":[{"cited_title":"Smith, and Yanai Elazar","cited_arxiv_id":null,"evidence_quote":"Supplies the CDAWG construction and query implementation used to find longest overlapping token sequences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Pythia training data and model recipe that Study 2 imitates with smaller models."},{"cited_title":"Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko","cited_arxiv_id":null,"evidence_quote":"The Natural Stories corpus, one of the five reading-time datasets used for overlap queries and fine-tuning."}],"review_version":1}