Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Natural Context Drift Undermines the Natural Language Understanding of Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM accuracy falls as Wikipedia passages drift from the text seen in training, even when questions stay answerable.

desk verdict Real robustness finding with an unvalidated independent variable; worth serious refereeing, but the similarity proxy needs a model-internal check. read the letter →

arxiv 2509.01093 v1 pith:KZ4GH3F3 submitted 2025-09-01 cs.CL cs.AI

Yulong Wu , Viktor Schlegel , Riza Batista-Navarro This is my paper
classification cs.CLcs.AI
keywords naturallanguageunderstandingquestionansweringcontextdriftsemanticsimilarityreadingcomprehensionWikipediarevisionhistoryLLMrobustnessbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what happens when the reading passages given to a question-answering LLM are not the passages it saw during pretraining but human-edited later versions of the same Wikipedia articles. The authors build a framework that digs each passage out of Wikipedia's revision history, measures its maximum semantic similarity to same-title Wikipedia text in the model's own training corpus, and tracks accuracy across similarity bins. Across six QA datasets and eight LLMs with publicly available training data, accuracy declines as passages drift: for example, BoolQ accuracy drops from about 71% to 12.5% between the most and least similar bins, and the aggregate regression slope is 65.78 ± 41.55. Human readers do not show the same decline, so the edited passages remain answerable. The paper's conclusion is that natural context evolution is a genuine and measurable challenge to LLM reading comprehension, not a benchmark artifact.

What carries the argument

The load-bearing object is a semantic similarity score: for each edited passage, the maximum similarity between that passage and any same-title Wikipedia content in the LLM's training corpus, computed with a sentence-embedding model (all-MiniLM-L6-v2, with three alternative embeddings tested as ablations). The authors group these scores into ten bins, plot average accuracy per bin, and fit a linear slope. Two filters make the comparison clean: answer-preservation checking ensures edited passages still contain the answer, and a context-free question pass removes instances the model can already answer from parametric memory. The natural-perturbation pipeline producing the edited variants is ad

What would settle it

Take the edited BoolQ passages in the lowest similarity bin that models answer incorrectly, have human editors rewrite them so they become semantically similar to the pretraining Wikipedia version without changing facts or answerability, and check whether model accuracy recovers to the high-bin level. If accuracy stays low, the decline is not caused by semantic distance to pretraining text; if it recovers, the similarity proxy is what carries the effect.

Watch

Extended reading notes

Core claim

The central claim is that a generative LLM's ability to answer questions from a passage degrades as the passage moves, through ordinary human editing rather than adversarial perturbation, away from the version of that article the model saw in pretraining. This holds even though the question and all necessary information remain present in the edited passage, and despite differences in model family, size, training corpus, and instruction tuning. The effect is specific to LLMs: human annotators' accuracy stays roughly flat across the same similarity bins. Aggregated across tasks and models, the mean slope is 65.78 ± 41.55 and the mean Pearson correlation is 0.684 ± 0.291; BoolQ and SQuAD 2.0 sh

Load-bearing premise

The result depends on the idea that the highest semantic similarity between an edited passage and same-title Wikipedia content in the model's training corpus really measures how close the passage is to what the model saw during pretraining; if edited passages resemble other, non-Wikipedia training text, the similarity bins misclassify passages and the apparent drift effect could be an artifact of the embedding model.

Editorial extensions

If this is right

  • Static QA benchmarks drawn from a single Wikipedia snapshot will overstate how well LLMs read in the wild, where articles are continually edited.
  • Robustness evaluations should include drifted or temporally shifted splits, ordering passages by their distance to training text rather than only by random noise or paraphrasing.
  • Surface-form-dependent tasks such as extractive QA and yes/no decisions are the most vulnerable, while reasoning-heavy tasks are less susceptible, suggesting different failure modes.
  • Instruction tuning and chain-of-thought prompting do not remove the degradation, so fixes will need training-time exposure to evolving text or explicit invariance to harmless rewording.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The similarity proxy only compares edited passages with the Wikipedia subset of training data; a natural extension is to search the full training corpus for near-duplicates, which would tell whether low similarity really means far from all pretraining exposure.
  • Because the effect is weakest on multi-hop reasoning, the phenomenon may reflect shallow lexical or surface matching rather than a general loss of comprehension; a controlled paraphrase test that varies wording while holding content fixed could isolate that mechanism.
  • This suggests a deployable diagnostic: measure the similarity of incoming documents to a model's known training distribution and flag low-similarity contexts as high-risk for accuracy degradation before answering.
  • The framework could transfer to other evolving corpora with revision logs, such as news article archives, code documentation, or product manuals, where similar drift curves might appear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a framework for studying how natural, human-driven evolution of Wikipedia passages affects question answering in LLMs. It extracts edited versions of reading passages from six QA benchmarks using Wikipedia revision histories, measures each edited passage's maximum semantic similarity (via sentence embeddings) to same-title Wikipedia content in the model's publicly available pretraining corpus, bins passages by this similarity score, and correlates bin membership with LLM accuracy. The authors evaluate eight open instruction-tuned LLMs (plus base variants) and report a general decline in accuracy as similarity decreases, with a mean slope of 65.78 and mean Pearson correlation of 0.684. They include controls: filtering out instances the model can answer without the passage, human-annotation experiments showing stable human performance across bins, an infini-gram check for verbatim leakage, and robustness experiments with alternative embedding models, base (non-instruction-tuned) models, and chain-of-thought prompting. The central conclusion is that performance degrades as context passages diverge from pretraining content even when the question and all necessary information are present.

Significance. If the central claim holds, this is a valuable empirical contribution: it moves beyond synthetic perturbations to real human edits and connects a concretely observable phenomenon—Wikipedia text evolution—to LLM robustness and training-data proximity. The study is unusually careful in several respects: it uses transparent models with known pretraining corpora, evaluates multiple datasets and model families, includes human annotation to rule out unanswerability, and reports leakage checks. The accuracy trend is measured rather than derived from the similarity scores, so the finding is not forced by construction. However, the independent variable—the maximum semantic similarity to same-title Wikipedia content—is not validated as a model-internal measure of pretraining exposure. Since the entire monotonic trend is defined over this proxy, this is the main threat to the paper's interpretation. The framework is reusable and the paper is worth publishing after the proxy concern is addressed with additional evidence or explicit reframing of the claims.

major comments (3)
  1. [§2 (Semantic similarity-LLM average accuracy correlation analysis)] The independent variable is the maximum all-MiniLM-L6-v2 similarity between an edited passage and any same-title Wikipedia content in the training corpus, and the abstract interprets this as 'how closely each variant aligns with content seen during pretraining.' This is load-bearing, yet the proxy is never validated against a model-internal signal of familiarity. The infini-gram check in §4/Table 1 rules out only verbatim inclusion; it does not address semantically near-duplicate content in non-Wikipedia web text or later Wikipedia snapshots. Low-similarity bins are also likely to contain larger or more complex edits, so the observed accuracy drop could reflect edit magnitude/text difficulty rather than distance from pretraining per se. The three alternative embedding models in Appendix E share the same sentence-embedding training objective and therefore do not break this confound. Pleas
  2. [§4 (slope analysis and Figure 5)] The aggregate claim that the relationship is 'consistent and statistically grounded' is not supported by the reported statistics. The mean slope is 65.78 ± 41.55 and the mean Pearson correlation is 0.684 ± 0.291, but the per-model/per-dataset values vary widely and include negative or near-zero slopes (e.g., OLMoE-1B-7B-0125-Instruct on D(RoBERTa): slope = -18.90, Figure 5(c); OLMoE on WikiWhy: slope = 4.71, Figure 5(e)). The paper does not report confidence intervals, p-values, or per-bin sample sizes for these slopes, and the regression appears to be fit on only ten bin means, some of which may have very few instances after filtering. Please provide per-dataset/per-model significance tests (e.g., permutation tests) and a clear statement of how many of the fitted slopes are significantly positive.
  3. [§3 (WikiWhy correctness; Appendix B/Table 3)] Two analysis choices are not justified with robustness evidence. First, the WikiWhy correctness threshold of 0.6 on all-MiniLM-L6-v2 similarity between the model response and the ground-truth explanation is hand-chosen; no sensitivity analysis is given. Second, the context-free-answerability filter removes up to 61.40% of BoolQ instances for OLMo-2-1124-13B-Instruct (Table 3). Filtering is a sensible control, but the paper does not report how the per-bin sample sizes or the monotonic trend change as a result of this filtering. These choices could affect the magnitude and even the direction of the reported trends, so they should be tested explicitly.
minor comments (6)
  1. [Abstract] Typo: 'BOOL Q' should be 'BoolQ'.
  2. [§4 / Figure 4 / Appendix H] Human annotation results are shown without error bars, per-bin sample sizes, or inter-annotator agreement. Since the human control is used to argue that edited passages remain answerable, these details should be reported.
  3. [Appendix A / Table 2] Table 2's column header 'A vg. per Passages' appears to be a formatting error; also the table layout makes it hard to distinguish original passage counts from edited passage counts. Please clarify the column definitions.
  4. [§3 / Figure 3] For the non-OLMo models, only three datasets (SQuAD 2.0, WikiWhy, HotpotQA) are reported, while the abstract and §4 imply coverage of all six datasets across all models. Please state explicitly which models were evaluated on which datasets and why.
  5. [§4 (leakage discussion)] The infini-gram percentages in Table 1 are small but not uniformly negligible (up to 8.69% for SQuAD 2.0 on OLMo-2/OLMoE), and several cells are missing (e.g., D(RoBERTa) for dolly, HotpotQA for all models). The statement that the impact 'may not be significant' should be substantiated with a clearer threshold or additional checks.
  6. [Appendix D / Figure 5] The slope analysis plots linear regressions over bin midpoints but does not show the underlying data points or per-bin confidence intervals. Adding these would help assess the reliability of the slopes, especially for bins with sparse data.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the accuracy-vs-similarity trend is measured, not constructed; the only self-citation is methodological provenance and is not load-bearing.

full rationale

The central claim—LLM QA accuracy declines as edited passages become less semantically similar to same-title Wikipedia content in the training corpus—is an empirical correlation, not a quantity derived from its own definition. The independent variable (maximum all-MiniLM-L6-v2 cosine similarity between the edited passage and same-title Wikipedia content from the model's training corpus, §2 and §3) is computed independently of the dependent variable (per-bin LLM accuracy). No parameter is fitted to accuracy data and then reported as a prediction; the slopes and Pearson correlations are post-hoc summaries of the measured relationship. The extraction pipeline is adapted from the authors' own prior work (Wu et al., 2025, cited in §2: "we adopt the natural perturbation pipeline proposed by Wu et al. (2025), with two slight modifications"), but this citation supplies a curation method, not a uniqueness theorem or evidence for the empirical trend, and Appendix A plus the human annotation protocol provide independent controls. The infini-gram verbatim-leakage check and the human-performance control test alternative explanations rather than constructing the result. The main substantive risk is construct validity of the similarity proxy—whether embedding-model cosine distance tracks the LLM's actual pretraining exposure—and the use of the same embedding model (all-MiniLM-L6-v2) for WikiWhy answer correctness is a potential shared-method confound. These are empirical validity concerns, not circular reductions of the conclusion to its inputs by definition. No circular step can be exhibited from the paper's own equations or construction.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The paper relies on several unproved background assumptions: the naturalness of Wikipedia edits, the validity of embedding-based similarity as a proxy for pretraining proximity, the representativeness of the Wikipedia subset in each training corpus, the adequacy of answer-preservation checks, the reliability of human annotations, and the completeness of leakage checks. No new entities are postulated.

free parameters (3)
  • WikiWhy correctness threshold = 0.6 (all-MiniLM-L6-v2 similarity)
    Hand-chosen threshold to classify generated answers as correct on WikiWhy; affects accuracy calculations and the reported trend for that dataset.
  • BoolQ minimum edited passage length = 56 characters
    Ad hoc filter applied only to BoolQ to remove short edited passages; a post hoc exclusion that affects which instances populate the similarity bins.
  • Number of similarity bins = 10
    Binning choice for the correlation analysis; affects slope estimates and bin-level stability, though trends are robust to alternative embeddings.
assumptions (6)
  • domain assumption Wikipedia revision histories provide valid instances of natural text evolution.
    The edited passages are mined from Wikipedia page histories and treated as realistic human edits; used throughout Section 2.
  • domain assumption Sentence-embedding similarity (e.g., all-MiniLM-L6-v2) is a valid proxy for semantic closeness to pretraining content.
    The independent variable is the maximum SBERT similarity between edited passage and same-title training-corpus Wikipedia text; if invalid, bins misrepresent training proximity.
  • domain assumption The Wikipedia subset of each model's training corpus (Dolma, RedPajama, SlimPajama, Pile) accurately reflects the Wikipedia content the model saw.
    Training corpora may contain multiple snapshots or derived versions; the paper queries the provided Wikipedia component and assumes it is representative.
  • domain assumption Answers Preserving Checking ensures edited passages remain answerable for extractive datasets.
    For extractive QA, the check verifies that at least one ground-truth answer string remains in the edited passage; for BoolQ a manual filter is used, and for WikiWhy no check is applied (Section A).
  • domain assumption Human annotations provide an unbiased measure of passage answerability.
    Doctoral-student annotators label samples across similarity bins; no inter-annotator agreement statistic is reported, and the sample is small.
  • domain assumption Infini-gram verbatim matching accurately estimates leakage of edited passages into the full training corpus.
    Used in Section 4 to argue that verbatim inclusion is negligible; only verbatim n-grams are checked, not semantic paraphrase leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Natural Context Drift Undermines the Natural Language Understanding of Large Language Models." pith.science (2026). https://pith.science/paper/KZ4GH3F3

@misc{pith2026250901093,
  author       = {Pith},
  title        = {Pith review of: Natural Context Drift Undermines the Natural Language Understanding of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZ4GH3F3}},
  note         = {Machine review of arXiv:2509.01093}
}
read the original abstract

How does the natural evolution of context paragraphs affect question answering in generative Large Language Models (LLMs)? To investigate this, we propose a framework for curating naturally evolved, human-edited variants of reading passages from contemporary QA benchmarks and for analyzing LLM performance across a range of semantic similarity scores, which quantify how closely each variant aligns with content seen during pretraining. Using this framework, we evaluate six QA datasets and eight LLMs with publicly available training data. Our experiments reveal that LLM performance declines as reading passages naturally diverge from the versions encountered during pretraining-even when the question and all necessary information remains present at inference time. For instance, average model accuracy on BoolQ drops by over 30% from the highest to lowest similarity bins, with slopes exceeding 70 across several LLMs. These findings suggest that natural text evolution poses a significant challenge to the language understanding capabilities of LLMs.

Figures

Figures reproduced from arXiv: 2509.01093 by the authors.

Figure 1
Figure 1. An overview of the analysis framework. Module [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Average accuracy of the instruction-finetuned [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Average accuracy of other instruction-finetuned LLMs across ten semantic similarity bins. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Accuracy of human annotators on QA tasks [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Average accuracy of instruction-finetuned [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Average accuracy of the instruction-finetuned [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Average accuracy of the Base OLMo LLMs across ten semantic similarity bins. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Average accuracy of other Base LLMs across ten semantic similarity bins. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Average accuracy of the instruction-finetuned [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Average accuracy of other instruction-finetuned LLMs across ten semantic similarity bins, with the [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    The LENS framework applied to 192 real-world settings shows moderate natural prompt distribution shifts cause 73% average performance loss in deployed LLMs, especially across user groups and regions.

Reference graph

Works this paper leans on

44 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. https://doi.org/10.1162/tacl_a_00338 Beat the AI : Investigating adversarial human annotation for reading comprehension . Transactions of the Association for Computational Linguistics, 8:662--678

    doi:10.1162/tacl_a_00338 2020

  2. [2]

    Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.147 Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive readers? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2514--2528, Miami, Florida, USA. Association for Co...

    doi:10.18653/v1/2024.emnlp-main.147 2024

  3. [3]

    Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. https://arxiv.org/abs/2201.07311 Datasheet for the pile . Preprint, arXiv:2201.07311

    arXiv 2022

  4. [4]

    Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University

    work page 2018

  5. [5]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language...

    doi:10.18653/v1/n19-1300 2019

  6. [6]

    Together Computer. 2023. https://github.com/togethercomputer/RedPajama-Data Redpajama: An open source recipe to reproduce llama training dataset

    work page 2023

  7. [7]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm Free dolly: Introducing the world's first truly open instruction-tuned llm

    2023

  8. [8]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025 a . https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement ...

    arXiv 2025

Show all 44 references
  1. [9]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025 b . https://arxiv.org/abs...

    2025 arXiv

  2. [10]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . P...

    2020 arXiv

  3. [11]

    Max Glockner, Xiang Jiang, Leonardo F. R. Ribeiro, Iryna Gurevych, and Markus Dreyer. 2025. https://arxiv.org/abs/2505.05949 Neoqa: Evidence-based question answering with generated news events . Preprint, arXiv:2505.05949

    2025 arXiv

  4. [12]

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 2...

    2024 doi

  5. [13]

    Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William Yang Wang. 2023. https://openreview.net/forum?id=vaxnu-Utr4l Wikiwhy: Answering and explaining cause-and-effect questions . In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. https://doi.org/10.1145/3600006.3613165 Efficient memory management for large language model serving with pagedattention . In Proceedings of the 29...

    2023

  7. [15]

    Mosh Levy, Shauli Ravfogel, and Yoav Goldberg. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.569 Guiding LLM to fool itself: Automatically manipulating machine reading comprehension shortcut triggers . In Findings of the Association for Computational Linguistics: EMNLP...

    2023 doi

  8. [16]

    Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.30 An open-source data contamination report for large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 528--541, Miami...

    2024 doi

  9. [17]

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024 a . https://openreview.net/forum?id=u2vAyMeLMm Infini-gram: Scaling unbounded n-gram language models to a trillion tokens . In First Conference on Language Modeling

    2024

  10. [18]

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, and 8 others. 2024 b . https://...

    2024

  11. [19]

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, and 5...

    2025 arXiv

  12. [20]

    Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. https://doi.org/10.18653/v1/2022.findings-acl.146 Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models . In Findings of the Association for Computa...

    2022 doi

  13. [21]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, an...

    2025 arXiv

  14. [22]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  15. [23]

    Medha Palavalli, Amanda Bertsch, and Matthew Gormley. 2024. https://doi.org/10.18653/v1/2024.conda-1.3 A taxonomy for data contamination in large language models . In Proceedings of the 1st Workshop on Data Contamination (CONDA), pages 22--40, Bangkok, Thailand. Association fo...

    2024 doi

  16. [24]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  17. [25]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  18. [26]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  19. [27]

    Viktor Schlegel, Marco Valentino, Andre Freitas, Goran Nenadic, and Riza Batista-Navarro. 2020. https://aclanthology.org/2020.lrec-1.660/ A framework for evaluation of machine reading comprehension gold standards . In Proceedings of the Twelfth Language Resources and Evaluatio...

    2020

  20. [28]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://www.cerebras.net/blog/slimpajama-a-62...

    2023

  21. [29]

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennigh...

    2024 doi

  22. [30]

    Zhen Wang. 2022. https://arxiv.org/abs/2206.15030 Modern question answering datasets and benchmarks: A survey . Preprint, arXiv:2206.15030

    2022 arXiv

  23. [31]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processi...

    2022

  24. [32]

    Edwin B. Wilson. 1927. https://doi.org/10.1080/01621459.1927.10502953 Probable inference, the law of succession, and statistical inference . Journal of the American Statistical Association, 22(158):209--212

    1927

  25. [33]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 doi

  26. [34]

    Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2021. https://doi.org/10.18653/v1/2021.eacl-main.311 Is the understanding of explicit discourse relations required in machine reading comprehension? In Proceedings of the 16th Conference of the European Chapter of the Assoc...

    2021 doi

  27. [35]

    Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-short.21 Are machine reading comprehension systems robust to context paraphrasing? In Proceedings of the 13th International Joint Conference on Natural Language Processing and t...

    2023 doi

  28. [36]

    Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2025. https://arxiv.org/abs/2502.16523 Pay attention to real world perturbations! natural robustness evaluation in machine reading comprehension . Preprint, arXiv:2502.16523

    2025 arXiv

  29. [37]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. https://doi.org/10.1145/3626772.3657878 C-pack: Packed resources for general chinese embeddings . In Proceedings of the 47th International ACM SIGIR Conference on Research and Develop...

    2024

  30. [38]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qw...

    2025 arXiv

  31. [39]

    Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. https://doi.org/10.18653/v1/D17-1213 Identifying semantic edit intentions from revisions in W ikipedia . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2000--2010,...

    2017 doi

  32. [40]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  33. [41]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. https://arxiv.org/abs/2401.02385 Tinyllama: An open-source small language model . Preprint, arXiv:2401.02385

    2024 arXiv

  34. [42]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2025. https://arxiv.org/...

    2025 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.