Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

AbsenceBench: Language Models Can't Tell What's Missing

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that current language models, though near-perfect at finding inserted content in long documents, fail to identify deliberately omitted lines, with the best model reaching 69.6% F1 on its new AbsenceBench benchmark—a gap…

desk verdict New and useful benchmark; the core empirical claim likely holds, but the insertion control and attention explanation are weaker than the paper suggests. read the letter →

arxiv 2506.11440 v1 pith:OQZP7MLE submitted 2025-06-13 cs.CL

Harvey Yiyun Fu , Aryan Shrivastava , Jared Moore , Peter West , Chenhao Tan , Ari Holtzman This is my paper · ORCID
classification cs.CL
keywords absencedetectionomissionlong-contextlanguagemodelsneedle-in-a-haystackself-attentionbenchmarkLLM-as-a-judgeinference-timecompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a sharp asymmetry in large language models: they are very good at locating content that has been added to a document, but noticeably bad at listing content that has been removed. To demonstrate this, the authors construct AbsenceBench, a benchmark built from poetry, numerical sequences, and GitHub pull requests, where a model is shown an original document and an edited copy with some lines or numbers deliberately deleted and must identify exactly what is missing. Across fourteen state-of-the-art models, the best score is 69.6% micro-F1 at an average context length of only 5K tokens, far below the near-perfect results the same models achieve on needle-in-a-haystack retrieval. The paper argues the failure is structural to Transformer attention, which can weight existing tokens but has nothing to attend to at a gap, and supports this by showing that inserting a placeholder token at each omission raises F1 substantially across three tested models.

What carries the argument

The mechanism under test is the omission itself: AbsenceBench presents both the original and the modified document, so the task reduces to identifying the set difference, yet models fail because a removed element leaves no token for Transformer attention to key onto. The paper's analytical instrument is the placeholder manipulation—replacing deletions with '<missing line>' or an underline run—which restores an attendable position and reliably improves F1, and the comparative instrument is the insertion-bench control, which swaps deletion for addition of foreign Harry Potter lines to isolate absence from the general difficulty of detecting document changes.

What would settle it

Run an insertion-bench control where the added lines are in-domain and plausible—e.g., other lines from the same poet or realistic diff lines from the same repository—rather than foreign Harry Potter text; if models keep near-99% F1 the absence explanation survives the salience probe, while a drop toward AbsenceBench levels would show the 56.9% contrast is largely a salience artifact.

Watch

Extended reading notes

Core claim

The central claim is that identifying absence is a distinct capability from identifying presence, and that current LLMs largely lack it. On AbsenceBench, a task any undergraduate could solve by line-by-line comparison in linear time, the strongest evaluated model family reaches only about 70% micro-F1, while the same models score near 99% on an insertion-bench control where foreign lines are added rather than removed; the paper reports a 56.9% average F1 drop when the task flips from insertion to omission. The proposed mechanism is that self-attention allocates weight among existing keys and therefore cannot anchor on a deletion, since a gap supplies no token to attend to. As evidence, the authors show that explicitly marking each deleted segment with a placeholder such as '<missing line>' raises the average F1 of three models by roughly 36–42% depending on the averaging, with the largest relative gain (+81.8%) in the GitHub pull-request domain.

Load-bearing premise

The paper's omission-versus-insertion comparison assumes that adding foreign, off-topic Harry Potter lines is the exact inverse of deleting original lines, even though inserted foreign content is far more salient than a missing original line; the control may be easier irrespective of whether absence detection is the bottleneck.

Editorial extensions

If this is right

  • If the gap holds up, current LLM-as-a-judge systems can silently fail to notice rubric criteria that are not addressed, even when the judged text is right in front of the model.
  • Diff-review and merge-conflict tools that rely on LLMs would benefit from explicitly marking deleted lines with placeholders, since that simple change substantially raises detection accuracy.
  • Longer contexts are likely to make absence detection harder, not easier, for at least the poetry domain, so NIAH-style long-context scores should not be read as evidence of absence-detection ability.
  • Inference-time compute buys only about a 7.9% average F1 improvement while generating roughly 8K thinking tokens per task, so reasoning about absence is currently an expensive workaround rather than a fix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A matched-salience control (inserting plausible in-domain lines rather than foreign Harry Potter text) would test whether the 56.9% insertion-versus-omission gap is partly a salience artifact rather than purely an absence effect.
  • The placeholder result suggests a design principle for future architectures: an explicit 'nothing here' token, or a mechanism that can attend to emptiness, may be necessary for semantic absence detection such as missing premises or missing rubric elements.
  • Because the benchmark is solvable by a linear-time program, prompting models to perform an explicit diff (rather than free recall) is a natural stress test that separates attention limitations from instruction-following failures.
  • AbsenceBench's surface-form simplicity implies it is a lower bound on difficulty; real-world absence detection—missing evidence, missing steps in a proof, missing criteria in a review—is likely much harder and deserves its own benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AbsenceBench, a benchmark for evaluating whether LLMs can identify deliberately omitted content. It covers three domains (poetry, numerical sequences, GitHub pull-request diffs), asks models to compare an original and a modified document, and measures micro F1 for recovering the omitted lines. The authors evaluate 14 LLMs, report uniformly low scores (best average 69.6% F1 at ~5K context), and compare this to a Needle-in-a-Haystack (NIAH) insertion control in which Harry Potter lines are added to the same documents. They also study the effect of context length, omission rate, prompt placement, thinking-mode reasoning, and placeholder markers. The central empirical claim is that LLMs are dramatically worse at detecting omissions than insertions, with the difference attributed to Transformer attention's inability to attend to gaps.

Significance. The benchmark itself is a useful and clearly specified probe of an understudied capability: detecting missing content, which matters for applications such as LLM-as-a-judge and code review. The paper deserves credit for releasing code, evaluating 14 models across three diverse domains, and using micro F1 rather than recall, which correctly penalizes the trivial copy-the-whole-context strategy. If the main result holds, it is a meaningful counterpoint to NIAH-style evaluations. However, the paper's headline asymmetry between omissions and insertions rests on a control that confounds the operation with the salience of the inserted content, and the attention-based explanation is supported only by an intervention that changes the task itself. The benchmark results and the descriptive trends are likely sound, but the mechanistic and comparative conclusions need substantial revision.

major comments (3)
  1. [§3.2 and Appendix C, Table 10] The claim that 'locating omissions are harder than insertions' is not established by the insertion-bench control. The inserted needles are Harry Potter lines, which are semantically and stylistically foreign to poetry and code diffs, whereas the omitted items are original in-domain lines. A Harry Potter line can be spotted by surface incongruity (prose in verse, narrative text in a diff), while a missing original line leaves no such cue. The near-ceiling performance (99.5% poetry, ≥86.2% GitHub) is therefore expected under a salience account and does not isolate absence detection as the bottleneck. To support the paper's central asymmetry, the control should match content salience: for example, insert random lines from other poems into the poetry haystack and random plausible diff lines (or shifted duplicates of existing lines) into the PR haystack, keeping the target items as in-domain as the omitted ones. Without such a matched control, the 56.9% gap cannot be attributed to presence versus absence.
  2. [§4.2 and Table 5] The placeholder intervention is over-interpreted as evidence that 'Transformer attention cannot attend to gaps.' Replacing an omission with an explicit '<missing line>' token changes the task from open-ended absence detection to locating and copying explicit markers; the model no longer needs to infer that something is absent. The large improvement may simply reflect that the placeholder provides the answer positions directly. A necessary control is to insert placeholders at random non-omitted positions (or to replace some present lines with placeholders) to test whether the benefit comes from marking actual gaps or from the availability of explicit position markers. The current design does not distinguish an attention-to-gaps mechanism from a prompt/format effect, so the mechanistic conclusion in §5 and the abstract is not yet supported.
  3. [§3.2, Table 3, and §7 (Statistical significance)] The paper reports comparative claims—for example, that inference-time compute improves performance by 7.9%, that closed-source models outperform open-weights models by a margin, and that context-length correlations have R² = 0.55 vs 0.08 across domains—without any error bars or significance testing. The authors acknowledge this in §7, but the missing variability quantification is load-bearing for several secondary claims, particularly the inference-time-compute benefit and the cross-domain correlation comparisons. I request at least a repeated-run or bootstrap analysis on a subsample of tasks (e.g., 5 runs on 100 instances per domain) to confirm that the reported gaps are not within-run noise, and to qualify the R² values accordingly.
minor comments (5)
  1. [Abstract] Typo: 'AbsenceBench to assesses' should be 'AbsenceBench assesses'.
  2. [§1, paragraph 4] Typo: 'observing a a massive 56.9% drop' contains a duplicated article.
  3. [Appendix C, Table 8] The Poetry (NIAH) prompt is internally inconsistent: the system prompt says 'List only the missing lines' while the user message asks for 'the extra lines.' Since this is the control that supports the main comparison, the prompt should be corrected and the NIAH numbers re-verified.
  4. [Table 5] Some percentage increases are missing the '%' symbol (e.g., '+52.0' and '+24.4'), and the table would be clearer if all deltas were formatted consistently.
  5. [Figure 5 and Appendix D] The color legend for omission rate is described only in the caption; consider adding a labeled colorbar to the figure itself for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AbsenceBench reports empirical measurements with no fitted parameters that are renamed as predictions, and no load-bearing self-citation. The insertion-bench comparison carries a salience confound, but that is an experimental validity concern, not a circular reduction.

full rationale

AbsenceBench is an empirical benchmark paper. The core results are F1 measurements of 14 LLMs on deliberately constructed omission tasks (Section 3.1, Table 3), and these measurements are not derived from the task definition by construction. The omission rate p=0.1, the document truncation, and the placeholder interventions are experimental choices, not fitted parameters that are later relabeled as predictions. The NIAH comparison (Section 3.2, Appendix C) uses the original NIAH test and a modified 'insertion bench' in which Harry Potter lines are inserted into poems and GitHub diffs. The large gap between insertion and omission performance is an observed empirical difference, not an equation that reduces to the definition of the benchmark. A critic could argue that the Harry Potter insertions are more salient than omitted original lines, so the comparison may overstate the role of absence per se; however, that is a threat to causal interpretation, not a circularity, because the asymmetry is not guaranteed by construction and could have gone the other way empirically. Likewise, the placeholder experiment in Section 4.2 is an intervention: the authors hypothesize that attention cannot anchor on gaps and then test this by inserting '<missing line>' placeholders. The 35.7% average improvement is an empirical outcome, not a consequence of the hypothesis being embedded in the benchmark definitions. The attention-based explanation is presented as a hypothesis with supporting evidence, and the Limitations section explicitly acknowledges the surface-form-only scope, the lack of significance testing, and the absence of prompt tuning, so the paper is transparent about the strength of its claims. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The benchmark is self-contained against external models and external NIAH results, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark design chooses several hyperparameters by hand (omission rate, step sizes, line ranges). The central claim is empirical and does not fit any parameters to outcomes. The main assumptions are that micro-F1 on element-level exact match measures absence detection, that the three domains are representative, and that the insertion-bench control isolates the omission/insertion distinction.

free parameters (4)
  • omission_probability_p = 0.1 (main set); varied as U(0, 0.5) in perturbation
    Chosen by hand for the main AbsenceBench set; affects task difficulty, but the paper additionally studies p as a variable.
  • numerical_sequence_step_sizes = 1, 4, 7, 13
    Chosen by hand for synthetic numerical sequences; not fitted to any outcome.
  • poem_line_count_range = 100 to 1000 lines, uniform
    Used to create document length diversity; a design choice.
  • pull_request_diff_line_range = 10 to 200 updated lines
    Filtering criterion for GitHub PRs; a design choice.
assumptions (4)
  • domain assumption Micro-F1 at the element level (exact match) is a valid measure of absence detection.
    The paper uses micro F1 to balance false positives from copy-pasting the original; this is reasonable, but it assumes element-level exact match reflects the ability being tested.
  • ad hoc to paper The insertion-bench control (inserting Harry Potter lines) is equivalent to the omission task except for presence versus absence of target content.
    The salience of foreign inserted content may make insertion detection much easier, confounding the omission versus insertion comparison.
  • domain assumption The three domains (poetry, numerical sequences, GitHub PRs) are representative of absence detection.
    The paper generalizes conclusions from these domains to 'LLMs cannot tell what is missing'; the authors themselves note generalization limits.
  • domain assumption NIAH is essentially solved for the compared models.
    The paper reports near-perfect NIAH performance from three models and uses it as a contrast; this is a stated comparison, not independently verified for all models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AbsenceBench: Language Models Can't Tell What's Missing." pith.science (2026). https://pith.science/paper/OQZP7MLE

@misc{pith2026250611440,
  author       = {Pith},
  title        = {Pith review of: AbsenceBench: Language Models Can't Tell What's Missing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQZP7MLE}},
  note         = {Machine review of arXiv:2506.11440}
}
read the original abstract

Large language models (LLMs) are increasingly capable of processing long inputs and locating specific information within them, as evidenced by their performance on the Needle in a Haystack (NIAH) test. However, while models excel at recalling surprising information, they still struggle to identify clearly omitted information. We introduce AbsenceBench to assesses LLMs' capacity to detect missing information across three domains: numerical sequences, poetry, and GitHub pull requests. AbsenceBench asks models to identify which pieces of a document were deliberately removed, given access to both the original and edited contexts. Despite the apparent straightforwardness of these tasks, our experiments reveal that even state-of-the-art models like Claude-3.7-Sonnet achieve only 69.6% F1-score with a modest average context length of 5K tokens. Our analysis suggests this poor performance stems from a fundamental limitation: Transformer attention mechanisms cannot easily attend to "gaps" in documents since these absences don't correspond to any specific keys that can be attended to. Overall, our results and analysis provide a case study of the close proximity of tasks where models are already superhuman (NIAH) and tasks where models breakdown unexpectedly (AbsenceBench).

Figures

Figures reproduced from arXiv: 2506.11440 by the authors.

Figure 1
Figure 1. (a) An overview of the difference between the Needle-in-a-haystack (NIAH) test setting [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The three domains in AbsenceBench test models’ abilities across a variety of document lengths and omission probabilities. Frequency reports the number of tasks in the domain within a given range of document lengths. The average context length across all tasks in AbsenceBench is 5K tokens. On the document level, the average document length is 2.7K, while it is 4.7K for poetry, 1.5K for numerical sequences, and 1.7K f… view at source ↗
Figure 3
Figure 3. Reasoning models often generate an order of magnitude more text than input doc￾ument. Distribution of the thinking token ratio (number of generated thinking tokens divided by number of tokens in the original document) for four inference-time compute models under each domain. We set the parameters of the boxplot to capture 99% of the distribution. The outliers are hidden for better clarity (see [PITH_FULL_IMAGE:figu… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Closed-source models (reds) perform better than open-weights models (blues) on [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: GPT-4.1-mini performs worse on longer tasks in Poetry, but the relationship is not clear in Numerical Sequences and Github PRs. Each plot shows the F1-score (y-axis) and the total context length (x-axis). Dark blue represents a lower and dark red represents a higher pe…
Figure 6
Figure 6. Figure 6: Micro-F1 score of Claude-3.7-Sonnet (y-axis) as a function of the total context length (x-axis) as well as the percentage of omission (color) 0 5000 10000 15000 20000 25000 0.0 0.2 0.4 0.6 0.8 1.0 Micro F1 Poetry R 2=0.60 0 2000 4000 6000 8000 0.0 0.2 0.4 0.6 0.8 1.0 N…
Figure 7
Figure 7. Figure 7: Micro-F1 score of Llama-4-Maverick (y-axis) as a function of the total context length (x-axis) as well as the percentage of omission (color) 22 [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: The thinking token ratio (number of generated thinking tokens divided by number of tokens [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The inattentional gap in task conditioned AI models that omit otherwise reportable safety critical signals

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    Task conditioning suppresses safety-critical signal reporting in language and vision models that unconstrained versions report at higher rates, creating an inattentional gap that decouples benchmark safety from real-w...

  2. When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Models

    cs.CL 2026-08 conditional novelty 6.0 of 10

    LLMs over-certify negative answers from partial evidence, especially when completeness is implied rather than stated, and prompting mainly trades over-closure for under-closure.

Reference graph

Works this paper leans on

47 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D

    Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning, 2024. URL https://arxiv.org/abs/2404.11018

    arXiv 2024

  2. [2]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu,...

    arXiv 2025

  3. [3]

    L-eval: Instituting standardized evaluation for long context language models, 2023

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023. URL https://arxiv.org/abs/2307.11088

    arXiv 2023

  4. [4]

    Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud

    Cem Anil, Esin DURMUS, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Naomi Bashkansky Rylan Schaeffer, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, Jam...

    work page 2024

  5. [5]

    Claude 3 haiku: our fastest model yet, 2024

    Anthropic. Claude 3 haiku: our fastest model yet, 2024. URL https://www.anthropic.com/news/claude-3-haiku

    work page 2024

  6. [6]

    Longbench: A bilingual, multitask benchmark for long context understanding, 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024. URL https://arxiv.org/abs/2308.14508

    arXiv 2024

  7. [7]

    Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley. Unlimiformer: Long-range transformers with unlimited length input, 2023. URL https://arxiv.org/abs/2305.01625

    arXiv 2023

  8. [8]

    Booookscore: A systematic exploration of book-length summarization in the era of llms, 2024

    Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. Booookscore: A systematic exploration of book-length summarization in the era of llms, 2024. URL https://arxiv.org/abs/2310.00785

    arXiv 2024

Show all 47 references
  1. [9]

    Extending context window of large language models via positional interpolation, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL https://arxiv.org/abs/2306.15595

    2023 arXiv

  2. [10]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https://arxiv.org/abs/2405.21060

    2024 arXiv

  3. [11]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  4. [12]

    Mathematical capabilities of chatgpt, 2023

    Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner. Mathematical capabilities of chatgpt, 2023. URL https://arxiv.org/abs/2301.13867

    2023 arXiv

  5. [13]

    Fu, Elliot L

    Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convolutions for sequence modeling, 2023. URL https://arxiv.org/abs/2302.06646

    2023 arXiv

  6. [14]

    How to train long-context language models (effectively), 2025

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively), 2025. URL https://arxiv.org/abs/2410.02660

    2025

  7. [15]

    Gemini 2.5: Our most intelligent ai model, 2025

    Google. Gemini 2.5: Our most intelligent ai model, 2025. URL https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/

    2025

  8. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  9. [17]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    2024 arXiv

  10. [18]

    Ruler: What's the real context size of your long-context language models?, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL https://arxiv.org/abs/2404.06654

    2024 arXiv

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  12. [20]

    Needle in a haystack - pressure testing llms, 2023

    Gregory Kamradt. Needle in a haystack - pressure testing llms, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack

    2023

  13. [21]

    Fables: Evaluating faithfulness and content selection in book-length summarization, 2024

    Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. Fables: Evaluating faithfulness and content selection in book-length summarization, 2024. URL https://arxiv.org/abs/2404.01261

    2024 arXiv

  14. [22]

    Benchmarking cognitive biases in large language models as evaluators, 2024

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2024. URL https://arxiv.org/abs/2309.17012

    2024 arXiv

  15. [23]

    The narrativeqa reading comprehension challenge, 2017

    Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017. URL https://arxiv.org/abs/1712.07040

    2017 arXiv

  16. [24]

    Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, a...

    2024 arXiv

  17. [25]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

    Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  18. [26]

    Openai o3-mini, pushing the frontier of cost-effective reasoning., 2025

    OpenAI. Openai o3-mini, pushing the frontier of cost-effective reasoning., 2025. URL https://openai.com/index/openai-o3-mini/

    2025

  19. [27]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  20. [28]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  21. [29]

    gutenberg-poetry-corpus: A corpus of poetry from project gutenberg, 2018

    Allison Parrish. gutenberg-poetry-corpus: A corpus of poetry from project gutenberg, 2018

    2018

  22. [30]

    Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Kris...

    2023 arXiv

  23. [31]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. URL https://arxiv.org/abs/2108.12409

    2022 arXiv

  24. [32]

    Qwen3 technical report, 2025 a

    Qwen. Qwen3 technical report, 2025 a . URL https://github.com/QwenLM/Qwen3/blob/main/Qwen3_Technical_Report.pdf

    2025

  25. [33]

    Qwq-32b: Embracing the power of reinforcement learning, 2025 b

    Qwen. Qwq-32b: Embracing the power of reinforcement learning, 2025 b . URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  26. [34]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  27. [35]

    Z ero SCROLLS : A zero-shot benchmark for long text understanding

    Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Z ero SCROLLS : A zero-shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977--7989,...

    2023 doi

  28. [36]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  29. [37]

    A length-extrapolatable transformer, 2022

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer, 2022. URL https://arxiv.org/abs/2212.10554

    2022 arXiv

  30. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  31. [39]

    Michelangelo: Long context evaluations beyond haystacks via latent structure queries, 2024

    Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Fira...

    2024 arXiv

  32. [40]

    Novelqa: Benchmarking question answering on documents exceeding 200k tokens, 2025

    Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Xiangkun Hu, Zheng Zhang, Qian Wang, and Yue Zhang. Novelqa: Benchmarking question answering on documents exceeding 200k tokens, 2025. URL https://arxiv.org/abs/2403.12766

    2025 arXiv

  33. [41]

    Grok 3 beta — the age of reasoning agents, 2025

    xAI. Grok 3 beta — the age of reasoning agents, 2025. URL https://x.ai/news/grok-3

    2025

  34. [42]

    Stress-testing long-context language models with lifelong icl and task haystack, 2024

    Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress-testing long-context language models with lifelong icl and task haystack, 2024. URL https://arxiv.org/abs/2407.16695

    2024 arXiv

  35. [43]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  36. [44]

    Long-context language modeling with parallel context encoding, 2024

    Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding, 2024. URL https://arxiv.org/abs/2402.16617

    2024 arXiv

  37. [45]

    Helmet: How to evaluate long-context language models effectively and thoroughly, 2025

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly, 2025. URL https://arxiv.org/abs/2410.02694

    2025 arXiv

  38. [46]

    bench: Extending long context evaluation beyond 100k tokens, 2024

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. bench: Extending long context evaluation beyond 100k tokens, 2024. URL https://arxiv.org/abs/2402.13718

    2024 arXiv

  39. [47]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.