Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that LLM accuracy on its GSM-Infinite arithmetic problems declines along a sigmoid as the required number of operations grows, and that exponential test-time compute yields only linear gains.

desk verdict A genuinely useful synthetic long-context reasoning benchmark, with a headline scaling-law claim that does not survive close reading as stated. read the letter →

arxiv 2502.05252 v1 pith:5NF7I72B submitted 2025-02-07 cs.CL cs.AI

Yang Zhou , Hongyi Liu , Zhuoming Chen , Yuandong Tian , Beidi Chen This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMreasoninglong-contextbenchmarkssyntheticdatagenerationcomputationalgraphscomplexityinferencescalingsigmoidperformancedecaygrade-schoolmath
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces GSM-Infinite, a generator that turns grade-school word problems into computational graphs and then into arbitrarily long, arbitrarily hard arithmetic problems. Using it, the authors claim two quantitative laws: model accuracy declines along a sigmoid as the number of required operations increases, and exponentially increasing inference computation yields only linear gains in aggregate accuracy. If true, these laws give a common yardstick for comparing LLM reasoning across models and contexts, and they suggest that current models fail from fundamental limits rather than from a lack of specific benchmark examples. The paper also claims that its graph-based noise resists retrieval-augmented systems, so the long-context degradation it measures is driven by genuine reasoning, not by easily filtered filler.

What carries the argument

The central construct is a computational graph whose nodes are arithmetic variables and whose edges are the four grade-school operations, with a query node at the end. Reasoning complexity is measured by the op count: the length of the shortest topological solution path from the given values to the query. Complexity is scaled by adding nodes and edges, and noise is added by extending the core graph outward with semantically close fake nodes and edges, a 'spider topology' that makes retrieved chunks hard to distinguish from essential ones. A reverse mode masks low-level variables so problems require implicit subtraction and division, and a template system maps graphs into natural language. The evaluation metric is AUC, a Riemann sum of accuracy over op counts from 2 until accuracy falls below 5%.

What would settle it

Run a single model on thousands of Medium problems at op counts 5, 8, 11, 14, and 17, and fit the sigmoid separately for addition-heavy and division-heavy graphs. If the inflection point moves with operation type at the same op count, the op-count axis is not the uniform complexity driver the paper's law requires.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM reasoning performance on arithmetically structured problems is a predictable function of quantitative complexity, and the function has the same shape across models. On the GSM-Infinite zero-noise tasks, accuracy as a function of op count is fitted by a sigmoid with $R^2 > 0.98$ for the models shown; models differ mainly in where the curve drops. A second claimed regularity is that running best-of-N repeated sampling produces aggregate AUC gains that are linear in $\log N$, so each additional unit of AUC costs exponentially more samples. On long-context versions, accuracy falls as context grows at fixed difficulty, and the paper reports that its graph-generated noise defeats both passive and active retrieval-augmented baselines, unlike the noise in earlier long-context benchmarks.

Load-bearing premise

The load-bearing premise is that the number of operations on the shortest solution path is a fair measure of how hard a problem is; if some operations or graph structures cost a model more than others, the sigmoid curves and model rankings built on that axis would change.

Editorial extensions

If this is right

  • Accuracy on GSM-Infinite problems can be predicted from op count by a sigmoid, so new models can be compared by a single inflection point rather than by heterogeneous benchmark scores.
  • Best-of-N sampling's benefit is logarithmic, meaning test-time compute is an expensive lever for hard multi-step reasoning.
  • Retrieval-augmented systems cannot substitute for long-context reasoning on these noise graphs, so the benchmark isolates reasoning rather than retrieval.
  • The generator can produce arbitrarily many problems at any op count, removing the data-scarcity barrier that prevents fine-grained difficulty evaluation on human-written sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the sigmoid law is driven by the number of sequential operations, then models that externalize operations through a scratchpad or chain-of-thought should shift the curve to the right; this is a testable prediction the paper does not make.
  • Beyond the paper: the log-linear repeated-sampling law suggests a compute-optimal frontier between sampling and model scale for reasoning tasks, which could be quantified from GSM-Infinite style curves.
  • Beyond the paper: the same graph generator could be extended beyond arithmetic to symbolic algebra, logic, or code traces to test whether the sigmoid is a property of multi-step reasoning in general or of arithmetic specifically.
  • Beyond the paper: because forward problems consistently outperform reverse problems, GSM-Infinite could be used to measure how much of a model's reasoning is constructive rather than genuinely bidirectional.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces GSM-Infinite, a synthetic grade-school math benchmark generated from computational graphs. The generator controls reasoning complexity (measured as the length of the topological solution path, called op count) and context length by adding noise through 'spider' graph extensions. The authors evaluate 18 LLMs on zero-noise problems and 10 LLMs at 8K/16K/32K contexts, reporting Area-Under-Curve (AUC) versus op count. They claim a consistent sigmoid decline in accuracy as complexity increases, a forward/reverse asymmetry, RAG failure on their noise, and a repeated-sampling result that exponentially increasing inference compute yields only linear AUC gains. The abstract and conclusion elevate these observations to general scaling laws.

Significance. If the claims hold, GSM-Infinite is a valuable controlled testbed: it provides fine-grained control of complexity and context length, a scalable synthetic corpus, and noise that defeats retrieval-based baselines. The forward/reverse asymmetry and the RAG failure analysis are concrete, reproducible contributions. The public code and benchmark release are strengths. However, the most striking claim—'exponentially increasing inference computation yields only linear performance gains'—is not supported by the reported measurements, and the op-count difficulty axis is used in ways that outrun its validation. The benchmark itself appears sound enough to be salvaged, but the paper's central framing needs substantial revision.

major comments (3)
  1. [§5.5, Appendix C.2, Eq. (2)]
  2. [§3.1, §5.2, §5.3, Table 1]
  3. [§5.2, Figure 10]
minor comments (5)
  1. [Table 1 caption]
  2. [§5.5]
  3. [Appendix C.2, Eq. (2)]
  4. [§4.3]
  5. [Figure 1]

Circularity Check

1 steps flagged · score 6.0 of 10

The repeated-sampling 'exponentially increasing compute yields only linear gains' finding reduces, via Appendix C.2 Eqs. (1)-(3), to a deterministic transformation of the per-task success probabilities fitted from 256 samples; the benchmark and sigmoid-decay results are otherwise self-contained.

  1. fitted input called prediction [Section 5.5 and Appendix C.2 (Eqs. 1-3)]
    "Compute per-task empirical accuracy: ptask = # Correct Samples / 256 (1). Estimate accuracy for N samples: Acctask = 1−(1−ptask)^N (2). Average results across 80 tasks: Final Accuracy = 1/80 Σ Acctask (3). ... If we plot the AUC score versus the number of repeated trial settings and take the log scale of the repeated trial N, the graph is linear ... Therefore, GSM-∞ helps reveal that Repeated Sampling leads to linear AUC Improvement from exponentially increasing computation cost."

    The claimed inference-scaling law is not an independent measurement at multiple realized N values. p_task is empirically estimated from 256 samples per task (Eq. 1), and every 'accuracy at N' used to build the AUC-versus-logN curve is defined by Eq. 2 as a deterministic transformation of those fitted p_task values. Averaging over tasks (Eq. 3) and summing into AUC then returns a function of the fitted p_task distribution; the linear-in-logN trend is an artifact of that transformation plus the chosen N range, not a separately measured property of LLM inference.

full rationale

The benchmark construction and the sigmoid-decay characterization are self-contained: op count is defined from the generated computational graph, and the sigmoid fits are descriptive curve fits to measured accuracy, so they do not reduce to their inputs. No load-bearing self-citation chain is present; the adopted 'abstract parameters' construct is cited to external work (Ye et al. 2024a), not to the authors' own prior results. The one clear circular reduction is the repeated-sampling scaling claim. In Appendix C.2 the per-task success probability p_task is estimated from 256 samples (Eq. 1), and then the accuracy at every trial count N is defined by Eq. 2 as 1-(1-p_task)^N, averaged in Eq. 3 and integrated into AUC. The reported linear AUC versus log N relation is therefore computed from the fitted p_task values under the i.i.d. formula; it is not a separately realized best-of-N measurement, and the paper does not report the p_task distribution or any N > 256 runs. Presenting this deterministic transformation as a discovered 'systematic inference scaling trend' is the fitted input renamed as a prediction. This makes the repeated-sampling finding partially circular, while the rest of the paper's content remains independent, so the score is 6 rather than 8 or 10.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's central results rest on the op-count complexity measure, the independence assumption for repeated sampling, and the benchmark generator's design choices. No new physical or mathematical entities are introduced; the benchmark itself is the artifact.

free parameters (3)
  • Sigmoid fit parameters (midpoint and growth rate) per model and subtask = not reported
    Fitted to accuracy versus operation count to support the claim of a consistent sigmoid decline (Section 5.2, Figure 10).
  • Linear fit slope and intercept for AUC versus log N = not reported
    Fitted to the computed AUC values over repeated sampling trials to claim a linear relationship with log N (Section 5.5, Figure 9b).
  • AUC cutoff accuracy threshold and op range = 5% accuracy, op 2 to when accuracy drops below 5%
    Choice of integration range for AUC affects all reported scores; set in Section 4.3.
assumptions (5)
  • domain assumption The number of operations in the shortest topological solution path is a valid uniform measure of reasoning complexity.
    The paper equates op count with complexity and uses it as the x-axis for all accuracy curves and AUC (Section 3.1).
  • standard math Independent sampling with temperature T = 1.0 makes best-of-N accuracy equal to 1-(1-p)^N per task.
    Used in Appendix C.2 to estimate accuracy for N repeated samples from per-task success probability p.
  • domain assumption The computational graph abstraction with abstract and instance parameters faithfully represents natural-language grade-school math problems.
    Basis for the problem generator, following Ye et al. (2024a); assumed to transfer to natural language reasoning.
  • domain assumption The three templates (animals-zoo, teachers-school, movies-festival) provide sufficient linguistic diversity and do not introduce memorization confounds.
    The paper acknowledges limited natural language diversity in Appendix B and relies on template variety for generality.
  • domain assumption Dense retriever all-mpnet-v2-base with a 2048-token budget is a representative RAG baseline.
    Used for RAG comparisons in Section 5.4 and Appendix C.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?." pith.science (2026). https://pith.science/paper/5NF7I72B

@misc{pith2026250205252,
  author       = {Pith},
  title        = {Pith review of: GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NF7I72B}},
  note         = {Machine review of arXiv:2502.05252}
}
read the original abstract

Long-context large language models (LLMs) have recently shown strong performance in information retrieval and long-document QA. However, to tackle the most challenging intellectual problems, LLMs must reason effectively in long and complex contexts (e.g., frontier mathematical research). Studying how LLMs handle increasing reasoning complexity and context length is essential, yet existing benchmarks lack a solid basis for quantitative evaluation. Inspired by the abstraction of GSM-8K problems as computational graphs, and the ability to introduce noise by adding unnecessary nodes and edges, we develop a grade school math problem generator capable of producing arithmetic problems with infinite difficulty and context length under fine-grained control. Using our newly synthesized GSM-Infinite benchmark, we comprehensively evaluate existing LLMs. We find a consistent sigmoid decline in reasoning performance as complexity increases, along with a systematic inference scaling trend: exponentially increasing inference computation yields only linear performance gains. These findings underscore the fundamental limitations of current long-context LLMs and the key challenges in scaling reasoning capabilities. Our GSM-Infinite benchmark provides a scalable and controllable testbed for systematically studying and advancing LLM reasoning in long and complex contexts.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning

    cs.CL 2026-07 conditional novelty 6.0 of 10

    GEAR adds a grounding reward for overlap with key evidence and a distractor penalty to the RL accuracy signal, reducing repetitive copying and improving long-context reasoning accuracy by up to 4.6 points over accurac...

  2. The Power of Power Law: Asymmetry Enables Compositional Reasoning

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    Power-law data sampling creates beneficial asymmetry in the loss landscape that lets models acquire high-frequency skill compositions first, enabling more efficient learning of rare long-tail skills than uniform distr...

  3. From Local Corrections to Generalized Skills: Improving Neuro-Symbolic Policies with MEMO

    cs.RO 2026-03 conditional novelty 6.0 of 10

    Clustering multi-user multi-task language corrections into a retrieval-augmented skillbook of code templates lets neuro-symbolic robots generate new skills and raise zero-shot success on novel tasks.

  4. vAttention: Verified Sparse Attention

    cs.LG 2025-10 conditional novelty 6.0 of 10

    vAttention is a sparse attention method that mixes heavy-hitter tokens with a statistically sized random sample to provide (ε, δ)-guaranteed approximation of full attention.

  5. LongReasonArena: A Long Reasoning Benchmark for Large Language Models

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A benchmark where AI models must run long algorithmic computations reveals a linear drop in accuracy as the log of reasoning steps increases, with top models scoring 7-16% on the hardest level.

  6. Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings

    cs.IR 2025-05 conditional novelty 6.0 of 10

    A new benchmark (ConTEB) and training method (InSeNT) show that context-aware chunk embeddings greatly improve retrieval on context-dependent queries, with minimal computational overhead.

Reference graph

Works this paper leans on

41 extracted references · 5 canonical work pages · cited by 6 Pith papers

  1. [1]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  2. [2]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  3. [3]

    variables, where only implicit addition/subtraction can be generated. (d) contrasts (c) and shows an example that with additional ``three-entity

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv

  4. [4]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  5. [5]

    Longbench: A bilingual, multitask benchmark for long context understanding, 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024. https://arxiv.org/abs/2308.14508

    arXiv 2024

  6. [6]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025. https://arxiv.org/abs/2412.15204

    arXiv 2025

  7. [7]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. https://arxiv.org/abs/2004.05150

    arXiv 2020

  8. [9]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024 b . https://arxiv.org/abs/2407.21787

    arXiv 2024

Show all 41 references
  1. [11]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021 b

    arXiv

  2. [12]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  3. [13]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flash A ttention: Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  4. [14]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  5. [15]

    Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy, 2023. https://arxiv.org/abs/2207.02098

    2023 arXiv

  6. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  7. [17]

    Needle in a haystack - pressure testing llms, 2023

    Github. Needle in a haystack - pressure testing llms, 2023. https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main

    2023

  8. [18]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. https://arxiv.org/abs/2103.03874

    2021 arXiv

  9. [20]

    Ruler: What's the real context size of your long-context language models?, 2024 b

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024 b . https://arxiv.org/abs/2404.06654

    2024 arXiv

  10. [21]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023 a

    arXiv

  11. [22]

    Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig

    Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation, 2023 b . https://arxiv.org/abs/2305.06983

    2023 arXiv

  12. [23]

    Needle in a haystack - pressure testing llms, 2023

    Gregory Kamradt. Needle in a haystack - pressure testing llms, 2023. https://github.com/gkamradt/LLMTest NeedleInAHaystack/tree/main

    2023

  13. [24]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024. https://arxiv.org/abs/2406.10149

    2024 arXiv

  14. [25]

    Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, a...

    2024 arXiv

  15. [26]

    Same task, more tokens: the impact of input length on the reasoning performance of large language models, 2024

    Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models, 2024. https://arxiv.org/abs/2402.14848

    2024 arXiv

  16. [27]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. https://arxiv.o...

    2021 arXiv

  17. [28]

    Long context vs

    Xinze Li, Yixin Cao, Yubo Ma, and Aixin Sun. Long context vs. rag for llms: An evaluation and revisits. arXiv preprint arXiv:2501.01880, 2024 a

    2024 arXiv

  18. [29]

    Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach

    Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, ...

    2024

  19. [30]

    Ring attention with blockwise transformers for near-infinite context, 2023 a

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023 a . https://arxiv.org/abs/2310.01889

    2023 arXiv

  20. [31]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023 b . https://arxiv.org/abs/2307.03172

    2023 arXiv

  21. [32]

    Dafnybench: A benchmark for formal software verification, 2024

    Chloe Loughridge, Qinyi Sun, Seth Ahrenbach, Federico Cassano, Chuyue Sun, Ying Sheng, Anish Mudide, Md Rakib Hossain Misu, Nada Amin, and Max Tegmark. Dafnybench: A benchmark for formal software verification, 2024. https://arxiv.org/abs/2406.08467

    2024 arXiv

  22. [33]

    Minimax-01: Scaling foundation models with lightning attention, 2025

    MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junha...

    2025 arXiv

  23. [34]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024

    arXiv

  24. [35]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  25. [36]

    Tree attention: Topology-aware decoding for long-context attention on gpu clusters, 2024

    Vasudev Shyam, Jonathan Pilault, Emily Shepperd, Quentin Anthony, and Beren Millidge. Tree attention: Topology-aware decoding for long-context attention on gpu clusters, 2024. https://arxiv.org/abs/2408.04093

    2024 arXiv

  26. [37]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. https://arxiv.org/abs/2408.03314

    2024 arXiv

  27. [38]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024 a

    2024 arXiv

  28. [39]

    Jamba-1.5: Hybrid transformer-mamba models at scale

    Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid transformer-mamba models at scale. arXiv preprint arXiv:2408.12570, 2024 b

    2024 arXiv

  29. [40]

    Modular elliptic curves and fermat's last theorem

    Andrew Wiles. Modular elliptic curves and fermat's last theorem. Annals of mathematics, 141 0 (3): 0 443--551, 1995

    1995

  30. [41]

    Physics of language models: Part 2.1, grade-school math and the hidden reasoning process, 2024 a

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process, 2024 a . https://arxiv.org/abs/2407.20311

    2024 arXiv

  31. [42]

    Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems, 2024 b

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems, 2024 b . https://arxiv.org/abs/2408.16293

    2024 arXiv

  32. [44]

    In defense of rag in the era of long-context language models, 2024 b

    Tan Yu, Anbang Xu, and Rama Akkiraju. In defense of rag in the era of long-context language models, 2024 b . https://arxiv.org/abs/2409.01666

    2024 arXiv

  33. [45]

    bench: Extending long context evaluation beyond 100k tokens, 2024

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. bench: Extending long context evaluation beyond 100k tokens, 2024. https://arxiv.org/abs/2402.13718

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.