Pith. sign in

REVIEW 4 major objections 5 minor 42 references

Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art long-context language models cannot reliably attribute a key to the set of documents that reference it, even in a 24K-token context far below their advertised limits.

desk verdict A genuinely new and well-built long-context benchmark, with a human-baseline confound that slightly overstates the capability gap but doesn't sink the contribution. read the letter →

arxiv 2507.09506 v2 pith:AUOM5ZVB submitted 2025-07-13 cs.CL

Junjie Wu , Gefei Gu , Yanan Zheng , Dit-Yan Yeung , Arman Cohan This is my paper · ORCID
classification cs.CL
keywords long-contextlanguagemodelsreferencingbenchmarkdocumentattributionretrievalexact-matchaccuracyevaluationLLMcapabilities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that today's long-context language models, despite strong performance on retrieval-style benchmarks, cannot reliably perform long-context referencing: given a set of indexed documents and a specific key, they fail to output the exact set of documents that reference that key. The authors build Ref-Long, a three-subset benchmark spanning synthetic to real-world documents, and evaluate 13 models on it. The headline evidence is that on the hardest synthetic setting, with the key occurring across multiple documents in a 24K-token context, the strongest tested model, GPT-4o, reaches only 19% exact-match accuracy, while human annotators score 92%. This matters because practical uses of long contexts, such as legal citation, financial reporting, and scientific literature review, depend on attributing information to the correct source document.

What carries the argument

The central object is the Ref-Long task itself: M numbered documents, each containing a subset of N distinct keys, with the model required to output the set of document indexes referencing a queried key, scored by exact match. The task forces a model to track key–document co-occurrence relationships rather than merely retrieve a needle string. The benchmark is built in three variants: Ref-Long-A, where a template sentence 'The little penguin counted {num} ★' is inserted into unrelated essays (abrupt key in fluent context); Ref-Long-F, built on SummHay with paraphrased insight statements as keys (fluent key in fluent context); and Ref-Long-Paper, where the keys are titles of arXiv papers and the target documents are papers that cite them (real-world citations). Alongside the benchmark, the paper's analysis machinery includes human evaluation, strategy-prompt variants, and error-type classification (reference less, reference more, both).

What would settle it

Run the dictionary-strategy prompt on the Multi-Hard-24K setting of Ref-Long-A with a frontier model; if it reaches near the human level of 92% exact-match accuracy (or if any LCLM exceeds roughly 50% exact-match without task-specific fine-tuning), the claim that current models fundamentally lack referencing capability would be undermined. A second check: recruit a larger group of annotators, say twenty, on the same 50 tasks; if their accuracy falls below 60%, the benchmark's human-feasibility premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that state-of-the-art long-context language models lack long-context referencing: given a set of indexed documents and a specific key, they cannot reliably output the indexes of exactly those documents that reference the key. The authors establish this through Ref-Long, a benchmark with three subsets that progress from synthetic to realistic scenarios, evaluated across 13 closed- and open-source LCLMs. The headline result is that on the hardest synthetic setting, where the key appears across multiple documents in a 24K-token context, the best model (GPT-4o) reaches only 19% exact-match accuracy, and performance continues to drop as input length grows. Human annotators scored 92% on the same tasks, and the paper shows that neither changing the query format, adding a human-style dictionary-building strategy, nor fine-tuning on Ref-Long data closes the gap.

Load-bearing premise

The central finding rests on the assumption that the synthetic Ref-Long-A task—a stylized template sentence inserted into unrelated essays—is a valid and representative test of general long-context referencing, and that human-level performance estimated from two annotators on 50 tasks is a sound yardstick for what a capable referencer should achieve.

Editorial extensions

If this is right

  • Advertised context windows overstate usable long-context ability: exact-match referencing accuracy collapses long before models reach their stated maximum input lengths.
  • Retrieval-style benchmarks such as needle-in-a-haystack, counting-stars, and RULER overestimate long-context understanding because they never force models to relate a key to the documents that mention it.
  • Long-context fine-tuning recipes (ProLong) and citation-QA fine-tuning (LongCite-8B) do not transfer to referencing; only fine-tuning directly on Ref-Long tasks helps, and only on the easiest settings.
  • Prompt-level interventions, including injecting the human dictionary-building strategy, trigger some referencing in GPT-4o but the gains are far below the human baseline.
  • Error analysis shows that models both miss documents that reference the key (numeric keys) and add documents that reference only unrelated keys (natural-language keys), meaning the deficit affects both recall and precision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the deficit persists in deployed systems, retrieval-augmented generation will inherit citation errors whenever the model itself selects sources; a direct test is whether fine-grained citation quality in long-context QA correlates with Ref-Long scores.
  • A natural mechanistic prediction, testable with attention-likelihood probes, is that positional biases (lost-in-the-middle effects, attention decay) drive the failure rather than the semantic content of the keys.
  • Because task difficulty should track key ambiguity, a graded Ref-Long diagnostic that interpolates from pure retrieval to full referencing could localize exactly where models transition from finding a string to associating it with a document.
  • The synthetic subset's clean ground truth makes Ref-Long-A a useful controlled probe for studying how long-context models track repeated key–document associations, independent of the realism debate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Ref-Long, a benchmark for evaluating long-context referencing: given a set of indexed documents and a key, a model must output the indexes of all documents that mention or reference that key. The benchmark has three subsets: Ref-Long-A (synthetic: a template sentence 'The little penguin counted {num} ★' inserted into Paul Graham essays), Ref-Long-F (fluent keys in fluent context, built on SummHay insights), and Ref-Long-Paper (real arXiv citation relationships). The authors evaluate 13 long-context language models (LCLMs) across input lengths from 8K to 75K tokens, reporting exact-match accuracy and F1. The central finding is that all models, including GPT-4o, perform poorly on the multi-key settings, with the best Exact Acc on Multi-Hard-24K being only 19.00%. The paper also reports a human evaluation on 50 Multi-Hard-24K tasks (two annotators), prompt-format variants, a strategy prompt inspired by human annotators, a fine-tuning experiment on Llama-3.1-8B, and an error analysis. The authors conclude that current LCLMs lack the capability for long-context referencing even at input lengths far below their claimed maximum context sizes.

Significance. If the benchmark is valid, it fills a real gap: existing long-context benchmarks emphasize retrieval or general QA, whereas Referencing requires attributing a key to all relevant documents, a task with practical applications in legal, financial, and scholarly domains. The paper's strengths include the breadth of evaluated models (13 LCLMs), the three-subset design spanning synthetic to realistic scenarios, the public release of data and code, and the multi-perspective analyses (human evaluation, prompt engineering, fine-tuning, error taxonomy). The realistic subsets (Ref-Long-F and Ref-Long-Paper) provide mitigating evidence that the synthetic task's format is not the sole driver of poor model performance. However, the human evaluation—a key pillar of the 'manageable for humans' claim—is currently too weak to support the strong capability-gap conclusion, and the synthetic task's reliance on visually distinctive repeated strings warrants a control condition. The benchmark itself is a useful contribution, but the strength of the paper's claims needs to be reined in or better supported.

major comments (4)
  1. [§4.3, Table 2] The human baseline is plausibly confounded by visual search. The two annotators averaged 123.95 seconds per 24K-token task, which is roughly 194 tokens per second—two orders of magnitude above normal silent reading rates (about 4–5 tokens/s). This pace is only credible if the annotators used search tools (e.g., Ctrl+F) or visual scanning of the distinctive 'The little penguin counted {num} ★' template, where the star symbol makes the target highly salient. The paper does not report whether annotators were allowed or observed using such tools. This confound undermines the claim that Ref-Long tasks are 'manageable for humans' in the sense of unaided reading-based referencing, and therefore weakens the inference that low LCLM scores represent a capability gap rather than a task artifact. Please specify the exact experimental protocol (instructions given to annotators, allowed tools, whether screens were recorded) or run a no-search human baseline on a subset of tasks.
  2. [§4.3, Table 2] The human evaluation uses only two annotators on 50 tasks, all drawn from the Multi-Hard-24K setting of Ref-Long-A. No human evaluation was performed on Ref-Long-F or Ref-Long-Paper, so the 'manageable for humans' conclusion does not transfer to the realistic subsets, where the keys are natural-language spans or paper titles and the task likely requires genuine reading rather than pattern matching. With two annotators, the reported 92% human Exact Acc has very wide uncertainty (e.g., the 95% Wilson interval is roughly 81–97%, and a single disagreement changes the point estimate by 2 percentage points). Please report per-annotator scores, an inter-annotator agreement statistic (e.g., Cohen's kappa on document-level judgments), and confidence intervals, or restrict the human-based claim to the specific synthetic setting.
  3. [§4.1, Table 1 and Table 9] The Ref-Long-A task may be measuring exhaustive multi-occurrence scanning rather than contextual referencing. The abrupt, visually distinctive template sentence is inserted into unrelated essays, and the key is a number with a star symbol; exact-match evaluation then requires the model to enumerate all documents containing that exact string. The sharp collapse from 94% Ex Acc on Single-Hard-24K to 19% on Multi-Hard-24K (Table 9) suggests that the difficulty lies in exhaustive enumeration across many occurrences rather than in understanding contextual relationships between a key and document-level content. The paper should include a control condition that isolates the enumeration component—for example, a retrieval-only version where the model must output any one document index for the key, or a version where distractors contain semantically related but not identical keys. Without such a control, the synthetic-subset results are ambiguous as evidence for a 'referencing capability' deficit, although the realistic subsets partially mitigate this concern.
  4. [§4.5, Table 5] The fine-tuning experiment uses a single base model (Llama-3.1-Ins-8B), only 500 training tasks from Multi-Easy-8K, and evaluation only on the Easy setting. The conclusion that 'fine-tuning on long-context data alone cannot fully overcome LCLMs' limitations on Ref-Long tasks' is too broad for this evidence. The experiment can show that one lightweight fine-tuning recipe does not solve the task, but it cannot support a general claim about the effectiveness of fine-tuning for long-context referencing. Please either soften the conclusion or add experiments with more training data, longer contexts, or additional base models.
minor comments (5)
  1. [§4.3] The sentence 'as the their time taken to solve each task is also recorded' contains a grammatical error ('as the their'); please rephrase.
  2. [Appendix B, Figures 11–12] The fruit-name prompts contain the typo 'eated' in 'The little penguin eated {fruit}'; this should be 'ate' or 'eats'.
  3. [§4.1 and Appendix A] The 'aggregative' sampling method is described only briefly; please define explicitly whether the 8K task is a strict prefix of the 16K task (same documents, same key, same order) and whether this dependency is accounted for in any statistical comparisons across input lengths.
  4. [§4.3] The paper states that o1 was evaluated but does not report the number of tasks or the reason its performance was poor; please provide at least the sample size and the exact prompting settings for o1, or state that it was evaluated on the same 50 tasks.
  5. [Limitations] The Limitations section appropriately acknowledges the restricted model coverage and the limited topic diversity of SummHay, but the same limitations are not echoed in the abstract or introduction; please add a sentence in the introduction noting that the benchmark covers only three news topics and one synthetic template family, so generalizability to other domains should be treated with caution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Ref-Long's conclusions are measured against externally defined ground truth, not derived from fitted parameters or self-citation.

full rationale

The paper's central claim—that LCLMs struggle with long-context referencing—is an empirical observation on a benchmark whose labels are independent of the evaluated models. In Ref-Long-A, synthetic template sentences are inserted at known document positions, so the ground-truth document indexes are fixed by construction and are not derived from any model behavior (§4.1). Ref-Long-F inherits insight-to-document assignments from SummHay (§5.1), and Ref-Long-Paper uses real arXiv citation relations (§5.2). No parameter is fitted to the models and then renamed a prediction; the task definitions do not entail the measured failure rates, since strong models score high on easy settings (e.g., GPT-4o reaches 100.00 Ex Acc on Single-Hard-8K in Table 9) and low on hard settings. The one self-citation (Wu et al., 2025, used in §1 only to motivate the human-evaluation design) is not load-bearing: the human baseline is an empirical measurement, and its possible confound from visual search or Ctrl+F (annotators averaged 123.95s per 24K-token task, §4.3) is a validity threat rather than circularity. Even if that confound were accepted, the realistic subsets are evaluated without relying on the human baseline. Thus no claim reduces by construction to its own inputs, and there is no meaningful circularity in the derivation chain. The skeptical concerns about the human evaluation pertain to benchmark validity, not to circularity, and do not raise the circularity score under the stated rules.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

Ref-Long is an evaluation benchmark, not a physical theory, so there are no invented entities. The free parameters are dataset design choices (key ranges, document counts, topic selection) that calibrate task difficulty but are not fitted to model performance. The axioms are the domain assumptions underlying the benchmark's validity as a measure of referencing capability, with the synthetic task representativeness and the small human evaluation being the most fragile.

free parameters (4)
  • Key ranges for Ref-Long-A (Easy/Medium/Hard) = [0,100), [0,60), [0,20)
    These ranges are hand-chosen to control key frequency across documents and thus task difficulty. A smaller range leads to more repeated keys and harder tasks. This is a design choice, not fitted to model performance.
  • Number of documents M per task = 8/16/24 (Ref-Long-A), 8/16/24/32/40/48/56 (Ref-Long-F), 8/12/16/20 (Ref-Long-Paper)
    The paper varies M to produce input lengths from 8K to 75K tokens. These values are experiment design parameters, chosen to create a length-difficulty gradient.
  • Topics selected from SummHay = Foot Locker, Twitter, Financial Market
    Only 3 of the 10 SummHay topics are used for Ref-Long-F, selected as representative news topics. This is a hand-chosen subset that limits topic diversity.
  • Number of seed papers and distractors in Ref-Long-Paper = 47 seed papers, 34 distractors
    The size of the paper collection was chosen to balance task realism and construction effort; the exact set is not enumerated in the paper, relying on the released dataset.
assumptions (5)
  • domain assumption Exact match accuracy on document index sets is a valid primary metric for referencing capability.
    §3.2 defines Ex Acc as the main metric, requiring exact set equality. This assumes that partial referencing is not considered successful, which may be stringent but is common in set-prediction benchmarks.
  • domain assumption The synthetic template sentence task (penguin counting stars) is a representative and fair proxy for real-world long-context referencing.
    §4.1 constructs Ref-Long-A by inserting irrelevant template sentences into essays. The paper assumes this captures the core skill of attributing a key to documents, an assumption supported by the follow-up realistic subsets but not proven.
  • domain assumption The human evaluation with two PhD student annotators on 50 tasks is sufficiently representative to estimate human-level performance.
    §4.3 reports 42/50 identical answers between two annotators. This small sample is used to claim humans can solve the tasks, but the sample size and annotator pool limit generalizability.
  • domain assumption The arXiv papers selected for Ref-Long-Paper (published after March 2024) are outside the training data of most evaluated LCLMs.
    §5.2 uses this assumption to avoid data contamination, but the authors themselves note that Qwen2.5-Ins-72B may have seen the papers, so the assumption is partially violated.
  • standard math Greedy decoding with temperature 0 yields stable, reproducible model outputs.
    Appendix D states greedy decoding; this is standard practice in evaluation and makes outputs deterministic, though it can understate a model's true capability if sampling helps.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models." pith.science (2026). https://pith.science/paper/AUOM5ZVB

@misc{pith2026250709506,
  author       = {Pith},
  title        = {Pith review of: Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUOM5ZVB}},
  note         = {Machine review of arXiv:2507.09506}
}
read the original abstract

Long-context language models (LCLMs) have exhibited impressive capabilities in long-context understanding tasks. Among these, long-context referencing -- a crucial task that requires LCLMs to attribute items of interest to specific parts of long-context data -- remains underexplored. To bridge this gap, this paper proposes Referencing Evaluation for Long-context Language Models (Ref-Long), a novel benchmark designed to assess the long-context referencing capability of LCLMs. Specifically, Ref-Long requires LCLMs to identify the indexes of documents that reference a specific key, emphasizing contextual relationships between the key and the documents over simple retrieval. Based on the task design, we construct three subsets ranging from synthetic to realistic scenarios to form the Ref-Long benchmark. Experimental results of 13 LCLMs reveal significant shortcomings in long-context referencing, even among advanced models like GPT-4o. To further investigate these challenges, we conduct comprehensive analyses, including human evaluations, task format adjustments, fine-tuning experiments, and error analyses, leading to several key insights. Our data and code can be found in https://github. com/wujunjie1998/Ref-Long.

Figures

Figures reproduced from arXiv: 2507.09506 by the authors.

Figure 1
Figure 1. An example Ref-Long task. Given a long￾context input with indexed documents that include sev￾eral NBA players names, an LCLM is asked to give the indexes of documents that reference “Durant” (marked as red). Names other than “Durant” are marked as blue. tioners need to quickly identify the specific chapter within the relevant legal code when presented with a particular case or provision, and financial profes￾sionals… view at source ↗
Figure 2
Figure 2. LCLMs’ performances drop consistently as input length grows. See Table [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Example Prompt for Ref-Long-A: Instructions [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Example Prompt for Ref-Long-A: Instructions [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 6
Figure 6. Figure 6: Example Prompt for Ref-Long-F: Instructions [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Example Prompt for Ref-Long-Paper: In￾structions Followed by Documents, where “ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs” is the specific key. Example Prompt (B) [User Input]: You will find several papers indexed with numbers below. Eac…
Figure 9
Figure 9. Figure 9: Example Prompt for Ref-Long-A: Instructions [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 8
Figure 8. Figure 8: Example Prompt for Ref-Long-Paper: In￾structions Preceding Documents, where "ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs" is the specific key. Example Prompt (A) [User Input]: You will find several documents indexed with numbers below. Ea…
Figure 12
Figure 12. Figure 12: Example Prompt for Ref-Long-A: Instruc￾tions Followed by Documents, with fruit name instead, where “apple” is the interested key. Example Prompt (A) [User Input]: You will find several documents indexed with numbers below. Each document is divided by ’\n\n’. Please re…
Figure 13
Figure 13. Figure 13: Example Prompt for Ref-Long-NBA: Instruc [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Example Prompt for Ref-Long-NBA: In￾structions Preceding Documents, where “Paul George” is the interested key. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 23 canonical work pages

  1. [1]

    Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan...

    arXiv 2024

  2. [2]

    AI@Meta. 2024. https://ai.meta.com/blog/meta-llama-3-1/ Introducing llama 3.1: Our most capable models to date

    work page 2024

  3. [3]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. https://aclanthology.org/2024.acl-long.776 L -eval: Instituting standardized evaluation for long context language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1...

    work page 2024

  4. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024 b . Longalign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376--1395

    work page 2024

  5. [6]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024 c . Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119--3137

    work page 2024

  6. [7]

    Jiajun Bao, Junjie Wu, Yiming Zhang, Eshwar Chandrasekharan, and David Jurgens. 2021. Conversations gone alright: Quantifying and predicting prosocial outcomes in online conversations. In Proceedings of the Web Conference 2021, pages 1134--1145

    work page 2021

  7. [8]

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660

    arXiv 2024

  8. [9]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465--6488

    work page 2023

Show all 42 references
  1. [10]

    Gemini. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . Preprint, arXiv:2403.05530

    2024 arXiv

  2. [11]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  3. [12]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

  4. [13]

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker - Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex ...

    2024 arXiv

  5. [14]

    Greg Kamradt. 2023 a . https://github.com/gkamradt/LLMTest_NeedleInAHaystack Needle in a haystack - pressure testing llms

    2023

  6. [15]

    Gregory Kamradt. 2023 b . Needle in a haystack - pressure testing llms. https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main

    2023

  7. [16]

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A “novel” challenge for long-context language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17048--17085

    2024

  8. [17]

    Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems, 37:106519--106554

    2024

  9. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  10. [19]

    Philippe Laban, Alexander Richard Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context llms and rag systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885--9903

    2024

  11. [20]

    Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2024

  12. [21]

    Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. 2024. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963

    2024

  13. [22]

    Yushan Liu, Zili Wang, and Ruifeng Yuan. 2024. Querysum: A multi-document query-focused summarization dataset augmented with similar query clusters. In AAAI , pages 18725--18732. AAAI Press

    2024

  14. [23]

    Mmlongbench-doc: Benchmarking long-context document understanding with visualizations

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. In The Thirty-eight Conference on Neural Information Processing System...

  15. [24]

    OpenAI. 2024 a . https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  16. [25]

    OpenAI. 2024 b . https://cdn.openai.com/o1-system-card-20241205.pdf Openai o1 system card

    2024

  17. [26]

    Needle threading: Can llms follow threads through near-million-scale haystacks? In The Thirteenth International Conference on Learning Representations

    Jonathan Roberts, Kai Han, and Samuel Albanie. Needle threading: Can llms follow threads through near-million-scale haystacks? In The Thirteenth International Conference on Learning Representations

  18. [27]

    Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403.11802

    2024 arXiv

  19. [28]

    Zecheng Tang, Keyan Zhou, Juntao Li, Baibei Ji, Jianye Hou, and Min Zhang. 2024. L-citeeval: Do long-context models truly leverage context for responding? arXiv preprint arXiv:2410.02115

    2024 arXiv

  20. [29]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  21. [30]

    Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. 2024. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:...

    2024 arXiv

  22. [31]

    Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.322 Leave no document behind: Benchmarking long-context LLM ...

    2024 doi

  23. [32]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  24. [33]

    Junjie Wu, Mo Yu, Lemao Liu, Dit-Yan Yeung, and Jie Zhou. 2025. https://aclanthology.org/2025.naacl-long.423/ Understanding LLM s' fluid intelligence deficiency: An analysis of the ARC task . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the A...

    2025

  25. [34]

    Liyan Xu, Jiangnan Li, Mo Yu, and Jie Zhou. 2024. Fine-grained modeling of narrative context: A coherence perspective via retrospective questions. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5822--5838

    2024

  26. [35]

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2024. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694

    2024 arXiv

  27. [36]

    Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xiaochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023. Personality understanding of fictional characters during book reading. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2023

  28. [37]

    Mo Yu, Lemao Liu, Junjie Wu, Tsz Ting Chung, Shunchi Zhang, Jiangnan Li, Dit-Yan Yeung, and Jie Zhou. 2025. https://aclanthology.org/2025.naacl-long.569/ The stochastic parrot on LLM `s shoulder: A summative assessment of physical concept understanding . In Proceedings of the ...

    2025

  29. [38]

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. 2024 a . Longcite: Enabling llms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897

    2024 arXiv

  30. [39]

    Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang, Junhao Liu, Longze Chen, Run Luo, and Min Yang. 2024 b . Marathon: A race through the realm of long context with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024

  31. [40]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024 c . ∞ bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational L...

    2024

  32. [41]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  33. [42]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  34. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.