Pith. sign in

REVIEW 5 cited by

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.17419 v2 pith:VHUHDIN3 submitted 2024-06-25 cs.CL cs.AI

Minzheng Wang , Longze Chen , Cheng Fu , Shengyi Liao , Xinghua Zhang , Bingli Wu , Haiyang Yu , Nan Xu
show 6 more authors
Lei Zhang Run Luo Yunshui Li Min Yang Fei Huang Yongbin Li
This is my paper · ORCID
classification cs.CLcs.AI
keywords long-contextdocumentloongllmsanswerbenchmarkscapabilitiescases
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually catching up. However, existing benchmarks employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-context applications. To bridge this gap, we propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Unlike typical document QA, in Loong's test cases, each document is relevant to the final answer, ignoring any document will lead to the failure of the answer. Furthermore, Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Extensive experiments indicate that existing long-context language models still exhibit considerable potential for enhancement. Retrieval augmented generation (RAG) achieves poor performance, demonstrating that Loong can reliably assess the model's long-context modeling capabilities.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

    cs.LG 2025-06 conditional novelty 7.0 of 10

    RAG in in-context linear regression has an exact bias-variance tradeoff and a finite-sample bound revealing a generalization ceiling as retrieved examples grow.

  2. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  3. Respecting Temporal-Causal Consistency: Entity-Event Knowledge Graphs for Retrieval-Augmented Generation

    cs.IR 2025-06 conditional novelty 6.0 of 10

    A mention-level entity-event knowledge graph for RAG modestly improves temporal-causal question answering on a new narrative benchmark, with gains mostly coming from adding HyDE-style hypothetical answers.

  4. NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    NovelHopQA is a new benchmark that pairs long novel excerpts with 1-4 hop questions and shows LLM accuracy drops consistently with both context length and reasoning depth.

  5. Evaluating Hierarchical Clinical Document Classification Using Reasoning-Based LLMs

    cs.CL 2025-07 conditional novelty 3.0 of 10

    Across 1,500 MIMIC-IV discharge summaries and 11 LLMs, no model exceeded 57% F1 on ICD-10 coding, with reasoning-labeled models slightly ahead of others, but the comparison is confounded by model differences.

Pith tools