Pith. sign in

REVIEW 16 cited by

LooGLE: Can Long-Context Language Models Understand Long Contexts?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.04939 v2 pith:PRJT5G4D submitted 2023-11-08 cs.CL cs.AI

Jiaqi Li , Mengmeng Wang , Zilong Zheng , Muhan Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords llmslongcontextdependencytaskslooglemodelsshort
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understanding. LooGLE features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains. Human annotators meticulously crafted more than 1,100 high-quality question-answer pairs to meet the long dependency requirements. These pairs underwent thorough cross-validation, yielding the most precise assessment of LLMs' long dependency capabilities. The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings: (i) commercial models outperformed open-sourced models; (ii) LLMs excelled in short dependency tasks like short question-answering and cloze tasks but struggled with more intricate long dependency tasks; (iii) in-context learning and chaining thoughts offered only marginal improvements; (iv) retrieval-based techniques demonstrated substantial benefits for short question-answering, while strategies for extending context window length had limited impact on long context understanding. As such, LooGLE not only provides a systematic and comprehensive evaluation schema on long-context LLMs, but also sheds light on future development of enhanced models towards "true long-context understanding".

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...

  2. Training and Evaluating Diffusion Policies with Long Context Lengths

    cs.RO 2026-06 conditional novelty 6.0 of 10

    Naive long-context Diffusion Policies succeed with UNet+Cross-Attention and sufficient data; variable-history training cuts sample complexity in the low-data regime.

  3. vAttention: Verified Sparse Attention

    cs.LG 2025-10 conditional novelty 6.0 of 10

    vAttention is a sparse attention method that mixes heavy-hitter tokens with a statistically sized random sample to provide (ε, δ)-guaranteed approximation of full attention.

  4. Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Sticker-TTS uses three collaborating models to distill and reuse compact summaries (stickers) of past reasoning attempts, improving math benchmark accuracy at a claimed equivalent inference cost.

  5. A Distributed Learned Hash Table

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A learned order-preserving hash inside a Chord-style DHT lets range queries finish in roughly the same cost as single-key lookups, cutting latency and messages by 80-90%+ in tests.

  6. Exploring the Potential of LLMs for Serendipity Evaluation in Recommender Systems

    cs.IR 2025-07 conditional novelty 6.0 of 10

    Basic and multi-model LLM prompts can evaluate recommendation serendipity as well as or better than standard proxy formulas, reaching 21.5% Pearson correlation with user-study ratings.

  7. CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    LLM-based retrieval-augmented tuning of EDA flow parameters found a 9.9% lower-power configuration on one industrial core than classical optimizers.

  8. Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis

    cs.SE 2025-06 conditional novelty 6.0 of 10

    LLM4PFA combines iterative LLM-based constraint extraction with Z3 solving to reduce false positives in static analysis, achieving 72-96% false positive reduction and 93% recall on a new benchmark.

  9. Hierarchical Tree Search-based User Lifelong Behavior Modeling on Large Language Model

    cs.IR 2025-05 conditional novelty 6.0 of 10

    HiT-LBM chunks lifelong user behavior, uses process-rating-guided tree search to select interest summaries, and fuses them temporally to improve recommendation performance.

  10. NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    NovelHopQA is a new benchmark that pairs long novel excerpts with 1-4 hop questions and shows LLM accuracy drops consistently with both context length and reasoning depth.

  11. MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.

  12. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

  13. The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A VLM-LLM agentic pipeline and a new 251-person benchmark show that ordinary personal photo sets can reveal private attributes, including abstract traits like income and MBTI, at rates above human evaluators.

  14. CoDec: Prefix-Shared Decoding Kernel for LLMs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoDec combines KV-cache reads across requests that share a prefix, yielding average 1.9x decode-attention speedup and 120.9x less global memory traffic versus FlashDecoding.

  15. Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.

  16. Structured Memory Mechanisms for Stable Context Representation in Large Language Models

    cs.CL 2025-05 reject novelty 2.0 of 10

    A gated memory module with attention-based reading and forgetting is reported to improve NarrativeQA and dialogue consistency over GPT-2, BART, Longformer, and RETRO.

Pith tools