Pith. sign in

REVIEW 15 cited by

HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.02694 v3 pith:QSZXCGND submitted 2024-10-03 cs.CL cs.AI

Howard Yen , Tianyu Gao , Minmin Hou , Ke Ding , Daniel Fleischer , Peter Izsak , Moshe Wasserblat , Danqi Chen This is my paper · ORCID
classification cs.CLcs.AI
keywords modelstaskslclmsbenchmarksdiversehelmetdownstreamlong-context
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Many benchmarks exist for evaluating long-context language models (LCLMs), yet developers often rely on synthetic tasks such as needle-in-a-haystack (NIAH) or an arbitrary subset of tasks. However, it remains unclear whether these benchmarks reflect the diverse downstream applications of LCLMs, and such inconsistencies further complicate model comparison. We investigate the underlying reasons behind these practices and find that existing benchmarks often provide noisy signals due to limited coverage of applications, insufficient context lengths, unreliable metrics, and incompatibility with base models. In this work, we introduce HELMET (How to Evaluate Long-context Models Effectively and Thoroughly), a comprehensive benchmark encompassing seven diverse, application-centric categories. We also address several issues in previous benchmarks by adding controllable lengths up to 128K tokens, model-based evaluation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, we demonstrate that HELMET offers more reliable and consistent rankings of frontier LCLMs. Through a comprehensive study of 59 LCLMs, we find that (1) synthetic tasks like NIAH do not reliably predict downstream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlations with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when tasks require full-context reasoning or following complex instructions -- the gap widens as length increases. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and better predict other downstream performance; ultimately, we advocate for a holistic evaluation across diverse tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

    cs.CL 2026-07 conditional novelty 8.0 of 10

    WILDTRACE evaluates long-context models on 481 natural multi-hop evidence trails from 214 real documents, with top systems at 75.3% and geometry-specific weaknesses.

  2. Internalized Reasoning for Long-Context Visual Document Understanding

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Synthetic page-ranked reasoning traces plus low-strength model merging give a 32B VLM 58.3 on MMLongBenchDoc, beating a 235B teacher while cutting output tokens ~12× versus explicit reasoning.

  3. Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models

    cs.CL 2025-07 conditional novelty 7.0 of 10

    Ref-Long is a new long-context referencing benchmark on which all 13 tested LCLMs perform poorly, revealing a capability gap that simple retrieval benchmarks miss.

  4. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...

  5. HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.

  6. E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A controllable synthetic multi-step tool-use benchmark shows frontier LLMs remain unreliable on product-state changes, with Pass^3 below 60% (below 70% even with code).

  7. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  8. Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

    cs.LG 2026-07 accept novelty 6.0 of 10

    Jet-Long is a tuning-free bifocal RoPE method that dynamically sets remote group size from sequence length, recovering the base model within the pretrained window and beating prior zero-shot extenders on RULER, HELMET...

  9. Uncertainty-gated selection for block-sparse attention

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An uncertainty-gated router expands block-sparse attention budgets only on low-cutoff-margin queries, lifting paired recall by up to 28 pp and preserving most dense accuracy at 128K faster than dense.

  10. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  11. AbsenceBench: Language Models Can't Tell What's Missing

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLMs that ace Needle-in-a-Haystack struggle to identify deliberately omitted content, a new benchmark called AbsenceBench shows.

  12. NoLiMa: Long-Context Evaluation Beyond Literal Matching

    cs.CL 2025-02 conditional novelty 6.0 of 10

    NoLiMa shows that long-context language models' strong retrieval performance largely depends on literal matches, and that accuracy drops sharply once those matches are removed.

  13. LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

    cs.AI 2026-08 conditional novelty 5.0 of 10

    The paper reports that LongCat Sparse Attention matches full attention quality while reducing indexing overhead and supporting one-million-token training.

  14. MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.

  15. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

Pith tools