Pith. sign in

REVIEW 10 cited by

NoLiMa: Long-Context Evaluation Beyond Literal Matching

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.05167 v3 pith:6P2E2DW3 submitted 2025-02-07 cs.CL

Ali Modarressi , Hanieh Deilamsalehy , Franck Dernoncourt , Trung Bui , Ryan A. Rossi , Seunghyun Yoon , Hinrich Schütze This is my paper · ORCID
classification cs.CL
keywords contextsmodelsneedlehaystackliterallongnolimacapabilities
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves retrieving a "needle" (relevant information) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reasoning. However, in these benchmarks, models can exploit existing literal matches between the needle and haystack to simplify the task. To address this, we introduce NoLiMa, a benchmark extending NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associations to locate the needle within the haystack. We evaluate 13 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. Even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts. We publicly release the dataset and evaluation code at https://github.com/adobe-research/NoLiMa.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...

  2. HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.

  3. Where Facts Go Missing: A Layerwise Taxonomy and Per-Layer Attribution of Information Omission in Air-Gapped LLMAgent Pipelines

    cs.MA 2026-07 conditional novelty 6.0 of 10

    In a controlled 75,476-trial stress test, about 73% of omitted-fact failures in LLM agent pipelines are traced to deterministic middleware (redaction, pagination, truncation) rather than model behavior.

  4. CORVUS: Context Optimization and Reduction Via Underlying Synchronization for LLM Coding Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Decoupling file reads from file contents and re-injecting current repository state at each step cuts LLM coding-agent context use by up to 50% without hurting pass rates.

  5. UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.

  6. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  7. Positional Biases Shift as Inputs Approach Context Window Limits

    cs.CL 2025-08 conditional novelty 6.0 of 10

    The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.

  8. From Alerts to Intelligence: A Novel LLM-Aided Framework for Host-based Intrusion Detection

    cs.CR 2025-07 conditional novelty 6.0 of 10

    SHIELD, an LLM-aided pipeline combining a masked autoencoder, deterministic data augmentation, and multi-level prompting, detects host-based attacks with high precision on three public datasets.

  9. ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A new benchmark converts text-only QA datasets into text images and shows that vision-language models degrade sharply on long visually presented contexts.

  10. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

Pith tools