Pith. sign in

REVIEW 11 cited by

L-Eval: Instituting Standardized Evaluation for Long Context Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.11088 v3 pith:WFIOVMV7 submitted 2023-07-20 cs.CL

Chenxin An , Shansan Gong , Ming Zhong , Xingjian Zhao , Mukai Li , Jun Zhang , Lingpeng Kong , Xipeng Qiu This is my paper · ORCID
classification cs.CL
keywords evaluationmodelscontextlongl-evallanguagelclmsmetrics
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Recently, there has been growing interest in extending the context length of large language models (LLMs), aiming to effectively process long inputs of one turn or conversations with more extensive histories. While proprietary models such as GPT-4 and Claude can largely preserve the reasoning ability in an extended context, open-source models are still progressing through the early stages of development. To bridge this gap, we propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k$\sim$200k tokens). On the other hand, we investigate the effectiveness in evalution metrics for LCLMs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment, and thus we strongly advocate for length-instruction-enhanced (LIE) evaluation and employing LLM judges. We conducted a comprehensive study of 4 popular commercial LLMs and 12 open-source counterparts using the L-Eval benchmark. Our empirical findings offer useful insights into the study of LCLMs and lay the groundwork for the development of more principled evaluation of these models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Language agents' believability and goal achievement decline over multi-episode social interactions, and curated memory summaries only partially close the gap with humans.

  2. AbsenceBench: Language Models Can't Tell What's Missing

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLMs that ace Needle-in-a-Haystack struggle to identify deliberately omitted content, a new benchmark called AbsenceBench shows.

  3. NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    NovelHopQA is a new benchmark that pairs long novel excerpts with 1-4 hop questions and shows LLM accuracy drops consistently with both context length and reasoning depth.

  4. X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

    cs.LG 2026-07 conditional novelty 5.0 of 10

    X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.

  5. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

  6. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  7. MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents

    cs.CL 2025-06 conditional novelty 5.0 of 10

    MemBench introduces a multi-scenario, multi-level memory benchmark for LLM agents, evaluating factual and reflective memory across participation and observation settings.

  8. MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.

  9. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

  10. SELF: Self-Extend the Context Length With Logistic Growth Function

    cs.CL 2025-05 conditional novelty 5.0 of 10

    SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.

  11. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

Pith tools