Pith. sign in

REVIEW 11 cited by

Long Code Arena: a Set of Benchmarks for Long-Context Code Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11612 v1 pith:VZMYT4CN submitted 2024-06-17 cs.LG cs.AIcs.IRcs.SE

Egor Bogomolov , Aleksandra Eliseeva , Timur Galimzyanov , Evgeniy Glukhov , Anton Shapkin , Maria Tigina , Yaroslav Golubev , Alexander Kovrigin
show 3 more authors
Arie van Deursen Maliheh Izadi Timofey Bryksin
This is my paper · ORCID
classification cs.LGcs.AIcs.IRcs.SE
keywords codeprocessingcontextbenchmarkshuggingfacelongarenadataset
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes have increased by orders of magnitude over the last few years. However, there is a shortage of benchmarks for code processing that go beyond a single file of context, while the most popular ones are limited to a single method. With this work, we aim to close this gap by introducing Long Code Arena, a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions based on popular LLMs to showcase the usage of the dataset and to simplify adoption by other researchers. We publish the benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace Hub for all the datasets, and link to the GitHub repository with baselines: https://huggingface.co/spaces/JetBrains-Research/long-code-arena.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ORCA-bench: How Ready Are Language Model Agents for Oncall?

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Frontier coding agents achieve at most 25.3% RCA accuracy on realistic medium oncall tasks and 10% on hard ones in a production-fidelity telemetry-plus-code benchmark.

  2. Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Frontier LLMs systematically fail simple long-context subsequence-retrieval tasks as difficulty is scaled along decoys, quantifier complexity, search space, and predicate arity.

  3. Retrieval-Oriented Code Representations in Agentic Bug Localization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Role-aware file summaries give the best cost-effectiveness for file-level bug localization, beating file paths by up to 40% Hit@5 at far smaller footprint than raw source.

  4. UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.

  5. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

    cs.DC 2026-02 unverdicted novelty 6.0 of 10

    SPEED-Bench is a new standardized benchmark for speculative decoding that supplies semantically diverse qualitative data and throughput-oriented splits across concurrency levels, integrated with vLLM and TensorRT-LLM.

  6. How can we assess human-agent interactions? Case studies in software agent design

    cs.AI 2025-10 conditional novelty 6.0 of 10

    PULSE combines sparse human ratings with prediction-powered inference to cut confidence intervals by ~40% and shows LLM choice matters more than scaffolding for user satisfaction.

  7. Robot builds a robot's brain: AI generated drone command and control station hosted in the sky

    cs.RO 2025-08 conditional novelty 6.0 of 10

    An AI model generated a fully functional drone command-and-control website, deployed on the drone itself, and a real flight demonstrated it.

  8. Challenge on Optimization of Context Collection for Code Completion

    cs.SE 2025-10 conditional novelty 5.0 of 10

    A new Python/Kotlin benchmark and competition show retrieval-based context can improve fill-in-the-middle code completion chrF scores, with a caveat about possible data leakage.

  9. CoRet: Improved Retriever for Code Editing

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoRet fine-tunes a code encoder with a repository-level likelihood loss and call-graph context, improving retrieval recall on SWE-bench and Long Code Arena.

  10. MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.

  11. SPRINT: An Assistant for Issue Report Management

    cs.SE 2025-02 conditional novelty 4.0 of 10

    SPRINT combines three machine-learning models for duplicate detection, severity prediction, and bug localization into a single open-source GitHub application, with a preliminary evaluation and a five-developer usabili...

Pith tools