Pith. sign in

REVIEW 6 cited by

RepoQA: Evaluating Long Context Code Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.06025 v1 pith:BIEOVCJD submitted 2024-06-10 cs.SE cs.CLcs.LG

Jiawei Liu , Jia Le Tian , Vijay Daita , Yuxiang Wei , Yifeng Ding , Yuhan Katherine Wang , Jun Yang , Lingming Zhang This is my paper · ORCID
classification cs.SEcs.CLcs.LG
keywords llmscoderepoqamodelscontextlong-contextneedleunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advances have been improving the context windows of Large Language Models (LLMs). To quantify the real long-context capabilities of LLMs, evaluators such as the popular Needle in a Haystack have been developed to test LLMs over a large chunk of raw texts. While effective, current evaluations overlook the insight of how LLMs work with long-context code, i.e., repositories. To this end, we initiate the RepoQA benchmark to evaluate LLMs on long-context code understanding. Traditional needle testers ask LLMs to directly retrieve the answer from the context without necessary deep understanding. In RepoQA, we built our initial task, namely Searching Needle Function (SNF), which exercises LLMs to search functions given their natural-language description, i.e., LLMs cannot find the desired function if they cannot understand the description and code. RepoQA is multilingual and comprehensive: it includes 500 code search tasks gathered from 50 popular repositories across 5 modern programming languages. By evaluating 26 general and code-specific LLMs on RepoQA, we show (i) there is still a small gap between the best open and proprietary models; (ii) different models are good at different languages; and (iii) models may understand code better without comments.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

    cs.SE 2026-07 conditional novelty 7.0 of 10

    RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.

  2. Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Frontier LLMs systematically fail simple long-context subsequence-retrieval tasks as difficulty is scaled along decoys, quantifier complexity, search space, and predicate arity.

  3. RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

    cs.SE 2026-08 conditional novelty 6.0 of 10

    RepoProbe uses GitHub Discussions and checklist-based verification to measure repository-level code understanding in LLMs, finding a persistent gap between fluent explanations and evidence-grounded correctness plus a ...

  4. Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

    cs.SE 2026-07 conditional novelty 6.0 of 10

    QA-driven pre-repair knowledge acquisition (Questioner + Answerer, then Resolver) raises SWE-bench Verified Pass@1 by up to 4.4 points over Mini-SWE-Agent and other pre-repair methods.

  5. Structural Code Search using Natural Language Queries

    cs.SE 2025-07 conditional novelty 6.0 of 10

    An LLM-based system translates English code search questions into Semgrep and GQL queries, reaching 55-70% precision/recall on a new synthetic 400-query Java benchmark.

  6. CODENS: Transforming Code Changes into Living, Accessible, and Queryable Documentation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    CODENS incrementally builds a semantic software knowledge graph from pull requests and answers repository-level questions via vector, multi-hop, and agent-guided retrieval.

Pith tools