Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CodeRepoQA extracts 585,687 multi-turn QA dialogues from GitHub issues and reports that current LLMs fall well short of human maintainer responses.

desk verdict A useful dataset idea undermined by a likely pull-request contamination in the crawling pipeline; the benchmark needs a data fix and stronger evaluation stats before it can be trusted. read the letter →

arxiv 2412.14764 v1 pith:4ER5YLLV submitted 2024-12-19 cs.SE cs.AI

Ruida Hu , Chao Peng , Jingyi Ren , Bo Jiang , Xiangxin Meng , Qinyun Wu , Pengfei Gao , Xinchen Wang
show 1 more author
Cuiyun Gao
This is my paper · ORCID
classification cs.SEcs.AI
keywords questionansweringsoftwareengineeringlargelanguagemodelsGitHubissuesmulti-turndialoguerepository-levelbenchmarkcodeQAevaluationmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CodeRepoQA, a public benchmark of 585,687 multi-turn question-answer dialogues mined from issues in 30 popular GitHub repositories covering five programming languages. Its purpose is to test language models on repository-level software engineering questions, the kind of messy, multi-participant exchanges that single-turn code QA benchmarks do not capture. Using the final reply of a repository maintainer as ground truth, the authors evaluate ten language models with BLEU, ROUGE-L, ROUGE-1, and Edit Similarity, finding that even the best model leaves a wide gap to real maintainer responses. They also report that medium-length contexts, around the 40 percent mark of the length distribution, produce the highest scores.

What carries the argument

The central object is the CodeRepoQA entry: a structured GitHub issue thread containing repository metadata, author association roles, and the full multi-turn dialogue, with the final maintainer reply designated as the reference answer. The construction pipeline filters raw issues by character count, duplicate markers, external links, event count, robot detection, and participant count to keep only authentic interactions. At evaluation time, the dialogue history is fed to a language model, and the generated reply is scored against the maintainer reference using BLEU, ROUGE-L, ROUGE-1, and Edit Similarity.

What would settle it

Manually annotate a random sample of, say, 200 CodeRepoQA threads: if human raters find that the final maintainer message is not clearly the best or most complete answer in a substantial share of cases, or that model responses judged correct by humans still have near-zero BLEU and ROUGE scores, the benchmark's ground-truth and metric assumptions would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that repository-level, multi-turn QA performance can be measured at scale by turning GitHub issue threads into benchmark entries, and that current LLMs are still far from matching human maintainers on this task. CodeRepoQA is built by crawling 30 popular repositories, filtering for genuine, non-duplicate, non-robotic, multi-participant conversations, and yielding 585,687 entries with an average of 6.62 dialogue turns. Evaluation treats the last response from a MEMBER, AUTHOR, or CONTRIBUTOR as the reference answer; across ten models the best average score is 0.1826, with ROUGE-1 reaching only 0.2470, which the authors read as evidence that generated answers miss key content from the real replies. On context length, the 40 percent length group scores highest on average (0.1698) while the longest contexts score lowest (0.1518), supporting the paper's conclusion that medium-length contexts are most favorable.

Load-bearing premise

The load-bearing premise is that the last reply from a repository maintainer is the correct answer to the whole preceding dialogue, and that word-overlap scores gauge answer quality.

Editorial extensions

If this is right

  • Researchers can score new or existing LLMs on 585,687 real software-engineering dialogues without additional crawling or manual annotation.
  • Model scale alone does not predict QA quality: the 7B CodeQwen model outperforms GPT-4, and the 236B DeepSeek-Coder-V2 is beaten by its 33B and 6.7B siblings.
  • The context-length result implies that feeding a model a medium-length slice of the thread is more likely to maximize measured scores than sending the shortest or longest contexts.
  • The benchmark structure supports per-language and per-repository breakdowns, allowing targeted analysis of where models struggle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because ground truth is defined as the last maintainer reply, the benchmark implicitly assumes a single correct resolution; a follow-up could compare human judgments of answer adequacy across all replies in a thread.
  • Lexical-overlap metrics reward surface similarity, so a model that paraphrases the maintainer's wording would score well; a semantic or human evaluation on a subset could test whether low BLEU scores truly indicate poor answers.
  • The context-length finding may be confounded with issue difficulty, since harder issues tend to attract longer discussions; matching issues of similar difficulty across length groups would separate the two explanations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CodeRepoQA, a large-scale multi-turn question-answering benchmark constructed from GitHub issues across 30 popular repositories in five programming languages, yielding 585,687 entries with an average of 6.62 dialogue turns per entry. The authors describe a two-stage pipeline of crawling via the GitHub REST API followed by automatic filtering, and they evaluate ten commercial and open-source LLMs using BLEU, ROUGE-1, ROUGE-L, and Edit Similarity. The main empirical claims are that LLMs remain limited in software-engineering QA and that medium-length contexts lead to better performance.

Significance. If the benchmark is valid as described, it would be a valuable public resource: it is substantially larger than existing QA benchmarks, it is repository-level, and it contains multi-turn dialogues, which are underrepresented in prior work. The authors make the dataset public and describe the crawling and filtering steps concretely, which are clear strengths for reproducibility. The significance of the experimental conclusions, however, depends on two assumptions that are not adequately verified: that the collected GitHub threads are genuine QA dialogues rather than a mixture of issues and pull requests, and that the last maintainer response is a reliable gold answer. The lexical evaluation also lacks statistical support, so the reported ranking and context-length findings should be treated as preliminary until those issues are addressed.

major comments (4)
  1. [Section 2.1 and 2.2] The GitHub REST API endpoint used for crawling, 'List repository issues,' returns both issues and pull requests by default, with pull requests identified by a non-null 'pull_request' field. The five filtering criteria described in Section 2.2 (comment scale, redundancy, external check, event count, robot detection, participant count) do not remove pull requests. If a substantial fraction of the 585,687 entries are pull-request threads, the dataset is not purely a QA benchmark, and the RQ1/RQ2 results in Tables 3 and 4 would not support the paper's central claims. Please report the issue/PR split in the released dataset and either exclude PRs or provide a reasoned justification for why code-review discussions should be treated as QA dialogues.
  2. [Section 3.2] The ground-truth construction assumes that the last response from a repository maintainer (MEMBER, AUTHOR, or CONTRIBUTOR) is a valid answer to the preceding dialogue. Many GitHub threads contain feature requests, bug triage, status updates, or unresolved discussions in which the final maintainer comment is not a factual answer. The paper provides no manual validation, inter-annotator agreement, or heuristic to confirm that the chosen response actually answers the question. This assumption is load-bearing for all reported model scores, so it needs to be validated, for example by sampling entries and measuring human agreement on whether the maintainer reply resolves the thread's question.
  3. [Table 4 and Section 3.5] The claim that 'medium-length contexts are more conducive to LLMs' performance' rests on differences that are very small and are reported without error bars or significance tests. For example, the average score for the 40% group is 0.1698 versus 0.1605 for the 20% group and 0.1518 for the 100% group; with 585,687 entries these differences could be statistically significant or could be artifacts of grouping or metric instability. Additionally, the paper does not define what 'question length' means (characters, tokens, or words?) or how the percentile groups were formed. Please provide confidence intervals, significance tests, and a precise definition of the length variable.
  4. [Tables 3 and 4 and Section 3.1] The model evaluation protocol is under-specified, which limits the reproducibility of the experimental results. The paper does not give the exact prompt template, the decoding parameters (temperature, top-p, max tokens), or the truncation strategy used to fit inputs within each model's context window. Without these details, readers cannot reproduce the reported scores or assess whether the comparisons are fair across models with different context lengths. Please include the full evaluation configuration, ideally in the repository or an appendix.
minor comments (6)
  1. [Table 1] The table uses '%' placeholders where checkmarks or 'No' values are needed; this appears to be a LaTeX rendering issue and should be fixed.
  2. [Author affiliation] The affiliation 'Haribin Institute of Technology, Shenzhen' contains a typo; it should be 'Harbin Institute of Technology.'
  3. [References] Reference [5] is incomplete ('github. [n.d.]') and the ACM reference format lists copyright years 2018 and 2025 inconsistently; please correct the metadata.
  4. [Section 2.2] The 'Event count' filter says 'more than ten events' but does not specify which GitHub event types are counted or why the threshold of ten was chosen; please clarify.
  5. [Figure 1] Figure 1 is difficult to read because the text boxes and arrows are visually cramped; a cleaner layout or a simplified example would improve comprehension.
  6. [Section 5 (Conclusion)] The conclusion states that the benchmark is 'repository-level' but the evaluation uses only the dialogue history, not the repository code or files. Please clarify whether the benchmark actually requires repository-level reasoning or only conversation-level understanding.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CodeRepoQA is a dataset-construction and evaluation paper, and its claims are not derived from fitted parameters or self-citation chains.

full rationale

The paper's central claims are that CodeRepoQA contains 585,687 multi-turn QA entries and that ten LLMs score poorly on lexical metrics. Each step in the construction chain is external: GitHub's REST API provides raw issues; six filtering rules are heuristic quality gates; ground truth is the last maintainer response; and scores are computed from model outputs. There is no equation in which a predicted quantity is defined as a fitted input. The context-length result in Table 4 is a post hoc stratification of the same benchmark, not a prediction from a fitted parameter, so it cannot reduce to an input by construction. Reference [11] is authored by overlapping authors (Pengfei Gao, Xinchen Wang, and Chao Peng), but it is cited only as one example among several of LLM-related software engineering research and is not load-bearing for the benchmark construction or evaluation. Possible data-validity concerns, such as whether the GitHub issues endpoint includes pull requests or whether maintainer replies are good gold answers, are correctness questions about the benchmark rather than circularity: they do not involve a claim being equivalent to its own inputs. Accordingly, no circular step is identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

All entries in this ledger are hand-chosen thresholds or unstated domain assumptions that the benchmark construction and evaluation rely on.

free parameters (3)
  • Minimum comment length threshold = 200 characters
    Chosen by hand in Section 2.2 to balance information content and model input limits; this directly shapes the benchmark composition.
  • Maximum issue size threshold = 10MB
    Hand-chosen upper bound in Section 2.2, also shaping which issues are included.
  • Maximum event count threshold = 10 events
    Hand-chosen in Section 2.2 to exclude auto-generated text; affects data composition.
assumptions (3)
  • domain assumption GitHub issues from popular repositories are representative of real software engineering QA tasks.
    The whole benchmark rests on the idea that issue dialogues reflect the complexity of real-world software development (Section 1).
  • domain assumption The final maintainer response in a dialogue is a correct and complete answer to the preceding discussion.
    Section 3.2 uses the last maintainer response as ground truth without human validation of correctness.
  • domain assumption Lexical overlap metrics (BLEU, ROUGE, Edit Similarity) can measure the quality of answers.
    Section 3.3 selects these metrics for evaluation; the paper does not validate them against human judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering." pith.science (2026). https://pith.science/paper/4ER5YLLV

@misc{pith2026241214764,
  author       = {Pith},
  title        = {Pith review of: CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ER5YLLV}},
  note         = {Machine review of arXiv:2412.14764}
}
read the original abstract

In this work, we introduce CodeRepoQA, a large-scale benchmark specifically designed for evaluating repository-level question-answering capabilities in the field of software engineering. CodeRepoQA encompasses five programming languages and covers a wide range of scenarios, enabling comprehensive evaluation of language models. To construct this dataset, we crawl data from 30 well-known repositories in GitHub, the largest platform for hosting and collaborating on code, and carefully filter raw data. In total, CodeRepoQA is a multi-turn question-answering benchmark with 585,687 entries, covering a diverse array of software engineering scenarios, with an average of 6.62 dialogue turns per entry. We evaluate ten popular large language models on our dataset and provide in-depth analysis. We find that LLMs still have limitations in question-answering capabilities in the field of software engineering, and medium-length contexts are more conducive to LLMs' performance. The entire benchmark is publicly available at https://github.com/kinesiatricssxilm14/CodeRepoQA.

Figures

Figures reproduced from arXiv: 2412.14764 by the authors.

Figure 1
Figure 1. A multi-turn QA entry illustrating the main com [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

    cs.SE 2026-07 conditional novelty 7.0 of 10

    RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.

  2. Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study

    cs.SE 2026-08 conditional novelty 6.0 of 10

    On SWE-QA, semantic search beat deep agentic search in accuracy (65.2% vs 46.2% Pass) and in cost per correct answer, with 41.8% of deep agent failures attributed to planner-to-subagent coordination breakdowns.

  3. Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

    cs.SE 2026-07 conditional novelty 6.0 of 10

    QA-driven pre-repair knowledge acquisition (Questioner + Answerer, then Resolver) raises SWE-bench Verified Pass@1 by up to 4.4 points over Mini-SWE-Agent and other pre-repair methods.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages · cited by 3 Pith papers

  1. [1]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1469–1481

    2023

  2. [2]

    Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, et al . 2023. Codeapex: A bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.01940 (2023)

    arXiv 2023

  3. [3]

    Xingyuan Bu Ge Bai, Jie Liu et al. 2024. MT-Bench-101: A Fine-Grained Bench- mark for Evaluating Large Language Models in Multi-Turn Dialogues. InProceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics . ACL, 7421–7454

    work page 2024

  4. [4]

    github. 2022. GitHub REST API documentation. https://docs.github.com/en/ rest?apiVersion=2022-11-28

    work page 2022

  5. [5]

    github. [n.d.]. marking-issues-or-pull-requests-as-a-duplicate. https://docs. github.com/en/issues/tracking-your-work-with-issues/administering-issues

    work page

  6. [6]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Under- standing. Proceedings of the International Conference on Learning Representations (ICLR) (2021)

    2021

  7. [7]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for soft- ware engineering: A systematic literature review. arXiv preprint arXiv:2308.10620 (2023)

    arXiv 2023

  8. [8]

    Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2022)

    work page 2022

Show all 16 references
  1. [9]

    Chenxiao Liu and Xiaojun Wan. 2021. CodeQA: A question answering dataset for source code comprehension. arXiv preprint arXiv:2109.08365 (2021)

    2021 arXiv

  2. [10]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large Language Model-Based Agents for Software Engineering: A Survey. arXiv preprint arXiv:2409.02977 (2024)

    2024 arXiv

  3. [11]

    Yizhou Liu, Pengfei Gao, Xinchen Wang, Chao Peng, and Zhao Zhang

  4. [12]

    Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Sy...

    2023

  5. [13]

    Yangtian Zi Raymond Li, Loubna Ben Allal et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  6. [14]

    Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023. The programmer’s assistant: Conversational interaction with a large language model for software development. InProceedings of the 28th International Conference on Intelligent User Inte...

    2023

  7. [15]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793 (2024)

    2024 arXiv

  8. [2024]

    arXiv preprint arXiv:2409.00899 (2024)

    MarsCode Agent: AI-native Automated Bug Fixing. arXiv preprint arXiv:2409.00899 (2024)

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.