{"id":"ad1b4537-8dd4-4513-a121-3ea98aac43ce","arxiv_id":"2412.14764","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper presents CodeRepoQA, a large-scale multi-turn repository-level QA benchmark, and evaluates ten LLMs on it.","lead":"CodeRepoQA is a new public benchmark of over 585,000 multi-turn question-answer dialogues mined from GitHub issues across 30 repositories and five programming languages. The authors evaluate ten large language models and report that all score low on lexical similarity metrics, with medium-length contexts performing best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset construction never excludes GitHub pull requests, which the issues endpoint returns by default; if PR threads are present, CodeRepoQA is not purely a QA benchmark and all downstream evaluation is affected.","rationale":"I read the paper as a dataset contribution whose load-bearing claim is that the released data are a QA benchmark. The most direct threat is that the construction pipeline does not filter pull requests even though GitHub's issues API returns them by default. The filtering rules in §2.2 are silent on this, and the unexpectedly large counts for 30 repos are consistent with PR threads being included. The reader's ground-truth concern remains valid but is downstream of this data-integrity question; if PRs are present, no metric or baseline adjustment can fix the benchmark label. I do not see an internal inconsistency in the pipeline as described, and the idea is useful; the concern is a concrete, testable data-validity issue. I therefore keep the reader's CONDITIONAL verdict unchanged: the condition should include a PR-contamination check before the benchmark is used for QA claims. If the check finds no PRs, my additional objection collapses and the reader's original conditions (confidence intervals, baselines, ground-truth validation) still apply.","tokens_in":6527,"tokens_out":9719,"duration_ms":66893,"concrete_test":"Download the released dataset or re-crawl the 30 repositories through the same issues endpoint and check each entry for a non-null `pull_request` field (or, in a re-crawl, inspect the pull_request object). Compute the fraction of the 585,687 entries that are pull requests. If that fraction exceeds a small tolerance (e.g., 1–2%), re-filter the benchmark to issues only and recompute Tables 3 and 4; the 'QA benchmark' claim should then be restated as a 'repository conversation' benchmark until re-validation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 says 'we use the GitHub REST API to crawl all issues from 30 repositories,' and Section 2.2 lists filtering rules: comment scale, redundancy, external check, event count, robot detection, and participant count. None of these rules removes GitHub pull requests. The GitHub 'List repository issues' endpoint returns both issues and pull requests, and a PR is marked by a non-null `pull_request` field. The raw count (~636,000) and final count (585,687) are plausible for 30 popular repositories only if PRs are included. A PR body plus review comments is a code-review discussion, not necessarily a question-answer exchange, so the central claim that CodeRepoQA is a software-engineering QA benchmark—and the RQ1/RQ2 results in Tables 3 and 4—would be unsupported if a substantial fraction of the entries are PRs. This is a data-validity problem that precedes the ground-truth and metric concerns in Sections 3.2–3.3.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CodeRepoQA, a large-scale multi-turn question-answering benchmark constructed from GitHub issues across 30 popular repositories in five programming languages, yielding 585,687 entries with an average of 6.62 dialogue turns per entry. The authors describe a two-stage pipeline of crawling via the GitHub REST API followed by automatic filtering, and they evaluate ten commercial and open-source LLMs using BLEU, ROUGE-1, ROUGE-L, and Edit Similarity. The main empirical claims are that LLMs remain limited in software-engineering QA and that medium-length contexts lead to better performance.","tokens_in":6713,"tokens_out":3217,"duration_ms":30385,"significance":"If the benchmark is valid as described, it would be a valuable public resource: it is substantially larger than existing QA benchmarks, it is repository-level, and it contains multi-turn dialogues, which are underrepresented in prior work. The authors make the dataset public and describe the crawling and filtering steps concretely, which are clear strengths for reproducibility. The significance of the experimental conclusions, however, depends on two assumptions that are not adequately verified: that the collected GitHub threads are genuine QA dialogues rather than a mixture of issues and pull requests, and that the last maintainer response is a reliable gold answer. The lexical evaluation also lacks statistical support, so the reported ranking and context-length findings should be treated as preliminary until those issues are addressed.","major_comments":[{"comment":"The GitHub REST API endpoint used for crawling, 'List repository issues,' returns both issues and pull requests by default, with pull requests identified by a non-null 'pull_request' field. The five filtering criteria described in Section 2.2 (comment scale, redundancy, external check, event count, robot detection, participant count) do not remove pull requests. If a substantial fraction of the 585,687 entries are pull-request threads, the dataset is not purely a QA benchmark, and the RQ1/RQ2 results in Tables 3 and 4 would not support the paper's central claims. Please report the issue/PR split in the released dataset and either exclude PRs or provide a reasoned justification for why code-review discussions should be treated as QA dialogues.","section":"Section 2.1 and 2.2"},{"comment":"The ground-truth construction assumes that the last response from a repository maintainer (MEMBER, AUTHOR, or CONTRIBUTOR) is a valid answer to the preceding dialogue. Many GitHub threads contain feature requests, bug triage, status updates, or unresolved discussions in which the final maintainer comment is not a factual answer. The paper provides no manual validation, inter-annotator agreement, or heuristic to confirm that the chosen response actually answers the question. This assumption is load-bearing for all reported model scores, so it needs to be validated, for example by sampling entries and measuring human agreement on whether the maintainer reply resolves the thread's question.","section":"Section 3.2"},{"comment":"The claim that 'medium-length contexts are more conducive to LLMs' performance' rests on differences that are very small and are reported without error bars or significance tests. For example, the average score for the 40% group is 0.1698 versus 0.1605 for the 20% group and 0.1518 for the 100% group; with 585,687 entries these differences could be statistically significant or could be artifacts of grouping or metric instability. Additionally, the paper does not define what 'question length' means (characters, tokens, or words?) or how the percentile groups were formed. Please provide confidence intervals, significance tests, and a precise definition of the length variable.","section":"Table 4 and Section 3.5"},{"comment":"The model evaluation protocol is under-specified, which limits the reproducibility of the experimental results. The paper does not give the exact prompt template, the decoding parameters (temperature, top-p, max tokens), or the truncation strategy used to fit inputs within each model's context window. Without these details, readers cannot reproduce the reported scores or assess whether the comparisons are fair across models with different context lengths. Please include the full evaluation configuration, ideally in the repository or an appendix.","section":"Tables 3 and 4 and Section 3.1"}],"minor_comments":[{"comment":"The table uses '%' placeholders where checkmarks or 'No' values are needed; this appears to be a LaTeX rendering issue and should be fixed.","section":"Table 1"},{"comment":"The affiliation 'Haribin Institute of Technology, Shenzhen' contains a typo; it should be 'Harbin Institute of Technology.'","section":"Author affiliation"},{"comment":"Reference [5] is incomplete ('github. [n.d.]') and the ACM reference format lists copyright years 2018 and 2025 inconsistently; please correct the metadata.","section":"References"},{"comment":"The 'Event count' filter says 'more than ten events' but does not specify which GitHub event types are counted or why the threshold of ten was chosen; please clarify.","section":"Section 2.2"},{"comment":"Figure 1 is difficult to read because the text boxes and arrows are visually cramped; a cleaner layout or a simplified example would improve comprehension.","section":"Figure 1"},{"comment":"The conclusion states that the benchmark is 'repository-level' but the evaluation uses only the dialogue history, not the repository code or files. Please clarify whether the benchmark actually requires repository-level reasoning or only conversation-level understanding.","section":"Section 5 (Conclusion)"}],"recommendation":"major_revision","confidential_remarks":"The PR/issue concern is the most important issue and is directly checkable in the public dataset. If a large fraction of entries are pull requests, the benchmark can still be salvaged by filtering on the 'pull_request' field and re-running the experiments, but the current manuscript's central claims cannot stand without that re-analysis. The ground-truth validity and lack of statistical testing are also fixable but require additional work. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: CodeRepoQA is a promising dataset idea that ships with a construction bug that undermines its central claim. The GitHub REST 'list issues' endpoint returns pull requests by default, and none of the filtering rules in Section 2.2 excludes them. If a nontrivial share of the 585,687 entries are PR threads, this is not a QA benchmark; it's a mixed bag of issues and code-review discussions. The 636k raw count for 30 repos is only plausible if PRs are included. That's the first thing a referee must check.\n\nWhat's genuinely useful: the paper offers a large, public, multi-turn, repository-level QA resource spanning five languages, built from real developer dialogues. The construction pipeline is described in enough detail to reproduce or audit. That is a real contribution, even if the final dataset needs cleaning.\n\nThe evaluation is the weak half. Using the last maintainer response as ground truth is a defensible heuristic, but it is unvalidated—some maintainer replies are 'thanks' or 'fixed in #123' rather than substantive answers. The lexical metrics (BLEU, ROUGE, Edit Similarity) are fine as coarse indicators, but the paper reports no confidence intervals or significance tests. The context-length conclusion in Table 4 is a two-point difference in average score (0.1698 vs 0.1605) with no error bars; calling that 'medium-length contexts are more conducive' overstates the evidence.\n\nThe citation and baseline picture is fair; the comparison to MMLU, CodeQA, CS1QA, and CodeApex is accurate, and the novelty claim of multi-turn repo-level QA holds up.\n\nIf the PR issue is confirmed, the dataset as shipped is not usable as a QA benchmark as-is. But the fix is straightforward (filter on the pull_request field), and the underlying idea is solid. This paper should go to peer review, not be desk-rejected, because a good referee can push the authors to clean the data and re-run the experiments. I'd also want the authors to release the actual API query and filtering code.\n\nFor the next 12 months I wouldn't cite the current version, but I'd watch for a corrected release.","headline":"A useful dataset idea undermined by a likely pull-request contamination in the crawling pipeline; the benchmark needs a data fix and stronger evaluation stats before it can be trusted.","tokens_in":7240,"tokens_out":2554,"would_cite":false,"duration_ms":16136,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeRepoQA extracts 585,687 multi-turn QA dialogues from GitHub issues and reports that current LLMs fall well short of human maintainer responses.","keywords":["question answering","software engineering","large language models","GitHub issues","multi-turn dialogue","repository-level benchmark","code QA","evaluation metrics"],"falsifier":"Manually annotate a random sample of, say, 200 CodeRepoQA threads: if human raters find that the final maintainer message is not clearly the best or most complete answer in a substantial share of cases, or that model responses judged correct by humans still have near-zero BLEU and ROUGE scores, the benchmark's ground-truth and metric assumptions would be falsified.","tokens_in":6354,"feed_emoji":"🐙","tokens_out":7244,"duration_ms":51815,"temperature":0.7,"pith_summary":"The paper introduces CodeRepoQA, a public benchmark of 585,687 multi-turn question-answer dialogues mined from issues in 30 popular GitHub repositories covering five programming languages. Its purpose is to test language models on repository-level software engineering questions, the kind of messy, multi-participant exchanges that single-turn code QA benchmarks do not capture. Using the final reply of a repository maintainer as ground truth, the authors evaluate ten language models with BLEU, ROUGE-L, ROUGE-1, and Edit Similarity, finding that even the best model leaves a wide gap to real maintainer responses. They also report that medium-length contexts, around the 40 percent mark of the length distribution, produce the highest scores.","feed_headline":"585,687 GitHub issue threads become an LLM testbed","feed_subtitle":"CodeRepoQA mines five languages from 30 repositories; best model averages 0.18 across four answer-overlap metrics.","key_machinery":"The central object is the CodeRepoQA entry: a structured GitHub issue thread containing repository metadata, author association roles, and the full multi-turn dialogue, with the final maintainer reply designated as the reference answer. The construction pipeline filters raw issues by character count, duplicate markers, external links, event count, robot detection, and participant count to keep only authentic interactions. At evaluation time, the dialogue history is fed to a language model, and the generated reply is scored against the maintainer reference using BLEU, ROUGE-L, ROUGE-1, and Edit Similarity.","core_discovery":"The central claim is that repository-level, multi-turn QA performance can be measured at scale by turning GitHub issue threads into benchmark entries, and that current LLMs are still far from matching human maintainers on this task. CodeRepoQA is built by crawling 30 popular repositories, filtering for genuine, non-duplicate, non-robotic, multi-participant conversations, and yielding 585,687 entries with an average of 6.62 dialogue turns. Evaluation treats the last response from a MEMBER, AUTHOR, or CONTRIBUTOR as the reference answer; across ten models the best average score is 0.1826, with ROUGE-1 reaching only 0.2470, which the authors read as evidence that generated answers miss key content from the real replies. On context length, the 40 percent length group scores highest on average (0.1698) while the longest contexts score lowest (0.1518), supporting the paper's conclusion that medium-length contexts are most favorable.","pith_inferences":["Because ground truth is defined as the last maintainer reply, the benchmark implicitly assumes a single correct resolution; a follow-up could compare human judgments of answer adequacy across all replies in a thread.","Lexical-overlap metrics reward surface similarity, so a model that paraphrases the maintainer's wording would score well; a semantic or human evaluation on a subset could test whether low BLEU scores truly indicate poor answers.","The context-length finding may be confounded with issue difficulty, since harder issues tend to attract longer discussions; matching issues of similar difficulty across length groups would separate the two explanations."],"forward_implications":["Researchers can score new or existing LLMs on 585,687 real software-engineering dialogues without additional crawling or manual annotation.","Model scale alone does not predict QA quality: the 7B CodeQwen model outperforms GPT-4, and the 236B DeepSeek-Coder-V2 is beaten by its 33B and 6.7B siblings.","The context-length result implies that feeding a model a medium-length slice of the thread is more likely to maximize measured scores than sending the shortest or longest contexts.","The benchmark structure supports per-language and per-repository breakdowns, allowing targeted analysis of where models struggle."],"supporting_citations":[{"why":"Supplies the GitHub REST API access used to crawl all issues from the 30 selected repositories.","marker":"[4]"},{"why":"Provides the event-count filtering heuristic used to remove robot-generated issue activity.","marker":"[13]"},{"why":"Defines the duplicate-issue markers used to filter redundant entries.","marker":"[5]"},{"why":"The MMLU benchmark that CodeRepoQA compares against as a prior QA evaluation resource.","marker":"[6]"},{"why":"The CodeQA dataset, a single-turn code QA benchmark whose scope CodeRepoQA extends.","marker":"[9]"},{"why":"The CS1QA dataset, another single-turn code QA benchmark used for comparison.","marker":"[8]"},{"why":"The CodeApex benchmark used as a single-turn, single-language comparison point.","marker":"[2]"},{"why":"Motivates the multi-turn dialogue design by showing that issue resolution needs multiple interactions.","marker":"[3]"}],"fun_headline_variants":["585K GitHub threads benchmark LLM code QA","New benchmark shows LLMs lag on repo-level QA","CodeRepoQA: 585K issue threads test LLM coding answers","LLMs score 0.18 on new 585K GitHub QA benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the last reply from a repository maintainer is the correct answer to the whole preceding dialogue, and that word-overlap scores gauge answer quality.","fun_headline_variants_meta":{"raw":{"variants":["585K GitHub threads benchmark LLM code QA","New benchmark shows LLMs lag on repo-level QA","CodeRepoQA: 585K issue threads test LLM coding answers","LLMs score 0.18 on new 585K GitHub QA benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1189,"prompt_tokens":925,"completion_tokens":264,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":193}},"tokens_in":541,"tokens_out":264,"duration_ms":2781,"temperature":1.0,"reasoning_tokens":193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:54:49.292249+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually annotate a random sample of, say, 200 CodeRepoQA threads: if human raters find that the final maintainer message is not clearly the best or most complete answer in a substantial share of cases, or that model responses judged correct by humans still have near-zero BLEU and ROUGE scores, the benchmark's ground-truth and metric assumptions would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GitHub REST API access used to crawl all issues from the 30 selected repositories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the duplicate-issue markers used to filter redundant entries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The CS1QA dataset, another single-turn code QA benchmark used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the multi-turn dialogue design by showing that issue resolution needs multiple interactions."}],"review_version":1}