{"id":"64603442-976a-4558-9335-bdb0b82f734e","arxiv_id":"2501.13742","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Retrieval-augmented code generation improves CodeGen, UniXcoder, and CodeT5, with BM25 and sequential integration recommended as the most cost-effective configuration.","lead":"This paper systematically tests whether adding retrieved similar code snippets improves code generation models, comparing retrieval methods and fusion strategies across three datasets. It finds retrieval augmentation broadly helps, with the simple BM25 retriever and sequential concatenation giving the best balance of performance and cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HearthStone's 41.60% EM gain is ~11 extra exact matches in 66 test items with no repeated runs; the universal-improvement claim needs per-dataset bootstrap CIs.","rationale":"The paper provides reproducible artifacts (retrieval-augmented datasets and code) and is transparent about cases where retrieval hurts in Section 5.3, which I credit. The central claim, however, is an empirical universal: with BM25 and Sequential Integration Fusion, all three models improve across datasets. For that claim to hold, the point estimates in Table 3 must be stable. The paper reports no repeated-seed variance anywhere, and the aggregate t-test in Section 5.2 is under-specified: it does not state the unit of analysis (models, datasets, metrics), whether it is paired, or how the p-value is computed. The HearthStone EM numbers are the most striking but the least stable: the 41.60% average relative improvement is an average of relative changes, and in absolute counts it is 7→10, 9→15, and 13→15 correct outputs on 66 test items. A difference of a few exact matches is within noise for a 66-item sample. The internal reporting inconsistencies (Table 3 vs Table 5 CodeBLEU for the same condition; abstract SFF percentages not reproducing from Table 5) reinforce that the quantitative record needs verification before the recommendation is taken as established. I therefore focus the stress test on statistical reliability rather than the database-coverage assumption emphasized by the reader; the latter is acknowledged in Section 6.3.2 and affects applicability beyond the tested benchmarks, whereas the former is a precondition for the observed comparisons themselves. The verdict remains conditional: the direction of evidence is consistent, but the strength of the flagship claim needs a bootstrap replication.","tokens_in":30572,"tokens_out":10421,"duration_ms":95579,"concrete_test":"Using the released retrieval-augmented datasets and code, rerun Table 3's nine model-dataset conditions with at least five fine-tuning seeds, then compute paired bootstrap 95% confidence intervals for the EM/BLEU/CodeBLEU deltas on the fixed test split, reporting per-dataset and per-model intervals. If any HearthStone EM interval includes zero, or the CoNaLa CodeGen EM regression is reproduced, the universal-improvement claim is not supported; also recompute the abstract's SFF percentages from Table 5 to confirm whether they reproduce.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is only as strong as the reliability of the point estimates in Table 3. The paper reports no repeated-seed variance for any condition, and the flagship quantitative anchor—the 41.60% average EM improvement on HearthStone—is calculated from 29 to 40 exact matches out of 66 test examples across three models (CodeGen 7→10, UniXcoder 9→15, CodeT5 13→15). Differences of 2–6 correct examples are within plausible seed-to-seed noise for a 66-item test set, so without per-dataset confidence intervals the 'universal improvement' claim is not established for EM, and the aggregate t-test in Section 5.2 (p=0.035) pools across models, datasets, and metrics without reporting the unit of analysis. Reporting instability compounds the concern: Table 3 and Table 5 give different CodeBLEU values for the same CodeT5+BM25+SIF configuration on CONCODE (43.48 vs 46.92), and the abstract's SFF improvements (14.83% BLEU, 8.05% CodeBLEU) do not reproduce from Table 5. If the HearthStone EM gain is within noise, the strongest evidence for universality weakens, even though BLEU/CodeBLEU trends may survive.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic empirical study of retrieval-augmented code generation. It evaluates three pre-trained code models (CodeGen, UniXcoder, CodeT5) and three LLMs (ChatGLM3-6B, CodeLlama-7B, DeepSeek-Coder) on CONCODE, CoNaLa, and HearthStone, comparing five retrieval techniques (BM25, RetroMAE, CodeBERT, UniXcoder, CoCoSoDa) and four fusion strategies (Sequential Integration Fusion, Sample Expansion Fusion, Vectorized Decoding Fusion, Sketch Filling Fusion). The main claims are that the retrieval-augmented framework reliably improves code generation across models and datasets, that BM25 and Sequential Integration Fusion are convenient and effective default choices, and that Sketch Filling Fusion can yield further gains at higher computational cost. The paper also reports cost measurements and releases code and augmented datasets.","tokens_in":30815,"tokens_out":5158,"duration_ms":42498,"significance":"If the results hold, this is a useful reference for practitioners: it provides a rare head-to-head comparison of retrieval techniques and fusion strategies for code generation, it covers multiple model architectures, and it quantifies the training and inference cost trade-offs. The BLEU and CodeBLEU improvements are directionally consistent across most conditions, which gives the central claim partial support, and the release of code and augmented datasets is a concrete reproducibility asset. However, the absence of variance estimates, the small HearthStone EM counts, and internal inconsistencies between tables mean that the specific numerical claims should be treated with caution until they are reconciled by the authors.","major_comments":[{"comment":"The headline HearthStone result, a 41.60% average EM improvement, is computed from very small absolute counts: 7→10, 9→15, and 13→15 correct predictions out of 66 test examples for CodeGen, UniXcoder, and CodeT5, respectively. With no repeated runs and no per-condition variance estimates, differences of 2–6 correct examples are within plausible seed-to-seed noise, so the claim that retrieval-augmentation universally improves EM on HearthStone is not established. Please report per-dataset bootstrap confidence intervals or repeated-seed results, and state explicitly the unit of analysis and pairing for the t-test reported as p=0.035.","section":"Section 5.2, Table 3"},{"comment":"Tables 3 and 5 describe the same configuration, CodeT5 with BM25 retrieval and Sequential Integration Fusion, but report inconsistent CodeBLEU values: 43.48 vs 46.92 on CONCODE and 49.80 vs 60.50 on HearthStone. This is a load-bearing inconsistency because the central recommendation to use BM25 and SIF depends on these numbers. Please explain the discrepancy (e.g., different k, different evaluation version, different preprocessing) and harmonize the tables so that readers can determine which configuration the reported improvements refer to.","section":"Tables 3 and 5"},{"comment":"The abstract and introduction claim that Sketch Filling Fusion yields an average improvement of 14.83% in BLEU and 8.05% in CodeBLEU across the three datasets for original CodeT5. These numbers do not reproduce from Table 5: comparing SFF to the baseline row gives roughly 39% average BLEU improvement and 25% average CodeBLEU improvement, while comparing SFF to SIF or SEF gives different values again. Please correct the abstract and Section 1 or state explicitly the comparison basis for these percentages.","section":"Abstract and Section 1, RQ3"},{"comment":"The recommendation on the optimal number of retrieved code snippets is based on a single run per k value for one model, but the underlying numerical values are not reported in the text or a table, and Figure 2 plots five metrics with two axes and a legend that is difficult to read. Since this recommendation is one of the paper's actionable findings, please provide the exact numbers or a companion table so that readers can assess the magnitude of the differences and the location of the claimed inflection points.","section":"Section 5.4.1, Figure 2"}],"minor_comments":[{"comment":"There are several typos and copyediting issues, including 'Specically' in Section 2.2, 'donotes' in Section 3.4, 'Expainsion' in Section 5.4.2, and a missing parenthesis in Figure 4(g); a careful proofread is needed.","section":"Throughout"},{"comment":"The statistical significance test is described only as 't-test' with p=0.035; please report whether it is paired, what the sample units are, whether all metrics are pooled, and whether any multiple-comparison correction was applied.","section":"Section 5.2"},{"comment":"The units in Table 7 are not fully clear: the header says 'retrieval costs per 50 instances,' but the 'total costs' column appears to scale the per-50-instance cost to the full test set; please clarify the exact computation and the number of instances used for the total.","section":"Table 7"},{"comment":"Section 6.1.1 reports substantial LLM improvements, while Section 6.4 states that generalization to larger models is uncertain; please make the relationship between these two statements explicit, since readers may otherwise see them as contradictory.","section":"Section 6.1.1 vs Section 6.4"},{"comment":"The SimAST metric is defined in Equation (18) but the table header uses 'SimilarityAST'; please unify the notation throughout the paper.","section":"Section 4.3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible and the study is potentially useful, but the internal inconsistency between Tables 3 and 5, the unreproducible SFF percentages in the abstract, and the lack of variance estimates are the main blockers. If the authors can reconcile the tables and add bootstrap confidence intervals or repeated-seed results, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. Bottom line: this is a genuinely useful systematic benchmark—the first single-framework comparison I know of that runs three generation models, five retrievers, and four fusion strategies on the same three datasets, and it also checks the effect on LLM prompting. The main BLEU/CodeBLEU trends are consistent across tables; the recommendation that cheap BM25 plus sequential concatenation is a strong default is plausible and practically useful. The paper is also honest about retrieval hurting on some configurations (the RetroMAE cases), and it releases data and code, so the empirical core is reproducible in principle.\n\nThe soft spots are real but not fatal. No repeated runs or variance estimates appear anywhere; the only significance support is one t-test pooling across models, datasets, and metrics (p=0.035) with no stated unit of analysis. The flagship HearthStone EM claim (41.60% average improvement) comes from 7→10, 9→15, and 13→15 exact matches on a 66-item test set—differences of 2–6 examples, plausibly seed noise. The universality statement should not lean on EM. Two tables disagree: CodeT5+BM25+SIF on CONCODE shows CodeBLEU 43.48 in Table 3 and 46.92 in Table 5. The abstract's Sketch Filling numbers (14.83% BLEU, 8.05% CodeBLEU) do not reproduce from Table 5 no matter how I average. Table 7's cost arithmetic is off (CoNaLa BM25 total 109 s looks like a factor-of-10 error against the per-50 rate and the 500-item test set). And SFF wrecking EM on CoNaLa (7.40→1.80) is passed over in silence.\n\nNone of this undermines the central practical conclusion—BLEU/CodeBLEU gains are broad and consistently positive—but the paper should either add bootstrap CIs or soften the universal-improvement language, and the reporting inconsistencies need fixing before the quantitative recommendations are taken at face value.\n\nWho it is for: practitioners choosing a cheap retrieval setup for code generation, and researchers wanting a map of the retrieval-augmented framework design space. It deserves serious peer review and likely acceptance after a revision that adds variance estimates and reconciles the tables. I would cite it.","headline":"Useful systematic benchmark of retrieval-augmented code generation, but the headline numbers need variance estimates and a few tables need reconciling before the quantitative recommendations can be trusted.","tokens_in":31339,"tokens_out":3028,"would_cite":true,"duration_ms":26548,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-augmented code generation works best with a simple no-training retriever and simple concatenation, an empirical study of three models and three datasets finds.","keywords":["retrieval-augmented generation","code generation","pre-trained code models","BM25","fusion strategies","code search","empirical study","CodeBLEU"],"falsifier":"Run the recommended pipeline (BM25 retrieval plus Sequential Integration Fusion on CodeT5) on a held-out benchmark whose test problems require APIs or coding patterns absent from the retrieval database; the paper's universal-effectiveness claim would be falsified if the augmented model does not beat the base model on BLEU or CodeBLEU.","tokens_in":1645,"feed_emoji":"⚙️","tokens_out":1873,"duration_ms":61562,"temperature":0.7,"pith_summary":"The paper is an empirical attempt to turn retrieval-augmented generation into a dependable recipe for code generation: retrieve code snippets similar to a natural-language request, splice them into the model input, and fine-tune the model on the augmented data. It claims this recipe improves three architecturally different pre-trained code models (CodeGen, UniXcoder, CodeT5) across three datasets (CONCODE, CoNaLa, HearthStone), with average Exact Match up 41.60% on HearthStone. It also claims BM25 and Sequential Integration Fusion are the convenient high-performing defaults, while Sketch Filling Fusion adds further gains at substantially higher training cost. If right, the study gives practitioners a model-agnostic way to improve code generation without changing the model architecture, and a cost-aware map of when to use more elaborate retrieval or fusion.","feed_headline":"BM25 retrieval lifts code generation across models","feed_subtitle":"Study of three code models and three benchmarks finds simple retrieved-code concatenation beats fancier setups.","key_machinery":"The machinery is a three-phase retrieval-augmented pipeline. In the retrieval phase, a retriever (BM25, RetroMAE, or a code search model such as CodeBERT, UniXcoder, or CoCoSoDa) finds k similar code snippets from a database of natural-language/code pairs; in the fusion phase, those snippets are combined with the original request by concatenation (Sequential Integration Fusion), by expanding the training sample (Sample Expansion Fusion), by encoding each snippet and feeding the vectors to the decoder (Vectorized Decoding Fusion), or by extracting a structural sketch (Sketch Filling Fusion); in the generation phase, the unchanged pre-trained model is fine-tuned on the augmented input. The load-bearing piece is that the retrieved snippets act as an external reference that supplies structural and semantic guidance, so the model does not have to infer the target shape from the description alone.","core_discovery":"On its own terms, the paper's central claim is that the retrieval-augmented framework generalizes: all three pre-trained models improve on BLEU and CodeBLEU on all three datasets when BM25-retrieved code is concatenated into the input, and the retriever requires no training. The strongest evidence is HearthStone, where the average EM improvement across models is 41.60%, with BLEU and CodeBLEU up about 9% and 8.7%. The paper further finds that more sophisticated retrievers are not better: BM25 outperforms the code search models on CONCODE and HearthStone and is best or near-best for CodeT5 on CoNaLa, while RetroMAE can degrade CodeGen and UniXcoder on CONCODE and HearthStone. Among fusion strategies, Sequential Integration Fusion and Sketch Filling Fusion lead, with Sketch Filling giving the largest CodeT5 gains (average 14.83% BLEU, 8.05% CodeBLEU) but requiring 2 to 7 times the training time; Sequential Integration is recommended as the cost-effective choice. The same retrieval augmentation also improves three large language models used in a prompting setup.","pith_inferences":["The paper's recommendation of BM25 likely does not transfer to very large or rapidly changing codebases, because its retrieval cost grows with corpus size; the paper's own cost tables hint at this trade-off, and a learned retriever may become preferable at scale.","If the mechanism is truly that retrieved code carries structural guidance, a direct testable extension is to degrade the retrieval database with random or wrong-language snippets and predict that gains disappear or reverse; the paper's RetroMAE result already points in that direction.","Active retrieval, deciding per request whether retrieval will help, could salvage the cases where noisy snippets hurt; the paper names this as future work, so treating it as an extension is consistent with the study's findings.","The main constraint is the assumption that the training set doubles as the retrieval database; for benchmarks whose test data excludes the training set, constructing an external retrieval database becomes the next bottleneck."],"forward_implications":["Fine-tuned code models can be improved without changing their architecture or parameters, simply by augmenting the training input with BM25-retrieved examples.","A no-training retriever can beat deep code-search models for generation, so teams without retrieval-label budgets need not adopt heavier retrievers.","The best number of retrieved snippets is dataset-dependent; short or structured datasets saturate or degrade, and long snippets get truncated, so the choice of k should follow from input/output length rather than being maximized.","For code with regular structure, extracting a sketch from the best-matching snippet yields the largest gains, making sketch-based fusion a candidate when training cost is acceptable.","Large language models used in prompting mode also benefit from retrieved code snippets, extending the finding beyond fine-tuned models."],"supporting_citations":[{"why":"Supplies CodeGen, one of three pre-trained code generation models evaluated in the study.","marker":"[50]"},{"why":"Supplies UniXcoder, the second generation model, and also one of the code-search retrievers tested.","marker":"[21]"},{"why":"Supplies CodeT5, the third model and the base model for all fusion-strategy experiments.","marker":"[68]"},{"why":"Supplies the CONCODE dataset of Java code with repository-based splits used to test generalization.","marker":"[32]"},{"why":"Supplies the CoNaLa dataset of Stack Overflow-derived Python request-code pairs.","marker":"[72]"},{"why":"Supplies the HearthStone dataset of structured card-code pairs where the largest EM gains appear.","marker":"[45]"},{"why":"Supplies the BM25 retrieval algorithm that the paper recommends as the no-training default.","marker":"[64]"},{"why":"Supplies RetroMAE, the text-retrieval technique whose retrieved snippets sometimes degrade performance.","marker":"[71]"},{"why":"Supplies CoCoSoDa, the code-search model that performs best with large language models in the study.","marker":"[60]"},{"why":"Supplies the sketch-extraction method behind Sketch Filling Fusion, the highest-gain fusion strategy.","marker":"[41]"}],"fun_headline_variants":["BM25 retrieval lifts code gen across three models","Simple BM25 beats fancier retrievers for code generation","BM25 wins on simplicity and performance for code gen","Retrieval-augmented code gen: simple BM25 is the winner","BM25 outperforms neural retrievers in code generation"],"cache_read_input_tokens":33536,"weakest_assumption_plain":"The load-bearing premise is that the retrieval database, here the training set, contains a snippet similar enough to every test request to add signal rather than noise; if a new domain's test data has no near neighbors, the recommended pipeline has nothing useful to retrieve and the claimed universal improvement can collapse.","fun_headline_variants_meta":{"raw":{"variants":["BM25 retrieval lifts code gen across three models","Simple BM25 beats fancier retrievers for code generation","BM25 wins on simplicity and performance for code gen","Retrieval-augmented code gen: simple BM25 is the winner","BM25 outperforms neural retrievers in code generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001438,"raw_usage":{"total_tokens":5857,"prompt_tokens":1069,"completion_tokens":4788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":4707}},"tokens_in":685,"tokens_out":4788,"duration_ms":31245,"temperature":1.0,"reasoning_tokens":4707,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:38:23.961884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the recommended pipeline (BM25 retrieval plus Sequential Integration Fusion on CodeT5) on a held-out benchmark whose test problems require APIs or coding patterns absent from the retrieval database; the paper's universal-effectiveness claim would be falsified if the augmented model does not beat the base model on BLEU or CodeBLEU.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BM25 retrieval algorithm that the paper recommends as the no-training default."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies CodeGen, one of three pre-trained code generation models evaluated in the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies UniXcoder, the second generation model, and also one of the code-search retrievers tested."},{"cited_title":"Joty, and Steven C","cited_arxiv_id":null,"evidence_quote":"Supplies CodeT5, the third model and the base model for all fusion-strategy experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CONCODE dataset of Java code with repository-based splits used to test generalization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CoNaLa dataset of Stack Overflow-derived Python request-code pairs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HearthStone dataset of structured card-code pairs where the largest EM gains appear."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies RetroMAE, the text-retrieval technique whose retrieved snippets sometimes degrade performance."},{"cited_title":"SkCoder: A Sketch-based Approach for Automatic Code Generation","cited_arxiv_id":"2302.06144","evidence_quote":"Supplies the sketch-extraction method behind Sketch Filling Fusion, the highest-gain fusion strategy."}],"review_version":1}