Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that on a large proprietary C++ codebase, retrieval-augmented generation with BM25 achieves higher code-completion accuracy than fine-tuning, and that the two approaches combine for the best results.

desk verdict Solid empirical comparison for industrial code completion: BM25 RAG beats fine-tuning on this proprietary C++ corpus, and the two combine, but the fine-tuning baseline lacks convergence evidence. read the letter →

arxiv 2505.15179 v1 pith:Y75PHOYL submitted 2025-05-21 cs.SE

Chaozheng Wang , Zezhou Yang , Shuzheng Gao , Cuiyun Gao , Ting Peng , Hailiang Huang , Yuetang Deng , Michael Lyu This is my paper
classification cs.SE
keywords codecompletionretrieval-augmentedgenerationfine-tuninglargemodelsBM25industrialcodebaseempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish an empirical ranking of two ways to adapt large code models (LCMs) to a big proprietary C++ codebase: retrieval-augmented generation (RAG) and fine-tuning. The authors test six LCMs on 50,000 line-completion instances drawn from 20,000 internal files at Tencent's WXG department, using 120,000 files as the retrieval corpus and fine-tuning set. Their central finding is that RAG, especially with the lexical BM25 retriever, reaches higher completion accuracy than fine-tuning alone, and that combining fine-tuning with RAG performs best of all. If the paper is right, companies with large private repositories can improve code completion without retraining the model, keep their model's general abilities intact, and get still more from retrieval as their codebase grows.

What carries the argument

The mechanism that carries the argument is function-level retrieval-augmented generation. The 120,000-file corpus is decomposed by Tree-sitter into 914,667 functions and classes; a retriever selects the top-K most similar items to a 20-line query, and those items are prepended to the query in ascending similarity order before the LCM generates the next line. The load-bearing retriever is BM25, which scores snippets by exact term-frequency and inverse-document-frequency matches: in C++ code with strict syntax and distinctive business-specific identifiers, lexical overlap turns out to be a better relevance signal than the semantic similarities computed by the four neural embedding models. The comparison baseline is autoregressive fine-tuning, which trains the LCM on file-level token sequences with a next-token prediction loss.

What would settle it

Re-run the 50,000-instance benchmark with per-model hyperparameter search for fine-tuning (varying learning rate, batch size, and epochs with a validation-set stopping rule) and check whether BM25-based RAG still beats the best fine-tuned model; separately, repeat the same RAG protocol on an industrial codebase in a dynamically typed language to test the paper's claim that lexical matching wins because of C++'s strict syntax and domain-specific identifiers.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a measured performance ordering. Averaged over six models, base exact-match accuracy is 24.79; fine-tuning raises it by 78.3% to 44.20, while similarity-based RAG with BM25 raises it by 116.9% to 53.76, and fine-tuned models with BM25-based RAG reach 57.43. BM25, a lexical keyword-matching algorithm, outperforms every neural embedding retriever studied (CodeBERT, UniXcoder, CodeT5, and CoCoSoDa), while CodeBERT-based retrieval performs no better than random retrieval. Dependency-based retrieval, which prepends the definitions of functions called in the query, gives only a modest 8.9% gain. The paper also finds that fine-tuning costs 3.5 to 41.4 hours of GPU time and degrades general code-generation ability (average HumanEval pass@1 falls 24.1%), whereas RAG costs minutes to prepare but slows inference by up to 78% as retrieved snippets lengthen prompts, and that RAG keeps gaining from larger codebases after fine-tuning plateaus around 90,000 files.

Load-bearing premise

The load-bearing premise is that the single shared fine-tuning configuration—one learning rate, batch size, sequence length, and warm-up, with no reported epoch count or per-model tuning—is a fair and representative operating point for all six models; if some were undertrained or need different hyperparameters, the claim that RAG has a higher ceiling than fine-tuning could be an artifact of that configuration.

Editorial extensions

If this is right

  • For a company with a large proprietary codebase, a lexical retriever plus an unchanged base LCM can beat fine-tuning on line-level completion accuracy while preserving the model's general code abilities.
  • Fine-tuning and RAG are complementary: fine-tuned models use retrieved context better, so the strongest configuration is fine-tuning followed by RAG, with average gains of 7.79% exact match, 5.27% edit similarity, and 7.91% BLEU over base-model RAG.
  • RAG's advantage grows with data: from 90,000 to 120,000 files, fine-tuning gains only about 0.35% exact match while RAG-BM25 and RAG-CoCoSoDa gain about 2.26% and 2.13%, so retrieval is the better bet in data-rich settings.
  • Fine-tuning carries a generalization cost that RAG does not: average pass@1 drops 24.1% on HumanEval and 11.7% on MBPP, with smaller drops on reasoning and translation.
  • The trade-off is runtime latency: prepending top-5 BM25 snippets lengthens the average input from about 246 to about 2,500 tokens and cuts throughput from about 4,031 to 874 tokens per second.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's explanation for BM25's edge—strict C++ syntax and domain-specific identifiers—could be tested directly by running the same protocol on a Python or JavaScript industrial codebase, where identifier vocabulary is less discriminative and neural embeddings might close the gap.
  • Editorial extension: because the paper reports a single shared fine-tuning configuration without per-model hyperparameter tuning or a stopping criterion, a fair re-run with tuned fine-tuning could change the size of the RAG-over-FT margin, even if the direction of the ranking holds.
  • Editorial extension: since retrieval accuracy directly drives RAG gains, a system that fine-tunes the retriever itself (rather than the generator) might push the combined RAG+FT ceiling higher without triggering the catastrophic forgetting the paper measures.
  • Editorial extension: the throughput loss suggests a concrete follow-up: retrieve fewer or shorter units (statement-level snippets) or compress retrieved contexts, then re-measure accuracy and speed to find the Pareto front between the two.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a large-scale empirical comparison of retrieval-augmented generation (RAG) and fine-tuning (FT) for adapting large code models (LCMs) to an industrial C++ codebase. The authors collect over 160,000 internal Tencent WXG C++ files, use 120,000 files both as a retrieval corpus and as fine-tuning data, and evaluate six LCMs (Qwen2.5-Coder 0.5B/1.5B/3B/7B and DeepSeek-Coder 1.3B/6.7B) on a fixed test set of 50,000 code-completion instances. They compare effectiveness (exact match, edit similarity, BLEU), efficiency, and parameter sensitivity. The main claims are that RAG with BM25 retrieval achieves higher accuracy than fine-tuning alone, that fine-tuning and RAG are complementary and their combination yields further gains, and that RAG shows better scalability with corpus size than fine-tuning.

Significance. If the results hold, this is a practically important and actionable study. The industrial scale, the clean file-level split between test and retrieval/training corpora, and the breadth of six LCMs are clear strengths. The finding that BM25, a simple lexical retriever, outperforms neural embedding-based retrievers for line-level C++ completion is a falsifiable and surprising result that would interest both practitioners and researchers. The study also provides explicit guidance on the efficiency trade-off between RAG and fine-tuning. However, the strength of the central claim (RAG > FT) depends on the fine-tuning baseline being a converged, representative operating point, and the paper currently does not provide the necessary convergence evidence. A data error in the generalization analysis further weakens the secondary claims.

major comments (3)
  1. [§3.5, Table 2, Table 5] The fine-tuning protocol does not report the number of training epochs, total optimization steps, or any stopping criterion; Table 2 lists only the optimizer, learning rate, batch size, and sequence length, while Table 5 gives wall-clock hours but not step counts. Consequently, the fine-tuned models in Table 3 may be undertrained checkpoints rather than converged operating points. The headline comparison (RAG-BM25 EM 53.76 vs. Fine-tuning EM 44.20) could therefore reflect a non-representative fine-tuning configuration, and the same missing control threatens RQ4: the reported fine-tuning plateau from 90K to 120K files could be an artifact of a fixed training budget rather than an intrinsic scaling property. Please report per-model epoch counts, loss curves, validation-loss-based stopping criteria, or a per-model hyperparameter search, and confirm that each fine-tuned model has converged.
  2. [Table 7] In Table 7, the QC-7B fine-tuned row reports MBPP and MBPP+Plus values of 36.09 and 29.82, which are identical to the QC-0.5B fine-tuned values and dramatically lower than the QC-7B base values (70.43 and 57.89). This is inconsistent with the other rows and strongly suggests a copy-paste error. Because Table 7 is the sole basis for the catastrophic-forgetting discussion in §5.1 and the corresponding implication in §5.2.1, the error directly undermines the generalization claims. Please correct the table, recompute the average row, and re-evaluate the conclusions that depend on it.
  3. [§4.4, Figure 3] The scalability analysis and Finding 4 are based on small differences between conditions: fine-tuning improves by 0.35 percentage points in EM from 90K to 120K files, while RAG-BM25 improves by 2.26 points and RAG-CoCoSoDa by 2.13 points. These values are reported without confidence intervals, repeated runs, or any statistical test. With a single fine-tuning run per condition, it is unclear whether the observed plateau is a reliable effect or within-run noise. Please provide error bars or statistical tests (e.g., bootstrap over the 50,000 test instances, or multiple fine-tuning runs) to support the scalability conclusion.
minor comments (5)
  1. [Abstract] The abstract states that RAG 'when implemented with appropriate embedding models that map code snippets into dense vector representations' achieves higher accuracy, but the paper's best RAG variant is BM25, which is lexical and does not use dense embeddings. Please rephrase the abstract to avoid this inconsistency.
  2. [§2.1.2] The role of the validation split (23,746 files) is not described; it is unclear whether the validation set was used for early stopping, hyperparameter selection, or only for reporting. Please clarify.
  3. [§3.3] The Random retrieval baseline is included but not explained in detail; it would be helpful to state whether the same top-K concatenation is used and whether the small average degradation (EM 24.16 vs. base 24.79) is expected.
  4. [§3.5] The fine-tuning hyperparameters are said to follow 'previous work', but no citation is given for the specific values of learning rate, batch size, and sequence length. Please add the reference.
  5. [Table 1] The test instance construction is described briefly: 20 consecutive input lines and the next line as target. Please specify how the sliding window is positioned (e.g., whether the target line is always the line after the window or whether the window can end at any offset) and whether the window moves by fixed steps across each file.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RAG-vs-fine-tuning comparison is an empirical measurement on a held-out test set, with self-citations used only as methodological references.

full rationale

This paper is an empirical comparison, not a derivation, and its central claims are measured rather than constructed. The headline conclusion that RAG-BM25 outperforms fine-tuning, and that their combination is best, is evaluated on a held-out test set of 50,000 instances drawn from 20,000 C++ files that are disjoint from the 120,000-file retrieval corpus and fine-tuning training set; Section 2.1.2 explicitly selects the test files first and then splits the remaining files for retrieval and training. No parameter is fitted to the test set, and no equation defines the reported accuracies in terms of the compared methods' own outputs. The self-citations to prior work by overlapping authors, such as [10, 12, 62, 68], are used only to justify standard fine-tuning procedures and the ordering of concatenated retrieved demonstrations; they do not certify the empirical ranking, which is generated by the paper's own experiments. The main weakness, namely that the fine-tuning section reports no epoch count or stopping criterion, is a fairness and reproducibility concern about whether the fine-tuning operating point is representative, but it is not circularity because it does not make the RAG result equivalent to the fine-tuning input by construction. Accordingly, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No parameters were fitted to the effectiveness data; the ledger records the hand-chosen evaluation settings and domain assumptions on which the reported rankings depend.

free parameters (3)
  • Top-K retrieved snippets (K=5) = 5
    Main RAG comparisons prepend five retrieved code snippets to the query (Section 3.4, RQ4); the headline BM25 versus FT result is reported at K=5, and sensitivity to K is analyzed separately.
  • Fine-tuning hyperparameters = LR 5e-6, batch 32, sequence length 4096, warm-up 100
    One shared hyperparameter set is used for all six models (Section 3.5, Table 2); no per-model tuning or epoch count is reported, so the FT baseline is a chosen operating point rather than a tuned best.
  • Test instance construction window = 20 input lines, 1 target line
    The evaluation benchmark is defined by this window choice (Section 2.1.2); a different context length or multi-line target could change the relative advantage of RAG.
assumptions (4)
  • domain assumption The 120,000-file retrieval/training corpus and the 20,000-file test split are file-disjoint, and near-duplicate code patterns between them do not materially inflate RAG's exact-match scores.
    Section 2.1.2 describes random splitting but does not report deduplication of similar functions or statements across splits.
  • domain assumption Exact match on a single predicted line is a meaningful proxy for industrial code completion quality.
    Section 3.2 adopts EM, edit similarity, and BLEU from prior work; no user study or acceptance metric calibrates these proxies for real IDE use.
  • domain assumption Retrieved function and class snippets from elsewhere in the same repository are appropriate context for predicting the next line.
    Section 2.2 defines prompt concatenation of Top-K items, assuming cross-file similarity helps line-level completion.
  • standard math Standard definitions of BM25, cosine similarity, Levenshtein distance, and BLEU apply without modification.
    Used as background computational definitions in Sections 2.2 and 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry." pith.science (2026). https://pith.science/paper/Y75PHOYL

@misc{pith2026250515179,
  author       = {Pith},
  title        = {Pith review of: RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y75PHOYL}},
  note         = {Machine review of arXiv:2505.15179}
}
read the original abstract

Code completion, a crucial practice in industrial settings, helps developers improve programming efficiency by automatically suggesting code snippets during development. With the emergence of Large Code Models (LCMs), this field has witnessed significant advancements. Due to the natural differences between open-source and industrial codebases, such as coding patterns and unique internal dependencies, it is a common practice for developers to conduct domain adaptation when adopting LCMs in industry. There exist multiple adaptation approaches, among which retrieval-augmented generation (RAG) and fine-tuning are the two most popular paradigms. However, no prior research has explored the trade-off of the two approaches in industrial scenarios. To mitigate the gap, we comprehensively compare the two paradigms including Retrieval-Augmented Generation (RAG) and Fine-tuning (FT), for industrial code completion in this paper. In collaboration with Tencent's WXG department, we collect over 160,000 internal C++ files as our codebase. We then compare the two types of adaptation approaches from three dimensions that are concerned by industrial practitioners, including effectiveness, efficiency, and parameter sensitivity, using six LCMs. Our findings reveal that RAG, when implemented with appropriate embedding models that map code snippets into dense vector representations, can achieve higher accuracy than fine-tuning alone. Specifically, BM25 presents superior retrieval effectiveness and efficiency among studied RAG methods. Moreover, RAG and fine-tuning are orthogonal and their combination leads to further improvement. We also observe that RAG demonstrates better scalability than FT, showing more sustained performance gains with larger scales of codebase.

Figures

Figures reproduced from arXiv: 2505.15179 by the authors.

Figure 1
Figure 1. Average exact match and throughput under differ [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Exact match of the base and fine-tuned model under different retrieved contexts. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Exact match and edit similarity of fine-tuning and RAG under different sizes of codebase. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example that the QC-7B base model fails to predict [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Position Paper: Programming Language Techniques for Bridging LLM Code Generation Semantic Gaps

    cs.SE 2025-07 unverdicted novelty 2.0 of 10

    A position paper arguing that PL techniques, especially formal verification and structure-aware representations, should be deeply integrated into LLM code generation.

Reference graph

Works this paper leans on

77 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [1]

    Toufique Ahmed, Christian Bird, Premkumar Devanbu, and Saikat Chakraborty

    work page

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2...

    work page 2020

  3. [3]

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost.arXiv preprint arXiv:1604.06174(2016)

    arXiv 2016

  4. [4]

    ClangD. 2025. ClangD. https://clangd.llvm.org/

    work page 2025

  5. [5]

    copilot. 2025. copilot. https://copilot.microsoft.com/

    work page 2025

  6. [6]

    cursor. 2025. cursor. https://www.cursor.com/

    work page 2025

  7. [7]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems35 (2022), 16344–16359

    work page 2022

  8. [8]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al . 2020. CodeBERT: A Pre- Trained Model for Programming and Natural Languages. InFindings of the Asso- ciation for Computational Linguistics: EMNLP 2020. 1536–1547

    2020

Show all 77 references
  1. [9]

    Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. 2024. Search-based llms for code optimization. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 254–266

    2024

  2. [10]

    Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lunyiu Nie, Xin Xia, and Michael R. Lyu. 2023. Code Structure-Guided Transformer for Source Code Summarization.ACM Trans. Softw. Eng. Methodol.32, 1 (2023), 23:1–23:32

    2023

  3. [11]

    Shuzheng Gao, Wenxin Mao, Cuiyun Gao, Li Li, Xing Hu, Xin Xia, and Michael R. Lyu. 2024. Learning in the Wild: Towards Leveraging Unlabeled Data for Ef- fectively Tuning Pre-trained Code Models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineerin...

    2024

  4. [12]

    Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R Lyu. 2023. What makes good in-context demonstrations for code intelligence tasks with llms?. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 761–773

    2023

  5. [13]

    Shuzheng Gao, Hongyu Zhang, Cuiyun Gao, and Chaozheng Wang. 2023. Keeping Pace with Ever-Increasing Data: Towards Continual Learning of Code Intelligence Models. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023....

    2023

  6. [14]

    Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27...

    2024

  7. [15]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. InPro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7212–7225

    2022

  8. [16]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  9. [17]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the Knowl- edge in a Neural Network.CoRRabs/1503.02531 (2015)

    2015 arXiv

  10. [18]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    2024 arXiv

  11. [19]

    Loshchilov Ilya and Hutter Frank. 2018. Decoupled Weight Decay Regularization. International Conference on Learning Representations, ICLR(2018)

    2018

  12. [20]

    Maliheh Izadi, Roberta Gismondi, and Georgios Gousios. 2022. CodeFill: Multi- token Code Completion by Jointly learning from Structure and Naming Sequences. In44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022. ...

    2022

  13. [21]

    Zhenlan Ji, Pingchuan Ma, Zongjie Li, and Shuai Wang. 2023. Benchmarking and explaining large language model-based code generation: A causality-centric approach.arXiv preprint arXiv:2310.06680(2023)

    2023 arXiv

  14. [22]

    Zhenlan Ji, Daoyuan Wu, Pingchuan Ma, Zongjie Li, and Shuai Wang. 2024. Test- ing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs.arXiv preprint arXiv:2404.17833(2024)

    2024 arXiv

  15. [23]

    Ziheng Jiang, Tianqi Chen, and Mu Li. 2018. Efficient Deep Learning Inference on Edge Devices.MLsys(2018)

    2018

  16. [24]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models.arXiv preprint arXiv:1911.00172(2019)

    2019 arXiv

  17. [25]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of ...

    2017

  18. [26]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  19. [27]

    Guochang Li, Chen Zhi, Jialiang Chen, Junxiao Han, and Shuiguang Deng. 2024. Exploring Parameter-Efficient Fine-Tuning of Large Language Model on Auto- mated Program Repair. InProceedings of the 39th IEEE/ACM International Confer- ence on Automated Software Engineering, ASE 20...

    2024

  20. [28]

    Lyu, and Irwin King

    Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018. Code Completion with Neural Attention and Pointer Networks. InProceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden. ijcai.org, 4159–4165

    2018

  21. [29]

    Zongjie Li, Pingchuan Ma, Huaijin Wang, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. 2022. Unleashing the Power of Compiler Intermediate Representation to Enhance Neural Program Embeddings. In44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsb...

    2022

  22. [31]

    Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. CCTEST: Testing and Repairing Code Completion Systems. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1238–1250

    2023

  23. [32]

    Zongjie Li, Chaozheng Wang, Pingchuan Ma, Chaowei Liu, Shuai Wang, Daoyuan Wu, Cuiyun Gao, and Yang Liu. 2024. On Extracting Specialized Code Abilities from Large Language Models: A Feasibility Study. InProceedings of the IEEE/ACM 46th International Conference on Software Engi...

    2024

  24. [33]

    Zongjie Li, Chaozheng Wang, Shuai Wang, and Gao Cuiyun. 2023. Protecting Intellectual Property of Large Language Model-Based Code Generation APIs via Watermarks. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denm...

    2023

  25. [34]

    Zongjie Li, Daoyuan Wu, Shuai Wang, and Zhendong Su. 2024. API-guided Dataset Synthesis to Finetune Large Code Models.arXiv preprint arXiv:2408.08343 (2024)

    2024 arXiv

  26. [35]

    Tianyang Liu, Canwen Xu, and Julian J. McAuley. 2024. RepoBench: Benchmark- ing Repository-Level Code Auto-Completion Systems. InThe Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net

    2024

  27. [36]

    Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024. GraphCoder: Enhancing Repository-Level Code Completion via Coarse-to-fine Retrieval Based on Code Context Graph. InPro- ceedings of the 39th IEEE/ACM International Conference o...

    2024

  28. [37]

    Ilya Loshchilov and Frank Hutter. 2016. SGDR: Stochastic Gradient Descent with Warm Restarts. InInternational Conference on Learning Representations

    2016

  29. [38]

    Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svy- atkovskiy. 2022. ReACC: A Retrieval-Augmented Code Completion Framework. FSE Companion ’25, June 23–28, 2025, Trondheim, Norway Wang et al. InProceedings of the 60th Annual Meeting of the Association for...

    2022

  30. [39]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. [n. d.]. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understand- ing and Generation. InThirty-fifth Conference on Neural I...

  31. [40]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and S...

    2021

  32. [41]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. InThe Twelfth International Conference on Learning Representations, ICLR 2024,...

    2024

  33. [42]

    Pingchuan Ma, Zhaoyu Wang, Zongjie Li, Zhenlan Ji, Ao Sun, Juergen Rahmel, and Shuai Wang. 2025. Reeq: Testing and Mitigating Ethically Inconsistent Suggestions of Large Language Models with Reflective Equilibrium. (2025). https://doi.org/10.1145/3722554

    2025 doi

  34. [43]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in con- nectionist networks: The sequential learning problem. InPsychology of learning and motivation. Vol. 24. Elsevier, 109–165

    1989

  35. [44]

    Noor Nashid, Taha Shabani, Parsa Alian, and Ali Mesbah. 2024. Contextual API Completion for Unseen Repositories Using LLMs.arXiv preprint arXiv:2405.04600 (2024)

    2024 arXiv

  36. [45]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA. ACL, 311–318

    2002

  37. [46]

    Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang

    Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval Augmented Code Generation and Summarization. InFindings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-2...

    2021

  38. [47]

    Yun Peng, Shuzheng Gao, Cuiyun Gao, Yintong Huo, and Michael R. Lyu. 2024. Domain Knowledge Matters: Improving Prompts with Fix Templates for Re- pairing Python Type Errors. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon,...

    2024

  39. [48]

    Lutz Prechelt. 2000. An empirical comparison of c, c++, java, perl, python, rexx and tcl.IEEE Computer33, 10 (2000), 23–29

    2000

  40. [49]

    qdrant. 2025. qdrant. https://qdrant.tech//

    2025

  41. [50]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297(2020)

    2020 arXiv

  42. [51]

    Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner. 2022. AI and ML accelerator survey and trends. In2022 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 1–10

    2022

  43. [52]

    Stephen Robertson, Hugo Zaragoza, et al . 2009. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends®in Information Retrieval 3, 4 (2009), 333–389

    2009

  44. [53]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  45. [54]

    Ensheng Shi, Yanlin Wang, Wenchao Gu, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2023. Cocosoda: Effective contrastive learning for code search. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2198–2210

    2023

  46. [55]

    Ze Tang, Jidong Ge, Shangqing Liu, Tingwei Zhu, Tongtong Xu, Liguo Huang, and Bin Luo. 2023. Domain adaptive code completion via language models and decoupled domain databases. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 421–433

    2023

  47. [56]

    tantivy. 2025. tantivy. https://github.com/quickwit-oss/tantivy

    2025

  48. [57]

    tree sitter. 2025. tree-sitter. https://tree-sitter.github.io/tree-sitter/

    2025

  49. [58]

    Chaozheng Wang, Shuzheng Gao, Cuiyun Gao, Wenxuan Wang, Chun Yong Chong, Shan Gao, and Michael R. Lyu. 2024. A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How. InProceedings of the 39th IEEE/ACM International Conference on Automated Software ...

    2024

  50. [59]

    Chaozheng Wang, Junhao Hu, Cuiyun Gao, Yu Jin, Tao Xie, Hailiang Huang, Zhenyu Lei, and Yuetang Deng. 2023. How Practitioners Expect Code Com- pletion?. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software En...

    2023

  51. [60]

    Chaozheng Wang, Zongjie Li, Cuiyun Gao, Wenxuan Wang, Ting Peng, Hail- iang Huang, Yuetang Deng, Shuai Wang, and Michael R Lyu. 2024. Exploring Multi-Lingual Bias of Large Code Models in Code Generation.arXiv preprint arXiv:2404.19368(2024)

    2024 arXiv

  52. [61]

    Chaozheng Wang, Zongjie Li, Yun Peng, Shuzheng Gao, Sirong Chen, Shuai Wang, Cuiyun Gao, and Michael R. Lyu. 2023. REEF: A Framework for Collecting Real-World Vulnerabilities and Fixes. In38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxem...

    2023

  53. [62]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R. Lyu. [n. d.]. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on ...

    2022

  54. [63]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R. Lyu. 2023. Prompt Tuning in Code Intelligence: An Experimental Evaluation.IEEE Trans. Software Eng.49, 11 (2023), 4869–4885

    2023

  55. [64]

    Weishi Wang, Yue Wang, Shafiq Joty, and Steven C. H. Hoi. 2023. RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair. InProceedings of the 31st ACM Joint European Software Engineering Con- ference and Symposium on the Foundations of Software E...

    2023

  56. [65]

    Yanlin Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng

  57. [66]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. CodeT5: Identifier- aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 8696–8708

    2021

  58. [67]

    CoRRabs/2407.19487 (2024)

    RLCoder: Reinforcement Learning for Repository-Level Code Completion. CoRRabs/2407.19487 (2024)

    2024 arXiv

  59. [68]

    Xin-Cheng Wen, Cuiyun Gao, Shuzheng Gao, Yang Xiao, and Michael R. Lyu

  60. [69]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering Code Generation with OSS-Instruct. InForty-first Inter- national Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

    2024

  61. [70]

    Wai Kin Wong, Huaijin Wang, Zongjie Li, Zhibo Liu, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. 2023. Refining decompiled c code with large language models. arXiv preprint arXiv:2310.06530(2023)

    2023 arXiv

  62. [71]

    InProceedings of the 33rd ACM SIGSOFT Inter- national Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024

    SCALE: Constructing Structured Natural Language Comment Trees for Software Vulnerability Detection. InProceedings of the 33rd ACM SIGSOFT Inter- national Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024. ACM, 235–247

    2024

  63. [72]

    F George Wilkie and B Hylands. 1998. Measuring complexity in C++ application software.Software: Practice and Experience28, 5 (1998), 513–546

    1998

  64. [73]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. InProceedings of the 2023 Conference on Empirical Methods in Natural La...

    2023

  65. [74]

    Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Chatqa 2: Bridging the gap to proprietary llms in long context and rag capabilities.arXiv preprint arXiv:2407.14482(2024)

    2024 arXiv

  66. [75]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 Technical Report.arXiv preprint arXiv:2412.15115(2024)

    2024 arXiv

  67. [77]

    Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020. Retrieval-based neural source code summarization. InICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020. ACM, 1385–1397

    2020

  68. [78]

    Kunpeng Zhang, Zongjie Li, Daoyuan Wu, Shuai Wang, and Xin Xia. 2025. Low- Cost and Comprehensive Non-textual Input Fuzzing with LLM-Synthesized Input Generators.arXiv preprint arXiv:2501.19282(2025)

    2025 arXiv

  69. [2024]

    Studying LLM Performance on Closed-and Open-source Data.arXiv preprint arXiv:2402.15100(2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.