Pith. sign in

REVIEW 4 major objections 6 minor 74 references

For repository-level code completion, a lexical-only retrieval pipeline built on ripgrep search commands, BM25 re-ranking, and line-interval deduplication outperforms semantic, graph-based, and reinforcement-learned retrieval methods on sta

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:08 UTC pith:36WWO25T

load-bearing objection A genuinely useful empirical result — a grep-plus-BM25 pipeline beats graph/RL RAG on CrossCodeEval — but the RepoEval_Updated half of the claim is weakened by test-set hyperparameter tuning and a couple of data hygiene issues. the 4 major comments →

arxiv 2601.23254 v3 pith:36WWO25T submitted 2026-01-30 cs.SE

Better Call Grep: Evaluating and Improving Grep-Like Lexical Retrieval for Repository-Level Code Completion

Baoyi Wang , Xingliang Wang , Guochang Li , Chen Zhi , Junxiao Han , Xinkui Zhao , Nan Wang , Shuiguang Deng
show 1 more author
Jianwei Yin
This is my paper
classification cs.SE
keywords repository-level code completionlexical retrievalripgrepretrieval-augmented generationBM25 re-rankingdeduplicationlarge language modelscross-file context
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Repository-level code completion needs cross-file context, and existing retrieval systems pay a heavy price for it: embedding indexes and code-context graphs take seconds to tens of seconds per query and grow stale as code changes. This paper asks how far a developer-style tool can go: the LLM inspects the incomplete code, writes a handful of grep search commands, and the raw hits are ranked and trimmed as context. The central claim is that this index-free lexical pipeline is not merely passable but state of the art—a naive version already matches graph-based and reinforcement-learned retrieval, and an optimized version improves exact-match accuracy by 7.04–15.58 percent relative over the best baseline on standard benchmarks while keeping retrieval at milliseconds.

Core claim

The paper establishes that explicit lexical matching of code identifiers is a stronger and cheaper signal for routing cross-file context into completion prompts than the semantic or structural similarity used by existing RAG methods. It reports that Naive GrepRAG—ten LLM-written ripgrep queries run unchanged, with snippets ranked by Jaccard similarity—already matches or beats graph-based and reinforcement-learned retrievers on Python and Java completions, and it shows through failure analysis that competing baselines usually lose the relevant code at the coarse-retrieval stage or demote it during re-ranking. The paper then diagnoses two failure modes of the naive pipeline—high-frequency ambi

What carries the argument

The load-bearing mechanism is the LLM-generated ripgrep query set, which converts the completion site into a handful of targeted identifier searches rather than a global similarity scan. GrepRAG then applies two post-processing steps: identifier-weighted re-ranking—BM25's inverse-document-frequency term demotes generic tokens like init while promoting rare, task-specific identifiers—and structure-aware deduplication, which reads each chunk's line-number interval to merge overlapping and adjacent hits into one contiguous block before the context is assembled.

Load-bearing premise

The completion site must contain a word (or a wildcard pattern) that also appears in the code the model needs to see, so a small set of generated search commands can find it; when the dependency is implicit—like an inheritance relation with no shared identifier—lexical retrieval has nothing to match and the approach fails.

What would settle it

Construct a completion benchmark (or filter an existing one) so that each target definition shares no lexical token with the completion site—for example, all dependencies introduced exclusively through inheritance, overrides, or factory registration—and measure GrepRAG's recall and exact-match accuracy against a structure-aware baseline; the claim predicts GrepRAG's recall falls toward its retrieval-miss rate (roughly 25–29% of failures) and its completion accuracy drops to or below the no-retrieval baseline on those cases.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, the default architecture for repository-level completion should start with a zero-index lexical retriever; graph indices and dense embeddings become an optional add-on rather than a prerequisite, since the paper measures lexical retrieval at roughly 1/35th the latency of graph-based methods on a 750K-line repository.
  • BM25's role is reversed: as a coarse global retriever it misses critical context, but as a re-ranker over a small, completion-aware candidate set it is the best available ranker, explaining why baselines fail in the coarse stage while GrepRAG succeeds by retrieving explicit identifiers first.
  • Structure-aware merging recovers several percentage points of exact match by keeping context contiguous and avoiding token waste; the ablation attributes roughly 3.3 EM points on Python to deduplication versus 0.5 points to re-ranking alone.
  • A distilled 0.6B model can generate the retrieval commands as effectively as much larger LLMs, bringing the total RAG pipeline to about two seconds on large repositories—under the practical latency threshold cited in the paper.
  • The performance gain is consistent across two different backbone LLMs, suggesting the benefit comes from the retrieval pipeline itself rather than from any single model's quirks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If lexical reachability decides success, the reported gains should shrink on cases where dependencies are implicit—such as inheritance with no shared identifier at the completion site; the paper's own Section 4.3.3 identifies this class but does not quantify its frequency, so the advantage over structure-aware retrieval on real repositories depends on how often such implicit cases occur.
  • The manuscript's Discussion explicitly flags that evaluation corpora may appear in LLM pretraining data, arguing that relative comparisons with identical backbones mitigate contamination; the absolute improvements should therefore be read as relative evidence, not absolute capability.
  • The same pipeline—query generation, identifier-weighted ranking, and interval fusion—transfers naturally to other code-retrieval tasks such as bug localization, code search, or agentic tool use, where identifier precision and low retrieval cost matter.
  • Because a distilled command generator preserved performance, the method's marginal cost could drop further; the remaining bottleneck for practical deployment is likely context-budget management and query generation latency, not retrieval itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper asks how far simple, index-free lexical retrieval can go for repository-level code completion. It introduces Naive GrepRAG, a pipeline in which an LLM generates ripgrep commands, the commands are executed against the repository, and raw matches are reranked with Jaccard similarity before being fed to a completion backbone. Experiments on CrossCodeEval show that this simple baseline already matches or exceeds graph-based, RL-based, and BM25-based RAG baselines on code EM. The paper then identifies two failure modes of the naive approach—keyword ambiguity causing ranking noise, and redundancy/fragmentation from overlapping retrieval hits—and proposes GrepRAG, which adds BM25 identifier-weighted reranking and line-interval based structural deduplication. On CrossCodeEval and RepoEval_Updated, with DeepSeek-V3.2-EXP and Qwen3-Coder-Plus backbones, GrepRAG is reported to consistently outperform all baselines; for example, on CrossCodeEval Python/DeepSeek, code EM rises from 38.61% (Naive) to 42.29% versus 36.59% for the best baseline RLCoder. The paper also distills command generation into a 0.6B model and reports that it retains the accuracy of large instruction generators at lower cost.

Significance. If the headline results hold, the paper makes a substantive point: for the benchmarks studied, a cheap lexical-retrieval pipeline with a small amount of post-processing can outperform substantially more complex structure-aware and RL-trained retrievers. The paper’s strengths are its breadth (two languages, two backbones, two benchmarks), its transparent ablations showing that deduplication contributes more than reranking, and its additional generalization and distillation experiments. The work also usefully quantifies retrieval latency and shows that index construction is a bottleneck for graph-based methods. However, the claim of consistent superiority is currently weakened by three concrete concerns that must be addressed: the central hyperparameter N is selected on RepoEval_Updated and then used when reporting that same benchmark; the main results table contains a suspicious copy-pasted row; and the reported retrieval times exclude LLM query-generation latency, which the paper itself later discloses is nontrivial.

major comments (4)
  1. [§5.6 and Table 5] The de-duplication candidate-pool size N is chosen on RepoEval_Updated itself. Fig. 7 sweeps N from 10% to 90% on RepoEval_Updated, identifies N=50% as the optimum, and §5.2 sets N=50% as the default; Table 5 then reports RepoEval_Updated results using that N. No held-out validation split is described. Consequently, the RepoEval_Updated half of the headline "consistent SOTA" claim is confounded with test-set hyperparameter selection. The API-level Java gain (45.67 vs 42.87) is the most exposed. Please report a validation-based selection (e.g., choose N on CrossCodeEval or on a split of RepoEval_Updated) and re-report the RepoEval_Updated numbers under that choice.
  2. [Table 5, RepoEval_Updated, Line-Level, Java] The DeepSeek half of the No RAG row for Java is bit-for-bit identical to the Python No RAG row (EM 34.25, ES 64.29, Recall 82.79, F1 80.36, Id-EM 40.80, Id-ES 66.16, Id-Recall 59.09, Id-F1 56.15). Since these are different repositories and languages, this is either a transcription error or an evaluation-harness bug. Because Table 5 is the paper's central evidence, this error must be corrected and adjacent rows rechecked before the comparative claims can be assessed.
  3. [§2, Table 5 vs. §6, Table 8] The "Retrieval Time" column in Tables 3 and 5 reports ~0.02s for Naive GrepRAG/GrepRAG, but this measures only ripgrep execution and excludes the LLM-generated ripgrep command generation. §6 states this additional inference is non-trivial: Table 8 lists GrepRAG (0.6B Distilled) end-to-end pipeline time as ~1.9s and omits GrepRAG with DeepSeek entirely "due to additional network latency." The efficiency motivation in §2 compares ripgrep execution to index-based retrieval without accounting for this cost. Please report end-to-end latency including query generation for all variants, and relabel the existing column so readers know it is raw grep time only.
  4. [§4.2.3, Eq. (1)] The failure-mode analysis defines the golden context C_gold as the set of fragments retrieved by Naive GrepRAG, and then classifies a baseline as having "recall failure" when its retrieved set covers less than 80% of C_gold's lines. Since S_unique consists of examples Naive GrepRAG solved, this is partially circular: a baseline that retrieves different but equally sufficient context is classified as recall failure. The taxonomy is still suggestive, but the paper should either use the actual ground-truth lines required for completion as C_gold, or demonstrate via an alternative golden-context definition that the recall-vs-reranking split is robust.
minor comments (6)
  1. [§3.3] The prompt is said to be "provided in the anonymized repository described in Section ??," but the cross-reference is unresolved. Please supply the actual repository link or include the prompt in the manuscript.
  2. [§4.1.3 and Fig. 4] The paragraph describing the Venn diagram refers to "GrepRAG uniquely resolved 161 instances," but at that point the method is Naive GrepRAG. Please keep the terminology consistent to avoid confusion with the later optimized GrepRAG.
  3. [Table 3 and Table 5] The 16-metric rows are very dense and difficult to parse. Consider splitting the results into separate tables per dataset or grouping code-match and identifier-match metrics into adjacent blocks.
  4. [Table 7] The claim that the choice of instruction generator has "little effect" is based on differences of roughly 0.3–1.3 EM points. No variance or significance estimate is provided, so the reader cannot assess whether these gaps are within noise.
  5. [§3.1 and §3.5] Please clarify whether the baselines (GraphCoder, RepoFuse, RLCoder) were re-run under the paper's exact setting (e.g., same K, same context window) or whether the numbers are taken from the original papers, and state any hyperparameter adaptations in a supplementary section.
  6. [§6] The knowledge-distillation experiment would benefit from more detail on the training data construction, the number of training examples, fine-tuning hyperparameters, and whether the 0.6B model was used for both query generation and backbones in Table 8.

Circularity Check

1 steps flagged

Main SOTA claim is measured on external benchmarks with shared backbones; the only self-referential step is the RQ2 failure decomposition, where 'golden context' is defined as Naive GrepRAG's own retrieved set.

specific steps
  1. self definitional [Section 4.2.3, Eq. (1) and Table 4]
    "Here, the set of code fragments retrieved by Naive GrepRAG for each sample serves as the golden context, denoted C_gold. For each baseline, we define the coverage of C_gold by its retrieved set C_retrieved as the line-level intersection ratio: I(C_retrieved, C_gold) = |Lines(C_retrieved) ∩ Lines(C_gold)| / |Lines(C_gold)| (1)"

    The paper's explanation of why baselines fail (Table 4: recall vs re-ranking failure) is scored against a 'golden context' that is by definition the output of the system being compared. A baseline that located the same correct definition but with different chunk boundaries or in differently bounded line intervals can be counted as not recalling the critical context, so the split into Type I/II failures is partly manufactured by the choice of reference set. This makes the causal conclusion that baselines mainly fail at coarse retrieval ('most baseline failures ... are due to coarse retrieval') self-referential rather than an independent measure of baseline retrieval quality. The end-to-end EM comparisons in Tables 3 and 5 are unaffected by this local definition.

full rationale

The central empirical claim—GrepRAG outperforms SOTA retrieval for repository-level code completion—is tested on the external CrossCodeEval and RepoEval_Updated benchmarks with identical backbone models (DeepSeek-V3.2-EXP and Qwen3-Coder-Plus) for every method, so no completion number is the output of a fitted parameter or an equivalent-by-construction equation. The ablation (§5.4) and command-generation generalization (§5.5) are also self-contained comparisons. Self-citations (e.g., GRACE [50]) appear only in related-work taxonomies and are not load-bearing. The one genuinely self-referential element is the failure decomposition in §4.2.3, where Naive GrepRAG's own retrieved set is called the 'golden context' (Eq. 1); this biases the Type I/II explanation but not the headline benchmark numbers. Two validity concerns are worth separating from circularity: §5.6 selects N=50% by sweeping N on RepoEval_Updated and then reports RepoEval_Updated results with that N, so the RepoEval_Updated half of the 'consistent SOTA' claim lacks a held-out validation split; and §3.3 promises the prompt in a repository 'described in Section ??', with code/artifacts not provided. These are evaluation-validity and reproducibility risks, not cases where a claimed prediction reduces by construction to its input.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's claims rest on benchmark assumptions, the lexically-reachable-context premise, and several hand-set hyperparameters (N, m, K, token budget). It introduces no new physical or ontological entities; GrepRAG is a software pipeline.

free parameters (5)
  • N (dedup candidate pool ratio) = 50%
    Set after observing an inverted-U sensitivity pattern on RepoEval_Updated (Fig. 7) and then applied to report gains on the same dataset; a tuned hyperparameter, not a parameter-free default.
  • m (number of ripgrep queries per completion) = 10
    Prompt instructs the LLM to generate about 10 commands (§3.3); affects recall and latency, with no sensitivity analysis.
  • K (top chunks fed to LLM) = 10
    Top-K retrieval budget standardized across all RAG baselines (§3.5); inherited from GraphCoder/RepoFuse conventions rather than derived.
  • context_token_limit = 4,096 tokens
    Fixed context window for all methods (§3.5); central to truncation and fragmentation behavior.
  • BM25 k1/b parameters = unstated (defaults assumed)
    BM25 is the core re-ranking component, but exact k1 and b values are not reported.
axioms (4)
  • domain assumption CrossCodeEval and RepoEval_Updated gold completions and cross-file filtering are valid proxies for repository-level completion quality.
    All conclusions are measured on these two benchmarks (§3.1); benchmark flaws would directly affect the findings.
  • domain assumption Relevant cross-file context is lexically reachable via a small set of exact or wildcard identifier queries.
    This is the enabling premise of grep-based retrieval; RQ3's 'Implicit Dependencies' failure shows it is not always true (§4.3.3).
  • ad hoc to paper Naive GrepRAG's retrieved set can serve as golden context for measuring baseline recall (Eq. 1).
    Failure analysis defines C_gold as Naive GrepRAG output (§4.2.3), which biases recall measurement against baselines that retrieve different-but-useful context.
  • domain assumption Relative gains measured with identical backbone models mitigate LLM pretraining contamination of benchmarks.
    The data-contamination discussion (§6) assumes shared bias cancels in relative comparisons; plausible but unverified.

pith-pipeline@v1.3.0-alltime-deepseek · 26220 in / 14925 out tokens · 159153 ms · 2026-08-03T06:08:09.479342+00:00 · methodology

0 comments
read the original abstract

Repository-level code completion remains challenging for large language models (LLMs) due to cross-file dependencies and limited context windows. Prior work addresses this challenge using Retrieval-Augmented Generation (RAG) frameworks based on semantic indexing or structure-aware graph analysis, but these approaches incur substantial computational overhead for index construction and maintenance. Motivated by common developer workflows that rely on lightweight search utilities (e.g., ripgrep), we revisit a fundamental yet underexplored question: how far can simple, index-free lexical retrieval support repository-level code completion before more complex retrieval mechanisms become necessary? To answer this question, we systematically investigate lightweight, index-free, intent-aware lexical retrieval through extensive empirical analysis. We first introduce Naive GrepRAG, a baseline framework in which LLMs autonomously generate ripgrep commands to retrieve relevant context. Despite its simplicity, Naive GrepRAG achieves performance comparable to sophisticated graph-based baselines. Further analysis shows that its effectiveness stems from retrieving lexically precise code fragments that are spatially closer to the completion site. We also identify key limitations of lexical retrieval, including sensitivity to noisy matches from high-frequency ambiguous keywords and context fragmentation caused by rigid truncation boundaries. To address these issues, we propose GrepRAG, which augments lexical retrieval with a lightweight post-processing pipeline featuring identifier-weighted re-ranking and structure-aware deduplication. Extensive evaluation on CrossCodeEval and RepoEval-Updated demonstrates that GrepRAG consistently outperforms state-of-the-art (SOTA) methods, achieving 7.04-15.58 percent relative improvement in code exact match (EM) over the best baseline on CrossCodeEval.

Figures

Figures reproduced from arXiv: 2601.23254 by Baoyi Wang, Chen Zhi, Guochang Li, Jianwei Yin, Junxiao Han, Nan Wang, Shuiguang Deng, Xingliang Wang, Xinkui Zhao.

Figure 1
Figure 1. Figure 1: Overview of the research framework. We first evaluate the effectiveness of Naive GrepRAG (RQ1), [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of GraphCoder and Grep in a method invocation scenario. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the Naive GrepRAG framework. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of distinct solved cases across systems in the CrossCodeEval dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Redundancy and Context Fragmentation. Two independent grep queries hit adjacent regions within [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Post-processing pipeline built upon the Naive GrepRAG framework. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sensitivity analysis of the de-duplication candidate pool size ( [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 1 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Imtiaz Ahmed, Sadman Islam, Partha Protim Datta, Imran Kabir, Naseef Ur Rahman Chowdhury, and Ahshanul Haque

  3. [3]

    Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2017. Learning to represent programs with graphs. arXiv preprint arXiv:1711.00740(2017)

    Pith/arXiv arXiv 2017

  4. [4]

    Sven Amann, Sebastian Proksch, Sarah Nadi, and Mira Mezini. 2016. A study of visual studio usage in practice. In2016 ieee 23rd international conference on software analysis, evolution, and reengineering (saner), Vol. 1. IEEE, 124–134

    2016

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923(2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Mark Chen. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  7. [7]

    Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.Advances in Neural Information Processing Systems36 (2023), 46701–46723

    2023

  8. [8]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. Cocomic: Code completion by jointly modeling in-file and cross-file context.arXiv preprint arXiv:2212.10007(2022)

    Pith/arXiv arXiv 2022

  9. [9]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow.arXiv preprint arXiv:2009.08366 (2020)

    Pith/arXiv arXiv 2020

  10. [10]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196(2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Junxiao Han, Shuiguang Deng, David Lo, Chen Zhi, Jianwei Yin, and Xin Xia. 2021. An empirical study of the landscape of open source projects in Baidu, Alibaba, and Tencent. In2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 298–307

    2021

  12. [12]

    Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs.Advances in Neural Information Processing Systems31 (2018)

    2018

  13. [13]

    Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016. On the naturalness of software. Commun. ACM59, 5 (2016), 122–131

    2016

  14. [14]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024). , Vol. 1, No. 1, Article . Publication date: February 2026. 20 Baoyi Wang, Xingliang Wang, Guochang Li, Chen Zhi, Junxiao Han, Xinkui Zhao, Nan Wang, Shuig...

    Pith/arXiv arXiv 2024

  15. [15]

    Paul Jaccard. 1901. Distribution de la flore alpine dans le bassin des Dranses et dans quelques régions voisines.Bull Soc Vaudoise Sci Nat37 (1901), 241–272

    1901

  16. [16]

    Levenshtein

    Vladimir I. Levenshtein. 1965. Binary codes capable of correcting deletions, insertions, and reversals.Soviet physics. Doklady10 (1965), 707–710. https://api.semanticscholar.org/CorpusID:60827152

    1965

  17. [17]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks(NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 793, 16 pages

    2020

  18. [18]

    Guochang Li, Yuchen Liu, Zhen Qin, Yunkun Wang, Jianping Zhong, Chen Zhi, Binhua Li, Fei Huang, Yongbin Li, and Shuiguang Deng. 2025. Empowering RepoQA-Agent based on Reinforcement Learning Driven by Monte-carlo Tree Search.arXiv preprint arXiv:2510.26287(2025)

    arXiv 2025

  19. [19]

    Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2023. Acecoder: Utilizing existing code to enhance code generation. arXiv preprint arXiv:2303.17780(2023)

    Pith/arXiv arXiv 2023

  20. [20]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161 (2023)

    Pith/arXiv arXiv 2023

  21. [21]

    Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, and Nan Duan. 2022. CodeRetriever: Large-scale contrastive pre-training for code search.arXiv preprint arXiv:2201.10866 (2022)

    Pith/arXiv arXiv 2022

  22. [22]

    Zeju Li, Changran Xu, Zhengyuan Shi, Zedong Peng, Yi Liu, Yunhao Zhou, Lingfeng Zhou, Chengyu Ma, Jianyuan Zhong, Xi Wang, et al . 2025. Deepcircuitx: A comprehensive repository-level dataset for rtl code understanding, generation, and ppa analysis.arXiv preprint arXiv:2502.18297(2025)

    Pith/arXiv arXiv 2025

  23. [23]

    Ming Liang, Xiaoheng Xie, Gehao Zhang, Xunjin Zheng, Peng Di, Hongwei Chen, Chengpeng Wang, Gang Fan, et al

  24. [24]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434(2024)

    Pith/arXiv arXiv 2024

  25. [25]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172(2023)

    Pith/arXiv arXiv 2023

  27. [27]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems.arXiv preprint arXiv:2306.03091(2023)

    Pith/arXiv arXiv 2023

  28. [28]

    Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024. Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model.arXiv preprint arXiv:2406.07003(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173(2024)

    Pith/arXiv arXiv 2024

  30. [30]

    Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022. Reacc: A retrieval- augmented code completion framework.arXiv preprint arXiv:2203.07722(2022)

    Pith/arXiv arXiv 2022

  31. [31]

    Mohamed Nejjar, Luca Zacharias, Fabian Stiehle, and Ingo Weber. 2025. Llms for science: Usage for code generation and data analysis.Journal of Software: Evolution and Process37, 1 (2025), e2723

    2025

  32. [32]

    Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, Shafiq Joty, Yingbo Zhou, Dragomir Radev, Arman Cohan, and Arman Cohan. 2024. L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models.Transactions of the Association for Computational Linguistics12 (2024...

    work page doi:10.1162/tacl_a_00705 2024

  33. [33]

    Zhenyu Pan, Xuefeng Song, Yunkun Wang, Rongyu Cao, Binhua Li, Yongbin Li, and Han Liu. 2025. Do Code LLMs Understand Design Patterns?. In2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE, 209–212

    2025

  34. [34]

    Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. InFindings of the Association for Computational Linguistics: EMNLP 2021. 2719– 2734

    2021

  35. [35]

    Huy Nhat Phan, Hoang Nhat Phan, Tien N Nguyen, and Nghi DQ Bui. 2024. Repohyper: Better context retrieval is all you need for repository-level code completion.CoRR(2024)

    2024

  36. [36]

    Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. InProceedings of the 35th ACM SIGPLAN conference on programming language design and implementation. 419–428. , Vol. 1, No. 1, Article . Publication date: February 2026. GrepRAG: An Empirical Study and Optimization of Grep-Like Retrieval for Code Completion 21

    2014

  37. [37]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084(2019)

    Pith/arXiv arXiv 2019

  38. [38]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends®in Information Retrieval3, 4 (2009), 333–389

    2009

  39. [39]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    Pith/arXiv arXiv 2023

  40. [40]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou

  41. [41]

    Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. 2023. Repofusion: Training code models to understand your repository.arXiv preprint arXiv:2306.10998(2023)

    Pith/arXiv arXiv 2023

  42. [42]

    Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. InInternational Conference on Machine Learning. PMLR, 31693–31715

    2023

  43. [43]

    Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, Mingze Ni, Li Li, and David Lo. 2025. Don’t complete it! Preventing unhelpful code completion for productive and sustainable neural code completion systems.ACM Transactions on Software Engineering and Methodology34, 1 (2025), 1–22

    2025

  44. [44]

    Hanzhuo Tan, Qi Luo, Ling Jiang, Zizheng Zhan, Jing Li, Haotian Zhang, and Yuqun Zhang. 2024. Prompt-based code completion via multi-retrieval augmented generation.ACM Transactions on Software Engineering and Methodology (2024)

    2024

  45. [45]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805(2023)

    Pith/arXiv arXiv 2023

  46. [46]

    Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q Vera Liao, and Jennifer Wortman Vaughan. 2025. Generation probabilities are not enough: Uncertainty highlighting in ai code completions.ACM Transactions on Computer-Human Interaction32, 1 (2025), 1–30

    2025

  47. [47]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  48. [48]

    Chaozheng Wang, Junhao Hu, Cuiyun Gao, Yu Jin, Tao Xie, Hailiang Huang, Zhenyu Lei, and Yuetang Deng. 2023. How practitioners expect code completion?. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1294–1306

    2023

  49. [49]

    Chong Wang, Jian Zhang, Yebo Feng, Tianlin Li, Weisong Sun, Yang Liu, and Xin Peng. 2025. Teaching code llms to use autocompletion tools in repository-level code generation.ACM Transactions on Software Engineering and Methodology 34, 7 (2025), 1–27

    2025

  50. [50]

    Xingliang Wang, Baoyi Wang, Chen Zhi, Junxiao Han, Xinkui Zhao, Jianwei Yin, and Shuiguang Deng. 2025. GRACE: Graph-Guided Repository-Aware Code Completion through Hierarchical Code Fusion.arXiv preprint arXiv:2509.05980 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    Yibo Wang, Lei Wang, Yue Deng, Keming Wu, Yao Xiao, Huanjin Yao, Liwei Kang, Hai Ye, Yongcheng Jing, and Lidong Bing. 2026. DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation.arXiv preprint arXiv:2601.09688(2026)

    arXiv 2026

  52. [52]

    Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. Rlcoder: Reinforce- ment learning for repository-level code completion.arXiv preprint arXiv:2407.19487(2024)

    Pith/arXiv arXiv 2024

  53. [53]

    Yunkun Wang, Yue Zhang, Guochang Li, Chen Zhi, Binhua Li, Fei Huang, Yongbin Li, and Shuiguang Deng. 2025. InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration.arXiv preprint arXiv:2510.18327(2025)

    arXiv 2025

  54. [54]

    Yunkun Wang, Yue Zhang, Zhen Qin, Chen Zhi, Binhua Li, Fei Huang, Yongbin Li, and Shuiguang Deng. 2025. Explo- raCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 18124–18145

    2025

  55. [55]

    Zixuan Wu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2025. LLMAppHub: A Large Collection of LLM-based Applications for the Research Community. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1254–1255

    2025

  56. [56]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval. 641–649

    2024

  57. [57]

    Haoran Xu, Chen Zhi, Junxiao Han, Xinkui Zhao, Jianwei Yin, and Shuiguang Deng. 2025. Revisiting Vulnerability Patch Localization: An Empirical Study and LLM-Based Solution.arXiv preprint arXiv:2509.15777(2025). , Vol. 1, No. 1, Article . Publication date: February 2026. 22 Baoyi Wang, Xingliang Wang, Guochang Li, Chen Zhi, Junxiao Han, Xinkui Zhao, Nan W...

    arXiv 2025

  58. [58]

    Haoran Xu, Chen Zhi, Tianyu Xiang, Zixuan Wu, Gaorong Zhang, Xinkui Zhao, Jianwei Yin, and Shuiguang Deng

  59. [59]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  60. [60]

    Jian Yang, Wei Zhang, Yibo Miao, Shanghaoran Quan, Zhenhe Wu, Qiyao Peng, Liqun Yang, Tianyu Liu, Zeyu Cui, Binyuan Hui, et al. 2025. Qwen2. 5-xCoder: Multi-Agent Collaboration for Multilingual Code Instruction Tuning. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 13121–13131

    2025

  61. [61]

    Zezhou Yang, Sirong Chen, Cuiyun Gao, Zhenhao Li, Xing Hu, Kui Liu, and Xin Xia. 2025. An empirical study of retrieval-augmented code generation: Challenges and opportunities.ACM Transactions on Software Engineering and Methodology(2025)

    2025

  62. [62]

    In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

    Prioritizing Large-Scale Natural Language Test Cases at OPPO. In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 458–468

  63. [63]

    Yang Yao, Yixu Wang, Yuxuan Zhang, Yi Lu, Tianle Gu, Lingyu Li, Dingyi Zhao, Keming Wu, Haozhe Wang, Ping Nie, et al. 2025. A Rigorous Benchmark with Multidimensional Evaluation for Deep Research Agents: From Answers to Reports.arXiv preprint arXiv:2510.02190(2025)

    arXiv 2025

  64. [64]

    Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2022. When language model meets private library.arXiv preprint arXiv:2210.17236(2022)

    Pith/arXiv arXiv 2022

  65. [65]

    Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2022. Large language models meet NL2Code: A survey.arXiv preprint arXiv:2212.09420(2022)

    Pith/arXiv arXiv 2022

  66. [66]

    thinking with long videos

    Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, Sicong Leng, Yifan Zhang, Bo Li, Chengwei Qin, Shijian Lu, Xingxuan Li, et al. 2025. Longvt: Incentivizing" thinking with long videos" via native tool calling.arXiv preprint arXiv:2511.20785(2025)

    Pith/arXiv arXiv 2025

  67. [67]

    Kaichen Zhang, Keming Wu, Zuhao Yang, Bo Li, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, and Lidong Bing. 2025. OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe.arXiv preprint arXiv:2511.16334(2025)

    arXiv 2025

  68. [68]

    Xinkui Zhao, Rongkai Liu, Yifan Zhang, Chen Zhi, Lufei Zhang, Guanjie Cheng, Yueshen Xu, Shuiguang Deng, and Jianwei Yin. 2025. Completion by Comprehension: Guiding Code Generation with Multi-Granularity Understanding. arXiv preprint arXiv:2512.04538(2025)

    arXiv 2025

  69. [69]

    Chen Zhi, Liye Cheng, Meilin Liu, Xinkui Zhao, Yueshen Xu, and Shuiguang Deng. 2024. LLM-powered Zero-shot Online Log Parsing. In2024 IEEE International Conference on Web Services (ICWS). IEEE, 877–887

    2024

  70. [70]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation.arXiv preprint arXiv:2303.12570(2023)

    Pith/arXiv arXiv 2023

  71. [74]

    Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. InProceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. 21–29. , Vol. 1, No. 1, Article . Publication date: February 2026

    2022

  72. [2023]

    InInternational Conference on Machine Learning

    Large language models can be easily distracted by irrelevant context. InInternational Conference on Machine Learning. PMLR, 31210–31227

  73. [2024]

    Repofuse: Repository-level code completion with fused dual context.arXiv preprint arXiv:2402.14323(2024)

    Pith/arXiv arXiv 2024

  74. [2025]

    Authorea Preprints(2025)

    Qwen 2.5: A comprehensive review of the leading resource-efficient llm with potentioal to surpass all competitors. Authorea Preprints(2025)

    2025