Pith. sign in

REVIEW 4 major objections 7 minor 51 references

Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that grounding LLM code completion in the project's actual dependency structure, then constraining token generation to valid APIs, reduces API hallucination by roughly three-quarters compared with retrieval-augmented…

desk verdict The engineering is solid and the method is sensible, but the headline gains over RAG are inflated by an evaluation design that puts the ground-truth API in the candidate list. read the letter →

arxiv 2505.05057 v2 pith:XIKT7OGG submitted 2025-05-08 cs.SE

Yujia Chen , Mingyu Chen , Cuiyun Gao , Zhihan Jiang , Zhongqi Li , Yuchi Ma This is my paper · ORCID
classification cs.SE
keywords APIhallucinationcodegenerationlargelanguagemodelshierarchicaldependencyminingconstraineddecodingprefixtreeretrieval-augmentedmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that most API hallucination in LLM-generated code is avoidable if the generator is told what the project actually contains and is then mechanically prevented from naming anything else. Its two-phase recipe is to mine local and global dependencies of the incomplete function by static analysis, and to decode under a mask that admits only the valid APIs reachable from that position. The paper argues this beats retrieval-augmented generation, which supplies isolated snippets and never verifies that a generated API exists. To support the claim it contributes a 416-sample benchmark of recent Java projects and two metrics, MiHN and MaHR, which count hallucinated elements and hallucinated APIs respectively. Across six open LLMs the framework reports average reductions of 67.52% in MiHN and 73.56% in MaHR relative to RAG, and similar reductions on internal industrial projects with proprietary models.

What carries the argument

The load-bearing mechanism is an API-name prefix tree, a trie over the token sequences of every valid API reachable at the generation point, built from the LLM's own tokenizer so that context-dependent tokenization is handled. During decoding a binary mask derived from the trie forces the next token to be a child of the current node whenever an API name is being emitted; a separate parameter indicator then admits only no-parameter closing tokens like `())` or parameter-opening tokens like `([` depending on whether the selected API takes arguments. The trie's contents come from the hierarchical dependency mining phase, which distinguishes local dependencies, namely called functions and reference APIs in the same function, from global dependencies, namely skeletonized class, field, and method signatures of the current and imported files. This design converts 'whatever the model might say next' into 'only what the project permits'.

What would settle it

Take a project where the correct API is reachable only through a mechanism the static analysis does not model, such as reflection, dynamic registration, a subclass in a file the import analysis excludes, or a generated source file, and measure how often MARIN's mask makes the ground-truth API impossible to generate. If that rate is high, dependency completeness, not decoding, is the bottleneck; if it is near zero on reflection-free projects, then MARIN is doing exactly what the paper claims.

Watch

Extended reading notes

Core claim

The central claim, stated in the paper's own terms, is that project-specific APIs, not third-party library APIs, are the dominant failure point, and that neither larger model size nor retrieval of similar code fixes it. What works is coupling enriched context with hard output constraints: the prompt carries a skeletonized view of related files, the current file, called functions, and reference APIs, while decoding is confined to an API-name prefix tree and a parameter-pattern classifier built from those dependencies. On the paper's new benchmark this design raises exact match by an average of 107.3% over RAG and lowers hallucination rates by over two-thirds across all six studied models; the same pattern holds when the framework is moved to internal industrial code with two proprietary models. The authors interpret this as evidence that API hallucination is primarily a problem of missing project context plus unconstrained decoding, not a fundamental capability limit of the base LLM.

Load-bearing premise

The load-bearing assumption is that static analysis can enumerate every valid API reachable at the generation position without missing anything and without stripping away the semantic detail that tells similar methods apart; if a valid API is not in the prefix tree the decoder can never produce it, and if the skeletons blur distinctions the model will still land on a wrong but syntactically allowed API.

Editorial extensions

If this is right

  • If the paper is right, retrieval corpora and their ongoing maintenance are unnecessary for API-hallucination mitigation; dependency mining on the fly is enough, which removes a scalability obstacle for large codebases.
  • Developers get help at both the start of a function and after most of the function is written: the benchmark's early-position and late-position splits both show large exact-match gains and hallucination drops.
  • The overhead is small enough for interactive use, about 0.02 seconds per completion on the open models and about 0.03 seconds on the proprietary industrial models, so the constraint mechanism is practical in real editing loops.
  • Because the gains appear across all six evaluated model sizes and carry over to proprietary industrial models, the framework's effectiveness does not depend on a particular base model or training recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same pipeline should transfer to any language with static import or include analysis; a cheap falsifiable test is Python or TypeScript, where the trie construction is unchanged and only the parser differs.
  • Editorial inference: the parameter constraint is binary, taking arguments or not, so a natural extension is a type-directed mask that also checks argument count and argument types, which would push MiHN lower than the paper's reported numbers.
  • Editorial inference: if the dominant cause is unconstrained decoding rather than context, a similar prefix-tree mask could be applied at other completion points, such as variables, fields, or module names, turning this into a general project-schema-constrained decoding layer rather than an API-specific fix.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes MARIN, a two-phase framework for reducing API hallucination in LLM-generated code. The first phase, Hierarchical Dependency Mining, uses static analysis to extract local dependencies (called functions, reference APIs) and global dependencies (skeletons of related and current files) and assembles them into a structured prompt. The second phase, Dependency Constrained Decoding, builds a prefix tree from the reference APIs and uses a binary mask (Eq. 1) to restrict the generated token sequence to valid API names and parameter patterns. To evaluate the approach, the authors introduce APIHulBench, a benchmark of 416 Java samples from 98 recent GitHub repositories, and two metrics, Micro Hallucination Number (MiHN) and Macro Hallucination Rate (MaHR). Experiments on six open-source LLMs (CodeLlama and DeepSeekCoder families) and on two Huawei proprietary models (PanguCoder) compare MARIN against Base, RAG, and De-Hallucinator baselines, reporting large reductions in MiHN and MaHR, large gains in EM/ES/IM, and negligible runtime overhead.

Significance. If the reported results are taken at face value, MARIN is a practically attractive method: it avoids maintaining a retrieval corpus, adds only about 0.02-0.03 seconds per generation, and directly prevents non-existent API calls through constrained decoding. The paper also contributes a new benchmark, two new metrics, and a released implementation, which are useful resources for the code-generation community. The central mechanism is sound in the sense that masking logits to a statically mined set of valid APIs deterministically eliminates the 'non-existent API' category of hallucination. However, the evaluation protocol is confounded: the prompt's 'Reference APIs' list and the decoding constraint set are both constructed from the same project in which the ground-truth API was masked, so the correct answer is always present in the candidate list. The significance of the empirical claims therefore depends on whether the authors can disentangle the oracle-list effect from the framework's own contribution; as written, the headline reductions over RAG are not a clean measurement of MARIN's mitigation capability.

major comments (4)
  1. [Section 3.2.1, Section 3.3.1, Section 4.2.2, Eq. (1), Table 1] The evaluation protocol confounds the effect of MARIN with the effect of an oracle candidate list that always contains the ground truth. APIHulBench samples are created by identifying an existing project-specific API call and masking it (Section 4.2.2), and local dependency analysis then 'identifies valid APIs available at the generation position' (Section 3.2.1), so the ground-truth method is necessarily a member of the Reference APIs block in the prompt (Figure 4). The same reference set is used to build the prefix tree in Section 3.3.1, and Eq. (1) forbids all token sequences outside it. The comparison against RAG and De-Hallucinator therefore measures the combination of (i) having a curated, answer-containing list and (ii) the model's ability to copy a listed name, rather than the isolated contribution of dependency mining and constrained decoding. The paper's own preliminary study, Figure 2(c), shows that adding an 'API reference' condition alone lowers MaHR from roughly 85% to 44%, which is a large share of the reported 73.56% average MaHR reduction over RAG. Please add a control baseline that receives the same Reference APIs block in the prompt without constrained decoding, and report selection accuracy (correctness among the valid APIs) as a separate metric.
  2. [Eq. (1), Section 5.1, Table 1] The masking mechanism makes non-existent API outputs impossible by construction whenever the prefix tree is complete, so the reduction in the 'existence' dimension of hallucination is deterministic rather than a learned capability of the LLM. This is a legitimate design choice, but the paper's headline numbers (average 67.52% MiHN decrease and 73.56% MaHR decrease over RAG) conflate the oracle constraint with the model's ability to choose correctly among valid options. MaHR still counts wrong-but-valid APIs as hallucinations, yet the narrative and the MiHN/MaHR definitions in Section 4.5.2 do not separate the 'existence' and 'selection' components. Please decompose the results into (a) the rate of non-existent API names, which should be zero whenever the constraint set is complete, and (b) the rate of wrong choices among the valid candidates, which is the meaningful measure of the model's remaining hallucination under MARIN.
  3. [Section 5.2, Figures 5 and 6] The -LG ablation variant is inconsistent with the method description and therefore weakens the attribution of improvements to the two phases. According to Section 5.2, '-LG w/wo CD' removes both local and global dependencies and provides only the incomplete function. Under Section 3.3.1, the prefix tree is built from the Reference APIs, which are part of the local dependency, so with local dependencies removed there should be no API-name constraint to apply in Eq. (1). Yet Figures 5 and 6 report that constrained decoding improves EM and MaHR even in the -LG variant (for example, Figure 5(d) shows a 19.0-point EM gain for DeepSeekCoder-1.3B under -LG with CD). Please clarify which constraints remain active in the -LG-with-CD setting, and if the reference list is in fact still used to build the prefix tree, revise the variant description and the conclusions drawn from the ablation.
  4. [Section 6.2] The threats-to-validity discussion does not address the completeness of the static analysis, which is load-bearing for the constrained-decoding guarantee. If the dependency miner misses a valid API at the generation position, Eq. (1) will never allow the correct output, and the benchmark's construction from existing project APIs means this failure mode cannot be observed in the current evaluation. Please add a threat discussing the precision and recall of the static analysis (e.g., a measurement on APIHulBench of how often the ground-truth API is absent from the mined reference set) and its consequences for the claimed guarantees.
minor comments (7)
  1. [Abstract, Section 1] The phrase 'with hierarchical dependency aware' should be 'with hierarchical dependency awareness'; as written, the title and abstract contain an ungrammatical fragment.
  2. [Table 1 caption, Section 5.1, Section 5.4] The table caption reports a 't-test with p-value < 0.001', while the text in Sections 5.1 and 5.4 reports the Wilcoxon signed-rank test; please use one consistent statistical test throughout and in the table footnotes.
  3. [Section 4.6] There is a typo in 'the average length of incompetent function'; it should read 'incomplete function'.
  4. [Section 4.6] 'vllms' should be 'vLLM', and the inference engine should be described consistently with the cited reference [18].
  5. [Section 4.4] Please justify running De-Hallucinator for only one iteration; the original method may require multiple grounding iterations, and a single iteration might understate its performance.
  6. [Section 5.1] The paper reports 't-test' and 'Wilcoxon signed-rank test' inconsistently, and it would help to state whether the significance tests are conducted per-metric across samples or across models; this affects how the reader interprets the asterisks in Table 1.
  7. [Figure 2(c)] Since the preliminary study already identifies an 'API reference' condition that substantially lowers MaHR, Figure 2(c) should be explicitly connected to the main evaluation as a control condition, rather than appearing only as motivation.

Circularity Check

1 steps flagged · score 6.0 of 10

A large share of MARIN's measured hallucination reduction is guaranteed by construction: the decoding mask and the benchmark ground truth are both drawn from the project's valid-API set, so eliminating non-existent API names is structurally forced.

  1. self definitional [Section 3.2.1; Section 3.3.1-3.3.2 (Algorithm 1, Eq. (1)); Section 4.2.2]
    "Through static analysis, we first identify valid APIs available at the generation position ... In this step, dependencies are used to build an API name prefix tree and identify parameter patterns, forming a constrained set of valid options ... We identify project-specific API calls within these functions and split each function into two parts: the prompt (code before the API) and an inference part (containing the ground truth API)."

    APIHulBench is built by masking an existing project-specific API call, so the ground-truth API is a member of the project's valid-API set at that position. MARIN's 'valid APIs available at the generation position' are mined from the same project and inserted as Reference APIs; Algorithm 1 builds the decoding prefix tree from this same set, and Eq. (1) forbids any token outside the set. Thus non-existent-API hallucinations are excluded by construction whenever the static analysis includes the ground-truth method. The reported MiHN/MaHR reductions therefore conflate the static analyzer's enumeration (a multiple-choice set containing the answer) and the decoder's hard mask with the LLM's actual API-selection ability.

full rationale

The central evaluation claim is partly forced by the protocol. Every sample's ground truth is one of the project's APIs, and the constrained decoder only permits the project's valid APIs, so the non-existent-API component of the hallucination metrics is eliminated by definition rather than by model competence. The paper's own Figure 2(c) shows that simply listing API references reduces MaHR to about 44%, and MARIN's hard mask explains most of the remaining gap. However, MARIN does not encode the exact target API: Table 1 shows substantial residual MaHR from choosing wrong valid APIs, and the RQ2 ablation shows that global dependency context improves accuracy independently of the mask. Self-citations (e.g., [3,4,22,31]) appear only as background or baseline support and are not load-bearing for the core derivation. The correct finding is partial circularity, not full tautology.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The method has no fitted parameters or new physical entities. The key assumptions are about the reliability of static analysis and the sufficiency of condensed context, which are domain assumptions rather than arbitrary postulates.

assumptions (3)
  • domain assumption Static analysis correctly resolves all valid API methods accessible at the generation point.
    The dependency constrained decoding relies on this list; if an API is missed, the model cannot generate it (Section 3.3).
  • domain assumption The file skeletons and called-function signatures provide sufficient context to select the correct API among valid ones.
    The paper assumes the compressed prompt retains enough semantics for the model to match intent (Section 3.2).
  • domain assumption Split tokenization with shared prefix removal yields the same API token sequence regardless of surrounding context.
    The paper states tokenization is context-dependent and handles it by tokenizing full calls and prefixes (Section 3.3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware." pith.science (2026). https://pith.science/paper/XIKT7OGG

@misc{pith2026250505057,
  author       = {Pith},
  title        = {Pith review of: Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XIKT7OGG}},
  note         = {Machine review of arXiv:2505.05057}
}
read the original abstract

Application Programming Interfaces (APIs) are crucial in modern software development. Large Language Models (LLMs) assist in automated code generation but often struggle with API hallucination, including invoking non-existent APIs and misusing existing ones in practical development scenarios. Existing studies resort to Retrieval-Augmented Generation (RAG) methods for mitigating the hallucination issue, but tend to fail since they generally ignore the structural dependencies in practical projects and do not indeed validate whether the generated APIs are available or not. To address these limitations, we propose MARIN, a framework for mitigating API hallucination in code generated by LLMs with hierarchical dependency aware. MARIN consists of two phases: Hierarchical Dependency Mining, which analyzes local and global dependencies of the current function, aiming to supplement comprehensive project context in LLMs input, and Dependency Constrained Decoding, which utilizes mined dependencies to adaptively constrain the generation process, aiming to ensure the generated APIs align with the projects specifications. To facilitate the evaluation of the degree of API hallucination, we introduce a new benchmark APIHulBench and two new metrics including Micro Hallucination Number (MiHN) and Macro Hallucination Rate (MaHR). Experiments on six state-of-the-art LLMs demonstrate that MARIN effectively reduces API hallucinations, achieving an average decrease of 67.52% in MiHN and 73.56% in MaHR compared to the RAG approach. Applied to Huaweis internal projects and two proprietary LLMs, MARIN achieves average decreases of 57.33% in MiHN and 59.41% in MaHR.

Figures

Figures reproduced from arXiv: 2505.05057 by the authors.

Figure 1
Figure 1. The motivation example: A Wrong API generated by CodeLLama-7B with RAG. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Analysis of API hallucinations across different dimensions. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overview of MARIN. 3.2 Hierarchical Dependency Mining As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An example illustrating the MARIN’ prompt tem [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Evaluation results of different variants on APIHulBench-F. “-LD” denotes removing local dependency, “-GD” denotes [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Evaluation results of different variants on APIHulBench-M. “-LD” denotes removing local dependency, “-GD” denotes [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Case study on API hallucination mitigation across Base, De-Hallucinator, RAG, and MARIN using CodeLlama-7B. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 27 canonical work pages

  1. [1]

    Amazon. 2024. https://aws.amazon.com/codewhisperer

    work page 2024

  2. [2]

    Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023. Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (2023), 85–111

    2023

  3. [3]

    Yujia Chen, Cuiyun Gao, Xiaoxue Ren, Yun Peng, Xin Xia, and Michael R. Lyu

    work page

  4. [4]

    Yujia Chen, Cuiyun Gao, Muyijie Zhu, Qing Liao, Yong Wang, and Guoai Xu. 2024. APIGen: Generative API Method Recommendation. In IEEE International Confer- ence on Software Analysis, Evolution and Reengineering, SANER 2024, Rovaniemi, Finland, March 12-15, 2024 . IEEE, 171–182

    work page 2024

  5. [5]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informatio...

    work page 2023

  6. [6]

    Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative grounding for llm-based code completion. arXiv preprint arXiv:2401.01701 (2024)

    arXiv 2024

  7. [7]

    Jia Feng, Jiachen Liu, Cuiyun Gao, Chun Yong Chong, Chaozheng Wang, Shan Gao, and Xin Xia. 2024. ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code. In Proceedings of the 39th IEEE/ACM Inter- national Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024 , Vladimir Filkov...

    work page 2024

  8. [8]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

    work page 2023

Show all 51 references
  1. [9]

    Such, and Guillermo Suarez-Tangil

    Vahid Ghafouri, Vibhor Agarwal, Yong Zhang, Nishanth Sastry, Jose M. Such, and Guillermo Suarez-Tangil. 2023. AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topics. In Proceedings of the 32nd ACM International...

    2023

  2. [10]

    Github. 2024. copilot.github.com

    2024

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guant- ing Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang

  4. [12]

    Dong Huang, Qingwen Bu, Jie Zhang, Xiaofei Xie, Junjie Chen, and Heming Cui

  5. [13]

    Huggingface Hub. 2024. https://huggingface.co/

    2024

  6. [14]

    Nihal Jain, Robert Kwiatkowski, Baishakhi Ray, Murali Krishna Ramanathan, and Varun Kumar. 2024. On Mitigating Code LLM Hallucinations with API Documentation. arXiv preprint arXiv:2407.09726 (2024)

    2024 arXiv

  7. [15]

    CoRR abs/2309.14345 (2023)

    Bias Assessment and Mitigation in LLM-based Code Generation. CoRR abs/2309.14345 (2023)

    2023 arXiv

  8. [16]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1–38

    2023

  9. [17]

    Jiasheng Jiang, Jingzheng Wu, Xiang Ling, Tianyue Luo, Sheng Qu, and Yanjun Wu. 2024. APP-Miner: Detecting API Misuses via Automatically Mining API Path Patterns. In IEEE Symposium on Security and Privacy, SP 2024, San Francisco, CA, USA, May 19-23, 2024 . IEEE, 4034–4052

    2024

  10. [18]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12 (2023), 248:1–248:38

    2023

  11. [19]

    Jingxuan Li, Rui Huang, Wei Li, Kai Yao, and Weiguo Tan. 2021. Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2021, Luxembourg, September 27 - October 1, 2021 . IEEE, 195–205

    2021

  12. [20]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023

  13. [21]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  14. [22]

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971 (2024)

    2024

  15. [23]

    McAuley, Han Hu, Torsten Scholak, Sébastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, and et al

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zi- jian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul...

    2024 arXiv

  16. [24]

    Bingchang Liu, Chaoyu Chen, Cong Liao, Zi Gong, Huan Wang, Zhichao Lei, Ming Liang, Dajun Chen, Min Shen, Hailian Zhou, Hang Yu, and Jianguo Li. 2024. MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discove...

    2024

  17. [25]

    Yao Wan nd Zhangqian Bi, Yang He, Jianguo Zhang, Hongyu Zhang, Yulei Sui, Guandong Xu, Hai Jin, and Philip S. Yu. 2024. Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit. ACM Comput. Surv. 56, 12 (2024), 309:1–309:41

    2024

  18. [26]

    Phuong Thanh Nguyen, Juri Di Rocco, Davide Di Ruscio, Lina Ochoa, Thomas Degueule, and Massimiliano Di Penta. 2019. FOCUS: a recommender system for mining API function calls and usage patterns. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2...

    2019

  19. [27]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. In The Twelfth International Conference on Learning Representations

    2023

  20. [28]

    OpenAI. 2022. Codex. https://openai.com/blog/openai-codex/

    2022

  21. [29]

    Yun Peng, Shuqing Li, Wenwei Gu, Yichen Li, Wenxuan Wang, Cuiyun Gao, and Michael R. Lyu. 2023. Revisiting, Benchmarking and Exploring API Recommen- dation: How Far Are We? IEEE Trans. Software Eng. 49, 4 (2023), 1876–1897

    2023

  22. [30]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. InThe Eleventh International Conference on Learning Representations, ICLR 2023, ...

    2023

  23. [31]

    Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang

  24. [32]

    Yuchen Tian, Weixiang Yan, Qian Yang, Qian Chen, Wen Wang, Ziyang Luo, and Lei Ma. 2024. CodeHalu: Code Hallucinations in LLMs Driven by Execution-based Verification. arXiv preprint arXiv:2405.00253 (2024)

    2024 arXiv

  25. [33]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xi- aoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défo...

    2023 arXiv

  26. [34]

    Tree-sitter. 2024. https://github.com/tree-sitter/tree-sitter

    2024

  27. [35]

    CoRR abs/2307.14936 (2023)

    PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback. CoRR abs/2307.14936 (2023)

    2023 arXiv

  28. [36]

    Moshi Wei, Nima Shiri Harzevili, Yuchao Huang, Junjie Wang, and Song Wang

  29. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, and Shruti Bhosale et al . 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint (2023)

    2023

  30. [38]

    Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Break- throughs in statistics: Methodology and distribution . Springer, 196–202

    1992

  31. [39]

    Chaozheng Wang, Junhao Hu, Cuiyun Gao, Yu Jin, Tao Xie, Hailiang Huang, Zhenyu Lei, and Yuetang Deng. 2023. Practitioners’ Expectations on Code Completion. CoRR abs/2301.03846 (2023)

    2023 arXiv

  32. [40]

    Hassan, and Zhenchang Xing

    Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E. Hassan, and Zhenchang Xing. 2017. What do developers search for on the web? Empir. Softw. Eng. 22, 6 (2017), 3149–3185

    2017

  33. [41]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  34. [42]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering Code Generation with OSS-Instruct. In Forty-first Inter- national Conference on Machine Learning

    2024

  35. [43]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. CoRR abs/2...

    2023 arXiv

  36. [44]

    McDaniel, and Chaowei Xiao

    Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick D. McDaniel, and Chaowei Xiao

  37. [45]

    CoRR abs/2402.18649 (2024)

    A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems. CoRR abs/2402.18649 (2024)

    2024 arXiv

  38. [48]

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2024. WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with Refined Data Generation. InProceedings of the 62nd Annual Meeting of the Association for Computatio...

    2024

  39. [50]

    Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023. Unifying the perspectives of nlp and software engineering: A survey on language models for code. arXiv preprint arXiv:2311.07989 (2023)

    2023 arXiv

  40. [51]

    Ziyao Zhang, Yanlin Wang, Chong Wang, Jiachi Chen, and Zibin Zheng. 2024. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation. arXiv preprint arXiv:2409.20550 (2024)

    2024 arXiv

  41. [2022]

    In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022

    CLEAR: Contrastive Learning for API Recommendation. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022 . ACM, 376–387

    2022

  42. [2023]

    IEEE Trans

    API Usage Recommendation Via Multi-View Heterogeneous Graph Repre- sentation Learning. IEEE Trans. Software Eng. 49, 5 (2023), 3289–3304

    2023

  43. [2024]

    CoRR abs/2401.14196 (2024)

    DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence. CoRR abs/2401.14196 (2024)

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.