Pith. sign in

REVIEW 4 cited by

On Mitigating Code LLM Hallucinations with API Documentation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.09726 v1 pith:SKHPRKYD submitted 2024-07-13 cs.CL cs.AIcs.LG

Nihal Jain , Robert Kwiatkowski , Baishakhi Ray , Murali Krishna Ramanathan , Varun Kumar This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords frequencyapiscloudapibenchcodehallucinationsabsolutedemonstratedocumentation
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

In this study, we address the issue of API hallucinations in various software engineering contexts. We introduce CloudAPIBench, a new benchmark designed to measure API hallucination occurrences. CloudAPIBench also provides annotations for frequencies of API occurrences in the public domain, allowing us to study API hallucinations at various frequency levels. Our findings reveal that Code LLMs struggle with low frequency APIs: for e.g., GPT-4o achieves only 38.58% valid low frequency API invocations. We demonstrate that Documentation Augmented Generation (DAG) significantly improves performance for low frequency APIs (increase to 47.94% with DAG) but negatively impacts high frequency APIs when using sub-optimal retrievers (a 39.02% absolute drop). To mitigate this, we propose to intelligently trigger DAG where we check against an API index or leverage Code LLMs' confidence scores to retrieve only when needed. We demonstrate that our proposed methods enhance the balance between low and high frequency API performance, resulting in more reliable API invocations (8.20% absolute improvement on CloudAPIBench for GPT-4o).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware

    cs.SE 2025-05 conditional novelty 7.0 of 10

    MARIN reduces API hallucinations in LLM-generated code by supplying project dependency context and constraining decoding to the project's valid APIs, as shown on a new 416-sample Java benchmark and Huawei internal projects.

  2. Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data

    cs.HC 2025-08 conditional novelty 6.0 of 10

    Non-programmer business users often fail to spot critical mistakes in AI-generated data analyses, even when explicitly warned and incentivized.

  3. Reducing Tool Hallucination via Reliability Alignment

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A reliability alignment framework, Relign, that expands the LLM tool-use action space with indecisive actions reduces tool hallucination rates and improves task success on the new RelyToolBench benchmark.

  4. Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges

    cs.SE 2025-04 conditional novelty 3.0 of 10

    This survey organizes code-LLM hallucinations into four types, reviews 52 related papers, and identifies open challenges in detection and mitigation.

Pith tools