{"id":"1ba3e01b-6d26-4b67-8835-cd1b8f47545a","arxiv_id":"2505.05057","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MARIN reduces API hallucinations in LLM-generated code by supplying project dependency context and constraining decoding to the project's valid APIs, as shown on a new 416-sample Java benchmark and Huawei internal projects.","lead":"This paper introduces MARIN, a system that stops AI code assistants from inventing fake functions by teaching them the project's real structure and limiting their choices to functions that actually exist. The authors report that this reduces API mistakes by roughly two-thirds compared to a retrieval-based approach.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reference API list in the prompt and in the decoding constraint set contains the ground-truth API, so the gains over RAG may come from answer leakage rather than from MARIN's proposed framework.","rationale":"The reader's rationale mentions a potential information leak because the reference API list always contains the ground truth, but the reader's stated weakest assumption is about completeness and semantic preservation of the skeleton, not about this answer-exposure confound. The concern is more severe than the reader's CONDITIONAL verdict suggests: the prompt itself names the correct API, and Eq. (1) restricts decoding to that same named set, so the comparison to RAG is not a fair test of MARIN. The paper's ablation studies (Figures 5 and 6) separate hierarchical dependencies from constrained decoding, but they never remove the Reference API list from the prompt while keeping decoding unconstrained, so they cannot reveal that the prompt already gives away the answer. The proposed Reference-Only and Constrained-Only baselines would settle this directly. If the Reference-Only baseline reproduces most of MARIN's gain, the central empirical claim is unsupported as written and the paper should be rejected or substantially revised; if the gain persists after removing the leak, the original verdict could stand. This is not an ad hominem criticism: static dependency mining that produces a candidate API set is a plausible and potentially useful idea, and the efficiency measurements are informative, but the current evaluation design does not distinguish the framework's contribution from answer exposure. The benchmark is released, though the framework code is not, so the test would require reimplementation from the paper's description.","tokens_in":21149,"tokens_out":7057,"duration_ms":79182,"concrete_test":"Run a \"Reference-Only\" baseline on APIHulBench and the Huawei benchmark: use MARIN's exact prompt, including the mined Reference APIs, but disable the trie masking from Eq. (1) so the LLM decodes normally. If this baseline already reaches MARIN's EM and MaHR levels in Table 1 and Table 3, then the reported gains over RAG are attributable to the ground-truth API being listed in the Reference APIs, not to dependency-constrained decoding. To further isolate the decoding contribution, also run a \"Constrained-Only\" variant that applies the trie mask but omits the Reference API list from the prompt; only the residual gain of MARIN over both variants can be claimed as the framework's actual effect.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.2.1 states that local dependency analysis \"identifies valid APIs available at the generation position,\" and these are placed verbatim in the prompt as \"Reference APIs\" (Figure 4). Section 3.3.1 then builds the prefix tree used in the masking scheme of Eq. (1) from the same reference API set. The APIHulBench samples are constructed by taking real project functions and masking an existing project-specific API call (Section 4.2.2). Therefore, for every sample, the receiver type's method set extracted by local dependency analysis necessarily contains the ground-truth method, and the prompt lists it explicitly alongside the other candidates. The constrained decoder then forbids every token sequence not in this list. The reported 67.52% MiHN and 73.56% MaHR reductions relative to RAG may accordingly measure the static analysis's ability to enumerate a small multiple-choice set containing the answer, plus the LLM's ability to copy a listed name, rather than a general mitigation of API hallucination. This confound is load-bearing because all six open-source LLM evaluations and the Huawei industrial evaluation use the same protocol. The paper's own preliminary study (Figure 2c) already shows that an \"API reference\" condition reduces MaHR to 44%, so the effect is not negligible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MARIN, a two-phase framework for reducing API hallucination in LLM-generated code. The first phase, Hierarchical Dependency Mining, uses static analysis to extract local dependencies (called functions, reference APIs) and global dependencies (skeletons of related and current files) and assembles them into a structured prompt. The second phase, Dependency Constrained Decoding, builds a prefix tree from the reference APIs and uses a binary mask (Eq. 1) to restrict the generated token sequence to valid API names and parameter patterns. To evaluate the approach, the authors introduce APIHulBench, a benchmark of 416 Java samples from 98 recent GitHub repositories, and two metrics, Micro Hallucination Number (MiHN) and Macro Hallucination Rate (MaHR). Experiments on six open-source LLMs (CodeLlama and DeepSeekCoder families) and on two Huawei proprietary models (PanguCoder) compare MARIN against Base, RAG, and De-Hallucinator baselines, reporting large reductions in MiHN and MaHR, large gains in EM/ES/IM, and negligible runtime overhead.","tokens_in":21360,"tokens_out":6070,"duration_ms":60425,"significance":"If the reported results are taken at face value, MARIN is a practically attractive method: it avoids maintaining a retrieval corpus, adds only about 0.02-0.03 seconds per generation, and directly prevents non-existent API calls through constrained decoding. The paper also contributes a new benchmark, two new metrics, and a released implementation, which are useful resources for the code-generation community. The central mechanism is sound in the sense that masking logits to a statically mined set of valid APIs deterministically eliminates the 'non-existent API' category of hallucination. However, the evaluation protocol is confounded: the prompt's 'Reference APIs' list and the decoding constraint set are both constructed from the same project in which the ground-truth API was masked, so the correct answer is always present in the candidate list. The significance of the empirical claims therefore depends on whether the authors can disentangle the oracle-list effect from the framework's own contribution; as written, the headline reductions over RAG are not a clean measurement of MARIN's mitigation capability.","major_comments":[{"comment":"The evaluation protocol confounds the effect of MARIN with the effect of an oracle candidate list that always contains the ground truth. APIHulBench samples are created by identifying an existing project-specific API call and masking it (Section 4.2.2), and local dependency analysis then 'identifies valid APIs available at the generation position' (Section 3.2.1), so the ground-truth method is necessarily a member of the Reference APIs block in the prompt (Figure 4). The same reference set is used to build the prefix tree in Section 3.3.1, and Eq. (1) forbids all token sequences outside it. The comparison against RAG and De-Hallucinator therefore measures the combination of (i) having a curated, answer-containing list and (ii) the model's ability to copy a listed name, rather than the isolated contribution of dependency mining and constrained decoding. The paper's own preliminary study, Figure 2(c), shows that adding an 'API reference' condition alone lowers MaHR from roughly 85% to 44%, which is a large share of the reported 73.56% average MaHR reduction over RAG. Please add a control baseline that receives the same Reference APIs block in the prompt without constrained decoding, and report selection accuracy (correctness among the valid APIs) as a separate metric.","section":"Section 3.2.1, Section 3.3.1, Section 4.2.2, Eq. (1), Table 1"},{"comment":"The masking mechanism makes non-existent API outputs impossible by construction whenever the prefix tree is complete, so the reduction in the 'existence' dimension of hallucination is deterministic rather than a learned capability of the LLM. This is a legitimate design choice, but the paper's headline numbers (average 67.52% MiHN decrease and 73.56% MaHR decrease over RAG) conflate the oracle constraint with the model's ability to choose correctly among valid options. MaHR still counts wrong-but-valid APIs as hallucinations, yet the narrative and the MiHN/MaHR definitions in Section 4.5.2 do not separate the 'existence' and 'selection' components. Please decompose the results into (a) the rate of non-existent API names, which should be zero whenever the constraint set is complete, and (b) the rate of wrong choices among the valid candidates, which is the meaningful measure of the model's remaining hallucination under MARIN.","section":"Eq. (1), Section 5.1, Table 1"},{"comment":"The -LG ablation variant is inconsistent with the method description and therefore weakens the attribution of improvements to the two phases. According to Section 5.2, '-LG w/wo CD' removes both local and global dependencies and provides only the incomplete function. Under Section 3.3.1, the prefix tree is built from the Reference APIs, which are part of the local dependency, so with local dependencies removed there should be no API-name constraint to apply in Eq. (1). Yet Figures 5 and 6 report that constrained decoding improves EM and MaHR even in the -LG variant (for example, Figure 5(d) shows a 19.0-point EM gain for DeepSeekCoder-1.3B under -LG with CD). Please clarify which constraints remain active in the -LG-with-CD setting, and if the reference list is in fact still used to build the prefix tree, revise the variant description and the conclusions drawn from the ablation.","section":"Section 5.2, Figures 5 and 6"},{"comment":"The threats-to-validity discussion does not address the completeness of the static analysis, which is load-bearing for the constrained-decoding guarantee. If the dependency miner misses a valid API at the generation position, Eq. (1) will never allow the correct output, and the benchmark's construction from existing project APIs means this failure mode cannot be observed in the current evaluation. Please add a threat discussing the precision and recall of the static analysis (e.g., a measurement on APIHulBench of how often the ground-truth API is absent from the mined reference set) and its consequences for the claimed guarantees.","section":"Section 6.2"}],"minor_comments":[{"comment":"The phrase 'with hierarchical dependency aware' should be 'with hierarchical dependency awareness'; as written, the title and abstract contain an ungrammatical fragment.","section":"Abstract, Section 1"},{"comment":"The table caption reports a 't-test with p-value < 0.001', while the text in Sections 5.1 and 5.4 reports the Wilcoxon signed-rank test; please use one consistent statistical test throughout and in the table footnotes.","section":"Table 1 caption, Section 5.1, Section 5.4"},{"comment":"There is a typo in 'the average length of incompetent function'; it should read 'incomplete function'.","section":"Section 4.6"},{"comment":"'vllms' should be 'vLLM', and the inference engine should be described consistently with the cited reference [18].","section":"Section 4.6"},{"comment":"Please justify running De-Hallucinator for only one iteration; the original method may require multiple grounding iterations, and a single iteration might understate its performance.","section":"Section 4.4"},{"comment":"The paper reports 't-test' and 'Wilcoxon signed-rank test' inconsistently, and it would help to state whether the significance tests are conducted per-metric across samples or across models; this affects how the reader interprets the asterisks in Table 1.","section":"Section 5.1"},{"comment":"Since the preliminary study already identifies an 'API reference' condition that substantially lowers MaHR, Figure 2(c) should be explicitly connected to the main evaluation as a control condition, rather than appearing only as motivation.","section":"Figure 2(c)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a practical, well-engineered system and useful benchmark resources, but the evaluation's main comparison is confounded: the reference API list in the prompt and the decoding constraint set are both derived from the same project whose ground-truth API was masked, so the correct answer is always in the candidate set. The authors' own Figure 2(c) shows that the 'API reference' condition alone accounts for a large portion of the reported improvement. I would ask for a re-analysis with an API-reference control baseline, a decomposition of existence versus selection accuracy, and a clarification of the -LG ablation before reconsidering publication. If the authors can provide those, the paper could become a solid contribution; without them, the headline numbers overstate the framework's independent effect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: MARIN is a plausible, well-engineered framework, but the reported 67–74% reductions in API hallucination over RAG are not a fair measure of the framework. The prompt and the decoding constraint both use a reference API list that, by construction, always contains the ground-truth API. So the model is effectively doing multiple-choice from a set that includes the answer. The stress-test note holds up on reading. Section 3.2.1 says the static analysis \"identifies valid APIs available at the generation position\" and those go verbatim into the prompt; Section 3.3.1 builds the prefix tree from the same set. Since the benchmark is made by masking an existing project API call, the answer is always in the list.\n\nThat said, this is not a fatal flaw, and the paper has real value. The combination of hierarchical dependency mining and constrained decoding with a prefix tree is new relative to RAG, De-Hallucinator, and DAG. The ablation shows both components help independently: providing dependencies alone cuts hallucination, and constrained decoding adds a further reduction. In fact, the paper's own preliminary study shows an API reference list alone gets MaHR down to ~44%; MARIN with constrained decoding gets it to ~12–28%. So the method does contribute beyond the list, but the headline comparison to RAG conflates two effects. A fair evaluation would include a RAG-plus-reference-list baseline, or report gains relative to the reference-list-only condition.\n\nThe efficiency results are credible and practically interesting: static analysis plus masking adds only ~0.02s overhead versus RAG's ~0.5s. The Huawei evaluation is a nice real-world check, though it shares the same protocol. The benchmark is auto-generated, and the authors acknowledge quality risks; they release the benchmark but not the MARIN implementation, which limits reproducibility.\n\nThis paper is worth a serious referee. The central mechanism is sound, and the framework is a reasonable design for a real pain point. But the authors should be pushed to release the code and re-analyze the results to separate the oracle-list effect from the constrained-decoding effect. Without that, the headline numbers overstate what is learned vs. what is guaranteed by construction.","headline":"The engineering is solid and the method is sensible, but the headline gains over RAG are inflated by an evaluation design that puts the ground-truth API in the candidate list.","tokens_in":21902,"tokens_out":3392,"would_cite":true,"duration_ms":36716,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that grounding LLM code completion in the project's actual dependency structure, then constraining token generation to valid APIs, reduces API hallucination by roughly three-quarters compared with retrieval-augmented…","keywords":["API hallucination","code generation","large language models","hierarchical dependency mining","constrained decoding","API prefix tree","retrieval-augmented generation","hallucination metrics"],"falsifier":"Take a project where the correct API is reachable only through a mechanism the static analysis does not model, such as reflection, dynamic registration, a subclass in a file the import analysis excludes, or a generated source file, and measure how often MARIN's mask makes the ground-truth API impossible to generate. If that rate is high, dependency completeness, not decoding, is the bottleneck; if it is near zero on reflection-free projects, then MARIN is doing exactly what the paper claims.","tokens_in":20950,"feed_emoji":"🤖","tokens_out":8876,"duration_ms":76553,"temperature":0.7,"pith_summary":"The paper tries to establish that most API hallucination in LLM-generated code is avoidable if the generator is told what the project actually contains and is then mechanically prevented from naming anything else. Its two-phase recipe is to mine local and global dependencies of the incomplete function by static analysis, and to decode under a mask that admits only the valid APIs reachable from that position. The paper argues this beats retrieval-augmented generation, which supplies isolated snippets and never verifies that a generated API exists. To support the claim it contributes a 416-sample benchmark of recent Java projects and two metrics, MiHN and MaHR, which count hallucinated elements and hallucinated APIs respectively. Across six open LLMs the framework reports average reductions of 67.52% in MiHN and 73.56% in MaHR relative to RAG, and similar reductions on internal industrial projects with proprietary models.","feed_headline":"Cut API hallucination in LLM code by 73 percent","feed_subtitle":"A two-stage framework feeds LLMs project dependencies, then masks out any API token the project doesn't contain.","key_machinery":"The load-bearing mechanism is an API-name prefix tree, a trie over the token sequences of every valid API reachable at the generation point, built from the LLM's own tokenizer so that context-dependent tokenization is handled. During decoding a binary mask derived from the trie forces the next token to be a child of the current node whenever an API name is being emitted; a separate parameter indicator then admits only no-parameter closing tokens like `())` or parameter-opening tokens like `([` depending on whether the selected API takes arguments. The trie's contents come from the hierarchical dependency mining phase, which distinguishes local dependencies, namely called functions and reference APIs in the same function, from global dependencies, namely skeletonized class, field, and method signatures of the current and imported files. This design converts 'whatever the model might say next' into 'only what the project permits'.","core_discovery":"The central claim, stated in the paper's own terms, is that project-specific APIs, not third-party library APIs, are the dominant failure point, and that neither larger model size nor retrieval of similar code fixes it. What works is coupling enriched context with hard output constraints: the prompt carries a skeletonized view of related files, the current file, called functions, and reference APIs, while decoding is confined to an API-name prefix tree and a parameter-pattern classifier built from those dependencies. On the paper's new benchmark this design raises exact match by an average of 107.3% over RAG and lowers hallucination rates by over two-thirds across all six studied models; the same pattern holds when the framework is moved to internal industrial code with two proprietary models. The authors interpret this as evidence that API hallucination is primarily a problem of missing project context plus unconstrained decoding, not a fundamental capability limit of the base LLM.","pith_inferences":["Editorial inference: the same pipeline should transfer to any language with static import or include analysis; a cheap falsifiable test is Python or TypeScript, where the trie construction is unchanged and only the parser differs.","Editorial inference: the parameter constraint is binary, taking arguments or not, so a natural extension is a type-directed mask that also checks argument count and argument types, which would push MiHN lower than the paper's reported numbers.","Editorial inference: if the dominant cause is unconstrained decoding rather than context, a similar prefix-tree mask could be applied at other completion points, such as variables, fields, or module names, turning this into a general project-schema-constrained decoding layer rather than an API-specific fix."],"forward_implications":["If the paper is right, retrieval corpora and their ongoing maintenance are unnecessary for API-hallucination mitigation; dependency mining on the fly is enough, which removes a scalability obstacle for large codebases.","Developers get help at both the start of a function and after most of the function is written: the benchmark's early-position and late-position splits both show large exact-match gains and hallucination drops.","The overhead is small enough for interactive use, about 0.02 seconds per completion on the open models and about 0.03 seconds on the proprietary industrial models, so the constraint mechanism is practical in real editing loops.","Because the gains appear across all six evaluated model sizes and carry over to proprietary industrial models, the framework's effectiveness does not depend on a particular base model or training recipe."],"supporting_citations":[{"why":"Defines API hallucination in practical code generation and provides the retrieval-augmented generation baseline and input-construction approach that MARIN is compared against.","marker":"[45]"},{"why":"Provides the iterative grounding baseline, generate then retrieve related APIs and refine, that MARIN must outperform without iterative retrieval.","marker":"[6]"},{"why":"Supplies the API-documentation-dependent augmentation approach that the paper excludes because many projects lack detailed docs, motivating a documentation-free dependency solution.","marker":"[14]"},{"why":"One of the two open code-model families used in the evaluation; the 7B, 13B, and 34B variants provide the size sweep across which MARIN's gains are measured.","marker":"[30]"},{"why":"The other open code-model family in the evaluation; the 1.3B, 6.7B, and 33B variants cover small to large models.","marker":"[11]"},{"why":"Source of the proprietary industrial models, 11B and 34B variants, used in the internal-project evaluation.","marker":"[31]"},{"why":"Its filtering and cross-file completion methodology informs how APIHulBench's Java projects and samples are constructed.","marker":"[5]"},{"why":"Referenced for benchmark construction practices, including duplicate removal and task filtering, when building APIHulBench.","marker":"[7]"},{"why":"The parser used to locate functions and API-call positions in Java files for both benchmark generation and dependency extraction.","marker":"[34]"}],"fun_headline_variants":["LLM API hallucination slashed 74% via dependency-aware decoding","MARIN framework cuts API hallucination by 74% on average","Dependency-aware decoding reduces LLM API hallucination 73%","New method slashes 74% API hallucination in LLM code","Cut API hallucination 74% with hierarchical dependency-aware decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that static analysis can enumerate every valid API reachable at the generation position without missing anything and without stripping away the semantic detail that tells similar methods apart; if a valid API is not in the prefix tree the decoder can never produce it, and if the skeletons blur distinctions the model will still land on a wrong but syntactically allowed API.","fun_headline_variants_meta":{"raw":{"variants":["LLM API hallucination slashed 74% via dependency-aware decoding","MARIN framework cuts API hallucination by 74% on average","Dependency-aware decoding reduces LLM API hallucination 73%","New method slashes 74% API hallucination in LLM code","Cut API hallucination 74% with hierarchical dependency-aware decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000734,"raw_usage":{"total_tokens":3321,"prompt_tokens":1024,"completion_tokens":2297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":2205}},"tokens_in":640,"tokens_out":2297,"duration_ms":16795,"temperature":1.0,"reasoning_tokens":2205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:14:01.608610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a project where the correct API is reachable only through a mechanism the static analysis does not model, such as reflection, dynamic registration, a subclass in a file the import analysis excludes, or a generated source file, and measure how often MARIN's mask makes the ground-truth API impossible to generate. If that rate is high, dependency completeness, not decoding, is the bottleneck; if it is near zero on reflection-free projects, then MARIN is doing exactly what the paper claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the two open code-model families used in the evaluation; the 7B, 13B, and 34B variants provide the size sweep across which MARIN's gains are measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Its filtering and cross-file completion methodology informs how APIHulBench's Java projects and samples are constructed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Referenced for benchmark construction practices, including duplicate removal and task filtering, when building APIHulBench."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The parser used to locate functions and API-call positions in Java files for both benchmark generation and dependency extraction."}],"review_version":1}