{"id":"5ce6f177-4f51-457d-9e1f-5ec7281594b6","arxiv_id":"2412.17754","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ADC fine-tunes Llama-3.1-8B with code execution traces and adversarial function-call data, improving BFCL v2 overall from 63.19 to 79.01, but with no released artifacts or error bars.","lead":"This paper fine-tunes an 8-billion-parameter language model on code traces and adversarial examples to improve function calling in AI agents, and reports results on a public benchmark. The gains look real, but the paper's own tables undercut its strongest claims, and no training code or data are released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section III-C's claim that ADC 'sets a new benchmark in the Execution metric' is contradicted by Table I: GPT-4-0125 (89.25), Meta-Llama-3-70B (88.04), and GPT-4o-mini (87.95) all exceed ADC's 87.50.","rationale":"The paper's contribution is an empirical claim that a training recipe leads to state-of-the-art function calling. The most load-bearing premise is therefore that the reported numbers actually beat existing models. That premise fails internally: Table I shows ADC's Execution score of 87.50 is below GPT-4-0125 (89.25), Meta-Llama-3-70B (88.04), GPT-4o-mini (87.95), and Claude-3-Opus (87.39), and the Overall score is also below several open and commercial models in the same table. Because the abstract and Section III-C present the Execution result as \"a new benchmark\" and \"a new standard\", the primary evidence contradicts the central claim regardless of how one views the training-data attribution question. The reader's weakest_assumption focused on the lack of a controlled baseline; the reader did mention the internal contradiction in the rationale, but did not make it the weakest assumption, so I mark partial agreement. If the authors intended \"best among 8B open models\", that must be stated explicitly and defended; as written, the claim cannot survive. This reinforces the reader's REJECT verdict, so no verdict adjustment is needed beyond UNCHANGED.","tokens_in":7645,"tokens_out":3908,"duration_ms":37496,"concrete_test":"Compute the maximum of the Execution column in Table I over all listed models and compare it with ADC's 87.50. If any listed model exceeds 87.50, then the Section III-C sentence is false for the table's own universe. To test the possible narrower interpretation, subset the table to models of comparable size (around 8B parameters) and check whether ADC is the highest; if so, the claim would need explicit requalification before it can be evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in the abstract and Section III-C is that ADC \"sets a new benchmark in the Execution metric with an outstanding score of 87.50.\" Table I, the paper's own only comparison table, lists four models with higher Execution scores: GPT-4-0125-Preview (89.25), Meta-Llama-3-70B-Instruct (88.04), GPT-4o-mini-2024-07-18 (87.95), and Claude-3-Opus-20240229 (87.39). ADC's 87.50 is therefore not a new benchmark, even within the models the paper itself compares against. The Overall score of 79.01 also trails GPT-4-0125 (85.79), GPT-4o-mini (83.35), Meta-Llama-3-70B (81.59), Functionary-Small-v3.1 (80.21), Gorilla-OpenFunctions-v2 (79.10), and xLAM-7b-fc-r (79.41). The claim could be salvaged only by reinterpreting \"benchmark\" as \"best among open 8B-class models\", but the text does not make that qualification and the table does not support the unqualified statement. This is an internal inconsistency in the primary evidence, not a disagreement with external consensus. The paper also releases no checkpoints, datasets, or evaluation code, so the reported numbers cannot be independently reproduced. As stated, the load-bearing element of the central claim — that ADC outperforms existing models on execution — is false.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADC, a training recipe that fine-tunes Llama3.1-8B-Instruct for function calling. It first fine-tunes on code snippets from CodeNet and POJ104 augmented with line-level variable-execution feedback, and then fine-tunes on function-calling data produced by an adversarial generator/discriminator process seeded from xlam-function-calling and ToolBench. The authors evaluate the resulting model on BFCL v2 and report an Overall score of 79.01, an Execution score of 87.50, and ablation results showing drops when either training dataset or the embedding strategy is changed. The paper claims that ADC 'sets a new benchmark in the Execution metric' and 'setting a new standard' for function calling.","tokens_in":8012,"tokens_out":8410,"duration_ms":82198,"significance":"The idea of using line-level execution feedback as process supervision and augmenting function-calling data with an adversarial generator/discriminator is interesting, and the evaluation is on an external benchmark (BFCL v2), so there is no direct circularity of fitting parameters to the test metric. The ablation in Table II is a reasonable first step toward understanding the contributions. However, the paper's central empirical claim is directly contradicted by its own Table I, the component contributions are not isolated by controlled fine-tuning baselines, and no checkpoints, datasets, evaluation code, training hyperparameters, or variance information are provided. On the evidence in the manuscript, the significance is far below what the abstract claims.","major_comments":[{"comment":"The sentence 'ADC sets a new benchmark in the Execution metric with an outstanding score of 87.50' is contradicted by Table I: GPT-4-0125-Preview (89.25), Meta-Llama-3-70B-Instruct (88.04), GPT-4o-mini-2024-07-18 (87.95), and Claude-3-Opus-20240229 (87.39) all report higher Execution scores than ADC's 87.50. The Overall score of 79.01 also trails several models in the same table, including GPT-4-0125-Preview (85.79), GPT-4o-mini (83.35), Meta-Llama-3-70B (81.59), Functionary-Small-v3.1 (80.21), Gorilla-OpenFunctions-v2 (79.10), and xLAM-7b-fc-r (79.41). Because the abstract and Section I make the same unqualified 'new standard/benchmark' claim, this is an internal inconsistency in the primary evidence, not a disagreement with external consensus. A qualified claim such as 'best Execution among the open 8B-class models in Table I' would need to be stated and defended as such.","section":"Section III-C, Table I"},{"comment":"No controlled fine-tuning baseline isolates the proposed components. The reported improvement over Meta-Llama3.1-8B-Instruct (63.19 to 79.01 Overall) compares an instruct-tuned checkpoint with a version fine-tuned on large code and function-calling corpora; it does not show that line-level execution feedback and adversarial generation, rather than any large fine-tuning corpus, drive the gain. Table II's ablations remove entire datasets ('w/o code dataset' and 'w/o function calling dataset'), not the feedback annotations or the adversarial generation step specifically, so the effect of each proposed mechanism is not identified. I would require at least: (i) SFT on the same code snippets without line-level feedback, (ii) SFT on the same function-calling seed data without adversarial refinement, and (iii) a matched data-volume standard instruction-tuning baseline.","section":"Section III-C, Table I, Table II"},{"comment":"The evaluation is not reproducible from the paper. Section II-A does not report how many code snippets survived redundancy reduction, length filtering, invalid-code filtering, and non-informative filtering; the thresholds of 10 steps and 2048 characters are stated, but final dataset sizes are not. Section II-B does not report the generator/discriminator models, prompts, or number of iterations, and Section III-A gives only hardware and the base model, with no learning rate, batch size, epochs, or inference settings. No checkpoints, datasets, or evaluation code are released, and the BFCL v2 results appear to come from a single run with no error bars or seed information. As a result the reported numbers cannot be independently verified or compared reliably.","section":"Section II and III-A"}],"minor_comments":[{"comment":"The Figure 1 example is internally inconsistent: the comment says 'Calculate the sum of odd numbers', but the loop sums even numbers, and the variable trace (e.g., 'total: 0 => 1' and 'total: 1 => 4') does not match the shown code. Please make the example self-consistent.","section":"Section II-A4, Figure 1"},{"comment":"Several references are malformed: [17] and [18] contain stray brackets and duplicated 'https:' text, and [19] has broken nested Markdown. The bibliography should be cleaned before resubmission.","section":"References [17]-[19]"},{"comment":"The paper should state explicitly that the adversarial seed dataset xlam-function-calling comes from the same Salesforce/APIgen ecosystem as the compared xLAM-7b-fc-r model; this does not affect the external BFCL v2 evaluation, but it is relevant context for the comparison in Table I.","section":"Section II-B1"},{"comment":"Please specify which snapshot/version of BFCL v2 was used and how the Overall Accuracy weights are computed, since leaderboard results can change over time and the metric definitions matter for reproducing the numbers.","section":"Section III-B"}],"recommendation":"reject","confidential_remarks":"The rejection is based on a factual contradiction in the primary evidence: the paper's headline 'new benchmark' Execution claim is false according to its own Table I. The missing controlled baselines and lack of artifacts compound this. If the authors re-scope the central claim and provide a reproducible evaluation with matched baselines, a future submission would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline number doesn't survive contact with the paper's own table. ADC's 87.50 Execution score is bested by GPT-4-0125 (89.25), Meta-Llama-3-70B (88.04), GPT-4o-mini (87.95), and Claude-3-Opus (87.39) in Table I, so Section III-C's 'sets a new benchmark in the Execution metric' is simply false as written. That is the load-bearing claim of the abstract and conclusion, and it contradicts the evidence.\n\nWhat's actually new is the recipe: line-level execution feedback from CodeNet/POJ104 embedded as comments, an adversarial generator-discriminator loop over seed function-calling data, and staged fine-tuning of Llama-3.1-8B-Instruct. That combination is not in xLAM, Granite-FC, or API生成. The data construction is described concretely enough that someone could rebuild it. The ablation study is a plus: dropping the code dataset or the function-calling dataset costs roughly 11 and 23 points overall, and line-level embedding beats prefix or suffix embedding by a wide margin. Those internal comparisons are the paper's most credible evidence.\n\nThe soft spots are exactly where the reader put them. One run on one benchmark, no error bars, no controlled fine-tuning baseline with matched data volume and compute, no released checkpoints or datasets. The ablation doesn't isolate the line-level feedback from simply training on a large code corpus, so the 15-point gain over the base model could be mostly 'fine-tuning on lots of code' rather than process supervision. The adversarial seed data (xlam-function-calling) comes from the same pipeline as one of the compared models, which is a minor contamination smell, not a fatal one.\n\nThe paper is not a waste of time; someone building a function-calling model would find the line-level feedback idea worth trying, and the ablation suggests the format matters. But as submitted, the central empirical claim is factually wrong in the paper's own evidence, and the result is unreproducible because nothing is released. I would desk-reject this version, with an explicit note that a corrected resubmission with honest claims, error bars, a matched fine-tuning baseline, and released artifacts would get a fair hearing. The idea deserves a rigorous study; this manuscript is not yet one.\n\nRegards.","headline":"The paper's 'new benchmark' Execution claim is contradicted by its own Table I, but the line-level feedback recipe is plausible and deserves a cleaner test.","tokens_in":8551,"tokens_out":2602,"would_cite":false,"duration_ms":24574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On the BFCL v2 function-calling benchmark, a fine-tuned 8B model reaches 87.50 Execution—the top execution score in the paper's comparison—and the paper argues this comes from line-level code execution feedback, adversarial data…","keywords":["function calling","large language models","line-level execution feedback","adversarial data generation","staged training","BFCL v2","parameter matching","process supervision"],"falsifier":"Fine-tune the same base model on the same datasets with the execution feedback removed and the adversarial filtering replaced by random sampling, holding data volume and training steps constant; if overall BFCL v2 accuracy stays near the base model's 63.19, the recipe's components are the cause, whereas a score near 79 would show the gain is mostly from added data.","tokens_in":7464,"feed_emoji":"🤖","tokens_out":5636,"duration_ms":46710,"temperature":0.7,"pith_summary":"The paper claims that function-calling ability in a small open LLM can be improved substantially by training on code annotated with line-level execution feedback, then on adversarially refined function-calling data, in two stages. On the BFCL v2 benchmark this recipe lifts a Llama-3.1-8B-Instruct model to an Execution score of 87.50, the best in the reported comparison, and an Overall score of 79.01. The author's case rests on ablations showing that removing the code dataset drops Overall to 67.73, removing the function-calling data drops it to 55.78, and embedding feedback line-by-line beats prefix or suffix embedding. A sympathetic reader would care because the result suggests fine-grained process supervision and adversarial data can substitute for larger models in tool-use tasks.","feed_headline":"Line-level code feedback lifts function-call execution to 87.5%","feed_subtitle":"A staged recipe of execution traces and adversarial data gives an 8B model the top execution score on BFCL v2.","key_machinery":"Line-level execution feedback: each code line is annotated with before-and-after variable values (formatted as v: x => y at line i) produced by actually running the snippet, and the annotations are embedded as comments directly under the corresponding line. The second mechanism is an adversarial loop in which an LLM generator creates harder function-call scenarios and an LLM discriminator keeps only those that meet diversity, realism, and pass-rate criteria. The third is staged training: first fine-tune on the code-plus-feedback dataset to instill format following, then fine-tune on the refined function-calling dataset for parameter matching. The line-level embedding placement is itself load-bearing: prefix and suffix embeddings drop Overall to 59.89 and 65.13, versus 79.01 for line-level.","core_discovery":"The central claim is that a three-part training recipe—process supervision from line-level execution traces, adversarial generation and filtering of function-call data, and staged fine-tuning—materially improves how well a model follows function-call formats and matches complex parameters. The evidence is the BFCL v2 result: an 8B-parameter model reaches an Execution accuracy of 87.50, surpassing several larger and commercial models on that metric, with an Overall accuracy of 79.01. The paper attributes the execution strength to line-level feedback teaching logical reasoning, the parameter-matching gains to the adversarial dataset, and the format gains to staged training that puts code data before function-call data.","pith_inferences":["A natural next experiment is a matched-data control that strips the feedback and adversarial filtering while holding data volume and training steps fixed, which would isolate the mechanisms; the paper does not include this control.","The adversarial loop could be extended to close the evaluation gap by having the discriminator actually execute generated function calls and check outputs, an idea the paper leaves implicit.","Because the code-feedback data is filtered to correct, terminating programs, the method implicitly prioritizes canonical solutions and may under-represent valid alternative coding styles, a consequence the paper does not discuss.","The same line-level feedback technique could transfer to other execution-rich domains, such as tool-use traces in browser automation, but that is an extrapolation beyond the reported experiments."],"forward_implications":["If the recipe holds, execution traces can provide a scalable form of process supervision that does not require human annotation.","Staged training that builds format following before parameter matching could apply to other structured-output tasks such as SQL generation or API orchestration.","The adversarial generator-discriminator loop is a viable way to expand small seed function-calling datasets without manual curation.","The strong Execution score suggests downstream tool-using agents may benefit more from execution-validated training than from larger model scale alone.","Placement of auxiliary feedback matters; the same data embedded line-by-line outperforms prefix or suffix attachment by a wide margin."],"supporting_citations":[{"why":"Provides the BFCL v2 benchmark and evaluation metrics on which all reported scores are measured.","marker":"[12]"},{"why":"Supplies the Llama-3.1-8B-Instruct base model that ADC starts from and one of the main comparison baselines.","marker":"[13]"},{"why":"CodeNet is the primary source of the code snippets used to build the line-level execution feedback dataset.","marker":"[23]"},{"why":"POJ104 supplements CodeNet to increase diversity of the code used for execution feedback.","marker":"[24]"},{"why":"Provides the 60k xlam-function-calling seed data that bootstraps the adversarial function-calling dataset.","marker":"[27]"},{"why":"ToolBench is the second seed dataset feeding the adversarial generator-discriminator process.","marker":"[28]"},{"why":"xLAM-7b-fc-r is the closest-function-calling competitor whose overall score slightly exceeds ADC and whose data pipeline ADC builds on.","marker":"[9]"}],"fun_headline_variants":["Adversarial datasets and line-level feedback lift function-call execution to 87.5%","8B model tops BFCL execution via adversarial and line-level training","Staged recipe of code and adversarial data yields 87.5% execution on BFCL","Line-level feedback and adversarial data help 8B model beat larger ones on function calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 15.82-point overall gain over the base model comes from the training recipe itself, not from the large volume of code and function-call data used, since no control with matched data, order, and compute is reported.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial datasets and line-level feedback lift function-call execution to 87.5%","8B model tops BFCL execution via adversarial and line-level training","Staged recipe of code and adversarial data yields 87.5% execution on BFCL","Line-level feedback and adversarial data help 8B model beat larger ones on function calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001113,"raw_usage":{"total_tokens":4578,"prompt_tokens":830,"completion_tokens":3748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":3658}},"tokens_in":446,"tokens_out":3748,"duration_ms":25513,"temperature":1.0,"reasoning_tokens":3658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:12:01.375733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same base model on the same datasets with the execution feedback removed and the adversarial filtering replaced by random sampling, holding data volume and training steps constant; if overall BFCL v2 accuracy stays near the base model's 63.19, the recipe's components are the cause, whereas a score near 79 would show the gain is mostly from added data.","supporting_citations":[{"cited_title":"Berkeley function calling leaderboard,","cited_arxiv_id":null,"evidence_quote":"Provides the BFCL v2 benchmark and evaluation metrics on which all reported scores are measured."},{"cited_title":"Meta llama 3,","cited_arxiv_id":null,"evidence_quote":"Supplies the Llama-3.1-8B-Instruct base model that ADC starts from and one of the main comparison baselines."},{"cited_title":"Convolutional neural networks over tree structures for programming language processing,","cited_arxiv_id":null,"evidence_quote":"POJ104 supplements CodeNet to increase diversity of the code used for execution feedback."},{"cited_title":"Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models,","cited_arxiv_id":null,"evidence_quote":"ToolBench is the second seed dataset feeding the adversarial generator-discriminator process."}],"review_version":1}