Pith. sign in

REVIEW 3 major objections 4 minor 29 references

ADC: Enhancing Function Calling Via Adversarial Datasets and Code Line-Level Feedback

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On the BFCL v2 function-calling benchmark, a fine-tuned 8B model reaches 87.50 Execution—the top execution score in the paper's comparison—and the paper argues this comes from line-level code execution feedback, adversarial data…

desk verdict The paper's 'new benchmark' Execution claim is contradicted by its own Table I, but the line-level feedback recipe is plausible and deserves a cleaner test. read the letter →

arxiv 2412.17754 v2 pith:MDOWLMNO submitted 2024-12-23 cs.SE

Wei Zhang , Yi Zhang , Li Zhu , Qianghuai Jia , Feijun Jiang , Hongcheng Guo , Zhoujun Li , Mengping Zhou This is my paper · ORCID
classification cs.SE
keywords functioncallinglargelanguagemodelsline-levelexecutionfeedbackadversarialdatagenerationstagedtrainingBFCLv2parametermatchingprocesssupervision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that function-calling ability in a small open LLM can be improved substantially by training on code annotated with line-level execution feedback, then on adversarially refined function-calling data, in two stages. On the BFCL v2 benchmark this recipe lifts a Llama-3.1-8B-Instruct model to an Execution score of 87.50, the best in the reported comparison, and an Overall score of 79.01. The author's case rests on ablations showing that removing the code dataset drops Overall to 67.73, removing the function-calling data drops it to 55.78, and embedding feedback line-by-line beats prefix or suffix embedding. A sympathetic reader would care because the result suggests fine-grained process supervision and adversarial data can substitute for larger models in tool-use tasks.

What carries the argument

Line-level execution feedback: each code line is annotated with before-and-after variable values (formatted as v: x => y at line i) produced by actually running the snippet, and the annotations are embedded as comments directly under the corresponding line. The second mechanism is an adversarial loop in which an LLM generator creates harder function-call scenarios and an LLM discriminator keeps only those that meet diversity, realism, and pass-rate criteria. The third is staged training: first fine-tune on the code-plus-feedback dataset to instill format following, then fine-tune on the refined function-calling dataset for parameter matching. The line-level embedding placement is itself load-bearing: prefix and suffix embeddings drop Overall to 59.89 and 65.13, versus 79.01 for line-level.

What would settle it

Fine-tune the same base model on the same datasets with the execution feedback removed and the adversarial filtering replaced by random sampling, holding data volume and training steps constant; if overall BFCL v2 accuracy stays near the base model's 63.19, the recipe's components are the cause, whereas a score near 79 would show the gain is mostly from added data.

Watch

Extended reading notes

Core claim

The central claim is that a three-part training recipe—process supervision from line-level execution traces, adversarial generation and filtering of function-call data, and staged fine-tuning—materially improves how well a model follows function-call formats and matches complex parameters. The evidence is the BFCL v2 result: an 8B-parameter model reaches an Execution accuracy of 87.50, surpassing several larger and commercial models on that metric, with an Overall accuracy of 79.01. The paper attributes the execution strength to line-level feedback teaching logical reasoning, the parameter-matching gains to the adversarial dataset, and the format gains to staged training that puts code data before function-call data.

Load-bearing premise

The load-bearing premise is that the 15.82-point overall gain over the base model comes from the training recipe itself, not from the large volume of code and function-call data used, since no control with matched data, order, and compute is reported.

Editorial extensions

If this is right

  • If the recipe holds, execution traces can provide a scalable form of process supervision that does not require human annotation.
  • Staged training that builds format following before parameter matching could apply to other structured-output tasks such as SQL generation or API orchestration.
  • The adversarial generator-discriminator loop is a viable way to expand small seed function-calling datasets without manual curation.
  • The strong Execution score suggests downstream tool-using agents may benefit more from execution-validated training than from larger model scale alone.
  • Placement of auxiliary feedback matters; the same data embedded line-by-line outperforms prefix or suffix attachment by a wide margin.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is a matched-data control that strips the feedback and adversarial filtering while holding data volume and training steps fixed, which would isolate the mechanisms; the paper does not include this control.
  • The adversarial loop could be extended to close the evaluation gap by having the discriminator actually execute generated function calls and check outputs, an idea the paper leaves implicit.
  • Because the code-feedback data is filtered to correct, terminating programs, the method implicitly prioritizes canonical solutions and may under-represent valid alternative coding styles, a consequence the paper does not discuss.
  • The same line-level feedback technique could transfer to other execution-rich domains, such as tool-use traces in browser automation, but that is an extrapolation beyond the reported experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ADC, a training recipe that fine-tunes Llama3.1-8B-Instruct for function calling. It first fine-tunes on code snippets from CodeNet and POJ104 augmented with line-level variable-execution feedback, and then fine-tunes on function-calling data produced by an adversarial generator/discriminator process seeded from xlam-function-calling and ToolBench. The authors evaluate the resulting model on BFCL v2 and report an Overall score of 79.01, an Execution score of 87.50, and ablation results showing drops when either training dataset or the embedding strategy is changed. The paper claims that ADC 'sets a new benchmark in the Execution metric' and 'setting a new standard' for function calling.

Significance. The idea of using line-level execution feedback as process supervision and augmenting function-calling data with an adversarial generator/discriminator is interesting, and the evaluation is on an external benchmark (BFCL v2), so there is no direct circularity of fitting parameters to the test metric. The ablation in Table II is a reasonable first step toward understanding the contributions. However, the paper's central empirical claim is directly contradicted by its own Table I, the component contributions are not isolated by controlled fine-tuning baselines, and no checkpoints, datasets, evaluation code, training hyperparameters, or variance information are provided. On the evidence in the manuscript, the significance is far below what the abstract claims.

major comments (3)
  1. [Section III-C, Table I] The sentence 'ADC sets a new benchmark in the Execution metric with an outstanding score of 87.50' is contradicted by Table I: GPT-4-0125-Preview (89.25), Meta-Llama-3-70B-Instruct (88.04), GPT-4o-mini-2024-07-18 (87.95), and Claude-3-Opus-20240229 (87.39) all report higher Execution scores than ADC's 87.50. The Overall score of 79.01 also trails several models in the same table, including GPT-4-0125-Preview (85.79), GPT-4o-mini (83.35), Meta-Llama-3-70B (81.59), Functionary-Small-v3.1 (80.21), Gorilla-OpenFunctions-v2 (79.10), and xLAM-7b-fc-r (79.41). Because the abstract and Section I make the same unqualified 'new standard/benchmark' claim, this is an internal inconsistency in the primary evidence, not a disagreement with external consensus. A qualified claim such as 'best Execution among the open 8B-class models in Table I' would need to be stated and defended as such.
  2. [Section III-C, Table I, Table II] No controlled fine-tuning baseline isolates the proposed components. The reported improvement over Meta-Llama3.1-8B-Instruct (63.19 to 79.01 Overall) compares an instruct-tuned checkpoint with a version fine-tuned on large code and function-calling corpora; it does not show that line-level execution feedback and adversarial generation, rather than any large fine-tuning corpus, drive the gain. Table II's ablations remove entire datasets ('w/o code dataset' and 'w/o function calling dataset'), not the feedback annotations or the adversarial generation step specifically, so the effect of each proposed mechanism is not identified. I would require at least: (i) SFT on the same code snippets without line-level feedback, (ii) SFT on the same function-calling seed data without adversarial refinement, and (iii) a matched data-volume standard instruction-tuning baseline.
  3. [Section II and III-A] The evaluation is not reproducible from the paper. Section II-A does not report how many code snippets survived redundancy reduction, length filtering, invalid-code filtering, and non-informative filtering; the thresholds of 10 steps and 2048 characters are stated, but final dataset sizes are not. Section II-B does not report the generator/discriminator models, prompts, or number of iterations, and Section III-A gives only hardware and the base model, with no learning rate, batch size, epochs, or inference settings. No checkpoints, datasets, or evaluation code are released, and the BFCL v2 results appear to come from a single run with no error bars or seed information. As a result the reported numbers cannot be independently verified or compared reliably.
minor comments (4)
  1. [Section II-A4, Figure 1] The Figure 1 example is internally inconsistent: the comment says 'Calculate the sum of odd numbers', but the loop sums even numbers, and the variable trace (e.g., 'total: 0 => 1' and 'total: 1 => 4') does not match the shown code. Please make the example self-consistent.
  2. [References [17]-[19]] Several references are malformed: [17] and [18] contain stray brackets and duplicated 'https:' text, and [19] has broken nested Markdown. The bibliography should be cleaned before resubmission.
  3. [Section II-B1] The paper should state explicitly that the adversarial seed dataset xlam-function-calling comes from the same Salesforce/APIgen ecosystem as the compared xLAM-7b-fc-r model; this does not affect the external BFCL v2 evaluation, but it is relevant context for the comparison in Table I.
  4. [Section III-B] Please specify which snapshot/version of BFCL v2 was used and how the Overall Accuracy weights are computed, since leaderboard results can change over time and the metric definitions matter for reproducing the numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the reported BFCL v2 results are external benchmark outcomes, and the training recipe is built from public code corpora and seed function-calling datasets.

full rationale

The paper's derivation chain is a training recipe rather than a formal derivation: line-level execution feedback is generated by executing public CodeNet/POJ104 code and recording variable changes; the adversarial function-calling data is produced from xlam-function-calling and ToolBench seed datasets via an LLM generator/discriminator; and the base model is fine-tuned in two stages. No equation in the paper defines an output in terms of the evaluation metric, and no fitted parameter is renamed as a prediction: the BFCL v2 Overall/Execution/AST/Irrelevance/Relevance scores are computed by an external benchmark from held-out test cases. The self-citations in the reference list ([1], [2], [11]) are contextual examples in the introduction and are not load-bearing for the method. The only notable defect is an accuracy/framing issue, not circularity: Section III-C states that ADC 'sets a new benchmark in the Execution metric with an outstanding score of 87.50,' but Table I lists GPT-4-0125-Preview (89.25), Meta-Llama-3-70B-Instruct (88.04), GPT-4o-mini-2024-07-18 (87.95), and Claude-3-Opus-20240229 (87.39) with higher Execution scores, so the unqualified claim is contradicted by the paper's own comparison table. Also, the paper does not provide a matched-data controlled ablation isolating the code-dataset stage from generic code-instruction fine-tuning, which weakens causal attribution but does not make any claimed result equivalent to an input by construction. The xlam-function-calling seed data originate from the same Salesforce ecosystem as the xLAM baseline, creating a possible data-overlap concern, but since evaluation is on the external BFCL v2 benchmark and no target label from BFCL is used in training, this does not constitute circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ADC method has no fitted parameters on the target benchmark, so circularity load is low. But the empirical claim rests on hand-set data thresholds, an unvalidated assumption about process supervision, and unreported training and adversarial loop settings, making the evidence weak.

free parameters (3)
  • Feedback step cap (per line-variable pair) = 10 steps
    Chosen by hand to bound trace length; reported in Section II-A3a as replacing intermediate steps with ellipses beyond 10 steps.
  • Length filtering threshold = 2048 characters
    Hand-set from a 90% statistical cutoff to exclude long code+feedback pairs; Section II-A3b.
  • Adversarial iteration count and discriminator thresholds
    Not reported in the paper; these choices directly determine the final function-calling dataset composition and are not auditable.
assumptions (4)
  • domain assumption BFCL v2 is an adequate and noise-free measure of function-calling quality.
    The paper uses a single benchmark with single-run scores and no error bars, so all conclusions depend on this assumption.
  • domain assumption Line-level execution feedback provides effective process supervision for function format following.
    The central hypothesis is supported only by an ablation that does not control for data volume, training order, or compute.
  • domain assumption The LLM generator and discriminator produce and select valid, challenging function-call data.
    The paper provides no human audit or automated validation of the generated data beyond the discriminator's own judgments.
  • domain assumption Regex-based modification of C/C++ snippets (adding bits/stdc++.h and using namespace std) preserves program semantics.
    Section II-A2 states this preprocessing is applied to fix common defects; if it changes behavior, the execution feedback is invalid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ADC: Enhancing Function Calling Via Adversarial Datasets and Code Line-Level Feedback." pith.science (2026). https://pith.science/paper/MDOWLMNO

@misc{pith2026241217754,
  author       = {Pith},
  title        = {Pith review of: ADC: Enhancing Function Calling Via Adversarial Datasets and Code Line-Level Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MDOWLMNO}},
  note         = {Machine review of arXiv:2412.17754}
}
read the original abstract

Large Language Models (LLMs) have made significant strides in Natural Language Processing and coding, yet they struggle with robustness and accuracy in complex function calls. To tackle these challenges, this paper introduces ADC, an innovative approach that enhances LLMs' ability to follow function formats and match complex parameters. ADC utilizes a high-quality code fine-tuning dataset with line-level execution feedback, providing granular process supervision that fosters strong logical reasoning and adherence to function formats. It also employs an adversarial dataset generation process to improve parameter matching. The staged training methodology capitalizes on both enriched code datasets and refined adversarial datasets, leading to marked improvements in function calling capabilities on the Berkeley Function-Calling Leaderboard (BFCL) Benchmark. The innovation of ADC lies in its strategic combination of process supervision, adversarial refinement, and incremental learning, setting a new standard for LLM proficiency in complex function calling.

Figures

Figures reproduced from arXiv: 2412.17754 by the authors.

Figure 1
Figure 1. Overview of ADC. We first create a detailed code dataset with line-level execution feedback by executing the code and embedding the feedback into [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 14 canonical work pages

  1. [1]

    Lemur: Log parsing with entropy sampling and chain-of-thought merging,

    W. Zhang, H. Guo, A. Le, J. Yang, J. Liu, Z. Li, T. Zheng, S. Xu, R. Zang, L. Zheng, and B. Zhang, “Lemur: Log parsing with entropy sampling and chain-of-thought merging,” 2024. [Online]. Available: https://arxiv.org/abs/2402.18205

    arXiv 2024

  2. [2]

    Eclipse: Semantic entropy-lcs for cross-lingual industrial log parsing,

    W. Zhang, X. Cheng, Y . Zhang, J. Yang, H. Guo, Z. Li, X. Yin, X. Guan, X. Shi, L. Zheng, and B. Zhang, “Eclipse: Semantic entropy-lcs for cross-lingual industrial log parsing,” 2024. [Online]. Available: https://arxiv.org/abs/2405.13548

    arXiv 2024

  3. [3]

    Code alpaca: An instruction-following llama model for code generation,

    S. Chaudhary, “Code alpaca: An instruction-following llama model for code generation,” https://github.com/sahil280114/codealpaca, 2023

    2023

  4. [4]

    Self-instruct: Aligning language models with self- generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” arXiv preprint arXiv:2212.10560 , 2022

    arXiv 2022

  5. [5]

    Wizardcoder: Empowering code large language models with evol-instruct,

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” arXiv preprint arXiv:2306.08568 , 2023

    arXiv 2023

  6. [6]

    Magicoder: Source code is all you need,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Source code is all you need,” arXiv preprint arXiv:2312.02120 , 2023

    arXiv 2023

  7. [7]

    Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning,

    Z. Yu, X. Zhang, N. Shang, Y . Huang, C. Xu, Y . Zhao, W. Hu, and Q. Yin, “Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2312.14187

    arXiv 2024

  8. [8]

    Efficient and scalable estimation of tool representations in vector space,

    S. Moon, S. Jha, L. E. Erdogan, S. Kim, W. Lim, K. Keutzer, and A. Gholami, “Efficient and scalable estimation of tool representations in vector space,” 2024. [Online]. Available: https://arxiv.org/abs/2409. 02141

    work page 2024

Show all 29 references
  1. [9]

    xlam: A family of large action models to empower ai agent systems,

    J. Zhang, T. Lan, M. Zhu, Z. Liu, T. Hoang, S. Kokane, W. Yao, J. Tan, A. Prabhakar, H. Chen, Z. Liu, Y . Feng, T. Awalgaonkar, R. Murthy, E. Hu, Z. Chen, R. Xu, J. C. Niebles, S. Heinecke, H. Wang, S. Savarese, and C. Xiong, “xlam: A family of large action models to empower a...

    2024 arXiv

  2. [10]

    Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks,

    I. Abdelaziz, K. Basu, M. Agarwal, S. Kumaravel, M. Stallone, R. Panda, Y . Rizk, G. Bhargav, M. Crouse, C. Gunasekara, S. Ikbal, S. Joshi, H. Karanam, V . Kumar, A. Munawar, S. Neelam, D. Raghu, U. Sharma, A. M. Soria, D. Sreedhar, P. Venkateswaran, M. Unuvar, D. Cox, S. Rouk...

    2024 arXiv

  3. [11]

    mabc: multi-agent blockchain-inspired collaboration for root cause analysis in micro-services architecture,

    W. Zhang, H. Guo, J. Yang, Y . Zhang, C. Yan, Z. Tian, H. Ji, Z. Li, T. Li, T. Zheng, C. Chen, Y . Liang, X. Shi, L. Zheng, and B. Zhang, “mabc: multi-agent blockchain-inspired collaboration for root cause analysis in micro-services architecture,” 2024. [Online]. Available: ht...

    2024 arXiv

  4. [12]

    Berkeley function calling leaderboard,

    F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez, “Berkeley function calling leaderboard,” https://gorilla. cs.berkeley.edu, 2024

    2024

  5. [13]

    Meta llama 3,

    M. AI, “Meta llama 3,” https://ai.meta.com/blog/meta-llama-3/, 2024

    2024

  6. [14]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  7. [15]

    Claude.ai onboarding,

    “Claude.ai onboarding,” https://www.anthropic.com/claude

  8. [16]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B....

  9. [17]

    Hermes-2-pro-llama-3-8b

    “Hermes-2-pro-llama-3-8b.” [Online]. Available: [https: //huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B]https: //huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B)

  10. [18]

    Hermes-2-pro-mistral-7b

    “Hermes-2-pro-mistral-7b.” [Online]. Available: [https: //huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B]https: //huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B)

  11. [19]

    Hermes-2-theta-llama-3-8b

    “Hermes-2-theta-llama-3-8b.” [Online]. Available: [https: //huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B] [NousResearch/Hermes-2-Theta-Llama-3-8B](https://huggingface. co/NousResearch/Hermes-2-Pro-Llama-3-8B))

  12. [20]

    Functionary-small-v3.1,

    “Functionary-small-v3.1,” https://huggingface.co/meetkai/ functionary-small-v3.1

  13. [21]

    Functionary-small-v3.2,

    “Functionary-small-v3.2,” https://huggingface.co/meetkai/ functionary-small-v3.2

  14. [22]

    Gorilla openfunctions v2,

    “Gorilla openfunctions v2,” 2024

    2024

  15. [23]

    Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,

    R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker et al., “Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,” arXiv preprint arXiv:2105.12655, 2021

    2021 arXiv

  16. [24]

    Convolutional neural networks over tree structures for programming language processing,

    L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin, “Convolutional neural networks over tree structures for programming language processing,” in Proceedings of the AAAI conference on artificial intelligence , vol. 30, no. 1, 2016

    2016

  17. [25]

    Aizu online judge,

    Y . Watanobe, “Aizu online judge,” https://onlinejudge.u-aizu.ac.jp

  18. [26]

    Atcoder,

    “Atcoder,” https://atcoder.jp/

  19. [27]

    Apigen: Automated pipeline for gen- erating verifiable and diverse function-calling datasets,

    Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, S. Kokane, J. Tan, W. Yao, Z. Liu, Y . Feng et al. , “Apigen: Automated pipeline for gen- erating verifiable and diverse function-calling datasets,” arXiv preprint arXiv:2406.18518, 2024

    2024 arXiv

  20. [28]

    Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models,

    Z. Guo, S. Cheng, H. Wang, S. Liang, Y . Qin, P. Li, Z. Liu, M. Sun, and Y . Liu, “Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models,” 2024

    2024

  21. [2020]

    Available: https://arxiv.org/abs/2005.14165

    [Online]. Available: https://arxiv.org/abs/2005.14165

    2005 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.