CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing

Andrey Boytsov; Anne Goujon; Cedric Lothritz; Jacques Klein; Saad Ezzini; Tegawend\'e F. Bissyand\'e; Ulrick Ble; Xunzhu Tang; Yewei Song

REVIEW 4 major objections 6 minor 1 cited by

CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CallNavi shows that realistic API routing needs unfiltered candidate pools, multi-step calls, and nested dependencies.

desk verdict CallNavi is a genuinely useful new benchmark for LLM function calling, but the empirical claims are softer than the abstract suggests because GPT-4o generated the data, judged the outputs, and tops the leaderboard. read the letter →

arxiv 2501.05255 v2 pith:5E3VCU3P submitted 2025-01-09 cs.SE cs.CL

Yewei Song , Xunzhu Tang , Cedric Lothritz , Saad Ezzini , Jacques Klein , Tegawendé F. Bissyandé , Andrey Boytsov , Ulrick Ble

show 1 more author

Anne Goujon

This is my paper · ORCID

classification cs.SEcs.CL

keywords functioncallingAPIroutingbenchmarklargelanguagemodelsnestedcallsASTmatchstabilityscoremulti-stepreasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CallNavi, a benchmark that tests whether large language models can pick the right API from an unfiltered list of more than 100 candidates, then build correct JSON calls, including multi-step and nested call sequences. Eighteen commercial, general-purpose, and fine-tuned models were evaluated on 729 questions across 10 domains, with difficulty levels from single calls to five-step dependent chains. The paper argues that existing benchmarks such as API-Bank, ToolEyes, and BFCL understate this challenge because they use smaller candidate pools or omit nested dependencies. It also proposes two inexpensive improvements: a two-step pipeline that lets a general LLM do routing while a parameter-focused model fills in arguments, and a backward-inference prompting strategy that plans API calls from the final goal backward. If the benchmark is accepted, it gives chatbot builders a more realistic stress test and simple recipes for improving API routing and parameter generation.

What carries the argument

The load-bearing object is the CallNavi dataset itself: 579 manually validated API functions and 729 questions, with an unfiltered candidate list of up to 115 APIs, organized into easy, medium, and hard levels, where hard questions require up to five sequential calls and nested dependencies. The evaluation pipeline includes syntax-validity, structural-accuracy, and AST exact-match checks on generated JSON, plus an LLM-as-a-judge score and a stability score computed as $(F_1-F_2)/(N-F_2)$ over repeated runs. The two proposed mechanisms are a two-step generation pipeline, which separates API routing from parameter generation, and a backward-inference prompting procedure that plans from the final answer's API backward through its parameter dependencies.

What would settle it

Independently sample, say, 50 CallNavi questions and have fresh annotators verify ground-truth API calls against the stated API list, checking for missing or invented functions and wrong parameter names; a substantial error rate would invalidate the benchmark's conclusions. As a second check, run the two-step and backward-inference pipelines on an existing independent function-calling benchmark with a large candidate pool; if the gains do not reproduce there, the paper's recipe is dataset-specific.

Watch

Extended reading notes

Core claim

CallNavi's central claim is that function-calling ability should be measured under realistic selection pressure: a model facing over 100 plausible APIs must name the right ones in order, fill in parameters, and chain calls where later parameters depend on earlier outputs. On this test, OpenAI's GPT-4o and GPT-4o mini lead, but even they drop sharply on hard questions, and most open-weight models that route correctly fail to produce valid parameter JSON. The paper shows that decomposing the task, first letting a general-purpose LLM select the API list and then letting a fine-tuned or smaller model generate the full call, improves overall accuracy over end-to-end generation, and that a backward-inference prompt, which starts from the final API and recursively searches for missing dependencies, raises GPT-4o's hard-level routing from 0.611 to 0.894 exact match and parameter accuracy from 0.388 to 0.729. A proposed election-style stability score quantifies run-to-run consistency across five repeated samples, complementing AST exact-match metrics.

Load-bearing premise

The load-bearing premise is that the synthetic API list and questions, generated by GPT-4o and then manually checked, faithfully represent realistic API collections and user queries; if that representativeness fails, the benchmark rankings and the measured gains of the two proposed methods may not carry over to production systems.

Editorial extensions

If this is right

API routing accuracy and parameter-generation accuracy are distinct capabilities: several open models route well but collapse on JSON parameter generation, so benchmarks should report both separately.
Splitting routing from parameter generation improves fine-tuned models like NexusRaven, Gorilla v2, and xLAM when GPT-4o supplies the routing, bringing them close to general LLMs.
Backward-inference prompting yields large gains on hard questions, suggesting that dependency-aware planning rather than larger context windows is the lever for multi-step API calls.
Context windows below roughly 6K tokens truncate CallNavi inputs and cause failures, while larger context alone does not fix reasoning errors.
The stability score reveals consistent-output differences among models that overall accuracy metrics miss, which matters for trust in finance and healthcare chatbots.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the dataset's API specs and questions are synthetic, a natural next test is to run the same routing protocols against live, versioned API documentation from a real product and see whether the model rankings and the gains from two-step routing persist.
The benchmark's hard-level improvements from backward inference might transfer to other long-horizon tool-use settings, such as code repair workflows where a final edit depends on earlier retrieval steps, but that transfer is not established by the paper.
The election stability score could be adopted alongside standard consistency metrics in other structured-generation benchmarks, since it directly penalizes ties and rewards one dominant output; its behavior with very small sample sizes deserves a check.
A practical deployment implication is that chatbot systems should cache or validate the most frequent output across repeated runs rather than trusting a single generation, because stability and accuracy are only weakly aligned for many models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the CallNavi paper. Short version: the benchmark is real and worth knowing about; the empirical results need more work before you quote them.

What's new: CallNavi gives you 729 questions over 579 APIs across 10 domains, with a difficulty split (easy/medium/hard) and, importantly, candidate pools of 100+ APIs and nested multi-step calls. That is a genuine gap relative to API-Bank, ToolEyes, and BFCL. The dataset is released, the statistics are detailed, and the manual validation is described plausibly. The Election Stability Score is a small but sensible addition—it measures majority dominance across repeated runs, which freq@topk misses. The two-step routing idea (general LLM selects APIs, fine-tuned model generates parameters) is also reasonable and worth testing.

The load-bearing weakness is the GPT-4o lifecycle: GPT-4o generated the API functions and the easy questions, and GPT-4o also acts as the judge for the 'Overall GPT Score.' That doesn't invalidate the benchmark, but it means the headline ranking (GPT-4o first) and the claimed improvements from the proposed pipelines are confounded. A self-authored, self-judged test can favor the author model in ways that have nothing to do with routing quality. The paper's threats section mentions LLM-as-a-judge subjectivity, but it treats it as a side note rather than a threat to the main conclusion.

Second, Table 6's 'overall' column is never defined, and the table omits the no-routing baseline rows. You can't verify the claimed gain from 2-step generation without seeing the same models without routing. Table 7's backward-thinking comparison is also single-point numbers with no variance or significance testing. Minor: the abstract says 'significantly improve' but there are no significance tests anywhere in the paper.

Worth noting: the stability score derivation and examples are clear; the case studies are useful. The paper would be strengthened by adding error bars and statistical tests, clarifying the overall metric, reporting baselines for Table 6, and ideally running a human or independent judge to decouple generation from evaluation.

Verdict: this deserves a serious referee. The core benchmark contribution is solid and the dataset release makes it checkable. But the paper should not be accepted in its current form—the empirical claims need to be either backed by proper comparisons and statistics or softened.

Referee Report

4 major / 6 minor

Summary. The paper introduces CallNavi, a new benchmark for evaluating LLMs on API function calling and routing. The benchmark contains 729 questions and 579 API functions across 10 domains, with easy, medium, and hard difficulty levels that require single calls, multiple sequential calls, and nested/dependent calls with up to five steps. The authors evaluate 18 commercial, general-purpose, and fine-tuned LLMs using several metrics: API routing exact match, AST-based syntax/structural/exact-match scores, an LLM-as-a-judge score using GPT-4o, and a newly proposed stability score. They also propose two optimizations for zero/few-shot settings: a two-step generation pipeline that separates API selection (routing) from parameter generation, and a backward-inference strategy for multi-step API planning. The paper reports that GPT-4o and GPT-4o mini perform best overall, that most open models are competitive at routing but weaker at parameter generation, and that the two proposed optimizations improve hard-level performance.

Significance. If the empirical claims hold, CallNavi would be a useful addition to the function-calling benchmark landscape. Unlike API-Bank, ToolEyes, and BFCL, CallNavi presents large unfiltered candidate pools, multi-call sequences, and nested dependency scenarios, which better reflect realistic chatbot deployments. The released dataset and the breadth of evaluated models are strengths: the authors provide detailed statistics, manual validation procedures, and stability measurements, and they compare against existing benchmarks. The proposed routing recipes (two-step generation and backward reasoning) are simple and potentially practical. However, the empirical evaluation currently has confounds that prevent unambiguous interpretation: GPT-4o generated part of the benchmark and also served as the judge, the key optimization tables omit baselines and define their headline metric only implicitly, and no variance or significance information is reported for the main optimization results. These issues are addressable, and the benchmark itself remains valuable, but the reported improvements and model rankings should be treated as provisional until they are resolved.

major comments (4)

[Section 4 and Section 5.2] GPT-4o has a double role that directly affects the headline ranking and the optimization claims. Section 4 states that GPT-4o generated the API functions and the easy questions, and Section 5.2 uses GPT-4o as the LLM-as-a-judge scorer for the 'Overall GPT Score' in Table 4. Thus GPT-4o is evaluated on data it helped create and is scored by itself. The Threats to Validity paragraph only acknowledges 'potential subjectivity in evaluation' and does not address this specific lifecycle. The Pearson correlation with AST match reported in Section 6.4 does not remove the confound, because both metrics can share the same model-specific bias. Please provide a concrete test: e.g., score a random sample with a different judge model or with human annotators and report agreement/reranking, or present model rankings using only the AST metrics. Without this, RQ1 and the claim that GPT-4o is the best model are not fully supported.
[Table 6 / Section 7.1] The 'overall' column in Table 6 is never defined in Section 7.1 or in the text around the table. From the numbers it appears to be the difficulty-weighted average of the easy/medium/hard columns (e.g., for NexusRaven: (0.657*456 + 0.457*188 + 0.188*85)/729 = 0.551), but the text should state this explicitly and specify which underlying metric is being averaged (presumably AST match). More importantly, Table 6 omits the corresponding one-step baseline rows from Table 4 (the 'without routing' condition). The claim that separating API selection from parameter generation 'significantly improves performance' cannot be verified without these baselines. Please add the baseline rows for each model and state the metric and aggregation formula.
[Tables 6 and 7 / Sections 7.1-7.2] The optimization results are reported as single-point estimates with no variance or significance testing, despite the paper itself using five runs for stability experiments in Section 6.4.1. Section 7.2 claims a '30% improvement in hard-level API calling tasks', but the two rows in Table 7 show absolute improvements of 0.283 and 0.341, so the source of the 30% figure is unclear. LLM outputs are stochastic, so a single run can produce misleading gains. Please report mean and standard deviation (or confidence intervals) over at least the same five runs used for the stability scores, and apply a paired significance test when claiming that the proposed methods improve routing or parameter generation.
[Section 5.3] The proposed Levenshtein stability score measures the average normalized similarity to the first output x0, not agreement among all outputs. If the first sample is an outlier and the remaining n-1 samples are identical, the score will be low even though the model is highly stable. This undermines the claim in Section 6.4.1 that the score measures 'the stability of the model's outputs across multiple runs'. Please replace it with a symmetric measure, such as the average pairwise similarity or similarity to the majority output. Also, the Election Stability Score definition does not specify F2 when there is only one unique output; the example sets F2=0, but this convention should be made part of the formal definition.

minor comments (6)

[Table 4] The column headings 'All Avg.' and 'Macro Avg.' are not defined. Please clarify whether they are weighted by question count or are simple means across the difficulty columns.
[Table 4] NexusRaven and Gorilla v2 have 'N/A' for Syntax Validity. Please explain why these fine-tuned models are not evaluated on that metric, since they do have entries for the AST-match columns.
[Section 6.1] The model selection criteria are described loosely ('based on their performance, architecture, and relevance'). Please state the inclusion criteria more concretely, especially because the paper excludes some models such as Firefunction v2 but does not give a complete list of considered models.
[Section 8.5 and Table 8] Table 8 is difficult to read because the rows for different models are run together in the text; please format it as a normal table with separate rows and clear column alignment.
[Section 9.2] The conclusion says '500 APIs and 700 questions', while Table 2 reports 579 API functions and 729 questions. Please use consistent numbers.
[Throughout] There are several minor typos and formatting inconsistencies, e.g., 'Mistal models' in Section 6.4 and 'Y AML' in Section 8.5 and Table 8. Please proofread the final version.

Circularity Check

1 steps flagged · score 3.0 of 10

GPT-4o authored the benchmark and served as its judge, making the headline ranking partly self-referential; no formal derivation reduces to its inputs, so circularity is mild.

other [Section 4 (Initial API Function Generation; Generation of Easy Questions), Section 5.2 (LLM-as-a-Judge Evaluation), Section 6.4 (Benchmarks Results), Threats to Validity]
"Using GPT-4o, we generated API function names, descriptions, parameters, and return values based on a variety of scenario descriptions spanning multiple domains. ... For the easy subset, we used GPT-4o to generate questions related to API usage. ... We also use GPT-4o language models to evaluate whether the generated JSON outputs correspond accurately to the ground truth [45]. ... OpenAI’s models, GPT4o and GPT4o mini, consistently outperform the others ... and overall GPT score (0.913 and 0.908). ... LLM-as-a-judge introduces potential subjectivity in evaluation."

The structural loop is that GPT-4o generated the API functions and easy questions, and GPT-4o also scored all model outputs through LLM-as-a-judge. The headline empirical result, that GPT-4o and GPT-4o mini consistently outperform the other models, is therefore measured on tasks written by GPT-4o and scored by a GPT-4o judge. The ground truth and the scoring function both originate from the very model being ranked, so the top ranking is not an independent measurement; it is partly a self-consistency result of GPT-4o with itself. This is not a full reduction by construction, because AST-match metrics and the reported correlation with BFCL provide some external signal, but the central 'best model' conclusion is weakened by the GPT-4o-authored dataset and GPT-4o judge.

full rationale

The paper's contributions are empirical rather than derivational. The two-step routing and backward-inference pipelines are evaluated on the CallNavi set, and their reported gains are not obtained by fitting a parameter and then predicting the same quantity; the stability score is an explicitly defined formula rather than a disguised input. Self-citations ([10, 34]) are used only to motivate the stability measurement and do not carry a load-bearing argument. The clearest circularity signal is the GPT-4o lifecycle: GPT-4o generated the API functions and easy questions, and GPT-4o served as the LLM judge behind the overall GPT score used in the leaderboard. This does not make the evaluation equivalent to its inputs by definition, but it weakens the independence of the conclusion that OpenAI models are best. External grounding through BFCL comparison and AST-based metrics limits the severity, so the score is moderate rather than high.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the validity of the synthetically generated dataset and on several evaluation design choices. No numerical parameters are fitted to data; the burden is mostly in dataset representativeness and metric conventions.

assumptions (5)

domain assumption GPT-4o-generated API functions and questions, after manual validation, are accurate and representative of real-world function-calling scenarios.
Section 4 'Initial API Function Generation' states the dataset was generated with GPT-4o and manually reviewed; the benchmark's validity rests on this.
domain assumption The unfiltered presentation of all 100+ API candidates in the prompt is a realistic setting for API routing.
Section 1 claims 'unfiltered API selection with over 100 candidates'; in practice, systems often retrieve a subset, so this design choice shapes all results.
domain assumption Evaluating local models with 4-bit quantization and default Ollama settings yields valid model comparisons.
Section 6.2 states all local models run with 4-bit quantization on default Ollama settings, which may disadvantage some models.
domain assumption GPT-4o as an LLM judge produces scores that align with ground-truth AST metrics.
Section 5.2 uses GPT-4o for evaluation; the paper reports a high correlation (0.934) with AST match, but judge bias is a known risk.
domain assumption Placeholder tokens for dependent parameters are a valid way to handle multi-step outputs where ground truth is not unique.
Section 5.1 says placeholder tokens are used and excluded from strict comparisons; the evaluation depends on this convention.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing." pith.science (2026). https://pith.science/paper/5E3VCU3P

@misc{pith2026250105255,
  author       = {Pith},
  title        = {Pith review of: CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5E3VCU3P}},
  note         = {Machine review of arXiv:2501.05255}
}

read the original abstract

API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.

Figures

Figures reproduced from arXiv: 2501.05255 by the authors.

**Figure 1.** Example of API Calling pipeline via LLM 2 Related Work Generating and executing accurate API calls is crucial to integrating LLM into real-world conversation applications. Existing benchmarks, such as API-Bank [18], ToolEyes [40], and ToolBench [21], evaluate API selection and execution capabilities but often rely on prefiltered API candidate pools, lack nested API tasks, or focus on narrow domain coverage. In con… view at source ↗

**Figure 3.** Pipeline of AST Match Score into object trees, comparing each node and leaf. A perfect match across all nodes results in a score of 1 for AST Exact Match. This multistep evaluation ensures a thorough assessment of the accuracy of API function calls and the structural integrity of the parameters, allowing for a granular analysis of the performance. 5.2 LLM-as-a-Judge Evaluation We also use GPT-4o language models to e… view at source ↗

**Figure 2.** Example of Evaluation Pipeline As [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: 2-Steps Generation Pipeline APIs are then provided to a fine-tuned/LAM model, which focuses on generating the correct API calls along with the necessary parameters. This sequential process allows the general LLM to efficiently handle API selection, while the fine-tune…

**Figure 5.** Figure 5: Backward Thinking Pipeline (2) Checking Parameter Completeness: The model verifies whether all required parameters for the final API are available. If any essential information is missing, the model does not proceed with execution but instead considers the necessary st…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Doc2Agent: Scalable Generation of Tool-Using Agents from API Documentation
cs.CL 2025-06 reject novelty 6.0 of 10

Doc2Agent automatically converts unstructured REST API documentation into validated, Python-based tools for AI agents, reporting a 55% relative WebArena improvement over direct API calling.

Reference graph

Works this paper leans on

48 extracted references · 16 canonical work pages · cited by 1 Pith paper

[1]

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 (2024)

arXiv 2024
[2]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023
[3]

Mistral Ai. 2024. AI in abundance. https://mistral.ai/news/september-24-release/

work page 2024
[4]

Mistral Ai. 2024. Mistral NeMo. https://mistral.ai/news/mistral-nemo/

work page 2024
[5]

Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2024. Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation. arXiv preprint arXiv:2403.06988 (2024)

arXiv 2024
[6]

Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gaši´c. 2018. Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278 (2018)

arXiv 2018
[7]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

arXiv 2021
[8]

Cohere. [n. d.]. The Command R model — Cohere. https://docs.cohere.com/v2/ docs/command-r

work page

Show all 48 references

[9]

Yihong Dong, Ge Li, and Zhi Jin. 2023. CODEP: grammatical seq2seq model for general-purpose code generation. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 188–198

2023
[10]

Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. 2023. Short- cut Learning of Large Language Models in Natural Language Understanding. arXiv:2208.11857 [cs.CL] https://arxiv.org/abs/2208.11857

2023 arXiv
[11]

Pawel Garbacki and Benny Chen. 2024. Firefunction-v2: Function calling ca- pability on par with GPT4o at 2.5x the speed and 10% of the cost. https: //fireworks.ai/blog/firefunction-v2-launch-post

2024
[12]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 (2025)

2025 arXiv
[13]

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. StableToolBench: Towards Stable Large- Scale Benchmarking on Tool Learning of Large Language Models. arXiv preprint arXiv:2403.07714 (2024)

2024 arXiv
[14]

Jinwei He and Feng Lu. 2024. CauseJudger: Identifying the Cause with LLMs for Abductive Logical Reasoning. arXiv preprint arXiv:2409.05559 (2024)

2024 arXiv
[15]

Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.02349 (2020)

2020 arXiv
[16]

Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352 (2023)

2023 arXiv
[17]

Patil Fanjia Yan, Tianjun Zhang, Ion Stoica, and Joseph E

Charlie Cheng-Jie Ji, Huanzhi Mao, Shishir G. Patil Fanjia Yan, Tianjun Zhang, Ion Stoica, and Joseph E. Gonzalez. 2024. Gorilla OpenFunctions v2. https: //gorilla.cs.berkeley.edu//blogs/7_open_functions_v2.html

2024
[18]

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244 (2023)

2023 arXiv
[19]

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092–1097

2022
[20]

We Need Structured Output

Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. 2024. "We Need Structured Output": Towards User-centered Constraints on Large Language Model Output. InExtended Abstracts of the CHI Conference on Human Factors...

2024
[21]

Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. 2024. ToolACE: Winning the Points of LLM Function Calling. arXiv preprint arXiv:2409.00920 (2024)

2024 arXiv
[22]

Meta-AI. 2024. Introducing Llama 3.1: our most capable models to date. https: //ai.meta.com/blog/meta-llama-3-1/

2024
[23]

Meta-AI. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, cus- tomizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision- edge-mobile-devices/

2024
[24]

Nexusflow.ai. 2023. NexusRaven-V2: Surpassing GPT-4 for Zero-shot Function Calling. https://nexusflow.ai/blogs/ravenv2

2023
[25]

Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334 (2023)

2023 arXiv
[26]

Jason Paul. 2024. NVIDIA announces first digital Human Technologies On-Device Small Language Model, improving conversation for game characters | NVIDIA blog. https://blogs.nvidia.com/blog/digital-human-technology-mecha-break/

2024
[27]

Yun Peng, Shuqing Li, Wenwei Gu, Yichen Li, Wenxuan Wang, Cuiyun Gao, and Michael R Lyu. 2022. Revisiting, benchmarking and exploring API recommenda- tion: How far are we? IEEE Transactions on Software Engineering 49, 4 (2022), 1876–1897

2022
[28]

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large lan- guage models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 (2023)

2023 arXiv
[29]

Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax net- works for code generation and semantic parsing. arXiv preprint arXiv:1704.07535 (2017)

2017 arXiv
[30]

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

2024 arXiv
[31]

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301 (2023)

2023 arXiv
[32]

Bissyande

Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawende F. Bissyande. 2024. CodeAgent: Autonomous Communicative Agents for Code Review. arXiv:2402.02172 [cs.SE] https://arxiv.org/abs/2402.02172

2024 arXiv
[33]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024)

2024 arXiv
[34]

Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F Bissyandé. 2023. Is ChatGPT the ultimate programming assistant–how far is it? arXiv preprint arXiv:2304.11938 (2023)

2023 arXiv
[35]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

2023 arXiv
[36]

Adina Trufinescu. 2024. Discover the new Multi-Lingual, High-Quality PHI- 3.5 SLMS. https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/ discover-the-new-multi-lingual-high-quality-phi-3-5-slms/ba-p/4225280

2024
[37]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

2022
[38]

Patil, Ion Stoica, and Joseph E

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley Function Calling Leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_ leaderboard.html

2024
[39]

Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. 𝜏- bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045 (2024)

2024 arXiv
[40]

Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. 2024. Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprint arXiv:2401.00741 (2024)

2024 arXiv
[41]

Shayan Zamanirad, Boualem Benatallah, Moshe Chai Barukh, Fabio Casati, and Carlos Rodriguez. 2017. Programming bots by synthesizing natural language ex- pressions into API invocations. In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 832–837

2017
[42]

Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. 2024. xLAM: A Family of Large Action Models to Empower AI Agent Systems. arXiv preprint arXiv:2409.03215 (2024)

2024 arXiv
[43]

Shengnan Zhang, Yan Hu, and Guangrong Bian. 2017. Research on string sim- ilarity algorithm based on Levenshtein Distance. In 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). IEEE, 2247–2251

2017
[44]

Yinger Zhang, Hui Cai, Xeirui Song, Yicheng Chen, Rui Sun, and Jing Zheng
[45]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

2023
[46]

EASE 2025, 17–20 June, 2025, Istanbul, Türkiye T rovato et al

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. EASE 2025, 17–20 June, 2025, Istanbul, Türkiye T rovato et al

2025
[2023]

arXiv preprint arXiv:2310.04474 (2023)

Reverse chain: A generic-rule for llms to master multi-api planning. arXiv preprint arXiv:2310.04474 (2023)

2023 arXiv
[2024]

arXiv preprint arXiv:2406.15877 (2024)

Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 (2024)

2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 (2024)

arXiv 2024

[2] [2]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023

[3] [3]

Mistral Ai. 2024. AI in abundance. https://mistral.ai/news/september-24-release/

work page 2024

[4] [4]

Mistral Ai. 2024. Mistral NeMo. https://mistral.ai/news/mistral-nemo/

work page 2024

[5] [5]

Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2024. Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation. arXiv preprint arXiv:2403.06988 (2024)

arXiv 2024

[6] [6]

Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gaši´c. 2018. Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278 (2018)

arXiv 2018

[7] [7]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

arXiv 2021

[8] [8]

Cohere. [n. d.]. The Command R model — Cohere. https://docs.cohere.com/v2/ docs/command-r

work page