Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Financial AI agents should be judged on whether their tool calls respect timeliness, intent, and regulatory domain, and this paper's 760-tool benchmark makes such violations measurable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:49 UTC pith:6X2Z7TGA

load-bearing objection A genuinely useful executable financial tool library and question set, wrapped in a compliance-evaluation framework that still needs validation before its headline numbers can be trusted. the 3 major comments →

arxiv 2603.08262 v2 pith:6X2Z7TGA submitted 2026-03-09 cs.AI

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

Jiaxuan Lu , Kong Wang , Yemin Wang , Qingmei Tang , Hongwei Zeng , Xiang Chen , Jiahao Pi , Shujian Deng
show 4 more authors
Lingzhi Chen Yi Fu Kehua Yang Xiao Sun
This is my paper
classification cs.AI
keywords LLM agentstool usefinancial benchmarkscompliance evaluationtimelinessintent restraintregulatory domainexecutable APIs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

FinToolBench argues that evaluating a financial LLM agent by whether its tool calls execute is not enough: a syntactically perfect call can be wrong for finance if the data is stale, if the call exceeds the user's intent, or if the tool belongs to the wrong market domain. To make those failures measurable, the paper builds a runnable benchmark coupling 760 real free-tier financial tools with 295 queries that genuinely require tool use, and tags each tool with three attributes: timeliness, intent type, and regulatory domain. Evaluation then reports both capability metrics (invocation and execution success) and compliance mismatch rates (TMR, IMR, DMR) computed from each run's auditable tool trace. The paper also introduces FATR, a retrieval-and-reasoning baseline that injects those finance attributes into tool cards, and reports that doing so improves conditional execution success and lowers mismatch rates. If the benchmark's judge-based compliance layer proves valid, financial AI evaluation gains a shared testbed that can distinguish a tool-using agent that is merely executable from one that is trustworthy under finance-specific constraints.

Core claim

On its own terms, the paper's central discovery is that finance-specific acceptability can be defined call-by-call and measured from execution traces. Each of the 760 tools carries three finance attributes—timeliness, intent type, and regulatory domain—and the evaluation records every tool call as an auditable trace entry. An LLM judge then marks each call as matched or mismatched to the question on each attribute; if any call in a trace violates a dimension, the question is counted as mismatched, yielding TMR, IMR, and DMR. The reported runs show no planner that is both eager and reliable: one backend invokes tools often but fails execution, another is conservative and precise, and the FATR

What carries the argument

The load-bearing machinery is the tool-trace plus finance-attribute schema. Every tool is annotated with three attributes—timeliness (realtime/daily/as_filed/periodic/static), intent type (informational/advisory/transactional), and a set-valued regulatory domain—and every evaluation run logs a structured trace recording step, tool name, parameters, output, and error. From a trace, judge functions assess per-call alignment in each dimension, and the paper aggregates them into question-level mismatch rates TMR, IMR, and DMR. FATR operationalizes the same schema at planning time by retrieving a small candidate set, formatting tool cards with the finance tags, inferring the question's constraint

Load-bearing premise

Everything the paper's compliance numbers say depends on the automated judge's per-call verdicts about timeliness, intent, and regulatory domain matching what a financial compliance expert would decide; the paper reports only spot-check human QA and no direct agreement measurement.

What would settle it

Have a panel of financial-compliance experts independently label a random sample of 50–100 tool traces as compliant or violating timeliness, intent restraint, and regulatory-domain alignment, and compare their verdicts with the judge outputs behind Eq. (1); if agreement is at chance on any dimension, the reported TMR/IMR/DMR rates measure the judge rather than the agents. In parallel, rerun the main protocol across multiple seeds to check whether model rankings change more than run-to-run variance.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Two agents with identical tool-execution success can now be separated by their compliance: TMR/IMR/DMR expose whether the trace used stale data, escalated beyond intent, or crossed market domains.
  • Because the benchmark is executable end-to-end on free-tier tools, the same 295 queries can serve as a shared regression suite for financial agents across model versions and providers.
  • Attribute injection changes planner behavior: FATR with finance tags lowers tool invocation slightly but raises conditional execution success and cuts mismatch rates, indicating that making constraints salient improves tool choice rather than executor stability.
  • The capability/compliance split converts diagnosis from a single pass/fail into a distinction between 'cannot call tools' and 'called the wrong kind of tool'.
  • The category-level breakdown shows that aggregate scores hide strong heterogeneity, so deployment decisions may need per-category compliance budgets rather than one global number.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the compliance layer survives a human-agreement check, the benchmark's attribute schema is a template for other regulated domains—healthcare, legal, or public administration—where stale, out-of-scope, or unrequested actions carry similar risks.
  • Because the 760 tools are live free-tier APIs, the benchmark could detect tool drift: re-running the same queries months apart would quantify how much of an agent's failures come from endpoints changing, a dimension the paper leaves to future work.
  • The reported trade-off between coverage and precision suggests a deployment calibration: a planner's invocation threshold should be tuned against the relative cost of a wrong-but-tool-backed answer versus a missed answer—an axis the paper does not quantify.
  • Given the paper's own acknowledgement that LLM judges can be unstable, a natural next step is to publish a human-agreement study on the judge labels and seed variance for the main results table.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. FinToolBench introduces a runnable benchmark for financial tool-use agents, coupling 760 executable free-tier tools (RapidAPI and AkShare) with 295 tool-required questions derived from FinanceBench and OpenFinData. Each tool is annotated with three finance attributes—timeliness, intent type, and regulatory domain—via a Qwen3-8B three-vote majority protocol. The evaluation separates capability metrics (TIR, TESR, CER, Soft Score, CSS) from compliance mismatch rates (TMR, IMR, DMR), with compliance scored by a GPT-5.1 judge checking each tool call's attributes against the question. The paper also proposes FATR, a finance-aware retrieval and ReAct-planning baseline that injects finance attributes into tool cards and adds execution stabilization. Experiments compare four LLM planners and an attribute-injection ablation, reporting trade-offs between coverage, execution reliability, and compliance, with case studies showing that attribute injection changes tooling behavior but does not guarantee final-answer correctness.

Significance. The benchmark construction is concrete and audit-friendly: the staged pipeline, rule-based executability filters (Appendix A), normalized manifest, and detailed trace schema (Table 2) are valuable assets. The case studies in Appendix I are a real strength, honestly showing that FATR can produce cleaner traces while still failing on numeric fidelity or task framing. If the compliance layer were externally validated, FinToolBench would fill a genuine gap as a standard, trace-level testbed for financial tool-use agents. However, the paper's central contribution—measuring timeliness, intent, and regulatory-domain compliance—currently rests on unvalidated LLM-generated labels and a single-run LLM judge, and all agent results are reported without seed variance. The significance is therefore conditional: the infrastructure is promising, but the metric validity and statistical reliability need to be established before the benchmark can support comparative conclusions.

major comments (3)
  1. [§3.2.4, §B.2, Eq. (1), §5.4] The compliance metrics TMR/IMR/DMR are computed by GPT-5.1 judging each tool call against tool attributes that were produced by Qwen3-8B majority vote from tool names and descriptions. No human agreement statistics, calibration, or error analysis are reported for either the attribute labels or the judge decisions. The human spot-check in §3.2.4 mentions a 'statistically significant sample' but provides no sample size, measured agreement, or protocol details. Given that §2.3 itself cites LLM-judge instability, relying on a single un-repeated GPT-5.1 decision per call (§5.4) is a serious risk. Without external validation against domain experts, the advertised TMR/IMR/DMR numbers may reflect agreement with the pipeline's own labels rather than finance acceptability. This is load-bearing for the benchmark's central claim and needs to be addressed with a human-agreement study, label-validity
  2. [§4.2, §6.2, Eq. (1)] FATR's compliance benefit is in part circular: FATR injects finance tags into tool cards, and the compliance judge checks exactly those tags when computing mismatch rates. A planner that follows the prompt rule 'informational > advisory; avoid transactional' can trivially lower IMR without necessarily improving true compliance. To establish that the TMR/IMR/DMR reductions in Figure 5 are not an artifact of shared label injection, the authors should run ablations with corrupted or permuted tags, blind the judge to the injected tool-card text, or compare against human-validated labels. Without such tests, the claim that attribute injection 'improves compliance alignment' is not identifiable from the reported experiments.
  3. [Table 3, §5.1, §5.4] All results in Table 3 are single-run, with no seed variance, confidence intervals, or error bars. Given that LLM planners are stochastic and that §2.3 argues LLM judges are unstable, the observed model differences (e.g., TIR 0.4407 vs. 0.2267, IMR 0.5000 vs. 0.7231) may be within run-to-run noise. The comparison between model backends and the FATR ablation needs at least 3–5 independent runs per condition with standard errors, and the compliance judge should either be repeated or accompanied by an agreement measure. This is necessary to support the paper's comparative conclusions.
minor comments (4)
  1. [§3.4, Eq. (1), D.2] The notation for the compliance judge is inconsistent: Eq. (1) writes J_T(q, A(t_k), τ_k), while D.2 defines J_T(q, t_k, x_k, o_k, T(t_k)). Please unify the notation and state explicitly what context the judge receives (question, tool name, arguments, output, and attribute tag).
  2. [Figure 7] The heatmap column header reads 'FMR' but the metric is called TMR in the text. Please correct the label.
  3. [§3.2.4] The phrase 'statistically significant sample' should either be backed with the sample size and the statistical test used, or softened to 'a spot-check sample' to avoid overstating the nature of the human audit.
  4. [§5.4] The decision to use a single judge decision per tool call for compliance is stated without justification, despite three repeats being used for Soft Score. Please explain and, if possible, report repeat agreement on a small sample.

Circularity Check

0 steps flagged

No significant circularity: compliance metrics are an operational definition, and FATR's use of the same attribute tags is a deliberate treatment, not a reduction of the result to its inputs.

full rationale

FinToolBench is an empirical benchmark paper rather than a derivation chain, and no central claim reduces to its own inputs by construction. The compliance metrics in Eq. (1) are operationally defined: TMR/IMR/DMR state that a question is mismatched when an LLM judge, looking at tool metadata A(t), finds a call inconsistent with the question's timeliness, intent, or domain requirement. FATR injects those same attribute tags into tool cards and instructs the planner to prefer tags that match the inferred requirements. This is the intended intervention, not circularity: the judge evaluates the resulting tool trace, and FATR improves compliance by making the labels salient to the planner. The labels themselves are produced by Qwen3-8B majority vote (§B.2) and judge decisions are not accompanied by human-agreement statistics, which is a real validity concern for the benchmark's external meaning, but it is not a circular step under the required standard of exhibiting an equation or fitted parameter that is equivalent to its input. The paper's self-citations (e.g., [13], [17], [25], [27]) appear only as related-work context and are not load-bearing for the benchmark's construction or evaluation claims. Appendix I further shows that attribute injection does not mechanically guarantee correctness, indicating the evaluation is not trivially gaming its own judge. Therefore, no significant circularity is present; the unvalidated label/judge pipeline should be assessed as a measurement-validity limitation, not as circular reasoning.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

This benchmark paper has no fitted constants and no derivations; its load-bearing inputs are hand-chosen protocol constants (Top-K=20, max_steps=5, timeout/retries, vote thresholds) and a chain of LLM-moderated assumptions: Qwen3-8B labels the tools, Qwen3-8B selects the questions, GPT-5.1 grades both correctness and compliance, and only spot-check human QA sits between the pipeline and domain truth. The honest ledger therefore lists the protocol constants plus these LLM-as-ground-truth assumptions.

free parameters (4)
  • Top-K retrieval candidate set = 20
    Determines the action space both in construction and in FATR: if the correct tool falls outside the top-20, no planner can succeed, coupling all results to retriever quality (§4.2, §C.2).
  • Interaction horizon max_steps = 5
    Caps ReAct loops at 5 tool-augmented steps; given multi-tool traces that use up to 4 tools (Fig. 6) plus retries, some questions may be truncated by this hand-chosen limit, affecting TESR and CSS (§4.3, §5.1).
  • Execution budget per call = 60s timeout, 2 retries
    Determines which live endpoints count as 'executable' during curation and which calls succeed during evaluation; no sensitivity analysis reported (§5.1, App A).
  • Vote/threshold constants = 3 samples majority (≥2 votes) in construction; 3 judge repeats for Soft Score; 1 judge decision per compliance call
    Hand-chosen protocol constants that decide which attribute labels and which compliance verdicts survive; their effect on benchmark validity is untested (§3.1.4, §B.2, §5.4).
axioms (5)
  • domain assumption GPT-5.1 judge output approximates human/regulatory truth for both correctness and compliance without calibration
    All Soft Score, CSS, TMR, IMR, and DMR values depend on GPT-5.1 judgments (§3.4, §5.4). The paper cites judge-instability results ([10], [11], [6]) but reports no human agreement, calibration, or error-bound study for its own judge.
  • domain assumption Qwen3-8B majority-vote finance attribute annotations are correct
    Tool attributes are generated by Qwen3-8B three-vote labeling with only spot-check human QA (§3.1.4, §B.2, §3.2.4). If labels are wrong, both FATR's tool cards and the compliance metrics inherit the same error.
  • domain assumption Qwen3-8B tool-required filtering and tool-question alignment are valid
    The 295-question set is defined by Qwen3-8B judgments (length ≤500 chars, ≥2-of-3 votes) with human spot-checks only (§3.2.1–3.2.4, §C). The '295 tool-required queries' claim inherits this model's judgment.
  • domain assumption Free-tier APIs remain executable and stable after construction
    Executability is verified at build time (App A); live endpoints drift, which the paper defers to future work (§7). Any drift changes TESR/CSS and the compliance denominators between runs.
  • domain assumption FinanceBench/OpenFinData gold answers are correct scoring targets
    Soft Score and CSS are computed against external gold answers from FinanceBench [12] and OpenFinData [19] (§3.2.1, §D.1); benchmark quality inherits the source QA datasets' quality.
invented entities (1)
  • None no independent evidence
    purpose: No new physical or conceptual entities are postulated; the contributions are a benchmark artifact, a metric family, and a baseline method.
    The paper introduces no new particles, forces, dimensions, or conserved quantities; FATR and the TMR/IMR/DMR metrics are methods, not entities.

pith-pipeline@v1.3.0-alltime-deepseek · 19535 in / 16567 out tokens · 165338 ms · 2026-08-04T05:49:56.278868+00:00 · methodology

0 comments
read the original abstract

The integration of Large Language Models (LLMs) into the financial domain is driving a paradigm shift from passive information retrieval to dynamic, agentic interaction. While general-purpose tool learning has witnessed a surge in benchmarks, the financial sector, characterized by high stakes, strict compliance, and rapid data volatility, remains critically underserved. Existing financial evaluations predominantly focus on static textual analysis or document-based QA, ignoring the complex reality of tool execution. Conversely, general tool benchmarks lack the domain-specific rigor required for finance, often relying on toy environments or a negligible number of financial APIs. To bridge this gap, we introduce FinToolBench, the first real-world, runnable benchmark dedicated to evaluating financial tool learning agents. Unlike prior works limited to a handful of mock tools, FinToolBench establishes a realistic ecosystem coupling 760 executable financial tools with 295 rigorous, tool-required queries. We propose a novel evaluation framework that goes beyond binary execution success, assessing agents on finance-critical dimensions: timeliness, intent type, and regulatory domain alignment. Furthermore, we present FATR, a finance-aware tool retrieval and reasoning baseline that enhances stability and compliance. By providing the first testbed for auditable, agentic financial execution, FinToolBench sets a new standard for trustworthy AI in finance. The tool manifest, execution environment, and evaluation code will be open-sourced to facilitate future research.

Figures

Figures reproduced from arXiv: 2603.08262 by Hongwei Zeng, Jiahao Pi, Jiaxuan Lu, Kehua Yang, Kong Wang, Lingzhi Chen, Qingmei Tang, Shujian Deng, Xiang Chen, Xiao Sun, Yemin Wang, Yi Fu.

Figure 1
Figure 1. Figure 1: FinToolBench overview. Left: the scope of our bench [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FinToolBench dataset construction pipeline. Stage 1 collects raw tool sources. Stage 2 performs tool curation and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of Finance-Aware Tool Routing (FATR). FATR takes a Question Inventory and a Tool Inventory & Retrieval [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Tool cards for attribute injection and constraint [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Attribute injection ablation in FATR. We compare [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Tool usage distribution on FinToolBench. Left: [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Metrics by category for Doubao-Seed-1.6 on FinToolBench. Rows are question categories and columns are evaluation [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Single-box, end-to-end comparison for financebench_9. The baseline fails due to an interface mismatch. FATR recovers a valid tool trace, but the final answer remains incorrect because it does not follow the dataset’s intended reasoning criterion. Case Study 2 (openfindata_release_155): injection prunes redundant tools while preserving correctness Question. As a financial analyst, evaluate the downside risk… view at source ↗
Figure 9
Figure 9. Figure 9: Single-box comparison for openfindata_release_155. Injected finance attributes reduce redundant and incompatible tool calls while preserving correctness [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Single-box comparison for financebench_33. Injected finance attributes change the tool strategy, but the resulting proxy-based estimate remains incorrect [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

    cs.AI 2026-04 unverdicted novelty 7.0

    FinTrace supplies trajectory-level metrics for LLM financial tool calling, exposing gaps in information use and output quality, while its preference dataset enables DPO training that boosts intermediate metrics.

  2. The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidence

    cs.CE 2026-05 accept novelty 5.0

    Reported alpha from end-to-end LLM trading agents does not constitute deployment evidence until it passes structural tests for temporal integrity, frictions, robustness, calibration, execution, and disaggregation.

Reference graph

Works this paper leans on

37 extracted references · 20 linked inside Pith · cited by 2 Pith papers

  1. [1]

    2025.𝜏 2-bench: Evaluating conversational agents in a dual-control environment

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik R Narasimhan. 2025.𝜏 2-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982(2025)

    Pith/arXiv arXiv 2025

  2. [2]

    Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. 2025. Finance agent benchmark: Benchmarking LLMs on real-world financial research tasks.arXiv preprint arXiv:2508.00828(2025)

    Pith/arXiv arXiv 2025

  3. [3]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216 (2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Hua...

    2021

  5. [5]

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. 2024. WorkArena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718(2024)

    Pith/arXiv arXiv 2024

  6. [6]

    Jennifer D’Souza, Hamed Babaei Giglou, and Quentin Münch. 2025. YESciEval: Robust LLM-as-a-judge for scientific question answering. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 13749–13783

    2025

  7. [7]

    Jiayu Guo, Yu Guo, Martha Li, and Songtao Tan. 2025. FLAME: Financial large-language model assessment and metrics evaluation.arXiv preprint arXiv:2501.06211(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, and Liwen Zhang

  9. [9]

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. StableToolBench: Towards stable large- scale benchmarking on tool learning of large language models.arXiv preprint arXiv:2403.07714(2024)

    Pith/arXiv arXiv 2024

  10. [10]

    Rajarshi Haldar and Julia Hockenmaier. 2025. Rating Roulette: Self-inconsistency in LLM-as-a-judge frameworks. InFindings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 24986– 25004

    2025

  11. [11]

    Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. 2024. LLM-Rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 13806–13834

    2024

  12. [12]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. FinanceBench: A new benchmark for financial question answering.arXiv preprint arXiv:2311.11944(2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Yankai Jiang, Wenjie Lou, Lilong Wang, Zhenyu Tang, Shiyang Feng, Jiaxuan Lu, Haoran Sun, Yaning Pan, Shuang Gu, Haoyang Su, Feng Liu, Wangxu Wei, Pan Tan, Dongzhan Zhou, Fenghua Ling, Cheng Tan, Bo Zhang, Xiaosong Wang, Lei Bai, and Bowen Zhou. 2025. SCP: Accelerating discovery with a global web of autonomous scientific agents.arXiv preprint arXiv:2512.2...

    arXiv 2025

  14. [14]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A comprehensive benchmark for tool-augmented LLMs.arXiv preprint arXiv:2304.08244(2023)

    Pith/arXiv arXiv 2023

  15. [15]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al . 2024. AgentBench: Evalu- ating LLMs as agents. InInternational Conference on Learning Representations. arXiv:2308.03688

    Pith/arXiv arXiv 2024

  16. [16]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303.16634(2023)

    Pith/arXiv arXiv 2023

  17. [17]

    Jiaxuan Lu, Ziyu Kong, Yemin Wang, Rong Fu, Haiyuan Wan, Cheng Yang, Wenjie Lou, Haoran Sun, Lilong Wang, Yankai Jiang, Xiaosong Wang, Xiao Sun, and Dongzhan Zhou. 2026. Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoning.arXiv preprint arXiv:2601.07641(2026)

    arXiv 2026

  18. [18]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. GAIA: a benchmark for general AI assistants.arXiv preprint arXiv:2311.12983(2023)

    Pith/arXiv arXiv 2023

  19. [19]

    OpenFinData. 2024. OpenFinData: An Open Financial Evaluation Dataset. GitHub repository. Release v0.1.0; Accessed: 2026-02-07

    2024

  20. [20]

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267)....

    2025

  21. [21]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2024. Go- rilla: Large language model connected with massive APIs.Advances in Neural Information Processing Systems37 (2024), 126544–126565

    2024

  22. [22]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al . 2024. ToolLLM: Facilitating large language models to master 16,000+ real-world APIs. InInternational Conference on Learning Representations. arXiv:2307.16789

    Pith/arXiv arXiv 2024

  23. [23]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems36 (2023), 68539–68551

    2023

  24. [24]

    Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Pa- tel, Esin Durmus, Spandana Gella, Karolina Stańczak, and Siva Reddy. 2025. SafeArena: Evaluating the safety of autonomous web agents.arXiv preprint arXiv:2503.04957(2025)

    Pith/arXiv arXiv 2025

  25. [25]

    Haiyuan Wan, Chen Yang, Junchi Yu, Meiqi Tu, Jiaxuan Lu, Di Yu, Jianbao Cao, Ben Gao, Jiaqing Xie, Aoran Wang, Wenlong Zhang, Philip Torr, and Dongzhan Zhou. 2025. DeepResearch Arena: The first exam of LLMs’ research abilities via seminar-grounded tasks.arXiv preprint arXiv:2509.01396(2025)

    arXiv 2025

  26. [26]

    Hongfei Xia, Hongru Wang, Zeming Liu, Qian Yu, Yuhang Guo, and Haifeng Wang. 2025. SafeToolBench: Pioneering a prospective benchmark to evaluating tool utilization safety in LLMs.arXiv preprint arXiv:2509.07315(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Cheng Yang, Jiaxuan Lu, Haiyuan Wan, Junchi Yu, and Feiwei Qin. 2025. From What to Why: A multi-agent system for evidence-based chemical reaction con- dition reasoning.arXiv preprint arXiv:2509.23768(2025)

    arXiv 2025

  28. [28]

    2024.𝜏- bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045(2024)

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. 2024.𝜏- bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations. arXiv:2210.03629

    Pith/arXiv arXiv 2023

  30. [30]

    Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, and Chen Ma. 2025. Crowd Comparative Reasoning: Unlocking comprehensive evaluations for LLM- as-a-judge.arXiv preprint arXiv:2502.12501(2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judg- ing LLM-as-a-Judge with MT-Bench and Chatbot Arena.Advances in Neural Information Processing Systems36 (2023), 46595–46623. arXiv:2306.05685

    Pith/arXiv arXiv 2023

  32. [32]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2024. WebArena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations. arXiv:2307.13854

    Pith/arXiv arXiv 2024

  33. [33]

    question requirement

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question an- swering benchmark on a hybrid of tabular and textual content in finance.arXiv preprint arXiv:2105.07624(2021). Lu et al. A Tool Curation Criteria This section spells out the criteria used to build the FinToolBench tool...

    Pith/arXiv arXiv 2021

  34. [35]

    Function name: {func_name}

  35. [36]

    <RAPIDAPI_KEY>

    Function parameters: - All input parameters: {param_str} - Additional fixed parameter: rapidapi_key: str = "<RAPIDAPI_KEY>"

  36. [37]

    Performance is not measured through operating margin

    The function must: - Use requests to send the HTTP request - Use X-RapidAPI-Key and X-RapidAPI-Host in headers - Return response.json() if possible, otherwise return response.text - Do not explain anything. Output pure Python code only - Parameter names must exactly match the keys in the parameter dictionary (case sensitive) Description: {desc} Parameter ...

    2022

  37. [2025]

    FinEval: A Chinese financial domain knowledge evaluation benchmark for large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Lingui...

    2025