REVIEW 5 major objections 6 minor 20 references
FinS-Pilot: A Benchmark for Online Financial RAG System
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read FinS-Pilot is a benchmark of 316 real queries from an online financial assistant that scores RAG systems on live and static data, ranking Chinese LLMs under retrieval settings.
desk verdict A useful benchmark idea compromised by a circular gold-standard construction for its content-based queries. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The benchmark itself is the central object: 316 queries derived from desensitized logs of an online financial assistant, with dual-channel retrieval of Tushare Pro API data for numerical questions and embedding-based text retrieval augmented by Bing search for content questions. Generation is performed by prompting multiple Chinese LLMs with task-specific prompts that incorporate retrieved documents, and judgment proceeds through a two-stage pipeline where an LLM (Doubao-1.5-pro) selects the most semantically relevant candidate answer, followed by manual review and correction by AI and finance experts to produce gold-standard answers; numerical ground truths are extracted manually from API responses with consensus. The two-level intent taxonomy, with 9 first-level and 62 second-level categories, is what enables fine-grained performance analysis across financial workflows.
What would settle it
Re-generate all gold-standard answers for content-based queries using a different LLM selector, such as DeepSeek-v3, then re-score the six models; if the relative ranking changes substantially, the original selection model biased the benchmark.
Extended reading notes
Core claim
The central claim is that FinS-Pilot fills a gap in financial RAG evaluation by combining real-world user queries with dynamic, real-time data sources and static text, organized by an operational intent taxonomy. The paper demonstrates this utility by evaluating six Chinese LLMs under three retriever configurations, finding that live API data is indispensable for numerical questions, external web search consistently improves content answers, and untuned BERT retrieval or no retrieval at all performs poorly. The benchmark ranks Xiaofa-1.0 as the best overall generator, with DeepSeek-v3, DeepSeek-R1, Doubao-1.5-pro, Moonshot-v1, and Baichuan-4 clustering below it, and the authors claim this demonstrates FinS-Pilot's effectiveness in identifying models suitable for financial applications.
Load-bearing premise
The gold-standard answers used to score the models are unbiased, even though the first pick of each content answer came from Doubao-1.5-pro, one of the very models the benchmark later ranks.
Editorial extensions
If this is right
- If FinS-Pilot is sound, financial RAG systems can be evaluated on authentic user queries with live data, closing the gap between static-report benchmarks and production conditions.
- The near-zero performance of non-retrieval baselines on numerical queries shows that real-time data integration is a requirement, not an option, for financial assistants.
- The consistent improvement under Bing retrieval over a private text corpus suggests that access to external fresh documents is a key lever for content quality.
- The benchmark's fine-grained intent taxonomy could let practitioners pinpoint which workflows a model fails, guiding pipeline optimization in production environments.
Reading between the lines
- The 62-intent taxonomy could be reused as a task scaffold for training or instruction-tuning financial assistants, not just for evaluation, though the paper does not propose this.
- Because numerical queries are tied to timestamped API calls, the benchmark could be re-run later with fresh market data to test models' temporal robustness and resistance to data contamination, a use the authors leave implicit.
- The strong showing of the closed-source in-house model Xiaofa-1.0 may reflect specialization on the target workflow, and a public re-analysis with additional open models would clarify whether the ranking generalizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. FinS-Pilot is a proposed benchmark for evaluating retrieval-augmented generation (RAG) systems in online financial applications. The benchmark is built from 316 real-world user queries logged by a financial assistant: 104 numerical queries whose answers are checked against real-time API data (Tushare), and 212 content-based queries whose answers are derived from retrieved text corpora. The queries are organized into a two-level intent taxonomy (9 first-level, 62 second-level categories). Gold answers for numerical queries are obtained by manual extraction from API data; gold answers for content-based queries are produced by having Doubao-1.5-pro select the most semantically aligned candidate answer from a set of LLM-generated candidates, followed by expert editing. The paper evaluates six Chinese LLMs (DeepSeek-v3, DeepSeek-R1, Baichuan-4, Doubao-1.5-pro, Moonshot-v1, and the authors' Xiaofa-1.0) under four retrieval settings (Base, Bing, Bert, Close) and reports that Xiaofa-1.0 performs best on most numerical and content-based metrics, while RAG-based retrieval improves over non-retrieval baselines.
Significance. If the benchmark construction were methodologically sound, FinS-Pilot would fill a real gap: existing financial QA benchmarks mostly use static reports, while production financial assistants need real-time data and RAG. The use of actual user logs, a fine-grained intent taxonomy, and a hybrid dynamic API/text retrieval setup are genuinely useful design choices. The numerical track is a well-motivated contribution, and the open release of code and data is a strength. However, the central comparative claim—that the benchmark identifies the most suitable models for financial applications—rests on the content-based evaluation, and that evaluation is compromised by a reference-construction loop involving one of the evaluated models. Additional problems with metric definition, reproducibility of dynamic data, and lack of statistical support further reduce the reliability of the reported conclusions. The benchmark has potential, but the evidence as presented does not support the paper's headline claims.
major comments (5)
- [Section 2.4 and Section 3.1, Table 1] The content-based gold-standard construction is circular with respect to the evaluated models. Section 2.4 states that Doubao-1.5-pro judges all candidate answers and selects the one 'most semantically aligned' as the sole candidate for the standard answer, which experts subsequently edit. Doubao-1.5-pro is also one of the generators evaluated in Section 3.1 and Table 1. Whenever Doubao-1.5-pro's own output is selected, the reference inherits Doubao's phrasing, and the ROUGE-L, BLEU, cosine similarity, and hallucination metrics—which compare against that reference—will be systematically inflated for Doubao relative to other models. The bias persists even after human editing because editing starts from the Doubao-selected text rather than from a model-independent source. The paper reports no analysis of how often Doubao's candidate was selected, no ablation removing Doubao from the selection step, and no independent reference construction. Consequently, the content-based results in Table 1, including the comparison between Xiaofa-1.0 and Doubao, cannot be interpreted as unbiased evidence of model capability.
- [Section 3.2, Table 1] The description of the 'maximum normalization' applied to Completeness (COM.) and Relevance (REL.) is self-contradictory: the text says the scores are linearly rescaled by a factor of 1/4 to a standardized [0,1] interval, yet maximum normalization would divide by the maximum, not by 4. No formula, raw-score range, or pre-normalization values are provided. Since COM. and REL. are used to support the claim that Xiaofa-1.0 'achieves best results in most metrics,' the missing definition makes Table 1 impossible to interpret or reproduce.
- [Section 2.4] The claim of '100% accuracy for time-sensitive numerical indicators' is unsupported. Manual extraction by a few experts with consensus is not a guarantee of ground truth, especially because the underlying API data are dynamic and can change between evaluation runs. The paper does not report inter-annotator agreement, validation against a second authoritative source, or any error analysis for the numerical gold standard. Without such evidence, the 100% accuracy assertion should be removed or substantially qualified.
- [Section 2.2 and Section 2.5] The benchmark's dynamic data sources are not versioned, which undermines reproducibility. Section 2.2 states that the text database 'will be updated regularly' and that real-time market data are obtained through the Tushare API; Section 2.5 refers to 'some APIs to retrieve stock price data.' Because numerical answers and the corresponding evaluation depend on the specific data available at a given time, two runs of the benchmark at different dates may yield different gold answers and different model scores. The paper should provide date-stamped snapshots of API responses and corpus versions, or clearly specify how to obtain them, so that comparisons over time are meaningful.
- [Section 3.2, Table 1 and Figure 2] The reported experimental results lack any uncertainty quantification. Figure 2 and Table 1 give point estimates only, with no confidence intervals, standard deviations, or significance tests. Several comparative gaps are small (e.g., Base COM for DeepSeek-v3 is 0.2152 versus Xiaofa-1.0's 0.2282; Bing REL ranges from 0.9333 to 0.9792), and the per-category query counts (104 numerical and 212 content-based) are modest. Without error bars or statistical testing, the claim that Xiaofa-1.0 'achieves best results in most metrics' is not established.
minor comments (6)
- [Section 2.4] There is a typo: 'proficient in data processingy' should be 'proficient in data processing.'
- [Section 1] The sentence 'resulting the gap between their evaluation results and the performance of models in the online environment' is ungrammatical; 'resulting the gap' should be 'resulting in a gap.'
- [Section 4] The reference to 'LAMBDA' in Section 4 is a misspelling of 'LAMBADA'; please correct it.
- [Section 2.2] The retrieval configuration is underspecified: the embedding-based dense retriever ('Base') and the 'Bert' retriever are named but no model names, embedding dimensions, top-k settings, or corpus sizes are given, making it difficult to reproduce the retrieval conditions.
- [Section 3.2] The sentence 'We process 104 numerical queries DeepSeek-v3 without any reference data, yields zero accuracy' is missing a word (likely 'with') and should be rewritten for clarity.
- [Figure 2] Figure 2 would benefit from error bars or per-model variance information, especially because the accuracy differences among several models are small (e.g., 0.740 vs. 0.760).
Circularity Check
Content-based gold answers are seeded by Doubao-1.5-pro's own selected output, so a large fraction of the benchmark's scoring is partially self-referential.
-
self definitional
[Section 2.4 (Judgment); evaluated in Section 3.1 and Table 1]
"For content-based queries, we first use a large language model (Doubao-1.5-pro) to judge the relevance between all candidate answers and user queries, and selects the most relevant answer as the only candidate for the standard answer. Subsequently, we first remove the content in the answer that is irrelevant to the user query, and then invites experts in the fields of artificial intelligence and finance to review, correct the wrong information therein, and finally obtain the standard answer for each query."
The gold-standard answer for each content-based query is, by construction, the Doubao-1.5-pro candidate judged 'most semantically aligned', later edited by experts. Section 3.1 then evaluates Doubao-1.5-pro on the same 212 content-based queries, and Table 1 reports its ROUGE-L/BLEU/cosine/hallucination/completeness/relevance scores against these references. Because the reference text is seeded by Doubao's own output, Doubao's n-gram and semantic similarity to the reference is inflated relative to models whose candidates were not selected; the hallucination metric ('whether the response contains information conflicting with references') is also trivially easier for Doubao, since the reference does not conflict with Doubao's selected statements.
full rationale
The numerical-query track is independent: ground-truth values are manually extracted from structured API responses, and Xiaofa-1.0's 91.5% accuracy is a real result. The self-citations (refs [2,3,16,19]) are contextual and not load-bearing. The central circularity is confined to the content-based track but affects 212/316 queries, the part of the benchmark that uses LLM-judged 'best answer' references, so the paper's broadest claim—that FinS-Pilot 'identifies models suitable for financial applications'—is compromised. There are also non-circularity concerns: unversioned dynamic data, absence of error bars, ad hoc normalization of COM./REL., and evaluating an author-affiliated model (Xiaofa-1.0, 'Ours') without explicit conflict-of-interest discussion. These do not count toward the circularity score but compound risk. Score 7 reflects partial circularity by construction, not full equivalence, because experts review the selected answers and the numerical track remains independent.
Assumptions & free parameters
free parameters (1)
- COM/REL scaling factor =
1/4
assumptions (4)
- domain assumption LLM-based metrics (hallucination, completeness, relevance) are valid measures of answer quality.
- domain assumption Manual expert review ensures accurate gold-standard answers.
- domain assumption The user logs are representative of real financial assistant usage.
- domain assumption Tushare API data is accurate and current.
Cite this review
Pith. "Pith review of FinS-Pilot: A Benchmark for Online Financial RAG System." pith.science (2026). https://pith.science/paper/COW6X5LY
@misc{pith2026250602037,
author = {Pith},
title = {Pith review of: FinS-Pilot: A Benchmark for Online Financial RAG System},
year = {2026},
howpublished = {\url{https://pith.science/paper/COW6X5LY}},
note = {Machine review of arXiv:2506.02037}
}
read the original abstract
Large language models (LLMs) have demonstrated remarkable capabilities across various professional domains, with their performance typically evaluated through standardized benchmarks. In the financial field, the stringent demands for professional accuracy and real-time data processing often necessitate the use of retrieval-augmented generation (RAG) techniques. However, the development of financial RAG benchmarks has been constrained by data confidentiality issues and the lack of dynamic data integration. To address this issue, we introduce FinS-Pilot, a novel benchmark for evaluating RAG systems in online financial applications. Constructed from real-world financial assistant interactions, our benchmark incorporates both real-time API data and text data, organized through an intent classification framework covering critical financial domains. The benchmark enables comprehensive evaluation of financial assistants' capabilities in handling both static knowledge and time-sensitive market information.Through systematic experiments with multiple Chinese leading LLMs, we demonstrate FinS-Pilot's effectiveness in identifying models suitable for financial applications while addressing the current gap in specialized evaluation tools for the financial domain. Our work contributes both a practical evaluation framework and a curated dataset to advance research in financial NLP systems. The code and dataset are accessible on GitHub.
Figures
Reference graph
Works this paper leans on
-
[1]
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023. Benchmarking Large Language Models in Retrieval-Augmented Generation. CoRR abs/2309.01431 (2023). https://doi.org/10.48550/ARXIV.2309.01431 arXiv:2309.01431
-
[2]
Yiqun Chen, Lingyong Yan, Weiwei Sun, Xinyu Ma, Yi Zhang, Shuaiqiang Wang, Dawei Yin, Yiming Yang, and Jiaxin Mao. 2025. Improving retrieval- augmented generation through multi-agent reinforcement learning. arXiv preprint arXiv:2501.15228 (2025)
arXiv 2025
-
[3]
Yiqun Chen, Erhan Zhang, Lingyong Yan, Shuaiqiang Wang, Jizhou Huang, Dawei Yin, and Jiaxin Mao. 2025. MAO-ARAG: Multi-Agent Orchestration for Adaptive Retrieval-Augmented Generation. arXiv preprint arXiv:2508.01005 (2025)
arXiv 2025
-
[4]
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al
-
[5]
Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
arXiv 2018
-
[6]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)
arXiv 2025
-
[7]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300 (2020)
arXiv 2020
-
[8]
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944 (2023)
arXiv 2023
Show all 20 references
-
[9]
Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. LatestEval: addressing data contamination in language model evaluation through dynamic and time- sensitive test construction. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Confe...
2024
-
[10]
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81
2004
-
[11]
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)
2024 arXiv
-
[12]
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018. 1941–1942
2018
-
[13]
Lang Mei, Siyu Mo, Zhihan Yang, and Chong Chen. 2025. A survey of multimodal retrieval-augmented generation. arXiv preprint arXiv:2504.08748 (2025)
2025 arXiv
-
[14]
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meetin...
2016
-
[15]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318
2002
-
[16]
Yiding Sun, Feng Wang, Yutao Zhu, Wayne Xin Zhao, and Jiaxin Mao. 2024. An integrated data processing framework for pretraining foundation models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2713–2718
2024
-
[17]
Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 (2018)
2018 arXiv
-
[18]
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. 2024. LiveBench: A Challenging, Contamination-Free LLM Benchmark. arXiv preprint arXiv:2406.19314 (2024)
2024 arXiv
-
[19]
Yutao Zhu, Kun Zhou, Kelong Mao, Wentong Chen, Yiding Sun, Zhipeng Chen, Qian Cao, Yihan Wu, Yushuo Chen, Feng Wang, et al. 2024. Yulan: An open-source large language model. arXiv preprint arXiv:2406.19853 (2024)
2024 arXiv
-
[2021]
arXiv preprint arXiv:2109.00122 (2021)
Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122 (2021)
2021 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.