{"id":"5b82d1d4-8084-42f9-8269-59e148466531","arxiv_id":"2506.06622","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"QuantMCP connects LLMs to financial data APIs via MCP, but its claims of reduced hallucination rest on a single anecdotal case study.","lead":"QuantMCP is a framework that lets large language models pull live financial data from APIs like Wind and yfinance through the Model Context Protocol, so answers are based on real numbers instead of the model's memory. The paper describes a prototype and one example, but provides no systematic test of whether this reduces errors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The guarantee is not end-to-end: QuantMCP verifies only the data source, so an LLM mis-parameterized tool call returns authoritative-looking wrong data and no pipeline step catches it.","rationale":"The load-bearing condition is that the LLM's translation of intent into tool calls is reliable enough to ground a guarantee. QuantMCP's design verifies only the provenance of the data, not the semantics of the request or the post-retrieval computation. Since Section 3.1 puts the LLM in charge of ticker/date/field extraction and Section 4.2 additionally has the LLM compute averages, there are two ungrounded links in the chain. The pre-retrieval link is more dangerous: a wrong parameter sends a valid API request that returns real, plausible, authoritative-looking data for the wrong entity or period, and no component in the framework can detect this. The paper reports a single successful walkthrough and releases no code or benchmark, so the central empirical claim is unsupported. This does not mean the MCP architecture is wrong; it means the 'guarantees' language in Section 3.2 is not justified. A systematic tool-call accuracy benchmark and output verification would be the minimal evidence needed. While such evidence could upgrade the paper, the current REJECT verdict is appropriate.","tokens_in":7316,"tokens_out":8034,"duration_ms":84888,"concrete_test":"Build a benchmark of 50-100 natural-language financial queries with ground-truth MCP calls (ticker, start/end date, field list, options), including ambiguous tickers (e.g., 300750.SZ vs 300750.HK), relative dates ('last quarter', 'Q1 2024'), and multi-entity requests. Run QuantMCP with DeepSeek-V3 (and at least one other LLM) and compare each emitted tool call to ground truth by exact match; also recompute every analytical output (averages, changes, ratios) against a reference implementation. If exact-match tool-call accuracy is below 100% or any downstream calculation diverges, the paper's 'guarantees data integrity' claim is falsified; report per-error-class rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 promises that grounding in external APIs 'mitigates hallucination and guarantees data integrity.' That guarantee holds only if every LLM-generated step is correct, but QuantMCP neither verifies nor constrains them. The weakest step is the natural-language-to-tool translation in Section 3.1: the LLM is solely responsible for ticker resolution, date-range conversion, and field-code mapping. If it emits a wrong ticker or field, the MCP server executes the call and returns real API data that is not what the user asked for. Because the response is genuinely from Wind/yfinance, the error is indistinguishable from a correct retrieval and is then passed to the analysis stage as 'verified.' The paper's only demonstration (Section 4.2) is one successful invocation; there is no benchmark, no adversarial query set, no measurement of tool-call accuracy, and no error handling for mismatched parameters. A second instance of the same gap is visible in Section 4.2 itself: after retrieval, the LLM computes averages and interprets trends, and these analytical outputs are not API data and are never independently checked. Thus the framework grounds data provenance, not the correctness of the overall answer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QuantMCP, a framework that uses the Model Context Protocol (MCP) to let a large language model (LLM) retrieve financial data from external APIs (Wind, yfinance, etc.) through a set of registered tools, and then use that retrieved data for downstream analysis. The architecture has three layers: an LLM-based natural language interface, an MCP server that exposes and invokes financial data tools, and a final LLM stage that performs analysis on the grounded data. The paper also describes a prototype implementation using DeepSeek-V3 and Wind, and illustrates the workflow with a single query about CATL stock data for Q1 2024, including the computation of average closing price and turnover rate. The central claim is that grounding LLM responses in API-retrieved data 'mitigates hallucination and guarantees data integrity.'","tokens_in":7543,"tokens_out":2727,"duration_ms":30536,"significance":"If the central claim were convincingly demonstrated, QuantMCP would be a useful contribution to applied LLM systems for finance: it addresses a real problem (LLM hallucination on financial facts), uses a timely standardization mechanism (MCP), and lowers the barrier to querying professional financial data. The paper is clearly written and the architecture description is plausible. However, the evidence presented is limited to a single successful invocation with no baseline, no error analysis, and no quantitative evaluation of tool-call accuracy, hallucination rates, or analysis correctness. The framework itself is not fundamentally flawed, but the claimed guarantees are not currently supported by the manuscript's experiments.","major_comments":[{"comment":"The central claim that QuantMCP 'mitigates hallucination and guarantees data integrity' (abstract, Section 3.2) is not supported by the evaluation. Section 4.2 reports a single successful query execution, with no error rate, no comparison against an LLM without grounding, no adversarial or out-of-distribution queries, and no quantitative metric for retrieval or computation accuracy. Because this is the only demonstration, the paper does not establish that the framework reliably produces correct answers across realistic queries. A revision should add a systematic evaluation: a benchmark of diverse natural language queries, measurement of tool-call parameterization accuracy (ticker, date range, field codes), comparison with a baseline LLM on the same questions, and an analysis of hallucination/error cases.","section":"Section 4.2"},{"comment":"The claim that grounding in external APIs 'guarantees data integrity' overstates what the architecture ensures. As described in Section 3.1, the LLM is solely responsible for translating natural language into correct MCP tool calls. If the model emits a wrong ticker, an incorrect date range, or a misnamed field, the MCP server will faithfully execute the call and return authoritative-looking data that does not answer the user's question. No component in the pipeline verifies that the retrieved data matches the user's intent. This is not a hypothetical concern: it is the standard failure mode of LLM tool use. The paper should either add a verification/feedback mechanism for tool-call correctness or explicitly scope the claim to 'data provenance is verified, not semantic correctness of retrieval.'","section":"Section 3.2"},{"comment":"The demonstration claims that the LLM computes the average closing price and average turnover rate, but there is no independent check of these computed values. The paper does not report the exact numbers, compare them against ground-truth calculations, or assess the LLM's arithmetic reliability. Since analytical outputs are not API data, they are not automatically grounded. The manuscript should include a verification of any derived metrics, or explicitly separate 'API-retrieved data' from 'LLM-computed analytics' and discuss the associated error modes.","section":"Section 4.2"}],"minor_comments":[{"comment":"The phrase 'guarantees data integrity' is too strong given the analysis above; consider replacing it with something like 'verifies data provenance' or 'reduces the risk of hallucinated data points.'","section":"Abstract / Section 3.2"},{"comment":"The figure contains a typo: 'Financial Database (e.g., yfinance, Widely, public or private)' should presumably be 'Windy' or 'Wind'; please correct.","section":"Figure 1"},{"comment":"The example JSON snippet has mismatched brackets: it opens with '[' but the inner dictionaries use '}' without a closing ']' for the list. This makes the example harder to parse; fix the formatting.","section":"Section 4.2"},{"comment":"The manuscript describes a prototypical implementation but does not provide a link to code or a repository. For reproducibility, please consider releasing the MCP server implementation and the example configurations.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline system paper. The architecture is sensible and the MCP angle is timely, but the evaluation is far too thin for the strength of the claims. If the authors add a solid benchmark and temper the 'guarantee' language, the paper could become acceptable for an applied venue. The self-citation (Zeng 2024) is not a concern for the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a routine MCP integration paper dressed up as a framework, with one worked example and no real evaluation. The architecture is sensible, and the description is honest about its prototype status, but the central promise of 'guaranteeing data integrity' is not supported.\n\nWhat's actually new: not much. Applying MCP to financial APIs is a direct extension of published tool-use work and the paper's own references (FinCon, MCP surveys). The design choices are reasonable: server-side credential management, JSON normalization, tool manifests. The Wind API example is concrete, and the figures are helpful.\n\nWhere it falls down: the evaluation is anecdotal. There is one successful query, no baseline, no adversarial set, no error analysis, no code release. The claimed guarantee is end-to-end, but the verification stops at the data source. The LLM is solely responsible for mapping natural language to tickers, date ranges, and field codes. If it gets a parameter wrong, the MCP server pulls real data from Wind/yfinance that isn't what the user asked for. That authoritative-looking wrong data then feeds the analysis stage. So QuantMCP grounds provenance, not correctness. The paper itself even notes the LLM's analytical outputs (averages, trends) are not independently checked, which undercuts the 'guarantees' language in Section 3.2.\n\nThe stress-test note is right on the mark. The reader's weakest assumption is actually the load-bearing one.\n\nWho's this for? Practitioners or students looking for a clean example of an MCP-based financial data agent. As a research contribution it's thin; the claim of novelty is overstated. The self-citation to Zeng (2024) is irrelevant to the core mechanism and harmless.\n\nRecommendation: I'd desk reject in its current form. The architecture is worth a blog post, not a full paper, unless the authors add a proper evaluation of tool-call accuracy under realistic queries, including mis-parameterization cases. If this comes in with that, it could be a useful empirical note.","headline":"Routine MCP integration with one anecdote, no evaluation, and an overstated guarantee.","tokens_in":8042,"tokens_out":2134,"would_cite":false,"duration_ms":21924,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QuantMCP grounds LLM financial answers in live API data, so numbers come from Wind or yfinance rather than from the model's memory.","keywords":["large language models","hallucination mitigation","model context protocol","financial data APIs","grounded data retrieval","natural language interface","tool invocation","financial analysis"],"falsifier":"Run a set of ambiguous financial queries through QuantMCP—company names that map to multiple tickers, relative dates like 'last quarter', and field codes that vary by provider—then compare the MCP tool-call parameters against human-annotated ground truth. If a meaningful share of calls retrieve data for the wrong security, date window, or field, the claim that grounding guarantees data integrity fails.","tokens_in":7136,"feed_emoji":"📈","tokens_out":9230,"duration_ms":78987,"temperature":0.7,"pith_summary":"QuantMCP sets out to show that an LLM can be made reliable for financial data tasks by never letting it answer factual market questions from memory. The framework routes each request through a Model Context Protocol server that translates the user's natural language into a call to an authoritative financial API such as Wind or yfinance, returns structured JSON to the model, and only then lets the model summarize or analyze. The intended effect is to turn LLMs from a source of plausible but often wrong financial facts into an interface to verifiable data, while preserving the conversational interface that makes the tools accessible. The paper supports this with a prototype using DeepSeek-V3-0324 wired through an MCP server to Wind, and an illustrative query about CATL's 2024 first-quarter daily closing price, price-to-book ratio, and turnover rate.","feed_headline":"QuantMCP grounds LLM financial answers in live API data","feed_subtitle":"Plain-language queries become verified Wind and yfinance calls; the model analyzes real numbers, not recalled ones","key_machinery":"The Model Context Protocol (MCP) server is the central mechanism: a standardized gateway that exposes financial data functions as discoverable tools with declared parameters, invokes the appropriate Python API adapter (such as WindPy or yfinance), manages API credentials server-side, and returns structured JSON to the LLM. The work it does is to turn the LLM's natural-language intent into a parameterized, authenticated data request, so the model's answer is built on externally sourced facts rather than memory.","core_discovery":"The paper claims that routing every specific financial data point through an MCP server—rather than letting the LLM answer from parametric memory—mitigates hallucination and guarantees data integrity in finance. The LLM's job is restricted to understanding intent, choosing a tool, and assembling parameters; the data itself comes from the API, so the user sees numbers with external provenance. The paper then argues that the same architecture unlocks analysis: with grounded data in hand, the LLM can compute averages, identify trends, compare benchmarks, and generate explanations it could not safely produce otherwise. The demonstration shows DeepSeek-V3-0324 correctly parsing a query about Contemporary Amperex Technology's first-quarter 2024 close, price-to-book, and turnover, retrieving the data through Wind, and returning both retrieved values and calculated averages.","pith_inferences":["Editorial inference: the paper's failure mode shifts but does not disappear—if the LLM mis-parameterizes a tool call with the wrong ticker, date window, or field code, the wrong data arrives with the same authoritative provenance, and the paper does not test this step.","Editorial inference: a natural extension is to benchmark the natural-language-to-tool-call step against a human-annotated set of ambiguous financial queries, which would give a quantitative test of the data-integrity guarantee.","Editorial inference: the same MCP-grounded pattern could transfer to other high-stakes domains where the model should never invent facts, such as medical records, legal dockets, or scientific databases, provided a comparable API layer exists."],"forward_implications":["Financial queries answered through QuantMCP carry data provenance: every number traces to a named API call at query time, so a user can audit where a figure came from.","Non-programmers can request professional datasets in plain language and get structured, analysis-ready data without learning Wind or yfinance call syntax.","The same MCP server can expose additional Python-accessible financial sources, so the framework scales beyond the prototype's Wind integration.","Analysis tasks such as averages, comparisons, trend detection, and visualization are performed on externally sourced data, making the reasoning less likely to be built on fabricated inputs.","Credential handling stays server-side, so users and models never see API keys, reducing the attack surface for financial data access."],"supporting_citations":[{"why":"supplies the Model Context Protocol that QuantMCP builds on for standardized tool discovery and invocation.","marker":"Singh et al., 2025"},{"why":"documents the LLM hallucination phenomenon that QuantMCP claims to mitigate by grounding data in external APIs.","marker":"Huang et al., 2025"},{"why":"technical report for DeepSeek-V3, the model used in the QuantMCP prototype and case study.","marker":"Liu et al., 2024"},{"why":"establishes the general LLM capabilities that the natural-language interface layer assumes for intent recognition and tool use.","marker":"Achiam et al., 2023"},{"why":"surveys LLM applications in finance, motivating the analytical tasks QuantMCP enables on grounded data.","marker":"Li et al., 2023"},{"why":"shows LLM systems supporting financial decision-making, the downstream use case QuantMCP extends to verified data.","marker":"Yu et al., 2024"}],"fun_headline_variants":["QuantMCP grounds LLM answers in verified live data","Ask in plain English, get verified financial numbers","LLMs stop guessing—QuantMCP fetches real data","Bridging conversational AI to real financial data","QuantMCP: from natural language to grounded analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM reliably turns natural-language intent into correct tool calls—right ticker, date range, and field codes—because any error at that step would deliver an authoritative-looking wrong answer, and the paper does not test that step.","fun_headline_variants_meta":{"raw":{"variants":["QuantMCP grounds LLM answers in verified live data","Ask in plain English, get verified financial numbers","LLMs stop guessing—QuantMCP fetches real data","Bridging conversational AI to real financial data","QuantMCP: from natural language to grounded analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3113,"prompt_tokens":897,"completion_tokens":2216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":2139}},"tokens_in":513,"tokens_out":2216,"duration_ms":18747,"temperature":1.0,"reasoning_tokens":2139,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:52:10.177937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a set of ambiguous financial queries through QuantMCP—company names that map to multiple tickers, relative dates like 'last quarter', and field codes that vary by provider—then compare the MCP tool-call parameters against human-annotated ground truth. If a meaningful share of calls retrieve data for the wrong security, date window, or field, the claim that grounding guarantees data integrity fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the Model Context Protocol that QuantMCP builds on for standardized tool discovery and invocation."}],"review_version":1}