{"id":"3e7d17b8-2ac8-4edd-a732-b805f38b34da","arxiv_id":"2506.08757","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A function-calling LLM that selects pre-approved SQL functions outperformed direct NL-to-SQL in human-evaluated correctness for nuclear plant data retrieval.","lead":"This paper describes a system that uses a function-calling large language model to retrieve nuclear plant data by selecting from pre-approved SQL functions, instead of generating SQL directly. It reports that this function-based approach scored higher on human expert correctness checks than a direct NL-to-SQL baseline, while LLM-based quality metrics were nearly identical.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human-evaluated accuracy advantage is uninterpretable without the test set; the paper never states how evaluation queries were selected or whether they were drawn from the pre-approved function library, and no maintainability metric is reported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation queries may not be representative and could be drawn from the function library's own examples. Reading the paper in good faith, the method is a plausible engineering pattern, and using GPT-4o for both systems is a sensible attempt to isolate the effect of the approach. However, Section 4.5 provides only aggregate scores and a figure, with no test-set size, selection procedure, or query inventory. Without that information, the human-evaluated correctness advantage cannot be separated from test-design artifacts. The maintainability claim is also completely unevidenced, but the accuracy comparison is the more immediate load-bearing pillar. I agree with the reader that the verdict should remain CONDITIONAL: the method could be convincingly demonstrated with a more complete evaluation, but the paper as written does not fully support its central claim. No change to the reader's verdict is needed.","tokens_in":7245,"tokens_out":1502,"duration_ms":18689,"concrete_test":"Publish the full evaluation query set along with the exact function-library version used, and independently score both systems on a held-out set of queries sampled from operational user logs that were not available during function-library construction, reporting per-query correctness and sample size per condition. Additionally, report a concrete maintainability measure, such as the time or cost to add one new query type to each system, to substantiate the maintainability claim in Section 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the function-calling method improves accuracy and maintainability over NL-to-SQL rests entirely on Section 4.5, yet that section specifies no sample size, no test set description, and no query-selection procedure. Section 4.5.2 reports higher average human-evaluated correctness scores and Figure 5 shows per-question scores, but without knowing whether those queries were representative of real operational use or were derived from the same pre-approved function library that defines the system, the comparison may be circular. If the test queries were the function library's own exemplars, the function-calling approach would trivially outperform NL-to-SQL because each query maps to a curated, expert-validated function, while the NL-to-SQL system must generalize from schema knowledge alone. The maintainability claim in Section 5 is likewise unsupported: no metric, function-library size, or update-cost comparison is provided. The methodology itself is reasonable and the same LLM was used for both systems, which is a strength, but the evidence presented does not yet establish the stated superiority.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a function-calling LLM architecture for retrieving operational data from nuclear plant databases, as an alternative to direct NL-to-SQL generation. The system routes user queries to sub-agents that select among pre-approved, expert-validated functions, each encapsulating fixed SQL logic. The authors compare this approach with a non-function-calling NL-to-SQL system using LLM-computed metrics (Answer Relevance, Relevance, Faithfulness) and human SME-evaluated correctness, reporting higher correctness for the function-calling approach. The conclusion claims improvements in both accuracy and maintainability.","tokens_in":7449,"tokens_out":2363,"duration_ms":28633,"significance":"If the reported results are robust, the function-calling paradigm is a practically valuable contribution to data retrieval in safety-critical domains: it reduces the risk of unvalidated SQL generation, enables expert oversight, and is architecturally transparent. A clear strength is that the same LLM (GPT-4o) is used in both systems, which isolates the effect of the function-calling wrapper. The human SME evaluation is also a step in the right direction, as it provides a non-LLM ground truth for correctness. However, the significance is presently limited by severe under-specification of the evaluation, and the maintainability claim has no supporting measurement at all.","major_comments":[{"comment":"The evaluation section provides no test set size, no query-selection procedure, and no error bars or statistical tests. Figures 3–5 report only average scores and a qualitative per-question comparison. Without knowing the number of queries, how they were chosen, and whether they overlap with the function library's design examples, the human-evaluated correctness advantage reported in Section 4.5.2 cannot be interpreted as evidence of general superiority. Please report the number of queries, the selection/construction process, and the per-query results with a paired significance test or confidence intervals.","section":"4.5"},{"comment":"The central claim of improved system maintainability is not supported by any measurement in the manuscript. No function-library size, update-cost comparison, maintenance metric, or qualitative maintenance-log analysis is reported. Either add a quantitative or well-defined qualitative maintainability evaluation (for example, time and effort required to add a new query type, or the frequency of function updates) or explicitly frame maintainability as a design hypothesis rather than an evaluated result.","section":"5"},{"comment":"The LLM-computed metrics (Answer Relevance, Relevance, Faithfulness) are not described with respect to which model computed them, the prompt design, or any validation against human judgments. Since the system under test is the same model family (GPT-4o), these metrics are likely to reflect the LLM's own response-generation bias rather than an independent quality assessment. The manuscript should state the metric model, provide evidence that the metric prompts are reliable, and ideally calibrate the LLM metrics against a human-annotated subset.","section":"4.5.1"},{"comment":"The potential circularity of the evaluation is not addressed. The function library is built by the authors to cover known use cases (Section 3.1), and the test queries are not disclosed. If the evaluation questions were drawn from the same use cases used to design the function library, the function-calling approach would trivially outperform NL-to-SQL, which must generalize from the schema alone. Please disclose the relationship between the test set and the function-library design, and ideally include held-out query types that are not directly supported by an existing function.","section":"4.5 and 3.1"}],"minor_comments":[{"comment":"The abstract contains a typographical artifact: \"dat a\" appears with an extra space in the phrase \"Retrieving operational data from nuclear power plants.\" Please correct it.","section":"Abstract"},{"comment":"The text says \"A comprehensive and quantitative assessment of function-calling performance is documented in the literature review section below,\" but the literature review appears in the Introduction (Section 1), not below Section 2.1. Please fix the cross-reference.","section":"2.1"},{"comment":"Figure 5 is described as showing \"detailed scores per question,\" but the manuscript does not include the per-question data in a machine-readable table or appendix. Adding the per-question scores, or at least the number of questions and the distribution of scores, would allow readers to verify the reported advantage.","section":"4.5.2"},{"comment":"The reference formatting is inconsistent: some entries include author lists, some include only a title and URL, and the access-date style varies. Please unify the reference style according to the conference template.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short conference paper, and the evaluation under-specification may partly reflect length constraints. However, the main claims—accuracy improvement and maintainability improvement—are both load-bearing and neither is currently established to an acceptable standard. The maintainability claim is entirely unmeasured, and the accuracy claim rests on an opaque test set. The architecture and methodology are reasonable, and the authors clearly have a real deployed system, so I believe the issues are fixable with additional disclosure and modest additional analysis; hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on arXiv:2506.08757. It's an applied systems paper from Ontario Power Generation comparing two ways to let plant staff query legacy nuclear databases in natural language: direct NL-to-SQL with RAG, and a function-calling approach where the LLM selects among pre-approved, expert-validated functions that wrap SQL. The core idea is sound and not new in general—function calling is established—but the application to this regulated setting with a curated function library is a reasonable contribution.\n\nWhat the paper does well: the architecture is described clearly, the authors deliberately used the same GPT-4o model for both systems to isolate the method, and they are candid about limitations—smaller models struggle with function calling, tool overload degrades selection, chained queries aren't supported yet. The human-evaluated correctness scores by SMEs give some independent signal that the function-calling system produces more acceptable answers; that is the strongest piece of evidence.\n\nThe soft spots are real and concentrated in the evaluation. There is no test set size, no description of how the evaluation queries were selected, no sample of queries shown, no error bars or statistical tests. The maintainability claim in the conclusion is asserted but never measured; no function library size, no update-cost comparison. The LLM-computed metrics are similar for both systems and those are produced by the same model family, so they don't discriminate. The circularity concern is legitimate: if the test questions were drawn from the use cases the function library was built to cover, the comparison is partly baked in. The paper never tells us. I don't think the human SME scores are worthless—they do favor the function-calling system—but without knowing the test distribution, the magnitude of the claimed improvement is uninterpretable.\n\nWho gets value from this paper: practitioners designing LLM front-ends for legacy databases in safety-constrained industries. It's a useful case study of what breaks in practice (jargon, retries, agent routing). It is not a rigorous research contribution; the evaluation would need serious strengthening to support the stated claims.\n\nI'd send it to peer review because the systems design is coherent and the domain is under-served, but I would make the authors disclose the test set, sample size, selection procedure, and add at least one objective maintainability measure. As is, I'd treat the accuracy claim as plausible but unproven, and the maintainability claim as a proposal.\n\nRecommendation: engage with it if you're working on enterprise NL-to-SQL with legacy schemas; otherwise a skim suffices.","headline":"A sensible applied pattern with honest engineering detail, but the evaluation is too sparse to back the accuracy claim and the maintainability claim is unmeasured.","tokens_in":7942,"tokens_out":2089,"would_cite":false,"duration_ms":22776,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Function-calling LLMs that select pre-approved SQL functions beat direct natural-language-to-SQL generation for nuclear plant queries.","keywords":["function-calling LLM","NL-to-SQL","nuclear plant data retrieval","legacy databases","multi-agent workflow","SQL validation","retrieval-augmented generation","maintainability"],"falsifier":"Ask plant operators who never saw the function library to write a fresh set of queries, including some request types the library was not designed for, and have SMEs score both systems blind; if the function-calling system no longer outperforms the NL-to-SQL baseline on human-evaluated correctness, the reported advantage is an artifact of overlapping test questions and functions.","tokens_in":7077,"feed_emoji":"⚛️","tokens_out":6740,"duration_ms":70204,"temperature":0.7,"pith_summary":"The paper argues that natural-language-to-SQL generation is the wrong interface for critical nuclear plant data retrieval because users cannot validate the generated SQL and the legacy databases are complex. It proposes instead a function-calling system in which an LLM routes a user's question to predefined, expert-validated functions that encapsulate the only SQL queries allowed. The authors report that human-scored correctness is notably higher for this function-calling method than for a comparable system that generates SQL directly, while LLM-computed relevance and faithfulness scores are nearly identical. If the finding holds, organizations with safety-critical legacy databases can get the flexibility of natural-language querying without surrendering control of the underlying SQL. The trade-off is an upfront and ongoing cost of building and maintaining the function library.","feed_headline":"Pre-approved functions beat freehand SQL for nuclear data queries","feed_subtitle":"Expert-validated SQL inside function calls lifted human-scored correctness and kept LLM autonomy in check.","key_machinery":"The load-bearing object is the function library: a set of pre-approved, purpose-specific functions, each encapsulating reviewed and optimized SQL for a common request type such as work orders or stock levels. Around it sits a multi-agent workflow in which a main agent decides whether a query needs data retrieval, routes it to the appropriate sub-agent, and the sub-agent selects and executes one of its functions, with typed schema validation on every parameter, constrained decoding for structured outputs, automatic retries when the wrong function or agent is chosen, and full logging of each step. The comparison baseline replaces this library with a retrieval-augmented generator that extracts intent, retrieves example queries, validates table and field names against a schema store, and generates new SQL when no example matches. The key contrast is that the function-calling method moves all SQL authoring out of generation time and into expert review time.","core_discovery":"On its own terms, the paper's central discovery is that the reliability bottleneck in NL-to-SQL over legacy plant databases is not the LLM's language understanding but the unrestricted generation of SQL, and that replacing that step with function selection removes most of the failure modes. In the evaluated system, the function-calling workflow routes queries through domain-specific sub-agents that can only invoke pre-approved functions; the non-function-calling baseline uses intent extraction, retrieval of example queries, and schema validation to build SQL from scratch. Human experts scored the function-calling outputs substantially higher, and question-by-question results show the baseline frequently produced entirely incorrect answers while the function-calling method was consistently correct or partially correct. The paper presents this as evidence that constraining SQL to validated functions improves accuracy and maintainability, at the cost of building the function library, and notes that NL-to-SQL tools can help generate initial function code for experts to validate.","pith_inferences":["A reader might infer that the accuracy gap would shrink on queries whose intent lies outside the existing function library, because the baseline can generate new SQL while the function-calling system is limited to pre-approved capabilities.","The maintainability claim carries a hidden cost: if real operator questions invent new patterns faster than experts can add functions, library upkeep could outweigh the savings in validation effort.","The paper's evidence that function-selection accuracy drops as tool count grows suggests a concrete test: measure selection accuracy on the same plant queries with and without retrieval-based filtering of the function library."],"forward_implications":["Operators can ask natural-language questions while the system only executes SQL that experts have already reviewed, reducing the risk of hallucinated or unsafe queries.","The upfront function-library cost can be partly offset by using NL-to-SQL tools to draft initial function code, with experts concentrating on validation rather than creation.","Systems with fewer than 70-billion-parameter models will struggle to combine conversation and function calling, so deployments should budget for larger models or specialized ones.","Retrieval-based filtering of the available functions should improve selection accuracy, since performance degrades when the LLM is offered too many tools.","Handling chained queries that require multiple function calls in sequence is not supported in the current version and is planned as future work with reasoning models."],"supporting_citations":[{"why":"Supplies the guidance that 70B+ parameters are recommended when conversation and function calling must coexist, which shapes model selection.","marker":"[1]"},{"why":"Provides the structured-outputs technique, constrained decoding, that the system relies on to enforce JSON schemas.","marker":"[2]"},{"why":"Establishes the function-calling leaderboard used to pick a frontier model for both evaluated systems.","marker":"[3]"},{"why":"Shows that small models can approach top-tier function-calling performance after targeted fine-tuning, informing the scalability discussion.","marker":"[4]"},{"why":"Reports that function-selection accuracy drops when the number of tools grows, motivating dynamic filtering strategies.","marker":"[7]"},{"why":"Introduces document-retrieval-based tool filtering that reduces the function set the LLM must consider.","marker":"[8]"}],"fun_headline_variants":["Function-calling LLMs beat NL-to-SQL for nuclear data","Pre-approved functions outperform freehand SQL in plants","Expert-validated functions lift nuclear query accuracy","Nuclear data retrieval: function calls trump raw SQL","LLM function calls outdo NL-to-SQL for critical queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation set is assumed to resemble real operator queries and not to be drawn mainly from the same pre-approved function library that the function-calling system was built around.","fun_headline_variants_meta":{"raw":{"variants":["Function-calling LLMs beat NL-to-SQL for nuclear data","Pre-approved functions outperform freehand SQL in plants","Expert-validated functions lift nuclear query accuracy","Nuclear data retrieval: function calls trump raw SQL","LLM function calls outdo NL-to-SQL for critical queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1309,"prompt_tokens":1011,"completion_tokens":298,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":219}},"tokens_in":627,"tokens_out":298,"duration_ms":3868,"temperature":1.0,"reasoning_tokens":219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:01:57.920147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask plant operators who never saw the function library to write a fresh set of queries, including some request types the library was not designed for, and have SMEs score both systems blind; if the function-calling system no longer outperforms the NL-to-SQL baseline on human-evaluated correctness, the reported advantage is an artifact of overlapping test questions and functions.","supporting_citations":[{"cited_title":"Traditional NL-to-SQL methods, while user-friendly, introduce risks when handling complex legacy databases","cited_arxiv_id":null,"evidence_quote":"Supplies the guidance that 70B+ parameters are recommended when conversation and function calling must coexist, which shapes model selection."},{"cited_title":"What day is it today?","cited_arxiv_id":null,"evidence_quote":"Provides the structured-outputs technique, constrained decoding, that the system relies on to enforce JSON schemas."},{"cited_title":"main agent","cited_arxiv_id":null,"evidence_quote":"Establishes the function-calling leaderboard used to pick a frontier model for both evaluated systems."},{"cited_title":"Show me all the work requests entered in by John Smith","cited_arxiv_id":null,"evidence_quote":"Shows that small models can approach top-tier function-calling performance after targeted fine-tuning, informing the scalability discussion."},{"cited_title":"This query also comes with an explanation of how it was built","cited_arxiv_id":null,"evidence_quote":"Reports that function-selection accuracy drops when the number of tools grows, motivating dynamic filtering strategies."},{"cited_title":"Any discrepancies are corrected so that the query conforms to the expected format and contains only valid tables and fields","cited_arxiv_id":null,"evidence_quote":"Introduces document-retrieval-based tool filtering that reduces the function set the LLM must consider."}],"review_version":1}