Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Without retrieval augmentation, an LLM produces zero correctly matching enterprise SQL or API calls; with iterative retrieval, it maintains accuracy even when database and API documentation are mixed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 06:06 UTC pith:XTO2BKU7

load-bearing objection Solid proof that RAG is essential for enterprise SQL/API generation, but the CoRAG advantage is confounded by unequal retrieval budgets and GPT-5-generated references. the 3 major comments →

arxiv 2602.07086 v2 pith:XTO2BKU7 submitted 2026-02-06 cs.SE cs.AIcs.CL

RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation

Tim Schlippe , Simon Martin , Michael Marketsm\"uller This is my paper
classification cs.SE cs.AIcs.CL
keywords retrieval-augmented generationtext-to-SQLAPI call generationCoRAGSelf-RAGenterprise softwareLLM evaluationdocumentation retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that for enterprise systems, retrieval-augmented generation is a functional prerequisite: with no retrieval, a state-of-the-art LLM scores 0% exact match on every task, while any RAG variant jumps execution accuracy to 71–79%. Among the three retrieval policies tested, CoRAG — which decomposes a user request into iterative sub-queries and retrieves iteratively — is the most robust in hybrid documentation settings, keeping SQL exact-match at 15.32% when standard RAG falls to 11.56% (p<0.01). The combined SQL-plus-API task shows the same pattern: CoRAG reaches 10.29% exact match versus 7.45% for standard RAG (p=0.0006), driven mainly by SQL. These findings matter because production assistants must handle mixed schema and endpoint documentation, and retrieval-policy choice is shown to be as important as the documentation itself.

Core claim

The central discovery is that retrieval policy, not just retrieval presence, determines whether enterprise structured generation survives documentation heterogeneity. In database-only or API-only contexts the three RAG variants perform comparably, but when database schemas and API specifications are both retrievable, standard RAG and Self-RAG degrade by 20–30% relative on SQL exact match, while CoRAG's iterative query decomposition holds essentially flat. CoRAG's advantage is statistically significant in the combined task (p=0.0006 vs standard RAG) and in hybrid SQL (p<0.01 vs both other variants). The paper attributes this to SQL's compositional structure: schema elements are spread across

What carries the argument

CoRAG (Chain-of-Retrieval Augmented Generation) is the central mechanism: it iteratively generates a sub-query from the user's request, retrieves top-5 documentation chunks for that sub-query, performs an aggregation-and-sufficiency check, and repeats until the LLM signals completion. This is contrasted with standard RAG (single top-5 retrieval, one generation call) and Self-RAG (top-5 retrieval followed by LLM-based relevance filtering). The paper's evaluation harness also matters: 631 execution-validated test cases derived from SAP Transactional Banking, with SQL and API gold outputs, scored by exact match, component match, execution accuracy, endpoint retrieval, and classification accurac

Load-bearing premise

The paper assumes its self-generated gold SQL/API calls are neutral ground truth, but since the same LLM family writes both the references and the candidates, stylistic self-similarity could inflate exact-match differences.

What would settle it

A concrete way to test the central claim: replace the GPT-5-generated reference outputs with references written by a different model family or by human experts, then rerun CoRAG vs standard RAG on the same 631 cases. If CoRAG's statistically significant exact-match advantage disappears or reverses, the claimed robustness is an artifact of self-generated references; if it persists, the claim survives.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Any production natural-language assistant for enterprise SQL/API should treat retrieval infrastructure as mandatory, not optional, since the no-RAG baseline is 0% exact match.
  • When documentation is homogeneous or partitionable by task type, standard RAG is sufficient; when it is mixed, CoRAG's extra two-to-three LLM calls are justified by statistically significant robustness.
  • Execution accuracy (68–79%) far exceeds exact match (10–15%), so production systems should use execution success as the primary correctness signal and treat structural match as a secondary check.
  • Unified assistants that route between SQL and API should consider task-specific retrieval policies, e.g., CoRAG for SQL, standard RAG for API, because API generation showed no significant variant differences.
  • Current systems remain best deployed in assisted mode: even the best configuration reaches only ~15% exact match, so interactive refinement and human fallback are necessary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's single-reference, single-backbone design likely understates all variants' true correctness; a multi-reference or semantic-equivalence metric could shrink the gap between exact match and execution accuracy, and might change the CoRAG-vs-RAG ranking.
  • Because the gold references were generated by the same model family (GPT-5) used as the generation backbone, the exact-match advantage could partly reflect stylistic self-similarity rather than functional superiority; re-testing with references written by humans or by a different model family would isolate this effect.
  • CoRAG's robustness in hybrid settings suggests a testable extension: feeding CoRAG only the final aggregated context, or using a fixed number of retrieval rounds, could separate the benefit of iteration from the benefit of query decomposition.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates three RAG variants—standard RAG, Self-RAG, and CoRAG—for natural-language-to-SQL and REST-API-call generation in an enterprise setting based on SAP Transactional Banking documentation. It introduces a new 631-case dataset (346 SQL, 285 API) built with an adapted APIGen pipeline, using GPT-5 to generate reference outputs, humanizing inputs, validating outputs against mock SQLite/Postman environments, and expert review. The authors report 18 experimental configurations across database-only, API-only, and hybrid documentation contexts. The central claims are: (i) RAG is functionally essential, with 0% exact-match accuracy for the no-RAG baseline across all tasks and execution accuracy rising to 71–79% with retrieval; and (ii) CoRAG provides statistically significant robustness advantages in hybrid documentation settings, especially for SQL generation (e.g., combined-task EM 10.29% vs. 7.45% for standard RAG, p=0.0006; SQL-hybrid EM 15.32% vs. 11.56%, p=0.0091/0.0026). The discussion frames retrieval-policy design as a primary determinant of production viability.

Significance. If the comparative findings hold, this would be a useful contribution: it addresses a real gap by jointly evaluating SQL and API generation under heterogeneous documentation, and it provides an execution-validated, publicly released test set. The RAG-essential result appears robust because the no-RAG baseline achieves 0% exact match while retrieval lifts execution accuracy to roughly 70–79%, measured against mock servers rather than only string matching. The paper also ships a concrete dataset and reports extensive experimental configurations. However, the headline comparative claim about CoRAG is currently not adequately supported. Two confounds—an unequal retrieval budget and reference outputs generated by the same model family being scored—undermine the attribution of the observed exact-match differences to iterative query decomposition. The practical significance of the paper therefore depends on resolving these issues.

major comments (3)
  1. [§3.1, §4.2, §4.3] The central CoRAG comparison is confounded by an unequal retrieval budget. Section 3.1 states that retrieval size was fixed at top-5 chunks, but CoRAG retrieves top-5 chunks per sub-query and repeats until the LLM signals completion, so it can access 10, 15, or more chunks while standard RAG and Self-RAG receive at most 5 chunks total. The reported hybrid-SQL and combined-task advantages (Table 2: 15.32% vs. 11.56%; Table 3: 10.29% vs. 7.45%) could therefore reflect a larger context window rather than a better retrieval policy. The paper does not report the total number of retrieved chunks per variant. To support the conclusion that iterative decomposition itself helps, the authors need an ablation that controls retrieval budget (e.g., standard RAG with top-10 or top-15 chunks, or CoRAG limited to 5 total chunks) and should report the average number of chunks used by each variant.
  2. [§3.3 step 1, §4.2, §5.5] The evaluation is partially self-referential. The reference SQL queries and API calls used for exact-match scoring were generated by GPT-5, and GPT-5 is also the backbone model whose RAG variants are being scored. Since exact match is a strict single-reference string comparison, any stylistic or structural bias in the reference set toward GPT-5 output patterns will disproportionately benefit variants whose outputs most resemble GPT-5's own generation style. This is especially relevant for CoRAG, whose multi-step reasoning may produce outputs more aligned with the generator's internal style. The limitations section (§5.5) lists single-reference and single-backbone issues but does not acknowledge this circularity. The authors should either add human-validated alternative references, evaluate with execution-based semantic equivalence as the primary metric, or analyze the degree of stylistic
  3. [§4.3, Table 3] The statistical significance claim rests on very small numbers of additional correct cases. In the combined task, CoRAG's exact-match advantage over standard RAG is 10.29% vs. 7.45%; over the full 631-case set this difference corresponds to roughly 18 cases. The SQL-subset difference of 15.90% vs. 11.56% corresponds to about 15 of 346 cases. These are plausible, but the paper reports only p-values from a paired t-test on binary outcomes and does not report the number of discordant pairs, effect sizes, or confidence intervals. Given the small absolute counts and the confounding in the previous comment, the authors should report McNemar's test details, exact confidence intervals, and ideally bootstrap results to show stability before the superiority claim can be accepted.
minor comments (5)
  1. [Abstract vs. §4.2 and §4.3] The abstract states CoRAG's SQL-hybrid exact match as 15.32%, while §4.3 reports the SQL subset as 15.90%. The relationship between these numbers and the combined-task SQL subset should be clarified.
  2. [§3.1] The description of Self-RAG's relevance threshold (≥0.2) is vague—what exactly is being thresholded, and how was this value chosen? The paper should provide the prompt or a precise definition.
  3. [§3.5] Paired t-tests on binary outcomes are unconventional; McNemar's test is more standard for paired binary data. At minimum, the reported p-values should be accompanied by the discordant-pair counts.
  4. [§5.5] The limitations section is missing the two most consequential threats to validity identified above: the self-generated reference outputs and the unequal retrieval budget. These should be acknowledged explicitly.
  5. [§4.1, Table 1] The API-only and hybrid baselines are identical (0.00, 20.31, 0.00, 28.45), which makes sense because baseline uses no retrieval, but the table could state this to avoid confusion.

Circularity Check

0 steps flagged

No significant circularity; the paper is an empirical comparison with external execution checks, not a derivation that reduces to its own inputs.

full rationale

The paper is an empirical benchmark study rather than a derivation chain: there are no fitted constants, no uniqueness theorems, and no quantity that is defined in terms of the quantity it is used to predict. The central RAG-essential claim is anchored in external execution checks against mock SQLite and Postman servers and in a no-RAG baseline of 0% exact match, neither of which depends on the reference labels alone. The CoRAG-vs-standard-RAG comparison does use exact-match against single GPT-5-generated reference outputs (Section 3.3 step 1), which is a genuine evaluation threat—it may favor outputs stylistically similar to GPT-5 and is confounded by CoRAG's larger effective retrieval budget (top-5 per sub-query vs. top-5 total for standard RAG). However, those references were execution-validated and expert-reviewed, and the reported differences are not true by construction; the experimental outcomes are contingent measurements, not equivalences. Self-citations (e.g., Idrisov and Schlippe 2024; Kivroglou et al. 2025) are contextual and not load-bearing. The paper itself flags single-reference and single-backbone limitations (Section 5.5), though it does not mention that references were generated by the same model family; that omission is a limitation or validity concern, not circularity. No load-bearing step reduces to its own input, so no significant circularity is found.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's central claims rest heavily on the dataset construction (GPT-5-generated references), the chosen retrieval hyperparameters, and the mock execution environment. These are documented but not all independently validated.

free parameters (5)
  • Retrieval top-k = 5
    Fixed at 5 chunks for all variants, following prior practice (Zhou et al., Chen et al.); affects the context available to generation and all downstream metrics.
  • Self-RAG relevance threshold = ≥0.2
    Ad hoc threshold for filtering chunks in Self-RAG; no ablation reported, yet it directly controls how many chunks survive to the prompt.
  • Document chunk size/overlap = 8,000 tokens; 800-token overlap (10%)
    Chunking scheme for API docs (12/145 chunks split); affects retrieval granularity and completeness.
  • Fixed reference date = 2025-05-01
    Fixed date imposed for temporal consistency; can influence SQL queries containing date predicates.
  • Mock execution environment = SQLite + Postman mock servers
    Execution 'validity' depends on this simulated environment, not the real SAP system; abstracting away authentication, side effects, and network behavior.
axioms (4)
  • domain assumption GPT-5 can generate valid reference outputs and natural-language test cases for TRBK tasks.
    Dataset construction (§3.3 step 1) relies on GPT-5 as the oracle for reference outputs, which is also the model under evaluation. This is a foundational assumption that may bias comparisons.
  • domain assumption Single-reference exact-match is an appropriate correctness metric.
    The paper acknowledges in §5.5 that multiple valid SQL/API outputs exist; this assumption can underestimate true correctness and may systematically favor the model that generated the reference.
  • domain assumption Paired t-tests on per-case binary outcomes are valid without multiple-comparison correction.
    Used in §4 to claim significance; binary outcomes with small success counts violate normality assumptions, and many pairwise tests are run without adjustment, inflating false-positive risk.
  • domain assumption SAP TRBK documentation is representative of enterprise domains.
    Generalization beyond TRBK is limited; §5.5 acknowledges this, but the paper's conclusions are implicitly generalized to 'enterprise structured generation'.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation." pith.science (2026). https://pith.science/paper/XTO2BKU7

@misc{pith2026260207086,
  author       = {Pith},
  title        = {Pith review of: RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XTO2BKU7}},
  note         = {Machine review of arXiv:2602.07086}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Enterprise software systems commonly expose business functionality through both relational databases and REST APIs. Accessing these interfaces requires specialized technical knowledge, as users must determine whether a request requires a database query or an API operation and understand the corresponding schemas, endpoints, and parameters. This creates demand for natural language interfaces that translate user requests into SQL queries and REST API calls. While large language models (LLMs) show promise for structured code generation, they typically lack reliable knowledge of enterprise-specific schemas, endpoints, and documentation. Retrieval-augmented generation (RAG) addresses this limitation by grounding generation in external documentation. However, prior work largely studies SQL query generation and REST API call generation separately, despite enterprise documentation environments often containing both database schemas and API specifications. We systematically evaluate standard RAG, Self-RAG, and CoRAG across SQL query generation, REST API call generation, and a combined task requiring routing between both operation types. Using SAP Transactional Banking as a realistic enterprise use case, we constructed an execution-validated dataset and compared retrieval strategies under database-only, API-only, and mixed-documentation settings. Retrieval augmentation proved essential for reliable enterprise structured generation, substantially improving performance over a no-retrieval baseline. CoRAG achieved the best results in the combined SQL query and REST API call setting, with statistically significant improvements in exact-match accuracy over standard RAG, primarily driven by stronger SQL query generation under mixed-documentation retrieval conditions. Overall, findings show that retrieval strategy substantially affects structured generation performance under mixed-documentation settings.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 16 linked inside Pith

  1. [1]

    Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation.arXiv preprint arXiv:2506.17277,

    [Amiri and Bocklitz, 2025] Mahsa Amiri and Thomas Bock- litz. Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation.arXiv preprint arXiv:2506.17277,

    Pith/arXiv arXiv 2025

  2. [4]

    NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls.arXiv preprint arXiv:2409.03797,

    [Basuet al., 2025 ] Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Ruchi Yara, Abhinav Selvam, Avi Sil, Avrilia Floratou, Achille Fok- oue, et al. NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls.arXiv preprint arXiv:2409.03797,

    Pith/arXiv arXiv 2025

  3. [5]

    Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Anal- ysis.arXiv preprint arXiv:2505.21700,

    [Bhatet al., 2025 ] Sai Rohith Bhat, Moritz Rudat, Julius Spiekermann, and Nick Flores-Herr. Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Anal- ysis.arXiv preprint arXiv:2505.21700,

    Pith/arXiv arXiv 2025

  4. [6]

    Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374,

    [Chenet al., 2021 ] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv 2021

  5. [8]

    A Survey on Large Language Models for Code Generation.arXiv preprint arXiv:2503.01245,

    [Huynh and Lin, 2025] Nhi Huynh and Bill Lin. A Survey on Large Language Models for Code Generation.arXiv preprint arXiv:2503.01245,

    Pith/arXiv arXiv 2025

  6. [9]

    Program Code Generation with Generative AIs.Algorithms, 17(2):62, January

    [Idrisov and Schlippe, 2024] Baskhad Idrisov and Tim Schlippe. Program Code Generation with Generative AIs.Algorithms, 17(2):62, January

    2024

  7. [10]

    [Idrisovet al., 2025 ] Baskhad Idrisov, Esther Eisenacher, and Tim Schlippe

    Special Issue: Feature Papers on Artificial Intelligence Algorithms and Their Applications. [Idrisovet al., 2025 ] Baskhad Idrisov, Esther Eisenacher, and Tim Schlippe. Program Code Generation: Single LLMs vs. Multi-Agent Systems. InThe 7th International Conference on Natural Language Processing (ICNLP), page TBD, Guangzhou, China, March

    2025

  8. [11]

    Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.arXiv preprint arXiv:2403.07974,

    [Jainet al., 2024 ] Naman Jain, King Han, Alex Gu, Wen- Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.arXiv preprint arXiv:2403.07974,

    Pith/arXiv arXiv 2024

  9. [12]

    SEAL: Suite for Evaluating API-Use of LLMs.arXiv preprint arXiv:2409.15523,

    [Kimet al., 2024 ] Wonyoung Kim, Ashish Jagmohan, and Aditya Vempaty. SEAL: Suite for Evaluating API-Use of LLMs.arXiv preprint arXiv:2409.15523,

    Pith/arXiv arXiv 2024

  10. [13]

    Investigating Retrieval Augmented Generation for LLM-Based Code Generation

    [Kivroglouet al., 2025 ] Paraskevi Kivroglou, Tim Schlippe, and Simon Martin. Investigating Retrieval Augmented Generation for LLM-Based Code Generation. InThe 3rd International Conference on F oundation and Large Language Models (FLLM), page TBD, Vienna, Austria, November

    2025

  11. [14]

    Natural Language Data Interfaces: A Data Access Odyssey.Leibniz International Proceedings in Informatics (LIPIcs), ICDT 2024,

    [Koutrika, 2024] Georgia Koutrika. Natural Language Data Interfaces: A Data Access Odyssey.Leibniz International Proceedings in Informatics (LIPIcs), ICDT 2024,

    2024

  12. [15]

    Evaluating Language Models on Real- World Enterprise Text-to-SQL Workflows.arXiv preprint arXiv:2411.07763,

    [Leiet al., 2024 ] Fuhui Lei, Junda Chen, Yiwei Ye, Ruixuan Cao, Dohyun Shin, Huiyao Su, Yiheng Gan, Jiaqi Liu, Lin Ma, and Tao Yu. Evaluating Language Models on Real- World Enterprise Text-to-SQL Workflows.arXiv preprint arXiv:2411.07763,

    Pith/arXiv arXiv 2024

  13. [16]

    User-Related Challenges of Self- Service Business Intelligence.Information Systems Man- agement, 38(4):309–323,

    [Lennerholtet al., 2020 ] Carl Lennerholt, Jorgen Van Laere, and Eva S ¨oderstr¨om. User-Related Challenges of Self- Service Business Intelligence.Information Systems Man- agement, 38(4):309–323,

    2020

  14. [18]

    API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs.arXiv preprint arXiv:2304.08244,

    [Liet al., 2023 ] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs.arXiv preprint arXiv:2304.08244,

    Pith/arXiv arXiv 2023

  15. [20]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    [Patilet al., 2023 ] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large Language Model Connected with Massive APIs.arXiv preprint arXiv:2305.15334,

    Pith/arXiv arXiv 2023

  16. [21]

    Postman Mock Servers Documentation

    [Postman, 2025] Postman. Postman Mock Servers Documentation. https://learning.postman.com/docs/ design-apis/mock-apis/overview/,

    2025

  17. [22]

    [Pourreza and Rafiei, 2023] Mohammadreza Pourreza and Davood Rafiei

    Accessed: 2025-05-21. [Pourreza and Rafiei, 2023] Mohammadreza Pourreza and Davood Rafiei. Evaluating Cross-Domain Text- to-SQL Models and Benchmarks.arXiv preprint arXiv:2310.18538,

    Pith/arXiv arXiv 2025

  18. [23]

    Calibrated Trust in Dealing with LLM Hallu- cinations: A Qualitative Study

    [Ryseret al., 2025 ] Adrian Ryser, Florian Allwein, and Tim Schlippe. Calibrated Trust in Dealing with LLM Hallu- cinations: A Qualitative Study. InThe 3rd International Conference on F oundation and Large Language Models (FLLM), page TBD, Vienna, Austria, November

    2025

  19. [24]

    SAP Transactional Banking API Documentation

    [SAP SE, 2025] SAP SE. SAP Transactional Banking API Documentation. https://api.sap.com/package/TRBK/ odatav4,

    2025

  20. [25]

    [Schicket al., 2024 ] Timo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom

    Accessed: 2025-05-21. [Schicket al., 2024 ] Timo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. InAdvances in Neural Information Processing Systems, volume 36,

    2025

  21. [26]

    DBTagger: Multi-Task Learning for Key- word Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks.arXiv preprint arXiv:2101.04226,

    [Ustaet al., 2021 ] Arif Usta, Ali Emre Karakayali, and ¨Ozg¨ur Ulusoy. DBTagger: Multi-Task Learning for Key- word Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks.arXiv preprint arXiv:2101.04226,

    Pith/arXiv arXiv 2021

  22. [27]

    Ribeiro, Alexandros Papangelis, Rajdeep Mukherjee, Ting-Yao Wang, Xi Zhao, Paul A

    [Wanget al., 2024 ] Zhuoer Wang, Leonardo F. Ribeiro, Alexandros Papangelis, Rajdeep Mukherjee, Ting-Yao Wang, Xi Zhao, Paul A. Crook, and Angeliki Metallinou. FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation Through State-Tracked Constrained Decoding and Reranking.arXiv preprint arXiv:2407.13945,

    Pith/arXiv arXiv 2024

  23. [28]

    Chain- of-Retrieval Augmented Generation.arXiv preprint arXiv:2501.14342,

    [Wanget al., 2025 ] Liang Wang, Haoyang Chen, Nan Yang, Xueguang Huang, Zhicheng Dou, and Furu Wei. Chain- of-Retrieval Augmented Generation.arXiv preprint arXiv:2501.14342,

    arXiv 2025

  24. [29]

    Xu, Uri Alon, Graham Neubig, and Vincent J

    [Xuet al., 2023 ] Frank F. Xu, Uri Alon, Graham Neubig, and Vincent J. Hellendoorn. ToolBench: A Benchmark for Tool Learning with Large Language Models.arXiv preprint,

    2023

  25. [30]

    Spider: A Large- Scale Human-Labeled Dataset for Complex and Cross- Domain Semantic Parsing and Text-to-SQL Task

    [Yuet al., 2018 ] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A Large- Scale Human-Labeled Dataset for Complex and Cross- Domain Semantic Parsing and Text-to-SQL Task. InThe 2018 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3911–...

    2018

  26. [2018]

    [Yuet al., 2019a ] Tao Yu, Rui Zhang, Heyang Yasunaga Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al

    Association for Computational Linguis- tics. [Yuet al., 2019a ] Tao Yu, Rui Zhang, Heyang Yasunaga Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases. InThe 2019 Confer- ence on Empirical Methods in Natural L...

    2019

  27. [2019]

    [Zhouet al., 2023 ] Shuyan Zhou, Uri Alon, Frank F

    Association for Com- putational Linguistics. [Zhouet al., 2023 ] Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. DocPrompting: Generating Code by Retrieving the Docs. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  28. [2020]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.Advances in Neural In- formation Processing Systems, 33:9459–9474,

    [Lewiset al., 2021 ] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.Advances in Neural In- formation Processing Systems, 33:9459–9474,

    2021

  29. [2021]

    When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?arXiv preprint arXiv:2503.15231,

    [Chenet al., 2025 ] Junda Chen, Shuzheng Chen, Jie Cao, Jing Shen, and Shing-Chi Cheung. When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?arXiv preprint arXiv:2503.15231,

    Pith/arXiv arXiv 2025

  30. [2023]

    APIGen: Automated Pipeline for Generating Verifiable and Diverse Function- Calling Datasets.arXiv preprint arXiv:2406.18518,

    [Liuet al., 2024 ] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. APIGen: Automated Pipeline for Generating Verifiable and Diverse Function- Calling Datasets.arXiv preprint arXiv:2406.18518,

    Pith/arXiv arXiv 2024

  31. [2024]

    Seven Failure Points When Engineering a Re- trieval Augmented Generation System.arXiv preprint arXiv:2401.05856,

    [Barnettet al., 2024 ] Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Ab- delrazek. Seven Failure Points When Engineering a Re- trieval Augmented Generation System.arXiv preprint arXiv:2401.05856,

    Pith/arXiv arXiv 2024

  32. [2025]

    Self-RAG: Learn- ing to Retrieve, Generate, and Critique Through Self- Reflection

    [Asaiet al., 2024 ] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learn- ing to Retrieve, Generate, and Critique Through Self- Reflection. InInternational Conference on Learning Rep- resentations (ICLR),

    2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.