REVIEW 2 major objections 5 minor 32 references
BAP-SQL: Budget-Aware Observation Planning for Agentic Text-to-SQL
T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Rewriting SQL before execution lifts tight-budget agent success
desk verdict A well-built empirical paper on pre-execution cost-aware SQL rewriting; the package-level gain is real, the estimator-accuracy isolation gap is openly acknowledged, and it deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a closed pre-execution loop built from three parts. First, an estimator combines EXPLAIN QUERY PLAN, a zero-row LIMIT 0 probe, and per-database catalog statistics to return calibrated p50 and p95 predictions for rows, result tokens, and database work. Second, the policy sees those estimates plus the remaining budget and may issue a Rewrite action that changes the query before execution; rewrites need not preserve the candidate's semantics, because success is judged against the user question and visible evidence. Third, an independent runtime shield configures VM-step, row, byte, and result-token caps before execution, so estimates expose risk while the shield guarantees feasibility. This loop is trained with a correctness-gated efficiency reward that gives zero reward to wrong or infeasible trajectories and ranks only the successful ones by evidence quality and tool-use count.
What would settle it
Run the full pipeline on held-out databases whose cardinality statistics differ sharply from the calibration databases, for example after multiplying base table sizes by ten or changing join selectivity, and check whether the tight-budget gains persist relative to matched supervised fine-tuning. A complementary check is a retrained shuffled-estimate placebo in which the policy sees permuted p50/p95 values while the shield still uses true estimates: if that retrained policy matches full BAP-SQL, the effect would not require trustworthy estimates.
Extended reading notes
Core claim
BAP-SQL's central claim is that exposing pre-execution cost estimates to a reinforcement-learned policy and letting it rewrite SQL before execution makes tool-using agents more likely to answer correctly under tight context, query, result-token, and database-work budgets. The paper shows this through a trained policy that observes remaining budget and p50/p95 estimates of rows, result tokens, and database work; chooses among inspect, estimate, rewrite, execute, manage-evidence, answer, and abstain actions; and is governed by a runtime shield that makes hard budget breaches unreachable. The primary result is a 3.4/3.6 percentage-point budgeted-success gain over matched SFT at tight budgets with 4.5/5.0 percent fewer total tokens. The paper attributes the gain to policy-visible planning and budget-sensitive rescue: in a locked retraining suite, removing the estimate-and-rewrite package costs 1.9/2.0 points, and a cost-matched placebo with shuffled estimates confirms that the policy relies on the estimate content. The advantage is regime-specific: it concentrates on budget-sensitive failures, shrinks with stronger generators and larger budgets, and reverses at the loosest budget, where a correctness-only SQL-RL baseline is stronger.
Load-bearing premise
The approach assumes the pre-execution estimator's p50/p95 forecasts are accurate enough to guide rewriting; if those estimates do not transfer to a new database's data distribution, the planning advantage could shrink or reverse.
Editorial extensions
If this is right
- Under tight observation budgets, a pre-execution estimate-and-rewrite loop is a reliable way to improve answer quality without increasing total tokens on general small and SQL-specialized generators.
- Removing the policy-visible estimates and the rewrite action costs roughly 2 percentage points of budgeted success, so the interface itself, not merely the reward, carries the effect.
- Correctness-gated efficiency shaping and pre-execution planning are complementary: the reward helps more when the policy can plan, with a planning-by-reward interaction of about 1.8/1.4 points.
- The saved tokens come from shorter trajectories and fewer turns, not from admitting fewer result tokens; database work does not decrease and may rise slightly.
- The benefit is largest at tight budgets and disappears or reverses at the loosest budget, so the method is a targeted complement to stronger task policies rather than a general accuracy booster.
Reading between the lines
- If estimator calibration transfers across database distributions, the same plan-before-execution principle could generalize to other tool-using agents whose actions determine the size and content of the next observation, such as search and API-call agents.
- The measured 30 percent estimator-noise penalty suggests a testable extension: training the policy under noisy or perturbed estimates could make it robust to calibration drift, which the current training does not explicitly address.
- The reversal at the loosest budget hints that a budget-conditional gating policy, one that rewrites aggressively at tight budgets and executes broad queries when there is headroom, could extend the gains beyond the current fixed regime.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes BAP-SQL, an agentic text-to-SQL loop with a pre-execution planning stage. Before executing SQL, an estimator returns p50/p95 predictions for rows, result tokens, and database work from EXPLAIN QUERY PLAN, a zero-row probe, and catalog statistics with calibration factors fitted on disjoint databases; a policy trained by RL across four joint budget levels can decide to inspect, estimate, rewrite, execute, manage evidence, answer, or abstain, while an independent runtime shield enforces hard limits. On a BIRD-derived dev set with Qwen3.5-4B, BAP-SQL reports Budgeted Success gains of 3.4/3.6 percentage points over matched SFT at XS/S with 4.5/5.0% fewer total tokens, and similar directional gains over interface-matched SQL-RL and adapted BACM-RL. The paper also reports transfer to FINER-SQL-4B and a 7B coder, a locked retraining factorial, fixed-policy diagnostics, and estimator calibration. The benefit attenuates as budgets loosen, reverses at the loosest budget L, and does not reduce database work.
Significance. The empirical package is careful and internally consistent. Strengths include paired database bootstrap intervals with per-seed stability, a locked 2x2 retraining suite, correctness-gated reward shaping, calibration factors fitted on disjoint databases, and explicit disclosure of limitations. The headline tight-budget gains are supported as an observed property of the trained system. The main weakness is that the training-level evidence for the advocated mechanism is at the package level: the no-planning control removes both policy-visible estimates and the Rewrite action, so the contribution of accurate cost estimates to learning is not isolated. Because the abstract attributes the gain to an estimate-then-rewrite loop, this gap is load-bearing for the paper's mechanistic claim rather than for the raw system-level result.
major comments (2)
- [Ablations; Table 4 (left panel)] The locked retraining suite's headline planning effect compares Full BAP-SQL RL (34.5/37.2) with No-planning RL (32.6/35.2), but the no-planning condition removes both policy-visible p50/p95 estimates and the Rewrite action. The +1.9/+2.0 pp effect therefore establishes the combined estimate-and-rewrite package under the shared reward, not the contribution of estimator accuracy. The fixed-policy shuffled-estimate row (31.9/34.4) shows inference-time sensitivity but cannot tell us whether a policy retrained on shuffled estimates would retain the gain; the paper's own Limitations section says exactly this. Because the abstract and contribution statements present the mechanism as 'estimates query risk, rewrites SQL when useful,' the estimate component is load-bearing. Please add a retrained shuffled-estimate (or equivalent no-estimate) condition to the locked protocol, or explicitly demote the mechanistic claim to package-level in the abstract and conclusion. The reward x planning interaction does not resolve the confound because both factors are defined at the package level.
- [Estimator behavior / Pre-execution query planning] The evidence for estimator quality is p95 coverage (98.8% rows, 97.0% result tokens, 96.0% database work) and a fixed-policy 30% noise injection costing 1.8 pp. Coverage of the tail does not measure the accuracy of the p50 signals that most plausibly drive the policy's rewrite decisions, and the noise-injection result is measured with the policy frozen. Please report calibration metrics for the p50 estimates (e.g., median absolute relative error or calibration slope by database and query shape) and state which quantile the policy actually consumes. Without this, the central 'estimate query risk' mechanism remains supported only by an inference-time sensitivity analysis, not by training-level evidence.
minor comments (5)
- [RL objective and configuration] The sentence 'configuration details appear in Section .' has an empty cross-reference; please fill in the section number or remove the dangling reference.
- [Experimental Setup, Metrics and inference] The term 'Database-t' is used as an analysis name but never defined; please define it in the Metrics and inference paragraph.
- [Problem Formulation] Evidence Support is defined as a metric vocabulary item but is not reported in any results table; either report it in the main results or remove the definition to avoid dead notation.
- [Results, RQ1 and Table 3 caption] The important caveat that bootstrap intervals condition on the three evaluated adapters appears in the caption and in Limitations, but it should also appear in the RQ1/RQ2 results text so that the headline intervals are not over-read as covering training randomness.
- [Results, Boundary conditions] The 7B coder experiment is reported in two sentences without model or training details; a short appendix entry describing the model, data split, and RL configuration would aid reproducibility.
Circularity Check
No significant circularity: the reported gains come from matched empirical comparisons with disjoint estimator calibration, and the reward is not identical to the evaluation metric.
full rationale
The paper's central claim is an empirical RL-versus-SFT comparison on a held-out dev set, and the ingredients that could create circularity are kept separate from the target metric. The estimator's calibration factors are fitted on disjoint databases ("Calibration factors are fitted on disjoint databases"), so the p50/p95 predictions used by the policy are not fitted to the evaluation tasks. The reward is explicitly correctness-gated and is not the Budgeted Success metric: R = 1[correct∧feasible](1 + 0.4e) r_tool(m,n), whereas Budgeted Success is s_i = [y_i = y*_i][z_i = 1]; the additional evidence and efficiency terms can only rank already-correct trajectories. The locked retraining suite compares full BAP-SQL against an action-matched no-planning control under a shared reward and curriculum, so the 1.9/2.0-point full-minus-no-planning difference isolates the policy-visible estimate-and-rewrite package rather than a fitted constant. The paper's own limitation statement—"It does not isolate estimator calibration or rewriting separately; a retrained shuffled-estimate placebo would address part of this gap"—is an attribution caveat about mechanism, not a reduction of the result to its inputs. No load-bearing step relies on a self-citation chain; cited baselines and prior methods (BIRD, OTC-PO, ContextBudget, SQL-R1, and others) are external and do not supply the reported gains. The 30% estimator-noise sensitivity is an honest boundary condition rather than evidence that the gain was preset. The estimator coverage numbers are presented as descriptive diagnostics, not as the source of the headline improvement. Therefore, no circular step is present.
Assumptions & free parameters
free parameters (2)
- Estimator calibration factors =
not reported in text
- Joint budget ladder (XS/S/M/L) =
Table 1 (e.g., XS: 2,400 context tokens, 1 query, 80 result tokens, 250,000 VM steps, 5 turns)
assumptions (3)
- domain assumption The BIRD-derived and dev datasets are representative of real text-to-SQL workloads for the purpose of the budgeted evaluation.
- domain assumption The environment's budget accounting (live context tokens, executed queries, cumulative admitted result tokens, SQLite VM steps) faithfully captures the costs that matter for serving text-to-SQL agents.
- domain assumption The database-cluster bootstrap over the 11 dev databases, conditioning on the three adapters, provides a valid uncertainty estimate for the paired comparisons.
Cite this review
Pith. "Pith review of BAP-SQL: Budget-Aware Observation Planning for Agentic Text-to-SQL." pith.science (2026). https://pith.science/paper/5S7NQ4UI
@misc{pith2026260802876,
author = {Pith},
title = {Pith review of: BAP-SQL: Budget-Aware Observation Planning for Agentic Text-to-SQL},
year = {2026},
howpublished = {\url{https://pith.science/paper/5S7NQ4UI}},
note = {Machine review of arXiv:2608.02876}
}
read the original abstract
Tool-using agents do not merely consume observations: their actions determine what arrives next. In agentic text-to-SQL, a broad query can spend context and database work before useful evidence appears, while post-hoc compression cannot recover omitted rows or expended work. We present BAP-SQL, which treats observation formation as a budget-control stage: it estimates query risk, rewrites SQL when useful, and delegates hard limits to an independent runtime shield. Across general 4B, specialized FINER-SQL 4B, and 7B backbones, BAP-SQL improves tight-budget success. On the primary BIRD-derived setting, it gains 3.4/3.6 percentage points over matched SFT while using 4.5/5.0% fewer tokens. Matched retraining and task-level transfer associate the gain with policy-visible planning and budget-sensitive rescue. The benefit attenuates as model capability and budget increase, reverses at the loosest setting, and does not reduce database work.
Figures
Reference graph
Works this paper leans on
-
[1]
Li, Jinyang and Hui, Binyuan and Qu, Ge and Yang, Jiaxi and Li, Binhua and Li, Bowen and Wang, Bailin and Qin, Bowen and Cao, Rongyu and Geng, Ruiying and Huo, Nan and Zhou, Xuanhe and Ma, Chenhao and Li, Guoliang and Chang, Kevin C. C. and Huang, Fei and Cheng, Reynold and Li, Yongbin , booktitle =. Can. 2023 , eprint =
work page 2023
-
[2]
Wang, Chenglong and Tatwawadi, Kedar and Brockschmidt, Marc and Huang, Po-Sen and Mao, Yi and Polozov, Oleksandr and Singh, Rishabh , journal =. Robust Text-to-. 2018 , url =
work page 2018
-
[3]
Ma, Peixian and Zhuang, Xialie and Xu, Chengjin and Jiang, Xuhui and Chen, Ran and Guo, Jian , journal =. 2025 , url =
work page 2025
-
[4]
Zhang, Yuxin and Fan, Meihao and Fan, Ju and Yi, Mingyang and Luo, Yuyu and Li, Guoliang and Wu, Bin and Zhou, Wenchao , journal =. Reward-. 2025 , url =
work page 2025
-
[5]
Xu, Zekun and Xia, Siyu and Yue, Chuhuai and Chai, Jiajun and Tian, Mingxue and Wang, Xiaohan and Lin, Wei and Li, Haoxuan and Yin, Guojun , journal =. 2025 , url =
work page 2025
-
[6]
and Rangwala, Huzefa , booktitle =
Hua, Harper and Han, Zhen and Shen, Zhengyuan and Lee, Meng-Chieh and Guan, Sheng and Zhu, Qi and Jeoung, Sullam and Chen, Yueyan and Bai, Yunfei and Wang, Shuai and Ioannidis, Vassilis N. and Rangwala, Huzefa , booktitle =. 2026 , pages =. doi:10.18653/v1/2026.acl-long.1677 , url =
-
[7]
Jiang, Jie and Shen, Siqi and Xie, Haining and Li, Yang and Shen, Yu and Huang, Danqing and Qian, Bo and Wu, Yinjun and Zhang, Wentao and Cui, Bin and Chen, Peng , journal =. 2025 , url =
work page 2025
-
[8]
Biswal, Asim and Lei, Chuan and Qin, Xiao and Li, Aodong and Narayanaswamy, Balakrishnan and Kraska, Tim , journal =. 2026 , url =
work page 2026
Show all 32 references
-
[9]
arXiv preprint arXiv:2504.14870 , year =
Acting Less is Reasoning More! Teaching Model to Act Efficiently , author =. arXiv preprint arXiv:2504.14870 , year =
-
[10]
arXiv preprint arXiv:2506.05256 , year =
Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning , author =. arXiv preprint arXiv:2506.05256 , year =
-
[11]
2026 , url =
Wu, Yong and Zheng, YanZhao and Xu, TianZe and Zhang, ZhenTao and Yu, YuanQiang and Zhu, JiHuai and Ma, Chao and Lin, BinBin and Dong, BaoHua and Zhu, HangCheng and Huang, RuoHui and Yu, Gang , journal =. 2026 , url =
2026
-
[12]
2025 , url =
Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Dai, Weinan and Fan, Tiantian and Liu, Gaohong and Liu, Lingjun and others , journal =. 2025 , url =
2025
-
[13]
Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , journal =. 2024 , url =
2024
-
[14]
and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , journal =
Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , journal =. 2021 , url =
2021
-
[15]
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics , year =
Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning , author =. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics , year =. 2504.03380 , archivePrefix =
-
[16]
2025 , url =
Jin, Bowen and Zeng, Hansi and Yue, Zhenrui and Yoon, Jinsung and Arik, Sercan and Wang, Dong and Zamani, Hamed and Han, Jiawei , journal =. 2025 , url =
2025
-
[17]
2026 , url =
Lin, Yuxiang and Wang, Zihan and Liu, Mengyang and Shan, Yuxuan and Bai, Longju and Zhang, Junyao and Jin, Xing and Chen, Boshan and Su, Jinyan and Wang, Xingyao and Pei, Jiaxin and Li, Manling , journal =. 2026 , url =
2026
-
[18]
2025 , url =
Huo, Nan and Xu, Xiaohan and Li, Jinyang and Jacobsson, Per and Lin, Shipei and Qin, Bowen and Hui, Binyuan and Li, Xiaolong and Qu, Ge and Si, Shuzheng and Han, Linheng and Alexander, Edward and Zhu, Xintong and Qin, Rui and Yu, Ruihan and Jin, Yiyao and Zhou, Feige and Zhong...
2025
-
[19]
Boosting Small Language Models for Text-to-
Hoang, Thanh Dat and Huynh, Thanh Trung and Weidlich, Matthias and Nguyen, Thanh Tam and Chen, Tong and Yin, Hongzhi and Nguyen, Quoc Viet Hung , booktitle =. Boosting Small Language Models for Text-to-. 2026 , url =
2026
-
[20]
2026 , howpublished =
2026
-
[21]
, journal =
Yang, Haolin and Zhang, Jipeng and He, Zhitao and Zhou, Alexander and Fung, Yi R. , journal =. 2025 , url =
2025
-
[22]
2025 , url =
Zhang, Haozhe and Mayer, Christoph and Abo Khamis, Mahmoud and Olteanu, Dan and Suciu, Dan , journal =. 2025 , url =
2025
-
[23]
Proceedings of the Conference on Innovative Data Systems Research , year =
Learned Cardinalities: Estimating Correlated Joins with Deep Learning , author =. Proceedings of the Conference on Innovative Data Systems Research , year =
-
[24]
2023 , pages =
Jiang, Huiqiang and Wu, Qianhui and Lin, Chin-Yew and Yang, Yuqing and Qiu, Lili , booktitle =. 2023 , pages =. doi:10.18653/v1/2023.emnlp-main.825 , url =
2023 doi
-
[25]
and Stoica, Ion and Gonzalez, Joseph E
Packer, Charles and Wooders, Sarah and Lin, Kevin and Fang, Vivian and Patil, Shishir G. and Stoica, Ion and Gonzalez, Joseph E. , journal =. 2023 , url =
2023
-
[26]
Proceedings of the AAAI Conference on Artificial Intelligence , year =
Safe Reinforcement Learning via Shielding , author =. Proceedings of the AAAI Conference on Artificial Intelligence , year =
-
[27]
2025 , url =
Aggarwal, Pranjal and Welleck, Sean , journal =. 2025 , url =
2025
-
[28]
Token-Budget-Aware
Han, Tingxu and Wang, Zhenting and Fang, Chunrong and Zhao, Shiyu and Ma, Shiqing and Chen, Zhenyu , journal =. Token-Budget-Aware. 2025 , url =
2025
-
[29]
A Survey on Advancing the
Lan, Hai and Bao, Zhifeng and Peng, Yuwei , journal =. A Survey on Advancing the. 2021 , doi =
2021
-
[30]
2025 , url =
Dai, Yaxun and Xie, Wenxuan and Zhuang, Xialie and Yang, Tianyu and Yang, Yiying and Yang, Haiqin and Zhao, Yuhang and Chao, Pingfu and Jiang, Wenhao , journal =. 2025 , url =
2025
-
[31]
Cost Trade-offs of Reasoning and Non-Reasoning Large Language Models in Text-to-
Deochake, Saurabh and Mukhopadhyay, Debajyoti , journal =. Cost Trade-offs of Reasoning and Non-Reasoning Large Language Models in Text-to-. 2025 , url =
2025
-
[32]
2025 , url =
Zhu, Yizhang and Jiang, Runzhi and Li, Boyan and Tang, Nan and Luo, Yuyu , journal =. 2025 , url =
2025
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.