Pith. sign in

REVIEW 4 major objections 4 minor 34 references

Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 7B analytics agent that decides behavior by rules and verifies results after execution outperforms a direct-prompted 32B baseline on business-answer reliability, with most of the gain coming from the deterministic layer rather than…

desk verdict The evaluation framework and honest negative results are the contribution; the headline +0.237 win rests on a test split the author admits was contaminated. read the letter →

arxiv 2608.09254 v1 pith:PPJGS2D3 submitted 2026-08-10 cs.AI cs.CL

Morris Lee This is my paper · ORCID
classification cs.AIcs.CL
keywords text-to-SQLbusinesstruthrateanalyticsagentambiguitydetectionabstentionsemanticlayerpost-executionverificationreliabilitybenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that analytics agents should be judged on whether the business number they return is correct and appropriate, not on whether their SQL matches a reference query. To make that measurable, it introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half of the correct responses are a clarification, abstention, or refusal. It then reports that QueryProof, a 7B agent that decides behavior by deterministic rules over a semantic layer and verifies results after execution, reaches a higher Business Truth Rate than a direct-prompted 32B baseline on an 80-task test split, at lower cost per correct answer. The author's claim is that most of that reliability gain comes from the deterministic layer, not from model size; the paper also discloses that the test ambiguous stratum was exposed during development via contaminated lexicon phrases, so the test figures are a frozen evaluation with prior exposure, not a clean confirmatory test.

What carries the argument

The central object is the deterministic behavior-and-verification layer: an ordered rule set over a semantic layer (metric definitions) and physical catalog (tables, columns, coverage) that decides each terminal behavior, plus static AST validation and post-execution checks on the result. It carries the argument by making the reliability decision auditable and independent of model judgment; the model is used only to propose candidate metric resolutions and SQL, and its outputs are re-checked before they can change the outcome.

What would settle it

Run the same pre-registered protocol on a freshly generated, uncontaminated warehouse with a new seed and no author knowledge of the questions; if the 7B rule-gated agent no longer beats the direct-prompted 32B baseline, or if the advantage falls inside noise when template families are resampled, the central claim is refuted. A second decisive check: scaffold the 32B model with the same semantic-layer rules and verification gates; if it then matches or exceeds the 7B agent, the credited source of the gain shifts from deterministic rules to scale.

Watch

Extended reading notes

Core claim

QueryProof is a state machine that calls a 7B language model twice but never lets the model decide behavior. From a semantic layer and physical catalog, rules choose REFUSE, ABSTAIN, or CLARIFY before the model is asked anything; the model only proposes which metric is meant and what SQL computes it, and every candidate is checked by static validation and post-execution verification. On the 80-task frozen test split, this system scores 0.537 Business Truth Rate routed (0.562 unrouted) against 0.300 for the 32B direct baseline, a paired difference of +0.237 [+0.112, +0.375]; it cuts false success to 0.351 of returned answers and returns no wrong number on an answerable task (0 of 24). The paper does not claim model size explains the gap; it claims the deterministic behavior and verification layer is where the reliability comes from, and it reports that the learned components — routing, confidence model, and post-training — did not transfer.

Load-bearing premise

The load-bearing premise is that the frozen test set is a fair yardstick; the paper discloses that five hand-written synonym rules matched every test question about ambiguous terms and could only have been written after seeing the held-out questions, so the +0.237 gap may partly reflect prior knowledge rather than the system's design.

Editorial extensions

If this is right

  • If the paper is right, text-to-SQL evaluation that relies on execution-match accuracy alone will overstate production readiness, because the expensive failures are wrong business numbers returned by valid SQL.
  • The measured gain is concentrated in strata where the correct response is not an answer; on the 32 standard questions the systems are near parity, so benchmark composition materially decides the apparent advantage.
  • A deterministic verification layer can absorb much of the reliability headroom that post-training would target: the pre-registered gate was not met because the remaining model-capability failure pool was 20 against a threshold of 100.
  • Learned reliability components may not transfer: routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced on both calibration and selective prediction.
  • The direction of the effect is better supported than its magnitude: resampling template families rather than tasks widens both accuracy intervals to include zero, so the ordering is evidence but the size of the benefit is poorly determined.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a component ablation is the decisive missing experiment; wrapping the 32B model in the same scaffold, or stripping the 7B agent of its verification gates, would cleanly separate scale from scaffolding.
  • Editorial inference: because the test ambiguous stratum was exposed during development, the benchmark's confirmatory value is limited; a pre-registered replication on a fresh seed-generated warehouse with no author knowledge would settle whether the direction of the effect survives.
  • Editorial inference: if the gain really lives in the deterministic layer, the practical bottleneck shifts to maintaining the semantic layer over time; organizations with unstable metric definitions may not reproduce the result without equivalent curation effort.
  • Editorial inference: the cost-per-correct-answer framing could be extended to count a correct abstention or refusal as a paid outcome, which would reward appropriate caution rather than only correct SQL.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces WarehouseReliabilityBench (WRB), a 400-task synthetic benchmark over two warehouses in which roughly half of the correct responses are clarifications, abstentions, or refusals, and defines reliability-focused metrics (Business Truth Rate, False Success Rate, coverage, cost per correct answer) with pinned denominators and a pre-registered paired bootstrap. It also presents QueryProof, a 7B analytics agent that uses a deterministic rule set over a semantic layer and physical catalog to decide behavior, and gates answers on static and post-execution verification. On an 80-task test split evaluated once, QueryProof is reported to outperform a direct-prompted 32B baseline by +0.237 in Business Truth Rate and to cut false success to 0.351 of returned answers. The paper also reports negative results: the routing layer does not transfer from validation to test, the fitted confidence model is worse than the heuristic it replaced, and post-training was declined by a pre-registered gate. The manuscript is transparent about its limitations, including test-split contamination, single-author labeling, and weak family-level bootstrap intervals.

Significance. If the empirical claims were reliable, this paper would make a useful contribution by demonstrating that deterministic semantic-layer and verification machinery can substantially improve the reliability of small analytics models on tasks that require clarification, abstention, or refusal, and by providing a benchmark and evaluation framework for such tasks. The paper is notable for its strengths: a real pre-registered protocol, a one-run rule on the test split, fully committed and replayable scoring artifacts, explicit denominator definitions for every metric, and unusually candid reporting of negative results and limitations. These strengths make the benchmark and the evaluation framework potentially reusable by the community. However, the central comparative result is undermined by the disclosed test-split contamination, and the family-level bootstrap intervals include zero, so the headline effect is not confirmed as a general result.

major comments (4)
  1. [Section 3.5 and Section 6.1] The central comparison is not a valid confirmatory test because the test split was exposed during development. The paper admits that five hand-authored lexicon phrases matched the entire 16-task test ambiguous stratum and could only have been written from the held-out split. Table 4 shows that QueryProof's advantage over the 32B baseline is concentrated in the ambiguous and unanswerable strata (0.250 vs 0.000 and 0.438 vs 0.000, respectively), exactly the strata whose labels the contaminated lexicon encoded. Removing the strings before the run does not undo the potential influence of prior test knowledge on taxonomy, rule design, or semantic-layer choices. The abstract's '+0.237 [+0.112, +0.375]' therefore should not be reported as an 'outperforms' result unless either replicated on an uncontaminated split or explicitly reframed as a hypothesis-generating observation from a single frozen evaluation with known exposure.
  2. [Section 6.3 / Table 8] The family-level resampling interval includes zero for both headline paired comparisons (+0.237 [−0.125, +0.562] against 32B; +0.262 [−0.075, +0.575] against few-shot). The text acknowledges that the task-level interval is optimistic and that the cluster bootstrap over 10 families is itself unreliable, but it nevertheless concludes that 'the direction of the effect is supported.' Given the acknowledged optimism of the task-level interval and the unreliability of the 10-cluster bootstrap, the evidence does not establish a directional effect. The paper should either provide a more reliable interval (for example, by generating more template families) or restrict the claim to a point estimate on this specific split.
  3. [Section 7 and Table 10] The claim that 'the gain tracks the deterministic layer' is not supported by the reported experiments. No component ablation is run; the only ablation (routing on/off, Table 10) shows the routed variant slightly underperforms the base variant on test (0.537 vs 0.562). The comparison between QueryProof and the baselines confounds the semantic layer, the verification checks, the repair loop, and the model prompting scheme, so the difference cannot be attributed to any specific deterministic component. The abstract and conclusion should either report that the attribution is currently unsupported by direct experimental evidence or add an ablation that isolates the deterministic components.
  4. [Section 3.4 and Section 7] The benchmark's labels, task taxonomy, semantic layer, and rules were all authored by a single person, and the re-adjudication (Table 1, κ = 0.887) is a test–retest measurement, not inter-annotator agreement. The paper states this limitation, but its central claim depends on the benchmark's validity as a fair measurement; single-author co-design plus the disclosed leakage means the benchmark has not been independently validated. A second annotator, or a review by a separate expert, would be needed to support the external-validity claims that the comparison implicitly makes.
minor comments (4)
  1. [Title and Abstract] The title and abstract emphasize '7B' versus '32B,' but the paper correctly states that the comparison is between systems rather than model sizes. Consider rewording the title to avoid implying a model-size comparison, since the 32B baseline receives none of the scaffolding.
  2. [Table 1] The column header 'Agreed n' is ambiguous; clarify that it is the number of labels on which the two adjudication passes agreed, and note that 'Agreement' is the raw proportion.
  3. [Section 5.3] The pre-registration is described as fixing artifact hashes and claim verbs, but the paper does not state whether the pre-registration document is publicly time-stamped or deposited; if it is, provide a link or repository path for verification.
  4. [Appendix A, Table 5] The 'Phase' column values (e.g., '6', '10', '11') are not defined in the table; consider adding a short explanation of what a phase is or aligning the labels with the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the comparison is empirical and the paper's disclosed threats are benchmark-validity concerns, not derivation steps that reduce to their own inputs.

full rationale

The paper's central claims are measured comparisons on a frozen, pre-registered test split: QueryProof's Business Truth Rate, false-success rate, and cost per correct answer are computed by explicit formulas (Appendix E) against a fixed correctness contract, not derived from the agent's own outputs. The rule-gated behavior decision uses the semantic layer and physical catalog, while the benchmark's labels are set independently by task design and re-adjudicated; there is no equation or fitted parameter that is renamed as a prediction. The disclosed contaminated lexicon and one-person co-design are genuine validity threats to the benchmark as a confirmatory test, but they are data-leakage and independence concerns, not circularity in the derivation chain. The paper also reports negative results (routing failing to transfer, the fitted confidence model losing to the heuristic), which is inconsistent with a claim that the conclusions are forced by construction. No self-citation is load-bearing, no uniqueness theorem is imported, and no known result is merely renamed. A score of 0 is appropriate: the derivation is self-contained as an empirical evaluation, with the paper's own limitations properly identifying why external replication is still needed.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central comparison rests on the benchmark being a fair and meaningful measurement. The warehouse, task taxonomy, semantic layer, rules, labels, and harness were all authored by one person, so co-design can favor QueryProof in ways no single check detects. Executable ground truth anchors answerable tasks, and the pre-registered frozen split with a one-run rule is a strong safeguard, but the disclosed test contamination and single-reviewer labels mean the benchmark cannot be treated as an independent external standard.

free parameters (3)
  • Routing confidence threshold = 0.918 operating point, selected on validation
    Selected on the validation split; over-abstains on test. Not central to the headline BTR because the base agent performs similarly, but it is a tuned parameter.
  • Logistic confidence model weights = Not reported numerically; fitted on development split
    L2-regularized logistic regression over join count, SQL length, repair attempts, and verification-issue count; fitted on dev and lost to the heuristic it replaced on test.
  • Amortized hardware cost rate = approximately $0.50 per hour
    Assumed cost basis for local generation; CPCA depends on this assumed rate, though relative comparisons are less sensitive to it.
assumptions (5)
  • domain assumption The synthetic warehouses, generated from a single pinned seed, are a meaningful proxy for production analytics workloads.
    Section 3.1: external validity is traded for executable ground truth; transfer to real warehouses is untested.
  • domain assumption Behavior contracts and result equivalence rules operationalize business truth.
    Section 3.3: column aliases excluded, column count included; no human study shows analysts can act on clarifications.
  • domain assumption Single-author labeling is stable enough for benchmark labels, with test-retest agreement 0.920 and kappa 0.887.
    Section 3.4: this is not inter-annotator agreement; all disagreements fell in one family and single-reviewer bias remains.
  • domain assumption Post-execution verification checks are a valid proxy for correctness despite measured false-clean rates.
    Section 4.3: false-clean rates were 0.163 on dev and 0.333 on validation; passing checks raises but does not establish correctness.
  • domain assumption Temperature-zero greedy decoding on Qwen2.5-Coder is deterministic enough for like-for-like comparison.
    Section 5.1 and Appendix B: generation reproducibility is not guaranteed across backends or hardware.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline." pith.science (2026). https://pith.science/paper/PPJGS2D3

@misc{pith2026260809254,
  author       = {Pith},
  title        = {Pith review of: Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPJGS2D3}},
  note         = {Machine review of arXiv:2608.09254}
}
read the original abstract

LLM analytics agents are evaluated on SQL syntax accuracy, but production failures look different: questions with two valid business definitions, questions the warehouse cannot answer, deprecated columns after a schema change, and queries that execute successfully while returning the wrong business number. No execution-match metric can score them. This paper introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal, with pinned denominators and a pre-registered paired bootstrap fixing each claim verb before the numbers existed. QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour, and gates every answer on deterministic post-execution checks. On an 80-task synthetic test split evaluated once, QueryProof outperforms a direct-prompted 32B baseline by +0.237 [+0.112, +0.375] Business Truth Rate at 71.0% lower cost per correct answer; against a cost-matched few-shot baseline the accuracy gain holds but the cost difference does not resolve. This compares systems rather than model sizes: the 32B baseline receives none of the scaffolding. False success falls from 0.754 to 0.351 of returned answers, and no wrong number was returned on an answerable task (0 of 24), though 13 answers went to questions requiring clarification or abstention. Removing the routing layer changes little (0.562 against 0.537), so the result does not depend on escalation. Routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced. Resampling template families rather than tasks widens both accuracy intervals to include zero, so the effect's direction is better supported than its magnitude. The gain tracks the deterministic layer, though no component ablation was run.

Figures

Figures reproduced from arXiv: 2608.09254 by the authors.

Figure 1
Figure 1. F4. WRB composition by category, domain and split. Splits are partitioned by template family, which is why the strata are uneven: the test ambiguous stratum holds two e-commerce families and no SaaS. Of the 400 tasks, 184 are answerable and carry an ex￾ecutable accepted result; the rest are scored against a behaviour contract. Schema drift A column the obvious query would use has been deprecated or renamed. The requ… view at source ↗
Figure 2
Figure 2. The decision ordering. Methods figure, not a result. Shape carries authority: box = rule decided from the semantic layer and the physical catalog, ellipse = model call, diamond = terminal behaviour. Every terminal behaviour is selected by a rule before the model is asked for anything, and an escalated answer is re￾checked by the same gates. The complete state machine, including the repair loop and the individual val… view at source ↗
Figure 3
Figure 3. F1. Business Truth Rate against cost per [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: F5. Every test outcome by kind, per system. Hatching distinguishes failure kinds without relying on [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: F2. Negative result. Risk–coverage for the fitted confidence model against the Phase 6 heuristic, on the same answer pool. Lower AURC is better. 0.0 0.2 0.4 0.6 0.8 1.0 Reported confidence 0.0 0.2 0.4 0.6 0.8 1.0 Empirical accuracy n=37 n=15 n=1 n=25 n=65 Pool: every r…
Figure 6
Figure 6. Figure 6: F3. Reliability diagram over 37 returned answers, with the ten bins fixed before the test run. Marker size [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: The complete agent state machine, of which Figure [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 13 canonical work pages

  1. [5]

    Text-to-

    Gao, Dawei and Wang, Haibin and Li, Yaliang and Sun, Xiuyu and Qian, Yichen and Ding, Bolin and Zhou, Jingren , journal =. Text-to-. 2024 , doi =. 2308.15363 , archivePrefix =

    arXiv 2024

  2. [6]

    Robust Text-to-

    Wang, Chenglong and Tatwawadi, Kedar and Brockschmidt, Marc and Huang, Po-Sen and Mao, Yi and Polozov, Oleksandr and Singh, Rishabh , journal =. Robust Text-to-. 2018 , eprint =

    work page 2018

  3. [10]

    2024 , eprint =

    Lee, Gyubok and Chay, Woosog and Cho, Seonhee and Choi, Edward , journal =. 2024 , eprint =

    work page 2024

  4. [14]

    Journal of Machine Learning Research , volume =

    On the Foundations of Noise-free Selective Classification , author =. Journal of Machine Learning Research , volume =. 2010 , url =

    work page 2010

  5. [18]

    Proceedings of the National Academy of Sciences , volume =

    The preregistration revolution , author =. Proceedings of the National Academy of Sciences , volume =. 2018 , doi =

    work page 2018

  6. [19]

    2024 , eprint =

    Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Dang, Kai and Yang, An and Men, Rui and Huang, Fei and Ren, Xingzhang and Ren, Xuancheng and Zhou, Jingren and Lin, Junyang , journal =. 2024 , eprint =

    work page 2024

  7. [21]

    1993 , isbn =

    An Introduction to the Bootstrap , author =. 1993 , isbn =

    work page 1993

  8. [22]

    dbt Semantic Layer , author =

    work page

Show all 34 references
  1. [23]

    Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.436 Benchmarking and improving text-to- SQL generation under ambiguity . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    2023 doi

  2. [24]

    Cube Dev, Inc. 2026. https://docs.cube.dev/introduction Cube : Introduction . Cube documentation. Accessed 2026-08-09

    2026

  3. [25]

    dbt Labs . 2026 a . https://docs.getdbt.com/docs/build/about-metricflow About MetricFlow . dbt Developer Hub documentation. Accessed 2026-08-09

    2026

  4. [26]

    dbt Labs . 2026 b . https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl dbt semantic layer . dbt Developer Hub documentation. Accessed 2026-08-09

    2026

  5. [27]

    Tibshirani

    Bradley Efron and Robert J. Tibshirani. 1993. https://doi.org/10.1201/9780429246593 An Introduction to the Bootstrap . Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, New York

    1993 doi

  6. [28]

    Ran El-Yaniv and Yair Wiener. 2010. https://www.jmlr.org/papers/v11/el-yaniv10a.html On the foundations of noise-free selective classification . Journal of Machine Learning Research, 11(53):1605--1641

    2010

  7. [29]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. https://doi.org/10.14778/3641204.3641221 Text-to- SQL empowered by large language models: A benchmark evaluation . Proceedings of the VLDB Endowment, 17(5):1132--1145

    2024

  8. [30]

    Yonatan Geifman and Ran El-Yaniv. 2017. https://arxiv.org/abs/1705.08500 Selective classification for deep neural networks . In Advances in Neural Information Processing Systems 30 (NIPS 2017)

    2017 arXiv

  9. [31]

    Google Cloud . 2026. https://cloud.google.com/looker/docs/what-is-lookml Introduction to LookML . Looker documentation. Accessed 2026-08-09

    2026

  10. [32]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. https://arxiv.org/abs/1706.04599 On calibration of modern neural networks . In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 13...

    2017 arXiv

  11. [33]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. https://arxiv.org/abs/2409.12186 Qwen2.5-Coder technical report ...

    2024 arXiv

  12. [34]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  13. [35]

    Gyubok Lee, Woosog Chay, Seonhee Cho, and Edward Choi. 2024. https://arxiv.org/abs/2403.15879 TrustSQL : Benchmarking text-to- SQL reliability with penalty-based scoring . arXiv preprint arXiv:2403.15879

    2024 arXiv

  14. [36]

    Wang, and Tao Yu

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida I. Wang, and Tao Yu. 2025. https://arxiv.org/abs/2411.07763 Spider 2.0: Evaluating langua...

    2025 arXiv

  15. [37]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. https://arxiv.org/abs/2305.03111 Can LLM already serve as a ...

    2023 arXiv

  16. [38]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://arxiv.org/abs/2205.14334 Teaching models to express their uncertainty in words . Transactions on Machine Learning Research

    2022 arXiv

  17. [39]

    Malloy Data . 2026. https://www.malloydata.dev/ Malloy : A modern open source language for analyzing, transforming, and modeling data . Project website and documentation. Accessed 2026-08-09

    2026

  18. [40]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. https://doi.org/10.1609/aaai.v29i1.9602 Obtaining well calibrated probabilities using bayesian binning . In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI), volume 29, pages...

    2015 doi

  19. [41]

    Nosek, Charles R

    Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. https://doi.org/10.1073/pnas.1708274114 The preregistration revolution . Proceedings of the National Academy of Sciences, 115(11):2600--2606

    2018 doi

  20. [42]

    Mohammadreza Pourreza and Davood Rafiei. 2023. https://arxiv.org/abs/2304.11015 DIN - SQL : Decomposed in-context learning of text-to- SQL with self-correction . In Advances in Neural Information Processing Systems 36 (NeurIPS 2023)

    2023 arXiv

  21. [43]

    Mark Raasveldt and Hannes M \"u hleisen. 2019. https://doi.org/10.1145/3299869.3320212 DuckDB : an embeddable analytical database . In Proceedings of the 2019 International Conference on Management of Data (SIGMOD '19), pages 1981--1984, Amsterdam, Netherlands. ACM

    2019

  22. [44]

    Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. 2023. https://doi.org/10.18653/v1/2023.findings-acl.352 Know what I don ' t know: Handling ambiguous and unknown questions for text-to- SQL . In Findings of the Association for Computational Linguistics: ACL 2023, pages 5701-...

    2023 doi

  23. [45]

    Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. https://arxiv.org/abs/1807.03100 Robust text-to- SQL generation with execution-guided decoding . arXiv preprint arXiv:1807.03100

    2018 arXiv

  24. [46]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. https://doi.org/10.18653/v1/D18-1425 S pider: A large-scale human-labeled dataset for complex and cross-domain seman...

    2018 doi

  25. [47]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging LLM -as-a-judge with MT -bench and chatbot arena ....

    2023 arXiv

  26. [48]

    Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.29 Semantic evaluation for text-to- SQL with distilled test suites . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396--411, Online....

    2020 doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.