Pith. sign in

REVIEW 2 major objections 5 minor 56 references

Standard text-to-SQL benchmarks ignore role-based access control, so high execution-accuracy scores conceal frequent unauthorized queries; the paper builds an RBAC-aware benchmark and shows many leading LLMs, especially open-weight ones, vi

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-01 05:44 UTC pith:25OMC6DI

load-bearing objection Useful RBAC-aware text-to-SQL benchmark with a real result, but the deny labels tie 'deny' to the gold SQL's exact footprint, so absolute violation rates are upper bounds. the 2 major comments →

arxiv 2607.22115 v1 pith:25OMC6DI submitted 2026-07-24 cs.DB cs.AI

Benchmarking Text-to-SQL under Role-Based Access Control

Yang Fei , Yangfan Jiang , Yin Yang , Xiaokui Xiao This is my paper
classification cs.DB cs.AI
keywords text-to-SQLrole-based access controlbenchmarkingexecution accuracySafe-EXLLM evaluationaccess control complianceRBAC violation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that current text-to-SQL benchmarks mislead practitioners because they assume unrestricted database access. The authors construct a benchmarking framework that layers realistic user roles and column-level access policies onto Spider, BIRD, and LiveSQLBench, producing 21,502 RBAC-annotated instances. They define new metrics — Safe-EX, violation rate, over-refusal rate, and AC-F1 — that separate SQL correctness from policy compliance. Across 14 models, they find that strong execution accuracy in the unrestricted setting does not transfer to access-controlled deployments: open-weight models in particular issue RBAC-violating SQL on a large share of instances, and common mitigations like schema hiding, in-context learning, and supervised fine-tuning do not reliably fix compliance. The paper's central claim is that access-control alignment must be a first-class evaluation dimension, not an afterthought left to a DBMS-side checker.

Core claim

The paper claims that a generated SQL query can be execution-correct yet still be an RBAC violation, and that this failure mode is common enough to invalidate standard execution accuracy as a proxy for deployability. Empirically, Llama3-SQLCoder-8b reaches 58.70 execution accuracy on unrestricted Spider but produces RBAC-violating SQL on 46.5% of RBAC instances; Snowflake-R1-7b reaches 41.88 on BIRD with a 63.8% violation rate; and on LiveSQLBench the same model has a 76.2% violation rate with only 5.91 execution accuracy on allowed instances. The authors further claim that restricting schema visibility to role-permitted columns reduces some explicit data leakage but does not enforce complia

What carries the argument

The core object is the RBAC-conditioned instance: a natural-language question paired with a role, a serialized column-operation-level policy, and a ground-truth allow/deny label computed by parsing the gold SQL with SQLGlot and checking whether its required permission set is a subset of the role's policy. Roles are synthesized by an LLM through a structured reasoning pipeline (context inference, responsibility definition, permission derivation) with automatic metrics and human validation. The evaluation hinges on Safe-EX, the fraction of ground-truth allowed instances where the system returns SQL that is both execution-correct and policy-compliant, alongside violation rate, over-refusal rate

Load-bearing premise

The deny labels are derived from the gold SQL alone: a question is labeled denied if the reference query touches a forbidden column, even if another fully compliant SQL query could answer the same question.

What would settle it

Take any denied instance and construct a policy-compliant SQL query that answers the question; score a system that produces that compliant query. If Safe-EX and violation rates shift materially when ground truth is 'answerable within policy' rather than 'gold SQL permitted,' then the benchmark's rankings measure adherence to a single access plan, not access-safe capability.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Standard execution accuracy overstates the deployability of text-to-SQL systems in access-controlled databases; RBAC-aware metrics are needed to measure real-world safety.
  • Open-weight text-to-SQL models, even those specialized for SQL generation, show systematically higher violation rates than commercial models, indicating a training-objective gap.
  • Schema hiding and role-aware prompting are insufficient defenses; models continue to hallucinate unauthorized columns and rarely refuse when they should.
  • Supervised fine-tuning on one RBAC dataset improves in-domain compliance but converts the model into a risk-averse refuser that does not generalize across domains.
  • The refusal-cliff pattern — correct reasoning about denial followed by non-compliant SQL output — suggests that policy adherence must be coupled into decoding or generation, not just prompted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The allow/deny ground truth is keyed to the gold SQL's specific access plan, so the benchmark measures whether a model reproduces that plan, not whether a question is answerable within the role's policy; a compliant alternative query would still be scored as a violation.
  • The refusal-cliff finding hints that decoding-time interventions — such as conditioned decoding that suppresses SQL tokens after a deny decision — could be more effective than prompt engineering or fine-tuning, a testable extension the paper does not pursue.
  • The framework's construction naturally extends to row-level predicates and to inference-based leakage, but each extension changes the character of the task; row-level checks align better with real data-masking deployments, while inference control would require policy models beyond RBAC.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a benchmarking framework for text-to-SQL under role-based access control (RBAC). It augments three existing text-to-SQL benchmarks (Spider, BIRD, LiveSQLBench) with LLM-synthesized roles and column/operation-level policies, using a human-in-the-loop validation pipeline, and derives allow/deny ground truth by checking whether the permissions required by the gold SQL are contained in the role policy. It then defines new metrics (Safe-EX, violation rate, over-refusal rate, AC-F1) and reports a systematic evaluation of fourteen models. The headline finding is that many systems, especially open-weight ones, show strong unrestricted execution accuracy but high RBAC violation rates once access constraints are present, and that common mitigations—schema hiding, in-context learning, and SFT—do not reliably fix the problem.

Significance. If the central claim is accurate, the paper identifies a real and practically important failure mode that existing text-to-SQL benchmarks miss: standard execution accuracy can overstate deployability in access-controlled databases. The empirical protocol is generally careful: 5-seed stratified sampling with standard deviations, cross-checking of unrestricted EX against public numbers, ablations such as removing DataOperator roles and comparing Full- vs Role-schema exposure, and a public release of the pipeline and datasets. However, the absolute violation rates that drive the central claim are computed from deny labels derived solely from the gold SQL. Because the paper does not quantify how many deny instances are actually answerable within the role policy by a compliant alternative query, the magnitude of the headline degradation is uncertain. The benchmark and metrics are nevertheless a useful contribution if the label-validity issue is addressed; the paper's core idea is worth publishing, but the current evidence does not yet support the strong quantitative claims as stated.

major comments (2)
  1. [§2.1, §3.3] The ground-truth label is y=allow iff Perm(Y*)⊆Π_r (§2.1), and §3.3 then treats any deny instance as requiring a refusal. This identifies "the gold SQL is outside the role" with "the question is not answerable within the role." A model that outputs a policy-compliant SQL answering Q but not matching Y* is scored as a violation (VC) and lowers AC-F1. The paper states this follows "the standard assumption" that gold SQL is a complete operational specification, but that assumption is exactly what is at stake for RBAC: the benchmark measures alignment with one specific access plan, not the general property that the question is answerable within policy. The headline numbers in Table 3 (e.g., Snowflake-R1-7b: 63.8% violation on BIRD, 76.2% on LiveSQLBench; Llama3-SQLCoder: 46.5% on Spider) are upper bounds if any non-trivial fraction of deny instances are answerable within policy. Since the ce
  2. [§4.1] The outcome-space and metric definitions inherit the same label-validity issue. In particular, the "Violation correct (VC)" category is defined as "incorrectly generated with execution-correct SQL (RBAC-rejected success)" even when the generated SQL is fully RBAC-compliant and answers the user's question; it is labeled a violation solely because the gold SQL is not authorized. Consequently, violation rate and AC-F1 treat a system that finds a legitimate compliant alternative as a false positive. Safe-EX is less affected because it is computed only over ground-truth allowed instances, but the metrics used to support the main empirical conclusion are not robust to the gold-SQL-minimality assumption. I ask the authors to report a sensitivity analysis—for example, by manually labeling a subsample of deny instances for answerability within policy, or by excluding instances where the denied co
minor comments (5)
  1. [Table 3] The column headers under "Avg. Cost/Task" are ambiguous: "w/o role" and "w/ role" could be read as applying to EX and Safe-EX rather than to cost. Please clarify with explicit subheaders or a note.
  2. [Figure 2] The legend for marker size and color is dense; consider a separate panel or a clearer caption explaining that size encodes accessible-column proportion and color encodes semantic-similarity bucket.
  3. [Table 6] The counts in columns such as #ValidG, #InvalidG, and #FailtoRefuse are raw counts averaged over five passes. Expressing them as rates (or reporting the base number of deny instances) would make the Full-Schema vs Role-Schema comparison easier to interpret.
  4. [§3.3] For denied instances, the expected output is "a standardized denial response." This design choice should be justified more explicitly against the alternative of allowing a policy-compliant reduced query, since that alternative is exactly what the deny-label audit in the major comments concerns.
  5. [References] The GitHub repository [46] should include a version/commit identifier or DOI so that the released artifacts are citable and reproducible as of the paper's publication date.

Circularity Check

0 steps flagged

No significant circularity: the headline RBAC violation and Safe-EX numbers are measured model behaviors against externally authored gold SQL and deterministic SQLGlot label parsing; no fitted parameter is renamed as a prediction.

full rationale

The paper's derivation chain is empirically self-contained. Ground-truth labels are computed deterministically: Section 2.1 defines y = allow iff Perm(Y*) is contained in the role policy (SQLGlot parsing plus set inclusion), and the roles/policies are LLM-synthesized but pass 3/4-annotator human screening plus deterministic quality filters (Section 3.2). The headline findings (Llama3-SQLCoder 46.5% violation on Spider; Snowflake-R1 63.8% on BIRD; 76.2% on LiveSQLBench) are measured outputs of the evaluated models under identical prompts, not values fitted from the labels. The same definition yields both 1.97% (DeepSeek-V3.2-Reasoner on Spider) and 76.15% violation (Snowflake-R1 on LiveSQLBench), so the metric discriminates genuine behavior and is not forced by construction. The weakest point is explicitly flagged by the authors: the deny label is derived from the gold SQL alone ('we treat the gold SQL as a correct and complete operational specification of the user's intended data access behavior'), which conflates 'question not answerable within policy' with 'gold SQL not permitted'. This could inflate absolute violation magnitudes (a construct-validity caveat the paper does not quantify), but it is a disclosed modeling assumption, not a by-construction reduction of a predicted quantity to an input: the model outputs are independent observables. Self-citation is not load-bearing: reference [46] is the artifact repository; [29] (a co-authored background citation on prompt-inference attacks) and [49] (refusal-cliff work, different authors) do not carry the central argument. The proposed metrics (Safe-EX, violation rate, AC-F1, six-category outcomes) are new operationalizations rather than renamings of a known result. Honest non-finding; score near the bottom of the 0-2 band.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

Central claims rest on three external inputs: gold SQL from the source benchmarks (assumed to define required access), LLM-synthesized role policies vetted by deterministic screens and four annotators (assumed realistic), and a deterministic SQL parser (SQLGlot) for permission extraction. No new physical entities are postulated; DataOperator roles, Safe-EX, AC-F1, and the six-category outcome space are benchmark constructs rather than entities of the kind this ledger tracks. Hand-set thresholds shape the allow/deny mix and thus absolute violation rates, but the headline finding — models frequently generate SQL for deny-labeled instances — is stable across benchmarks and models.

free parameters (4)
  • DataOperator sampling parameters = p_star=0.5, p_col=0.7, column cap=0.9, n=3 (n=2 for <3 tables), seed=42
    Appendix B, Algorithm 1. Hand-chosen to balance star/column access; cap prevents full coverage. These fix the deny-case distribution and hence the difficulty of every RBAC metric, without being fitted to model outputs.
  • Policy-quality screening thresholds = deny-rate bounds 5%/90%; max pairwise role overlap 0.8; semantic-similarity auxiliary threshold 0.60
    Appendix A.1. Hand-set acceptance criteria encoding the authors' definition of 'realistic, non-degenerate' roles; looser or tighter thresholds would change role coverage and deny rates.
  • Evaluation protocol constants = k=5 seeds, temperature 0, one random role per query per pass
    Section 4.2. Protocol choices that determine the variance of reported means; each query is judged against only a subset of roles per pass.
  • SFT hyperparameters = QLoRA rank 64, lr 2e-4, 8 epochs, 4-bit, bf16
    Appendix C, Table 9. Pertain to the secondary fine-tuning study; a different budget could change the remedy-effectiveness conclusions.
axioms (4)
  • domain assumption The gold SQL defines the complete and minimal set of permissions needed to answer the question.
    Section 2.1: 'we treat the gold SQL as a correct and complete operational specification of the user's intended data access behavior.' Load-bearing: every allow/deny label in the 21,502-instance dataset derives from it.
  • domain assumption SQLGlot parsing of reference SQL recovers all required (table, column, operation) triples.
    Section 3.3: permission extraction is 'deterministic' via SQLGlot with special handling of JOIN/GROUP BY/ORDER BY/predicates; no audit is reported for CTEs, window functions, or views, so under-extraction would mislabel deny as allow.
  • domain assumption LLM-synthesized roles screened by metrics and approved by 3/4 annotators are a faithful proxy for real-world RBAC policies.
    Sections 3.2 and A.3: realism of roles rests on one pipeline whose LLM backend is unnamed, four credentialed annotators reviewing 53 database-level configurations, and embedding-based semantic checks.
  • domain assumption Cosine similarity between role text and schema text is a meaningful alignment signal.
    Section 3.2 and Figure 2: used as a proxy to reject incoherent roles; the 0.60 threshold is auxiliary and experts make the final call, but the proxy still shapes feedback-driven regeneration.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Text-to-SQL under Role-Based Access Control." pith.science (2026). https://pith.science/paper/25OMC6DI

@misc{pith2026260722115,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Text-to-SQL under Role-Based Access Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25OMC6DI}},
  note         = {Machine review of arXiv:2607.22115}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Given a database S and a natural language question Q, text-to-SQL systems aim to generate an SQL query that correctly answers Q when executed against S. Currently, popular text-to-SQL benchmarks mostly assume unrestricted access to S; in practice, however, user access is often restricted, e.g., through role-based access control (RBAC) policies. This leads to a potential disconnect between benchmarking results and real-world performance: an LLM with high benchmark scores might perform poorly in an access-controlled environment, by frequently violating RBAC, or rejecting a query q that could be answered with only permitted data in S. Motivated by this, we present a comprehensive text-to-SQL benchmarking framework with realistic RBAC constraints, which features an LLM-assisted workflow that augments existing text-to-SQL benchmarks with plausible user roles and access policies. To do so, we formulate the problem of role synthesis as a structured reasoning process over the database schema, in which the LLM first infers the application context from the schema, and then derives role responsibilities and access scopes consistent with this context. This process is audited by human-in-the-loop quality control, in which domain experts perform metric-guided screening on the generated roles. Besides the augmented dataset, the proposed framework also contains evaluation metrics that identify RBAC-specific failure modes, and disentangle SQL utility from access-control compliance. We apply the proposed framework to several widely-used benchmarks, and conduct a systematic empirical study of state-of-the-art text-to-SQL systems. The results show that many solutions (especially open-weight LLMs) with high benchmarking scores under an unrestricted setting suffer sharp performance degradation once access constraints are in place, due to frequent RBAC violations.

Figures

Figures reproduced from arXiv: 2607.22115 by Xiaokui Xiao, YangFan Jiang, Yang Fei, Yin Yang.

Figure 1
Figure 1. Figure 1: Overview of the proposed RBAC-aware dataset construction and benchmarking pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Role Coverage and Semantic Alignment. Each row (e.g., wta_1) corresponds to a database. Each circular marker [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Query distribution and difficulty. Stacked bars show [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: A case study illustrating RBAC failures in reasoning. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Analysis of heuristic remedies [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pipeline of data generation with human in the loop. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: A concrete challenge-level RBAC SELECT task. Difficulty annotations are present for both Spider and BIRD. Spider categorized task difficulty into four levels: easy, medium, hard, and extra hard, based on the complexity of SQL components, such as the number of SELECT columns, WHERE conditions, use of GROUP BY, nested subqueries, and advanced operations like EXCEPT or INTERSECT. BIRD provides a difficulty la… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 9 linked inside Pith

  1. [1]

    Saeed Almheiri, Yerulan Kongrat, Adrian Santosh, Ruslan Tasmukhanov, Jose- maria Loza Vera, Muhammad Dehan Al Kautsar, and Fajri Koto. 2025. Role-Aware Language Models for Secure and Contextualized Access Control in Organizations. arXiv preprint arXiv:2507.23465(2025)

    arXiv 2025

  2. [2]

    Anthropic. 2025. System Card: Claude Opus 4 & Claude Sonnet 4. https://www- cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf

    2025

  3. [3]

    BIRD-SQL Team. 2025. LiveSQLBench-base-full-v1. https://huggingface.co/ datasets/birdsql/livesqlbench-base-full-v1. Hugging Face Datasets

    2025

  4. [4]

    Shuaichen Chang and Eric Fosler-Lussier. 2023. Selective Demonstrations for Cross-domain Text-to-SQL. InFindings of EMNLP. 14174–14189

    2023

  5. [5]

    Kaiwen Chen, Yueting Chen, Nick Koudas, and Xiaohui Yu. 2025. Reliable Text- to-SQL with Adaptive Abstention. InSIGMOD. 1–30

    2025

  6. [6]

    Kakkar, Yu Gan, Brenton Milne, and Fatma Özcan

    Yeounoh Chung, Gaurav T. Kakkar, Yu Gan, Brenton Milne, and Fatma Özcan

  7. [7]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Defog. 2024. llama-3-sqlcoder-8b. https://huggingface.co/defog/llama-3- sqlcoder-8b/

    2024

  9. [9]

    Ju Fan, Zihui Gu, Songyue Zhang, Yuxin Zhang, Zui Chen, Lei Cao, Guoliang Li, Samuel Madden, Xiaoyong Du, and Nan Tang. 2024. Combining small language models and large language models for zero-shot NL2SQL.PVLDB17, 11 (2024), 2750–2763

    2024

  10. [10]

    Yuankai Fan, Zhenying He, Tonghui Ren, Can Huang, Yinan Jing, Kai Zhang, and X Sean Wang. 2024. Metasql: A generate-then-rank framework for natural language to sql translation. InICDE. 1765–1778

    2024

  11. [11]

    2003.Role- based access control

    David Ferraiolo, D Richard Kuhn, and Ramaswamy Chandramouli. 2003.Role- based access control. Artech house

    2003

  12. [12]

    William Fleshman, Aleem Khan, Marc Marone, and Benjamin Van Durme. 2024. AdapterSwap: Continuous Training of LLMs with Data Removal and Access- Control Guarantees.arXiv preprint arXiv:2404.08417(2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Han Fu, Chang Liu, Bin Wu, Feifei Li, Jian Tan, and Jianling Sun. 2023. Catsql: Towards real world natural language to sql applications.PVLDB16, 6 (2023), 1534–1547

    2023

  14. [14]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation.PVLDB17, 5 (2024), 1132–1145

    2024

  15. [15]

    Zihui Gu, Ju Fan, Nan Tang, Lei Cao, Bowen Jia, Sam Madden, and Xiaoyong Du. 2023. Few-shot text-to-sql translation using structure and content prompt learning. InSIGMOD. 1–28

    2023

  16. [16]

    Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-generation database interfaces: A survey of llm-based text-to-sql.IEEE TKDE(2025)

    2025

  17. [17]

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2024. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. InCOLM

    2024

  18. [18]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    Pith/arXiv arXiv 2024

  19. [19]

    Bargav Jayaraman, Virendra J Marathe, Hamid Mozaffari, William F Shen, and Krishnaram Kenthapadi. 2025. Permissioned LLMs: Enforcing Access Control in Large Language Models.arXiv preprint arXiv:2505.22860(2025)

    arXiv 2025

  20. [20]

    Hyeonji Kim, Byeong-Hoon So, Wook-Shin Han, and Hongrae Lee. 2020. Natural language to SQL: Where are we today?PVLDB13, 10 (2020), 1737–1750

    2020

  21. [21]

    Ðorđe Klisura, Joseph Khoury, Ashish Kundu, Ram Krishnan, and Anthony Rios

  22. [22]

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. InICLR

    2025

  23. [23]

    Role-Conditioned Refusals: Evaluating Access Control Reasoning in Large Language Models.arXiv preprint arXiv:2510.07642(2025)

    arXiv 2025

  24. [24]

    Fei Li and Hosagrahar V Jagadish. 2014. NaLIR: an interactive natural language interface for querying relational databases. InSIGMOD. 709–712

    2014

  25. [25]

    Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The Dawn of Natural Language to SQL: Are We Fully Ready?PVLDB17, 11 (2024), 3318–3331

    2024

  26. [26]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al . 2023. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. InNeurIPS. 42330–42357

    2023

  27. [27]

    Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. Codes: Towards building open-source language models for text-to-sql. InSIGMOD. 1–28

    2024

  28. [28]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. 2024. Table-gpt: Table fine-tuned gpt for diverse table tasks. InSIGMOD. 1–28

    2024

  30. [30]

    Kyle Luoma and Arun Kumar. 2025. Snails: Schema naming assessments for improved llm-based sql inference. InSIGMOD. 1–26

    2025

  31. [31]

    Xinjian Luo, Ting Yu, and Xiaokui Xiao. 2025. Prompt Inference Attack on Distributed Large Language Model Inference Frameworks. InCCS. 1739–1753

    2025

  32. [32]

    OpenAI. 2025. OpenAI Platform. https://platform.openai.com/docs/models/

    2025

  33. [33]

    Toby Mao. 2023. SQLGlot. https://github.com/tobymao/sqlglot

    2023

  34. [34]

    Sean Wang

    Tonghui Ren, Yuankai Fan, Zhenying He, Ren Huang, Jiaqi Dai, Can Huang, Yinan Jing, Kai Zhang, Yifan Yang, and X. Sean Wang. 2024. PURPLE: Making a Large Language Model a Better SQL Writer. InICDE. 15–28

    2024

  35. [35]

    Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in- context learning of text-to-sql with self-correction. InNeurIPS. 36339–36348

    2023

  36. [36]

    Ravi S Sandhu, Edward J Coyne, Hal L Feinstein, and Charles E Youman. 1996. Role-Based Access Control Models.IEEE Computer29, 2 (1996), 38–47

    1996

  37. [37]

    Soumadeep Saha, Akshay Chaturvedi, Joy Mahapatra, and Utpal Garain. 2025. sudoLLM: On Multi-role Alignment of Language Models.arXiv preprint arXiv:2505.14607(2025)

    arXiv 2025

  38. [38]

    Richard Shay, Uri Blumenthal, Vijay Gadepally, Ariel Hamlin, John Darby Mitchell, and Robert K Cunningham. 2019. Don’t even ask: Database access control through query control.ACM SIGMOD Record47, 3 (2019), 17–22

    2019

  39. [39]

    Debdeep Sanyal, Umakanta Maharana, Yash Sinha, Hong Ming Tan, Shirish Karande, Mohan Kankanhalli, and Murari Mandal. 2025. OrgAccess: A Bench- mark for Role Based Access Control in Organization Scale LLMs.arXiv preprint arXiv:2505.19165(2025)

    Pith/arXiv arXiv 2025

  40. [40]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report.arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  41. [41]

    Pranav Subramaniam and Sanjay Krishnan. 2024. DePLOI: Applying NL2SQL to Synthesize and Audit Database Access Control.arXiv preprint arXiv:2402.07332 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Matthias Urban and Carsten Binnig. 2024. CAESURA: Language Models as Multi-Modal Query Planners. InCIDR

    2024

  43. [43]

    Qwen Team. 2024. Qwen2.5 Technical Report.arXiv preprint arXiv:2412.15115 (2024)

    Pith/arXiv arXiv 2024

  44. [44]

    Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al

  45. [45]

    Lianggui Weng, Dandan Liu, Rong Zhu, Bolin Ding, and Jingren Zhou. 2026. BridgeScope: A Universal Toolkit for Bridging Large Language Models and Databases. InCIDR

    2026

  46. [46]

    Fei Yang, Yangfan Jiang, Yin Yang, and Xiaokui Xiao. 2026. RBAC-Text2SQL Benchmark: Code and Data. GitHub repository. https://github.com/2020dfff/ RBAC-Text2SQL-Benchmark

    2026

  47. [47]

    Yicun Yang, Zhaoguo Wang, Yu Xia, Zhuoran Wei, Haoran Ding, Ruzica Piskac, Haibo Chen, and Jinyang Li. 2025. Automated Validating and Fixing of Text-to- SQL Translation with Execution Consistency. InSIGMOD. 1–28

    2025

  48. [48]

    Xiangjin Xie, Guangwei Xu, Lingyan Zhao, and Ruijie Guo. 2025. Opensearch-sql: Enhancing text-to-sql with dynamic few-shot and consistency alignment. In SIGMOD. 1–24

    2025

  49. [49]

    Qingyu Yin, Chak Tou Leong, Linyi Yang, Wenxuan Huang, Wenjie Li, Xiting Wang, Jaehong Yoon, Jinjin Gu, et al. 2025. Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?arXiv preprint arXiv:2510.06036(2025)

    arXiv 2025

  50. [50]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. InEMNLP. 3911–3921

    2018

  51. [51]

    Zhewei Yao, Guoheng Sun, Lukasz Borchmann, Zheyu Shen, Minghang Deng, Bohan Zhai, Hao Zhang, Ang Li, and Yuxiong He. 2025. Arctic-Text2SQL-R1: Sim- ple Rewards, Strong Reasoning in Text-to-SQL.arXiv preprint arXiv:2505.20315 (2025)

    arXiv 2025

  52. [52]

    Tianshu Zhang, Kun Qian, Siddhartha Sahai, Yuan Tian, Shaddy Garg, Huan Sun, and Yunyao Li. 2025. Evoschema: Towards Text-to-SQL Robustness against Schema Evolution.PVLDB18, 10 (2025), 3655–3668

    2025

  53. [53]

    roles are overly similar,

    Fan Zhou, Siqiao Xue, Danrui Qi, Wenhui Shi, Wang Zhao, Ganglin Wei, Hongyang Zhang, Caigai Jiang, Gangwei Jiang, Zhixuan Chu, et al. 2024. Db- gpt-hub: Towards open benchmarking text-to-sql empowered by large language models.arXiv preprint arXiv:2406.11434(2024). 14 Benchmarking Text-to-SQL under Role-Based Access Control A Details of Human-in-the-Loop V...

    Pith/arXiv arXiv 2024

  54. [54]

    Chao Zhang, Yuren Mao, Yijiang Fan, Yu Mi, Yunjun Gao, Lu Chen, Dongfang Lou, and Jinshu Lin. 2024. Finsql: Model-agnostic llms-based text-to-sql framework for financial analysis. InSIGMOD. 93–105

    2024

  55. [2022]

    UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models. InEMNLP. 602–631

  56. [2025]

    Is Long Context All You Need? Leveraging LLM’s Extended Context for NL2SQL.PVLDB18, 8 (2025), 2735–2747

    2025

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.