Research Challenges in Relational Database Management Systems for LLM Queries

Anurag Chakraborty; Kerem Akillioglu; M. Tamer \"Ozsu; Sairaj Voruganti

REVIEW 3 major objections 5 minor 51 references

Research Challenges in Relational Database Management Systems for LLM Queries

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that SQL queries invoking LLMs fail in current open-source systems because the LLM is treated as a black box, and that the key to scalability is integrating LLM cost and structured output handling into the database's plann

desk verdict Qualitative failure analysis is solid; the latency comparisons are confounded and need major revision. read the letter →

arxiv 2508.20912 v1 pith:J2TDPY34 submitted 2025-08-28 cs.DB cs.AI

Kerem Akillioglu , Anurag Chakraborty , Sairaj Voruganti , M. Tamer Özsu This is my paper

classification cs.DBcs.AI

keywords relationalLLMqueriesSQLinvocationqueryoptimizationconstraineddecodingGPUutilizationplanningretrieval-augmentedgenerationDuckDB

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The authors take five SQL query patterns that call LLMs—projection, filtering, multi-invocation, aggregation, and RAG—and run them on two open-source stacks (PostgreSQL with pgAI, DuckDB with FlockMTL) and an enterprise service (MotherDuck). Most queries either crash or run extremely slowly on the open-source stacks: the default open-source configurations run at most two of the five queries, while the enterprise service runs four of five. The paper traces the failures to three root causes: LLMs do not reliably produce output that fits the SQL schema, requests are batched so poorly that the GPU sits idle, and query planners optimize joins and scans but not the cost of the LLM calls themselves. It then implements small fixes—grammar-constrained decoding, row-level batching, and a manually pushed-down plan for the RAG query—and reports large latency improvements, concluding that tight DBMS+LLM integration is the path to scalable LLM queries.

What carries the argument

The paper's central object is the 'relational LLM query': an ordinary SQL statement with LLM invocations embedded in SELECT, WHERE, or aggregate clauses, following the five patterns defined in Liu et al. The load-bearing mechanism is the interface between the LLM and the relational engine: whether the LLM call is exposed to the planner as a black-box UDF or as a structured, typed operator. The authors show that three properties of that interface—guaranteed output format, batch scheduling, and cost modeling—determine whether a query runs at all and how fast.

What would settle it

Run a broader benchmark, say 20 LLM-in-SQL queries covering ORDER BY, GROUP BY, HAVING, DISTINCT, and chained LLM calls, on the same three systems. If a new failure mode appears that does not reduce to structured output, batching, or planner cost-awareness, the paper's three-challenge taxonomy is incomplete. Alternatively, if a black-box UDF system that simply parallelizes 256 row-level requests matches the latency of a tightly integrated operator on the same local model, the tight-integration thesis is undercut.

Watch

Extended reading notes

Core claim

The central discovery is a systematic map from LLM query shape to failure mode. A projection query (Q1) runs everywhere but slowly, because the system sends rows one at a time. Filter and multi-invocation queries (Q2, Q3) fail on open-source systems because the LLM adds text around 'Yes', breaking the equality check; grammar-based constrained decoding fixes this. The aggregation query (Q4) fails even with constrained decoding because FlockMTL's llm_reduce returns an untyped text blob instead of a numeric type, so the planner cannot apply AVG. The RAG query (Q5) fails because DuckDB's planner does not trigger the HNSW index and instead materializes a cross join; pgAI only succeeds after the a

Load-bearing premise

The load-bearing premise is that the five query patterns borrowed from Liu et al. fairly represent how LLM invocations appear in real SQL workloads; if actual workloads include other patterns, the failure analysis and the three proposed challenges may not generalize.

Editorial extensions

If this is right

Structured output (constrained decoding) is not optional for LLM queries; without it, filter and aggregation queries fail to parse.
Query optimizers that factor in LLM cost can reorder rows and reuse prompt prefixes, reducing prefill overhead on low-cardinality columns.
Separating constrained and unconstrained requests into different batches prevents GPU idle time caused by CPU-side token validation.
Even with these optimizations, local single-GPU inference is far slower than hosted API inference; enterprise-grade parallelism is a major factor in the performance gap.
Planner-execution mismatches (like a missing HNSW lookup turning into a cross join) can change a query's complexity from logarithmic to quadratic, causing memory blowups.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The five query patterns likely under-sample the space of relational LLM queries; patterns like LLM in ORDER BY, LLM in GROUP BY, or chained LLM calls with data dependencies could reveal additional planner and logic gaps.
If LLM cost becomes a first-class optimizer metric, existing techniques like prefix caching, row reordering, and approximate query processing could be composed in new ways; the paper gestures at this but does not build it.
A natural benchmark extension is to vary the local model size and serving engine (e.g., vLLM with continuous batching) to separate model/serving effects from DBMS integration effects on latency.
The paper's success at making Q5 run only after manually pushing down filters suggests that automatic semantic pushdown for vector similarity and LLM predicates is a concrete research target.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper does something genuinely useful: it actually runs five LLM-in-SQL queries from Liu et al. on three systems and documents where they break. The failure/success matrix (Table 1) and the diagnosed planner mismatches—Q4's untyped llm_reduce, Q5's HNSW index turning into a cross join—are concrete, credible findings. The qualitative claim that open-source systems lack structured output enforcement and that optimizers ignore LLM costs is well supported. If you work on DB-LLM integration, this is worth reading.

The soft spots are on the quantitative side. The performance comparison is not controlled. All FlockMTL latency numbers come from a version the authors modified to send one row per prompt, because stock row batching failed. Compare that to MotherDuck, which uses GPT-4o-mini (much smaller than LLaMA-3.1-8B) with up to 256 concurrent requests on proprietary hardware. The paper even says the enterprise comparison is "not directly comparable," yet the abstract and the 12-days-per-million-rows extrapolation lean on that gap. You can't attribute that order-of-magnitude difference to DBMS integration quality—system, model, hardware, and concurrency all change at once. Also, the paper says Q1 on 17,000 rows takes 5 hours in Section 1, but Table 2 reports 719.5 minutes (~12 hours) for the same setup, with no explanation. Single-run latencies and no released code further limit the empirical weight.

The workload representativeness is a minor concern. The five queries cover common shapes but may not generalize; that's acceptable for an "early exploration."

The qualitative findings hold up. The performance claims need fixing. This is a workshop paper, and I'd send it to review with the expectation of major revisions: release artifacts, report repeated measurements, and either drop the enterprise comparison or clearly present it as an upper bound from a different model class.

Who benefits: DB researchers and systems builders looking for concrete pain points in SQL LLM integrations. I'd cite it for the failure analysis, not for the latency numbers.

Recommendation: accept for peer review, conditional on major revisions.

Referee Report

3 major / 5 minor

Summary. The paper is an early exploration of LLM invocation inside relational SQL query processing. It evaluates two open-source integrations—PostgreSQL with pgAI and DuckDB with FlockMTL—using local LLaMA-3.1-8B inference via Ollama/vLLM, and one enterprise platform (MotherDuck with GPT-4o-mini). Five representative queries are taken from Liu et al. (2024) and run on the Rotten Tomatoes and SQuAD datasets. The paper reports a success/failure matrix, query latencies, and GPU utilization. It identifies three research challenges: enforcing structured outputs, optimizing resource utilization, and improving query planning. It argues that tighter LLM+DBMS integration is the key to scalable and efficient processing of LLM queries.

Significance. If the qualitative findings are credited, this is a useful early snapshot of an emerging area. The functional matrix (Table 1) is direct evidence: only Q1 runs on both open-source stacks; Q2–Q4 fail due to structured-output and planner mismatches; constrained decoding via vLLM rescues Q2/Q3 but not Q4; and Q5 fails on most systems, succeeding on pgAI only after a manual plan rewrite. These observations are specific, reproducible, and valuable for researchers working on LLM+DBMS integration. The paper does not introduce fitted parameters or self-referential definitions, and the queries come from an external benchmark. However, the quantitative performance claim in the abstract ('open-source solutions currently have limited functionality and poor performance') is not supported by a controlled comparison: the open-source and enterprise measurements differ simultaneously in system, model, hardware, and concurrency, and the open-source FlockMTL numbers were obtained after a code modification that serializes batches. The paper's most durable contribution is the functional analysis and the challenge taxonomy; the performance evidence needs substantial reframing.

major comments (3)

[§3.3, Tables 2 and 3] The central quantitative claim of 'poor performance' is based on a confounded comparison. Section 3.3 states that stock FlockMTL row batching failed to map outputs reliably, so the authors modified FlockMTL's code to keep the batch size to exactly one row. Table 2 therefore measures a serialized workaround, not stock FlockMTL behavior. The MotherDuck numbers, in contrast, use a different model (GPT-4o-mini vs LLaMA-3.1-8B), up to 256 concurrent requests, and proprietary hardware. The paper itself says the enterprise comparison is 'not directly comparable' (first paragraph of §3.3). Yet the abstract and Section 1 use the resulting order-of-magnitude gap to support 'limited functionality and poor performance.' This attribution is not valid: multiple variables change at once. I recommend either removing the enterprise comparison from the quantitative claim, running a controlled comparison w
[Section 1 vs Table 2] There is an internal inconsistency in the key motivating latency. Section 1 says Q1 on 17,000 rows takes '5 hours' in the pgai-ollama setup. Table 2 reports 719.5 minutes (approximately 12 hours) for the same query and system. The Section 1 text then extrapolates 'around 12 days to process one million rows' based on the 5-hour figure. This discrepancy is not explained and materially affects the paper's impact. Please reconcile the two numbers and state the exact dataset size and configuration underlying each.
[Section 2, workload representativeness] The paper states, without evidence, that the five queries from Liu et al. are 'a good representation for embedding LLM invocations at various points within SQL queries.' The three challenges in Section 4 are generalized from this small, hand-picked set on two open-source systems. In particular, Q5 required a manual plan change to run at all, so its behavior is not out-of-the-box. If the claimed three challenges are meant to characterize the field, the authors should either discuss what other LLM invocation patterns exist (e.g., LLM joins with similarity operators, multi-invocation per row, nested streaming calls) and why the five cover them, or soften the generality claims. This does not undermine the observed failures, but it is load-bearing for the paper's broader challenge taxonomy.

minor comments (5)

[§3.2 / §4.1] The numeric range for Q4 is stated inconsistently: the query prompt says 'between 0 (bad) and 5 (good)', §3.2 says 'numeric output between 1 and 4', and §4.1 says 'single number between 1 and 5'. This makes it hard to know what the actual constraint was in the experiments.
[Figures 6–9] Figure 8 is not explicitly discussed in the text, while the text references only Figures 6, 7, and 9. The figures also show single runs without confidence intervals or repeated trials; a sentence stating how representative these traces are would help.
[Section 1] Typo: 'the eminent systems are slow' should likely be 'the existing systems' or 'the current systems'.
[Table 3] The table title says 'MotherDuck platform' but the column entries are labeled 'motherduck-gpt'; the text sometimes calls it 'MotherDuck Prompt()'. Unify terminology.
[References] Reference [23] points to a GitHub release page rather than a paper or versioned artifact; consider citing the FlockMTL technical report or a stable DOI. Reference [1] is a bare URL with no access date.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical evaluation with no derivation, no fitted parameters, and no self-citations.

full rationale

The paper makes no derived predictions or first-principles claims that reduce to their inputs. It is an empirical exploration of existing systems (pgAI, FlockMTL, MotherDuck) using five external queries from Liu et al. [30] and external datasets (Rotten Tomatoes, SQuAD). The central claims of 'limited functionality and poor performance' are supported by measured latencies, GPU utilization graphs, and query success/failure tables, not by any definitional equivalence. There are no fitted parameters that are later relabeled as predictions, no uniqueness theorems imported from the authors' own prior work, and no author self-citations in the reference list. The paper explicitly qualifies the enterprise comparison: 'We then present MotherDuck's results separately since it is a closed enterprise service running on proprietary hardware, which is not directly comparable to the open-source systems' (Section 3.3). It also discloses modifications to FlockMTL ('we modified FlockMTL's code to keep the batch size for each prompt to exactly one row') and manual plan enforcement for pgAI's Q5. These are validity/confound limitations, not circular reasoning. The representativeness assumption about the five queries is a workload-generalization concern, not a circular step. Therefore no circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are introduced; the evaluation rests on domain assumptions about workload representativeness and deployment stack. The paper's claims are empirical rather than derived, so the main epistemic risk is generalizability rather than internal circularity.

assumptions (3)

domain assumption The five queries from Liu et al. are a good representation for embedding LLM invocations at various points within SQL queries.
Section 2 adopts this framing without an independent workload analysis; failures on these queries are used to generalize to 'LLM queries' broadly.
domain assumption Local inference of LLaMA 3.1 8B on a single A100 with Ollama/vLLM is representative of open-source LLM+DBMS deployments.
Section 3.1 fixes this stack; performance conclusions may change with different hardware, model sizes, or serving frameworks.
domain assumption The observed query failures stem from system design gaps (structured outputs, planning) rather than suboptimal prompts or configuration.
Section 3.2 attributes failures to structured output and planner mismatches; prompt sensitivity is acknowledged as a challenge in Section 4.1, so this assumption is not fully tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Research Challenges in Relational Database Management Systems for LLM Queries." pith.science (2026). https://pith.science/paper/J2TDPY34

@misc{pith2026250820912,
  author       = {Pith},
  title        = {Pith review of: Research Challenges in Relational Database Management Systems for LLM Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2TDPY34}},
  note         = {Machine review of arXiv:2508.20912}
}

read the original abstract

Large language models (LLMs) have become essential for applications such as text summarization, sentiment analysis, and automated question-answering. Recently, LLMs have also been integrated into relational database management systems to enhance querying and support advanced data processing. Companies such as Amazon, Databricks, Google, and Snowflake offer LLM invocation directly within SQL, denoted as LLM queries, to boost data insights. However, open-source solutions currently have limited functionality and poor performance. In this work, we present an early exploration of two open-source systems and one enterprise platform, using five representative queries to expose functional, performance, and scalability limits in today's SQL-invoked LLM integrations. We identify three main issues: enforcing structured outputs, optimizing resource utilization, and improving query planning. We implemented initial solutions and observed improvements in accommodating LLM powered SQL queries. These early gains demonstrate that tighter integration of LLM+DBMS is the key to scalable and efficient processing of LLM queries.

Figures

Figures reproduced from arXiv: 2508.20912 by the authors.

**Figure 3.** Multi-LLM Query. Q4: LLM Aggregation – The LLM is called to assign satisfaction ratings for reviews for each movie title to qualitatively measure average sentiment for overall customer feedback. Then these numerical scores are aggregated using an average function. This approach synthesizes qualitative data into a quantitative summary metric. SELECT AVG ( LLM (" Rate a satisfaction score between 0 ( bad ) and 5 ( go… view at source ↗

**Figure 1.** LLM Projection Query. Q2: LLM Filter – Here, the LLM is invoked in the WHERE clause to evaluate and filter rows based on semantic criteria. The LLM function determines if the input text meets a specified condition, thereby controlling which records are included in the result set. SELECT m . movie_title FROM Movies m JOIN Reviews r ON r . rotten_tomatoes_link = m . rotten_tomatoes_link WHERE LLM (" Analyze whether th… view at source ↗

**Figure 5.** RAG Query. 3 SYSTEMS TESTING Our objectives are two-fold: (1) we want to understand what is required to be able to execute the queries discussed in the previous section, and (2) we want to test the performance of executing them and understand the factors that affect query performance. As we discuss below, most of these queries cannot be executed using these open-source systems. Some of the issues can be addressed by… view at source ↗

Figures from the paper (3 more)

**Figure 6.** Figure 6: FlockMTL (Ollama) GPU Utilization for Q1 [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

**Figure 9.** Figure 9: pgAI (Ollama) GPU Utilization for Q5 does not allow subtransactions during parallel operations, an error2 gets triggered, and the query does not run. To resolve this issue, we forced the query to run in a singleprocess context where subtransactions are permitted. Whil…

**Figure 10.** Figure 10: Query plans for relational-LLM queries 4.3 Query Plans There are two main strategies in integrating LLM functions in the query planning stage. Databricks [2], Amazon Redshift [3], and pgAI use UDFs to call LLMs, while FlockMTL uses custom C++ functions that make LLM c…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 30 canonical work pages

[1]

https://motherduck.com/docs/sql-reference/motherduck-sql-reference/ ai-functions/prompt

[n.d.]. "https://motherduck.com/docs/sql-reference/motherduck-sql-reference/ ai-functions/prompt"

work page
[2]

AI Functions on Databricks

2025. AI Functions on Databricks. https://docs.databricks.com/aws/en/large- language-models/ai-functions. Databricks Documentation. Last updated Feb 10, 2025

work page 2025
[3]

Large Language Models for Sentiment Analysis with Amazon Redshift ML Preview

2025. Large Language Models for Sentiment Analysis with Amazon Redshift ML Preview. https://aws.amazon.com/blogs/big-data/large-language-models-for- sentiment-analysis-with-amazon-redshift-ml-preview/. AWS Big Data Blog

work page 2025
[4]

LLM with Vertex AI: Only Using SQL Queries in BigQuery

2025. LLM with Vertex AI: Only Using SQL Queries in BigQuery. https://cloud.google.com/blog/products/ai-machine-learning/llm-with- vertex-ai-only-using-sql-queries-in-bigquery. Google Cloud Blog

work page 2025
[5]

Snowflake Cortex: LLM Functions

2025. Snowflake Cortex: LLM Functions. https://docs.snowflake.com/en/user- guide/snowflake-cortex/llm-functions. Snowflake Documentation

work page 2025
[6]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023
[7]

Anyscale. 2025. Constrained generation with JSON mode. https://docs.anyscale. com/llms/serving/guides/json_mode/. Accessed: 17 March 2025

work page 2025
[8]

RJ Atwal, Peter Boncz, Ryan Boyd, Antony Courtney, Till Döhmen, Florian Gerlinghoff, Jeff Huang, Joseph Hwang, Raphael Hyde, Elena Felder, Jacob La- couture, Yves Le Maout, Boaz Leskes, Yao Liu, Alex Monahan, Dan Perkins, Tino Tereshko, Jordan Tigani, Nick Ursa, Stephanie Wang, and Yannick Welsch. 2024. MotherDuck: DuckDB in the Cloud and in the Client. I...

work page 2024

Show all 51 references

[9]

BentoML and Red Hat. 2025. Structured Decoding in vLLM: A Gentle Introduc- tion. https://www.bentoml.com/blog/structured-decoding-in-vllm-a-gentle- introduction Accessed: 15 March 2025

2025
[10]

Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages 7, PLDI (2023), 1946–1969

2023
[11]

Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2024. Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation. arXiv:2403.06988 [cs.LG] https://arxiv.org/abs/2403.06988

2024 arXiv
[12]

Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. 2024. On the Worst Prompt Performance of Large Language Models. In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 3...

2024
[13]

Sonia Cromp, Satya Sai Srinath Namburi GNVV, Mohammed Alkhudhayri, Catherine Cao, Samuel Guo, Nicholas Roberts, and Frederic Sala. [n.d.]. Tabby: Tabular Data Synthesis with Language Models. ([n. d.])
[14]

Hanjun Dai, Bethany Yixin Wang, Xingchen Wan, Bo Dai, Sherry Yang, Azade Nova, Pengcheng Yin, Phitchaya Mangpo Phothilimthana, Charles Sutton, and Dale Schuurmans. 2024. UQE: A Query Engine for Unstructured Databases. In Advances in Neural Information Processing Systems (NeurIPS 2024)

2024
[15]

Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. TURL: table understanding through representation learning. Proc. VLDB Endow. 14, 3 (Nov. 2020), 307–319. https://doi.org/10.14778/3430915.3430921

2020
[16]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLORA: efficient finetuning of quantized LLMs. In Proceedings of the 37th In- ternational Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Re...

2023
[17]

Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, and Tianqi Chen

Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, and Tianqi Chen. 2024. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv:2411.15100 [cs.CL] https://arxiv.org/ abs/2411.15100

2024 arXiv
[18]

Anas Dorbani, Sunny Yasser, Jimmy Lin, and Amine Mhedhbi. 2025. Be- yond Quacking: Deep Integration of Language Models and RAG into DuckDB. arXiv:2504.01157 [cs.DB] https://arxiv.org/abs/2504.01157

2025 arXiv
[19]

dottxt ai. 2025. Outlines: Structured Text Generation. https://github.com/dottxt- ai/outlines. Accessed: 15 March 2025

2025
[20]

Dubey, A

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, and et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

2024 arXiv
[21]

Yael Einy, Tova Milo, and Slava Novgorodov. 2024. Cost-Effective LLM Utilization for Machine Learning Tasks over Tabular Data. In Proceedings of the Conference on Governance, Understanding and Integration of Data for Effective and Responsible AI (Santiago, AA, Chile) (GUIDE-AI...

2024
[22]

Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun (Jane) Qi, Scott Nickleach, Diego Socolinsky, "SHS" Srinivasan Sengamedu, and Christos Faloutsos. 2024. Large language models (LLMs) on tabular data: Prediction, gener- ation, and understanding — a survey. Tr...

2024
[23]

FlockMTL. 2024. FlockMTL. http://github.com/dsg-polymtl/flockmtl/releases

2024
[24]

guidance ai. 2025. Guidance: A Guidance Language for Controlling Large Lan- guage Models. https://github.com/guidance-ai/guidance. Accessed: 5 June 2025

2025
[25]

Xingyu Ji, Aditya Parameswaran, and Madelon Hulsebos. [n.d.]. TARGET: Bench- marking Table Retrieval for Generative Tasks. In NeurIPS 2024 Third Table Repre- sentation Learning Workshop

2024
[26]

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Lun-W...

2024 doi
[27]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings ...

2020
[28]

Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano. 2024. A Declarative System for Optimizing AI Workloads. arXiv:2405.14696 [cs.CL]

2024 arXiv
[29]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. https://doi.org/10.11...

2024 doi
[30]

Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E Gonza- lez, Ion Stoica, and Matei Zaharia. 2024. Optimizing llm queries in relational workloads. arXiv preprint arXiv:2403.05821 (2024)

2024 arXiv
[31]

Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina J Semnani, Chen Jie Yu, and Monica S Lam. 2023. SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models. arXiv preprint arXiv:2311.09818 (2023)

2023 arXiv
[32]

https://motherduck.com/blog/sql-llm-prompt-function- gpt-models/

MotherDuck. [n.d.]. "https://motherduck.com/blog/sql-llm-prompt-function- gpt-models/"
[33]

MotherDuck. 2025. Introducing the embedding() function: Semantic search made easy with SQL. https://motherduck.com/blog/sql-embeddings-for-semantic- meaning-in-text-and-rag/

2025
[34]

Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner. 2025. A Closer Look at System Prompt Robustness. arXiv preprint arXiv:2502.12197 (2025)

2025 arXiv
[35]

Zan Ahmad Naeem, Mohammad Shahmeer Ahmad, Mohamed Eltabakh, Mourad Ouzzani, and Nan Tang. 2024. RetClean: Retrieval-Based Data Cleaning Using LLMs and Data Lakes. Proc. VLDB Endow. 17, 12 (Aug. 2024), 4421–4424. https: //doi.org/10.14778/3685800.3685890

2024
[36]

Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, and Jinwoo Shin. 2024. Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning. arXiv:2406.08527 [cs.LG] https://arxiv.org/abs/ 2406.08527

2024 arXiv
[37]

OpenAI. 2023. Introducing Structured Outputs in the API . https://openai.com/ index/introducing-structured-outputs-in-the-api/ Accessed: 15 March 2025

2023
[38]

Bo Pang and Lillian Lee. 2005. Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales. InProceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL‘05) , Kevin Knight, Hwee Tou Ng, and Kemal Ofl...

2005 doi
[39]

Liana Patel, Siddharth Jha, Carlos Guestrin, and Matei Zaharia. 2024. LOTUS: En- abling Semantic Queries with LLMs Over Tables of Unstructured and Structured Data. arXiv:2407.11418 [cs.DB]

2024 arXiv
[40]

pgAI. 2024. pgAI. https://github.com/timescale/pgai

2024
[41]

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018)

2018 arXiv
[42]

Dario Satriani, Enzo Veltri, Donatello Santoro, Sara Rosato, Simone Varriale, and Paolo Papotti. 2025. Logical and Physical Optimizations for SQL Query Execution over Large Language Models

2025
[43]

Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing ...

2021 doi
[44]

Parameswaran, and Eugene Wu

Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2025. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. arXiv:2410.12189 [cs.DB] https://arxiv.org/abs/2410.12189

2025 arXiv
[45]

Matthias Urban and Carsten Binnig. 2024. ELEET: Efficient Learned Query Execution over Text and Tables.Proc. VLDB Endow. 17, 13 (Sept. 2024), 4867–4880. https://doi.org/10.14778/3704965.3704989

2024
[46]

Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. 2024. HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection. In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. B...

2024
[47]

Yu Wang, Luyao Zhou, Yuan Wang, Zhenwan Peng, and Surya Prakash. 2024. Leveraging Pretrained Language Models for Enhanced Entity Matching: A Com- prehensive Study of Fine-Tuning and Prompt Learning Paradigms. Int. J. Intell. Syst. 2024 (Jan. 2024), 14. https://doi.org/10.1155/...

2024 doi
[48]

Brandon T Willard and Rémi Louf. 2023. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702 (2023)

2023 arXiv
[49]

Qinyuan Ye, Mohamed Ahmed, Reid Pryzant, and Fereshte Khani. 2024. Prompt Engineering a Prompt Engineer. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Ban...

2024 doi
[50]

Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. Jasper and Stella: distillation of SOTA embedding models. arXiv:2412.19048 [cs.IR] https: //arxiv.org/abs/2412.19048

2025 arXiv
[51]

Müller, Dalitso Banda, Fotis Psallidas, and Jignesh M

Yunjia Zhang, Avrilia Floratou, Joyce Cahoon, Subru Krishnan, Andreas C. Müller, Dalitso Banda, Fotis Psallidas, and Jignesh M. Patel. 2023. Schema Matching using Pre-Trained Language Models. In2023 IEEE 39th International Conference on Data Engineering (ICDE). 1558–1571. http...

2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

https://motherduck.com/docs/sql-reference/motherduck-sql-reference/ ai-functions/prompt

[n.d.]. "https://motherduck.com/docs/sql-reference/motherduck-sql-reference/ ai-functions/prompt"

work page

[2] [2]

AI Functions on Databricks

2025. AI Functions on Databricks. https://docs.databricks.com/aws/en/large- language-models/ai-functions. Databricks Documentation. Last updated Feb 10, 2025

work page 2025

[3] [3]

Large Language Models for Sentiment Analysis with Amazon Redshift ML Preview

2025. Large Language Models for Sentiment Analysis with Amazon Redshift ML Preview. https://aws.amazon.com/blogs/big-data/large-language-models-for- sentiment-analysis-with-amazon-redshift-ml-preview/. AWS Big Data Blog

work page 2025

[4] [4]

LLM with Vertex AI: Only Using SQL Queries in BigQuery

2025. LLM with Vertex AI: Only Using SQL Queries in BigQuery. https://cloud.google.com/blog/products/ai-machine-learning/llm-with- vertex-ai-only-using-sql-queries-in-bigquery. Google Cloud Blog

work page 2025

[5] [5]

Snowflake Cortex: LLM Functions

2025. Snowflake Cortex: LLM Functions. https://docs.snowflake.com/en/user- guide/snowflake-cortex/llm-functions. Snowflake Documentation

work page 2025

[6] [6]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023

[7] [7]

Anyscale. 2025. Constrained generation with JSON mode. https://docs.anyscale. com/llms/serving/guides/json_mode/. Accessed: 17 March 2025

work page 2025

[8] [8]

RJ Atwal, Peter Boncz, Ryan Boyd, Antony Courtney, Till Döhmen, Florian Gerlinghoff, Jeff Huang, Joseph Hwang, Raphael Hyde, Elena Felder, Jacob La- couture, Yves Le Maout, Boaz Leskes, Yao Liu, Alex Monahan, Dan Perkins, Tino Tereshko, Jordan Tigani, Nick Ursa, Stephanie Wang, and Yannick Welsch. 2024. MotherDuck: DuckDB in the Cloud and in the Client. I...

work page 2024