REVIEW 3 major objections 25 references
When the database and workload are known, synthesizing a cardinality estimator as executable code beats generic statistics and turns better estimates into faster plans.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 18:09 UTC pith:NTCUE5XA
load-bearing objection Real third path for cardinality estimation: LLM-synthesized executable estimators with staged feedback deliver large JOB/JOB-Complex gains, but headline numbers rest on single synthesis runs and a couple of write-up inconsistencies. the 3 major comments →
Bespoke-Card: Why Tune When You Can Generate? Synthesizing Workload-Specific Cardinality Estimators
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Bespoke-Card shows that a planner/coder agent loop, closed by deterministic subplan-level feedback against true cardinalities and a production baseline, can synthesize an inspectable Python cardinality estimator specialized to a fixed database and tuned for a declared workload. When those estimates are injected into PostgreSQL’s optimizer, total runtime falls 33% on JOB and 68% on JOB-Complex, and median q-error over all JOB subplans falls from 190.5 to 11.5, with the upper tail compressed by roughly two orders of magnitude.
What carries the argument
The structured synthesis harness: a planning agent that designs workload-specific statistics and estimation strategy, a coding agent that emits a card_estimator module over structured SPAJ requests (not raw SQL), and a deterministic evaluator that returns staged, structured feedback—join-only, filter-only, then full subplans—including q-error percentiles, regression rates versus PostgreSQL, and concrete outlier subplans, with archival selection of the best candidate.
Load-bearing premise
A representative, relatively stable workload is known up front, and true cardinalities can be collected once for the subplans used to score and repair the estimator.
What would settle it
Resynthesize an estimator for JOB or JOB-Complex under the same harness, inject its estimates into the same optimizer, and check whether end-to-end total runtime and the full subplan q-error distribution fail to beat the production baseline by the reported margins; a null result on either accuracy or plan quality would undercut the central claim.
If this is right
- For databases with known schemas and recurring workloads, optimizers can ship a regenerable, inspectable estimator artifact instead of only tuning a general statistics catalog.
- Staged isolation of join errors, filter errors, and their interaction becomes a reusable repair curriculum for any synthesized estimator under measurable feedback.
- Estimator quality can be improved without choosing a fixed neural architecture or collecting large labeled training sets for every model family.
- When data, schema, or workload shifts, the same loop can rebuild or re-specialize the estimator rather than maintaining a permanent universal component.
- The same feedback-driven code-synthesis pattern is proposed as applicable to other optimizer pieces that admit deterministic scoring, such as cost models.
Where Pith is reading between the lines
- If true-cardinality labeling for feedback remains the expensive step, cheaper approximate oracles or partial labeling schedules would become the main lever for wider adoption.
- The largest gains on the harder complex-join workload suggest the method’s value is highest where independence and uniformity assumptions break most, not on simple single-table filters.
- Human audit of the modest-size generated code could become part of production governance, since the artifact is readable Python rather than an opaque model checkpoint.
- Joint synthesis of cardinality logic with neighboring optimizer components could compound plan-quality gains beyond cardinality alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Bespoke-Card reframes cardinality estimation as LLM-driven synthesis of an executable, workload-specific estimator for a fixed database. A planner designs statistics and estimation strategies; a coder implements them as Python; a deterministic harness scores candidates against true subplan cardinalities and PostgreSQL, with a staged curriculum (join-only, filter-only, full subplans), structured q-error/regression feedback, outlier cases, and archival selection. On IMDb, injecting the synthesized estimates into PostgreSQL reduces total runtime by 33% on JOB and 68% on JOB-Complex versus PostgreSQL’s own estimator, with large median and tail q-error reductions (Table 3), small statistics footprints, and synthesis cost under one hour and $10.
Significance. If the results hold under modest robustness checks, the paper opens a genuine third path beside classical catalogs and fixed learned architectures: treating an optimizer component as code to be synthesized under measurable feedback. Strengths include end-to-end plan injection (not only offline q-error), a clear staged-feedback ablation (Figure 5), an inspectable strategy inventory (Table 4), artifact availability, and explicit cost/size reporting. The database-and-workload contract is a coherent specialization thesis rather than an unstated limitation. That combination is significant for query optimization and for the broader “bespoke systems” agenda.
major comments (3)
- Headline runtime and q-error claims (§5.2–5.4, Fig. 1, Table 3) rest on a single GPT-5.4 synthesis trajectory per workload (§5.1, §5.8: 88 turns, $9.74 on JOB). LLM coding is stochastic; different seeds can change the statistics plan, join/filter logic, and residual regressions. Without multi-seed runs (or at least variance on median/tail q-error and total runtime), the 33% and 68% deltas are not yet established as stable properties of the method. Please report several independent synthesis runs, or justify why a single trajectory is sufficient and add a sensitivity discussion.
- Accuracy and runtime evaluation use the same workload subplans that drive the staged feedback curriculum (§4.3, §5.3–5.4). The paper also claims support for arbitrary SPAJ requests over the target database (§3.1, §4.2). Those claims are not separated: there is no hold-out query set, template shift, or literal perturbation. At minimum, evaluate on held-out JOB/JOB-Complex queries or a second SPAJ sample over IMDb that was not used in feedback, and report whether plan quality and q-error degrade.
- Numerical reporting of the JOB median q-error is inconsistent in a load-bearing way. Table 3 and the arXiv-style abstract give PostgreSQL q50 = 190.5 and Bespoke-Card = 11.5 (~94% reduction). §5.4 states “from 19.5 to 11.5 (a 41% reduction),” and the workshop abstract claims a 41% median reduction. Please reconcile abstract, §5.4, and Table 3 so the central accuracy claim is unambiguous.
Circularity Check
No circular derivation: runtime and q-error claims are measured against external ground truth and an independent baseline after the estimator is frozen.
full rationale
Bespoke-Card is an engineering synthesis system, not a first-principles derivation that claims to predict quantities already used as definitional inputs. The load-bearing results (JOB/JOB-Complex end-to-end runtime with injected estimates; subplan q-error quantiles) are obtained by (i) freezing a generated estimator, (ii) comparing estimates to true cardinalities from EXPLAIN ANALYZE, (iii) comparing to PostgreSQL’s built-in estimator, and (iv) measuring wall-clock execution under PG-Lab injection with join enumeration and cost model held fixed. None of these outcomes is algebraically forced by the synthesis inputs: the planner’s design hypothesis and the coder’s patches are scored by a deterministic external harness, not by re-labeling a fitted parameter as a prediction. Self-citations to the authors’ Bespoke DBMS / Bespoke-OLAP vision and to JOB-Complex supply research context and a harder workload, but they do not underwrite uniqueness theorems, forbid alternatives, or substitute for the empirical measurements. Specializing to a declared workload and using that same workload’s subplans in the feedback curriculum is a stated design contract and a possible overfitting concern, not a by-construction identity between claim and input. No self-definitional step, fitted-input-as-prediction, load-bearing self-citation chain, uniqueness import, smuggled ansatz, or renamed known result is present in the derivation of the headline claims.
Axiom & Free-Parameter Ledger
free parameters (4)
- LLM backbone (GPT-5.4)
- Feedback curriculum stages and grouping dimensions
- Outlier count (10 worst over- and under-estimates)
- Archival selection criterion
axioms (5)
- domain assumption A representative workload and fixed schema are known a priori and form a stable “contract” for specialization.
- domain assumption True cardinalities of all evaluation subplans can be obtained once (via EXPLAIN ANALYZE) and reused as ground truth.
- ad hoc to paper PostgreSQL’s cardinality estimates serve as a useful “easy-to-beat” baseline that distinguishes fixable from intrinsically hard errors.
- domain assumption SPAJ queries with complex predicates are a sufficient query class for the prototype.
- standard math Standard relational algebra and q-error (max(est/true, true/est)) are the correct accuracy metrics.
invented entities (2)
-
Bespoke-Card multi-agent synthesis loop (planner + coder + staged deterministic harness)
no independent evidence
-
Database-and-workload contract
no independent evidence
read the original abstract
Cardinality estimators are built to support arbitrary schemas and workloads, forcing them to rely on generic statistics even when the schema and workload is known in advance, leaving optimizers prone to large errors and poor plans. We present Bespoke-Card, an agent-driven system that synthesizes workload-specific cardinality estimators as executable code: a planning agent designs the estimators' strategies, a coding agent implements them, and a validator scores the estimates against true cardinalities and PostgreSQL estimates, forming a robust and deterministic harness. Going beyond naive prompting, Bespoke-Card uses structured q-error feedback, regression analysis, concrete outlier subplans, a curriculum isolating join-only, filter-only, and full-subplan errors, and archival selection of the best implementation. Injecting its estimates into the optimizer cuts total PostgreSQL runtime on JOB by 33% and reduces median q-error over all JOB subplans from 190.5 to 11.5 (-94%), while synthesizing a strong estimator in under one hour for less than $10. Bespoke-Card is opening a new avenue for cardinality estimation next to classical generic estimators and learned estimator architectures.
Figures
Reference graph
Works this paper leans on
-
[1]
Gibbons, Viswanath Poosala, and Sridhar Ra- maswamy
Swarup Acharya, Phillip B. Gibbons, Viswanath Poosala, and Sridhar Ra- maswamy. 1999. Join Synopses for Approximate Query Answering. InSIGMOD. 275–286
1999
-
[2]
Rico Bergmann, Claudio Hartmann, Dirk Habich, and Wolfgang Lehner. 2025. An Elephant Under the Microscope: Analyzing the Interaction of Optimizer Components in PostgreSQL.SIGMOD3, 1 (2025), 9:1–9:28
2025
-
[3]
Garofalakis, Peter J
Graham Cormode, Minos N. Garofalakis, Peter J. Haas, and Chris Jermaine
-
[4]
Foundations and Trends in Databases4, 1-3 (2012), 1–294
Synopses for Massive Data: Samples, Histograms, Wavelets, Sketches. Foundations and Trends in Databases4, 1-3 (2012), 1–294
2012
-
[5]
Timo Eckmann, Matthias Jasny, Johannes Wehrstein, and Carsten Binnig. 2026. The Future Is Bespoke: Synthesizing One-Size-Fits-One DBMSs with LLM Coding Agents.IEEE Data Engineering Bulletin50, 1 (2026), 88–103
2026
-
[6]
Ullman, and Jennifer Widom
Hector Garcia-Molina, Jeffrey D. Ullman, and Jennifer Widom. 2009.Database Systems - The Complete Book (2. ed.)
2009
-
[7]
Benjamin Hilprecht, Andreas Schmidt, Moritz Kulessa, Alejandro Molina, Kris- tian Kersting, and Carsten Binnig. 2020. DeepDB: Learn from Data, not from Queries!VLDB13, 7 (2020), 992–1005
2020
-
[8]
Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, and Alfons Kemper. 2019. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. InCIDR
2019
-
[9]
Oleksii Kliukin. 2014. PgTune – Tuning PostgreSQL Config by Your Hardware
2014
-
[10]
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann. 2015. How Good Are Query Optimizers, Really?VLDB9, 3 (2015), 204–215
2015
-
[11]
Viktor Leis, Bernhard Radke, Andrey Gubichev, Alfons Kemper, and Thomas Neumann. 2017. Cardinality Estimation Done Right: Index-Based Join Sampling. InCIDR
2017
-
[12]
Yao Lu, Srikanth Kandula, Arnd Christian König, and Surajit Chaudhuri. 2021. Pre-training Summarization Models of Structured Datasets for Cardinality Esti- mation.VLDB15, 3 (2021), 414–426
2021
-
[13]
Ioannidis
Viswanath Poosala and Yannis E. Ioannidis. 1997. Selectivity Estimation Without the Attribute Value Independence Assumption. InVLDB. 486–495
1997
-
[14]
Ioannidis, Peter J
Viswanath Poosala, Yannis E. Ioannidis, Peter J. Haas, and Eugene J. Shekita
-
[15]
In SIGMOD
Improved Histograms for Selectivity Estimation of Range Predicates. In SIGMOD. 294–305
-
[16]
Selinger, Morton M
Patricia G. Selinger, Morton M. Astrahan, Donald D. Chamberlin, Raymond A. Lorie, and Thomas G. Price. 1979. Access Path Selection in a Relational Database Management System. InSIGMOD. 23–34
1979
-
[17]
One Size Fits All
Michael Stonebraker and Ugur Çetintemel. 2005. "One Size Fits All": An Idea Whose Time Has Come and Gone (Abstract). InICDE. 2–11
2005
-
[18]
Alexander van Renen, Dominik Horn, Pascal Pfeil, Kapil Vaidya, Wenjian Dong, Murali Narayanaswamy, Zhengchun Liu, Gaurav Saxena, Andreas Kipf, and Tim Kraska. 2024. Why TPC Is Not Enough: An Analysis of the Amazon Redshift Fleet.VLDB17, 11 (2024), 3694–3706
2024
-
[19]
Johannes Wehrstein, Carsten Binnig, Fatma Özcan, Shobha Vasudevan, Yu Gan, and Yawen Wang. 2025. Towards Foundation Database Models. InCIDR
2025
-
[20]
Johannes Wehrstein, Timo Eckmann, Roman Heinrich, and Carsten Binnig
-
[21]
JOB-Complex: A Challenging Benchmark for Traditional & Learned Query Optimization.VLDB(2025)
2025
-
[22]
Johannes Wehrstein, Timo Eckmann, Matthias Jasny, and Carsten Binnig. 2026. Bespoke OLAP: Synthesizing Workload-Specific One-size-fits-one Database En- gines.arXiv preprint arXiv:2603.02001(2026)
arXiv 2026
-
[23]
Peizhi Wu and Gao Cong. 2021. A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation. InSIGMOD. 2009–2022
2021
-
[24]
Zongheng Yang, Amog Kamsetty, Sifei Luan, Eric Liang, Yan Duan, Xi Chen, and Ion Stoica. 2020. NeuroCard: One Cardinality Estimator for All Tables.VLDB14, 1 (2020), 61–73
2020
-
[25]
Tianjing Zeng, Junwei Lan, Jiahong Ma, Wenqing Wei, Rong Zhu, Pengfei Li, Bolin Ding, Defu Lian, Zhewei Wei, and Jingren Zhou. 2024. PRICE: A Pretrained Model for Cross-Database Cardinality Estimation.VLDB18, 3 (2024), 637–650. 10
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.