REVIEW 4 major objections 5 minor 14 references
Training-Free Query Optimization via LLM-Based Plan Similarity
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that frozen LLM embeddings of execution plans can guide optimizer-hint selection by nearest-neighbour transfer, yielding a 21.1% latency reduction on the JOB-CEB workload without any training.
desk verdict A training-free plan-embedding hint recommender with a real win over string matching, but the 21% headline likely reflects template memorization under a query-level split. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the plan-embedding space: a frozen LLM encoder maps a physical execution plan to a vector, and ordinary vector distance in this space is treated as plan similarity. Around that space the paper builds a two-stage retrieval mechanism: first, N-nearest-neighbour voting over default-plan embeddings selects the most popular non-default hint set; second, a K-neighbourhood consistency check compares mean measured runtimes around the default plan and around the re-planned candidate, accepting the hint only if the candidate neighbourhood is faster. A fallback adaptive search over the 128 binary hint configurations, pruned by a fastest-so-far timeout and a plan-latency cache, supplies reference triplets and covers novel queries. The two neighbourhoods do the safety work: the first maximizes coverage of likely useful hints, the second blocks hints that would move the plan into a slower region of embedding space.
What would settle it
Run the same evaluation with a reference store that excludes all queries sharing the test queries' templates, creating a template-disjoint split. If the reported 21% latency reduction largely disappears or turns negative, the claimed generalization rests on near-duplicate plan retrieval rather than on embedding-based plan similarity.
Extended reading notes
Core claim
The central discovery is that pre-trained LLM embeddings of execution plans form a neighbourhood structure in which structurally related plans share beneficial hint sets. Given a new query, the system embeds the optimizer's default plan, finds its N nearest neighbours in a reference store of plan, optimal-hint, and runtime triplets, and lets the neighbours vote for the most frequent non-default hint set. It then re-plans the query with that candidate hint, embeds the new plan, and compares the average runtimes of the K nearest neighbours around the default plan with those around the candidate plan; the hint is adopted only if the candidate neighbourhood is faster. The paper reports that this two-stage scheme, with a fallback exhaustive search over 128 binary hint configurations, reduces total workload latency by 21.1% on JOB-CEB over openGauss, with the slowest fold still 8% faster and the 90th-percentile runtime down 24.7%. The ablation shows that LLM embeddings without the consistency check still beat string matching, with +16.1% versus -64.4% and 87 timeouts, so both the embedding space and the verification step are presented as necessary.
Load-bearing premise
The results assume that a random 10% query-level holdout from 3,133 template-generated JOB-CEB queries is independent enough from the 90% reference store to show real generalization; because the workload is generated from only 16 query templates, the same template can appear in both folds, so nearest-neighbour hint transfer may reduce to retrieving near-duplicate plans.
Editorial extensions
If this is right
- Porting LLM-PM to any database system with a hint interface requires only an embedder and a reference store of executed plans; no optimizer internals are modified.
- Because the reference store grows through the fallback search, the fraction of queries needing exhaustive hint search should shrink over time, improving end-to-end latency as the system accumulates experience.
- Plan embeddings should be preferred over plan-string edit distance for hint transfer: the string edit-distance baseline regresses performance by 64.4% and causes 87 timeouts, versus one timeout for the full system.
- The two-neighbourhood consistency check is essential to safe transfer: removing it drops the speed-up from 21.1% to 16.1% and increases timeouts fivefold.
- On workloads like JOB-CEB the gains concentrate in the long tail: the 90th-percentile runtime falls 24.7% while the median improves only 2.1%, so the method targets the slowest queries.
Reading between the lines
- Because JOB-CEB derives all 3,133 queries from just 16 templates, the random 10% holdout likely places near-duplicate queries in the reference store; a template-disjoint split would test whether the 21% gain reflects true generalization to unseen plan structures or retrieval of near-identical plans.
- The same two-neighbourhood consistency check could be applied to other black-box configuration tasks, such as cardinality-correction features or cost-model constants, where nearby configurations have measured outcomes and a frozen encoder is available.
- The paper's own oracle analysis attributes almost 40% of the attainable speed-up to the candidate-selection stage, suggesting that a domain-tuned embedding space could push performance closer to the 62.5% workload ceiling without changing the framework.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LLM-PM, a training-free hint-recommendation system for relational query optimization. For an incoming query, LLM-PM embeds the optimizer's default execution plan with a frozen LLM (OpenAI text-embedding-3-large), retrieves the N nearest default-plan embeddings from a reference store, and uses majority voting over the associated optimal hint sets to select a candidate hint. A consistency check compares average runtimes in K-neighborhoods around the default and candidate plan embeddings, and the hint is adopted only if the candidate neighborhood has lower average runtime; otherwise the query falls back to an adaptive exhaustive search over 128 binary hint sets. The evaluation on JOB-CEB/openGauss uses ten-fold cross-validation and reports a 21.1% aggregate latency reduction for the Plan-Mapping core, a 16.1% reduction without the consistency check, and a -64.4% regression for a Levenshtein plan-string baseline. The authors conclude that pre-trained plan embeddings can guide hint selection without training.
Significance. If the result holds, the paper provides useful evidence that frozen LLM embeddings of execution plans carry enough structure for k-NN hint transfer, and the two-stage voting-plus-consistency design is a reasonable safeguard. The ten-fold cross-validation and the explicit ablation against Levenshtein string matching are commendable: they go beyond a single train/test split and give the reader a concrete lower bound on what the embeddings contribute. The main significance, however, is conditional on two evaluation issues: the query-level split over a 16-template workload may allow near-duplicate retrieval, and the headline latency numbers appear to exclude embedding and search overhead. Because the method is deliberately simple and training-free, the evaluation rigor is the main determinant of the paper's value.
major comments (4)
- [§4.2, with §1 and §4.1] The 10% query-level holdout does not control for template leakage. JOB-CEB consists of 3,133 instances generated from only 16 templates, so a random query-level split places many near-identical queries from the same template in the reference store for each test query. Plan-Mapping can then retrieve a near-duplicate default plan and transfer its exhaustive-search optimal hint, which tests template memorization rather than generalization to unseen plan structures. The Levenshtein baseline is measured under the same split, so it does not resolve this issue. Please add a template-level holdout, or at least report the results separately for test queries whose nearest reference neighbor comes from a different template, and compare same-template versus cross-template transfer. This is decisive for the central claim that pre-trained plan embeddings generalize.
- [§4.2 and §3.4] The reported 21.1% aggregate speed-up appears to sum query execution runtimes only. Plan-Mapping also incurs (i) an OpenAI text-embedding-3-large API call for the default plan and another for the candidate plan, (ii) two k-NN searches over the reference store, and (iii) re-planning for the candidate hint. When Plan-Mapping returns no hint or the consistency check rejects the candidate, the full pipeline of §3.4 invokes Algorithm 1's exhaustive adaptive search, whose cost is also absent from the headline number. If the evaluation covers only the core Plan-Mapping path, that should be stated explicitly; either way, the embedding, search, and re-planning overhead should be quantified or the speed-up claim should be restricted to query execution time. As written, the 'practical, low-overhead' positioning of the paper is not supported by the reported metric.
- [§4.3, Table 1] The timeout policy is not specified. If a timed-out execution is counted as 450 s, then the 87 timeouts in the Levenshtein variant contribute roughly 39,150 s to the total and dominate the -64.4% figure; if timed-out queries are instead excluded or censored, the comparison across variants is not apples-to-apples. Please state exactly how timeouts enter the aggregate latency and report totals both with and without the timeout contribution.
- [§3.3, Algorithm 2 (steps 10–14)] The consistency check compares \bar{t}_0, the average default runtime of the K nearest stored default plans, with \bar{t}_{cand}, the average optimal runtime of the K nearest stored optimized plans. The latter is not the runtime of the test query under H_cand; it is an average over other queries' runtimes under their own optimal hint sets. The check can therefore accept a hint that is slow for the test query whenever the candidate plan embedding happens to lie near historically fast optimized plans. This weakens the claim that the consistency check 'validates' the candidate hint. Please either rephrase the check as a heuristic transfer criterion or evaluate it against actual test-query runtimes under H_cand.
minor comments (5)
- [§4.2] The text says a 19.1% reduction means queries run 'on average 1.19× faster', but a 19.1% reduction in runtime corresponds to a speed-up factor of about 1.24×, not 1.19×.
- [§3.3 and Algorithm 2] The phrase 'all hints enabled' is used in ways that can be misread: in the bit convention of §3.2 a 1 means 'disable the operator', so the default is the all-zero vector. Please define 'all hints enabled', 'default hints', and 'all operators enabled' explicitly at first use.
- [§4.1] The figure captions and in-text references are inconsistent: the text refers to 'Table 4 (a)' and 'Figure 4 (b)', but the displayed material appears to be a single table followed by a histogram. Please renumber and label the parts consistently.
- [§4.2] Per-fold results are reported as a range (8% to 32%) without confidence intervals or a significance test, although the section introduction promises an analysis of statistical significance. Adding a paired test or confidence interval would strengthen the claim of consistent improvement.
- [§3.3] The hyper-parameters N, K, and the distance metric are fixed at N=K=16 and Euclidean distance with no sensitivity analysis. A short study showing that the aggregate speed-up is stable over a range of N and K would make the result less configuration-dependent.
Circularity Check
No circularity: Plan-Mapping is a training-free retrieval system whose reference labels come from an independent exhaustive hint search, and no fitted parameter is renamed as a prediction.
full rationale
LLM-PM is a retrieval system: it embeds the default plan, retrieves the N nearest reference triples (d_i, H*_i, o_i), votes on the most popular non-default hint set, and verifies the candidate with a two-neighborhood consistency check. The reference labels H*_i are produced independently by Algorithm 1, an exhaustive timeout-pruned search over all 128 binary hint sets; they are not derived from the embedding predictor. The hyperparameters N=16, K=16, and the Euclidean distance are stated as fixed experimental choices, not fitted against the test folds, so the reported 21.1% speed-up is not a fitted quantity being called a prediction. The evaluation measures actual query latencies after applying the recommended hint and compares them with default latencies, so the outcome is not built into the method by construction. The only self-citation, reference [5] (Vasilenko et al., ACM), appears in the Related Work section as a description of prior work on cost-model calibration and is not used as evidence for any claim in this paper; it is therefore not load-bearing. The skeptical concern about query-level holdout over only 16 templates is a generalization or leakage question, not a circularity question: even if near-duplicate retrieval drives the gains, the reported speedup is still an empirically measured effect rather than an identity or a refit. No equation in the paper defines the prediction in terms of the measured outcome, and no uniqueness or ansatz is imported from the authors' prior work. The derivation chain is therefore self-contained with respect to circularity.
Assumptions & free parameters
free parameters (3)
- N (voting neighbourhood size) =
16
- K (consistency neighbourhood size) =
16
- distance metric =
Euclidean
assumptions (4)
- domain assumption Plans close in LLM embedding space tend to share optimal hint sets.
- domain assumption A frozen pre-trained LLM (OpenAI text-embedding-3-large) gives a meaningful semantic representation of execution plans.
- domain assumption The average runtime of K known neighbouring plans is a reliable proxy for an unseen plan's runtime and can decide whether a hint helps.
- domain assumption A random 10% query-level holdout from a 16-template workload yields independent test and reference sets.
Cite this review
Pith. "Pith review of Training-Free Query Optimization via LLM-Based Plan Similarity." pith.science (2026). https://pith.science/paper/UFSQO535
@misc{pith2026250605853,
author = {Pith},
title = {Pith review of: Training-Free Query Optimization via LLM-Based Plan Similarity},
year = {2026},
howpublished = {\url{https://pith.science/paper/UFSQO535}},
note = {Machine review of arXiv:2506.05853}
}
read the original abstract
Large language model (LLM) embeddings offer a promising new avenue for database query optimization. In this paper, we explore how pre-trained execution plan embeddings can guide SQL query execution without the need for additional model training. We introduce LLM-PM (LLM-based Plan Mapping), a framework that embeds the default execution plan of a query, finds its k nearest neighbors among previously executed plans, and recommends database hintsets based on neighborhood voting. A lightweight consistency check validates the selected hint, while a fallback mechanism searches the full hint space when needed. Evaluated on the JOB-CEB benchmark using OpenGauss, LLM-PM achieves an average speed-up of 21% query latency reduction. This work highlights the potential of LLM-powered embeddings to deliver practical improvements in query performance and opens new directions for training-free, embedding-based optimizer guidance systems.
Reference graph
Works this paper leans on
-
[1]
Proceedings of the VLDB Endowment 12(11), 1705–1718 (2019) https://doi.org/10.14778/3342263
Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M., Kraska, T., Papaem- manouil, O., Tatbul, N.: Neo: a learned query optimizer. Proceedings of the VLDB Endowment 12(11), 1705–1718 (2019) https://doi.org/10.14778/3342263. 3342644
-
[2]
https://arxiv.org/ abs/1809.00677
Kipf, A., Kipf, T., Radke, B., Leis, V., Boncz, P., Kemper, A.: Learned Cardinal- ities: Estimating Correlated Joins with Deep Learning (2018). https://arxiv.org/ abs/1809.00677
arXiv 2018
-
[3]
Woltmann, L., Hartmann, C., Thiele, M., Habich, D., Lehner, W.: Cardi- nality estimation with local deep learning models. In: Proceedings of the 17 Second International Workshop on Exploiting Artificial Intelligence Tech- niques for Data Management. aiDM ’19. Association for Computing Machin- ery, New York, NY, USA (2019). https://doi.org/10.1145/3329859....
-
[4]
In: Pro- ceedings of PGCon 2017, Ottawa, Canada (2017)
Ivanov, O., Bartunov, S.: Adaptive query optimization in PostgreSQL. In: Pro- ceedings of PGCon 2017, Ottawa, Canada (2017). Conference presentation and software available at https://github.com/postgrespro/aqo
work page 2017
-
[5]
Adaptive Cost Model for Query Optimization
Vasilenko, N., Demin, A., Ponomaryov, D.: Adaptive Cost Model for Query Optimization (2024). https://arxiv.org/abs/2409.17136
work page Pith review arXiv 2024
-
[6]
In: Proceedings of the 2021 International Conference on Management of Data
Marcus, R., Negi, P., Mao, H., Tatbul, N., Alizadeh, M., Kraska, T.: Bao: Making learned query optimization practical. In: Proceedings of the 2021 International Conference on Management of Data. SIGMOD ’21, pp. 1275–1288. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/ 3448016.3452838 . https://doi.org/10.1145/34480...
arXiv 2021
-
[7]
Anneser, C., Tatbul, N., Cohen, D., Xu, Z., Pandian, P., Laptev, N., Marcus, R.: Autosteer: Learned query optimization for any sql database. Proc. VLDB Endow. 16(12), 3515–3527 (2023) https://doi.org/10.14778/3611540.3611544
arXiv 2023
-
[8]
Woltmann, L., Thiessat, J., Hartmann, C., Habich, D., Lehner, W.: Fastgres: Making learned query optimizer hinting effective. Proc. VLDB Endow. 16(11), 3310–3322 (2023) https://doi.org/10.14778/3611479.3611528
arXiv 2023
Show all 14 references
-
[9]
https://arxiv.org/ abs/2304.04407
Xu, X., Zhao, Z., Zhang, T., Kang, R., Sun, L., Chen, J.: COOOL: A Learning- To-Rank Approach for SQL Hint Recommendations (2023). https://arxiv.org/ abs/2304.04407
2023 arXiv
-
[10]
Zhu, R., Chen, W., Ding, B., Chen, X., Pfadler, A., Wu, Z., Zhou, J.: Lero: A learning-to-rank query optimizer. Proc. VLDB Endow. 16(6), 1466–1479 (2023) https://doi.org/10.14778/3583140.3583160
2023
-
[11]
https://arxiv.org/abs/2412.02372
Zinchenko, S., Iazov, S.: HERO: Hint-Based Efficient and Reliable Query Opti- mizer (2024). https://arxiv.org/abs/2412.02372
2024 arXiv
-
[12]
https://arxiv.org/abs/2502.05562
Tan, J., Zhao, K., Li, R., Yu, J.X., Piao, C., Cheng, H., Meng, H., Zhao, D., Rong, Y.: Can Large Language Models Be Query Optimizer for Relational Databases? (2025). https://arxiv.org/abs/2502.05562
2025 arXiv
-
[13]
https://arxiv.org/abs/2411.02862
Akioyamen, P., Yi, Z., Marcus, R.: The Unreasonable Effectiveness of LLMs for Query Optimization (2024). https://arxiv.org/abs/2411.02862
2024 arXiv
-
[14]
https://arxiv.org/abs/2404.12872 18
Li, Z., Yuan, H., Wang, H., Cong, G., Bing, L.: LLM-R2: A Large Language Model Enhanced Rule-based Rewrite System for Boosting Query Efficiency (2024). https://arxiv.org/abs/2404.12872 18
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.