{"id":"c1aa75ea-399b-4e3e-831e-4a1094c83465","arxiv_id":"2608.07945","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ScaleSense predicts per-query resource and latency quantiles from execution-plan graphs and uses them to choose cost-effective compute configurations, reporting a 76.7% relative improvement in constraint satisfaction over baselines.","lead":"Alibaba and Zhejiang University researchers built ScaleSense, a system that predicts how much CPU, memory, disk, and network each database query will use at several cluster sizes, then picks the cheapest size that meets a user's speed target. On production and TPC-DS workloads it reports better constraint satisfaction than the baselines, with up to 5.22x lower cost under a performance-optimization policy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-plan assumption is load-bearing: ScaleSense scores every candidate CU configuration from one optimizer plan, so any CU-dependent plan change invalidates the counterfactual resource/latency predictions that drive the 76.7% CSA improvement.","rationale":"The paper is careful in several respects: it uses physically measurable labels, reports quantile coverage, isolates the FIHC contribution via SS-naive ablation, and evaluates on TPC-DS under all six CU configurations. Those design choices make the central claim credible in-distribution. My stress test focused on what must be true for the counterfactual CU comparisons to be meaningful: a single plan DAG must remain a sufficient representation of the query across CU configurations. This assumption is load-bearing because Problem 1, Algorithm 1, and the controller in Section 4.3 all consume one plan and vary only the CU feature. A direct empirical check can settle the issue without access to production data beyond the TPC-DS setup already used. If plans are stable across CUs, the concern evaporates and the conditional verdict could be upgraded; if plans change, the quantitative headline is weakened because predictions are being scored against plans that would not run. This is the same concern the Reader identified as the weakest assumption, so my agreement is 'agree'. I do not find an internal inconsistency or a more fundamental flaw; the limitation is that the paper does not provide this plan-stability check, and the conditional verdict appropriately reflects that missing evidence.","tokens_in":958,"tokens_out":1016,"duration_ms":99828,"concrete_test":"On TPC-DS, select 10-20 representative queries and force the AnalyticDB optimizer to generate the physical plan at 16, 64, 128, 256, and 512 CUs (via session-level CU parameters or optimizer hints). Record operator sequences, join algorithms, exchange types, and estimated rows/bytes, and compare plans using graph-edit or operator-sequence distance. If a majority of queries change plan structure between 16 and 512 CUs, rerun the Section 5.3 CSA evaluation with per-config plans fed to the encoder (i.e., encode the plan actually used at each candidate CU) and check whether the 27.2 pp average improvement and 76.7% relative improvement persist. If CSA drops materially (e.g., more than 10 pp), the single-plan assumption is load-bearing; if CSA is unchanged, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central scaling claims (76.7% relative CSA improvement, up to 5.22x cost reduction under performance optimization) require the predictor to forecast resource usage and latency for every candidate CU configuration from a single physical plan DAG. Section 4.1 and Problem 1 construct one plan embedding z and feature vector f, and Algorithm 1 chunks that same plan for all candidates; Section 4.3 then varies only the CU-valued feature when scoring each configuration. But physical plans are not configuration-invariant: at 512 CUs an optimizer can legally choose different join orders, exchange distributions, or parallelism, and a larger memory budget changes spill behavior. Table 1 itself shows footprints changing with scale, but nothing in Section 5 records whether the plan DAG fed to the encoder is the plan that would actually run under each candidate CU. Section 7's Lesson 1 explicitly admits that counterfactual ground truth across configurations was not collected, so the paper offers no evidence of plan stability. Because the FIHC controller's pressure and position signals are computed from Stage 1 predictions for those same stale plans, a plan change would propagate directly into the recommendation and could make the reported CSA gains an artifact of scoring plans that would not execute.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ScaleSense, a per-query auto-scaling framework for Alibaba AnalyticDB. It encodes optimizer physical plan DAGs with a graph transformer and plan-tree summarization, predicts quantiles (Q10, Q50, Q90) of CPU time, peak memory, disk scan volume, shuffle size, and end-to-end latency via a two-stage XGBoost quantile regression model with zero-inflated handling, and then selects a CU configuration with a Factor-Informed Hurwicz (FIHC) controller that blends optimistic and pessimistic latency estimates using resource-pressure and interval-position signals. The evaluation uses two production datasets (D1, D2) for prediction accuracy and the TPC-DS 1TB benchmark for auto-scaling. The abstract claims a 76.7% relative improvement in constraint satisfaction accuracy (CSA) over the best baseline and up to 5.22x monetary cost reduction under the performance-optimization policy.","tokens_in":20772,"tokens_out":4678,"duration_ms":51910,"significance":"If the results are robust, the work is significant: it targets a real industrial problem (per-query resource provisioning in serverless data warehouses), uses large-scale production data, and introduces an uncertainty-aware controller that goes beyond point-estimate scaling. The strengths are the explicit multi-dimensional resource prediction, the quantile interval framework, the zero-inflated classifier for sparse targets, and the low reported inference overhead. These are useful contributions to the learned resource estimation and auto-scaling literature. However, the central auto-scaling claim rests on a single-plan assumption that is not validated, and the CSA results are reported without variance or significance testing on a small benchmark (99 queries). The paper also acknowledges in Section 7 that counterfactual ground truth across CU configurations was not collected, which further limits the strength of the end-to-end claim.","major_comments":[{"comment":"The auto-scaling evaluation assumes that the optimizer's physical plan DAG is invariant to the CU configuration. Problem 1 defines f(q, a) with a single plan embedding z, Algorithm 1 chunks that same plan for all candidates, and Section 4.3 varies only the CU-valued feature when scoring configurations. However, physical plans can change with resources (join orders, exchange distributions, memory-driven spill behavior), and the paper never records whether the plan DAG used for prediction is the plan that would actually run under each candidate CU. Table 1 shows resource footprints changing with scale, but it does not show plan DAG differences. Section 7's Lesson 1 explicitly states that full replay across CU configurations was infeasible, so no evidence of plan stability is provided. Since the FIHC pressure and position signals are computed from Stage 1 predictions on those plans, a plan change would propagate into the recommendation and could make the reported 76.7% CSA improvement an artifact of scoring plans that would not execute. This is a load-bearing gap for the paper's main claim.","section":"Section 4.1, 4.3, and Problem 1"},{"comment":"The CSA results are reported on only the 99 TPC-DS queries, with no repeated train/test splits, no confidence intervals, and no significance tests. The average improvement of 27.2 pp over the best baseline is the mean over six hand-picked constraint settings; given the small test set and the variance visible in Figures 9 and 10, this difference may be within noise. Similarly, the 'up to 5.22x' cost reduction is a maximum over individual queries, not a typical or average outcome, and the paper presents it without a confidence estimate. The paper should provide per-setting results with repeated evaluation, or at least bootstrap intervals, to support the quantitative headline claim.","section":"Section 5.3"},{"comment":"The FIHC controller's alpha parameter is assigned by a heuristic threshold: alpha=0 for performance-optimization policies and alpha=1 for cost-optimization policies, based on a threshold of 5 on the sum of user scaling ratios. This assignment is not derived from the optimization problem and no sensitivity analysis is given. Because the paper attributes the majority of the CSA gain to FIHC (+27.1 pp over SS-naive), the result is contingent on this hand-chosen mapping. The paper should either provide a principled way to set alpha from user preferences, or report how CSA varies with alpha to show the gain is not a tuning artifact.","section":"Section 5.3, FIHC alpha assignment"},{"comment":"The claim that the quantile prediction acts as a 'reliable safety net' is weakened by under-coverage on the most safety-critical dimension. On D1, the 80% prediction interval for peak memory achieves only 68.03% coverage, which is well below the nominal 80%. Since memory under-estimation is the most catastrophic failure mode (OOM), the paper should discuss the calibration gap, consider wider intervals or conformal calibration for memory, and explain why the current coverage is acceptable for production deployment.","section":"Table 2"}],"minor_comments":[{"comment":"The abstract states 'Evaluations on over 1.36 million production queries,' but the auto-scaling evaluation is performed only on TPC-DS 1TB (99 queries); the production datasets D1 and D2 are used for prediction accuracy, not for end-to-end scaling. Please clarify this distinction in the abstract and introduction.","section":"Abstract and Section 5.1"},{"comment":"The relative Q-error plots would be more informative if accompanied by absolute Q-error values or a table, and if error bars or significance markers were included. The text reports improvements like '1.6%' and '5.1%' for latency and CPU, which are small; the presentation should make clear which differences are practically meaningful.","section":"Section 5.2, Figure 8"},{"comment":"The complexity analysis states that the full-plan Laplacian eigen-decomposition costs O(N^3), then claims the summarization reduces this to linear in N for fixed C. This is correct only if the eigen-decomposition is the dominant cost and the chunk size C is constant; please state the assumption about the sparsity of the plan DAG and the cost of eigen-decomposition for the typical plan sizes encountered.","section":"Section 4.1, Algorithm 1"},{"comment":"Reference [5] is listed as 'The SafeLoad authors' with a GitHub URL, which is not a proper citation. This should be replaced with a concrete author list and venue, or merged with the SafeLoad citation [33].","section":"References"},{"comment":"The text says each CU is 'one CPU core paired with 4GB of main memory' and later that each CU is 'provisioned as an isolated lightweight VM.' These statements are not obviously consistent; please clarify whether a CU is a VM or a resource unit within a VM, as this affects the resource pressure signal definition.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a strong industrial group and the prediction results on production data are plausible. The main reservation is that the end-to-end scaling claim depends on an untested single-plan assumption and on a small benchmark without uncertainty quantification. The authors may be able to address this by adding plan-stability analysis (even on a subset of TPC-DS queries where different CU plans can be captured), by reporting repeated-split or bootstrap results, and by providing a sensitivity analysis for the FIHC alpha. The paper is not ready for acceptance in its current form, but the core idea is worth a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a real industrial systems paper with a genuinely new controller, but the headline scaling claims rest on an assumption about plan stability that the paper never checks. Worth sending to serious reviewers, though they should push hard on that assumption.\n\nWhat it does well: the integration of plan-topology encoding (graph transformer with Laplacian PE) with two-stage quantile regression for CPU, memory, disk, network, and latency is well thought out, and the zero-inflated classifier is a sensible response to real data skew. The Factor-Informed Hurwicz criterion is the genuinely novel piece—adaptively blending optimistic and pessimistic quantiles using resource pressure and interval-skew signals is a practical way to make uncertainty actionable, and the ablation against SS-naive shows it matters. The scale of the production evaluation (1.36M queries) is a serious strength, and the authors are candid about deployment constraints in Section 7.\n\nThe soft spot that matters: the predictor scores every candidate CU configuration from a single optimizer plan DAG. The paper never verifies whether the physical plan is stable across CU configurations. In most distributed engines, changing CUs changes parallelism and exchange structure; if that happens here, the resource and latency predictions for alternative CUs are for plans that would not run. This is load-bearing because the FIHC controller's pressure and position signals are computed from those very predictions. The authors could have validated it on TPC-DS—they already measured all 99 queries at six CU sizes—but they don't report whether plans were identical. Lesson 1 says production queries can't be replayed, but that doesn't excuse the TPC-DS case, where they could have compared plans.\n\nSecondary issues: the auto-scaling evaluation is on 99 queries with no error bars or repeated splits, the FIHC alpha is hand-set per policy (α=0 for PO, α=1 for CO), and a couple of the baselines (LSR, SS-naive) are trivial. These are real but minor next to the plan-stability question.\n\nOverall: the central thrust is sound and the system is genuinely deployed. I'd send it to peer review with a clear request that the authors either demonstrate plan stability across CU sizes or re-frame the claims to acknowledge the limitation. The ideas are citable even with that caveat.","headline":"A serious industrial scaling paper whose Factor-Informed Hurwicz controller is genuinely new and whose production evaluation is large, but the headline CSA gains rest on a single-plan assumption the paper never validates.","tokens_in":21269,"tokens_out":3822,"would_cite":true,"duration_ms":44550,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScaleSense claims that choosing a query's compute units should be driven by quantile forecasts of four physical resource bottlenecks rather than by a point latency estimate.","keywords":["serverless data warehouse","auto-scaling","resource estimation","quantile regression","query plan encoding","graph transformer","performance-cost trade-off","AnalyticDB"],"falsifier":"Run a sample of TPC-DS queries at 16 and 512 CUs, capture the optimizer's physical plans at each size, and compare the DAGs: if join orders, exchange placement, or operator choices differ materially between configurations, or if actual cardinalities fall outside the predictor's stated intervals, then ScaleSense's forecasts are for a plan that would not run, which would contradict the assumption that one plan supports all candidates. Comparing predicted versus actual CPU, memory, disk, and network footprints under re-optimized plans would settle the claim directly.","tokens_in":20281,"feed_emoji":"☁️","tokens_out":6720,"duration_ms":70038,"temperature":0.7,"pith_summary":"ScaleSense is a framework for deciding, for each ad-hoc query in a serverless data warehouse, how many Compute Units (CUs) to allocate before the query runs. It claims that the right basis for this decision is not a single predicted latency but a set of quantile forecasts for four physical bottlenecks—CPU time, peak memory, table scan volume, and shuffle size—conditioned on each candidate configuration, together with a latency interval derived from them. An auto-scaling controller then blends the optimistic and pessimistic latency bounds adaptively, using the predicted resource pressure as a risk signal, so the chosen configuration respects user-defined cost and performance constraints. The paper reports that on 1.36 million production queries and TPC-DS, this yields a 76.7% relative improvement in constraint-satisfying configuration selection over the best baseline, with up to a 5.22x monetary cost reduction under the performance-optimization policy. If correct, this replaces reactive over-provisioning with proactive, preference-aware allocation.","feed_headline":"Quantile bottleneck forecasts cut cloud query cost by up to 5.22x","feed_subtitle":"ScaleSense picks per-query compute size from predicted CPU, memory, disk, and network ranges, not point estimates.","key_machinery":"Three components carry the argument. (1) Multi-faceted Query Encoder: a Graph Transformer with Laplacian positional encoding on the plan DAG, chunked by plan-tree summarization (200-node chunks) when plans are large, produces a 64-dimensional plan embedding that is concatenated with 157 plan-statistic features, giving a 221-dimensional query representation. (2) Quantile Resource Predictor: Stage 1 gradient-boosted quantile regression predicts Q10, Q50, and Q90 for CPU time, peak memory, table scan volume, and shuffle size, with monotonicity clipping and a zero-inflated hurdle classifier for sparse dimensions; Stage 2 learns latency quantiles from these factor quantiles. (3) Factor-Informed Hurwicz Criterion: the controller combines a resource-pressure signal, based on predicted per-CU memory and CPU load, and a factor-position signal, based on where the median sits inside each prediction interval, into an adaptive optimism parameter that blends Q10 and Q90 latency before solving the user's constrained minimization over candidate CU sets.","core_discovery":"At the paper's center is a two-stage learned predictor. Stage 1 estimates, for each candidate CU configuration, the 10th, 50th, and 90th percentiles of CPU time, peak memory, table scan volume, and shuffle size, using XGBoost quantile regression on a 221-dimensional feature vector built from a Graph Transformer plan embedding and plan statistics; a zero-inflated hurdle classifier handles the large fraction of exact-zero I/O and memory values. Stage 2 predicts latency quantiles from those factor quantiles plus features, so the controller receives calibrated uncertainty rather than a point estimate. The Factor-Informed Hurwicz Criterion then chooses the configuration by interpolating Q10 and Q90 latency with a per-configuration weight derived from resource pressure (predicted Q90 memory and CPU per CU) and from the skewness of the factor prediction intervals. The claim is that this uncertainty-aware pipeline selects configurations satisfying both latency and cost constraints far more often than point-estimate baselines: average constraint satisfaction accuracy improves by 27.2 percentage points, and the gain is attributable mainly to the controller rather than to the encoder.","pith_inferences":["Going beyond the paper: because the predictor is trained on one physical plan per query, the approach would need re-validation in settings where the optimizer re-optimizes the plan as the CU allocation changes; a testable extension is comparing predicted footprints against actual execution under plan changes.","Going beyond the paper: the same quantile-then-controller pattern could transfer to other serverless SQL systems with standardized compute units, such as Redshift RPUs or Snowflake warehouses, as long as execution logs expose the four physical counters.","Going beyond the paper: the prediction intervals themselves could serve as admission-control signals, not just scaling signals; a query whose Q90 memory exceeds the largest candidate configuration could be flagged before scheduling.","Going beyond the paper: a natural stress test is to measure whether the 27.2 percentage point gain persists when candidate configurations are not a small discrete ladder but a continuous range, since the controller's pressure signal normalizes across candidates."],"forward_implications":["Users can shift between performance-optimization and cost-optimization policies by changing two parameters, epsilon and rho, without retraining the predictor.","I/O-bound or network-bound queries will not be blindly given more CUs, because the predictor identifies the bottleneck dimension that scaling cannot remove.","Memory-sensitive queries get conservative allocations because the Q90 memory bound is used as a safety guardrail against OOM.","The framework's relative ranking across candidate configurations matters more than absolute accuracy, so out-of-distribution robustness can be evaluated on benchmarks like TPC-DS.","Inference overhead below 1% of query execution time makes per-query decisions feasible in the critical path."],"supporting_citations":[{"why":"Supplies the plan-level statistics feature set and the production OOM analysis that motivates the safety net; also defines the XGB baseline ScaleSense is compared against.","marker":"[33]"},{"why":"Provides the Laplacian-positional-encoding Graph Transformer architecture that the encoder adapts for query plan DAGs.","marker":"[18]"},{"why":"Basis for the gradient-boosted quantile regression used to produce calibrated prediction intervals.","marker":"[26]"},{"why":"RAIS is the state-of-the-art intelligent scaling system used as a baseline for the auto-scaling comparison.","marker":"[21]"},{"why":"LearnedWMP's MLP memory-demand predictor is used as a point-estimate baseline for resource estimation.","marker":"[24]"},{"why":"Original Hurwicz criterion and its query-evaluation uses that the Factor-Informed Hurwicz Criterion extends.","marker":"[13, 14, 32]"},{"why":"QueryFormer's tree-transformer plan encoding is the approach the encoder distinguishes itself from.","marker":"[37]"}],"fun_headline_variants":["Quantile resource forecasts cut cloud query costs up to 5.22x","Uncertainty-aware scaling selects compute size, cuts cost 5.22x","Predict resource ranges, not point estimates: cloud scaling saves 5.22x","ScaleSense: quantile-based auto-scaling reduces cloud query costs 5.22x","Per-query quantile forecasts pick optimal compute, cutting costs 5.22x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the physical query plan and the optimizer's row and cardinality estimates stay valid for every candidate CU configuration, so one plan per query can predict resource use at 16, 64, and 512 CUs alike.","fun_headline_variants_meta":{"raw":{"variants":["Quantile resource forecasts cut cloud query costs up to 5.22x","Uncertainty-aware scaling selects compute size, cuts cost 5.22x","Predict resource ranges, not point estimates: cloud scaling saves 5.22x","ScaleSense: quantile-based auto-scaling reduces cloud query costs 5.22x","Per-query quantile forecasts pick optimal compute, cutting costs 5.22x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000894,"raw_usage":{"total_tokens":3900,"prompt_tokens":1039,"completion_tokens":2861,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":2753}},"tokens_in":655,"tokens_out":2861,"duration_ms":25749,"temperature":1.0,"reasoning_tokens":2753,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:38:12.125231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a sample of TPC-DS queries at 16 and 512 CUs, capture the optimizer's physical plans at each size, and compare the DAGs: if join orders, exchange placement, or operator choices differ materially between configurations, or if actual cardinalities fall outside the predictor's stated intervals, then ScaleSense's forecasts are for a plan that would not run, which would contradict the assumption that one plan supports all candidates. Comparing predicted versus actual CPU, memory, disk, and network footprints under re-optimized plans would settle the claim directly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the plan-level statistics feature set and the production OOM analysis that motivates the safety net; also defines the XGB baseline ScaleSense is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Laplacian-positional-encoding Graph Transformer architecture that the encoder adapts for query plan DAGs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Basis for the gradient-boosted quantile regression used to produce calibrated prediction intervals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RAIS is the state-of-the-art intelligent scaling system used as a baseline for the auto-scaling comparison."},{"cited_title":"LearnedWMP: Workload Memory Prediction Using Distribution of Query Templates","cited_arxiv_id":"2401.12103","evidence_quote":"LearnedWMP's MLP memory-demand predictor is used as a point-estimate baseline for resource estimation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QueryFormer's tree-transformer plan encoding is the approach the encoder distinguishes itself from."}],"review_version":1}