{"id":"82493ac4-5d5f-4384-bcc2-b0de30c61bff","arxiv_id":"2506.16379","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PBench synthesizes benchmark workloads that match the CPU time, scanned bytes, and operator distributions of real cloud traces, reducing approximation error by up to 6x over prior tools.","lead":"PBench is a tool that turns anonymized cloud workload traces into synthetic benchmark workloads by mixing queries from TPC-H, TPC-DS, and other benchmarks, then assigning timestamps and using an LLM to generate extra queries. A generalist reader might care because it gives cloud database providers a way to test their systems on realistic, privacy-preserving workloads without exposing customer SQL.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1's additive sum of per-query profiled features is the load-bearing assumption; it ignores concurrency interactions, and the linear slowdown model only patches the timestamp phase, while §6 never isolates whether aggregate fidelity survives under varying concurrency.","rationale":"The reader's weakest_assumption correctly identifies the ILP additivity assumption as the most load-bearing step. The paper's central claim—that synthetic workloads closely approximate real execution statistics—depends on Eq. 1's linear combination of profiled per-component features being a valid predictor of the aggregate statistics of the executed workload. If concurrent queries interact nonlinearly, the selection phase can produce a workload that fits the optimization objective perfectly while failing in actual replay. The paper's only concession to concurrency is the linear slowdown model in the timestamp assignment phase (Section 4.2.1), which does not fix the ILP's underlying additivity assumption. The experiments do not isolate this risk: they show end-to-end approximation errors but do not test how error scales with concurrency or with the degree of parallel execution. A targeted experiment varying concurrency and comparing measured aggregates to the additive prediction would settle whether the concern is material. Because the reader already recommended conditional acceptance based on this same concern, the verdict should remain unchanged rather than being adjusted.","tokens_in":35984,"tokens_out":3874,"duration_ms":41813,"concrete_test":"Run a controlled additivity check: (1) profile a set of 20 components individually on the target cluster; (2) execute mixtures of these components at concurrency levels 1, 4, and 8 (or at the cluster's actual setting) and measure aggregate CPU time, scanned bytes, and operator counts; (3) compare measured aggregates to the linear sum of individual profiles. If the mismatch grows with concurrency or is comparable to the claimed 6x improvement margin (e.g., exceeds 20% relative error), the ILP objective cannot be trusted for realistic concurrent loads. In the paper's own pipeline, replay the same synthesized workload with the concurrency constraint l varied (e.g., 1x and 8x) and check whether window-level GMAPE for CPU time stays within the reported range (17.37% in Table 3) or degrades sharply.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central selection objective (Section 4.1.1, Eq. 1) minimizes the sum of absolute relative errors between the target window statistics and a linear combination of individually profiled component features, summing x_j * m'_hj and x_j * o'_uj across components. This is an additivity assumption: it presumes that when multiple components execute together, total CPU time, scanned bytes, and operator counts equal the sum of single-query profiles. The paper itself concedes that parallel execution changes per-query execution time (Section 4.2.1, citing Auto-WLM [27]) and addresses this only in the timestamp-phase linear slowdown model; the ILP selection phase still treats component features as concurrency-invariant. The duration constraint (total duration <= window length times max concurrency) does not correct for shared scans, cache effects, or resource contention. Consequently, the selected mixture may have near-zero predicted error yet fail to reproduce the target statistics in actual replay on the target cluster. The experimental section never isolates this risk: §6.2–§6.4 compare end-to-end approximation errors but do not report whether error grows with concurrency or with the degree to which the selected workload deviates from the sum-of-parts assumption. Given that the headline claim (up to 6x error reduction) depends on this additive selection being valid, the most load-bearing concern is that Eq. 1's additivity is unvalidated, and the paper's own acknowledgment of inter-query interference is confined to the timestamp stage only.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PBench, a workload synthesizer that takes real cloud workload traces (Snowset, Redset), which contain execution statistics but no SQL or databases, and constructs an executable synthetic workload from benchmark components (TPC-H, TPC-DS, JOB, YCSB). The synthesis pipeline has three stages: an ILP-based component selection that jointly minimizes relative error for performance metrics and operator distributions, a simulated-annealing timestamp assignment phase that refines the temporal placement of selected components, and an LLM-based component augmentation step that generates new queries and databases to close the gap between benchmark components and target traces. The paper evaluates PBench on Snowset and Redset against CAB and Stitcher, reporting substantially lower approximation error, with up to a 6x reduction in GMAPE for CPU Time, and it releases artifacts on GitHub.","tokens_in":36297,"tokens_out":7415,"duration_ms":82875,"significance":"The problem is timely and practical: anonymized cloud traces are increasingly available, but they cannot be executed directly, and existing benchmark-based synthesizers like CAB and Stitcher either use coarse heuristics or ignore operator distributions. If the reported results hold, PBench would be a useful tool for turning real trace statistics into replayable, privacy-preserving benchmarks. The paper deserves credit for formulating the synthesis problem precisely, providing a detailed ILP/SA/LLM pipeline, evaluating on two real traces, and open-sourcing the code. I do not see a hidden circularity in the main evaluation: the error metrics are computed from actually replayed synthetic workloads, not from the ILP objective alone. However, the quantitative claims currently rest on an undisclosed LLM, on stochastic experiments reported without variance, and on a baseline comparison whose candidate pools may not be matched; these issues are fixable but they block acceptance as-is.","major_comments":[{"comment":"The LLM augmenter is described only as \"a state-of-the-art LLM model\" with the model name withheld due to Amazon policy. Since component augmentation is one of the three core stages and the ablation study shows it has a large effect on operator-ratio accuracy, the experimental results are not reproducible without disclosing the model, the model version, and the decoding parameters (temperature, max tokens, prompt format). Please provide this information or a reproducible API-based alternative.","section":"Section 5 and Section 6.1 (footnote 1)"},{"comment":"The evaluation involves multiple sources of randomness: random selection of one-hour traces from Snowset, stochastic simulated annealing, and nondeterministic LLM generation, yet all tables report a single run with no error bars or confidence intervals. This is particularly concerning in Table 3, where PBench reports exactly 0.00 MAE for all four operator ratios; the reader cannot tell whether this is rounding, an exact fit, or an artifact of the aggregation. Please report means and standard deviations over repeated runs and clarify the exact-zero values.","section":"Section 6.2 and Tables 3-6"},{"comment":"For CAB, the paper states that \"we use all our prepared workload components queries to construct the query pool for a fair comparison,\" but the LLM-generated components are introduced as part of PBench's augmentation stage. If CAB and Stitcher did not receive the same augmented candidate pool, the comparison conflates the effect of the synthesis algorithm with the effect of the enlarged and better-matched candidate set. Please rerun the baselines with the identical augmented pool, or clearly state which candidate pool each method used and justify the asymmetry.","section":"Section 6.1, Baseline setup"},{"comment":"The ILP objective sums individually profiled component features (CPU time, scanned bytes, operator counts), which assumes that the aggregate statistics of concurrently running queries equal the sum of single-query profiles. Section 4.2.1 acknowledges that parallel execution changes per-query execution times and patches only the timestamp phase with a linear slowdown model, but the paper never isolates whether the additive selection assumption degrades aggregate fidelity under concurrency. The end-to-end replay results are encouraging, but the paper should add a predicted-vs-actual error decomposition at the window level, similar to the query-level decomposition in Figure 9, and report whether the error grows with concurrency or with the degree to which the selected workload deviates from the sum-of-parts assumption.","section":"Section 4.1.1, Eq. (1)"},{"comment":"As printed, the GMAPE formula uses an exponent of 1/(n-1) and has no final \"-1\", which is not the standard geometric-mean absolute percentage error. If the formula was implemented literally, all GMAPE values in Tables 3, 4, 6, and 8 would be computed incorrectly. Please correct the equation and verify the reported numbers, or explicitly define the intended statistic if a nonstandard metric is used.","section":"Section 6.1, Eq. (3)"}],"minor_comments":[{"comment":"\"This section first some presents background information\" is ungrammatical; revise to \"This section first presents background information.\"","section":"Section 2, opening sentence"},{"comment":"\"Therfore\" should be \"Therefore.\"","section":"Section 6.2, Efficiency Evaluation"},{"comment":"The definition of O as a \"percentage distribution\" conflicts with the additive use of o'_uj in Eq. (1), which treats operator counts (e.g., \"Agg Num\" in Figure 4) as quantities that can be summed. Please clarify whether the objective operates on ratios, counts, or both, since this affects how the operator terms should be normalized.","section":"Section 2.2 and Eq. (1)"},{"comment":"Profiling each query three times and averaging is a reasonable cache-handling strategy, but the paper does not report the variance across the three runs; a brief statement on run-to-run variability would help the reader judge the reliability of the profiled features.","section":"Section 3.2.2"},{"comment":"The K-Means centroid count k used in Section 5 is not listed among the hyperparameters. Please specify k and, ideally, report sensitivity of the augmentation stage to this parameter.","section":"Section 6.1, Hyperparameter settings"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for PVLDB, the problem is well motivated, and the proposed pipeline is sensible. The main obstacles to acceptance are experimental reproducibility and fairness: the undisclosed LLM, the absence of variance reporting, and the possibility that baselines were compared without the augmented candidate pool. I recommend major revision because these issues are fixable with additional experiments and disclosure rather than by reworking the core approach. Please also ask the authors to confirm explicitly in the revised version whether CAB and Stitcher had access to the LLM-generated components; if they did not, that is the most important correctness issue to address."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: PBench is worth taking seriously. It formalizes a problem that hasn't been crisply posed before — synthesize executable benchmark workloads that match real execution statistics from traces without queries or data — and the three-stage pipeline (ILP selection, simulated annealing time assignment, LLM augmentation) is a sensible decomposition. The experiments on Snowset and Redset are extensive, and the open-sourced code is more than most papers in this area offer.\n\nWhat the paper does well: the ILP formulation with duration, diversity, and count constraints is a reasonable first cut; the timestamp phase visibly improves interval-level fit; the LLM augmentation with positive/negative examples and trial-and-error profiling is a pragmatic use of LLMs. The ablation study (w/o TA, w/o LLM) isolates each contribution, and the error decomposition in Figure 9 (model vs run) is a nice touch that suggests the linear modeling assumption is workable in practice.\n\nThe soft spots, in order of severity. First, the LLM is not named because of Amazon policy. That is an obstacle to reproducibility; the paper should at least give the model family or a reproducible open-weights alternative. Second, no error bars or multiple seeds anywhere, though the SA, the LLM, and the random trace selection are all stochastic. This makes the exact 0.00 MAE values for operator ratios in Table 3 look too clean — those are almost certainly rounded, but the paper should show the underlying distribution. Third, the headline 'up to 6x' cherry-picks the best GMAPE on CPU time; the typical improvement, while still solid, is smaller, and the paper should quote the range. Fourth and most important: the selection objective in Eq. 1 assumes profiled features add linearly across components. The paper acknowledges concurrency interference but only patches it with a linear slowdown model in the timestamp phase. The end-to-end experiments on real traces give some comfort — if the assumption were badly wrong, the replay errors would likely be worse — but no experiment directly varies concurrency or isolates the additive-selection error. That is a gap to fix, not a fatal flaw.\n\nBottom line: this paper deserves a serious referee. It is a solid contribution to the benchmarking literature, addresses a real gap, and its code and data are available. I would accept it with major revision requiring: model disclosure or a workable substitute, error bars, a more measured headline, and an explicit concurrency-stress experiment. Send it out.","headline":"PBench is a genuinely new and well-engineered workload synthesis pipeline for cloud analytics benchmarking, but the evaluation needs transparency on LLM, error bars, and a direct test of the additivity assumption.","tokens_in":36867,"tokens_out":3759,"would_cite":false,"duration_ms":38103,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PBench claims that anonymized cloud workload traces, which contain execution statistics but no SQL or data, can be turned into executable synthetic workloads that closely reproduce those statistics, cutting approximation error by up to 6×…","keywords":["workload synthesis","cloud analytics benchmarking","workload traces","integer linear programming","simulated annealing","LLM query generation","operator distributions","performance metrics"],"falsifier":"Take a real trace dominated by a few very large queries whose resource consumption does not split linearly across benchmark components (e.g., a query whose cost is dominated by a single hash join with a large memory footprint), synthesize a workload with PBench, and compare the replayed CPU time and scanned bytes to the trace; if the GMAPE exceeds the reported range even with augmentation, the additivity and transfer assumptions fail in a detectable way.","tokens_in":35788,"feed_emoji":"📊","tokens_out":4871,"duration_ms":46747,"temperature":0.7,"pith_summary":"The paper introduces a new problem: turning anonymized cloud workload traces, which contain performance metrics and operator distributions but no SQL or underlying databases, into synthetic workloads that reproduce those statistics when executed. PBench solves this by selecting and combining queries and databases from existing benchmarks, assigning timestamps to preserve temporal dynamics, and augmenting the candidate set with LLM-generated components. The authors claim this reduces approximation error by up to 6 times compared to prior synthesis tools like Stitcher and CAB. If correct, PBench lets cloud vendors and researchers benchmark and tune database systems on realistic, privacy-preserving workloads derived from real production statistics.","feed_headline":"Synthetic workload generator cuts cloud benchmark error by 6x","feed_subtitle":"Mixes benchmark queries, smart timestamps, and LLM-generated SQL so anonymized traces become executable benchmarks.","key_machinery":"The central object is the workload component—a benchmark query paired with its populated database—with profiled performance features. The argument is carried by an integer linear program (ILP) whose objective is the sum of relative errors between the synthesized and real window-level performance features, solved to choose repetition counts of each component; a simulated annealing pass that refines query start times to match interval-level CPU time and scanned bytes; and an LLM augmentation loop that generates new components from centroid targets, using nearest and farthest examples as hints and trial-and-error regeneration.","core_discovery":"PBench claims that the statistics of a real cloud workload can be closely approximated by a carefully chosen mixture of profiled benchmark components, even though the original queries and data are unknown. Component selection is cast as an integer linear program that minimizes relative error across both performance metrics (CPU time, scanned bytes) and operator distributions (filter, aggregate, join, sort ratios), subject to duration, diversity, and count constraints. A simulated-annealing step then assigns timestamps to the selected components under a linear concurrency slowdown model to match the target's temporal peaks and valleys at fine intervals. When benchmark components cannot cover the target statistics, an LLM-based augmenter generates new queries on existing or rescaled databases, guided by positive and negative examples and iterative profiling feedback. On Snowset and Redset traces, PBench reports up to 6× lower approximation error than Stitcher and CAB.","pith_inferences":["Editorial inference: The additivity assumption behind the ILP could be stress-tested by synthesizing workloads that deliberately combine components with known nonlinear interactions (e.g., memory-bound joins competing for cache); if the error grows with interaction strength, the objective would need interaction terms.","Editorial inference: The LLM augmentation step is not specific to TPC-H or TPC-DS; it could generate components for other benchmarks or domain-specific schemas, provided profiling feedback is available, which would extend the method to workloads with unusual operator mixes.","Editorial inference: Because the method only needs aggregated statistics, it could be applied to traces collected at coarser granularity (e.g., minute-level logs) or to metrics like memory and IOPS, assuming those features are also additive.","Editorial inference: If many vendors adopt this approach, benchmark-based research could shift from fixed TPC workloads to trace-driven workload synthesis, changing how experimental sections of database papers are designed."],"forward_implications":["Cloud vendors can release executable benchmarks derived from their anonymized traces without exposing customer SQL or data, since only statistics are used.","Database optimizers and cost models can be evaluated on workloads whose operator distributions match production, potentially changing which join and aggregation strategies are chosen.","The synthesis is fast enough—about 14 minutes per one-hour trace—to support near-real-time workload replay, making it practical for continuous benchmarking.","Query-level synthesis with one-to-many ILP matching outperforms one-to-one matching, suggesting workload combination, not single-query mimicry, is the right granularity."],"supporting_citations":[{"why":"Supplies the Redset cloud workload trace used as a target and for query-level experiments.","marker":"[29]"},{"why":"Supplies the Snowset cloud workload trace used as the main window-level target.","marker":"[31]"},{"why":"Provides the CAB baseline and the query arrival pattern summarization method reused in PBench's trace selection.","marker":"[30]"},{"why":"Provides the Stitcher baseline that PBench compares against and whose Bayesian optimization approach it contrasts with ILP.","marker":"[33]"},{"why":"Supplies the linear concurrency slowdown model used in timestamp assignment to estimate query durations under parallel execution.","marker":"[27]"}],"fun_headline_variants":["PBench mixes benchmarks to mimic cloud traces 6x closer","LLM-augmented workload synthesis closes cloud gap by 6x","Synthetic cloud workloads now 6x more accurate via PBench","Cloud benchmark synthesis cut error 6x with LLM and ILP","Workload synthesizer mirrors real cloud stats, 6x better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole construction assumes the performance features of a mixture are the weighted sum of individually profiled components, so if queries interact nonlinearly under concurrency or profiling costs do not transfer to the target cluster, the selected workload will not reproduce the target statistics.","fun_headline_variants_meta":{"raw":{"variants":["PBench mixes benchmarks to mimic cloud traces 6x closer","LLM-augmented workload synthesis closes cloud gap by 6x","Synthetic cloud workloads now 6x more accurate via PBench","Cloud benchmark synthesis cut error 6x with LLM and ILP","Workload synthesizer mirrors real cloud stats, 6x better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":3054,"prompt_tokens":992,"completion_tokens":2062,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1970}},"tokens_in":608,"tokens_out":2062,"duration_ms":14823,"temperature":1.0,"reasoning_tokens":1970,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:27:13.293096+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real trace dominated by a few very large queries whose resource consumption does not split linearly across benchmark components (e.g., a query whose cost is dominated by a single hash join with a large memory footprint), synthesize a workload with PBench, and compare the replayed CPU time and scanned bytes to the trace; if the GMAPE exceeds the reported range even with augmentation, the additivity and transfer assumptions fail in a detectable way.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Redset cloud workload trace used as a target and for query-level experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Snowset cloud workload trace used as the main window-level target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CAB baseline and the query arrival pattern summarization method reused in PBench's trace selection."},{"cited_title":"Ciortea, Konstantinos Karanasos, and Subru Krishnan","cited_arxiv_id":null,"evidence_quote":"Provides the Stitcher baseline that PBench compares against and whose Bayesian optimization approach it contrasts with ILP."}],"review_version":1}