Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling

Jonathan Will; Lauritz Thamsen; Nico Treide; Odej Kao

REVIEW 4 major objections 5 minor 17 references

Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Autoscaling Spark jobs: no efficiency gain over static setups

desk verdict New dataset and a plausible negative result, but the 'no significant gain' claim is stronger than single-run data can support and needs error bars or softer wording. read the letter →

arxiv 2501.14456 v1 pith:OK4TXZ24 submitted 2025-01-24 cs.DC

Jonathan Will , Nico Treide , Lauritz Thamsen , Odej Kao This is my paper · ORCID

classification cs.DC

keywords autoscalingresourceefficiencyApacheSparkDataprocServerlessdistributeddataflowcloudallocationexecutorsecondsbenchmarktrace

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autoscaling is often presented as a way to avoid choosing static resources for Spark batch jobs. This paper tests that claim by running 18 Spark jobs on Google Dataproc Serverless, comparing an autoscaling configuration with two static allocations and ten further fixed configurations. Across the 234 executions, the autoscaling configuration produced the lowest resource use in only 2 of 18 jobs, while the two static configurations did so in 9 and 7 jobs. The authors conclude that autoscaling offers no significant resource-efficiency gain over static allocations, and they attribute this to the fact that autoscaling only adjusts the number of executors, leaving node size and the memory-to-CPU ratio fixed.

What carries the argument

The central object is Dataproc Serverless's autoscaling mechanism, which adjusts only the scale-out (number of executors) within a configured range while keeping the instance type (n2-standard-4) and hence the per-node memory-to-CPU ratio fixed. The evaluation metric is executor-seconds, with vCPU seconds and GiB-seconds used as secondary cost metrics under a 1:6 pricing weight. The configuration space R01-R10 is designed to isolate the effects of cluster memory, CPU cores, and scale-out separately, which lets the authors show that node type and resource mix matter more than dynamic executor count. The argument works by comparing this 'inelastic' autoscaling against static allocations that are free to vary node size and resource mix.

What would settle it

Run the same 18 jobs on the same 13 configurations several times each, and check whether the autoscaling configuration S1 becomes the lowest-cost option for more than a small minority of jobs; additionally, if any job can be shown to be most efficient under autoscaling while node size and memory-to-CPU ratio are both fixed, the paper's central conclusion would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that horizontal autoscaling of Spark batch jobs, as implemented in Google Dataproc Serverless, does not yield a significant resource-efficiency benefit over static resource allocations for the tested workloads. The evidence is a trace of 234 job executions: the dynamic configuration S1, scaling from 2 to 32 executors, achieved the lowest normalized executor-seconds for only 2 of 18 jobs, whereas the static configurations S2 (8 executors) and S3 (16 executors) were best in 9 and 7 jobs respectively. Autoscaling did tend to produce shorter runtimes, but resource consumption was comparable to or slightly higher than the best static choice. The authors locate the cause in the inelasticity of node size and of the memory-to-CPU ratio: the autoscaling mechanism selects an optimal node type once and then varies only the number of nodes, so it cannot adapt the resource mix to the job's actual demands.

Load-bearing premise

Each job was executed only once per cloud configuration, so the measured differences in executor-seconds could be outweighed by run-to-run variance; if repeated runs showed large variation, the ranking of autoscaling versus static allocations could change.

Editorial extensions

If this is right

If the claim holds, users can treat autoscaling as a convenient default that shortens runtimes, but not as a way to minimize resource cost; per-job static configuration remains the more efficient option.
Managed cloud services could improve efficiency by letting autoscaling vary node size and the memory-to-CPU ratio, not just the number of executors.
Choosing the single best configuration across all jobs leaves about a quarter of potential cost savings on the table compared with selecting the best configuration per job.
Under fluctuating resource prices, a fixed RAM-to-CPU ratio forces autoscaling to be less cost-efficient than a configuration matched to the current cost structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The single-run design means the 2-of-18 count could shift with repeated executions; a replication study with multiple runs per configuration would clarify whether the ranking is stable.
The negative result for Spark's scale-out-only autoscaling likely extends to other dataflow systems (e.g., Flink) that implement similar horizontal autoscaling, though this was not tested here.
A direct test of the mechanism would be to give autoscaling the ability to change node size or memory-to-CPU ratio at runtime; the paper predicts this would close much of the efficiency gap.
The dataset could be reused to build a per-job configuration recommender, since the results show that knowing the job's resource access pattern matters more than runtime scaling.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the paper. The useful thing here is the dataset: 234 Spark executions across 13 configurations, with public code and data. That alone makes this worth a look. The empirical comparison of Dataproc Serverless autoscaling against static allocations is new, as far as I know, and the cost-model sensitivity analysis (Figure 3) is a genuine strength — it shows how conclusions shift as the relative price of memory and CPU changes.

The central claim, however, is stated too strongly. The abstract says 'no significant resource efficiency gain,' but the data cannot support the word 'significant.' Each job-configuration pair was run exactly once. Many entries in Table V are within 5–10% of 1.0, and run-to-run variance on shared cloud infrastructure for distributed Spark jobs is often at least that large. Without repeated runs, confidence intervals, or a statistical test, the 2/9/7 win-count for S1 vs S2/S3 could easily shuffle. The paper itself concedes this in Section III.C, noting susceptibility to outliers, so the mismatch between the abstract and the evidence is an internal inconsistency.

That said, the conclusion is likely in the right direction. The conceptual argument — autoscaling only changes scale-out, not node size or the memory-to-CPU ratio — is sensible and matches what the data show: selecting the right static configuration often matters more than adjusting node count at runtime. The discussion in Section V is more measured than the abstract and basically says 'roughly comparable on average, with a theoretical 25% additional saving if you pick the best static config per job.' I buy that.

The citation pattern looks fair. The authors cite their own prior work on cluster configuration, which is relevant, and the literature on static allocation is adequately covered. No signs of overclaiming there.

So: this paper deserves peer review because it ships a reproducible benchmark and a plausible negative result that challenges a common assumption. But the 'significant' wording must be fixed, and the authors should either run repeats for at least a subset of jobs or add bootstrapped error estimates to the normalized values. If the editor wants a decision, I'd say conditional accept with those changes.

Referee Report

4 major / 5 minor

Summary. The paper presents a benchmarking suite (benchspark), a new trace dataset of 234 Spark job executions on Google Cloud Dataproc and Dataproc Serverless, and an experimental comparison of autoscaling (configuration S1) against static resource configurations (S2, S3, and R01-R10). The central empirical claim, stated in the abstract and Section IV.B, is that autoscaling yields 'no significant resource efficiency gain' over static allocations, supported by the observation that S1 achieved the lowest normalized executor-seconds in only 2 of 18 jobs, while static S2 and S3 did so in 9 and 7 jobs. The paper also reports a cost-model sensitivity analysis and discusses conceptual limitations of autoscaling, namely the inelasticity of node size and of the memory-to-CPU ratio.

Significance. If the central claim is supported, the paper is a useful empirical contribution for practitioners choosing between autoscaling and static resource allocation for managed Spark services, and for system designers working on autoscaling algorithms. The public benchmark suite and trace dataset are concrete artifacts that can be reused by other researchers, and the sensitivity analysis in Figure 3 adds insight beyond a single pricing point. However, the paper's headline claim is currently an interpretation of point estimates from single executions, so the significance of the finding is limited until the uncertainty in those measurements is addressed.

major comments (4)

[Abstract; Section IV.B; Table V; Section III.C] The claim that there is 'no significant resource efficiency gain' from autoscaling is not established by the reported data. Every cell in Table V is generated from exactly one execution per job-configuration pair, as Section III.C concedes, and many of the normalized ratios are close to 1.0 (e.g., 1.036, 1.057, 1.023). No confidence intervals, repeated runs, or statistical tests are provided, so the word 'significant' is unsupported. To support the inferential statement, the authors should either add repeated executions with variance estimates or apply a paired significance test on the per-job normalized costs; otherwise the abstract and conclusion should be explicitly weakened to 'no observed advantage in this sample of single executions.'
[Section IV.B; Table V] The win-count argument (S1 best in 2 of 18 jobs, S2 in 9, S3 in 7) depends entirely on point estimates that could change under run-to-run variation. With differences of a few percent in many entries, a rank reversal is plausible if variance is comparable to these effect sizes. The paper should provide stability evidence, for example repeated runs for at least a subset of jobs, or an explicit statement that the counts are point estimates with no stability guarantee. Without this, the central comparison between autoscaling and static allocations is not robust.
[Section IV.C; Table VI] The cost comparison between the autoscaling configuration S1 and the static configurations R08/R09 is confounded by the platform difference: S1 runs on Dataproc Serverless while R08/R09 run on regular Dataproc, and the paper reports a mean performance deviation of about 17.6% with a standard deviation of about 18.5% between the corresponding configurations. The statement that 'one can expect that S1 would produce a cost that is at least not exceedingly far' is informal. The cost comparison should either be adjusted for this platform effect or clearly labeled as confounded, since the conclusion that autoscaling is not better than static allocation depends on this comparison.
[Section V] The conclusion that the limitations of autoscaling are caused by 'the inelasticity of node size as well as the inelasticity of the ratio of memory to CPU cores' is plausible but not directly tested. In the experimental design, S1 varies only scale-out with a fixed node type, and none of the configurations varies node size or memory-to-CPU ratio for the autoscaling service itself. The discussion should be framed as a hypothesis derived from the comparison with static configurations, not as an experimentally established causal mechanism.

minor comments (5)

[Section III.C] The sentence 'we expect the overall evaluation to be accurate as it is mostly based on averages' is misleading in the context of Table V, because the win counts are based on single executions rather than on averages over repeated runs. Please clarify which parts of the evaluation are based on averages.
[Section IV.B; Figure 2] The text says that 'the variation in runtimes is much larger than the variation in expended executor seconds,' but the point is asserted only visually. Reporting the actual ranges or coefficients of variation for the normalized values would make this claim more precise.
[Figure 1] The caption text 'size : = si' appears to contain a formatting artifact; please use a consistent notation such as 'size = si' and 'size = 2·si'.
[Sections I, II] There are minor typos, including 'availablility' and 'viablility'. The paper would benefit from a careful proofreading pass.
[Figure 3] The labels A and B are mentioned in the text but not defined in the caption; please define them in the caption for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the resource-efficiency comparison is a direct empirical measurement against externally supplied pricing, with no fitted parameter or self-citation doing load-bearing work.

full rationale

The paper's central claim—that Dataproc Serverless autoscaling shows no significant resource efficiency gain over static allocations—is an empirical conclusion drawn from direct measurements of runtime and resource usage across 234 job executions (Section III.C) and from the normalized executor-seconds comparison in Table V. There is no fitted model, no prediction derived from the data, and no equation in which an output is defined in terms of the conclusion. The pricing model used in Section IV.C is an externally provided Google Dataproc pricing ratio, not a parameter fitted to the outcome. The self-citations [11], [12], [16], and [17] appear only as background or prior-work context and are not load-bearing for the experimental result. The paper itself flags a genuine limitation in Section III.C: 'each job was executed only once on each cloud resource configuration, which may make this measured test job data somewhat susceptible to individual outliers.' This is a statistical robustness concern about whether 'significant' is supported by single runs, but it is not circularity: the conclusion is not assumed by its inputs, and the measured ranks could change under noise without implying that the result was definitionally forced. The 'conceptual limitations' discussed in Section V (inelastic node size and fixed memory-to-CPU ratio) are interpretive explanations of the observed data, not assumptions smuggled in to produce the result. Therefore, under the specified circularity patterns, no step reduces to its own inputs, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted model is used; all conclusions rest on direct measurements. The listed axioms are the main domain assumptions the evaluation relies on.

assumptions (4)

domain assumption Spark's dynamic allocation on Dataproc Serverless adjusts only the number of executors, while node size and the per-executor memory-to-CPU ratio remain fixed.
Assumed from the service design and used throughout Sections IV and V to interpret why autoscaling cannot adjust node type.
domain assumption Executor-seconds and GiB-memory-seconds are valid proxies for resource efficiency.
Used in Tables IV-VI to compare configurations.
domain assumption The 18 jobs and their two dataset sizes form a representative sample of common Spark workloads.
The evaluation's conclusions about autoscaling are generalized from this workload set (Table I).
domain assumption Memory and CPU costs are compared at a fixed ratio of 6 GiB per vCPU core-second, based on Dataproc Serverless pricing.
Section IV-C cost analysis; sensitivity is explored in Figure 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling." pith.science (2026). https://pith.science/paper/OK4TXZ24

@misc{pith2026250114456,
  author       = {Pith},
  title        = {Pith review of: Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OK4TXZ24}},
  note         = {Machine review of arXiv:2501.14456}
}

read the original abstract

Distributed dataflow systems like Spark and Flink enable data-parallel processing of large datasets on clusters. Yet, selecting appropriate computational resources for dataflow jobs is often challenging. For efficient execution, individual resource allocations, such as memory and CPU cores, must meet the specific resource requirements of the job. An alternative to selecting a static resource allocation for a job execution is autoscaling as implemented for example by Spark. In this paper, we evaluate the resource efficiency of autoscaling batch data processing jobs based on resource demand both conceptually and experimentally by analyzing a new dataset of Spark job executions on Google Dataproc Serverless. In our experimental evaluation, we show that there is no significant resource efficiency gain over static resource allocations. We found that the inherent conceptual limitations of such autoscaling approaches are the inelasticity of node size as well as the inelasticity of the ratio of memory to CPU cores.

Figures

Figures reproduced from arXiv: 2501.14456 by the authors.

**Figure 1.** Time series of Dataproc Serverless executor allocation for executors [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 3.** Comparing cloud configuration selection approaches for varying [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

[1]

Spark: Cluster Computing with Working Sets,

M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: Cluster Computing with Working Sets,” in Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud). USENIX, 2010

work page 2010
[2]

Apache Flink™: Stream and Batch Processing in a Single Engine,

P. Carbone, A. Katsifodimos, S. Ewen, V . Markl, S. Haridi, and K. Tzoumas, “Apache Flink™: Stream and Batch Processing in a Single Engine,” IEEE Data Engineering Bulletin (Data Eng. Bull.) , 2015

work page 2015
[3]

AROMA: Automated Resource Allocation and Configuration of Mapreduce Environment in the Cloud,

P. Lama and X. Zhou, “AROMA: Automated Resource Allocation and Configuration of Mapreduce Environment in the Cloud,” in Proceedings of the 9th International Conference on Autonomic Computing (ICAC) . ACM, 2012

work page 2012
[4]

PerfOrator: Eloquent Performance Models for Resource Optimization,

K. Rajan, D. Kakadia, C. Curino, and S. Krishnan, “PerfOrator: Eloquent Performance Models for Resource Optimization,” in Proceedings of the Seventh ACM Symposium on Cloud Computing (SoCC) . Association for Computing Machinery, 2016

work page 2016
[5]

Bubble-flux: Precise Online QoS Management for Increased Utilization in Warehouse Scale Computers,

H. Yang, A. Breslow, J. Mars, and L. Tang, “Bubble-flux: Precise Online QoS Management for Increased Utilization in Warehouse Scale Computers,” ACM SIGARCH Computer Architecture News , 2013

work page 2013
[6]

A Measurement Study of Server Utilization in Public Clouds,

H. Liu, “A Measurement Study of Server Utilization in Public Clouds,” in IEEE Design Automation Standards Committee (DASC). IEEE, 2011

work page 2011
[7]

Quasar: Resource-efficient and QoS- aware Cluster Management,

C. Delimitrou and C. Kozyrakis, “Quasar: Resource-efficient and QoS- aware Cluster Management,” ACM Special Interest Group on Program- ming Languages (SIGPLAN) Notices , 2014

work page 2014
[8]

Scaling big data mining infrastructure: the twitter experience,

J. Lin and D. Ryaboy, “Scaling big data mining infrastructure: the twitter experience,” ACM Special Interest Group on Knowledge Discovery in Data (SIGKDD) Explorations Newsletter , 2013

work page 2013

Show all 17 references

[9]

Juggler: Autonomous Cost Optimization and Performance Prediction of Big Data Applications,

H. Al-Sayeh, B. Memishi, M. A. Jibril, M. Paradies, and K.-U. Sattler, “Juggler: Autonomous Cost Optimization and Performance Prediction of Big Data Applications,” in Proceedings of the 2022 International Conference on Management of Data (SIGMOD) , 2022

2022
[10]

Blink: Lightweight Sample Runs for Cost Optimization of Big Data Applica- tions,

H. Al-Sayeh, M. A. Jibril, B. Memishi, and K.-U. Sattler, “Blink: Lightweight Sample Runs for Cost Optimization of Big Data Applica- tions,” in New Trends in Database and Information Systems . Springer, 2022

2022
[11]

Get Your Memory Right: The Crispy Resource Allocation Assistant for Large- Scale Data Processing,

J. Will, L. Thamsen, J. Bader, D. Scheinert, and O. Kao, “Get Your Memory Right: The Crispy Resource Allocation Assistant for Large- Scale Data Processing,” in IEEE International Conference on Cloud Engineering (IC2E), 2022

2022
[12]

Selecting Efficient Cluster Resources for Data Analytics: When and How to Allocate for In-Memory Processing?

J. Will, L. Thamsen, D. Scheinert, and O. Kao, “Selecting Efficient Cluster Resources for Data Analytics: When and How to Allocate for In-Memory Processing?” in International Conference on Scientific and Statistical Database Management (SSDBM) . ACM, 2023

2023
[13]

Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics,

S. Venkataraman, Z. Yang, M. Franklin, B. Recht, and I. Stoica, “Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics,” in 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI) . USENIX, 2016

2016
[14]

CherryPick: Adaptively Unearthing the Best Cloud Con- figurations for Big Data Analytics,

O. Alipourfard, H. H. Liu, J. Chen, S. Venkataraman, M. Yu, and M. Zhang, “CherryPick: Adaptively Unearthing the Best Cloud Con- figurations for Big Data Analytics,” in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI) . USENIX, 2017

2017
[15]

Arrow: Low-Level Augmented Bayesian Optimization for Finding the Best Cloud VM,

C.-J. Hsu, V . Nair, V . W. Freeh, and T. Menzies, “Arrow: Low-Level Augmented Bayesian Optimization for Finding the Best Cloud VM,” in 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS). IEEE, 2018

2018
[16]

Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation,

D. Scheinert, H. Zhu, L. Thamsen, M. K. Geldenhuys, J. Will, A. Acker, and O. Kao, “Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation,” in IEEE International Perfor- mance Computing and Communications Conference (IPCCC) . IEEE, 2021

2021
[17]

Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs,

J. Will, J. Bader, and L. Thamsen, “Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs,” in IEEE International Conference on Big Data , 2020

2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Spark: Cluster Computing with Working Sets,

M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: Cluster Computing with Working Sets,” in Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud). USENIX, 2010

work page 2010

[2] [2]

Apache Flink™: Stream and Batch Processing in a Single Engine,

P. Carbone, A. Katsifodimos, S. Ewen, V . Markl, S. Haridi, and K. Tzoumas, “Apache Flink™: Stream and Batch Processing in a Single Engine,” IEEE Data Engineering Bulletin (Data Eng. Bull.) , 2015

work page 2015

[3] [3]

AROMA: Automated Resource Allocation and Configuration of Mapreduce Environment in the Cloud,

P. Lama and X. Zhou, “AROMA: Automated Resource Allocation and Configuration of Mapreduce Environment in the Cloud,” in Proceedings of the 9th International Conference on Autonomic Computing (ICAC) . ACM, 2012

work page 2012

[4] [4]

PerfOrator: Eloquent Performance Models for Resource Optimization,

K. Rajan, D. Kakadia, C. Curino, and S. Krishnan, “PerfOrator: Eloquent Performance Models for Resource Optimization,” in Proceedings of the Seventh ACM Symposium on Cloud Computing (SoCC) . Association for Computing Machinery, 2016

work page 2016

[5] [5]

Bubble-flux: Precise Online QoS Management for Increased Utilization in Warehouse Scale Computers,

H. Yang, A. Breslow, J. Mars, and L. Tang, “Bubble-flux: Precise Online QoS Management for Increased Utilization in Warehouse Scale Computers,” ACM SIGARCH Computer Architecture News , 2013

work page 2013

[6] [6]

A Measurement Study of Server Utilization in Public Clouds,

H. Liu, “A Measurement Study of Server Utilization in Public Clouds,” in IEEE Design Automation Standards Committee (DASC). IEEE, 2011

work page 2011

[7] [7]

Quasar: Resource-efficient and QoS- aware Cluster Management,

C. Delimitrou and C. Kozyrakis, “Quasar: Resource-efficient and QoS- aware Cluster Management,” ACM Special Interest Group on Program- ming Languages (SIGPLAN) Notices , 2014

work page 2014

[8] [8]

Scaling big data mining infrastructure: the twitter experience,

J. Lin and D. Ryaboy, “Scaling big data mining infrastructure: the twitter experience,” ACM Special Interest Group on Knowledge Discovery in Data (SIGKDD) Explorations Newsletter , 2013

work page 2013