arxiv: 2605.12272 · v1 · submitted 2026-05-12 · 💻 cs.IR · cs.DB

Recognition: no theorem link

BatchBench: Toward a Workload-Aware Benchmark for Autoscaling Policies in Big Data Batch Processing -- A Proposed Framework

Venkata Krishna Prasanth Budigi , Siri Chandana Sirigiri

Authors on Pith no claims yet

Pith reviewed 2026-05-13 03:18 UTC · model grok-4.3

classification 💻 cs.IR cs.DB

keywords autoscalingbenchmarking frameworkbatch processingworkload taxonomycloud computingLLM agentspolicy comparisonbig data

0 comments

The pith

BatchBench proposes an open framework to compare rule-based, learned, and agentic autoscaling policies on shared workloads and metrics.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies that autoscaling evaluations for big data batch processing cannot be compared across studies because each uses different workloads, baselines, and metrics. It proposes BatchBench as a solution consisting of a synthesized six-class workload taxonomy, a validated generator, a multi-axis evaluation harness, and a unified interface for policies. This setup would let rule-based controllers, reinforcement learning methods, and large language model agents be tested under identical conditions. A reader would care because autoscaling directly affects cloud costs and performance, yet progress is slowed by the inability to determine which approaches are actually superior.

Core claim

The paper claims that a standardized benchmarking framework called BatchBench can place rule-based, learned, and agentic autoscaling policies on equal experimental footing. It does so by contributing a workload taxonomy of six batch processing classes drawn from published benchmarks and traces, a parameterized generator validated with Kolmogorov-Smirnov and earth-mover distance methods, a five-axis evaluation harness that incorporates cost, SLA attainment, scaling responsiveness, scaling thrash, and decision interpretability while accounting for LLM inference costs, and a common agent interface for evaluation.

What carries the argument

BatchBench, the proposed open benchmarking framework whose workload taxonomy, validated generator, five-axis evaluation harness, and standardized agent interface carry the argument for comparable policy testing.

If this is right

Rule-based heuristics, reinforcement learning controllers, and LLM agents can be evaluated side by side on identical workloads and cost models.
LLM-based policies can be assessed with explicit inclusion of their inference costs alongside traditional resource costs.
Open research questions about policy behavior across different batch workload classes can be addressed systematically.
Future empirical studies can use the framework to produce results that other researchers can replicate and extend directly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Widespread adoption could reduce the number of one-off experiments by letting researchers start from a common reference point.
The taxonomy might serve as a starting point for creating similar benchmarks in streaming or interactive big data workloads.
Insights gained could guide the creation of hybrid policies that switch among rule-based, learned, and agentic approaches depending on observed workload class.
Community contributions to the open-source implementation could iteratively improve the generator and harness over time.

Load-bearing premise

The six-class workload taxonomy synthesized from existing benchmarks and traces, once validated statistically, will produce workloads representative enough of real production batch processing to support generalizable policy comparisons.

What would settle it

Re-running previously published autoscaling policies inside the BatchBench workloads and finding that their relative performance rankings differ markedly from the rankings reported in the original papers would indicate the workloads fail to support generalizable comparisons.

read the original abstract

Autoscaling has become a baseline expectation for cloud-native big data processing, and the design space has expanded beyond rule-based heuristics to include learned controllers and, most recently, large language model (LLM) agents. Yet despite a growing body of work spanning these paradigms, the community lacks a shared benchmark for comparing them. Existing evaluations rely on synthetic TPC-style queries, vendor blog posts with proprietary baselines, or narrow trace replays. Each new policy reports favorable numbers against a different baseline, on a different workload, with a different cost model, making cross-paper comparison effectively impossible. This is a position paper. We propose BatchBench, an open benchmarking framework designed to place rule-based, learned, and agentic autoscaling policies on equal experimental footing. The contribution is the design of the framework, not empirical results. We contribute: (1) a workload taxonomy of six batch processing classes synthesized from published autoscaling benchmarks and publicly released cluster traces; (2) the design of a parameterized workload generator with a validation methodology based on two-sample Kolmogorov-Smirnov and earth-mover distance; (3) a five-axis evaluation harness specification covering cost, SLA attainment, scaling responsiveness, scaling thrash, and decision interpretability, with first-class accounting for LLM inference cost; and (4) a standardized agent interface that lets LLM-based and reinforcement-learning autoscalers be evaluated alongside rule-based controllers with a single API. We discuss the expected evaluation surface, identify open research questions the framework is designed to answer, and outline a roadmap for the empirical paper that will follow. BatchBench's reference implementation is in active development and will be released as open source.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a position paper sketching a benchmark framework for autoscaling policies, but it offers no code, data, or validation yet.

read the letter

The main thing to know is that BatchBench is still a design document rather than a working benchmark. The authors lay out a workload taxonomy of six batch classes drawn from existing traces, a generator validated by Kolmogorov-Smirnov and earth-mover distance, a five-axis harness that folds in LLM inference cost, and a common API for rule-based, RL, and LLM agents. That combination of elements is not in the prior literature they cite, and the problem they target—impossible cross-paper comparisons—is real and well documented in the cloud autoscaling area.

Referee Report

0 major / 3 minor

Summary. The manuscript is a position paper proposing BatchBench, an open benchmarking framework to enable fair comparisons of rule-based, learned, and agentic autoscaling policies for big data batch processing. It contributes (1) a six-class workload taxonomy synthesized from published benchmarks and public traces, (2) a parameterized workload generator with a two-sample Kolmogorov-Smirnov and earth-mover distance validation procedure, (3) a five-axis evaluation harness covering cost, SLA attainment, scaling responsiveness, scaling thrash, and decision interpretability (with explicit LLM inference cost accounting), and (4) a standardized agent interface allowing unified evaluation of different policy types. The paper explicitly states that its contribution is the framework design rather than empirical results and outlines a roadmap for a follow-on empirical paper.

Significance. If implemented as described, BatchBench would address a clear gap in the autoscaling literature by providing a shared, workload-aware evaluation substrate that could replace the current patchwork of TPC-style queries, proprietary baselines, and narrow trace replays. The explicit support for agentic/LLM policies and first-class treatment of inference cost are timely strengths. The open-source commitment and the use of distribution-distance validation metrics are concrete assets that, once realized, would support reproducible policy comparisons.

minor comments (3)

[Abstract / motivation] The motivation section asserts that cross-paper comparison is 'effectively impossible' due to differing baselines, workloads, and cost models. Citing two or three concrete examples from the autoscaling literature (with specific divergent metrics or traces) would make this claim more persuasive and directly support the need for the proposed harness.
[Workload taxonomy] The workload taxonomy is described as 'synthesized from published autoscaling benchmarks and publicly released cluster traces,' yet the manuscript does not enumerate the source traces or benchmarks used. Listing the primary references (even in a table or appendix) would allow readers to assess the coverage and potential biases of the six-class taxonomy.
[Workload generator and validation] The validation procedure for the workload generator invokes two-sample Kolmogorov-Smirnov and earth-mover distance but does not specify acceptance thresholds, sample sizes, or how ties between synthetic and real distributions are resolved. Adding these operational details would clarify how 'representative' workloads are certified.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive and accurate summary of our position paper on BatchBench. The assessment correctly notes that the contribution is the framework design (workload taxonomy, parameterized generator with KS/EMD validation, five-axis harness including LLM cost, and standardized agent interface) rather than new empirical results, and we appreciate the recognition of its potential to address the reproducibility gap in autoscaling policy evaluation. The recommendation for minor revision is noted; no specific major comments were enumerated in the report.

Circularity Check

0 steps flagged

No significant circularity

full rationale

This is a position paper whose sole contribution is the specification of a benchmarking framework (workload taxonomy synthesized from external published benchmarks and traces, parameterized generator design, KS+EMD validation procedure, five-axis harness, and agent API). No equations, fitted parameters, derivations, or quantitative results are presented that could reduce to their own inputs. The central claims are forward-looking design choices rather than predictions or theorems derived from self-referential steps. No load-bearing self-citations exist in the provided text.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The framework rests on the assumption that a small set of workload classes plus statistical matching to traces is sufficient to represent production batch behavior; no new physical constants or fitted parameters are introduced because this is a design proposal rather than a fitted model.

axioms (2)

domain assumption Real-world batch workloads can be usefully partitioned into six classes synthesized from existing benchmarks and traces.
Stated in the contribution (1) of the abstract; no proof or empirical coverage argument is supplied.
domain assumption Two-sample Kolmogorov-Smirnov and earth-mover distance provide adequate validation that generated workloads match target distributions.
Mentioned in contribution (2); the paper does not demonstrate that these metrics capture the dimensions that matter for autoscaling decisions.

pith-pipeline@v0.9.0 · 5612 in / 1431 out tokens · 65006 ms · 2026-05-13T03:18:24.264225+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

28 extracted references · 28 canonical work pages · 2 internal anchors

[1]

Dynamic Resource Allocation,

Apache Spark Project, “Dynamic Resource Allocation,” Apache Spark Documentation, https://spark.apache.org/docs/latest/job-scheduling. html

work page
[2]

Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics,

M. Armbrust et al., “Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics,” inProc. CIDR, 2021

work page 2021
[3]

Introducing Databricks Optimized Auto-scaling on Apache Spark,

Databricks, “Introducing Databricks Optimized Auto-scaling on Apache Spark,” Databricks Engineering Blog, 2018

work page 2018
[4]

Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling,

J. Will et al., “Experimentally Evaluating the Resource Efficiency of Big Data Autoscaling,” arXiv:2501.14456, 2025

work page arXiv 2025
[5]

Db2une: Tuning Under Pressure via Deep Learning,

X. Liang et al., “Db2une: Tuning Under Pressure via Deep Learning,” inProc. VLDB, 2024

work page 2024
[6]

LOFTune: A Low-Overhead and Flexible Approach for Spark SQL Configuration Tuning,

Y . Zhang et al., “LOFTune: A Low-Overhead and Flexible Approach for Spark SQL Configuration Tuning,”IEEE TKDE, 2025

work page 2025
[7]

EAST: An Interpretable Knob Estimation System for Cloud Database,

R. Zhou et al., “EAST: An Interpretable Knob Estimation System for Cloud Database,” inProc. ICDE, 2025

work page 2025
[8]

The Database Gym,

X. Trummer, “The Database Gym,” inProc. SIGMOD, 2025

work page 2025
[9]

AgentTune: An Agent-Based Large Language Model Framework for Database Knob Tuning,

W. Wang et al., “AgentTune: An Agent-Based Large Language Model Framework for Database Knob Tuning,” inProc. SIGMOD, 2025

work page 2025
[10]

Rabbit: Retrieval-Augmented Generation Enables Better Automatic Database Knob Tuning,

H. Sun et al., “Rabbit: Retrieval-Augmented Generation Enables Better Automatic Database Knob Tuning,” inProc. ICDE, 2025

work page 2025
[11]

D-Bot: An LLM-Powered DBA Copilot,

X. Zhou et al., “D-Bot: An LLM-Powered DBA Copilot,” inProc. SIGMOD-Companion, 2025

work page 2025
[12]

GaussMaster: An LLM-based Database Copilot Sys- tem,

Huawei Cloud, “GaussMaster: An LLM-based Database Copilot Sys- tem,” arXiv preprint, 2025

work page 2025
[13]

NeurDB: An AI-powered Autonomous Data System,

J. Tan et al., “NeurDB: An AI-powered Autonomous Data System,” arXiv:2408, 2024

work page 2024
[14]

Why the Apache Spark Default Autoscaler Fails Your Lakehouse,

Onehouse, “Why the Apache Spark Default Autoscaler Fails Your Lakehouse,” Onehouse Engineering Blog, Sep. 2025

work page 2025
[15]

Improve reliability and reduce costs of your Apache Spark workloads with vertical autoscaling on Amazon EMR on EKS,

Amazon Web Services, “Improve reliability and reduce costs of your Apache Spark workloads with vertical autoscaling on Amazon EMR on EKS,” AWS Big Data Blog, 2023

work page 2023
[16]

Hyper: Hybrid Physical Design Advisor with Multi-agent Reinforcement Learning,

J. Doe et al., “Hyper: Hybrid Physical Design Advisor with Multi-agent Reinforcement Learning,” inProc. ICDE, 2025

work page 2025
[17]

Proximal Policy Optimization Algorithms

J. Schulman et al., “Proximal Policy Optimization Algorithms,” arXiv:1707.06347, 2017

work page internal anchor Pith review Pith/arXiv arXiv 2017
[18]

Human-level control through deep reinforcement learn- ing,

V . Mnih et al., “Human-level control through deep reinforcement learn- ing,”Nature, vol. 518, 2015

work page 2015
[19]

Language Models are Few-Shot Learners,

T. Brown et al., “Language Models are Few-Shot Learners,” inProc. NeurIPS, 2020

work page 2020
[20]

Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks,

P. Lewis et al., “Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks,” inProc. NeurIPS, 2020

work page 2020
[21]

Self-Consistency Improves Chain of Thought Reason- ing,

X. Wang et al., “Self-Consistency Improves Chain of Thought Reason- ing,” inProc. ICLR, 2023

work page 2023
[22]

Dropout as a Bayesian Approximation,

Y . Gal and Z. Ghahramani, “Dropout as a Bayesian Approximation,” in Proc. ICML, 2016

work page 2016
[23]

Better Bootstrap Confidence Intervals,

B. Efron, “Better Bootstrap Confidence Intervals,”J. Amer. Statist. Assoc., 1987

work page 1987
[24]

A Simple Sequentially Rejective Multiple Test Procedure,

S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scand. J. Statist., 1979

work page 1979
[25]

Alibaba Cluster Trace Program,

Alibaba Group, “Alibaba Cluster Trace Program,” https://github.com/ alibaba/clusterdata, 2018–2023

work page 2018
[26]

Google Cluster-Usage Traces v3,

J. Wilkes, “Google Cluster-Usage Traces v3,” Technical Report, Google Inc., 2020

work page 2020
[27]

The Case for Evaluating MapReduce Performance Using Workload Suites (SWIM),

Y . Chen et al., “The Case for Evaluating MapReduce Performance Using Workload Suites (SWIM),” inProc. MASCOTS, 2011

work page 2011
[28]

Beyond Similarity Search: A Unified Data Layer for Production RAG Systems

V . K. P. Budigi and S. C. Sirigiri, “Beyond Similarity Search: A Unified Data Layer for Production RAG Systems,” arXiv:2605.03275, 2026

work page internal anchor Pith review Pith/arXiv arXiv 2026