pith. machine review for the scientific record. sign in

arxiv: 2604.18043 · v1 · submitted 2026-04-20 · 💻 cs.DC

Recognition: unknown

Optimizing Memory Allocation in Distributed Clusters with Predictive Modeling

Jonathan Bader , Edgar Blumenthal , Marten Eckardt , Justus Krebs , Joel Witzke , Xemena Wysokinska , Haci Ismail Aslan , Odej Kao
Authors on Pith no claims yet

Pith reviewed 2026-05-10 04:12 UTC · model grok-4.3

classification 💻 cs.DC
keywords memory allocationquantile regressiondistributed systemspredictive modelingresource optimizationbuild jobsgradient boostingsafety factor
0
0 comments X

The pith

An ensemble of gradient boosting models predicts high memory quantiles to cut under-allocation failures and resource waste in distributed clusters.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a regression approach that trains LightGBM and XGBoost models to forecast high conditional quantiles of memory demand for jobs running on distributed systems. It adds a multiplicative safety factor to guard against the higher cost of under-allocation, which can cause job failures, while still limiting wasteful over-allocation. Tested on a real collection of build jobs from SAP, the method lowered the share of under-allocated jobs from 4.17 percent to 2.89 percent and dropped average overallocation from 148 percent to 44.51 percent. A reader would care because these gains improve cluster reliability and lower memory expenses without altering the underlying scheduler or hardware. The work also maps the trade-off surface between the two error types.

Core claim

We propose a regression method based on a LightGBM and XGBoost ensemble trained to predict high conditional quantiles. To further account for the high cost of underallocations we add a multiplicative safety factor. With our method we are able to reduce the number of under-allocated jobs from 4.17% to 2.89% and average overallocation from 148% to 44.51% on a real-world dataset of build jobs provided by SAP. We further explore the pareto frontier between optimization for underallocation and for overallocation.

What carries the argument

LightGBM and XGBoost ensemble trained to output high conditional quantiles of memory usage, combined with a multiplicative safety factor that raises the final allocation target.

If this is right

  • Fewer job failures from memory shortages during production runs on the same cluster.
  • Measurable reduction in total memory footprint and associated operating costs.
  • A tunable Pareto curve that lets operators choose their preferred balance between under- and overallocation.
  • Direct applicability to other resource types such as CPU or disk once similar quantile models are trained.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same quantile-plus-safety approach could be embedded inside dynamic schedulers that re-predict allocations mid-job.
  • If memory traces from many different workloads are pooled, the model might generalize across organizations without per-site retraining.
  • Periodic retraining on recent jobs would be needed to keep the assumption of stationary demand distributions from drifting.

Load-bearing premise

The distribution of memory demand seen in past training jobs will stay representative of future jobs so the learned quantiles continue to bound actual usage.

What would settle it

Apply the trained model to a fresh collection of jobs whose memory usage patterns differ markedly from the original training set and check whether the under-allocation rate rises above 2.89 percent.

Figures

Figures reproduced from arXiv: 2604.18043 by Edgar Blumenthal, Haci Ismail Aslan, Joel Witzke, Jonathan Bader, Justus Krebs, Marten Eckardt, Odej Kao, Xemena Wysokinska.

Figure 1
Figure 1. Figure 1: Pareto frontier for our LightGBM+XGBoost ensemble model with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Job distribution by allocation quality for our method vs. the baseline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
read the original abstract

In modern distributed systems, efficient resource allocation is a vital aspect to maintain scalability, reduce operational costs, and ensure fast execution even across heterogeneous workloads. Predictive models for resource usage are essential tools for optimizing allocation and preventing system bottlenecks. Predictive memory allocation has asymmetric costs as a key challenge: underallocation causes failures while overallocation wastes memory. We propose a regression method based on a LightGBM and XGBoost ensemble trained to predict high conditional quantiles. To further account for the high cost of underallocations we add a multiplicative safety factor. With our method we are able to reduce the number of under-allocated jobs from 4.17% to 2.89% and average overallocation from 148% to 44.51% on a real-world dataset of build jobs provided by SAP. We further explore the pareto frontier between optimization for underallocation and for overallocation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper proposes a regression method using an ensemble of LightGBM and XGBoost models trained to predict high conditional quantiles of memory demand, augmented by a multiplicative safety factor to mitigate the asymmetric costs of underallocation versus overallocation. On a real-world dataset of build jobs from SAP, the approach reduces the fraction of under-allocated jobs from 4.17% to 2.89% and average overallocation from 148% to 44.51%, while also exploring the Pareto frontier between the two objectives.

Significance. If the empirical gains prove robust under temporal validation and the learned quantiles continue to bound future workloads, the work offers a practical, deployable technique for memory allocation in heterogeneous distributed clusters. The use of production SAP build-job data is a strength, as are the concrete before-and-after metrics and the explicit treatment of asymmetric failure costs.

major comments (3)
  1. [Abstract] The central performance claims (under-allocation 4.17%→2.89%, overallocation 148%→44.51%) are reported in the abstract without any description of the train/test split (chronological vs. random/k-fold), cross-validation procedure, chosen quantile levels, feature set, or statistical significance tests. This information is load-bearing for the generalization claim, because the weakest assumption is that historical demand distributions remain representative of future jobs.
  2. [Method] The safety factor is introduced as a free multiplicative parameter to control underallocation risk, yet no value, selection procedure, or sensitivity analysis is supplied. Because the reported deltas depend on this choice, its impact must be quantified (e.g., via ablation or Pareto curves) to substantiate the headline improvements.
  3. [Results] The Pareto-frontier exploration between underallocation and overallocation objectives is mentioned but lacks concrete metrics, operating points, or figures that would allow readers to judge the practical trade-off surface achieved by the ensemble-plus-safety-factor method.
minor comments (2)
  1. [Abstract] The abstract states that the models predict “high conditional quantiles” but does not specify the exact quantile(s) (e.g., 0.95 or 0.99) or the loss function used; this notation should be clarified in the methods section.
  2. [Method] No mention is made of how the LightGBM and XGBoost models are combined (stacking, averaging, or selection), which affects reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback, which helps clarify the reproducibility and robustness aspects of our empirical claims. We address each major comment in turn and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] The central performance claims (under-allocation 4.17%→2.89%, overallocation 148%→44.51%) are reported in the abstract without any description of the train/test split (chronological vs. random/k-fold), cross-validation procedure, chosen quantile levels, feature set, or statistical significance tests. This information is load-bearing for the generalization claim, because the weakest assumption is that historical demand distributions remain representative of future jobs.

    Authors: We agree that the abstract should be self-contained on these points to support the generalization argument. The full manuscript already describes a chronological train/test split (earlier jobs for training, later jobs for testing) to respect temporal ordering, the quantile levels used for the ensemble, the feature set of job metadata and historical usage, and the direct evaluation procedure without formal statistical tests. In revision we will condense these details into the abstract while preserving length constraints. revision: yes

  2. Referee: [Method] The safety factor is introduced as a free multiplicative parameter to control underallocation risk, yet no value, selection procedure, or sensitivity analysis is supplied. Because the reported deltas depend on this choice, its impact must be quantified (e.g., via ablation or Pareto curves) to substantiate the headline improvements.

    Authors: The referee correctly notes that the specific safety-factor value and its selection were not stated. We will revise the method section to report the value employed, the procedure used to choose it (validation-set minimization of a combined under- and over-allocation cost), and an ablation showing metric sensitivity across a range of factors. Updated Pareto curves will be included to quantify the impact. revision: yes

  3. Referee: [Results] The Pareto-frontier exploration between underallocation and overallocation objectives is mentioned but lacks concrete metrics, operating points, or figures that would allow readers to judge the practical trade-off surface achieved by the ensemble-plus-safety-factor method.

    Authors: We acknowledge that the current text only mentions the frontier without sufficient quantitative support. In the revised results section we will add a table of concrete operating points (under-allocation rate versus average overallocation for multiple quantile-safety combinations) and a figure visualizing the achieved trade-off surface. revision: yes

Circularity Check

0 steps flagged

No significant circularity; results are empirical evaluation on held-out data

full rationale

The paper trains an ensemble of LightGBM and XGBoost models to predict high conditional quantiles of memory demand, augments them with a multiplicative safety factor, and reports empirical reductions (under-allocation 4.17%→2.89%, overallocation 148%→44.51%) when the resulting allocator is applied to a real-world SAP build-job dataset. These performance figures are obtained by running the fitted model on separate test data rather than by algebraic identity or by renaming training statistics as predictions. No equations, self-citations, or uniqueness theorems are invoked that would collapse the claimed improvements back into the training inputs by construction. The weakest assumption (stationarity of the demand distribution) is stated openly as a limitation rather than smuggled in. This is a standard supervised-learning evaluation pipeline whose central claims remain falsifiable against external data.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The claim rests on standard supervised-learning assumptions plus one tunable safety multiplier whose value is not reported in the abstract.

free parameters (1)
  • safety factor
    Multiplicative constant applied to the quantile prediction to guard against underallocation; its specific value is not given.
axioms (1)
  • domain assumption Future job memory demands are drawn from the same conditional distribution as the training data
    Required for any out-of-sample quantile prediction to remain valid.

pith-pipeline@v0.9.0 · 5474 in / 1152 out tokens · 56968 ms · 2026-05-10T04:12:53.366798+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 1 canonical work pages

  1. [1]

    Reconsidering cus- tom memory allocation,

    E. D. Berger, B. G. Zorn, and K. S. McKinley, “Reconsidering cus- tom memory allocation,” inProceedings of the 17th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2002, pp. 1–12

    2002

  2. [2]

    More than bin packing: Dynamic resource allocation strategies in cloud data centers,

    A. Wolke, B. Tsend-Ayush, C. Pfeiffer, and M. Bichler, “More than bin packing: Dynamic resource allocation strategies in cloud data centers,” Information Systems, vol. 52, pp. 83–95, 2015

    2015

  3. [3]

    Predictive performance modeling for distributed batch processing using black box monitoring and machine learning,

    C. Witt, M. Bux, W. Gusew, and U. Leser, “Predictive performance modeling for distributed batch processing using black box monitoring and machine learning,”Information Systems, vol. 82, pp. 33–52, 2019

    2019

  4. [4]

    Workload prediction for cloud cluster using a recurrent neural network,

    W. Zhang, B. Li, D. Zhao, F. Gong, and Q. Lu, “Workload prediction for cloud cluster using a recurrent neural network,” in2016 International Conference on Identification, Information and Knowledge in the Internet of Things (IIKI), 2016, pp. 104–109

    2016

  5. [5]

    Improving resource utilization in data centers using an lstm-based prediction model,

    K. Thonglek, K. Ichikawa, K. Takahashi, H. Iida, and C. Nakasan, “Improving resource utilization in data centers using an lstm-based prediction model,” in2019 IEEE International Conference on Cluster Computing (CLUSTER), 2019, pp. 1–8

    2019

  6. [6]

    Sizey: Memory-efficient execution of scientific workflow tasks,

    J. Bader, F. Skalski, F. Lehmann, D. Scheinert, J. Will, L. Thamsen, and O. Kao, “Sizey: Memory-efficient execution of scientific workflow tasks,” in2024 IEEE International Conference on Cluster Computing (CLUSTER), 2024, pp. 370–381

    2024

  7. [7]

    Quantile regression forests,

    N. Meinshausen, “Quantile regression forests,”Journal of Machine Learning Resesearch, vol. 7, pp. 983–999, 2006

    2006

  8. [8]

    XGBoost: A scalable tree boosting system,

    T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” inProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 785–794

    2016

  9. [9]

    LightGBM: A highly efficient gradient boosting decision tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T. Liu, “LightGBM: A highly efficient gradient boosting decision tree,” inAdvances in Neural Information Processing Systems (NeurIPS), 2017, pp. 3146–3154

    2017

  10. [10]

    Forecasting emergency room patient volumes using extreme gradient boosting with temporal and seasonal feature engineering: A comparative study across hospitals,

    K. A. Huang, W. M. Hardin, N. S. Prakash, and W. Hardin, “Forecasting emergency room patient volumes using extreme gradient boosting with temporal and seasonal feature engineering: A comparative study across hospitals,”Cureus, vol. 17, no. 6, 2025

    2025

  11. [11]

    Optuna: A next- generation hyperparameter optimization framework,

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next- generation hyperparameter optimization framework,” inProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 2623–2631

    2019

  12. [12]

    Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance,

    S. Watanabe, “Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance,”

  13. [13]

    arXiv preprint arXiv:2304.11127 , year=

    [Online]. Available: https://arxiv.org/abs/2304.11127

    work page arXiv

  14. [14]

    Practical bayesian optimiza- tion of machine learning algorithms,

    J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimiza- tion of machine learning algorithms,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 25, 2012

    2012

  15. [15]

    Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement,

    G. Forman and M. Scholz, “Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement,”SIGKDD Explorations Newsletter, vol. 12, no. 1, pp. 49–57, Nov. 2010

    2010

  16. [16]

    Regression quantiles,

    R. Koenker and G. Bassett, “Regression quantiles,”Econometrica, vol. 46, no. 1, pp. 33–50, 1978

    1978

  17. [17]

    Ensemble selection from libraries of models,

    R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes, “Ensemble selection from libraries of models,” inProceedings of the Twenty-First International Conference on Machine Learning (ICML), 2004, p. 18

    2004

  18. [18]

    Quantile aggregation of density forecasts,

    F. Busetti, “Quantile aggregation of density forecasts,”Oxford Bulletin of Economics and Statistics, vol. 79, no. 4, pp. 495–512, 2017

    2017

  19. [19]

    Why do tree-based models still outperform deep learning on typical tabular data?

    L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35. Curran Associates, Inc., 2022, pp. 507–520

    2022

  20. [20]

    Tabular data: Deep learning is not all you need,

    R. Shwartz-Ziv and A. Armon, “Tabular data: Deep learning is not all you need,”Information Fusion, vol. 81, pp. 84–90, 2022

    2022