REVIEW 4 major objections 5 minor 40 references
Right-sizing Recommendations (RSR): Cloud Workload Conformal Prediction for Virtual Machines in Data Center Operations
T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read The paper claims that bootstrapped conformal prediction converts 5-minute vCPU utilization series into reliable mid- and long-term prediction intervals that can drive VM right-sizing recommendations.
desk verdict An applied benchmark with a real validity problem: the reliability claim for the conformal intervals rests on an untested residual-independence assumption, and the reporting is too thin to carry the conclusions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is bootstrap conformal prediction: after training a point forecaster, one-step-ahead residuals are resampled and added to the model's predictions to synthesize a forecast distribution, from which the alpha/2 and 1-alpha/2 empirical quantiles define the interval. This avoids normality assumptions on residuals and requires only that residuals be uncorrelated. The forecasting core uses a direct-recursive multi-step strategy (DirRec) inside the forecaster, with four backtesting regimes for evaluation.
What would settle it
Compute the Ljung-Box or similar autocorrelation test on the one-step-ahead residuals of a random sample of the long-lived VM series; if significant autocorrelation is found, or if empirical coverage of the reported intervals on a held-out test set falls materially below the nominal 1-alpha level, the central 'reliable prediction intervals' claim is falsified.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that bootstrapped residual conformal prediction turns recursive multi-step forecasts of vCPU utilization into prediction intervals whose upper bound is accurate enough for provisioning decisions, and that XGBoost and CatBoost are the consistently top-ranked regressors for long-lived VM workload profiles across all backtesting strategies. The ranking holds for upper-bound (PI) evaluation; for target-value MAE, linear regression is reported as the top model. The paper stops short of claiming the intervals are optimal, presenting them instead as a practical, modular route to uncertainty-aware right-sizing.
Load-bearing premise
The entire interval-reliability claim rests on the assumption that the one-step-ahead forecast residuals are uncorrelated, yet the vCPU time series are described as non-stationary, seasonal, and bursty, and the paper reports no test of residual autocorrelation.
Editorial extensions
If this is right
- If the intervals are valid, cloud operators can replace conservative historical-max upper bounds with statistically calibrated upper bounds from conformal intervals, lowering over-provisioning.
- XGBoost and CatBoost emerge as the practical choices for large-scale long-horizon vCPU forecasting because they balance accuracy and runtime.
- Refitting strategies, especially fixed-origin and rolling-origin backtesting, improve top-1 model placements, indicating that updating models on recent data sharpens the forecasts.
- The modular pipeline can be applied to other VM metrics or datasets without redesign.
Reading between the lines
- The paper does not test residual autocorrelation, so a natural extension is to diagnose whether the uncorrelated-residual premise actually holds on these workloads; if it fails, coverage guarantees weaken.
- The ranking is based on limited VM candidates and focused on long-lived VMs; the same model hierarchy may not hold for short-lived or highly bursty workloads.
- Because the study is univariate and vCPU-only, the approach could likely be extended to memory and multivariate forecasting, where memory footprint is currently absent from public cloud traces.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a forecasting pipeline for Azure VM vCPU utilization, combining seven ML regressors with bootstrap conformal prediction (via skforecast) to produce prediction intervals, and evaluates four backtesting strategies on long-lived VM candidates from the Azure Public Datasets V1/V2. The central claim is that the resulting intervals are reliable enough to support right-sizing recommendations, and that XGBoost/CatBoost are the top-performing models across backtesting strategies. The experimental section presents rankings as bar-chart frequencies and discusses efficiency trade-offs, while a limitations section acknowledges workload unpredictability and missing service-level metadata.
Significance. If established, the paper would provide a useful practical evaluation of conformal prediction for cloud workload forecasting and a model-ranking benchmark over publicly available Azure traces. Strengths include the use of public datasets, a reproducible GitHub repository, a broad set of regression models, and four backtesting strategies. However, the central reliability claim is not yet supported by the evidence: the bootstrap CP intervals are not validated for the non-stationary, autocorrelated time series studied, and the empirical results lack quantitative forecast-error and coverage numbers, baselines, and a precisely specified VM subset. The paper is therefore of interest to the cloud operations community, but the current draft requires substantial additional analysis before its conclusions can be accepted.
major comments (4)
- [§III-D, §III-C2, §IV-C] The validity of the proposed conformal prediction intervals is not established. The paper states that skforecast's bootstrapping 'requires only that the residuals (errors) be uncorrelated' (§III-D), but the vCPU series are explicitly described as non-stationary, seasonal, and bursty (§IV-C), and the DirRec/recursive forecasting strategy propagates errors across steps (§III-C2). No autocorrelation test (e.g., Ljung-Box) is reported, and no empirical coverage evaluation (e.g., PICP per horizon) is provided. The only PICP-based ranking is mentioned as 'Likely' with no numeric support (§V-A). Without these diagnostics, the bootstrap intervals do not carry the conformal guarantee claimed, and the RSR upper-bound use case (where a miss can cause SLA violations) is not protected. This is a load-bearing issue for the paper's central claim and should be addressed either by switching to a split-co
- [§V, Fig. 9] The empirical evaluation is presented only as absolute-frequency bar charts for Top-1 and Top-3 model counts. No numeric MAE or PICP values, no error bars, and no baseline comparisons (e.g., naive, SARIMA, or prior-work methods) are reported. The statement in §IV-C that 'we need models that are capable of making skillful forecasts compared to naive models and tuned SARIMA models' is never followed by such a comparison. Consequently, the claims of 'promising forecasting results' and the model rankings are not quantitatively substantiated. The paper should include tables with per-model and per-backtesting-strategy error and coverage metrics, and at least one baseline comparison.
- [§V-A, Fig. 9] The VM subset used for the ranking experiments is unspecified. The caption of Fig. 9 refers to 'limited VM candidates due to expensive computation,' but the exact number of VMs, the selection criteria, the distribution across workload classes (Delay-insensitive/Interactive/Unknown), and the train/test split used for the ranking are not given. Without this information, the generalizability of the 'global top-tier models' claim cannot be assessed. Please state the sample size and the sampling procedure explicitly.
- [§IV] There is a discrepancy in the data split. Section IV says 'we allocate the first 75% of each VM workload time series to training and validation/calibration. The remaining 15% ... is reserved for testing,' which leaves 10% of the data unaccounted for. For medium-scale forecasting, the ratio is stated as 85% training and 15% testing, but no equivalent allocation is given for the large-scale case. Since conformal prediction requires a proper calibration set, the missing 10% may be critical. Please clarify the exact split and the role of the calibration data in the interval construction.
minor comments (5)
- [§III-A, Eq. (1)] Equation (1) is a tautological definition rather than a measurement equation. Consider removing or reformulating it to avoid confusion.
- [§II, Fig. 3] The survey counts in Fig. 3 are presented without a methodology or a reference to the underlying survey. Please either cite the survey source or describe how the 63 publications were compiled.
- [§IV-A, Fig. 7] Fig. 7 shows an 'Interactive-V2' trace with 5,422 observations (about 18 days), which appears inconsistent with the stated long-lifetime criterion of '≥29 consecutive days.' Please clarify whether this subplot is an example from a different filtering stage.
- [§V-A] The sentence beginning 'Likely, the results of ranked ML-powered conformal forecasts...' is phrased as a conjecture. If PICP-based rankings were computed, present them as definitive results with values.
- [References] The reference list has inconsistent formatting (e.g., incomplete entries, missing journal names, and inconsistent access-date conventions). A thorough cleanup is needed.
Circularity Check
No significant circularity: intervals and rankings come from held-out residuals and public-data backtests, not from fitted constants or self-citations.
full rationale
Walked the paper's derivation chain. Multi-step forecasts are built with DirRec (Sec. III-C2), and prediction intervals are constructed from bootstrapped residuals via empirical α/2 and 1−α/2 quantiles (Sec. III-D, Eq. 4); these intervals are then evaluated on held-out test portions of public Azure V1/V2 traces (Sec. IV), and model rankings come from backtested MAE/PICP and runtime (Sec. V). No quantity predicted is a fitted constant, no calibration parameter is renamed as a prediction, and no load-bearing claim is justified by a self-citation or by an imported uniqueness theorem. The only notable weakness is the validity premise that bootstrap CP 'requires only that the residuals (errors) be uncorrelated' (Sec. III-D), which may be violated by the non-stationary, seasonal, bursty vCPU series the paper itself describes (Sec. IV-C); this is an assumption/correctness risk, not a reduction of the output to the input. The Limitations section (Sec. V-C) acknowledges inherent unpredictability and missing metadata but discloses no circular step. Thus the result is self-contained against external public data and standard methodology, and the finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Train/test split ratio =
75%/15% (large-scale), 85%/15% (medium-scale)
- Savitzky-Golay filter parameters =
not reported
- Z-score outlier threshold and clipping bounds =
not reported
- Lag count / DirRec window =
not reported
- Bootstrap replications and alpha =
not reported
assumptions (4)
- domain assumption Model residuals from recursive multi-step forecasting are uncorrelated.
- domain assumption Long-lived Azure V1/V2 VM traces are representative of hyperscaler workloads for right-sizing.
- domain assumption Average vCPU utilization is a sufficient univariate target; min/max and memory are ignorable.
- standard math Conformal quantile calibration requires exchangeability of calibration and test residuals.
Cite this review
Pith. "Pith review of Right-sizing Recommendations (RSR): Cloud Workload Conformal Prediction for Virtual Machines in Data Center Operations." pith.science (2026). https://pith.science/paper/PFP73MKV
@misc{pith2026260724773,
author = {Pith},
title = {Pith review of: Right-sizing Recommendations (RSR): Cloud Workload Conformal Prediction for Virtual Machines in Data Center Operations},
year = {2026},
howpublished = {\url{https://pith.science/paper/PFP73MKV}},
note = {Machine review of arXiv:2607.24773}
}
read the original abstract
Managing cloud infrastructure efficiently, especially in environments of large cloud providers or hyperscalers, requires optimizing the use of physical resources to minimize costs and maximize performance. Selecting the right virtual machine (VM) sizes is crucial to achieving cost efficiency in these dynamic environments. However, traditional VM allocation and scheduling approaches often fail to account for the fluctuating and unpredictable nature of VM utilization, leading to inefficiencies such as over- or under-provisioning of resources. High-quality interval prediction helps accurately capture uncertainty in cloud resource demand and supports cloud operators in efficient instance provisioning. As an effective and reliable framework for constructing prediction intervals (PIs), conformal prediction (CP) is used for mid- and long-term forecasting tasks in cloud computing environments. This study proposes a new data-driven PI construction approach using bootstrapping conformal prediction for modern, dynamic, data-driven Right-sizing Recommendations (RSR) to enhance provisioning for diverse application workloads on hyperscalers. By learning workload utilization patterns, identifying correlations across multiple time series, and predicting medium- to long-term utilization trends, this research seeks to improve the efficiency of cloud and data center operations through an AI/ML-based provisioning pipeline. Our study demonstrates that AI-driven models, powered by machine learning regression techniques and evaluated using backtesting, achieve promising forecasting results for cloud resource utilization. Additionally, we rank the selected models to identify top-performing approaches for long-life VM candidates. The proposed framework enhances right-sizing recommendations and supports more cost-effective resource allocation in dynamic cloud environments.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Resource central: Understanding and predicting workloads for improved resource management in large cloud platforms
Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bianchini. Resource central: Understanding and predicting workloads for improved resource management in large cloud platforms. InProceedings of the 26th Symposium on Operating Systems Principles, pages 153–167, 2017
2017
-
[2]
A review on machine learning methods for workload prediction in cloud computing
Mohammad Yekta and Hadi Shahriar Shahhoseini. A review on machine learning methods for workload prediction in cloud computing. In2023 13th International Conference on Computer and Knowledge Engineering (ICCKE), pages 306–311. IEEE, 2023
2023
-
[3]
Resource utilization prediction model for cloud datacentre: Survey.International Journal of Advanced Computer Science & Applications, 16(3), 2025
Doaa Bliedy, Mohamed H Khafagy, and Rasha M Badry. Resource utilization prediction model for cloud datacentre: Survey.International Journal of Advanced Computer Science & Applications, 16(3), 2025
2025
-
[4]
Application-oriented cloud workload prediction: A survey and new perspectives.Tsinghua Science and Technology, 30(1):34–54, 2024
Binbin Feng and Zhijun Ding. Application-oriented cloud workload prediction: A survey and new perspectives.Tsinghua Science and Technology, 30(1):34–54, 2024
2024
-
[5]
Tianshu Huang, Arjun Ramesh, Emily Ruppel, Nuno Pereira, Anthony Rowe, and Carlee Joe-Wong. Interference-aware edge runtime prediction with conformal matrix completion.arXiv preprint arXiv:2503.06428, 2025
arXiv 2025
-
[6]
Forecasting workload in cloud computing: towards uncertainty-aware predictions and transfer learning.Cluster Computing, 28(4):258, 2025
Andrea Rossi, Andrea Visentin, Diego Carraro, Steven Prestwich, and Kenneth N Brown. Forecasting workload in cloud computing: towards uncertainty-aware predictions and transfer learning.Cluster Computing, 28(4):258, 2025
2025
-
[7]
Multivariate resource usage forecasting and temporal accuracy in private cloud systems, 2025
Nils Persson Suorra. Multivariate resource usage forecasting and temporal accuracy in private cloud systems, 2025
2025
-
[8]
OTexts, 2018
Rob J Hyndman and George Athanasopoulos.Forecasting: principles and practice. OTexts, 2018
2018
Show all 40 references
-
[9]
Machine learning ap- proaches for estimation of prediction interval for the model output
Durga L Shrestha and Dimitri P Solomatine. Machine learning ap- proaches for estimation of prediction interval for the model output. Neural networks, 19(2):225–235, 2006
2006
-
[10]
Linearregression: Ordinary least squares linear model
Scikit-learn Developers. Linearregression: Ordinary least squares linear model. https://scikit-learn.org/stable/modules/generated/sklearn.linear model.LinearRegression.html
-
[11]
Svr: Support vector regression
Scikit-learn Developers. Svr: Support vector regression. https:// scikit-learn.org/0.16/modules/generated/sklearn.svm.SVR.html
-
[12]
Mlpregressor: Multi-layer perceptron re- gressor
Scikit-learn Developers. Mlpregressor: Multi-layer perceptron re- gressor. https://scikit-learn.org/stable/modules/generated/sklearn.neural network.MLPRegressor.html
-
[13]
Randomforestregressor: Random forest ensemble method for regression
Scikit-learn Developers. Randomforestregressor: Random forest ensemble method for regression. https://scikit-learn.org/stable/modules/ generated/sklearn.ensemble.RandomForestRegressor.html. GitHub source: https://github.com/scikit-learn/scikit-learn/blob/c5497b7f7/ sklearn/ens...
2024
-
[14]
Xgboost: Scalable and flexible gradient boost- ing
XGBoost Developers. Xgboost: Scalable and flexible gradient boost- ing. https://xgboost.readthedocs.io/en/stable/. GitHub repository: https: //github.com/dmlc/xgboost, accessed: 2024
2024
-
[15]
Catboost: Gradient boosting on decision trees
CatBoost Developers. Catboost: Gradient boosting on decision trees. https://catboost.ai/docs/en/. GitHub repository: https://github.com/ catboost/catboost/issues, accessed: 2024
2024
-
[16]
Lightgbm: A fast, distributed, high-performance gradient boosting framework
LightGBM Developers. Lightgbm: A fast, distributed, high-performance gradient boosting framework. https://lightgbm.readthedocs.io/en/latest/ index.html. GitHub repository: https://github.com/microsoft/LightGBM/ tree/master, accessed: 2024
2024
-
[17]
Machine Learning Mastery, 2017
Jason Brownlee.Introduction to time series forecasting with python: how to prepare data and develop models to predict the future. Machine Learning Mastery, 2017
2017
-
[18]
Backtesting forecasts — skforecast user guide
skforecast Developers. Backtesting forecasts — skforecast user guide. https://skforecast.org/0.16.0/user guides/backtesting. Accessed: Feb. 08, 2024
2024
-
[19]
The three types of backtests.Available at SSRN, 2024
Jacques Joubert, Dragan Sestovic, Illya Barziy, Walter Distaso, and Marcos Lopez de Prado. The three types of backtests.Available at SSRN, 2024
2024
-
[20]
skforecast, 5 2025
Joaquin Amat Rodrigo and Javier Escobar Ortiz. skforecast, 5 2025
2025
-
[21]
On the use of cross-validation for time series predictor evaluation.Information Sciences, 191:192–213, 2012
Christoph Bergmeir and Jos ´e M Ben´ıtez. On the use of cross-validation for time series predictor evaluation.Information Sciences, 191:192–213, 2012
2012
-
[22]
Con- formal prediction sets for ordinal classification.Advances in Neural Information Processing Systems, 36:879–899, 2023
Prasenjit Dey, Srujana Merugu, and Sivaramakrishnan R Kaveri. Con- formal prediction sets for ordinal classification.Advances in Neural Information Processing Systems, 36:879–899, 2023
2023
-
[23]
Integrating physics and data-driven approaches: An explainable and uncertainty- aware hybrid model for wind turbine power prediction.arXiv preprint arXiv:2502.07344, 2025
Alfonso Gij ´on, Simone Eiraudo, Antonio Manjavacas, Daniele Salvatore Schiera, Miguel Molina-Solana, and Juan G ´omez-Romero. Integrating physics and data-driven approaches: An explainable and uncertainty- aware hybrid model for wind turbine power prediction.arXiv preprint ar...
2025 arXiv
-
[24]
Wind speed conformal prediction in wind farm based on algorithmic randomness theory
Guo-Rui Ji, Ze Dong, Dong-Feng Wang, Pu Han, and Da-Ping Xu. Wind speed conformal prediction in wind farm based on algorithmic randomness theory. In2008 International conference on machine learning and cybernetics, volume 1, pages 131–135. IEEE, 2008
2008
-
[25]
Conformal prediction in clinical medical sciences.Journal of Healthcare Informatics Research, 6(3):241– 252, 2022
Janette Vazquez and Julio C Facelli. Conformal prediction in clinical medical sciences.Journal of Healthcare Informatics Research, 6(3):241– 252, 2022
2022
-
[26]
John Wiley & Sons, 2017
William Q Meeker, Gerald J Hahn, and Luis A Escobar.Statistical intervals: a guide for practitioners and researchers. John Wiley & Sons, 2017
2017
-
[27]
A review and comparative analysis of univariate conformal regression methods
Jie Bao, Nicolo Colombo, Valery Manokhin, Suqun Cao, and Rui Luo. A review and comparative analysis of univariate conformal regression methods. InFourteenth Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2025), pages 282–304. PMLR, 2025
2025
-
[28]
Jing Bi, Haitao Yuan, Shuang Li, Kaiyi Zhang, Jia Zhang, and Mengchu Zhou. Arima-based and multiapplication workload prediction with wavelet decomposition and savitzky–golay filter in clouds.IEEE Transactions on Systems, Man, and Cybernetics: Systems, 54(4):2495– 2506, 2024
2024
-
[29]
Accurate workload prediction for edge data centers: Savitzky-golay filter, cnn and bilstm with attention mechanism.Applied Intelligence, 52(11):13027–13042, 2022
Lei Chen, Weiwen Zhang, and Haiming Ye. Accurate workload prediction for edge data centers: Savitzky-golay filter, cnn and bilstm with attention mechanism.Applied Intelligence, 52(11):13027–13042, 2022
2022
-
[30]
Telemetry data gaps in azure public dataset
Eli Cortez. Telemetry data gaps in azure public dataset. https://github. com/Azure/AzurePublicDataset/issues/4, 2019. Accessed: 2025-08-08
2019
-
[31]
Saits: Self-attention-based impu- tation for time series.Expert Systems with Applications, 219:119619, 2023
Wenjie Du, David C ˆot´e, and Yan Liu. Saits: Self-attention-based impu- tation for time series.Expert Systems with Applications, 219:119619, 2023
2023
-
[32]
Diffusion-based time series data imputation for cloud failure prediction at microsoft 365
Fangkai Yang, Wenjie Yin, Lu Wang, Tianci Li, Pu Zhao, Bo Liu, Paul Wang, Bo Qiao, Yudong Liu, M ˚arten Bj ¨orkman, et al. Diffusion-based time series data imputation for cloud failure prediction at microsoft 365. InProceedings of the 31st ACM Joint European Software Engineeri...
-
[33]
Novel attack vector to abuse aws for cryptojacking
Nahfid Nissar, S Arjun, Kvk Siddartha, and Shaik Muzamil Raheman. Novel attack vector to abuse aws for cryptojacking. In2024 3rd Inter- national Conference on Applied Artificial Intelligence and Computing (ICAAIC), pages 1369–1374. IEEE, 2024
2024
-
[34]
A survey of attack instances of cryptojacking targeting cloud infrastructure
Keshani Jayasinghe and Guhanathan Poravi. A survey of attack instances of cryptojacking targeting cloud infrastructure. InProceedings of the 2020 2nd Asia pacific information technology conference, pages 100– 107, 2020
2020
-
[35]
Defense mechanisms against ddos attacks in a cloud computing environment: State-of-the-art and research challenges.IEEE Communications Surveys & Tutorials, 21(4):3769–3795, 2019
Neha Agrawal and Shashikala Tapaswi. Defense mechanisms against ddos attacks in a cloud computing environment: State-of-the-art and research challenges.IEEE Communications Surveys & Tutorials, 21(4):3769–3795, 2019
2019
-
[36]
Ddos attacks in cloud computing: Issues, taxonomy, and future directions.Computer communications, 107:30–48, 2017
Gaurav Somani, Manoj Singh Gaur, Dheeraj Sanghi, Mauro Conti, and Rajkumar Buyya. Ddos attacks in cloud computing: Issues, taxonomy, and future directions.Computer communications, 107:30–48, 2017
2017
-
[37]
A note on the validity of cross-validation for evaluating autoregressive time series prediction.Computational Statistics & Data Analysis, 120:70–83, 2018
Christoph Bergmeir, Rob J Hyndman, and Bonsoo Koo. A note on the validity of cross-validation for evaluating autoregressive time series prediction.Computational Statistics & Data Analysis, 120:70–83, 2018
2018
-
[38]
Towards measuring predictability: To which extent data-driven approaches can extract deterministic relations from data exemplified with time series prediction and classification
Saleh GHOLAM ZADEH, Vaisakh Shaj, Patrick Jahnke, Gerhard Neu- mann, and Tim Breitenbach. Towards measuring predictability: To which extent data-driven approaches can extract deterministic relations from data exemplified with time series prediction and classification. Transact...
2025
-
[39]
A simple and effective predictive resource scaling heuristic for large-scale cloud applications
Quentin Rebjock, Valentin Flunkert, Tim Januschowski, Laurent Callot, and Joel Castellon. A simple and effective predictive resource scaling heuristic for large-scale cloud applications. InAIDB@ VLDB, 2020
2020
-
[40]
Temporal conformal prediction (tcp): A distribution-free statistical and machine learning framework for adaptive risk forecasting.arXiv preprint arXiv:2507.05470, 2025
Agnideep Aich, Ashit Baran Aich, and Dipak C Jain. Temporal conformal prediction (tcp): A distribution-free statistical and machine learning framework for adaptive risk forecasting.arXiv preprint arXiv:2507.05470, 2025
2025
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.