REVIEW 3 major objections 4 minor 40 references
Sustainable Machine Learning Retraining: Optimizing Energy Efficiency Without Compromising Accuracy
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Retraining on recent data or on drift signals cuts model-upkeep energy by up to 40 percent without meaningful accuracy loss.
desk verdict Solid empirical study of retraining energy; the sliding-window claim holds up, but the 40% drift-triggered savings rest on an unreported KS trigger rule that should be fixed before acceptance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the pairing of a data-selection rule with a retraining trigger. The data rule is the sliding window, which bounds the training set to the most recent period and is compared against full-history training. The trigger is an unsupervised Kolmogorov-Smirnov test, a statistical test that compares two distributions, which is applied to training data versus incoming inference data and fires retraining on significant divergence. Three variants differ in how features enter the comparison: all features, PCA-reduced features, and features selected by Gini importance. The KS test is what converts retrain-when-needed into an executable policy, and the variants are what make the energy comparison concrete. Energy is measured per pipeline stage with the CodeCarbon tool using RAPL and NVML.
What would settle it
Re-run the experiments while sweeping the KS test's significance level between 0.01 and 0.10 and count retraining events; if the one-year energy gap between informed and periodic retraining falls below the reported 7 to 40 percent at any setting in that range, the headline savings are trigger-threshold dependent rather than a stable property of drift-based retraining.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is empirical: over real-world failure-prediction datasets, the dominant driver of lifecycle energy is not the model itself but the retraining policy. Sliding-window retraining, which keeps the training set at a fixed recent size, consumes up to 25 percent less training energy than full-history retraining while matching or nearly matching ROC AUC. Drift-based informed retraining, when the detector is neither blind nor hypersensitive, is estimated to reduce one-year training-plus-drift energy by roughly 7 percent on Backblaze, 40 percent on Alibaba, and 10 percent on Google compared with periodic retraining. Inference energy is essentially unaffected by the retraining strategy. No single configuration wins everywhere, so the authors frame the contribution as a quantitative guide for selecting retraining techniques.
Load-bearing premise
The findings for drift-based retraining rest on the trigger rule of the KS detector, yet the paper never states the significance threshold or how per-feature p-values are combined, and a different threshold changes how often the model retrains and therefore how large the reported savings are.
Editorial extensions
If this is right
- A model maintained with a sliding window should consume up to 25 percent less retraining energy than one retrained on full history, with ROC AUC differences of at most a few percentage points on these datasets.
- Deploying a well-chosen unsupervised drift detector instead of a fixed schedule is estimated to cut one-year training-plus-drift energy by about 7 to 40 percent depending on the dataset.
- Detector overhead is small, under 4 percent of combined training and detection energy, so the main cost of a poor detector is unnecessary retraining rather than detection itself.
- Retraining strategy does not materially change inference energy, so energy optimization should focus on training and retraining frequency.
- There is no universally best technique; the recommendation is to pair drift-based triggering with sliding-window data and to validate detector sensitivity per application.
Reading between the lines
- If the same relationship holds beyond random forests and failure prediction, retraining policy, not model architecture, should be the first lever in green-ML audits, and studies reporting only training energy may miss the dominant lifecycle decision.
- The ranking of the three drift detectors is likely tied to the unstated KS significance threshold, so sweeping that threshold would show whether the reported 40 percent saving is robust or an artifact of one trigger sensitivity.
- A testable extension is to run the drift detector on a subsample or at lower frequency than every inference batch, since detector energy is small but nonzero, which could improve net savings or reveal a floor where detection cost outweighs retraining saved.
- The experiments exhibit both failure modes of drift detectors, missing drift and crying wolf, so a practical decision rule is to choose the least sensitive detector that still catches the drifts that actually degrade AUC.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports an empirical study of energy consumption in retraining strategies for failure-prediction ML models on three public datasets (Backblaze, Google, Alibaba). It compares sliding-window versus full-history retraining and periodic versus informed (drift-detection-based) retraining, measuring training, drift-detection, and inference energy with CodeCarbon across multiple random seeds, and evaluating accuracy with ROC AUC. The paper claims that sliding-window retraining reduces energy by up to 25% with negligible accuracy loss, and that informed retraining with a well-chosen drift detector can reduce annual energy by up to 40% based on a one-year extrapolation. It concludes that there is no universally optimal retraining strategy, and that the choice of drift detector is decisive.
Significance. If the findings hold, the paper makes a useful empirical contribution to Green AI: it provides direct measurements of retraining energy in a realistic AIOps domain, quantifies the overhead of unsupervised drift detectors (Table I shows under 4%), and demonstrates that the sustainability of informed retraining depends heavily on detector choice. The sliding-window versus full-history result is the most solid contribution, supported by direct measurements, multiple seeds, and Wilcoxon tests. The drift-based savings claim is more fragile because it depends on an unreported detector trigger specification and on a linear one-year extrapolation. The replication package and reliance on public datasets are strengths; the paper is empirically grounded rather than a derivation.
major comments (3)
- [Section V.C and Table II] The trigger rule for the KS drift detector is not specified. The text describes KS-ALL, KS-PCA, and KS-FI in terms of feature handling, but it never states the significance level alpha, how per-feature p-values are combined into a retraining decision, the sample size used for each test, or the cadence at which drift is evaluated. Since the number of retraining events determines training energy, and Table I shows detector overhead is under 4%, the entire informed-versus-periodic comparison and Table II depend on this hidden parameter. The reported results already show the sensitivity: KS-PCA and KS-FI are described as too sensitive on Google and consume more energy than periodic retraining, while KS-ALL detects no drift on Alibaba and yields accuracy matching the Static baseline. Without the trigger specification, Table II cannot be reproduced from the manuscript alone, and small changes in alpha could move any detector between under- and over-triggering regimes. Please report the exact trigger rule (including p-value aggregation and evaluation cadence) and include a sensitivity analysis over the threshold and aggregation rule.
- [Section VI.A.3 and Table II] The 'up to 40%' energy saving is a one-year extrapolation that assumes constant energy and drift frequency, as stated in the text. The measured periods are short (6 months for Backblaze, 1 month for Alibaba, 2 weeks for Google), and the 40% figure is driven by a single detector-dataset combination (KS-FI on Alibaba). Because retraining energy is roughly proportional to the number of drift-triggered retraining events, the estimate is only as good as the assumption that drift frequency in the measured window continues unchanged. The authors should present this as a bounded estimate, report the number of retraining events per configuration, and either justify the stationarity assumption or provide a range under alternative drift-frequency scenarios. The abstract and conclusion currently state the 40% value without this caveat.
- [Section VI.A.3 and Finding 2] The claim that informed retraining saves up to 40% 'provided a reliable data change detector is in place' is based on selecting, after seeing the results, the best detector for each dataset (KS-FI for Backblaze and Alibaba, KS-ALL for Google). The paper offers no a priori method to choose a detector that is neither too sensitive nor too insensitive; the condition 'reliable detector' is therefore outcome-defined rather than operationally specified. As a result, the headline claim is not yet actionable for practitioners. Please add a selection heuristic or validation procedure (for example, using a held-out period to estimate retraining frequency), or soften the claim to state that savings were observed for at least one detector per dataset and that the choice is currently hindsight-based.
minor comments (4)
- [Section II.B] There is a duplicated word: 'the distribution of the data data learned during the training process' should read 'the distribution of the data learned during the training process.'
- [Table II caption] The caption is incomplete: 'over the period of one .' should be 'over the period of one year.' Also, '1.000' and '1.000.000' should be written as '1,000' and '1,000,000' for consistency.
- [Section VI.A.3] The sentence 'followed by Alibaba (approx. 1 megajoules)' has a singular-plural agreement error; it should be 'approx. 1 megajoule.'
- [Throughout] The paper uses 'accuracy' interchangeably with ROC AUC. Although the authors define this choice, using 'ROC AUC' consistently in the abstract and findings would avoid potential confusion among readers.
Circularity Check
No circular derivation: the paper reports measured energy and accuracy values, and its conditional 'up to 40%' claim is an explicit extrapolation with the detector-choice caveat stated, not a fitted parameter disguised as a prediction.
full rationale
The paper is an empirical measurement study, not a derivation, so the central claims do not reduce to their inputs by construction. The 25% sliding-window saving is read directly from measured training energy in Fig. 2, and the 40% informed-retraining saving is an explicit one-year linear extrapolation of measured energy for a detector that Table II identifies as 'the most sustainable drift detection-based retraining technique for each dataset.' The extrapolation assumption is stated verbatim: 'we assume no change in the energy consumed during the given period and the rest of the year,' and the claim is qualified as conditional on 'a reliable data change detector' being in place and on the detector being 'properly chosen.' The self-citations ([10], [19], [27]) appear as motivation, background, or sources of the retraining and KS techniques; the paper explicitly says it 'validate[s] the claim of the authors [10]' with new experimental evidence, so the cited prior work is not load-bearing for the measured results. No equation defines energy or accuracy in terms of the claimed savings, and no fitted parameter is renamed as a prediction. The main weakness is not circularity: Section V.C never specifies the KS significance level, per-feature p-value aggregation, or evaluation cadence, and the per-dataset best detector is selected post hoc, both of which are reproducibility and external-validity threats. They do not make the measurements circular. The only reason the score is at the low nonzero end is the presence of several minor self-citations; none is load-bearing, and the measured results are independently self-contained.
Assumptions & free parameters
free parameters (3)
- KS test significance level =
not reported (default 0.05 likely)
- Per-feature KS p-value aggregation rule =
not reported
- Sliding window size =
one period (day/week/month depending on dataset)
assumptions (3)
- domain assumption CodeCarbon/RAPL energy measurements reflect the true energy consumption of the retraining and inference workloads
- domain assumption Distributional deviation detected by per-feature KS tests is a valid and sufficient signal that retraining is needed
- domain assumption The measured observation window (6 months, 1 month, or 2 weeks) is representative of the full year for extrapolation
Cite this review
Pith. "Pith review of Sustainable Machine Learning Retraining: Optimizing Energy Efficiency Without Compromising Accuracy." pith.science (2026). https://pith.science/paper/LMMMN2XB
@misc{pith2026250613838,
author = {Pith},
title = {Pith review of: Sustainable Machine Learning Retraining: Optimizing Energy Efficiency Without Compromising Accuracy},
year = {2026},
howpublished = {\url{https://pith.science/paper/LMMMN2XB}},
note = {Machine review of arXiv:2506.13838}
}
read the original abstract
The reliability of machine learning (ML) software systems is heavily influenced by changes in data over time. For that reason, ML systems require regular maintenance, typically based on model retraining. However, retraining requires significant computational demand, which makes it energy-intensive and raises concerns about its environmental impact. To understand which retraining techniques should be considered when designing sustainable ML applications, in this work, we study the energy consumption of common retraining techniques. Since the accuracy of ML systems is also essential, we compare retraining techniques in terms of both energy efficiency and accuracy. We showcase that retraining with only the most recent data, compared to all available data, reduces energy consumption by up to 25\%, being a sustainable alternative to the status quo. Furthermore, our findings show that retraining a model only when there is evidence that updates are necessary, rather than on a fixed schedule, can reduce energy consumption by up to 40\%, provided a reliable data change detector is in place. Our findings pave the way for better recommendations for ML practitioners, guiding them toward more energy-efficient retraining techniques when designing sustainable ML software systems.
Figures
Reference graph
Works this paper leans on
-
[1]
A synthesis of green architectural tactics for ml-enabled systems
Heli Järvenpää, Patricia Lago, Justus Bogner, Grace Lewis, Henry Muccini, and Ipek Ozkaya. A synthesis of green architectural tactics for ml-enabled systems. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Society , ICSE-SEIS’24, page 130–141, New York, NY , USA, 2024. Association for Computing Machinery
work page 2024
-
[2]
Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. The ml test score: A rubric for ml production readiness and technical debt reduction. 2017 IEEE International Conference on Big Data (Big Data), pages 1123–1132, 2017
work page 2017
-
[3]
Ahmed, and Andreas Kassler
Firas Bayram, Bestoun S. Ahmed, and Andreas Kassler. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Based Systems, 245:108632, 2022
2022
-
[4]
A survey on concept drift adaptation
João Gama, Indrundefined Žliobaitundefined, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM Comput. Surv., 2014
work page 2014
-
[5]
Data management challenges in production machine learning
Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinke- vich. Data management challenges in production machine learning. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD ’17, page 1723–1726, 2017
work page 2017
-
[6]
Yingzhe Lyu, Heng Li, Mohammed Sayagh, Zhen Ming (Jack) Jiang, and Ahmed E. Hassan. An empirical study of the impact of data splitting decisions on the performance of aiops solutions. ACM Trans. Softw. Eng. Methodol., 30(4), jul 2021
work page 2021
-
[7]
Towards a consistent interpretation of aiops models
Yingzhe Lyu, Gopi Krishnan Rajbahadur, Dayi Lin, Boyuan Chen, and Zhen Ming (Jack) Jiang. Towards a consistent interpretation of aiops models. ACM Trans. Softw. Eng. Methodol. , 31(1), 2021
work page 2021
-
[8]
Carbon emissions and large neural network training, 04 2021
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluís-Miquel Munguía, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training, 04 2021
work page 2021
Show all 40 references
-
[9]
Sustainable ai: Environmental implications, challenges and opportunities
Carole-Jean Wu et al. Sustainable ai: Environmental implications, challenges and opportunities. In Proceedings of Machine Learning and Systems, volume 4, pages 795–813, 2022
2022
-
[10]
Poenaru-Olaru, J
L. Poenaru-Olaru, J. Sallou, L. Cruz, J. S. Rellermeyer, and A. van Deursen. Retrain ai systems responsibly! use sustainable concept drift adaptation techniques. In 2023 IEEE/ACM 7th International Workshop on Green And Sustainable Software (GREENS) , pages 17–18, 2023
2023
-
[11]
Bagdanov
Tomaso Trinci, Simone Magistri, Roberto Verdecchia, and Andrew D. Bagdanov. How green is continual learning, really? analyzing the energy consumption in continual training of vision foundation models, 2024
2024
-
[12]
Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry pro- cessing: Watts driving the cost of ai deployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , FAccT ’24, page 85–99, New York, NY , USA, 2024. Association for Com...
2024
-
[13]
Aiops: Real-world challenges and research innovations
Yingnong Dang, Qingwei Lin, and Peng Huang. Aiops: Real-world challenges and research innovations. In 2019 IEEE/ACM 41st Inter- national Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 4–5, 2019
2019
-
[14]
Yingzhe Lyu, Heng Li, Zhen Ming Jiang, and Ahmed E. Hassan. Assessing the maturity of model maintenance techniques for aiops solutions. arXiv preprint arXiv:2311.03213 , 2023
2023 arXiv
-
[15]
Hassan, Cheng He, Ruirui Huang, Zhengda Zeng, Mian Wang, and Pinan Chen
Yangguang Li, Zhen Ming (Jack) Jiang, Heng Li, Ahmed E. Hassan, Cheng He, Ruirui Huang, Zhengda Zeng, Mian Wang, and Pinan Chen. Predicting node failures in an ultra-large-scale cloud computing platform: An aiops solution. ACM Trans. Softw. Eng. Methodol. , 2020
2020
-
[16]
Predicting node failure in cloud service systems
Qingwei Lin, Ken Hsieh, Yingnong Dang, Hongyu Zhang, Kaixin Sui, Yong Xu, Jian-Guang Lou, Chenggang Li, Youjiang Wu, Randolph Yao, Murali Chintalapati, and Dongmei Zhang. Predicting node failure in cloud service systems. In ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint...
2018
-
[17]
Data-Centric Green AI: An Exploratory Empirical Study
Roberto Verdecchia, Luís Cruz, June Sallou, Michelle Lin, James Wick- enden, and Estelle Hotellier. Data-Centric Green AI: An Exploratory Empirical Study. In ICT4S 2022 - 8th International Conference on ICT for Sustainability, 2022
2022
-
[18]
How to sustainably monitor ml-enabled systems? accuracy and energy efficiency tradeoffs in concept drift detection
Rafiullah Omar, Justus Bogner, Joran Leest, Vincenzo Stoico, Patricia Lago, and Henry Muccini. How to sustainably monitor ml-enabled systems? accuracy and energy efficiency tradeoffs in concept drift detection. In 2024 International Conference on ICT for Sustainability (ICT4S), 2024
2024
-
[19]
Poenaru-Olaru, L
L. Poenaru-Olaru, L. Cruz, A. van Deursen, and J. S. Rellermeyer. Are concept drift detectors reliable alarming systems? - a comparative study. In 2022 IEEE International Conference on Big Data (Big Data) , pages 3364–3373, 2022
2022
-
[20]
AIOps: Predictive Analytics & Machine Learning in Operations
Adnan Masood and Adnan Hashmi. AIOps: Predictive Analytics & Machine Learning in Operations . 2019
2019
-
[21]
Qian Cheng, Doyen Sahoo, Amrita Saha, Wenjing Yang, Chenghao Liu, Gerald Woo, Manpreet Singh, Silvio Saverese, and Steven C. H. Hoi. Ai for it operations (aiops) on cloud platforms: Reviews, opportunities and challenges. ArXiv, 2023
2023
-
[22]
Proac- tive error prediction to improve storage system reliability
Farzaneh Mahdisoltani, Ioan Stefanovici, and Bianca Schroeder. Proac- tive error prediction to improve storage system reliability. In 2017 USENIX Annual Technical Conference (USENIX ATC 17) , pages 391– 402, 2017
2017
-
[23]
Predicting disk replacement towards reliable data centers
Mirela Madalina Botezatu, Ioana Giurgiu, Jasmina Bogojeska, and Dorothea Wiesmann. Predicting disk replacement towards reliable data centers. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD ’16, page 39–48, 2016
2016
-
[24]
Learning under concept drift: A review
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering , 31:2346–2363, 2019
2019
-
[25]
Learn- ing with drift detection
João Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learn- ing with drift detection. In Ana L. C. Bazzan and Sofiane Labidi, editors, Advances in Artificial Intelligence – SBIA 2004 , 2004
2004
-
[26]
Con- cept drift detection through resampling
Maayan Harel, Koby Crammer, Ran El-Yaniv, and Shie Mannor. Con- cept drift detection through resampling. In Proceedings of the 31st Inter- national Conference on International Conference on Machine Learning - Volume 32, 2014
2014
-
[27]
Rellermeyer, and Arie van Deursen
Lorena Poenaru-Olaru, Luís Cruz, Jan S. Rellermeyer, and Arie van Deursen. Is your anomaly detector ready for change? adapting aiops solutions to the real world. In CAIN24, 2nd International Conference on AI Engineering - Software Engineering for AI , 2024
2024
-
[28]
Robust and rapid adaption for concept drift in software system anomaly detection
Minghua Ma, Shenglin Zhang, Dan Pei, Xin Huang, and Hongwei Dai. Robust and rapid adaption for concept drift in software system anomaly detection. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) , pages 13–24, 10 2018
2018
-
[29]
Improving service availability of cloud systems by predicting disk error
Yong Xu, Kaixin Sui, Randolph Yao, Hongyu Zhang, Qingwei Lin, Yingnong Dang, Peng Li, Keceng Jiang, Wenchi Zhang, Jian-Guang Lou, Murali Chintalapati, and Dongmei Zhang. Improving service availability of cloud systems by predicting disk error. In Proceedings of the 2018 USENIX...
2018
-
[30]
Schwartz, J
R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni. Green ai. In Communications of the ACM , volume 63, pages 54–63, 2020
2020
-
[31]
Energy efficiency of training neural network architectures: An empirical study
Yinlena Xu, Silverio Martínez-Fernández, Matias Martinez, and Xavier Franch. Energy efficiency of training neural network architectures: An empirical study. 01 2023
2023
-
[32]
Backblaze hard drive stats
Backblaze Inc. Backblaze hard drive stats. backblaze b2 cloud storage. https://www.backblaze.com/cloud-storage/resources/ hard-drive-test-data
-
[33]
Hellerstein
Charles Reiss, John Wilkes, and Joseph L. Hellerstein. Google cluster- usage traces: format + schema. Technical report, Google Inc., Novem- ber 2011. Revised 2012.03.20. Posted at http://code.google.com/p/ googleclusterdata/wiki/TraceVersion2
2011
-
[34]
Alibaba Group. 2021. alibaba cluster trace program. https://github.com/ alibaba/clusterdata
2021
-
[35]
Mlaas in the wild: Workload analysis and scheduling in large-scale heterogeneous gpu clusters
Weng Qizhen, Xiao Wencong, Yu Yinghao, Wang Wei, Wang Cheng, He Jian, Li Yong, Zhang Liping, Lin Wei, and Ding Yu. Mlaas in the wild: Workload analysis and scheduling in large-scale heterogeneous gpu clusters. In 9th USENIX Symposium on Networked Systems Design and Implementat...
2022
-
[36]
Learning from failure across multiple clusters: A trace-driven approach to understand- ing, predicting, and mitigating job terminations
Nosayba El-Sayed, Hongyu Zhu, and Bianca Schroeder. Learning from failure across multiple clusters: A trace-driven approach to understand- ing, predicting, and mitigating job terminations. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) , pa...
2017
-
[37]
Lewis, Sebastián Echeverría, Lena Pons, and Jeffrey Chrabaszcz
Grace A. Lewis, Sebastián Echeverría, Lena Pons, and Jeffrey Chrabaszcz. Augur: a step towards realistic drift detection in production ml systems. In Proceedings of the 1st Workshop on Software Engineering for Responsible AI , 2023
2023
-
[38]
Green software engineering done right: a scientific guide to set up energy efficiency experiments
Luís Cruz. Green software engineering done right: a scientific guide to set up energy efficiency experiments. Blog post, 2021
2021
-
[39]
de Araújo, JPW, and Minerva Books
Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, Marion Coutarel, Boris Feld, Jérémy Lecourt, Liam Connell, Amine Saboni, Inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de La- vorei...
2024
-
[40]
Towards aiops enabled services in continuously evolving software-intensive embedded systems
Anas Dakkak, Jan Bosch, and Helena Olsson. Towards aiops enabled services in continuously evolving software-intensive embedded systems. Journal of Software: Evolution and Process , page e2592, 2023
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.