REVIEW 4 major objections 4 minor 2 cited by
Carbon- and Precedence-Aware Scheduling for Data Processing Clusters
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A scheduler that weighs task precedence against grid carbon can cut data-processing emissions by up to 39.7 percent, while keeping end-to-end completion time nearly unchanged.
desk verdict PCAPS is a real systems contribution, but the theory is bookkeeping and the paper's central assumption—that softmax probability tracks task criticality—is asserted, not validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the relative-importance ratio $r_{v,t} := p_{v,t}/\max_{u\in A_t} p_{u,t}$, computed from the probability distribution over ready tasks generated by a machine-learned scheduler, together with the exponential threshold function $\Psi_\gamma(r)$ that turns this ratio into a run-or-defer decision. When the current carbon intensity $c(t)$ exceeds $\Psi_\gamma(r_{v,t})$ and at least one machine is busy, the sampled task is idled rather than executed; otherwise it runs. Because $\Psi_\gamma(1)=U$, the highest-importance task is always allowed to run regardless of carbon, which is the mechanism that protects the DAG's critical path. The proofs also lean on Graham's list-scheduling bound, which turns deferrals into a bounded makespan increase and yields the carbon stretch factor. CAP, by contrast, uses a k-search-inspired threshold set to set a cluster-wide resource quota, so it captures the carbon-shifting half of the idea but not the per-task precedence half.
What would settle it
Run PCAPS on a test batch where the scheduler's highest-probability tasks are known not to lie on the critical path, or replace its importance scores with random permutations while keeping the same deferral thresholds; if the carbon-versus-ECT trade-off does not degrade, then relative importance is not what is producing the savings.
Extended reading notes
Core claim
The central claim is that fine-grained, precedence-aware deferral is what makes carbon-aware scheduling worthwhile for data processing. PCAPS assumes that the softmax probabilities produced by a probabilistic scheduler such as Decima contain implicit information about which tasks are bottlenecks: the ratio $r_{v,t} = p_{v,t}/\max_{u\in A_t} p_{u,t}$ measures how much the scheduler wants task $v$ to run now. An exponential threshold function $\Psi_\gamma(r)$, parameterized by a carbon-awareness knob $\gamma$ and bounded by the known minimum and maximum carbon intensities, determines whether $v$ is dispatched to a machine or deferred. Critical tasks (those with $r$ near 1) are always dispatched; tasks with low relative importance are delayed when carbon intensity is high, and the freed capacity is used later. The paper proves that this filter gives PCAPS a carbon stretch factor of $1 + \frac{D(\gamma,c)K}{2 - 1/K}$, characterizes its carbon savings, and demonstrates experimentally that the idea beats both a carbon-agnostic ML scheduler and a uniform resource-quota wrapper (CAP) on the carbon-versus-end-to-end-time frontier.
Load-bearing premise
PCAPS's benefit rests on treating the probabilities emitted by the learned scheduler as a reliable ranking of how costly it is to delay each task, and the paper does not directly verify that high-probability tasks are the true bottlenecks.
Editorial extensions
If this is right
- A moderately carbon-aware PCAPS reduces carbon emissions by up to 39.7% in the simulator and 32.9% in the prototype, with average end-to-end completion time increasing by only 4.5% in the simulator and 1.3% in the prototype.
- The same setting raises average job completion time by 38.1% in the prototype and about 119% relative to Decima in the simulator, so the carbon savings come from queue build-up during high-carbon periods rather than from faster execution.
- CAP, which only scales cluster resources without looking at task dependencies, achieves smaller carbon savings (up to 25.1%) and pays more end-to-end time for the same carbon reduction, confirming that per-task precedence information is the differentiator.
- The trade-off is tunable: $\gamma=0$ reproduces the carbon-agnostic scheduler, $\gamma=1$ is maximally carbon-aware, and the achievable carbon savings grow with the variability of the grid's carbon intensity.
- On grids with stable, mostly coal-based intensity (ZA in the paper's data), the schedulers defer little and save little; on renewable-dominated grids with strong diurnal swings (CAISO), they wait for solar hours and save more.
Reading between the lines
- If the softmax probability is a faithful criticality proxy, then PCAPS is a drop-in carbon filter for any probabilistic DAG scheduler, not just Decima; a natural stress test is to replace the learned probabilities with critical-path slacks and compare the frontier.
- The large gap between average JCT and ECT suggests PCAPS is effectively trading fairness across jobs for aggregate throughput; in settings with per-job service-level objectives, the deferral policy would need a deadline or a per-job slack guardrail.
- The prototype's 32.9% carbon cut at 1.3% ECT cost implies that under internal carbon pricing, such a filter could pay for itself largely as an accounting change, but the realized cost saving depends on how tightly the carbon price tracks grid intensity.
- A direct ablation—running PCAPS with random importance scores while keeping the same threshold function—would isolate how much of the benefit is due to precedence information rather than to the carbon-triggered idling alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PCAPS, a carbon-aware scheduler for precedence-constrained data processing workloads, and CAP, a simpler resource-provisioning wrapper. PCAPS augments a probabilistic ML scheduler (e.g., Decima) with a threshold filter: using the ratio r_{v,t} = p_{v,t}/max_u p_{u,t} as a proxy for task importance, it runs high-importance tasks even during high-carbon periods and defers low-importance ones. CAP instead varies the cluster resource quota according to carbon intensity while leaving the underlying scheduler unchanged. The authors give analytical results on carbon stretch factor and carbon savings (Theorems 4.3–4.6), and evaluate both schedulers in a Spark-on-Kubernetes prototype on 100 executors and in a Spark simulator across six carbon-intensity traces and TPC-H/Alibaba workloads. The headline results are a 32.9% average carbon reduction in the prototype with near-neutral ECT relative to the default, and up to 39.7% carbon reduction in the simulator with modest ECT increases.
Significance. If the core mechanism is valid, the paper offers a practical, configurable way to reduce data-processing carbon footprint without severely degrading cluster throughput, and it provides one of the few evaluations of carbon-aware scheduling on a real 100-node Spark cluster. The prototype and the six-grid simulator study are substantial, the code is made available, and Appendix A.1.2 honestly documents the simulator/prototype baseline differences. The main limitation is that the distinguishing mechanism—interpreting Decima's softmax probability ratio as precedence-driven criticality—is an unverified proxy, and several analytical results are definitional rather than substantive. The contribution is therefore promising but not yet fully supported.
major comments (4)
- [§4.1, Definition 4.2 and Algorithm 1] The paper's distinguishing mechanism treats the softmax probability ratio r_{v,t} = p_{v,t} / max_{u in A_t} p_{u,t} as 'precedence-driven importance' and uses it to decide which tasks run during high-carbon periods. This identification is an assumption, not a derived property of Decima or of any probabilistic scheduler. The only direct evidence, Fig. 13, compares PCAPS to CAP-Decima on one grid (DE), one batch size (50 TPC-H jobs), and without confidence intervals, and the two systems differ in more than the r computation (fine-grained per-task deferral vs. cluster-level quotas). If r does not track criticality, PCAPS may defer exactly the tasks that block downstream DAGs, and the claimed small ECT increase will not generalize. I recommend a direct validation of the proxy: for example, compare deferral decisions against critical-path or descendant-count measures of task importance, and run an ablation in which r is replaced by random or anti-correlated priorities across several grids and batch sizes.
- [§4.1, Proof of Theorem 4.3 (Appendix B.1.1)] The proof bounds the total number of deferrals by E[D] <= n-1, arguing that at least one machine is active at all times. This does not bound D_v: a task can be sampled and deferred many times, especially because Algorithm 1 treats carbon-intensity changes as scheduling events, so the same ready task can be deferred repeatedly during a long high-carbon interval. Moreover, D(γ,c) is defined post hoc from E[D] and the longest task runtimes, so the 'carbon stretch factor' formula is an identity that holds for any scheduler and any γ; it carries no predictive content unless D(γ,c) is bounded a priori in terms of γ, K, L, U, and the DAG. Please either prove such a bound or recast Theorem 4.3 as a definition/measurement rather than a theorem.
- [§4.2, Proof of Theorem 4.5 (Appendix B.2.1)] The chain CAPK(J | M) <= AG_M(J) assumes that having additional machines and time-varying quotas cannot hurt. This is not generally true for list scheduling: Graham's anomalies show that adding processors can increase makespan for precedence-constrained jobs, and dynamic quota changes can fragment the schedule. Since Theorem 4.5 is presented as an analytical bound on CAP's carbon stretch factor, the monotonicity step needs a proof or a counterexample-free justification; otherwise the theorem should be stated under explicit assumptions on the scheduler and quota dynamics.
- [§4.1–4.2, Theorems 4.4 and 4.6 (Appendices B.1.2, B.2.2)] The carbon-savings results are exact algebraic rearrangements of Definition 3.2 after defining W, s_-, s_+, and c(T,T') in terms of the schedules being compared. They therefore cannot fail and do not, by themselves, establish that any particular scheduling policy saves carbon; the sign of the savings depends entirely on whether the weighted intensities satisfy s_- > s_+ + c(T,T'). I recommend presenting these as accounting identities and moving the burden of the trade-off claim to the empirical evaluation and to a priori bounds on the deferred work W.
minor comments (4)
- [Abstract and §6.3, Table 2] The abstract's 'up to 32.9%' is an average over traces and trials in Table 2, not a maximum; also the text in §6.3 reports 32.8% for PCAPS while Table 2 reports 32.9% (and 24.6% vs. 24.7% for CAP). Please align the wording and the numbers.
- [§6.5] The takeaway 'reduces carbon emissions by up to 39.7%' is again an average over the six carbon traces in Table 3, not a per-trace maximum; please state it as an average or report the actual maximum separately.
- [Definition 3.2 and Appendix B.1.2] The carbon-savings definition is written with integrals, while the proofs use discrete sums; state explicitly that the integrals are over the continuous time model and the sums are the discretized analogue.
- [Fig. 13] The cubic polynomial fit is used to compare PCAPS and CAP-Decima, but the fit is shown without confidence intervals and is extrapolated to regions with sparse trials; please report per-parameter-point means and standard errors or another uncertainty measure.
Circularity Check
No significant circularity: the theoretical bounds are parameterized applications of external list-scheduling results, and the headline carbon/ECT trade-off is an empirical measurement rather than a definitional or fitted prediction.
full rationale
The derivation chain in Sections 3-4 and Appendix B does not reduce the paper's central claims to its inputs. Theorems 4.3 and 4.5 are genuine applications of Graham's list-scheduling bound to deferral/provisioning parameters D(gamma,c) and M(B,c); they state conditional bounds, not empirical predictions, and the parameters are not fitted from the reported carbon savings. Theorems 4.4 and 4.6 decompose Definition 3.2 into W and weighted-average carbon intensities; this is an algebraic accounting identity, and the paper explicitly notes the savings can be negative (e.g., strictly increasing carbon traces), so the identity is not used as a self-fulfilling guarantee. The relative-importance score r_{v,t} (Definition 4.2) is an unvalidated proxy for precedence criticality, and the Fig. 13 ablation is limited; this is a correctness/robustness concern, not circularity, because r is taken from a JCT-trained scheduler and is not fitted to carbon outcomes, so Algorithm 1's decisions are not definitionally equal to the measured carbon reductions. Self-citations appear only as background for bounded carbon signals and threshold design; the load-bearing bounds come from external sources (Graham [25], k-search [45], Decima [48]). Prototype and simulator results are measurements against real traces and baselines, so the headline trade-off does not collapse into an input.
Assumptions & free parameters
free parameters (4)
- gamma (PCAPS carbon-awareness hyperparameter) =
0.5 (moderate config used for headline results)
- B (CAP minimum resource quota) =
20 executors out of 100 (moderate config)
- theta (GreenHadoop adaptation parameter) =
0.5
- time scale factor =
1 real minute = 1 experiment hour
assumptions (7)
- standard math Graham's list scheduling bound: makespan on K identical machines is at most (2 - 1/K) times optimal.
- domain assumption Carbon intensity c(t) is bounded by known L and U over the relevant horizon.
- domain assumption Carbon footprint of a schedule is C(t) = c(t)E(t), linear in the number of active executors, excluding idle power, cooling, and embodied carbon.
- domain assumption A carbon-agnostic probabilistic scheduler (e.g., Decima) is an instance of list scheduling with a random list.
- ad hoc to paper The relative importance r_{v,t} = p_{v,t} / max_u p_{u,t} measures the precedence-driven importance of task v.
- ad hoc to paper Experimental time scaling of 1 real minute = 1 experiment hour keeps hourly carbon granularity meaningful for job durations.
- domain assumption Each job is a DAG of precedence-constrained tasks (stages) scheduled non-preemptively onto K machines.
Cite this review
Pith. "Pith review of Carbon- and Precedence-Aware Scheduling for Data Processing Clusters." pith.science (2026). https://pith.science/paper/6VSEEFKX
@misc{pith2026250209717,
author = {Pith},
title = {Pith review of: Carbon- and Precedence-Aware Scheduling for Data Processing Clusters},
year = {2026},
howpublished = {\url{https://pith.science/paper/6VSEEFKX}},
note = {Machine review of arXiv:2502.09717}
}
abstract
As large-scale data processing workloads continue to grow, their carbon footprint raises concerns. Prior research on carbon-aware schedulers has focused on shifting computation to align with availability of low-carbon energy, but these approaches assume that each task can be executed independently. In contrast, data processing jobs have precedence constraints (i.e., outputs of one task are inputs for another) that complicate decisions, since delaying an upstream ``bottleneck'' task to a low-carbon period will also block downstream tasks, impacting the entire job's completion time. In this paper, we show that carbon-aware scheduling for data processing benefits from knowledge of both time-varying carbon and precedence constraints. Our main contribution is $\texttt{PCAPS}$, a carbon-aware scheduler that interfaces with modern ML scheduling policies to explicitly consider the precedence-driven importance of each task in addition to carbon. To illustrate the gains due to fine-grained task information, we also study $\texttt{CAP}$, a wrapper for any carbon-agnostic scheduler that adapts the key provisioning ideas of $\texttt{PCAPS}$. Our schedulers enable a configurable priority between carbon reduction and job completion time, and we give analytical results characterizing the trade-off between the two. Furthermore, our Spark prototype on a 100-node Kubernetes cluster shows that a moderate configuration of $\texttt{PCAPS}$ reduces carbon footprint by up to 32.9% without significantly impacting the cluster's total efficiency.
Figures
Figures from the paper (14 more)
Forward citations
Cited by 2 Pith papers
-
A Survey on Task Scheduling in Carbon-Aware Container Orchestration
A systematic survey categorizing carbon-aware Kubernetes scheduling algorithms along hardware/software and energy/carbon axes, with a proposed taxonomy.
-
LLM Harms: A Taxonomy and Discussion
This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.
Reference graph
Works this paper leans on
-
[1]
Cluster data collected from production clus- ters in alibaba for cluster management research, 2018
Alibaba. Cluster data collected from production clus- ters in alibaba for cluster management research, 2018. URL https://github.com/alibaba/clusterdata/ tree/master/cluster-trace-v2018
work page 2018
-
[2]
Resource Quotas – kuber- 12 netes documentation
The Kubernetes Authors. Resource Quotas – kuber- 12 netes documentation. https://kubernetes.io/docs/ concepts/policy/resource-quotas/, 2025. [Ac- cessed 23-01-2025]
work page 2025
-
[3]
A learning-based scheduler for high volume processing in data warehouse using graph neural networks
Vivek Bengre, M Reza HoseinyFarahabady, Mohammad Pivezhandi, Albert Y Zomaya, and Ali Jannesari. A learning-based scheduler for high volume processing in data warehouse using graph neural networks. In International Conference on Parallel and Distributed Computing: Applications and Technologies, pages 175–
-
[4]
Hanafy, Adam Le- chowicz, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili
Roozbeh Bostandoost, Walid A. Hanafy, Adam Le- chowicz, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili. Data-driven Algorithm Selection for Carbon-Aware Scheduling. In Proceedings of the 3rd Workshop on Sustainable Computer Systems, HotCarbon ’24, July 2024
work page 2024
-
[5]
Hanafy, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili
Roozbeh Bostandoost, Adam Lechowicz, Walid A. Hanafy, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili. LACS: Learning-Augmented Algo- rithms for Carbon-Aware Resource Scaling with Un- certain Demand. In Proceedings of the 15th ACM International Conference on Future and Sustainable Energy Systems, e-Energy ’24, page 27–45, New York, NY , USA, 202...
arXiv 2024
-
[6]
Kubernetes and the path to cloud native
Eric Brewer. Kubernetes and the path to cloud native. Santa Clara, CA, July 2015. USENIX Association
work page 2015
-
[7]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christo- pher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...
work page 2020
-
[8]
Decluttering the data mess in llm train- ing
Maximilian Böther, Dan Graur, Xiaozhe Yao, and Ana Klimovic. Decluttering the data mess in llm train- ing. Austin, 2024. HotInfra 2024. doi: 10.3929/ ethz-b-000717691. 2nd Workshop on Hot Topics in System Infrastructure (HotInfra 2024); Conference Lo- cation: Austin, TX, USA; Conference Date: November 3, 2024
work page 2024
Show all 75 references
-
[9]
Data-juicer: A one-stop data processing system for large language models
Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. Data-juicer: A one-stop data processing system for large language models. In Companion of the 2024 International ...
2024
-
[10]
A tutorial survey of job-shop scheduling problems using genetic algorithms—i
Runwei Cheng, Mitsuo Gen, and Yasuhiro Tsujimura. A tutorial survey of job-shop scheduling problems using genetic algorithms—i. representation. Computers & industrial engineering, 30(4):983–997, 1996
1996
-
[11]
Approximation algorithms for precedence-constrained scheduling prob- lems on parallel machines that run at different speeds
Fabián A Chudak and David B Shmoys. Approximation algorithms for precedence-constrained scheduling prob- lems on parallel machines that run at different speeds. Journal of Algorithms, 30(2):323–343, February 1999. ISSN 0196-6774. doi: 10.1006/jagm.1998.0987. URL http://dx.doi....
1999
-
[12]
Scheduler plugins, 2021
Kubernetes Community. Scheduler plugins, 2021. URL https://github.com/kubernetes-sigs/ scheduler-plugins
2021
-
[13]
Competitive algorithms for online knap- sack with succinct predictions, 2024
Mohammadreza Daneshvaramoli, Helia Karisani, Adam Lechowicz, Bo Sun, Cameron Musco, and Mohammad Hajiesmaili. Competitive algorithms for online knap- sack with succinct predictions, 2024. URL https: //arxiv.org/abs/2406.18752
2024 arXiv
-
[14]
Scheduling with communication delays via lp hierarchies and clustering,
Sami Davies, Janardhan Kulkarni, Thomas Rothvoss, Jakub Tarnawski, and Yihao Zhang. Scheduling with communication delays via lp hierarchies and clustering,
-
[15]
Scheduling with Communication Delays via LP Hierarchies and Cluster- ing II: Weighted Completion Times on Related Machines, page 2958–2977
Sami Davies, Janardhan Kulkarni, Thomas Rothvoss, Jakub Tarnawski, and Yihao Zhang. Scheduling with Communication Delays via LP Hierarchies and Cluster- ing II: Weighted Completion Times on Related Machines, page 2958–2977. Society for Industrial and Applied Mathematics, Janua...
2021 doi
-
[16]
Job shop scheduling with genetic al- gorithms
Lawrence Davis. Job shop scheduling with genetic al- gorithms. In Proceedings of the first International Con- ference on Genetic Algorithms and their Applications, pages 136–140. Psychology Press, 2014
2014
-
[17]
Karp, and G
Ran El-Yaniv, Amos Fiat, Richard M. Karp, and G. Turpin. Optimal Search and One-Way Trading On- line Algorithms. Algorithmica, 30(1):101–139, May 2001
2001
-
[18]
Electricity Map
Electricity Maps. Electricity Map. https://www. electricitymap.org/map, Accessed September 2023. 13
2023
-
[19]
The state of internal carbon pric- ing
Jessica Fan, Werner Rehm, Giulia Siccardo, and McK- insey & Company. The state of internal carbon pric- ing. https://www.mckinsey.com/capabilities/ strategy-and-corporate-finance/ our-insights/the-state-of-internal-carbon-pricing , 2021
2021
-
[20]
Configuration – Spark Documentation
The Apache Software Foundation. Configuration – Spark Documentation. https://spark.apache.org/ docs/3.5.3/configuration.html, 2024. [Accessed 12-12-2024]
2024
-
[21]
Running Spark on Kubernetes – Spark Documenta- tion
The Apache Software Foundation. Running Spark on Kubernetes – Spark Documenta- tion. https://spark.apache.org/docs/3.5.3/ running-on-kubernetes.html, 2024. [Accessed 12-12-2024]
2024
-
[22]
Job Scheduling – Spark Documentation
The Apache Software Foundation. Job Scheduling – Spark Documentation. https://spark.apache.org/ docs/3.5.3/job-scheduling.html, 2024. [Accessed 12-12-2024]
2024
-
[23]
A faster reinforcement learning ap- proach to efficient job scheduling in apache spark
Arkadiy Gertsman. A faster reinforcement learning ap- proach to efficient job scheduling in apache spark. Mas- ter’s thesis, University of Illinois at Urbana-Champaign, 2023
2023
-
[24]
Nguyen, Jordi Guitart, Jordi Torres, and Ricardo Bianchini
Íñigo Goiri, Kien Le, Thu D. Nguyen, Jordi Guitart, Jordi Torres, and Ricardo Bianchini. GreenHadoop: Leveraging Green Energy in Data-Processing Frame- works. In Proceedings of the 7th ACM European Conference on Computer Systems, EuroSys ’12, page 57–70, New York, NY , USA, 20...
2012
-
[25]
R. L. Graham. Bounds for certain multiprocessing anomalies. The Bell System Technical Journal , 45 (9):1563–1581, 1966. doi: 10.1002/j.1538-7305.1966. tb01709.x
1966
-
[26]
Geometric deep reinforce- ment learning for dynamic dag scheduling
Nathan Grinsztajn, Olivier Beaumont, Emmanuel Jean- not, and Philippe Preux. Geometric deep reinforce- ment learning for dynamic dag scheduling. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), page 258–265. IEEE, December 2020. doi: 10.1109/ssci47803.2020.9...
2020
-
[27]
Hanafy, Roozbeh Bostandoost, Noman Bashir, David Irwin, Mohammad Hajiesmaili, and Prashant Shenoy
Walid A. Hanafy, Roozbeh Bostandoost, Noman Bashir, David Irwin, Mohammad Hajiesmaili, and Prashant Shenoy. The War of the Efficiencies: Understanding the Tension between Carbon and Energy Optimization. In Proc. of the 2nd Workshop on Sustainable Computer Systems. ACM, Jul 2023
2023
-
[28]
Hanafy, Qianlin Liang, Noman Bashir, David Irwin, and Prashant Shenoy
Walid A. Hanafy, Qianlin Liang, Noman Bashir, David Irwin, and Prashant Shenoy. CarbonScaler: Leverag- ing Cloud Workload Elasticity for Optimizing Carbon- Efficiency. Proc. of the ACM on Measurement and Analysis of Computing Systems, 7(3), Dec 2023
2023
-
[29]
Learning interpretable scheduling al- gorithms for data processing clusters, 2024
Zhibo Hu, Chen Wang, Helen, Paik, Yanfeng Shu, and Liming Zhu. Learning interpretable scheduling al- gorithms for data processing clusters, 2024. URL https://arxiv.org/abs/2405.19131
2024 arXiv
-
[30]
Performance and cost-efficient spark job scheduling based on deep reinforcement learning in cloud computing environments
Muhammed Tawfiqul Islam, Shanika Karunasekera, and Rajkumar Buyya. Performance and cost-efficient spark job scheduling based on deep reinforcement learning in cloud computing environments. IEEE Transactions on Parallel and Distributed Systems, 33(7):1695–1710, 2021
2021
-
[31]
Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs
Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. Lessons learned from the chameleon testbed. In Proceedin...
2020
-
[32]
Minimalistic predictions to schedule jobs with online precedence constraints
Alexandra Anna Lassota, Alexander Lindermayr, Nicole Megow, and Jens Schlöter. Minimalistic predictions to schedule jobs with online precedence constraints. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Pr...
2023
-
[33]
The Online Pause and Re- sume Problem: Optimal Algorithms and An Application to Carbon-Aware Load Shifting
Adam Lechowicz, Nicolas Christianson, Jinhang Zuo, Noman Bashir, Mohammad Hajiesmaili, Adam Wier- man, and Prashant Shenoy. The Online Pause and Re- sume Problem: Optimal Algorithms and An Application to Carbon-Aware Load Shifting. Proc. of the ACM on Measurement and Analysis ...
2023
-
[34]
Online Conversion with Switch- ing Costs: Robust and Learning-augmented Algorithms
Adam Lechowicz, Nicolas Christianson, Bo Sun, No- man Bashir, Mohammad Hajiesmaili, Adam Wierman, and Prashant Shenoy. Online Conversion with Switch- ing Costs: Robust and Learning-augmented Algorithms. In Proc. of the 2024 SIGMETRICS/Performance Joint International Conference...
2024
-
[35]
Chasing Convex Functions with Long-term Constraints
Adam Lechowicz, Nicolas Christianson, Bo Sun, No- man Bashir, Mohammad Hajiesmaili, Adam Wierman, and Prashant Shenoy. Chasing Convex Functions with Long-term Constraints. In Proceedings of the 41st Inter- national Conference on Machine Learning, ICML’24. JMLR, 2024
2024
-
[36]
J. K. Lenstra and A. H. G. Rinnooy Kan. Complexity of scheduling under precedence constraints. Opera- tions Research, 26(1):22–35, 1978. ISSN 0030364X, 15265463. URL http://www.jstor.org/stable/ 169889
1978
-
[37]
Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service
Baolin Li, Siddharth Samsi, Vijay Gadepally, and De- vesh Tiwari. Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Anal- ysis, SC ’23, New ...
2023
-
[38]
Energy-aware scheduling for spark job based on deep reinforcement learning in cloud
Hongjian Li, Liang Lu, Wenhu Shi, Gangfan Tan, and Hao Luo. Energy-aware scheduling for spark job based on deep reinforcement learning in cloud. Computing, 105(8):1717–1743, March 2023. ISSN 1436-5057. doi: 10.1007/s00607-023-01171-z. URL http://dx.doi. org/10.1007/s00607-023-01171-z
2023 doi
-
[39]
Scheduling to minimize total weighted comple- tion time via time-indexed linear programming relax- ations
Shi Li. Scheduling to minimize total weighted comple- tion time via time-indexed linear programming relax- ations. In 2017 IEEE 58th Annual Symposium on Foun- dations of Computer Science (FOCS) , page 283–294. IEEE, October 2017. doi: 10.1109/focs.2017.34. URL http://dx.doi.or...
2017 doi
-
[40]
Cost efficient job scheduling scheme for large scale data center
Xinran Li and Zhaohao Ding. Cost efficient job scheduling scheme for large scale data center. In 2023 IEEE/IAS Industrial and Commercial Power Sys- tem Asia (I&CPS Asia), pages 2267–2272, 2023. doi: 10.1109/ICPSAsia58343.2023.10294452
2023
-
[41]
Data-efficient fine-tuning for llm-based recommendation
Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. Data-efficient fine-tuning for llm-based recommendation. In Pro- ceedings of the 47th International ACM SIGIR Confer- ence on Research and Development in Information Re- trieval, SIGIR ’24,...
-
[42]
When moe meets llms: Parameter efficient fine-tuning for multi- task medical applications
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets llms: Parameter efficient fine-tuning for multi- task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Info...
2024
-
[43]
Online job scheduling scheme for low-carbon data center operation: An information and energy nexus perspective
Wenyu Liu, Yuejun Yan, Yimeng Sun, Hongju Mao, Ming Cheng, Peng Wang, and Zhaohao Ding. Online job scheduling scheme for low-carbon data center operation: An information and energy nexus perspective. Applied Energy, 338:120918, 2023
2023
-
[44]
ISBN 9798400704314
Association for Computing Machinery. ISBN 9798400704314. doi: 10.1145/3626772.3657807. URL https://doi.org/10.1145/3626772.3657807
-
[45]
Optimal Algorithms for k-Search with Applica- tion in Option Pricing
Julian Lorenz, Konstantinos Panagiotou, and Angelika Steger. Optimal Algorithms for k-Search with Applica- tion in Option Pricing. Algorithmica, 55(2):311–328, August 2008
2008
-
[46]
Competitive Caching with Machine Learned Advice
Thodoris Lykouris and Sergei Vassilvtiskii. Competitive Caching with Machine Learned Advice. In Jennifer Dy and Andreas Krause, editors, Proc. of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proc. of Machine Learning Research, pages 3296–3305. PMLR, 10...
2018
-
[47]
Under- standing llms: A comprehensive overview from training to inference, 2024
Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, Yi Pan, Shaochen Xu, Zihao Wu, Zhengliang Liu, Xin Zhang, Shu Zhang, Xintao Hu, Tuo Zhang, Ning Qiang, Tianming Liu, and Bao Ge. Under- standing llms: A...
2024 arXiv
-
[48]
Learning Scheduling Algorithms for Data Processing Clusters
Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh. Learning Scheduling Algorithms for Data Processing Clusters. In Proceedings of the ACM Special Interest Group on Data Communication, SIGCOMM ’19, page 270–288, New York, NY , USA,...
2019
-
[49]
Torgraphina: A scheduler for data pro- cessing during high-frequency job arrival using upside down reinforcement learning
Yamini Mathur. Torgraphina: A scheduler for data pro- cessing during high-frequency job arrival using upside down reinforcement learning. Master’s thesis, Iowa State University, 2023
2023
-
[50]
Schedul- ing precedence-constrained jobs on related machines with communication delay, 2020
Biswaroop Maiti, Rajmohan Rajaraman, David Stalfa, Zoya Svitkina, and Aravindan Vijayaraghavan. Schedul- ing precedence-constrained jobs on related machines with communication delay, 2020. URL https:// arxiv.org/abs/2004.10776
2020 arXiv
-
[51]
On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines
Marius Mosbach, Maksym Andriushchenko, and Diet- rich Klakow. On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations,
-
[52]
A genetic algorithm for the flexible job-shop scheduling problem
Ferdinando Pezzella, Gianluca Morganti, and Giampiero Ciaschetti. A genetic algorithm for the flexible job-shop scheduling problem. Computers & operations research, 35(10):3202–3212, 2008
2008
-
[53]
Docker: lightweight linux containers for consistent development and deployment
Dirk Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239):2, 2014. 15
2014
-
[54]
Carbon- Aware Computing for Datacenters
Ana Radovanovic, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, et al. Carbon- Aware Computing for Datacenters. IEEE Transactions on Power Systems, 2022
2022
-
[55]
A comparison of priority rules for the job shop scheduling problem under different flow time-and tardiness-related objective functions
Veronique Sels, Nele Gheysen, and Mario Vanhoucke. A comparison of priority rules for the job shop scheduling problem under different flow time-and tardiness-related objective functions. International Journal of Production Research, 50(15):4255–4270, 2012
2012
-
[56]
Cloud-native workflow scheduling using a hybrid priority rule, dy- namic resource allocation, and dynamic task partition
Jungeun Shin, Diana Arroyo, Asser Tantawi, Chen Wang, Alaa Youssef, and Rakesh Nagi. Cloud-native workflow scheduling using a hybrid priority rule, dy- namic resource allocation, and dynamic task partition. In Proceedings of the 2024 ACM Symposium on Cloud Computing, SoCC ’24,...
2024
-
[57]
Im- proving Online Algorithms via ML Predictions
Manish Purohit, Zoya Svitkina, and Ravi Kumar. Im- proving Online Algorithms via ML Predictions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018
2018
-
[58]
Communication-aware scheduling of precedence- constrained tasks on related machines
Yu Su, Shai Vardi, Xiaoqi Ren, and Adam Wier- man. Communication-aware scheduling of precedence- constrained tasks on related machines. Opera- tions Research Letters , 51(6):709–716, 2023. ISSN 0167-6377. doi: https://doi.org/10.1016/j.orl.2023. 11.001. URL https://www.science...
2023 doi
-
[59]
Learning-augmented energy-aware list scheduling for precedence-constrained tasks
Yu Su, Vivek Anand, Jannie Yu, Jian Tan, and Adam Wierman. Learning-augmented energy-aware list scheduling for precedence-constrained tasks. ACM Trans. Model. Perform. Eval. Comput. Syst., 2024. doi: 10.1145/3680278. URL https://doi.org/10.1145/ 3680278
2024 doi
-
[60]
The tpc-h benchmarks, 2018
TPC-H. The tpc-h benchmarks, 2018. URL https: //www.tpc.org/tpch/
2018
-
[61]
We’re increasing our carbon fee as we double down on sustainability
Brad Smith and Microsoft Corporation. We’re increasing our carbon fee as we double down on sustainability. https://blogs. microsoft.com/on-the-issues/2019/04/15/ were-increasing-our-carbon-fee-as-we-double-down-on-sustainability/ , 2019
2019
-
[62]
WattTime
WattTime. WattTime. https://www.watttime.org, Accessed June 2024
2024
-
[63]
Let’s Wait Awhile: How Temporal Workload Shifting Can Re- duce Carbon Emissions in the Cloud
Philipp Wiesner, Ilja Behnke, Dominik Scheinert, Ko- rdian Gontarska, and Lauritz Thamsen. Let’s Wait Awhile: How Temporal Workload Shifting Can Re- duce Carbon Emissions in the Cloud. In Proceed- ings of the 22nd International Middleware Conference, Middleware ’21, page 260–2...
-
[64]
Sus- tainable AI: Environmental Implications, Challenges and Opportunities
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sus- tainable AI: Environmental Implications, Challenges and Opportunities. Proceedings of Machine Learning and Systems (MLSys), ...
2022
-
[65]
Will we run out of data? limits of llm scaling based on human- generated data, 2024
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Be- siroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human- generated data, 2024. URL https://arxiv.org/abs/ 2211.04325
2024 arXiv
-
[66]
Franklin, Scott Shenker, and Ion Stoica
Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. Resilient distributed datasets: a fault-tolerant abstraction for in- memory cluster computing. In Proceedings of the 9th USENIX Confere...
2012
-
[67]
Xin, Patrick Wendell, Tatha- gata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J
Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tatha- gata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. Apache spark: a unified engine for big data process...
-
[68]
ISBN 9781450385343
Association for Computing Machinery. ISBN 9781450385343. doi: 10.1145/3464298.3493399. URL https://doi.org/10.1145/3464298.3493399
-
[69]
green window
Yunhong Zhou, Deeparnab Chakrabarty, and Rajan Lukose. Budget Constrained Bidding in Keyword Auc- tions and Online Knapsack Problems. In Lecture Notes in Computer Science, pages 566–576. Springer Berlin Heidelberg, 2008. 17 Appendix A Evaluation Supplements In this section, we...
2008
-
[70]
Adaptive DAG Tasks Scheduling with Deep Reinforce- ment Learning, page 477–490
Qing Wu, Zhiwei Wu, Yuehui Zhuang, and Yuxia Cheng. Adaptive DAG Tasks Scheduling with Deep Reinforce- ment Learning, page 477–490. Springer International Publishing, 2018. ISBN 9783030050542. doi: 10.1007/ 978-3-030-05054-2_37. URL http://dx.doi.org/ 10.1007/978-3-030-05054-2_37
2018 doi
-
[74]
Learning to optimize dag scheduling in heterogeneous environment
Yunfan Zhou, Xijun Li, Jinhong Luo, Mingxuan Yuan, Jia Zeng, and Jianguo Yao. Learning to optimize dag scheduling in heterogeneous environment. In 2022 23rd IEEE International Conference on Mobile Data Management (MDM), pages 137–146, 2022. doi: 10.1109/MDM55031.2022.00040
2022
-
[2016]
doi: 10.1145/2934664
ISSN 0001-0782. doi: 10.1145/2934664. URL https://doi.org/10.1145/2934664. 16
-
[2020]
URL https://arxiv.org/abs/2004.09682
2004 arXiv
-
[2021]
URL https://openreview.net/forum?id= nzpLWnVAyah
-
[2024]
Association for Computing Machinery. 14
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.