Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Carbon- and Precedence-Aware Scheduling for Data Processing Clusters

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A scheduler that weighs task precedence against grid carbon can cut data-processing emissions by up to 39.7 percent, while keeping end-to-end completion time nearly unchanged.

desk verdict PCAPS is a real systems contribution, but the theory is bookkeeping and the paper's central assumption—that softmax probability tracks task criticality—is asserted, not validated. read the letter →

arxiv 2502.09717 v1 pith:6VSEEFKX submitted 2025-02-13 cs.DC cs.CYcs.SYeess.SY

classification cs.DCcs.CYcs.SYeess.SY
keywords carbon-awareschedulingprecedence-constrainedDAGsdataprocessingclustersreinforcementlearningschedulercarbonstretchfactorthreshold-basedonlinealgorithmsApacheSparkKubernetes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that carbon-aware scheduling for data-processing clusters should use both the time-varying carbon intensity of the grid and the precedence structure of each job's task graph. It proposes PCAPS, a scheduler that wraps an existing machine-learned probabilistic scheduler and decides at each scheduling event whether to run the sampled task or idle it until a cleaner period. The decision uses a relative-importance score: tasks the learned scheduler considers critical run even during high-carbon hours, while less important tasks wait. In a 100-node Spark/Kubernetes prototype and a high-fidelity simulator, a moderately carbon-aware PCAPS cuts carbon by about a third to 39.7 percent while raising end-to-end completion time by only a few percent, at the cost of substantially longer average per-job completion times. The paper also gives analytical bounds on the carbon-versus-completion-time trade-off for PCAPS and for a simpler provisioning-only variant, CAP.

What carries the argument

The load-bearing object is the relative-importance ratio $r_{v,t} := p_{v,t}/\max_{u\in A_t} p_{u,t}$, computed from the probability distribution over ready tasks generated by a machine-learned scheduler, together with the exponential threshold function $\Psi_\gamma(r)$ that turns this ratio into a run-or-defer decision. When the current carbon intensity $c(t)$ exceeds $\Psi_\gamma(r_{v,t})$ and at least one machine is busy, the sampled task is idled rather than executed; otherwise it runs. Because $\Psi_\gamma(1)=U$, the highest-importance task is always allowed to run regardless of carbon, which is the mechanism that protects the DAG's critical path. The proofs also lean on Graham's list-scheduling bound, which turns deferrals into a bounded makespan increase and yields the carbon stretch factor. CAP, by contrast, uses a k-search-inspired threshold set to set a cluster-wide resource quota, so it captures the carbon-shifting half of the idea but not the per-task precedence half.

What would settle it

Run PCAPS on a test batch where the scheduler's highest-probability tasks are known not to lie on the critical path, or replace its importance scores with random permutations while keeping the same deferral thresholds; if the carbon-versus-ECT trade-off does not degrade, then relative importance is not what is producing the savings.

Watch

Extended reading notes

Core claim

The central claim is that fine-grained, precedence-aware deferral is what makes carbon-aware scheduling worthwhile for data processing. PCAPS assumes that the softmax probabilities produced by a probabilistic scheduler such as Decima contain implicit information about which tasks are bottlenecks: the ratio $r_{v,t} = p_{v,t}/\max_{u\in A_t} p_{u,t}$ measures how much the scheduler wants task $v$ to run now. An exponential threshold function $\Psi_\gamma(r)$, parameterized by a carbon-awareness knob $\gamma$ and bounded by the known minimum and maximum carbon intensities, determines whether $v$ is dispatched to a machine or deferred. Critical tasks (those with $r$ near 1) are always dispatched; tasks with low relative importance are delayed when carbon intensity is high, and the freed capacity is used later. The paper proves that this filter gives PCAPS a carbon stretch factor of $1 + \frac{D(\gamma,c)K}{2 - 1/K}$, characterizes its carbon savings, and demonstrates experimentally that the idea beats both a carbon-agnostic ML scheduler and a uniform resource-quota wrapper (CAP) on the carbon-versus-end-to-end-time frontier.

Load-bearing premise

PCAPS's benefit rests on treating the probabilities emitted by the learned scheduler as a reliable ranking of how costly it is to delay each task, and the paper does not directly verify that high-probability tasks are the true bottlenecks.

Editorial extensions

If this is right

  • A moderately carbon-aware PCAPS reduces carbon emissions by up to 39.7% in the simulator and 32.9% in the prototype, with average end-to-end completion time increasing by only 4.5% in the simulator and 1.3% in the prototype.
  • The same setting raises average job completion time by 38.1% in the prototype and about 119% relative to Decima in the simulator, so the carbon savings come from queue build-up during high-carbon periods rather than from faster execution.
  • CAP, which only scales cluster resources without looking at task dependencies, achieves smaller carbon savings (up to 25.1%) and pays more end-to-end time for the same carbon reduction, confirming that per-task precedence information is the differentiator.
  • The trade-off is tunable: $\gamma=0$ reproduces the carbon-agnostic scheduler, $\gamma=1$ is maximally carbon-aware, and the achievable carbon savings grow with the variability of the grid's carbon intensity.
  • On grids with stable, mostly coal-based intensity (ZA in the paper's data), the schedulers defer little and save little; on renewable-dominated grids with strong diurnal swings (CAISO), they wait for solar hours and save more.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the softmax probability is a faithful criticality proxy, then PCAPS is a drop-in carbon filter for any probabilistic DAG scheduler, not just Decima; a natural stress test is to replace the learned probabilities with critical-path slacks and compare the frontier.
  • The large gap between average JCT and ECT suggests PCAPS is effectively trading fairness across jobs for aggregate throughput; in settings with per-job service-level objectives, the deferral policy would need a deadline or a per-job slack guardrail.
  • The prototype's 32.9% carbon cut at 1.3% ECT cost implies that under internal carbon pricing, such a filter could pay for itself largely as an accounting change, but the realized cost saving depends on how tightly the carbon price tracks grid intensity.
  • A direct ablation—running PCAPS with random importance scores while keeping the same threshold function—would isolate how much of the benefit is due to precedence information rather than to the carbon-triggered idling alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PCAPS, a carbon-aware scheduler for precedence-constrained data processing workloads, and CAP, a simpler resource-provisioning wrapper. PCAPS augments a probabilistic ML scheduler (e.g., Decima) with a threshold filter: using the ratio r_{v,t} = p_{v,t}/max_u p_{u,t} as a proxy for task importance, it runs high-importance tasks even during high-carbon periods and defers low-importance ones. CAP instead varies the cluster resource quota according to carbon intensity while leaving the underlying scheduler unchanged. The authors give analytical results on carbon stretch factor and carbon savings (Theorems 4.3–4.6), and evaluate both schedulers in a Spark-on-Kubernetes prototype on 100 executors and in a Spark simulator across six carbon-intensity traces and TPC-H/Alibaba workloads. The headline results are a 32.9% average carbon reduction in the prototype with near-neutral ECT relative to the default, and up to 39.7% carbon reduction in the simulator with modest ECT increases.

Significance. If the core mechanism is valid, the paper offers a practical, configurable way to reduce data-processing carbon footprint without severely degrading cluster throughput, and it provides one of the few evaluations of carbon-aware scheduling on a real 100-node Spark cluster. The prototype and the six-grid simulator study are substantial, the code is made available, and Appendix A.1.2 honestly documents the simulator/prototype baseline differences. The main limitation is that the distinguishing mechanism—interpreting Decima's softmax probability ratio as precedence-driven criticality—is an unverified proxy, and several analytical results are definitional rather than substantive. The contribution is therefore promising but not yet fully supported.

major comments (4)
  1. [§4.1, Definition 4.2 and Algorithm 1] The paper's distinguishing mechanism treats the softmax probability ratio r_{v,t} = p_{v,t} / max_{u in A_t} p_{u,t} as 'precedence-driven importance' and uses it to decide which tasks run during high-carbon periods. This identification is an assumption, not a derived property of Decima or of any probabilistic scheduler. The only direct evidence, Fig. 13, compares PCAPS to CAP-Decima on one grid (DE), one batch size (50 TPC-H jobs), and without confidence intervals, and the two systems differ in more than the r computation (fine-grained per-task deferral vs. cluster-level quotas). If r does not track criticality, PCAPS may defer exactly the tasks that block downstream DAGs, and the claimed small ECT increase will not generalize. I recommend a direct validation of the proxy: for example, compare deferral decisions against critical-path or descendant-count measures of task importance, and run an ablation in which r is replaced by random or anti-correlated priorities across several grids and batch sizes.
  2. [§4.1, Proof of Theorem 4.3 (Appendix B.1.1)] The proof bounds the total number of deferrals by E[D] <= n-1, arguing that at least one machine is active at all times. This does not bound D_v: a task can be sampled and deferred many times, especially because Algorithm 1 treats carbon-intensity changes as scheduling events, so the same ready task can be deferred repeatedly during a long high-carbon interval. Moreover, D(γ,c) is defined post hoc from E[D] and the longest task runtimes, so the 'carbon stretch factor' formula is an identity that holds for any scheduler and any γ; it carries no predictive content unless D(γ,c) is bounded a priori in terms of γ, K, L, U, and the DAG. Please either prove such a bound or recast Theorem 4.3 as a definition/measurement rather than a theorem.
  3. [§4.2, Proof of Theorem 4.5 (Appendix B.2.1)] The chain CAPK(J | M) <= AG_M(J) assumes that having additional machines and time-varying quotas cannot hurt. This is not generally true for list scheduling: Graham's anomalies show that adding processors can increase makespan for precedence-constrained jobs, and dynamic quota changes can fragment the schedule. Since Theorem 4.5 is presented as an analytical bound on CAP's carbon stretch factor, the monotonicity step needs a proof or a counterexample-free justification; otherwise the theorem should be stated under explicit assumptions on the scheduler and quota dynamics.
  4. [§4.1–4.2, Theorems 4.4 and 4.6 (Appendices B.1.2, B.2.2)] The carbon-savings results are exact algebraic rearrangements of Definition 3.2 after defining W, s_-, s_+, and c(T,T') in terms of the schedules being compared. They therefore cannot fail and do not, by themselves, establish that any particular scheduling policy saves carbon; the sign of the savings depends entirely on whether the weighted intensities satisfy s_- > s_+ + c(T,T'). I recommend presenting these as accounting identities and moving the burden of the trade-off claim to the empirical evaluation and to a priori bounds on the deferred work W.
minor comments (4)
  1. [Abstract and §6.3, Table 2] The abstract's 'up to 32.9%' is an average over traces and trials in Table 2, not a maximum; also the text in §6.3 reports 32.8% for PCAPS while Table 2 reports 32.9% (and 24.6% vs. 24.7% for CAP). Please align the wording and the numbers.
  2. [§6.5] The takeaway 'reduces carbon emissions by up to 39.7%' is again an average over the six carbon traces in Table 3, not a per-trace maximum; please state it as an average or report the actual maximum separately.
  3. [Definition 3.2 and Appendix B.1.2] The carbon-savings definition is written with integrals, while the proofs use discrete sums; state explicitly that the integrals are over the continuous time model and the sums are the discretized analogue.
  4. [Fig. 13] The cubic polynomial fit is used to compare PCAPS and CAP-Decima, but the fit is shown without confidence intervals and is extrapolated to regions with sparse trials; please report per-parameter-point means and standard errors or another uncertainty measure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical bounds are parameterized applications of external list-scheduling results, and the headline carbon/ECT trade-off is an empirical measurement rather than a definitional or fitted prediction.

full rationale

The derivation chain in Sections 3-4 and Appendix B does not reduce the paper's central claims to its inputs. Theorems 4.3 and 4.5 are genuine applications of Graham's list-scheduling bound to deferral/provisioning parameters D(gamma,c) and M(B,c); they state conditional bounds, not empirical predictions, and the parameters are not fitted from the reported carbon savings. Theorems 4.4 and 4.6 decompose Definition 3.2 into W and weighted-average carbon intensities; this is an algebraic accounting identity, and the paper explicitly notes the savings can be negative (e.g., strictly increasing carbon traces), so the identity is not used as a self-fulfilling guarantee. The relative-importance score r_{v,t} (Definition 4.2) is an unvalidated proxy for precedence criticality, and the Fig. 13 ablation is limited; this is a correctness/robustness concern, not circularity, because r is taken from a JCT-trained scheduler and is not fitted to carbon outcomes, so Algorithm 1's decisions are not definitionally equal to the measured carbon reductions. Self-citations appear only as background for bounded carbon signals and threshold design; the load-bearing bounds come from external sources (Graham [25], k-search [45], Decima [48]). Prototype and simulator results are measurements against real traces and baselines, so the headline trade-off does not collapse into an input.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

No new physical or theoretical entities are proposed. PCAPS and CAP are software artifacts with released code and their own experimental falsifiable handles. The central claims rest on standard scheduling analysis (Graham's list scheduling), domain assumptions about bounded carbon signals and the linear c(t)E(t) model, and two ad hoc-to-paper design choices: the relative-importance proxy and the time scaling factor.

free parameters (4)
  • gamma (PCAPS carbon-awareness hyperparameter) = 0.5 (moderate config used for headline results)
    Hand-chosen in prototype and simulator experiments to represent 'moderate' carbon-awareness; the reported 32.9% and 39.7% carbon reductions depend on this setting, and results vary substantially with gamma (Figures 7 and 11).
  • B (CAP minimum resource quota) = 20 executors out of 100 (moderate config)
    Hand-chosen to balance carbon savings and ECT for the headline CAP results; the trade-off is sensitive to B (Figures 8 and 12).
  • theta (GreenHadoop adaptation parameter) = 0.5
    Tunable carbon-awareness parameter in the adapted GreenHadoop baseline (Appendix A.1.1); affects the baseline comparison but not the central PCAPS/CAP claims.
  • time scale factor = 1 real minute = 1 experiment hour
    All job durations are scaled by 1/60 (Section 6.1) so that hourly carbon intensity granularity is meaningful for multi-hour jobs; this makes carbon deferral possible and shapes the empirical savings.
assumptions (7)
  • standard math Graham's list scheduling bound: makespan on K identical machines is at most (2 - 1/K) times optimal.
    Used in the proofs of Theorem 4.3 (Appendix B.1.1) and Theorem 4.5 (Appendix B.2.1), cited as [25].
  • domain assumption Carbon intensity c(t) is bounded by known L and U over the relevant horizon.
    Section 2.1, following prior work [5,33]; required for the threshold functions Psi_gamma and Phi to be well-defined and for the k-search thresholds in CAP.
  • domain assumption Carbon footprint of a schedule is C(t) = c(t)E(t), linear in the number of active executors, excluding idle power, cooling, and embodied carbon.
    Section 3 defines C_ALG(t) := c(t)E_ALG(t); all reported carbon savings use this linear accounting model.
  • domain assumption A carbon-agnostic probabilistic scheduler (e.g., Decima) is an instance of list scheduling with a random list.
    Appendix B.1.1 relies on this to apply Graham's bound to the PB baseline; it presumes a single task is greedily assigned per idle machine without preemption.
  • ad hoc to paper The relative importance r_{v,t} = p_{v,t} / max_u p_{u,t} measures the precedence-driven importance of task v.
    Definition 4.2 and Algorithm 1; this is the load-bearing design assumption for PCAPS's deferral decisions and is not directly validated against critical-path metrics.
  • ad hoc to paper Experimental time scaling of 1 real minute = 1 experiment hour keeps hourly carbon granularity meaningful for job durations.
    Section 6.1 scales all job durations by 1/60; this modeling choice is specific to this paper's evaluation setup.
  • domain assumption Each job is a DAG of precedence-constrained tasks (stages) scheduled non-preemptively onto K machines.
    Section 2.1 formalizes the job model; inherited from Spark and prior DAG scheduling work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Carbon- and Precedence-Aware Scheduling for Data Processing Clusters." pith.science (2026). https://pith.science/paper/6VSEEFKX

@misc{pith2026250209717,
  author       = {Pith},
  title        = {Pith review of: Carbon- and Precedence-Aware Scheduling for Data Processing Clusters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VSEEFKX}},
  note         = {Machine review of arXiv:2502.09717}
}
abstract

As large-scale data processing workloads continue to grow, their carbon footprint raises concerns. Prior research on carbon-aware schedulers has focused on shifting computation to align with availability of low-carbon energy, but these approaches assume that each task can be executed independently. In contrast, data processing jobs have precedence constraints (i.e., outputs of one task are inputs for another) that complicate decisions, since delaying an upstream ``bottleneck'' task to a low-carbon period will also block downstream tasks, impacting the entire job's completion time. In this paper, we show that carbon-aware scheduling for data processing benefits from knowledge of both time-varying carbon and precedence constraints. Our main contribution is $\texttt{PCAPS}$, a carbon-aware scheduler that interfaces with modern ML scheduling policies to explicitly consider the precedence-driven importance of each task in addition to carbon. To illustrate the gains due to fine-grained task information, we also study $\texttt{CAP}$, a wrapper for any carbon-agnostic scheduler that adapts the key provisioning ideas of $\texttt{PCAPS}$. Our schedulers enable a configurable priority between carbon reduction and job completion time, and we give analytical results characterizing the trade-off between the two. Furthermore, our Spark prototype on a 100-node Kubernetes cluster shows that a moderate configuration of $\texttt{PCAPS}$ reduces carbon footprint by up to 32.9% without significantly impacting the cluster's total efficiency.

Figures

Figures reproduced from arXiv: 2502.09717 by the authors.

Figure 1
Figure 1. Four scheduling policies for a motivating DAG and 18-hour-long carbon intensity trace (on the left hand side). [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. PCAPS interfaces with a probabilistic (PB) schedul￾ing policy. Given a probability distribution over nodes ➊, PCAPS computes a relative importance score ➋ that is used to determine which nodes should run based on the current carbon intensity ➌ – e.g., bottleneck nodes impeding job com￾pletion run regardless of carbon ➍, while less important nodes can be deferred for lower carbon periods ➎. 4.1 PCAPS From the discuss… view at source ↗
Figure 3
Figure 3. Illustrating PCAPS’s carbon-awareness filter. Jobs A and B are DAGs found in TPC-H queries and Alibaba traces, respectively [1, 60]. Highlighted nodes explain two schedul￾ing outcomes. In job A, the sampled node has low relative importance, so it is deferred. In contrast, job B’s sampled node is a bottleneck task with high relative importance: even when the current carbon intensity is high, such tasks are scheduled … view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: Time-varying carbon intensity for six grids (detailed [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualizing executor usage over time for three schedulers, [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Relative carbon footprint and end-to-end completion [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Relative carbon footprint and end-to-end completion [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 11
Figure 11. Figure 11: Relative carbon footprint and end-to-end com [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Relative carbon footprint and end-to-end comple [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Relative carbon footprint vs. end-to-end com [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 15
Figure 15. Figure 15: Executor usage (left) and number of jobs in the system (right) for an identical group of 50 TPC-H jobs in both the [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: plots the average carbon reduction, end-to-end completion time, and average per-job completion time with respect to FIFO for PCAPS, Decima, and CAP on top of FIFO in the simulator environment, using experiments with 12, 25, 50, 100, and 200 jobs. We find that the rela…
Figure 17
Figure 17. Figure 17: (a) Carbon reduction, (b) end-to-end completion time, and (c) average job completion time achieved by PCAPS, CAP, and Decima (relative to the Spark/Kubernetes default) in a single grid region for varying experiment sizes. Shaded regions denote the standard deviation a…
Figure 18
Figure 18. Figure 18: (a) Carbon reduction, (b) end-to-end completion time, and (c) average job completion time achieved by PCAPS, CAP-FIFO, and Decima (relative to FIFO) in a single grid region for varying Poisson interarrival times. Shaded regions denote the standard deviation across the…
Figure 19
Figure 19. Figure 19: (a) Carbon reduction, (b) end-to-end completion time, and (c) average job completion time achieved by PCAPS, CAP, and Decima (relative to the Spark/Kubernetes default) in a single grid region for varying Poisson interarrival times. Shaded regions denote the standard d…
Figure 20
Figure 20. Figure 20: (a) Average latency with N jobs in the queue and (b) average normalized time in the scheduler for PCAPS, CAP-FIFO, Decima, and FIFO in a single grid region for varying experiment sizes. Shaded region denotes the standard deviation across all 1000 trials. B Deferred An…
Figure 21
Figure 21. Figure 21: An example to contextualize how the optimal makespan differs when a job is given different amounts of machines. In [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Task Scheduling in Carbon-Aware Container Orchestration

    cs.SE 2025-08 conditional novelty 4.0 of 10

    A systematic survey categorizing carbon-aware Kubernetes scheduling algorithms along hardware/software and energy/carbon axes, with a proposed taxonomy.

  2. LLM Harms: A Taxonomy and Discussion

    cs.CY 2025-12 unverdicted novelty 3.0 of 10

    This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.

Reference graph

Works this paper leans on

75 extracted references · 58 canonical work pages · cited by 2 Pith papers

  1. [1]

    Cluster data collected from production clus- ters in alibaba for cluster management research, 2018

    Alibaba. Cluster data collected from production clus- ters in alibaba for cluster management research, 2018. URL https://github.com/alibaba/clusterdata/ tree/master/cluster-trace-v2018

  2. [2]

    Resource Quotas – kuber- 12 netes documentation

    The Kubernetes Authors. Resource Quotas – kuber- 12 netes documentation. https://kubernetes.io/docs/ concepts/policy/resource-quotas/, 2025. [Ac- cessed 23-01-2025]

  3. [3]

    A learning-based scheduler for high volume processing in data warehouse using graph neural networks

    Vivek Bengre, M Reza HoseinyFarahabady, Mohammad Pivezhandi, Albert Y Zomaya, and Ali Jannesari. A learning-based scheduler for high volume processing in data warehouse using graph neural networks. In International Conference on Parallel and Distributed Computing: Applications and Technologies, pages 175–

  4. [4]

    Hanafy, Adam Le- chowicz, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili

    Roozbeh Bostandoost, Walid A. Hanafy, Adam Le- chowicz, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili. Data-driven Algorithm Selection for Carbon-Aware Scheduling. In Proceedings of the 3rd Workshop on Sustainable Computer Systems, HotCarbon ’24, July 2024

  5. [5]

    Hanafy, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili

    Roozbeh Bostandoost, Adam Lechowicz, Walid A. Hanafy, Noman Bashir, Prashant Shenoy, and Moham- mad Hajiesmaili. LACS: Learning-Augmented Algo- rithms for Carbon-Aware Resource Scaling with Un- certain Demand. In Proceedings of the 15th ACM International Conference on Future and Sustainable Energy Systems, e-Energy ’24, page 27–45, New York, NY , USA, 202...

  6. [6]

    Kubernetes and the path to cloud native

    Eric Brewer. Kubernetes and the path to cloud native. Santa Clara, CA, July 2015. USENIX Association

  7. [7]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christo- pher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

  8. [8]

    Decluttering the data mess in llm train- ing

    Maximilian Böther, Dan Graur, Xiaozhe Yao, and Ana Klimovic. Decluttering the data mess in llm train- ing. Austin, 2024. HotInfra 2024. doi: 10.3929/ ethz-b-000717691. 2nd Workshop on Hot Topics in System Infrastructure (HotInfra 2024); Conference Lo- cation: Austin, TX, USA; Conference Date: November 3, 2024

Show all 75 references
  1. [9]

    Data-juicer: A one-stop data processing system for large language models

    Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. Data-juicer: A one-stop data processing system for large language models. In Companion of the 2024 International ...

  2. [10]

    A tutorial survey of job-shop scheduling problems using genetic algorithms—i

    Runwei Cheng, Mitsuo Gen, and Yasuhiro Tsujimura. A tutorial survey of job-shop scheduling problems using genetic algorithms—i. representation. Computers & industrial engineering, 30(4):983–997, 1996

  3. [11]

    Approximation algorithms for precedence-constrained scheduling prob- lems on parallel machines that run at different speeds

    Fabián A Chudak and David B Shmoys. Approximation algorithms for precedence-constrained scheduling prob- lems on parallel machines that run at different speeds. Journal of Algorithms, 30(2):323–343, February 1999. ISSN 0196-6774. doi: 10.1006/jagm.1998.0987. URL http://dx.doi....

  4. [12]

    Scheduler plugins, 2021

    Kubernetes Community. Scheduler plugins, 2021. URL https://github.com/kubernetes-sigs/ scheduler-plugins

  5. [13]

    Competitive algorithms for online knap- sack with succinct predictions, 2024

    Mohammadreza Daneshvaramoli, Helia Karisani, Adam Lechowicz, Bo Sun, Cameron Musco, and Mohammad Hajiesmaili. Competitive algorithms for online knap- sack with succinct predictions, 2024. URL https: //arxiv.org/abs/2406.18752

  6. [14]

    Scheduling with communication delays via lp hierarchies and clustering,

    Sami Davies, Janardhan Kulkarni, Thomas Rothvoss, Jakub Tarnawski, and Yihao Zhang. Scheduling with communication delays via lp hierarchies and clustering,

  7. [15]

    Scheduling with Communication Delays via LP Hierarchies and Cluster- ing II: Weighted Completion Times on Related Machines, page 2958–2977

    Sami Davies, Janardhan Kulkarni, Thomas Rothvoss, Jakub Tarnawski, and Yihao Zhang. Scheduling with Communication Delays via LP Hierarchies and Cluster- ing II: Weighted Completion Times on Related Machines, page 2958–2977. Society for Industrial and Applied Mathematics, Janua...

  8. [16]

    Job shop scheduling with genetic al- gorithms

    Lawrence Davis. Job shop scheduling with genetic al- gorithms. In Proceedings of the first International Con- ference on Genetic Algorithms and their Applications, pages 136–140. Psychology Press, 2014

  9. [17]

    Karp, and G

    Ran El-Yaniv, Amos Fiat, Richard M. Karp, and G. Turpin. Optimal Search and One-Way Trading On- line Algorithms. Algorithmica, 30(1):101–139, May 2001

  10. [18]

    Electricity Map

    Electricity Maps. Electricity Map. https://www. electricitymap.org/map, Accessed September 2023. 13

  11. [19]

    The state of internal carbon pric- ing

    Jessica Fan, Werner Rehm, Giulia Siccardo, and McK- insey & Company. The state of internal carbon pric- ing. https://www.mckinsey.com/capabilities/ strategy-and-corporate-finance/ our-insights/the-state-of-internal-carbon-pricing , 2021

  12. [20]

    Configuration – Spark Documentation

    The Apache Software Foundation. Configuration – Spark Documentation. https://spark.apache.org/ docs/3.5.3/configuration.html, 2024. [Accessed 12-12-2024]

  13. [21]

    Running Spark on Kubernetes – Spark Documenta- tion

    The Apache Software Foundation. Running Spark on Kubernetes – Spark Documenta- tion. https://spark.apache.org/docs/3.5.3/ running-on-kubernetes.html, 2024. [Accessed 12-12-2024]

  14. [22]

    Job Scheduling – Spark Documentation

    The Apache Software Foundation. Job Scheduling – Spark Documentation. https://spark.apache.org/ docs/3.5.3/job-scheduling.html, 2024. [Accessed 12-12-2024]

  15. [23]

    A faster reinforcement learning ap- proach to efficient job scheduling in apache spark

    Arkadiy Gertsman. A faster reinforcement learning ap- proach to efficient job scheduling in apache spark. Mas- ter’s thesis, University of Illinois at Urbana-Champaign, 2023

  16. [24]

    Nguyen, Jordi Guitart, Jordi Torres, and Ricardo Bianchini

    Íñigo Goiri, Kien Le, Thu D. Nguyen, Jordi Guitart, Jordi Torres, and Ricardo Bianchini. GreenHadoop: Leveraging Green Energy in Data-Processing Frame- works. In Proceedings of the 7th ACM European Conference on Computer Systems, EuroSys ’12, page 57–70, New York, NY , USA, 20...

  17. [25]

    R. L. Graham. Bounds for certain multiprocessing anomalies. The Bell System Technical Journal , 45 (9):1563–1581, 1966. doi: 10.1002/j.1538-7305.1966. tb01709.x

  18. [26]

    Geometric deep reinforce- ment learning for dynamic dag scheduling

    Nathan Grinsztajn, Olivier Beaumont, Emmanuel Jean- not, and Philippe Preux. Geometric deep reinforce- ment learning for dynamic dag scheduling. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), page 258–265. IEEE, December 2020. doi: 10.1109/ssci47803.2020.9...

  19. [27]

    Hanafy, Roozbeh Bostandoost, Noman Bashir, David Irwin, Mohammad Hajiesmaili, and Prashant Shenoy

    Walid A. Hanafy, Roozbeh Bostandoost, Noman Bashir, David Irwin, Mohammad Hajiesmaili, and Prashant Shenoy. The War of the Efficiencies: Understanding the Tension between Carbon and Energy Optimization. In Proc. of the 2nd Workshop on Sustainable Computer Systems. ACM, Jul 2023

  20. [28]

    Hanafy, Qianlin Liang, Noman Bashir, David Irwin, and Prashant Shenoy

    Walid A. Hanafy, Qianlin Liang, Noman Bashir, David Irwin, and Prashant Shenoy. CarbonScaler: Leverag- ing Cloud Workload Elasticity for Optimizing Carbon- Efficiency. Proc. of the ACM on Measurement and Analysis of Computing Systems, 7(3), Dec 2023

  21. [29]

    Learning interpretable scheduling al- gorithms for data processing clusters, 2024

    Zhibo Hu, Chen Wang, Helen, Paik, Yanfeng Shu, and Liming Zhu. Learning interpretable scheduling al- gorithms for data processing clusters, 2024. URL https://arxiv.org/abs/2405.19131

  22. [30]

    Performance and cost-efficient spark job scheduling based on deep reinforcement learning in cloud computing environments

    Muhammed Tawfiqul Islam, Shanika Karunasekera, and Rajkumar Buyya. Performance and cost-efficient spark job scheduling based on deep reinforcement learning in cloud computing environments. IEEE Transactions on Parallel and Distributed Systems, 33(7):1695–1710, 2021

  23. [31]

    Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs

    Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. Lessons learned from the chameleon testbed. In Proceedin...

  24. [32]

    Minimalistic predictions to schedule jobs with online precedence constraints

    Alexandra Anna Lassota, Alexander Lindermayr, Nicole Megow, and Jens Schlöter. Minimalistic predictions to schedule jobs with online precedence constraints. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Pr...

  25. [33]

    The Online Pause and Re- sume Problem: Optimal Algorithms and An Application to Carbon-Aware Load Shifting

    Adam Lechowicz, Nicolas Christianson, Jinhang Zuo, Noman Bashir, Mohammad Hajiesmaili, Adam Wier- man, and Prashant Shenoy. The Online Pause and Re- sume Problem: Optimal Algorithms and An Application to Carbon-Aware Load Shifting. Proc. of the ACM on Measurement and Analysis ...

  26. [34]

    Online Conversion with Switch- ing Costs: Robust and Learning-augmented Algorithms

    Adam Lechowicz, Nicolas Christianson, Bo Sun, No- man Bashir, Mohammad Hajiesmaili, Adam Wierman, and Prashant Shenoy. Online Conversion with Switch- ing Costs: Robust and Learning-augmented Algorithms. In Proc. of the 2024 SIGMETRICS/Performance Joint International Conference...

  27. [35]

    Chasing Convex Functions with Long-term Constraints

    Adam Lechowicz, Nicolas Christianson, Bo Sun, No- man Bashir, Mohammad Hajiesmaili, Adam Wierman, and Prashant Shenoy. Chasing Convex Functions with Long-term Constraints. In Proceedings of the 41st Inter- national Conference on Machine Learning, ICML’24. JMLR, 2024

  28. [36]

    J. K. Lenstra and A. H. G. Rinnooy Kan. Complexity of scheduling under precedence constraints. Opera- tions Research, 26(1):22–35, 1978. ISSN 0030364X, 15265463. URL http://www.jstor.org/stable/ 169889

  29. [37]

    Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service

    Baolin Li, Siddharth Samsi, Vijay Gadepally, and De- vesh Tiwari. Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Anal- ysis, SC ’23, New ...

  30. [38]

    Energy-aware scheduling for spark job based on deep reinforcement learning in cloud

    Hongjian Li, Liang Lu, Wenhu Shi, Gangfan Tan, and Hao Luo. Energy-aware scheduling for spark job based on deep reinforcement learning in cloud. Computing, 105(8):1717–1743, March 2023. ISSN 1436-5057. doi: 10.1007/s00607-023-01171-z. URL http://dx.doi. org/10.1007/s00607-023-01171-z

  31. [39]

    Scheduling to minimize total weighted comple- tion time via time-indexed linear programming relax- ations

    Shi Li. Scheduling to minimize total weighted comple- tion time via time-indexed linear programming relax- ations. In 2017 IEEE 58th Annual Symposium on Foun- dations of Computer Science (FOCS) , page 283–294. IEEE, October 2017. doi: 10.1109/focs.2017.34. URL http://dx.doi.or...

  32. [40]

    Cost efficient job scheduling scheme for large scale data center

    Xinran Li and Zhaohao Ding. Cost efficient job scheduling scheme for large scale data center. In 2023 IEEE/IAS Industrial and Commercial Power Sys- tem Asia (I&CPS Asia), pages 2267–2272, 2023. doi: 10.1109/ICPSAsia58343.2023.10294452

  33. [41]

    Data-efficient fine-tuning for llm-based recommendation

    Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. Data-efficient fine-tuning for llm-based recommendation. In Pro- ceedings of the 47th International ACM SIGIR Confer- ence on Research and Development in Information Re- trieval, SIGIR ’24,...

  34. [42]

    When moe meets llms: Parameter efficient fine-tuning for multi- task medical applications

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets llms: Parameter efficient fine-tuning for multi- task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Info...

  35. [43]

    Online job scheduling scheme for low-carbon data center operation: An information and energy nexus perspective

    Wenyu Liu, Yuejun Yan, Yimeng Sun, Hongju Mao, Ming Cheng, Peng Wang, and Zhaohao Ding. Online job scheduling scheme for low-carbon data center operation: An information and energy nexus perspective. Applied Energy, 338:120918, 2023

  36. [44]

    ISBN 9798400704314

    Association for Computing Machinery. ISBN 9798400704314. doi: 10.1145/3626772.3657807. URL https://doi.org/10.1145/3626772.3657807

  37. [45]

    Optimal Algorithms for k-Search with Applica- tion in Option Pricing

    Julian Lorenz, Konstantinos Panagiotou, and Angelika Steger. Optimal Algorithms for k-Search with Applica- tion in Option Pricing. Algorithmica, 55(2):311–328, August 2008

  38. [46]

    Competitive Caching with Machine Learned Advice

    Thodoris Lykouris and Sergei Vassilvtiskii. Competitive Caching with Machine Learned Advice. In Jennifer Dy and Andreas Krause, editors, Proc. of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proc. of Machine Learning Research, pages 3296–3305. PMLR, 10...

  39. [47]

    Under- standing llms: A comprehensive overview from training to inference, 2024

    Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, Yi Pan, Shaochen Xu, Zihao Wu, Zhengliang Liu, Xin Zhang, Shu Zhang, Xintao Hu, Tuo Zhang, Ning Qiang, Tianming Liu, and Bao Ge. Under- standing llms: A...

  40. [48]

    Learning Scheduling Algorithms for Data Processing Clusters

    Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh. Learning Scheduling Algorithms for Data Processing Clusters. In Proceedings of the ACM Special Interest Group on Data Communication, SIGCOMM ’19, page 270–288, New York, NY , USA,...

  41. [49]

    Torgraphina: A scheduler for data pro- cessing during high-frequency job arrival using upside down reinforcement learning

    Yamini Mathur. Torgraphina: A scheduler for data pro- cessing during high-frequency job arrival using upside down reinforcement learning. Master’s thesis, Iowa State University, 2023

  42. [50]

    Schedul- ing precedence-constrained jobs on related machines with communication delay, 2020

    Biswaroop Maiti, Rajmohan Rajaraman, David Stalfa, Zoya Svitkina, and Aravindan Vijayaraghavan. Schedul- ing precedence-constrained jobs on related machines with communication delay, 2020. URL https:// arxiv.org/abs/2004.10776

  43. [51]

    On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines

    Marius Mosbach, Maksym Andriushchenko, and Diet- rich Klakow. On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations,

  44. [52]

    A genetic algorithm for the flexible job-shop scheduling problem

    Ferdinando Pezzella, Gianluca Morganti, and Giampiero Ciaschetti. A genetic algorithm for the flexible job-shop scheduling problem. Computers & operations research, 35(10):3202–3212, 2008

  45. [53]

    Docker: lightweight linux containers for consistent development and deployment

    Dirk Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239):2, 2014. 15

  46. [54]

    Carbon- Aware Computing for Datacenters

    Ana Radovanovic, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, et al. Carbon- Aware Computing for Datacenters. IEEE Transactions on Power Systems, 2022

  47. [55]

    A comparison of priority rules for the job shop scheduling problem under different flow time-and tardiness-related objective functions

    Veronique Sels, Nele Gheysen, and Mario Vanhoucke. A comparison of priority rules for the job shop scheduling problem under different flow time-and tardiness-related objective functions. International Journal of Production Research, 50(15):4255–4270, 2012

  48. [56]

    Cloud-native workflow scheduling using a hybrid priority rule, dy- namic resource allocation, and dynamic task partition

    Jungeun Shin, Diana Arroyo, Asser Tantawi, Chen Wang, Alaa Youssef, and Rakesh Nagi. Cloud-native workflow scheduling using a hybrid priority rule, dy- namic resource allocation, and dynamic task partition. In Proceedings of the 2024 ACM Symposium on Cloud Computing, SoCC ’24,...

  49. [57]

    Im- proving Online Algorithms via ML Predictions

    Manish Purohit, Zoya Svitkina, and Ravi Kumar. Im- proving Online Algorithms via ML Predictions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018

  50. [58]

    Communication-aware scheduling of precedence- constrained tasks on related machines

    Yu Su, Shai Vardi, Xiaoqi Ren, and Adam Wier- man. Communication-aware scheduling of precedence- constrained tasks on related machines. Opera- tions Research Letters , 51(6):709–716, 2023. ISSN 0167-6377. doi: https://doi.org/10.1016/j.orl.2023. 11.001. URL https://www.science...

  51. [59]

    Learning-augmented energy-aware list scheduling for precedence-constrained tasks

    Yu Su, Vivek Anand, Jannie Yu, Jian Tan, and Adam Wierman. Learning-augmented energy-aware list scheduling for precedence-constrained tasks. ACM Trans. Model. Perform. Eval. Comput. Syst., 2024. doi: 10.1145/3680278. URL https://doi.org/10.1145/ 3680278

  52. [60]

    The tpc-h benchmarks, 2018

    TPC-H. The tpc-h benchmarks, 2018. URL https: //www.tpc.org/tpch/

  53. [61]

    We’re increasing our carbon fee as we double down on sustainability

    Brad Smith and Microsoft Corporation. We’re increasing our carbon fee as we double down on sustainability. https://blogs. microsoft.com/on-the-issues/2019/04/15/ were-increasing-our-carbon-fee-as-we-double-down-on-sustainability/ , 2019

  54. [62]

    WattTime

    WattTime. WattTime. https://www.watttime.org, Accessed June 2024

  55. [63]

    Let’s Wait Awhile: How Temporal Workload Shifting Can Re- duce Carbon Emissions in the Cloud

    Philipp Wiesner, Ilja Behnke, Dominik Scheinert, Ko- rdian Gontarska, and Lauritz Thamsen. Let’s Wait Awhile: How Temporal Workload Shifting Can Re- duce Carbon Emissions in the Cloud. In Proceed- ings of the 22nd International Middleware Conference, Middleware ’21, page 260–2...

  56. [64]

    Sus- tainable AI: Environmental Implications, Challenges and Opportunities

    Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sus- tainable AI: Environmental Implications, Challenges and Opportunities. Proceedings of Machine Learning and Systems (MLSys), ...

  57. [65]

    Will we run out of data? limits of llm scaling based on human- generated data, 2024

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Be- siroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human- generated data, 2024. URL https://arxiv.org/abs/ 2211.04325

  58. [66]

    Franklin, Scott Shenker, and Ion Stoica

    Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. Resilient distributed datasets: a fault-tolerant abstraction for in- memory cluster computing. In Proceedings of the 9th USENIX Confere...

  59. [67]

    Xin, Patrick Wendell, Tatha- gata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J

    Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tatha- gata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. Apache spark: a unified engine for big data process...

  60. [68]

    ISBN 9781450385343

    Association for Computing Machinery. ISBN 9781450385343. doi: 10.1145/3464298.3493399. URL https://doi.org/10.1145/3464298.3493399

  61. [69]

    green window

    Yunhong Zhou, Deeparnab Chakrabarty, and Rajan Lukose. Budget Constrained Bidding in Keyword Auc- tions and Online Knapsack Problems. In Lecture Notes in Computer Science, pages 566–576. Springer Berlin Heidelberg, 2008. 17 Appendix A Evaluation Supplements In this section, we...

  62. [70]

    Adaptive DAG Tasks Scheduling with Deep Reinforce- ment Learning, page 477–490

    Qing Wu, Zhiwei Wu, Yuehui Zhuang, and Yuxia Cheng. Adaptive DAG Tasks Scheduling with Deep Reinforce- ment Learning, page 477–490. Springer International Publishing, 2018. ISBN 9783030050542. doi: 10.1007/ 978-3-030-05054-2_37. URL http://dx.doi.org/ 10.1007/978-3-030-05054-2_37

  63. [74]

    Learning to optimize dag scheduling in heterogeneous environment

    Yunfan Zhou, Xijun Li, Jinhong Luo, Mingxuan Yuan, Jia Zeng, and Jianguo Yao. Learning to optimize dag scheduling in heterogeneous environment. In 2022 23rd IEEE International Conference on Mobile Data Management (MDM), pages 137–146, 2022. doi: 10.1109/MDM55031.2022.00040

  64. [2016]

    doi: 10.1145/2934664

    ISSN 0001-0782. doi: 10.1145/2934664. URL https://doi.org/10.1145/2934664. 16

  65. [2020]

    URL https://arxiv.org/abs/2004.09682

  66. [2021]

    URL https://openreview.net/forum?id= nzpLWnVAyah

  67. [2024]

    Association for Computing Machinery. 14

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.