Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

EcoServe: Designing Carbon-Aware AI Inference Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EcoServe argues that LLM inference carbon can be cut nearly in half by co-designing capacity planning, resource allocation, and scheduling around both embodied and operational emissions, rather than optimizing energy or cost alone.

desk verdict Solid carbon-aware LLM serving framework with real CPU reuse work; the 47% headline is model-predicted from the same objective it optimizes, so the quantitative claim needs validation before it is cited as fact. read the letter →

arxiv 2502.05043 v2 pith:ASXSVAGJ submitted 2025-02-07 cs.DC

Yueying Li , Zhanqiu Hu , Esha Choukse , Rodrigo Fonseca , G. Edward Suh , Udit Gupta This is my paper · ORCID
classification cs.DC
keywords carbon-awareinferenceembodiedcarbonoperationalLLMservingresourceprovisioningILPschedulingheterogeneoushardwareofflinebatch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EcoServe argues that minimizing LLM inference's carbon footprint requires co-optimizing capacity planning, resource allocation, and runtime scheduling, not just cutting energy or cost. It grounds this in three production-observed facts: GPUs dominate operational carbon, host systems (CPU, memory, storage) dominate embodied carbon, offline batch inference can consume up to 55 percent of serving capacity, and hardware/workload heterogeneity is large enough to exploit. The paper proposes the 4R principles—Reduce, Reuse, Rightsize, Recycle—implemented in a cross-stack integer linear program that picks GPU types, GPU counts, CPU cores, memory, and workload placement under TTFT/TPOT SLOs. Against performance-, energy-, and cost-optimized baselines, EcoServe reports up to 47 percent carbon savings (1.4–2.2x total carbon reduction) with performance within about 3 percent of the performance-optimized design.

What carries the argument

The load-bearing object is EcoServe's cross-stack integer linear program (ILP), which jointly decides GPU assignment, GPU counts, CPU cores, and memory for sliced prompt/decode workloads under TTFT and TPOT SLOs, minimizing a weighted sum of hardware cost and carbon. Around it, the 4R principles provide the search space: Reuse shifts offline decode work to underutilized CPUs, Rightsize matches heterogeneous GPUs to workload phases, Reduce trims DRAM and SSD to formulaic minimums (e.g., $C_{SSD} = 1.2 C_{GPU}$), and Recycle sets asymmetric lifetimes for host versus GPU. The carbon objective uses a fine-grained embodied-carbon model with per-GB memory, per-area PCB, and per-chip SoC constants, plus operational power measured from hardware power counters and grid carbon intensity. This machinery is what lets the paper claim a 47 percent reduction rather than an incremental efficiency gain.

What would settle it

Build a cluster with known supply-chain emissions (supplier LCAs for each component), run EcoServe's ILP output against a performance-optimized allocation on the same workload traces for a week, and meter wall power continuously; if the measured total-carbon gap is far below 47 percent, the model constants are optimistic. A cheaper check: re-run the ILP with DRAM embodied carbon doubled and halved, and see whether the reported savings range survives.

Watch

Extended reading notes

Core claim

The central claim is that the carbon cost of serving LLMs can be substantially reduced without violating latency SLOs by treating embodied and operational emissions as a single co-design problem. Three measurements motivate the design: GPUs are the largest operational-carbon source while host processors dominate embodied carbon; offline inference is a large, schedulable fraction of production demand; and the optimal GPU varies with prompt length, decode phase, and batch size. EcoServe operationalizes this through four principles: Reuse idle host CPUs for offline decoding, Rightsize GPU provisioning across heterogeneous accelerators, Reduce host memory and storage to what the workload actually needs, and Recycle by extending host lifetimes (to roughly 9 years) while upgrading GPUs more often (3 years). The paper reports that the combined strategies yield up to 47 percent carbon savings over performance-, energy-, and cost-optimized designs, and 1.4–2.2x total carbon reduction across open-source and production traces, with TPOT/TTFT degradation mostly under 3 percent.

Load-bearing premise

The claimed savings assume the embodied-carbon constants embedded in the optimizer (per GB of DRAM/SSD, per area of PCB, per chip, per watt of cooling/PDN) accurately describe real hardware, and because the evaluation uses the same model the optimizer was tuned on, a systematic error in those constants would change the reported 47 percent directly.

Editorial extensions

If this is right

  • Cloud LLM serving can cut total carbon by nearly half while keeping SLOs, so carbon need not be traded against user-facing latency.
  • Offline and batch inference, a large fraction of real capacity, can be scheduled onto idle host CPUs to avoid buying extra GPUs.
  • Energy-optimal and cost-optimal provisioning are different from carbon-optimal provisioning; choosing hardware by energy or price alone leaves carbon on the table.
  • Extending host lifetimes to about 9 years while refreshing GPUs every 3 years reduces embodied carbon more than uniform refresh cycles.
  • Fine-grained workload slicing lets a heterogeneous fleet outperform a homogeneous one on carbon with equal throughput.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 4R logic applies beyond LLM serving: any workload with a compute-bound phase and a memory-bound phase on underutilized hosts could reuse idle CPU capacity for its memory-bound phase, so the same ILP pattern may transfer to vision or embedding serving.
  • Because the optimizer and evaluation share the same carbon model, the 47 percent number is best read as savings under the paper's embodied-carbon accounting; an independent supply-chain LCA audit of each hardware SKU would be the natural stress test.
  • As grids decarbonize, the paper's own equations imply the optimum shifts further toward embodied-carbon minimization, so the relative benefit of EcoServe should grow as renewable penetration rises.
  • The result suggests a market-level consequence: GPU and host refresh cycles should decouple, and cloud SKUs with lean memory and storage could become standard carbon-efficient offerings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper analyzes carbon emissions of LLM inference systems, presenting three observations based on modeling and production traces from two cloud services: GPUs dominate operational carbon, host systems dominate embodied carbon, offline/batch inference accounts for a significant share of serving capacity, and workloads/hardware exhibit heterogeneity. Based on these observations, the authors design EcoServe, a carbon-aware resource provisioning and scheduling framework with four principles (Reduce, Reuse, Rightsize, Recycle) and a cross-layer ILP formulation. The paper claims carbon savings of up to 47% (also stated as 1.4–2.2× reduction) relative to performance-, energy-, and cost-optimized baselines while meeting SLOs, with individual strategies saving 12–41% depending on scenario. The evaluation combines real CPU inference experiments with simulation-based carbon and latency models for the combined 4R configurations.

Significance. If the quantitative claims held up, this would be a valuable contribution to carbon-aware systems design. The paper's strengths are its fine-grained embodied-carbon model for AI systems, the use of real production traces from two LLM services, a clear ILP formulation, and real-hardware validation of the CPU reuse kernel optimizations. The observation that host systems dominate embodied carbon while GPUs dominate operational carbon is an important systems insight. However, the central quantitative claim of 47% carbon savings is currently supported only by a model-based evaluation in which the optimizer and the evaluator use the same carbon and latency models; no independent end-to-end measurement of the combined 4R configurations is reported, and no code or data are released for reproduction. The Recycle component additionally depends on a confidential reliability model. The paper is therefore better characterized as a promising design framework with preliminary validation than as a demonstrated deployment result.

major comments (4)
  1. [Section 6.1, Figure 15, and Section 4.2.2] The headline 47% carbon saving is computed with the same profiling-based latency and carbon models that serve as the ILP objective; no independent end-to-end carbon or latency measurement is reported for the combined 4R allocations. Since Carbon(s,g,l,Phi_s,M_s) = sum_k gamma_g(t) * Lat_k(...) is minimized by construction, a systematic overestimate of embodied-carbon constants or an optimistic latency model would directly inflate the reported savings, and the SLO-compliance claim inherits the same risk. The paper should add independent validation, or at minimum a sensitivity analysis over the embodied-carbon constants and latency-model errors, before the 47% figure can be treated as a measured result.
  2. [Section 4.1.4, Figure 14, and Section 6.1] The Recycle strategy's 16.8% savings rest on a confidential 7nm CPU aging model whose 'details are omitted due to confidentiality,' so this component of the 4R claim is not independently checkable. The paper should either replace that evidence with public reliability data such as the cited DRAM and SSD studies, or relegate the Recycle contribution to an explicitly qualified sensitivity result rather than including it in the aggregate savings.
  3. [Section 1 and Section 6.1] The headline benefits are stated inconsistently: the introduction reports 'up to 47%' and '1.4–2.2x reduction' (i.e., 29–55% reduction), while Section 6.1 says 'On average EcoServe yields 47% carbon savings' and individual-strategy numbers differ between the two sections (e.g., Reduce 34% in the introduction vs. 12.4–28.6% in Section 6.1; Recycle 41% vs. 16.8%). The paper should define exactly which deployment scenario, carbon intensity, and baseline each number refers to and report a single consistent set of values.
  4. [Section 6.1 and Figure 15] The evaluation reports no error bars, confidence intervals, or scenario distributions for the claimed savings, and the caption's 'iso-throughput' assumption is not defined in the text. Since the central claim is an average/maximum over settings, the paper should present the distribution across workloads, request rates, and carbon intensities and state how the 47% value is aggregated; otherwise it is impossible to assess whether the result is robust or driven by a favorable subset of settings.
minor comments (5)
  1. [Section 6.3] The text contains an unresolved reference 'Figure ??'; the citation should be completed or removed.
  2. [Section 4.2.2] The description of gamma_g(t) as 'both linear with operational power of the workload times the carbon intensity, and with unit-time embodied carbon' is ambiguous; separate the operational and embodied terms clearly in the formulation.
  3. [Section 4.1.3, Equation (1)] The symbols n, d, h_kv, and l in the minimum DRAM capacity equation are not fully defined; clarify units and the derivation so the formula can be reproduced.
  4. [Section 6.1, Figure 15 caption] The 'iso-throughput' assumption appears only in the figure caption; define it in the text and state how throughput parity was enforced across baselines.
  5. [Section 5] The statement that the authors 'sample from Watttime and GreenSKU to get the real-time or average carbon intensity' is at odds with the fixed embodied-carbon constants in Table 1; clarify which components use dynamic carbon-intensity data and which use static life-cycle values.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline 47% carbon saving and SLO compliance are computed with the same profiling-based carbon and latency models that the ILP optimizes, so the central evaluation largely measures the solver's own objective.

  1. self definitional [Section 4.2.2 (ILP formulation); Section 5 (Implementation and Experimental Setup); Section 6.1 (End-to-end Evaluation), Figure 15]
    "We implement the profiling-based performance, energy and carbon models using Python based on those frameworks under various hardware, workload traces and request generators. ... On average EcoServe yields 47% carbon savings."

    The ILP in Section 4.2.2 minimizes Carbon(s,g,l,Phi_s,M_s) = sum_k gamma_g(t) * Lat_k(...) subject to Lat <= SLO. The 47% headline saving in Figure 15 is computed with the same profiling-based carbon and latency models, so it measures the reduction in the exact objective the solver optimizes. Baselines (Perf-opt, Energy-opt, Melange, SplitWise) are scored with the same model. Since no independent end-to-end carbon measurement of the combined 4R configurations is reported, the claimed real-world carbon saving is, within the evaluation, the ILP's objective gap by construction rather than an externally validated result.

  2. self definitional [Section 4.2.2 (ILP constraints); Section 6.1.1 (Performance Analysis)]
    "∀s,j where A_{s,j}=1, Lat_{p/d}(s,j,l,Φ_s,M_s)≤ SLO_{p/d} ... From Figure 15, we demonstrate that the performance of EcoServe variants is close to Perf-opt within 3 percent, except for the reuse case."

    The ILP hard-codes the latency model as a constraint, and the evaluation reports TPOT/TTFT from the same profiling-based latency model. Therefore 'maintaining performance targets and SLOs' is enforced by construction within the simulated evaluation, not verified against real SLO behavior for the co-designed allocations. The real-hardware latency evidence in Section 6.3 covers isolated CPU kernels, not the combined 4R configurations whose SLO compliance is claimed in the headline result.

full rationale

The dominant circularity is the evaluation loop: the carbon and latency models used to score EcoServe in Section 6.1 are the same models that define the ILP objective and constraints in Section 4.2.2. Consequently, the reported up-to-47% carbon saving and the SLO-compliance claim are guaranteed improvements on the optimizer's own surrogate, and any systematic bias in the latency predictor or embodied-carbon constants directly inflates the apparent benefit. This is partial, not total, circularity: the embodied-carbon constants come from external LCAs and tools (Dell R740, TechInsights, ACT, GreenSKU), and Section 6.3 presents real CPU-reuse kernel measurements. The self-citations to ACT, GreenSKU, and Splitwise involve overlapping authors but are externally published tools and therefore are not themselves the circular step. The confidential reliability model in Section 4.1.4 makes the 16.8% Recycle saving non-checkable, but that is an evidence gap rather than a definitional reduction. Overall, the central quantitative claim needs independent end-to-end carbon and latency validation before it can be taken as a measured result.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a carbon model built from prior life-cycle data and on the representativeness of production traces. The 1.2 SSD factor and the lifetime assumptions are ad hoc choices. The confidential reliability model is the most opaque assumption.

free parameters (4)
  • SSD sizing coefficient = 1.2
    Equation (2) sets min SSD capacity as 1.2 times GPU memory; the 1.2 factor is chosen by hand without derivation.
  • Baseline hardware lifetime = 4 years (host and GPU)
    Section 6.5 baseline assumes 4-year upgrades for both; EcoServe uses 9-year host, 3-year GPU. These lifetimes are assumptions, not measured.
  • ILP weight alpha = 1 (default)
    The ILP objective uses alpha=1 unless stated, meaning carbon is the sole objective; results may be sensitive to this choice.
  • Profiled latency model coefficients = not reported
    Lat(s,g,l,phi,m) is fit from profiling runs on vLLM/llama.cpp; no coefficients are given, so this is a fitted model whose accuracy is untested outside the paper.
assumptions (5)
  • domain assumption Embodied carbon per component values (e.g., 0.29 kgCO2e/GB DDR4, 0.048 kgCO2e/cm2 PCB) from Dell R740 LCA and TechInsights apply to the evaluated Azure/Lambda systems.
    Used in Table 1 and Figure 5; if these LCAs are not representative, the host-embodied-dominance observation weakens.
  • domain assumption ACT architectural carbon model correctly estimates processor embodied carbon.
    Used in Section 3.1.1 to estimate CPU/GPU die carbon; no independent validation for the specific chips.
  • domain assumption Average CPU utilization in production is about 6% of a 16-core processor.
    Section 3.2 Observation 4, based on profiling several models on vLLM/SGLang; generalizes to the production traces used for capacity planning.
  • ad hoc to paper The confidential 7nm CPU aging model accurately projects 0.8 years effective age at 20% utilization over 5 years.
    Section 4.1.4: details omitted due to confidentiality, so the Recycle strategy's reliability premise cannot be checked.
  • domain assumption Carbon intensity data from Watttime and GreenSKU represent regional grid carbon.
    Used as gamma_g(t) in the ILP and in Figure 6/13 simulations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EcoServe: Designing Carbon-Aware AI Inference Systems." pith.science (2026). https://pith.science/paper/ASXSVAGJ

@misc{pith2026250205043,
  author       = {Pith},
  title        = {Pith review of: EcoServe: Designing Carbon-Aware AI Inference Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASXSVAGJ}},
  note         = {Machine review of arXiv:2502.05043}
}
read the original abstract

The rapid increase in LLM ubiquity and scale levies unprecedented demands on computing infrastructure. These demands not only incur large compute and memory resources but also significant energy, yielding large operational and embodied carbon emissions. In this work, we present three main observations based on modeling and traces from the production deployment of two Generative AI services in a major cloud service provider. First, while GPUs dominate operational carbon, host processing systems (e.g., CPUs, memory, storage) dominate embodied carbon. Second, offline, batch inference accounts for a significant portion (up to 55\%) of serving capacity. Third, there are different levels of heterogeneity across hardware and workloads for LLM inference. Based on these observations, we design EcoServe, a carbon-aware resource provision and scheduling framework for LLM serving systems. It is based on four principles - Reduce, Reuse, Rightsize, and Recycle (4R). With a cross-stack ILP formulation and design, we demonstrate that EcoServe can lower carbon emissions by up to 47\%, compared to performance, energy, and cost-optimized design points, while maintaining performance targets and SLOs.

Figures

Figures reproduced from arXiv: 2502.05043 by the authors.

Figure 1
Figure 1. (Left) Breakdown of thermal design power pro [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. EcoServe’s carbon modeling framework with more fine-grained embodied carbon estimation on memory, storage, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Trends in bit density (left) and embodied carbon [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (13 more)
Figure 6
Figure 6. Figure 6: Embodied and operational carbon breakdowns of [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: EcoServe system diagram: An optimization framework that minimizes operational and embodied carbon across [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Roofline models of an Intel Sapphire Rapids CPU [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]
Figure 9
Figure 9. Figure 9: CPU offline inference requires carefully balanc [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Breakdown of online and offline demand for [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 12
Figure 12. Figure 12: Relative energy and carbon of prompt and de [PITH_FULL_IMAGE:figures/full_fig_p007_12.png]
Figure 13
Figure 13. Figure 13: Relative carbon savings with various hardware [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 14
Figure 14. Figure 14: Effective age with deployment time. 5 years, CPU aging is only 0.8 years, indicating significant potential for extended use ( [PITH_FULL_IMAGE:figures/full_fig_p008_14.png]
Figure 15
Figure 15. Figure 15: Carbon vs. performance trade-off. Lower-right is better. (Left, Center) We compare EcoServe to performance, en [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]
Figure 16
Figure 16. Figure 16: Carbon savings and reuse configurations sam [PITH_FULL_IMAGE:figures/full_fig_p010_16.png]
Figure 17
Figure 17. Figure 17: Comparison of Bloom-176B (Left) and Llama [PITH_FULL_IMAGE:figures/full_fig_p011_17.png]
Figure 18
Figure 18. Figure 18: Beginning with llama.cpp as a baseline, we op [PITH_FULL_IMAGE:figures/full_fig_p011_18.png]
Figure 21
Figure 21. Figure 21: Trends in annual embodied and operational [PITH_FULL_IMAGE:figures/full_fig_p012_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enabling Spatially Fine-Grained DVFS in Neural Processing Units for Energy-Efficient LLM Serving

    cs.AR 2026-07 conditional novelty 7.0 of 10

    Component-level DVFS on NPUs, with pipeline refactoring and compiler-coordinated voltage/frequency selection, cuts LLM-serving energy by 25.8–35.2% at sub-4% area overhead in simulation.

  2. Routing LLM Inference to the Cleanest Grid in Real Time

    cs.DC 2026-08 conditional novelty 6.0 of 10

    Carbon-aware request routing driven by marginal-emissions signals moved live LLM inference across regions and, under a modeled perfect-foresight replay, could reduce GPU-attributable emissions by 50.9 percent with a 9...

  3. Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulations

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A simulation framework couples an LLM inference simulator with a GPU power model and an energy-grid co-simulator to estimate energy and carbon emissions across deployment configurations.

  4. A Vertical Approach to Designing and Managing Sustainable Heterogeneous Edge Data Centers

    eess.SY 2025-06 reject novelty 3.0 of 10

    The paper presents a vertical integration framework for carbon-aware edge data center design, but all quantitative results are borrowed from prior work and the cross-layer benefit is untested.

Reference graph

Works this paper leans on

90 extracted references · 56 canonical work pages · cited by 4 Pith papers

  1. [1]

    https:// lambdalabs.com/service/gpu-cloud

    GPU Cloud - VMs for Deep Learning | Lambda — lambdalabs.com. https:// lambdalabs.com/service/gpu-cloud. [Accessed 05-05-2024]

    2024

  2. [2]

    Powering intelligence: Analyzing artificial intelligence and data center energy consumption

  3. [3]

    https://news.skhynix.com/hbm2e- opens-the-era-of-ultra-speed-memory-semiconductors/

    SK hynix Newsroom — news.skhynix.com. https://news.skhynix.com/hbm2e- opens-the-era-of-ultra-speed-memory-semiconductors/. [Accessed 05-05-2024]

    2024

  4. [4]

    [Available Online] https://developer.nvidia.com/tensorrt/, 2023

    Nvidia tensorrt. [Available Online] https://developer.nvidia.com/tensorrt/, 2023

    2023

  5. [5]

    Carbon explorer: A holis- tic framework for designing carbon aware datacenters

    Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Kiwan Maeng, Udit Gupta, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. Carbon explorer: A holis- tic framework for designing carbon aware datacenters. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages 118–...

    2023

  6. [6]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve. arXiv preprint arXiv:2403.02310, 2024

    arXiv 2024

  7. [7]

    Deepspeed- inference: Enabling efficient inference of trans- former models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of trans- former models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and An...

    2022

  8. [8]

    Aws recommended gpu instances, 2024

    AWS. Aws recommended gpu instances, 2024. Accessed: 2024-02-26

    2024

Show all 90 references
  1. [9]

    Azure gpu optimized virtual machine sizes, 2024

    Azure. Azure gpu optimized virtual machine sizes, 2024. Accessed: 2024-02-26

    2024

  2. [10]

    Life cycle assessment – dell r740

    Andreas Busa, Malcolm Hegeman, Jeff Vickers, Natalia Duque-Ciceri, and Constantin Hermann. Life cycle assessment – dell r740. Technical report, Thinkstep, 2019. [Available Online:] https://www.delltechnologies.com/asset/en- us/products/servers/technical-support/Full_LCA_Dell_R740.pdf

    2019

  3. [11]

    Flashdecoding: Accelerating llm inference by paralleling token generation

    Center for Research on Foundation Models (CRFM). Flashdecoding: Accelerating llm inference by paralleling token generation. https://crfm.stanford.edu/2023/10/ 12/flashdecoding.html, October 2023

    2023

  4. [12]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1–113, 2023

    2023

  5. [13]

    Sharegpt: A dataset of multi-turn chat interactions with large language models

    ShareGPT Contributors. Sharegpt: A dataset of multi-turn chat interactions with large language models. https://sharegpt.com/, 2024. Accessed: 2024-11-18

    2024

  6. [14]

    Clipper: A{Low-Latency} online prediction serving system

    Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. Clipper: A{Low-Latency} online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 613–627, 2017

    2017

  7. [15]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  8. [16]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022

    2022

  9. [17]

    Hanebutte, Rahul Khanna, and Chris- tian Le

    Howard David, Eugene Gorbatov, Ulf R. Hanebutte, Rahul Khanna, and Chris- tian Le. Rapl: memory power estimation and capping. In Proceedings of the 16th ACM/IEEE International Symposium on Low Power Electronics and Design , ISLPED ’10, page 189–194, New York, NY, USA, 2010. As...

    2010

  10. [18]

    Openblas: An optimized blas library

    OpenBLAS Developers. Openblas: An optimized blas library. https://www. openblas.net/, 2024. Accessed: 2024-11-18

    2024

  11. [19]

    Cvxpy: A python-embedded modeling language for convex optimization, 2024

    Steven Diamond, Stephen Boyd, et al. Cvxpy: A python-embedded modeling language for convex optimization, 2024. Accessed: 2024-11-19

    2024

  12. [20]

    Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models

    Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Li, and Yiran Chen. Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models. ArXiv, abs/2310.18859, 2023

    arXiv

  13. [21]

    Focal: A first-order carbon model to assess processor sustain- ability

    Lieven Eeckhout. Focal: A first-order carbon model to assess processor sustain- ability. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages 401–415, 2024

    2024

  14. [22]

    72-hour hourly map

    Electricity Maps. 72-hour hourly map. https://app.electricitymaps.com/map/72h/ hourly. Accessed: 2025-02-09

    2025

  15. [23]

    Llmcarbon: Modeling the end-to-end carbon footprint of large language models

    Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Parteek Sharma, Fan Chen, and Lei Jiang. Llmcarbon: Modeling the end-to-end carbon footprint of large language models. arXiv preprint arXiv:2309.14393, 2023

    2023 arXiv

  16. [24]

    Mobius: Fine tuning large-scale models on commodity gpu servers

    Yangyang Feng, Minhui Xie, Zijie Tian, Shuo Wang, Youyou Lu, and Jiwu Shu. Mobius: Fine tuning large-scale models on commodity gpu servers. International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS, 2:489–501, 1 2023

    2023

  17. [25]

    Garcia Bardon, P

    M. Garcia Bardon, P. Wuytens, L.-Å. Ragnarsson, G. Mirabelli, D. Jang, G. Willems, A. Mallik, A. Spessot, J. Ryckaert, and B. Parvais. Dtco including sustainability: Power-performance-area-cost-environmental score (ppace) analysis for logic technologies. In 2020 IEEE Internati...

    2020

  18. [26]

    Llama.cpp: Inference of llama models in pure c/c++

    Georgi Gerganov. Llama.cpp: Inference of llama models in pure c/c++. https: //github.com/ggerganov/llama.cpp, 2024. Accessed: 2024-11-18. 13

    2024

  19. [27]

    Gemma.cpp: Efficient inference for large language models

    Google. Gemma.cpp: Efficient inference for large language models. https://github. com/google/gemma.cpp, 2024. Accessed: 2024-11-18

    2024

  20. [28]

    Google sustainability report, 2024

    Google. Google sustainability report, 2024. Accessed: 2024-02-21

    2024

  21. [29]

    Mélange: Cost efficient large language model serving by exploiting gpu heterogeneity, 2024

    Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. Mélange: Cost efficient large language model serving by exploiting gpu heterogeneity, 2024

    2024

  22. [30]

    Serving{DNNs} like clockwork: Performance predictability from the bottom up

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving{DNNs} like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) , pages 443–...

    2020

  23. [31]

    Lee, David Brooks, and Carole-Jean Wu

    Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. Act: Designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual International Symposium on Computer Architecture ...

    2022

  24. [32]

    Chasing carbon: The elusive environmental footprint of computing

    Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu- Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing carbon: The elusive environmental footprint of computing. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages ...

    2021

  25. [33]

    FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines

    Jiaao He and Jidong Zhai. FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines. arXiv preprint arXiv:2403.11421, 2024

    2024 arXiv

  26. [34]

    Flashdecoding++: Faster large language model inference on gpus

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus. arXiv preprint arXiv:2311.01282, 2023

    2023 arXiv

  27. [35]

    Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee

    Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference, June 2023. arXiv:2303.06182 [cs]

    2023 arXiv

  28. [36]

    Jones, and Peipei Zhou

    Shixin Ji, Zhuoping Yang, Xingzhen Chen, Jingtong Hu, Yiyu Shi, Alex K. Jones, and Peipei Zhou. Towards Data-center Level Carbon Modeling and Optimization for Deep Learning Inference, March 2024. arXiv:2403.04976 [cs]

    2024 arXiv

  29. [37]

    Neo: Saving gpu memory crisis with cpu offloading for online llm inference, 2024

    Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. Neo: Saving gpu memory crisis with cpu offloading for online llm inference, 2024

    2024

  30. [38]

    Scotten W. Jones. Modeling 300mm wafer fab carbon emissions. In 2023 Interna- tional Electron Devices Meeting (IEDM) , pages 1–4, 2023

    2023

  31. [39]

    Profiling a warehouse-scale computer

    Svilen Kanev, Juan Pablo Darago, Kim Hazelwood, Parthasarathy Ranganathan, Tipp Moseley, Gu-Yeon Wei, and David Brooks. Profiling a warehouse-scale computer. In Proceedings of the 42nd annual international symposium on computer architecture, pages 158–169, 2015

    2015

  32. [40]

    Backblaze hard drive stats for q2 2021, 2021

    Andy Klein. Backblaze hard drive stats for q2 2021, 2021. Analysis of drive failure rates across 181,464 hard drives used in production

    2021

  33. [41]

    Serving MoE Models on Resource-constrained Edge Devices via Dynamic Expert Swapping, August 2023

    Rui Kong, Yuanchun Li, Qingtian Feng, Weijun Wang, Linghe Kong, and Yunxin Liu. Serving MoE Models on Resource-constrained Edge Devices via Dynamic Expert Swapping, August 2023. arXiv:2308.15030 [cs]

    2023 arXiv

  34. [42]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023

    2023

  35. [43]

    Amp: Automatically finding model parallel strategies with heterogeneity awareness

    Dacheng Li, Hongyi Wang, Eric Xing, and Hao Zhang. Amp: Automatically finding model parallel strategies with heterogeneity awareness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, p...

    2022

  36. [44]

    In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23) , pages 663–679, 2023

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al.{AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Des...

    2023

  37. [45]

    Gonzalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. AlpaServe: Statistical multiplexing with model parallelism for deep learning serv- ing. In 17th USENIX Symposium on Operating ...

    2023

  38. [46]

    New insight into the aging induced retention time degraded of advanced dram technology

    Yong Liu, Pengpeng Ren, Da Wang, Longda Zhou, Zhigang Ji, Junhua Liu, Run- sheng Wang, and Ru Huang. New insight into the aging induced retention time degraded of advanced dram technology. In 2022 IEEE International Reliability Physics Symposium (IRPS), pages 6B–2. IEEE, 2022

    2022

  39. [47]

    Cachegen: Kv cache compression and streaming for fast large language model serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: Kv cache compression and streaming for fast large language model ser...

    2024

  40. [48]

    Longbench: A bilingual long-context benchmark for large language models

    Yushi Liu, Qingxiu Xie, Yufei Wang, Yujie Jin, Huanxuan Zhang, Hai Wang, Weizhu Yin, Binxing Yang, and Jie Tang. Longbench: A bilingual long-context benchmark for large language models. https://github.com/THUDM/LongBench, 2023

    2023

  41. [49]

    Deja vu: Contextual sparsity for efficient LLMs at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, An- shumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient LLMs at inference time. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara ...

    2023

  42. [50]

    Power hungry processing: Watts driving the cost of ai deployment? In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 85–99, 2024

    Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of ai deployment? In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 85–99, 2024

    2024

  43. [51]

    Helix: Distributed serving of large language models via max- flow on heterogeneous gpus

    Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. Helix: Distributed serving of large language models via max- flow on heterogeneous gpus. arXiv preprint arXiv:2406.01566, 2024

    2024 arXiv

  44. [52]

    A large-scale study of flash memory failures in the field

    Justin Meza, Qiang Wu, Sanjev Kumar, and Onur Mutlu. A large-scale study of flash memory failures in the field. In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems , SIGMETRICS ’15, page 177–190, New York, NY, USA,...

    2015

  45. [53]

    Microsoft sustainability report, 2024

    Microsoft. Microsoft sustainability report, 2024. Accessed: 2024-02-21

    2024

  46. [54]

    Azure public dataset

    Microsoft Azure. Azure public dataset. https://github.com/Azure/ AzurePublicDataset, 2024. Microsoft Azure Public Dataset Repository

    2024

  47. [55]

    Roofline Performance Model - NERSC Documentation — docs.nersc.gov

    NERSC. Roofline Performance Model - NERSC Documentation — docs.nersc.gov. https://docs.nersc.gov/tools/performance/roofline/. [Accessed 08-05-2024]

    2024

  48. [56]

    Nvml api reference, 2024

    Nvidia. Nvml api reference, 2024. Accessed: 2024-09-21

    2024

  49. [57]

    Deep learning performance guide: Matrix multiplication (gemm)

    NVIDIA Corporation. Deep learning performance guide: Matrix multiplication (gemm). https://docs.nvidia.com/deeplearning/performance/dl-performance- matrix-multiplication/index.html, 2024. NVIDIA Documentation

    2024

  50. [58]

    onednn: Deep neural network library

    oneAPI Deep Neural Network Library (oneDNN) Developers. onednn: Deep neural network library. https://github.com/oneapi-src/oneDNN, 2024. Accessed: 2024-11-18

    2024

  51. [59]

    Instinfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference

    Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. Instinfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference. arXiv preprint arXiv:2409.04992, 2024

    2024 arXiv

  52. [60]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. arXiv preprint arXiv:2311.18677, 2023

    2023 arXiv

  53. [61]

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. Power-aware deep learning model serving with mu-Serve. In 2024 USENIX Annual Technical Conference (USENIX ATC 24) , pages 75...

    2024

  54. [62]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD ’2...

    2020

  55. [63]

    Xnnpack: High-performance neural network inference frame- work

    Google Research. Xnnpack: High-performance neural network inference frame- work. https://github.com/google/XNNPACK, 2020

    2020

  56. [64]

    {INFaaS}: Automated model-less inference serving

    Francisco Romero, Qian Li, Neeraja J Yadwadkar, and Christos Kozyrakis. {INFaaS}: Automated model-less inference serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 397–411, 2021

    2021

  57. [65]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  58. [66]

    Life-cycle emissions of ai hardware: A cradle-to-grave approach and generational trends

    Ian Schneider, Hui Xu, Stephan Benecke, David Patterson, Keguo Huang, Parthasarathy Ranganathan, and Cooper Elsworth. Life-cycle emissions of ai hardware: A cradle-to-grave approach and generational trends. arXiv preprint arXiv:2502.01671, 2025

    2025 arXiv

  59. [67]

    Data center lifecycle co2e calculator, 2024

    Schneider Electric. Data center lifecycle co2e calculator, 2024. Accessed: 2024- 11-19

    2024

  60. [68]

    Bianca Schroeder and Garth A. Gibson. Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? In 5th USENIX Conference on File and Storage Technologies (FAST 07) , San Jose, CA, February 2007. USENIX Association

    2007

  61. [69]

    Flash reliability in production: The expected and the unexpected

    Bianca Schroeder, Raghav Lagisetty, and Arif Merchant. Flash reliability in production: The expected and the unexpected. In 14th USENIX Conference on File and Storage Technologies (FAST 16), pages 67–80, 2016

    2016

  62. [70]

    FlexGen: High- Throughput Generative Inference of Large Language Models with a Single GPU

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High- Throughput Generative Inference of Large Language Models with a Single GPU. In International Conference on Machine Learning, pag...

    2023

  63. [71]

    Lifetime memory reliability data from the field, 2017

    Taniya Siddiqua et al. Lifetime memory reliability data from the field, 2017. Accessed: 2025-02-19

    2017

  64. [72]

    Powerinfer: Fast large lan- guage model serving with a consumer-grade gpu

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large lan- guage model serving with a consumer-grade gpu. arXiv preprint arXiv:2312.12456, 2023. 14

    2023 arXiv

  65. [73]

    Dynamollm: Designing llm inference clusters for performance and energy effi- ciency

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. Dynamollm: Designing llm inference clusters for performance and energy effi- ciency. arXiv preprint arXiv:2408.00741, 2024

    2024

  66. [74]

    Summarizing cpu and gpu design trends with product data

    Yifan Sun, Nicolas Bohm Agostini, Shi Dong, and David Kaeli. Summarizing cpu and gpu design trends with product data. arXiv preprint arXiv:1911.11313, 2019

    1911 arXiv

  67. [75]

    Swamit Tannu and Prashant J. Nair. The dirty secret of ssds: Embodied carbon. SIGENERGY Energy Inform. Rev., 3(3):4–9, oct 2023

    2023

  68. [76]

    Accelerating self-attentions for llm serving with flashinfer, 2024

    FlashInfer team. Accelerating self-attentions for llm serving with flashinfer, 2024. Accessed: 2024-02-24

    2024

  69. [77]

    Micron 1𝛼 dram technology, Nov 2024

    TechInsights. Micron 1𝛼 dram technology, Nov 2024. Accessed: [insert access date here]

    2024

  70. [78]

    Mlperf power: Benchmarking the energy efficiency of machine learning systems from microwatts to megawatts for sustainable ai.arXiv preprint arXiv:2410.12032, 2024

    Arya Tschand, Arun Tejusve Raghunath Rajan, Sachin Idgunji, Anirban Ghosh, Jeremy Holleman, Csaba Kiraly, Pawan Ambalkar, Ritika Borkar, Ramesh Chukka, Trevor Cockrell, et al. Mlperf power: Benchmarking the energy efficiency of machine learning systems from microwatts to megaw...

    arXiv

  71. [79]

    vllm v0.6.0: 2.7x throughput improvement and 5x latency reduction, September 2024

    vLLM Team. vllm v0.6.0: 2.7x throughput improvement and 5x latency reduction, September 2024. Accessed: 2024-11-21

    2024

  72. [80]

    Designing cloud servers for lower carbon

    Jaylen Wang, Daniel S Berger, Fiodar Kazhamiaka, Celine Irvene, Chaojie Zhang, Esha Choukse, Kali Frost, Rodrigo Fonseca, Brijesh Warrier, Chetan Bansal, et al. Designing cloud servers for lower carbon. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architect...

    2024

  73. [81]

    Coverage map

    WattTime. Coverage map. https://watttime.org/docs-dev/coverage-map/. Ac- cessed: 2025-02-09

    2025

  74. [82]

    Roofline: an insightful visual performance model for multicore architectures

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009

    2009

  75. [83]

    TwinPilots: A New Computing Paradigm for GPU-CPU Parallel LLM Inference

    Chengye Yu, Tianyu Wang, Zili Shao, Linjie Zhu, Xu Zhou, and Song Jiang. TwinPilots: A New Computing Paradigm for GPU-CPU Parallel LLM Inference. In Proceedings of the 17th ACM International Systems and Storage Conference , pages 91–103, 2024

    2024

  76. [84]

    Decentralized training of foundation models in heterogeneous environments, 2022

    Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments, 2022

    2022

  77. [85]

    Llm inference unveiled: Survey and roofline model insights

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, et al. Llm inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363, 2024

    2024 arXiv

  78. [86]

    In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) , pages 787–808, 2023

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica.{SHEPHERD}: Serving{DNNs} in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) , pages 787–808, 2023

    2023

  79. [87]

    H2o: Heavy- hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems , 36, 2024

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy- hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Syste...

    2024

  80. [88]

    HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices

    Xuanlei Zhao, Bin Jia, Haotian Zhou, Ziming Liu, Shenggan Cheng, and Yang You. HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices. arXiv preprint arXiv:2403.01164, 2024

    2024 arXiv

  81. [89]

    Efficiently programming large language models using sglang

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Efficiently programming large language models using sglang. arXiv preprint arXiv:2312.07104, 2023

    2023 arXiv

  82. [90]

    Distserve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput- optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024. 15

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.