Towards Sustainable Large Language Model Serving

Beihao Zhou; Sihang Liu; Sophia Nguyen; Yi Ding

REVIEW 3 major objections 5 minor 40 references

Towards Sustainable Large Language Model Serving

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that for compute-light LLM serving workloads, an older, slower GPU can consume less energy than a newer flagship GPU, and that once manufacturing emissions are included, older hardware can yield lower total carbon in…

desk verdict A genuinely useful GPU-energy measurement study whose 'total carbon' headline overreaches the GPU-only power data; worth a serious referee if the accounting boundary is fixed. read the letter →

arxiv 2501.01990 v1 pith:NOMIZW2G submitted 2024-12-31 cs.LG cs.DC

Sophia Nguyen , Beihao Zhou , Yi Ding , Sihang Liu This is my paper · ORCID

classification cs.LGcs.DC

keywords sustainabilitycarbonemissionslargelanguagemodelservingGPUembodiedoperationalintensityenergyefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the carbon footprint of serving a large language model depends less on how new the GPU is than on the match between workload, hardware, and grid. By profiling LLaMA with 1B, 3B, and 7B parameters on an RTX6000 Ada and an older T4, it finds that at batch size 1 the older T4 uses less energy per prompt (28% less for the 1B model, 20% less for the 7B model), and that in the memory-bound decode phase T4 uses 27.1% less per-token energy at batch size 1. Combining operational carbon ($E \cdot CI$) and embodied carbon amortized over a 5-year lifetime, the paper argues that in low-carbon grids older GPUs like the T4 can have lower total carbon emissions than a newer, faster GPU. The reason a sympathetic reader should care: if this is right, hardware generation is not a reliable proxy for sustainability, and decisions about which GPU serves which request should be made per configuration and per grid.

What carries the argument

The load-bearing mechanism is a two-term carbon model rather than a single energy number. Operational carbon is energy times grid carbon intensity ($C_{\mathrm{op}} = E\cdot CI$); embodied carbon is the manufacturing carbon of the GPU, estimated from chip area and memory via an architectural carbon model, discounted by the fraction of the GPU's lifetime a prompt occupies ($C_{\mathrm{em}} = (t/LT)\cdot C_{\mathrm{em,GPU}}$). Because T4 draws up to only 70 W against RTX6000 Ada's 300 W TDP, a light batch-size-1 load runs almost as fast on T4 while using less power, and its smaller chip area and memory give it roughly 2.6x lower embodied carbon (10.3 kg vs 26.6 kg). The argument is carried by this power-versus-time tradeoff and by how embodied cost is amortized over an assumed 5-year lifetime.

What would settle it

Run the same LLaMA workloads at batch size 1 on both GPUs while metering total system power at the wall (including CPU, DRAM, and cooling); if T4's per-prompt total energy is not below RTX6000 Ada's, the paper's central carbon conclusion fails.

Watch

Extended reading notes

Core claim

The central claim is that the older and slower T4 has higher energy efficiency than the newer and faster RTX6000 Ada when processing less compute-intensive requests (e.g., batch size 1), and that after including embodied carbon, strategically using older GPUs like T4 could effectively reduce total carbon emissions by amortizing the embodied carbon emissions of GPUs over time. The paper measures latency and GPU-only power (sampled every 100 ms with NVML) for LLaMA 1B, 3B, and 7B on both GPUs, splits serving into compute-bound prefill and memory-bound decode phases, and models total per-prompt carbon as $C_{\mathrm{prompt}} = E_{\mathrm{prompt}}\cdot CI + (t_{\mathrm{prompt}}/LT)\cdot C_{\mathrm{em}}$ for three grids (QC, CISO, and PACE). It reports that throughput-maximizing batch sizes are not energy-minimizing ones, and that embodied carbon can be up to 30.7% of total per-prompt carbon for RTX6000 Ada in a low-carbon grid, making older hardware attractive in such regions.

Load-bearing premise

The whole carbon ranking rests on measuring only GPU power; if cooling, CPU, memory, and power distribution overhead are counted, the older T4's longer execution time could turn its energy advantage into a disadvantage.

Editorial extensions

If this is right

For latency-flexible workloads in low-carbon grids, datacenters can cut total carbon by routing some requests to older GPUs rather than always buying the newest generation.
The batch size that maximizes throughput differs from the batch size that minimizes energy or carbon, so throughput-centric scheduling should not be assumed carbon-optimal.
Extending GPU lifetime from 4 to 8 years shrinks the embodied share of per-token carbon, most visibly in low-carbon grids where embodied carbon already dominates.
Carbon per token, not energy per token, should be the optimization target, since energy-minimal configurations are not always carbon-minimal once embodied emissions are included.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test: measure whole-system power (CPU, DRAM, cooling, PUE) for the same batch-size-1 prompts; if T4's longer runtime lifts system energy above RTX6000 Ada's, the carbon ranking could reverse, because this paper counts only GPU power.
The prefill/decode split suggests a heterogeneous scheduling policy: run compute-heavy prefill on new GPUs and memory-bound decode on older ones, extending the paper's phase-level findings into a concrete system design.
If embodied carbon were attributed to the whole server or the datacenter build rather than the GPU alone, the absolute numbers would change but the relative advantage of smaller, older chips would likely persist; this is a sensitivity check the paper does not run.
For interactive serving where latency targets are strict, T4's 1.1-2.2x slowness at batch size 1 may rule it out despite the carbon benefit, so the result applies mainly to latency-flexible or batch workloads.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a solid, honestly-scoped measurement study of LLaMA serving energy on two GPU generations, with a carbon overlay that is mostly arithmetic. The genuinely new thing is the per-token, per-phase energy/carbon matrix across model size, batch size, and GPU generation, and the finding that the older T4 can beat the RTX6000 Ada on GPU energy at batch size 1. That is a real data point and a useful counterexample to the assumption that newer hardware is always the greener choice.

What the paper does well: the measurement methodology is clear (NVML sampling at 100ms, median latency, average power), the prefill/decode split is informative, and Equations 2–4 are correct given their stated definitions. The embodied carbon treatment follows ACT, and they sweep the GPU lifetime assumption, which is the main free parameter. The three-region carbon intensity comparison is a nice addition and is consistent with how operational carbon should scale.

The load-bearing problem is the boundary of the carbon accounting. The paper admits in Section 2.1 that it only measures GPU power, but Equation 4 and the abstract call the result "total carbon emission." At batch size 1, T4's GPU-energy advantage is 28% for 1B and 20% for 7B, but T4 is 1.1x and 2.2x slower, respectively. If the surrounding node—CPU, DRAM, cooling, PSU losses, PUE—draws even a modest fraction of the GPU's power, the longer wall-clock time erodes or reverses T4's advantage. The stress-test breakeven for the 7B case (roughly one sixth of Ada's average GPU power as non-GPU overhead) is low enough that real serving nodes likely cross it. This does not invalidate the energy measurements, but it does invalidate the "total carbon" claim as stated. The fix is either system-level power measurements or a clearly-labeled "GPU-attributable carbon" claim with sensitivity analysis.

Other soft spots are minor to moderate: there are no error bars or trial counts, the exact LLaMA checkpoints and serving framework are not specified, and Section 3.3 says it uses QC's CI while Figures 5 and 6 say CISO. That last one is a sloppy inconsistency, not a substantive flaw, but it should be fixed before publication.

If you work on carbon-aware scheduling or hardware reuse, this is worth reading and citing as a measurement data point, not as a definitive answer on total system carbon. It deserves a serious referee: the core question is important, the measurements are plausible and reproducible in principle, and the paper is honest about its scope even where the framing overreaches. Send it to review, but expect the authors to tighten the boundary claim and add the missing experimental details.

Referee Report

3 major / 5 minor

Summary. This paper presents a measurement study of LLaMA 1B/3B/7B inference on an RTX6000 Ada and an older T4 GPU across batch sizes from 1 to 64. It reports latency and GPU-only energy consumption, then combines these measurements with grid-specific carbon intensities (QC, CISO, PACE) and ACT-style embodied-carbon estimates to compute per-prompt and per-token operational, embodied, and 'total' carbon emissions. The central empirical finding is that the older T4 has lower GPU energy per prompt at batch size 1, and the paper argues that after amortizing embodied carbon, strategically using older GPUs like the T4 could reduce total carbon emissions, especially in low-carbon-intensity regions. The paper concludes with future directions on hardware reuse, carbon-aware scheduling, and sustainable LLM infrastructure.

Significance. If the headline finding survives system-level accounting, it is practically important: hardware generation would not be a reliable proxy for carbon efficiency, and the greenest GPU would depend on batch size, model size, grid carbon intensity, and hardware lifetime. The paper's measurements are transparent, the arithmetic in Equations (2)-(4) is internally consistent, and the use of three carbon-intensity regimes is a sensible way to separate operational from embodied contributions. The paper also makes a useful conceptual point that energy efficiency and carbon efficiency are not the same. The main limitation is that the 'total carbon' claim is currently computed from GPU-only power, so the central conclusion is conditional on non-GPU overhead being negligible.

major comments (3)

[Section 2.1, Eq. (4)] The headline claim in the abstract and the Introduction that 'strategically using older GPUs like T4 could effectively reduce total carbon emissions' is not yet supported, because Eq. (4) labels GPU-only operational energy as 'total carbon emission.' Section 2.1 explicitly says 'this study focuses on GPU power consumption,' and Eq. (1) uses NVML GPU power only. For the 7B model at batch size 1, the paper reports that T4's GPU energy is 20% lower but its latency is 2.2x higher than RTX6000 Ada's. Under those numbers, any fixed non-GPU power (CPU, DRAM, cooling, PSU losses, PUE overhead) exceeding roughly one sixth of the RTX6000 Ada's average GPU power makes T4's system-level carbon per prompt higher than Ada's. That is a low threshold for a real serving node. Please either add node-level energy measurements or a defensible PUE/overhead model, or relabel the metric as 'GPU-power-based operational carbon plus embodied carbon' and soften the total-carbon claims accordingly.
[Section 2.2] The empirical characterization reports median latency and average power but gives no number of trials, no variance, and no statistical significance. The batch-size-1 energy advantage of T4 over RTX6000 Ada is 28% for the 1B model and 20% for the 7B model, while the 3B comparison is a 1.4x disadvantage; these are small margins that could reverse under measurement noise, thermal variation, or prompt heterogeneity. Please report run counts, error bars, and ideally confidence intervals for the latency and energy values that underlie the main comparisons.
[Section 3.1 and Section 3.4, Eq. (3)] The embodied-carbon analysis assumes a single fixed 5-year lifetime for both GPUs, and the sensitivity study in Section 3.4 sweeps only the T4's lifetime while keeping RTX6000 Ada's at 5 years. Since the total-carbon comparison between older and newer GPUs depends directly on the lifetime ratio, the conclusion that older GPUs reduce total carbon is conditional on an assumed ratio that is plausible but not demonstrated. The paper should show how the batch-size-1 total-carbon ordering changes when both lifetimes vary over a realistic range, not just T4's lifetime.

minor comments (5)

[Figures 5 and 6] The captions say the figures are 'under the CISO grid,' but the body text says 'We use the QC's CI value' and the figure legends label the operational component as 'Operational (QC).' Please align the captions, legends, and text.
[Section 2.1] The number of prompts used in the evaluation is not reported, only that prompts generating more than 150 tokens are considered. Reporting the dataset size and the distribution of prompt lengths would help assess the representativeness of the median latency and average power.
[Table 1] The technology node for RTX6000 Ada is listed as 5 nm, but the actual process is NVIDIA's 4N custom node; please use the vendor-specified process name or add a citation.
[Section 2.2] The abbreviation 'OOM' is used in Figure 1 but is not defined at first use; please spell out 'out of memory' in the text or caption.
[References] Reference [34] is a blog citation for ChatGPT's carbon per query; a primary or peer-reviewed source would be more appropriate for a quantitative claim in the introduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: carbon results are arithmetic compositions of measured energy/time, external carbon intensities, and ACT embodied values.

full rationale

The paper's derivation chain is a transparent accounting model. Energy consumption is measured directly with NVML (Eq. 1: E_prompt = P_prompt * t_prompt). Operational carbon is the product of that measured energy and externally sourced grid carbon intensities (Eq. 2). Embodied carbon is the execution-time-weighted fraction of ACT-derived chip-area/memory embodied values (Eq. 3), and total carbon is the sum (Eq. 4). No parameter is fitted to the paper's conclusions, and no 'prediction' is derived from the result it is supposed to establish. The finding that T4 can be more energy-efficient at batch size 1 is an empirical measurement reported in Section 2.2, not an artifact of the model. The carbon conclusions are logical consequences of multiplying measured quantities by stated external factors. The only notable caveat — that the study focuses on GPU power rather than full node power — is a scope limitation and a possible correctness risk, not a circularity: the equations honestly state what is included. The paper does not rely on self-citations to justify its central premise, and the embodied-carbon values are taken from an independent architectural carbon modeling tool (ACT) and compared to another independent study. Therefore the analysis is self-contained as an accounting model and warrants a circularity score of 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The analysis has no fitted parameters; it is an accounting model that multiplies measured GPU energy by grid carbon intensity and amortizes ACT-based embodied carbon over a hand-chosen 5-year lifetime. The central claims rest on four domain assumptions: GPU-only power is a sufficient proxy for serving energy, ACT embodied values are accurate, static annual average carbon intensities capture operational emissions, and Alpaca prompts generating more than 150 tokens represent LLM serving workloads. No new entities are introduced.

free parameters (1)

GPU lifetime assumption (LT) = 5 years
Equation 3 amortizes embodied carbon over LT; 5 years is assumed as a typical datacenter component lifetime, and Figure 7 shows the embodied carbon share is sensitive to this choice.

assumptions (4)

domain assumption GPU power consumption is the only significant contributor to LLM serving energy.
Section 2.1 says the study focuses on GPU power because LLMs mainly utilize GPU resources; Equation 4 treats E_prompt as the total energy for carbon accounting, excluding CPU, DRAM, cooling, and PUE.
domain assumption ACT embodied carbon estimates based on chip area and memory size are accurate for these two GPUs.
Table 1's embodied carbon values come from the ACT model; the total-carbon comparison relies on the relative embodied values being correct.
domain assumption Static annual average carbon intensities per region capture operational emissions.
Equation 2 multiplies measured energy by 2023 average CIs from Electricity Maps; real-time or marginal CIs vary and would change the operational carbon ranking.
domain assumption Alpaca prompts generating more than 150 tokens represent LLM serving workloads.
Section 2.1 selects only prompts producing more than 150 tokens for end-to-end comparisons, which excludes shorter, common serving requests.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Sustainable Large Language Model Serving." pith.science (2026). https://pith.science/paper/NOMIZW2G

@misc{pith2026250101990,
  author       = {Pith},
  title        = {Pith review of: Towards Sustainable Large Language Model Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOMIZW2G}},
  note         = {Machine review of arXiv:2501.01990}
}

read the original abstract

In this work, we study LLMs from a carbon emission perspective, addressing both operational and embodied emissions, and paving the way for sustainable LLM serving. We characterize the performance and energy of LLaMA with 1B, 3B, and 7B parameters using two Nvidia GPU types, a latest-generation RTX6000 Ada and an older-generation T4. We analytically model operational carbon emissions based on energy consumption and carbon intensities from three grid regions -- each representing a different energy source mix, and embodied carbon emissions based on chip area and memory size. Our characterization and modeling provide us with an in-depth understanding of the performance, energy, and carbon emissions of LLM serving. Our findings highlight the potential for optimizing sustainable LLM serving systems by considering both operational and embodied carbon emissions simultaneously.

Figures

Figures reproduced from arXiv: 2501.01990 by the authors.

**Figure 2.** Throughput and energy in the prefill phase (1Bparameter LLaMA). 1E+0 1E+2 1E+4 1 2 4 8 16 32 64 Throughput (tokens/s) Batch Size RTX6000 Ada T4 (a) Throughput. 0.01 0.1 1 10 1 2 4 8 16 32 64 Energy (J/token) Batch Size RTX6000 Ada T4 (b) Per-token energy [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Throughput and energy in the decode phase (1Bparameter LLaMA). RTX6000 Ada and T4. We observe that RTX6000 Ada is more efficient from both a throughput and energy perspective during the prefill phase. This is because the prefill phase is compute-bounded, which prefers more powerful GPUs like RTX6000 Ada. We also notice that the throughput reaches the peak when the batch size is 8 on T4 and 32 on RTX6000 Ada. Energ… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Per-prompt carbon emission under the QC, CISO, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 7.** Figure 7: Embodied carbon emissions of T4 GPU under dif [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

**Figure 6.** Figure 6: Per-token carbon emission in the decode phase under the CISO grid (1B-parameter LLaMA). Takeaway 3: The balance between operational and embodied carbon emissions varies, depending on the CI. In high-CI regions, operational carbon makes up a significant portion of per-…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 29 canonical work pages

[1]

AWS Trainium

AWS. AWS Trainium. https://aws.amazon.com/machine-learning/trainium/

work page
[2]

Reducing the carbon impact of generative AI inference (today and in 2035)

Andrew A Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. Reducing the carbon impact of generative AI inference (today and in 2035). In Proceedings of the 2nd Workshop on Sustainable Computer Systems (HotCarbon), pages 1–7, 2023

work page 2023
[3]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

arXiv 2022
[4]

Electricity maps

Electricity Maps ApS. Electricity maps. https://app.electricitymaps.com/, 2024

work page 2024
[5]

Fine- tuning giant neural networks on commodity hardware with automatic pipeline model parallelism

Saar Eliad, Ido Hakimi, Alon De Jagger, Mark Silberstein, and Assaf Schuster. Fine- tuning giant neural networks on commodity hardware with automatic pipeline model parallelism. In USENIX Annual Technical Conference (ATC), 2021

work page 2021
[6]

LLMCarbon: Modeling the end-to-end carbon footprint of large language models

Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang. LLMCarbon: Modeling the end-to-end carbon footprint of large language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024

work page 2024
[7]

AttentionStore: Cost-effective attention reuse across multi-turn conversations in large language model serving

Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. AttentionStore: Cost-effective attention reuse across multi-turn conversations in large language model serving. arXiv preprint arXiv:2403.19708, 2024

arXiv 2024
[8]

Accelerate AI development with Google cloud TPUs

Google. Accelerate AI development with Google cloud TPUs. https://cloud. google.com/tpu

work page

Show all 40 references

[9]

Why your internet habits are not as clean as you think

Sarah Griffiths. Why your internet habits are not as clean as you think. https://www.bbc.com/future/article/20200305-why-your-internet-habits- are-not-as-clean-as-you-think, 2020

2020
[10]

Lee, David Brooks, and Carole-Jean Wu

Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. ACT: Designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual International Symposium on Computer Architecture ...

2022
[11]

Small language models fine-tuned to coordinate larger language models improve complex reasoning

Gurusha Juneja, Subhabrata Dutta, Soumen Chakrabarti, Sunny Manchanda, and Tanmoy Chakraborty. Small language models fine-tuned to coordinate larger language models improve complex reasoning. arXiv preprint arXiv:2310.18338, 2024

arXiv
[12]

Fast inference from transform- ers via speculative decoding

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transform- ers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML) , 2023. Towards Sustainable Large Language Model Serving HotCarbon’24, July 9, 2024, San...

2023
[13]

Toward sustainable HPC: Carbon footprint estimation and environmental implications of HPC systems

Baolin Li, Rohan Basu Roy, Daniel Wang, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari. Toward sustainable HPC: Carbon footprint estimation and environmental implications of HPC systems. In Proceedings of the International Conference for High Performance Computing, Networ...

2023
[14]

Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024

2024
[15]

Counting carbon: A survey of factors influencing the emissions of machine learning

Alexandra Sasha Luccioni and Alex Hernandez-Garcia. Counting carbon: A survey of factors influencing the emissions of machine learning. arXiv preprint arXiv:2302.08476, 2023

2023 arXiv
[16]

Estimating the carbon footprint of BLOOM, A 176B parameter language model

Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of BLOOM, A 176B parameter language model. J. Mach. Learn. Res., 24(1), mar 2024

2024
[17]

Bringing carbon awareness to multi-cloud application delivery

Diptyaroop Maji, Ben Pfaff, Vipin P R, Rajagopal Sreenivasan, Victor Firoiu, Sreeram Iyer, Colleen Josephson, Zhelong Pan, and Ramesh K Sitaraman. Bringing carbon awareness to multi-cloud application delivery. In Proceedings of the 2nd Workshop on Sustainable Computer Systems ...

2023
[18]

Sitaraman

Diptyaroop Maji, Prashant Shenoy, and Ramesh K. Sitaraman. CarbonCast: Multi-day forecasting of grid carbon intensity. In Proceedings of the 9th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys), 2022

2022
[19]

Sitaraman, and Prashant Shenoy

Diptyaroop Maji, Ramesh K. Sitaraman, and Prashant Shenoy. DACF: Day- ahead carbon intensity forecasting of power grids using machine learning. In Proceedings of the Thirteenth ACM International Conference on Future Energy Systems (e-Energy), e-Energy ’22, 2022

2022
[20]

MTIA v1: Meta’s first-generation AI inference accelerator

Meta. MTIA v1: Meta’s first-generation AI inference accelerator. https://ai.meta. com/blog/meta-training-inference-accelerator-AI-MTIA/, 2023

2023
[21]

NVIDIA HGX AI Supercomputer

Nvidia. NVIDIA HGX AI Supercomputer. https://www.nvidia.com/en-us/data- center/hgx/, 2024

2024
[22]

NVIDIA management library (NVML)

Nvidia. NVIDIA management library (NVML). https://developer.nvidia.com/ management-library-nvml, 2024

2024
[23]

Ashraf, Christian Engelmann, Mallikarjun Shankar, and James H

George Ostrouchov, Don Maxwell, Rizwan A. Ashraf, Christian Engelmann, Mallikarjun Shankar, and James H. Rogers. GPU lifetimes on titan supercomputer: Survival analysis and reliability. In Proceedings of the International Conference for High Performance Computing, Networking, ...

2020
[24]

Splitwise improves GPU usage by splitting LLM inference phases

Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. Splitwise improves GPU usage by splitting LLM inference phases. In International Symposium on Computer Architecture (ISCA) , 2024

2024
[25]

So, Maud Texier, and Jeff Dean

David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. The carbon footprint of machine learning training will plateau, then shrink. Computer, 2022

2022
[26]

HuggingGPT: Solving AI tasks with ChatGPT and its friends in hugging face

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving AI tasks with ChatGPT and its friends in hugging face. Advances in Neural Information Processing Systems (NeurIPS) , 36, 2024

2024
[27]

PowerInfer: Fast large language model serving with a consumer-grade GPU

Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. PowerInfer: Fast large language model serving with a consumer-grade GPU. arXiv preprint arXiv:2312.12456, 2023

2023 arXiv
[28]

Hashimoto

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Car- los Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_ alpaca, 2023

2023
[29]

NVIDIA Tesla T4

TechPowerUP. NVIDIA Tesla T4. https://www.techpowerup.com/gpu-specs/ tesla-t4.c3316/, 2018

2018
[30]

NVIDIA RTX 6000 Ada Generation

TechPowerUP. NVIDIA RTX 6000 Ada Generation. https://www.techpowerup. com/gpu-specs/rtx-6000-ada-generation.c3933/, 2023

2023
[31]

LLaMA: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. LLaMA: Open and efficient foundation langua...

2023 arXiv
[32]

FreshLLMs: Refreshing large lan- guage models with search engine augmentation

Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. FreshLLMs: Refreshing large lan- guage models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023

arXiv
[33]

Peeling back the carbon curtain: Carbon optimization challenges in cloud computing

Jaylen Wang, Udit Gupta, and Akshitha Sriraman. Peeling back the carbon curtain: Carbon optimization challenges in cloud computing. In Workshop on Sustainable Computer Systems (HotCarbon) , 2023

2023
[34]

Gen AI’s environmental ledger: A closer look at the carbon footprint of ChatGPT

Vinnie Wong. Gen AI’s environmental ledger: A closer look at the carbon footprint of ChatGPT. https://piktochart.com/blog/carbon-footprint-of-chatgpt/, 2023

2023
[35]

Small models are valuable plug-ins for large language models

Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian McAuley. Small models are valuable plug-ins for large language models. arXiv preprint arXiv:2305.08848, 2023

2023 arXiv
[36]

ASPEN: High-throughput LoRA fine-tuning of large language models with a single GPU

Zhengmao Ye, Dengchun Li, Jingqi Tian, Tingfeng Lan, Jie Zuo, Lei Duan, Hui Lu, Yexi Jiang, Jian Sha, Ke Zhang, and Mingjie Tang. ASPEN: High-throughput LoRA fine-tuning of large language models with a single GPU. arXiv preprint arXiv:2312.02515, 2023

2023 arXiv
[37]

Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E

Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. RAFT: Adapting language model to domain specific RAG. arXiv preprint arXiv:2403.10131, 2024

2024 arXiv
[38]

A GNN-based day ahead carbon intensity forecasting model for cross-border power grids

Xiaoyang Zhang and Dan Wang. A GNN-based day ahead carbon intensity forecasting model for cross-border power grids. In Proceedings of the 14th ACM International Conference on Future Energy Systems (e-Energy) , e-Energy ’23, 2023

2023
[39]

Embodied carbon account- ing through spatial-temporal embodied carbon models

Xiaoyang Zhang, Yijie Yang, and Dan Wang. Embodied carbon account- ing through spatial-temporal embodied carbon models. arXiv preprint arXiv:2312.06364, 2023

2023 arXiv
[40]

DistServe: Disaggregating prefill and decoding for goodput- optimized large language model serving

Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput- optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024

2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

AWS Trainium

AWS. AWS Trainium. https://aws.amazon.com/machine-learning/trainium/

work page

[2] [2]

Reducing the carbon impact of generative AI inference (today and in 2035)

Andrew A Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. Reducing the carbon impact of generative AI inference (today and in 2035). In Proceedings of the 2nd Workshop on Sustainable Computer Systems (HotCarbon), pages 1–7, 2023

work page 2023

[3] [3]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

arXiv 2022

[4] [4]

Electricity maps

Electricity Maps ApS. Electricity maps. https://app.electricitymaps.com/, 2024

work page 2024

[5] [5]

Fine- tuning giant neural networks on commodity hardware with automatic pipeline model parallelism

Saar Eliad, Ido Hakimi, Alon De Jagger, Mark Silberstein, and Assaf Schuster. Fine- tuning giant neural networks on commodity hardware with automatic pipeline model parallelism. In USENIX Annual Technical Conference (ATC), 2021

work page 2021

[6] [6]

LLMCarbon: Modeling the end-to-end carbon footprint of large language models

Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang. LLMCarbon: Modeling the end-to-end carbon footprint of large language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024

work page 2024

[7] [7]

AttentionStore: Cost-effective attention reuse across multi-turn conversations in large language model serving

Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. AttentionStore: Cost-effective attention reuse across multi-turn conversations in large language model serving. arXiv preprint arXiv:2403.19708, 2024

arXiv 2024

[8] [8]

Accelerate AI development with Google cloud TPUs

Google. Accelerate AI development with Google cloud TPUs. https://cloud. google.com/tpu

work page