REVIEW 1 cited by
An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy.
Forward citations
Cited by 1 Pith paper
-
Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
No single LLM quantization method dominates performance, energy, and quality; the best choice depends on task, request length, load, parallelism, and GPU type.
Discussion (0). Continue with ORCID to comment.