REVIEW 5 cited by
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM service providers. To efficiently use GPU resources and boost throughput, batching multiple requests has emerged as a popular paradigm; to further speed up batching, LLM quantization techniques reduce memory consumption and increase computing capacity. However, prevalent quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully leverage the capabilities of modern GPUs, such as 4-bit integer operators, resulting in sub-optimal performance. To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss. Atom significantly boosts serving throughput by using low-bit operators and considerably reduces memory consumption via low-bit quantization. It attains high accuracy by applying a novel mixed-precision and fine-grained quantization process. We evaluate Atom on 4-bit weight-activation quantization in the serving context. Atom improves end-to-end throughput (token/s) by up to $7.7\times$ compared to the FP16 and by $2.5\times$ compared to INT8 quantization, while maintaining the same latency target.
Forward citations
Cited by 5 Pith papers
-
PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
PolyQ makes fractional-bit CPU LLM inference practical via per-channel bit allocation from {2,3,4,8,16} and compile-time channel-order merging.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models
A training-free framework jointly quantizes a VLA model to 4 bits and prunes visual tokens, recovering or exceeding full-precision success rates at 1.93x speedup.
-
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...
-
Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs
Adaptively quantizing parts of an LLM's layers to FP4 can improve win rates and trading yields in latency-sensitive agent tasks, but the reported gains come from choosing the best compression level after seeing test results.
Discussion (0). Continue with ORCID to comment.