Pith. sign in

REVIEW 12 cited by

FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.11421 v1 pith:SCX77GS2 submitted 2024-03-18 cs.DC cs.PF

Jiaao He , Jidong Zhai This is my paper
classification cs.DCcs.PF
keywords memoryservingbandwidthbatchefficientkv-cachemodelspart
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Cost of serving large language models (LLM) is high, but the expensive and scarce GPUs are poorly efficient when generating tokens sequentially, unless the batch of sequences is enlarged. However, the batch size is limited by some constantly reused intermediate results, namely KV-Cache. They occupy too much memory to fit more sequences into a GPU simultaneously. While they could be offloaded to host memory, the CPU-GPU bandwidth is an inevitable bottleneck. We find a way to decompose the transformer models into two parts of different characteristics, one of which includes the memory-bound KV-Cache accessing. Our key insight is that the aggregated memory capacity, bandwidth, and computing power of CPUs across multiple nodes is an efficient option to process this part. Performance improvement comes from reduced data transmission overhead and boosted GPU throughput to process the other model part. Moreover, we address efficiency challenges brought by heterogeneity at both temporal and inter-device scopes using scheduling and performance modeling techniques. Evaluation results show that our system achieves 1.88x - 5.04x the throughput of vLLM when serving modern LLMs with the same GPU.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training-Free Hashing-Based Attention via Binary Principal Components

    cs.LG 2026-08 conditional novelty 6.0 of 10

    BinaryPC uses binary principal components of keys to pick the most relevant tokens for attention, preserving accuracy with a 2% budget.

  2. Understanding Rate-Distortion Performance in Distributed Transformer Inference

    cs.LG 2026-01 unverdicted novelty 6.0 of 10

    Deeper transformer layers produce intermediate representations that are harder to lossy-compress, and the paper links this to growing covariance and Rademacher complexity.

  3. Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism

    cs.DC 2025-09 conditional novelty 6.0 of 10

    Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.

  4. KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    By downsampling key and value head dimensions and retraining with distillation, KV-Latent cuts KV cache memory by about 50% on 7-8B LLMs while keeping average benchmark scores within about 1 point of the base model.

  5. Kinetics: Rethinking Test-Time Scaling Laws

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.

  6. Hardware-Efficient Attention for Fast Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.

  7. DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

    cs.DC 2026-07 conditional novelty 5.0 of 10

    DualDecoder prefetches sparse KV-cache entries using a co-executed speculative token, removing GPU-resident auxiliary state and improving long-context LLM decoding throughput by up to 2.62x.

  8. MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MoQAE learns chunk-level KV cache bit-width selection with a mixture-of-experts router, improving accuracy and efficiency over prior quantization methods.

  9. Beyond the Buzz: A Pragmatic Take on Inference Disaggregation

    cs.DC 2025-06 conditional novelty 5.0 of 10

    Disaggregated serving (separate prefill and decode GPU pools) expands the throughput-interactivity Pareto frontier mainly for prefill-heavy workloads and models larger than about 10B parameters, provided the prefill-t...

  10. Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding

    cs.LG 2025-05 reject novelty 5.0 of 10

    LFPS predicts which KV cache positions a decoder will attend to by reusing historical vertical and slash patterns plus local expansion, then runs exact Top-k only on the predicted candidate set.

  11. HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs

    cs.CL 2025-07 conditional novelty 4.0 of 10

    HCAttention combines key quantization, CPU value offloading, and cumulative-attention eviction to run long-context LLMs with 12.5% to 25% of the GPU KV cache while keeping LongBench accuracy close to full attention.

  12. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

Pith tools