REVIEW 22 cited by
LLM Inference Unveiled: Survey and Roofline Model Insights
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The field of efficient Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise framework that analyzes the various methods of LLM Inference to provide a clear understanding of this domain. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems, such as why LLMs are memory-bound, how much memory and computation they need, and how to choose the right hardware. We systematically collate the latest advancements in efficient LLM inference, covering crucial areas such as model compression (e.g., Knowledge Distillation and Quantization), algorithm improvements (e.g., Early Exit and Mixture-of-Expert), and both hardware and system-level enhancements. Our survey stands out by analyzing these methods with roofline model, helping us understand their impact on memory access and computation. This distinctive approach not only showcases the current research landscape but also delivers valuable insights for practical implementation, positioning our work as an indispensable resource for researchers new to the field as well as for those seeking to deepen their understanding of efficient LLM deployment. The analyze tool, LLM-Viewer, is open-sourced.
Forward citations
Cited by 22 Pith papers
-
TileSight: A First-Principles Tile-Centric Analytical GPU Performance Model from Cores to Clusters
A tile-centric analytical model predicts GPU kernel latency and cache behavior purely from microbenchmark-calibrated hardware rates, reaching about 12% GEMM error and 13% end-to-end LLM serving error across five GPU lines.
-
XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization
XQuant caches quantized layer inputs and rematerializes Keys and Values on the fly, achieving up to 12.5x KV cache memory savings with about 0.1 perplexity degradation.
-
DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model
A near-core decompression accelerator plus a 3D roofline model speeds up compressed LLM matrix multiplication by up to 4x in simulation.
-
FlatAttention: Dataflow and Fabric Collectives Co-Optimization for Efficient Multi-Head Attention on Tile-Based Many-PE Accelerators
FlatAttention groups tiles and uses NoC collectives to reduce HBM traffic by up to 16x and reach 89.3% utilization for multi-head attention on many-PE accelerators.
-
MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems
A dimensionally strict analytical framework codifies 22 ML systems walls into 28 composable resolvers for sub-second full-stack design-space exploration and hardware synthesis.
-
SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving
Throughput saturation in LLM serving is caused by DRAM-bandwidth saturation in decode-phase attention kernels, and SLIM predicts this knee from sparse profiling to avoid wasteful batching.
-
NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement
A heterogeneous GPU-PNM serving system that routes short prompts to GPUs and long prompts to near-memory sparse-attention devices, with runtime migration for growing contexts, reports large throughput and tail-latency...
-
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
A two-stage token-then-element KV cache pruning scheme with a reconfigurable sorting accelerator is reported to reduce decoding memory traffic by ~7x and energy by up to 90% while keeping accuracy within ~1%.
-
Think Before You Grid-Search: Floor-First Triage for LLM Serving
LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.
-
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
Sliding-window bidirectional-attention KV compression with Token2Chunk and periodic PagedAttention eviction keeps near-full reasoning accuracy at ~20% cache while lifting memory-constrained throughput ~12.75%.
-
Sim-FA: A GPGPU Simulator Framework for Fine-Grained Asynchronous Pipeline Analysis
Sim-FA is a new simulator that instruments FlashAttention-3 for cycle-accurate GPGPU analysis, achieving 5.7% average error on H800 while explaining inaccuracies in existing DRAM traffic models.
-
Teaching LLMs to Speak Spectroscopy
A LLaMA-3.1-8B model fine-tuned with LoRA on digit-serialized SDSS spectra predicts redshifts with MAE 0.043 and retains 85% of its astronomy QA performance.
-
Kinetics: Rethinking Test-Time Scaling Laws
A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.
-
LatentLLM: Attention-Aware Joint Tensor Compression
LatentLLM compresses pretrained LLMs and multimodal models with attention-aware joint low-rank tensor decomposition, outperforming SVD-based baselines on OPT perplexity and LLaVA ScienceQA.
-
Scaling Law for Quantization-Aware Training
A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.
-
A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing
Tunable energy landscapes whose thermal averages equal sigmoid, softmax, and matrix-vector products can, in principle, form the basis of a low-energy analog computer, with a superconducting double-well device as a fir...
-
HPIM: Heterogeneous Processing-In-Memory-based Accelerator for Large Language Models Inference
A heterogeneous processing-in-memory accelerator that runs attention in SRAM and weight-heavy matrix-vector products in HBM reportedly cuts single-batch LLM decoding latency by up to 34x in simulation.
-
APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
A bit-sliced Tensor Core scheme with a bipolar integer encoding and adaptive kernel autotuning reports 1.65-3.99x speedups for ultra-low-bit GPTQ-quantized LLM inference on three NVIDIA GPUs.
-
RAILS: Retrieval-Augmented Intelligence for Learning Software Development
RAILS, a RAG pipeline with FAISS retrieval and a compile-error feedback loop, is reported to outperform baseline GPT-3.5-Turbo prompting on 78 Java import error cases.
-
Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.
-
$\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.
-
How to keep pushing ML accelerator performance? Know your rooflines!
A JSSC survey organizing ML accelerator techniques around two roofline curves, one for throughput and one for energy efficiency, with the insight that their bottleneck knees often sit at different arithmetic intensities.
Discussion (0). Sign in to comment.