REVIEW 9 cited by
Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have shown remarkable performance across a wide range of applications, often outperforming human experts. However, deploying these gigantic models efficiently for diverse inference use cases requires carefully designed hardware platforms with ample computing, memory, and network resources. With constant innovation in LLM serving optimizations and model architecture evolving at breakneck speed, the hardware requirements to meet Service Level Objectives (SLOs) remain an open research question. To answer the question, we present an analytical tool, GenZ, to efficiently navigate the relationship between diverse LLM model architectures(Dense, GQA, MoE, Mamba), LLM serving optimizations(Chunking, Speculative decoding, quanitization), and AI platform design parameters. Our tool estimates LLM inference performance metrics for the given scenario. We have validated against real hardware platforms running various different LLM models, achieving a max geomean error of 5.82.We use GenZ to identify compute, memory capacity, memory bandwidth, network latency, and network bandwidth requirements across diverse LLM inference use cases. We also study diverse architectural choices in use today (inspired by LLM serving platforms from several vendors) to help inform computer architects designing next-generation AI hardware accelerators and platforms. The trends and insights derived from GenZ can guide AI engineers deploying LLMs as well as computer architects designing next-generation hardware accelerators and platforms. Ultimately, this work sheds light on the platform design considerations for unlocking the full potential of large language models across a spectrum of applications. The source code is available at https://github.com/abhibambhaniya/GenZ-LLM-Analyzer . Users can also be tried it on at https://genz-llm-analyzer.streamlit.app/ without any setup on your web browser.
Forward citations
Cited by 9 Pith papers
-
RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
RAG-Stack jointly optimizes RAG algorithm choices and serving-system settings via sub-metric-aware multi-objective Bayesian optimization plus an analytical performance model, reporting Pareto frontiers covering 52.5% ...
-
TileSight: A First-Principles Tile-Centric Analytical GPU Performance Model from Cores to Clusters
A tile-centric analytical model predicts GPU kernel latency and cache behavior purely from microbenchmark-calibrated hardware rates, reaching about 12% GEMM error and 13% end-to-end LLM serving error across five GPU lines.
-
MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems
A dimensionally strict analytical framework codifies 22 ML systems walls into 28 composable resolvers for sub-second full-stack design-space exploration and hardware synthesis.
-
SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving
Throughput saturation in LLM serving is caused by DRAM-bandwidth saturation in decode-phase attention kernels, and SLIM predicts this knee from sparse profiling to avoid wasteful batching.
-
LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
Scaling on-chip M3D caches to 1–4 GB reduces simulated LLM serving energy by up to 44% in prefill and 30% in decode, but benefits saturate and are workload-dependent.
-
Think Before You Grid-Search: Floor-First Triage for LLM Serving
LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.
-
Sim-FA: A GPGPU Simulator Framework for Fine-Grained Asynchronous Pipeline Analysis
Sim-FA is a new simulator that instruments FlashAttention-3 for cycle-accurate GPGPU analysis, achieving 5.7% average error on H800 while explaining inaccuracies in existing DRAM traffic models.
-
Scaling Intelligence: Designing Data Centers for Next-Gen Language Models
A simulation study argues that flat optical networks with uniform bandwidth improve LLM training throughput and reduce tuning sensitivity compared to two-tier networks.
-
Memory Offloading for Large Language Model Inference with Latency SLO Guarantees
Select-N meets LLM latency SLOs by choosing and continuously adjusting an offloading interval that controls how much model state lives in CPU memory, increasing host memory usage and throughput versus prior offloading...
Discussion (0). Continue with ORCID to comment.