Pith. sign in

REVIEW 8 cited by

Full Stack Optimization of Transformer Inference: a Survey

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.14017 v1 pith:IGWKGQP2 submitted 2023-02-27 cs.CL cs.LG

Sehoon Kim , Coleman Hooper , Thanakul Wattanawong , Minwoo Kang , Ruohan Yan , Hasan Genc , Grace Dinh , Qijing Huang
show 4 more authors
Kurt Keutzer Michael W. Mahoney Yakun Sophia Shao Amir Gholami
This is my paper · ORCID
classification cs.CLcs.LG
keywords transformermodelsarchitectureapproachesinferencebeendesignoperations
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advances in state-of-the-art DNN architecture design have been moving toward Transformer models. These models achieve superior accuracy across a wide range of applications. This trend has been consistent over the past several years since Transformer models were originally introduced. However, the amount of compute and bandwidth required for inference of recent Transformer models is growing at a significant rate, and this has made their deployment in latency-sensitive applications challenging. As such, there has been an increased focus on making Transformer models more efficient, with methods that range from changing the architecture design, all the way to developing dedicated domain-specific accelerators. In this work, we survey different approaches for efficient Transformer inference, including: (i) analysis and profiling of the bottlenecks in existing Transformer architectures and their similarities and differences with previous convolutional models; (ii) implications of Transformer architecture on hardware, including the impact of non-linear operations such as Layer Normalization, Softmax, and GELU, as well as linear operations, on hardware design; (iii) approaches for optimizing a fixed Transformer architecture; (iv) challenges in finding the right mapping and scheduling of operations for Transformer models; and (v) approaches for optimizing Transformer models by adapting the architecture using neural architecture search. Finally, we perform a case study by applying the surveyed optimizations on Gemmini, the open-source, full-stack DNN accelerator generator, and we show how each of these approaches can yield improvements, compared to previous benchmark results on Gemmini. Among other things, we find that a full-stack co-design approach with the aforementioned methods can result in up to 88.7x speedup with a minimal performance degradation for Transformer inference.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A distillation-teleportation protocol for fault-tolerant QRAM

    quant-ph 2025-05 accept novelty 8.0 of 10

    An adaptive distillation-teleportation protocol implements a fault-tolerant QRAM query with poly(n) quantum resources and 1/poly(n) device fidelity, at the cost of an exponential classical dataset update each round.

  2. Edge-Inference Governors Need Memory-Clock State

    cs.PF 2026-06 unverdicted novelty 6.0 of 10

    EMC state is required in latency models for edge inference governors; EMC-blind CPU/GPU fits miss 25-28% deadlines while EMC-aware refits limit misses to 1.3% and identify feasible energy points across vision and LLM ...

  3. Learning to Remember, Learn, and Forget in Attention-Based Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Palimpsa adds a per-slot importance/precision state to gated linear attention, letting a fixed-size memory forget stale information and protect important information, and recovers Mamba2 as a high-forgetting limit.

  4. vAttention: Verified Sparse Attention

    cs.LG 2025-10 conditional novelty 6.0 of 10

    vAttention is a sparse attention method that mixes heavy-hitter tokens with a statistically sized random sample to provide (ε, δ)-guaranteed approximation of full attention.

  5. COMET: A Framework for Modeling Compound Operation Dataflows with Explicit Collectives

    cs.AR 2025-08 conditional novelty 6.0 of 10

    COMET adds explicit collective-communication costs to compound-operation dataflow modeling, and reports simulated fused dataflow speedups of 1.42x, 3.46x, and 1.82x over unfused baselines.

  6. SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Sharing one MLP layer's weights across several layers plus low-rank adapters recovers most of a pretrained LLM's quality with a fraction of the storage and faster phone inference.

  7. HAFM: Hierarchical Autoregressive Foundation Model for Music Accompaniment Generation

    cs.SD 2026-04 unverdicted novelty 5.0 of 10

    HAFM uses a hierarchical autoregressive model with dual-rate HuBERT and EnCodec tokens to generate coherent instrumental music from vocals, achieving FAD 2.08 on MUSDB18 while matching prior systems with fewer parameters.

  8. SD-Acc: Accelerating Stable Diffusion through Phase-aware Sampling and Hardware Co-Optimizations

    cs.AR 2025-07 conditional novelty 5.0 of 10

    Phase-aware sampling cuts Stable Diffusion's compute by roughly 2.4x to 5.7x with only small CLIP-score changes, and the accompanying FPGA accelerator turns this into 2.7x to 6.0x energy savings over an Nvidia V100 GPU.

Pith tools