Pith. sign in

REVIEW 8 cited by

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.11912 v3 pith:EMSCH3AS submitted 2024-04-18 cs.CL cs.LG

Hanshi Sun , Zhuoming Chen , Xinyu Yang , Yuandong Tian , Beidi Chen This is my paper · ORCID
classification cs.CLcs.LG
keywords triforcecachegenerationlongmodelsequencetimesa100
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

With large language models (LLMs) widely deployed in long content generation recently, there has emerged an increasing demand for efficient long-sequence inference support. However, key-value (KV) cache, which is stored to avoid re-computation, has emerged as a critical bottleneck by growing linearly in size with the sequence length. Due to the auto-regressive nature of LLMs, the entire KV cache will be loaded for every generated token, resulting in low utilization of computational cores and high latency. While various compression methods for KV cache have been proposed to alleviate this issue, they suffer from degradation in generation quality. We introduce TriForce, a hierarchical speculative decoding system that is scalable for long sequence generation. This approach leverages the original model weights and dynamic sparse KV cache via retrieval as a draft model, which serves as an intermediate layer in the hierarchy and is further speculated by a smaller model to reduce its drafting latency. TriForce not only facilitates impressive speedups for Llama2-7B-128K, achieving up to 2.31$\times$ on an A100 GPU but also showcases scalability in handling even longer contexts. For the offloading setting on two RTX 4090 GPUs, TriForce achieves 0.108s/token$\unicode{x2014}$only half as slow as the auto-regressive baseline on an A100, which attains 7.78$\times$ on our optimized offloading system. Additionally, TriForce performs 4.86$\times$ than DeepSpeed-Zero-Inference on a single RTX 4090 GPU. TriForce's robustness is highlighted by its consistently outstanding performance across various temperatures. The code is available at https://github.com/Infini-AI-Lab/TriForce.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A training-free self-speculative decoding system with a recallable sparse KV cache and entropy-guided adaptive speculation achieves up to 2.79× speedup while preserving the target distribution.

  2. Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Lossy speculative-decoding verification splits into truncation-based and collaborative methods; truncation-based methods underperform their matched baselines, and capping draft overshoot preserves quality.

  3. Rectified Sparse Attention

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ReSA keeps sparse decoding accurate by refreshing the KV cache with dense attention every f tokens, achieving near-lossless quality with a large speedup.

  4. POSS: Position Specialist Generates Better Draft for Speculative Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Using position-specialized draft layers instead of one single draft model improves later-token acceptance in speculative decoding, yielding modest speedups on Llama-3-8B and Llama-2-13B.

  5. CLaSp: In-Context Layer Skip for Self-Speculative Decoding

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A training-free, context-adaptive layer-skipping method for self-speculative decoding that reports roughly 1.1x to 1.8x speedups on LLaMA models while preserving output distribution.

  6. RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RAD selects redundant attention layers via self-speculative decoding throughput, replaces them with SSM blocks, and distills them into hybrid models that beat the original on GSM8K and CRUX.

  7. KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

    cs.LG 2026-07 conditional novelty 5.0 of 10

    KAP compiles structured knowledge priors into runtime access plans that cut proposal-time KV access to 5.5% at 128K while matching full-context QA quality and yielding up to 1.19× decode throughput.

  8. On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention

    cs.LG 2025-06 conditional novelty 5.0 of 10

    On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.

Pith tools