Pith. sign in

REVIEW 5 cited by

FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.14856 v2 pith:67UY2WMH submitted 2025-02-20 cs.CL cs.AIcs.LG

Weilin Zhao , Tengyu Pan , Xu Han , Yudi Zhang , Ao Sun , Yuxiang Huang , Kaihuo Zhang , Weilun Zhao
show 4 more authors
Yuxuan Li Jianyong Wang Zhiyuan Liu Maosong Sun
This is my paper
classification cs.CLcs.AIcs.LG
keywords samplingspeculativedraftfr-speclanguageacceleratingcompressionfrequency-ranked
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75% while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12$\times$ speedup over the state-of-the-art speculative sampling method EAGLE-2. Code available at https://github.com/thunlp/FR-Spec.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpeLLM: Character-Level Multi-Head Decoding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    SpeLLM converts a standard token-based LLM into a character-spelling model with multiple parallel output heads, achieving competitive downstream performance with a 5.1% average decoding speedup.

  2. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  3. Out-of-Vocabulary Sampling Boosts Speculative Decoding

    cs.LG 2025-06 reject novelty 6.0 of 10

    RDK redistributes a pruned drafter's probability mass using a token co-occurrence prior to restore acceptance rates in speculative decoding, but its formal guarantees are not established.

  4. Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EAGLE-2 loses most of its speedup on 4-bit quantized LLMs, and a hierarchical draft-then-sequence scheme restores 1.31x speedup over EAGLE-2 on W4A16 Llama-3-70B.

  5. MiniCPM4: Ultra-Efficient LLMs on End Devices

    cs.CL 2025-06 conditional novelty 5.0 of 10

    MiniCPM4-8B reportedly matches Qwen3-8B on standard benchmarks while using about 22% of the training tokens, and achieves large long-context speedups on edge devices.

Pith tools