Pith. sign in

REVIEW 5 cited by

Faster Cascades via Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.19261 v2 pith:NSEOJZWU submitted 2024-05-29 cs.CL cs.AIcs.LG

Harikrishna Narasimhan , Wittawat Jitkrittum , Ankit Singh Rawat , Seungyeon Kim , Neha Gupta , Aditya Krishna Menon , Sanjiv Kumar This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords speculativecascadesdecodingruleapproachesdeferralmodelmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Cascades and speculative decoding are two common approaches to improving language models' inference efficiency. Both approaches involve interleaving models of different sizes, but via fundamentally distinct mechanisms: cascades employ a deferral rule that invokes the larger model only for "hard" inputs, while speculative decoding uses speculative execution to primarily invoke the larger model in parallel verification mode. These mechanisms offer different benefits: empirically, cascades offer better cost-quality trade-offs, often even outperforming the large model, while theoretically, speculative decoding offers a guarantee of quality-neutrality. In this paper, we leverage the best of both these approaches by designing new speculative cascading techniques that implement their deferral rule through speculative execution. We characterize the optimal deferral rule for our speculative cascades, and employ a plug-in approximation to the optimal rule. Experiments with Gemma and T5 models on a range of language benchmarks show that our approach yields better cost quality trade-offs than cascading and speculative decoding baselines.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Lossy speculative-decoding verification splits into truncation-based and collaborative methods; truncation-based methods underperform their matched baselines, and capping draft overshoot preserves quality.

  2. Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Hidden-state traces of frozen LLMs/VLMs can be read by lightweight trained heads to predict when to defer, clarify, call tools, or abstain, cutting routed inference cost 27–90%.

  3. BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.

  4. CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing

    cs.CL 2025-02 conditional novelty 5.0 of 10

    CITER trains a token-level router with preference optimization to route non-critical tokens to a small model and critical tokens to a large model, reducing inference cost on QA and math benchmarks.

  5. Consultant Decoding: Yet Another Synergistic Mechanism

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Consultant Decoding speeds up LLM generation by accepting draft tokens whose negative log-likelihood under the target model falls below a fixed threshold, reaching 2-3x speedups with comparable quality.

Pith tools