REVIEW 1 cited by
Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present a simple on the fly method for faster inference of large language models. Unlike other (self-)speculative decoding techniques, our method does not require fine-tuning or black-box optimization to generate a fixed draft model, relying instead on simple rules to generate varying draft models adapted to the input context. We show empirically that our light-weight algorithm is competitive with the current SOTA for self-speculative decoding, while being a truly plug-and-play method.
Forward citations
Cited by 1 Pith paper
-
AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference
A runtime-adaptive speculative decoder with an 11-rule hierarchy and multi-policy engine keeps wasted draft compute under ~32% and bounds latency variance on CPU-constrained GGUF inference.
Discussion (0). Continue with ORCID to comment.