{"id":"aa36d20f-76b3-4c17-bab5-db3e3f1569ff","arxiv_id":"2412.18910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AdaEAGLE learns to predict the number of accepted draft tokens from the last hidden state and uses that prediction as the adaptive draft length in EAGLE-style speculative decoding.","lead":"The paper adds a small neural network, LDLP, to the EAGLE speculative decoding framework to predict how many draft tokens to generate before each verification step. It reports modest throughput gains over fixed-length EAGLE and about 1.6x speedup over ordinary autoregressive decoding, with no change to output quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that LDLP predicts the optimal draft length depends on untested Assumption 1 in Appendix A; if acceptance probability is not monotone in official-history length, the training labels and EAGLE-Oracle upper bound are not latency-optimal.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the proof that the optimal draft length equals the acceptance length depends on Assumption 1, which is plausible but never validated. My reading of the paper confirms this is the central structural weakness. The abstract and contributions repeatedly use 'optimal' to describe the predicted draft length, and §2.3/Appendix A explicitly define the oracle objective as minimizing target forward passes. The paper itself states the counterexample before introducing Assumption 1, so the dependence is not hidden — it is acknowledged and then assumed away. A direct empirical check of Assumption 1 is feasible with the authors' own pipeline and would settle whether the claimed optimality is real. The other issues noted by the reader — headline inconsistency (1.62× vs 1.61× vs 1.61%), missing variance, and no released code — are real but secondary; they affect reproducibility and reporting, not the core argument. Since the reader already set CONDITIONAL and my concern is the same one, no verdict change is needed. I am not raising an external-consensus disagreement; this is an internal logical dependency that the paper itself flags.","tokens_in":21606,"tokens_out":4318,"duration_ms":42548,"concrete_test":"Run a controlled experiment on the same Vicuna-7B/EAGLE setup that directly measures Assumption 1. For many positions m in a set of target continuations, generate draft tokens at position m from two official prefixes: a longer prefix T_out[:j] and a shorter prefix T_out[:j′] with j′<j, following the data-collection procedure of §3.3. For each draft token, compute paccept = min(1, p_target/p_draft) using the target and draft model probabilities, and aggregate over all m and all pairs (j′, j). If paccept(ˆtm|j) < paccept(ˆtm|j′) occurs for a nontrivial fraction of cases, Assumption 1 is violated and the optimality argument collapses. Report the fraction and magnitude of violations, ideally broken down by benchmark and by position distance j−j′.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LDLP predicts the 'optimal' draft length is load-bearing for both the method design and the EAGLE-Oracle upper bound, and it depends entirely on Appendix A's Assumption 1: paccept(ˆtm|j) ≥ paccept(ˆtm|j′) for j′<j. The paper's own Appendix A concedes that without this assumption a shorter draft can change the draft hidden-state trajectory and produce a different token at the next position that is accepted, so the acceptance length is not necessarily the latency-optimal length. Yet Assumption 1 is never tested. The training label in §3.3 is the longest common prefix between the draft and target outputs, i.e., the acceptance length; if Assumption 1 fails, LDLP is regressing to the wrong target, the Table 2 'EAGLE-Oracle' is not an upper bound, and the abstract's 'optimal draft length' wording is unsupported. The reported average gain over the best fixed length is only 64.44 vs 62.88 Tok/s, so the distinction between optimizing for acceptance length and optimizing latency is not just philosophical; if the assumption is false, a predictor trained on true latency-optimal lengths could plausibly do better, and the current results could be a heuristic artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaEAGLE, an extension of the EAGLE speculative decoding framework in which a lightweight MLP (LDLP) predicts, at each decoding iteration, the number of draft tokens to be generated. The predictor takes the embedding and final-layer-normalized hidden state of the last accepted token and is trained with an asymmetric L1 loss against the length of the longest common prefix between draft and target outputs. Experiments on Vicuna-7B across six benchmarks report average throughput of 64.44 Tok/s for AdaEAGLE and 65.20 for AdaEAGLE-DDD, versus 62.88 for the best fixed-length EAGLE and 64.43 for DDD. An oracle-based analysis in Table 2 motivates the adaptive approach by showing a large potential gain, and ablations support the loss penalty. The paper claims this is the first explicit model of adaptive draft structures and that LDLP predicts the optimal draft length.","tokens_in":21776,"tokens_out":9445,"duration_ms":86276,"significance":"The contribution is practically relevant: the base variant removes manual thresholds, the predictor is cheap, and the combination with DDD improves throughput on four of six benchmarks relative to the compared baselines. The oracle analysis in §2.4 is a useful way to quantify headroom, and the limitations section is candid about the sequential and greedy scope. The strengths are the clean architecture, the ablation showing the benefit of the loss penalty, and the explicit modeling idea that can be transferred to other speculative decoding frameworks. However, the 'optimality' framing is not fully supported, and the throughput advantage over the best fixed-length baseline is small enough that variance reporting is needed before the headline claim can be taken at face value.","major_comments":[{"comment":"The optimality claim for the predicted draft length is load-bearing and depends on an untested monotonicity assumption. In Appendix A, the paper explicitly notes that a shorter draft can change the hidden-state trajectory and produce an accepted token that a longer draft would miss, and it then introduces Assumption 1 (paccept(ˆtm|j) ≥ paccept(ˆtm|j′) for j′<j) to prove Lemma 1 that the acceptance length equals the latency-optimal draft length. Section 3.3 uses exactly this quantity, the longest common prefix k_j, as the LDLP training label, and Table 2's EAGLE-Oracle is presented as an upper bound on achievable throughput. If Assumption 1 fails on real data, the training labels are not latency-optimal and the oracle need not bound the achievable speedup. Please validate the monotonicity empirically on the evaluation benchmarks, or replace the acceptance-length oracle with a true latency-minimizing oracle, or revise the abstract and Section 2.3 to say 'predicted acceptance length' rather than 'optimal draft length'.","section":"Appendix A; §2.3; §3.3"},{"comment":"The headline empirical gain is not supported by variance estimates. The average throughput difference between AdaEAGLE (64.44 Tok/s) and the best fixed-length EAGLE (62.88) is about 2.5%, and AdaEAGLE is not uniformly better per benchmark (for example, HumanEval: EAGLE-5 77.95 vs AdaEAGLE 77.55; Natural Questions: EAGLE-3 57.44 vs AdaEAGLE 57.41). No error bars, repeated runs, or multiple seeds are reported for a single-GPU batch-1 throughput measurement that is known to be sensitive to hardware noise. Please report at least 3-5 runs with standard deviations, or provide a paired comparison on identical hardware, to establish that the average gain is not within noise.","section":"Table 3; §4.2"}],"minor_comments":[{"comment":"Please specify the numerical values of λ in Eq. (4) and kmax in Eq. (3) in the experimental setup; these are needed to reproduce the reported results.","section":"§4.1; Eq. (3); Eq. (4)"},{"comment":"The abstract's '1.62x speedup' and the conclusion's '1.61'/'1.61%' do not obviously match Table 3 (65.20/40.56 ≈ 1.61, 64.44/40.56 ≈ 1.59); state the aggregation rule used for the reported speedup ratio.","section":"Abstract; §6"},{"comment":"There are several typos: 'a a Lightweight' in §3.1, 'accpeted' in the Table 3 caption, and 'avxchieves' and '1.61%' in §6.","section":"§3.1; Table 3 caption; §6"},{"comment":"The classification baseline ('w/ Cls') is not described in enough detail; specify the class set (for example, binned lengths) and the loss used.","section":"Table 4; §4.3.2"},{"comment":"The statement of Assumption 1 would be clearer if the positions j and j′ were explicitly tied to the iteration index in which the draft begins; consider adding a short notational remark.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest and the architecture is simple; the main risk is the gap between the 'optimality' language and the empirical support. I would be comfortable with acceptance after the authors add the monotonicity test and variance-aware throughput numbers. The scope fits the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: AdaEAGLE is a simple, plausible extension of EAGLE that adds a small MLP to predict draft length from the last hidden state. The gains over fixed-length EAGLE are real but small, around 1-4% throughput, and the paper's strongest claim—that it predicts the 'optimal' length—rests on an untested monotonicity assumption. Still, it is clearly written and deserves a serious referee.\n\nWhat is actually new: earlier adaptive-length methods (EAGLE-2, DDD, SpecDec++, DISCO, PEARL, OPT-Tree) rely on logit confidence or thresholds; AdaEAGLE is the first to train a dedicated regression head for the acceptance length. The oracle analysis in Section 2 is useful and honest: it quantifies a 29% headroom, which motivates the approach. The asymmetric L1 loss (penalizing under-prediction more than over-prediction) is simple, and the ablation shows it helps. Training is cheap and the architecture is light enough to re-implement from the text, though no code is released.\n\nSoft spots, in proportion:\n1. Appendix A's Assumption 1 is load-bearing but never tested. The paper itself concedes that a shorter draft changes the hidden-state trajectory for the next iteration and can produce a different token at the same position, so acceptance length stops being the latency-optimal length unless a longer verified history always improves draft accuracy. The training label and the EAGLE-Oracle upper bound both inherit this caveat. The method may still work as a heuristic, but the 'optimal' wording oversells it.\n2. No error bars or repeated seeds. Average throughput is 64.44 tok/s against 62.88 for the best fixed length, so the headline gain is only a few percent; on several benchmarks the gap is under 1 tok/s, well within possible noise.\n3. The abstract says 1.62x, the introduction says 1.61, and the conclusion says '1.61%' (and misspells 'achieves'). Minor but should be caught.\n4. No code. For a systems-oriented SD paper, lack of artifacts weakens reproducibility.\n\nThe citation pattern is fine; the related-work table is fair and the baselines are the right ones.\n\nWho this is for: researchers working on speculative decoding, especially EAGLE-style draft structures. A practitioner might pick up a few percent throughput, but should treat the 'optimal' claim with salt.\n\nRecommendation: send to peer review. Ask for repeated runs, code or a detailed reproducibility package, and either a direct test of Assumption 1 or a softened claim. The paper is currently a conditional accept, not a reject.","headline":"Plausible, modest extension of EAGLE; the 'optimality' claim exceeds the evidence, but the method is simple, reproducible, and deserves peer review.","tokens_in":22425,"tokens_out":4701,"would_cite":true,"duration_ms":41214,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdaEAGLE predicts the optimal draft length at each decoding step with a lightweight three-layer MLP, reaching a 1.62x speedup over vanilla autoregressive decoding and beating the best fixed-length EAGLE baseline on six benchmarks while…","keywords":["speculative decoding","adaptive draft length","draft length prediction","EAGLE","LLM inference acceleration","hidden state prediction","lossless decoding","greedy decoding"],"falsifier":"Run greedy EAGLE decoding on a fixed prompt set and measure $p_{accept}(\\hat{t}_m | j)$ for drafts conditioned on true verified histories of length $j$ and on shorter draft-influenced histories $j' < j$; if any longer history yields lower acceptance probability, Assumption 1 is violated. Then compare total target forward passes for drafts stopped exactly at the first rejection versus one token earlier: if the shorter draft ever reaches the same formal output with fewer target forward passes, Lemma 1's optimality claim collapses.","tokens_in":21325,"feed_emoji":"⚡","tokens_out":10958,"duration_ms":88337,"temperature":0.7,"pith_summary":"Speculative decoding, in which a small model drafts tokens that a large model verifies in one parallel pass, usually fixes the draft length in advance. The paper argues that this is wasteful because the number of draft tokens the target model will accept fluctuates sharply from step to step. AdaEAGLE therefore adds a Lightweight Draft Length Predictor (LDLP), a three-layer MLP that reads the last verified token's embedding and hidden state and predicts how many tokens to draft before drafting begins. Under greedy decoding, LDLP is trained to match the acceptance length, and the paper proves, under a monotonicity assumption, that drafting exactly the acceptance length minimizes target-model forward passes. On six benchmarks with Vicuna-7B, AdaEAGLE reaches 64.44 tokens/s versus 62.88 tokens/s for the best fixed-length EAGLE, and combining it with the threshold-based method DDD reaches 65.20 tokens/s, a 1.62x speedup over vanilla autoregressive decoding with unchanged output quality.","feed_headline":"One MLP predicts draft length, lifting LLM speed 1.62x","feed_subtitle":"A small MLP predicts when to stop drafting, lifting throughput from 62.9 to 65.2 tokens/s without manual tuning.","key_machinery":"The load-bearing component is the Lightweight Draft Length Predictor (LDLP), a three-layer MLP with residual connections. It takes as input the embedding $e_j$ and the final-layer-normalized hidden state $f_j$ of the last accepted token, computes a scalar score, and outputs the next draft length as $\\bar{k}_{r+1} = \\mathrm{Round}(\\mathrm{MLP}(e_j, f_j))$, clamped to $[0, k_{max}]$. Training labels are produced offline by comparing a target-generated continuation with a $k_{max}$-token EAGLE draft and taking the longest common prefix; the loss is $\\lambda|\\bar{k} - k^\\circ|$ when the prediction is too short and $|\\bar{k} - k^\\circ|$ otherwise, with $\\lambda > 1$. The mechanism carries the argument by replacing a manually chosen fixed draft length with a context-dependent prediction derived from the hidden state's plan for future tokens.","core_discovery":"The central claim is that the optimal draft length for EAGLE-style speculative decoding is the acceptance length: the largest prefix of the draft that the target model accepts, provided that longer official token histories never make the draft less accurate (Assumption 1 in the appendix). Lemma 1 shows that under that assumption, stopping the draft exactly at the first rejection minimizes target forward passes. AdaEAGLE turns this length into a predicted quantity: the LDLP module takes the embedding $e_j$ and final-layer-normalized hidden state $f_j$ of the last accepted token and outputs a clamped, rounded scalar draft length, trained with a penalized L1 loss that over-penalizes predictions shorter than the true acceptance length because a target forward pass costs about 20 times a draft step. The paper reports that this explicit length control outperforms every fixed-length EAGLE setting and matches or exceeds the threshold-based DDD baseline across six benchmarks, with AdaEAGLE at 64.44 tokens/s and AdaEAGLE-DDD at 65.20 tokens/s.","pith_inferences":["Editorial inference: if Assumption 1 ever fails, meaning a longer verified history can make the draft worse, then the LDLP training label is no longer guaranteed to minimize latency, and stopping one token before the apparent acceptance boundary could occasionally be faster.","Editorial inference: the penalized L1 surrogate could be replaced by a latency-aware objective, such as minimizing expected target forward passes per token directly, which might close more of the gap to the EAGLE-Oracle topline.","Editorial inference: extending this explicit-modeling idea to tree-based and non-greedy decoding would require predicting width and depth of subtrees, not just a single length; the LDLP design suggests the same lightweight predictor family could be reused there."],"forward_implications":["Fixed draft length disappears as a hyperparameter: one AdaEAGLE model adapts across tasks, drafting longer on predictable code and shorter on more dynamic question-answering, without per-dataset tuning.","Because the target model's verification step is unchanged, the acceleration is lossless and the output distribution remains identical to vanilla autoregressive decoding.","Explicit length prediction and threshold-based stopping are complementary: AdaEAGLE-DDD achieves the highest throughput on four of six benchmarks, reaching 65.20 tokens/s.","The measured gap to the EAGLE-Oracle topline, 83.46 versus 66.35 tokens/s on MT-Bench, shows the remaining headroom for adaptive draft-length modeling."],"supporting_citations":[{"why":"Defines EAGLE, the speculative decoding framework with draft features that AdaEAGLE builds on and that supplies the fixed-length baselines.","marker":"(Li et al., 2024b)"},{"why":"Supplies the observation that acceptance length depends on generated context and provides EAGLE-2 as a dynamic-tree comparison.","marker":"(Li et al., 2024a)"},{"why":"DDD is the threshold-based adaptive-length method AdaEAGLE compares against and combines with to form AdaEAGLE-DDD.","marker":"(Brown et al., 2024)"},{"why":"SpecDec++ supplies the definition of optimal draft length as the length minimizing target forward passes, which the oracle analysis adapts.","marker":"(Huang et al., 2024)"},{"why":"Draft & Verify is an implicit self-speculative adaptive method listed as a comparison in the analysis of draft-structure modeling.","marker":"(Zhang et al., 2024)"},{"why":"Supports the premise that hidden states of the last token plan future tokens, justifying LDLP's choice of input features.","marker":"(Wu et al., 2024)"}],"fun_headline_variants":["MLP predicts optimal draft length for 1.62x faster LLM decoding","Adaptive draft length via MLP: LLM speedup of 1.62x without tuning","Small MLP sets draft length, lifting LLM speed 1.62x","Predict draft length to stop early: LLM speed 1.62x","Context-aware draft length predictor speeds LLM decoding 1.62x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that drafting exactly the acceptance length is optimal assumes that a longer verified history never makes the draft model's guesses worse, and that monotonicity is treated as reasonable rather than tested.","fun_headline_variants_meta":{"raw":{"variants":["MLP predicts optimal draft length for 1.62x faster LLM decoding","Adaptive draft length via MLP: LLM speedup of 1.62x without tuning","Small MLP sets draft length, lifting LLM speed 1.62x","Predict draft length to stop early: LLM speed 1.62x","Context-aware draft length predictor speeds LLM decoding 1.62x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1558,"prompt_tokens":930,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":546,"tokens_out":628,"duration_ms":5099,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:20:29.187508+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run greedy EAGLE decoding on a fixed prompt set and measure $p_{accept}(\\hat{t}_m | j)$ for drafts conditioned on true verified histories of length $j$ and on shorter draft-influenced histories $j' < j$; if any longer history yields lower acceptance probability, Assumption 1 is violated. Then compare total target forward passes for drafts stopped exactly at the first rejection versus one token earlier: if the shorter draft ever reaches the same formal output with fewer target forward passes, Lemma 1's optimality claim collapses.","supporting_citations":[],"review_version":1}