Pith. sign in

REVIEW 3 major objections 5 minor 44 references

A single signal—how much a token's attention shifts between denoising steps—can make diffusion LLMs far faster without hurting accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:41 UTC pith:TPYR3ZNQ

load-bearing objection A well-engineered drift-based cache/commit system with a real idea, but the throughput claim rests on a commitment premise that is only validated on LLaDA-8B/GSM8K and needs more evidence before the Pareto claim is credible. the 3 major comments →

arxiv 2607.14107 v1 pith:TPYR3ZNQ submitted 2026-05-07 cs.CL cs.AI

Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs

Mingyu Lee , Akshat Ramachandran , Souvik Kundu , Tushar Krishna This is my paper
classification cs.CL cs.AI
keywords diffusion LLMstoken driftKV cacheparallel decodinginference accelerationattention distributionKL divergencetraining-free
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the two main inefficiencies of diffusion large language models — stale key-value caches from bidirectional attention, and conservative token-by-token decoding — are two faces of one phenomenon: token representations drift as newly decoded tokens reshape the context. It defines drift as the KL divergence between a token's attention distribution at consecutive denoising steps, and uses it two ways: to refresh only the cache positions that actually go stale, and to commit tokens early when a sharp drift spike signals convergence to the final token. The paper argues this single signal subsumes the separate heuristics of earlier work and, if correct, lets diffusion LLMs decode 2–4× more tokens per forward pass at parity accuracy without any training. The evidence is accuracy–throughput gains on mathematics and coding benchmarks across several dLLM families.

Core claim

Token representational drift is an inherent property of bidirectional diffusion decoding: when a masked position is unmasked, its hidden-state change propagates through attention to every other position that attends to it, so all representations move even under exact recomputation. The paper shows that this drift is not uniform — it concentrates near the active decoding block and it advances in sharp events that coincide with the onset of convergence toward a token's eventual prediction. Polestar exploits the first property to refresh only high-drift KV-cache positions (Polestar-Cache) and the second to commit tokens earlier, including suffix tokens that are not yet being denoised (Polestar-

What carries the argument

Token drift, defined as the KL divergence between a token's attention distribution at consecutive denoising steps (Eq. 4), is the central object. For cache refresh, drift identifies which KV positions are stale enough to warrant a sparse forward pass, using spherical K-means centroids as proxies for hidden states and residual NVFP4 quantization to keep them cheap. For commitment, a drift-delta — current drift minus the mean of the past h steps — is compared against a confidence-conditioned threshold α(τ_s − c)², so lower-confidence tokens can still be committed when they exhibit a sharp drift event; suffix tokens become eligible when their centroids are among the top-k drifting clusters, and

Load-bearing premise

The entire throughput gain rests on the premise that a sharp increase in a token's attention-drift signals the beginning of stable convergence to its final value — a correlation established only on LLaDA-8B's GSM8K trajectories, and one that, if it fails on other models or tasks, would unmask wrong tokens and erase the Pareto gains.

What would settle it

Run Polestar-Commit with the paper's default parameters on a coding or vision-language benchmark with a dLLM family not in the paper, and measure the agreement between drift-triggered early commits and the token eventually selected after full denoising for suffix positions. If that agreement is not significantly above a static-confidence baseline, the claim that drift marks convergence is false. A cheaper proxy: compute the correlation between drift-delta and ΔC (cosine progress to the final commitment state) on those tasks; near-zero correlation would falsify the mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is correct, diffusion LLMs can decode 2–4× more tokens per forward pass at parity accuracy, substantially closing the throughput gap to autoregressive LLMs without any retraining.
  • The drift signal subsumes the separate heuristics of prior work — static confidence thresholds, most-attended-token cosine dissimilarity, and dependency graphs — since a single KL measurement serves both cache refresh and commitment.
  • Sparse drift-triggered KV refresh makes cache reuse practical for bidirectional attention, reducing per-step recomputation and enabling diffusion LLMs on longer generations.
  • Across the reported benchmarks, the method reaches up to 3.67 tokens per forward pass and up to ~3.7× higher throughput at accuracy equal to or above the full-recomputation baseline — a direct corollary of the paper's results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same drift signal might transfer beyond diffusion LLMs to any bidirectional or iterative-refinement model (e.g., non-autoregressive or consistency models), where representation evolution is a natural convergence indicator — a testable extension.
  • If commitment via drift spikes is validated on more models and tasks, it could be combined with speculative decoding or adaptive compute to allocate denoising steps based on drift magnitude rather than confidence alone.
  • The paper's own ablations show aggressive commitment rules (α=2.5, h=0, τ_s=0.7) collapse accuracy, so the practical gains hinge on the drift threshold landing within a narrow window; robust deployment would need per-family calibration or a fallback.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Polestar, a training-free inference framework for diffusion LLMs that uses KL-divergence-based token representational drift (Eq. 4) as a unified signal for two optimization axes: KV-cache refresh and token commitment. Polestar-Cache clusters local-window hidden states, quantizes residuals, and selectively refreshes the top-k drifting clusters; Polestar-Commit commits current-block tokens using a drift-delta dynamic threshold and pre-commits high-drift suffix tokens gated by confidence. System optimizations (CPU offloading, CUDA streams) hide the added overhead. Across GSM8K, MATH, HumanEval, MBPP, ParallelBench, and multimodal MathVista/MathVerse on LLaDA-8B-Instruct, LLaDA-1.5, Dream-7B-Instruct, and LLaDA-V, the paper reports up to 3.67 TPF, up to 20.5× TPS over the baseline, and claims a new state of the art on the accuracy–throughput Pareto frontier.

Significance. If the drift signal is as predictive as claimed, this is a valuable unification: the same attention-derived quantity addresses both cache staleness and commit readiness without any training, and the paper provides unusually thorough engineering evidence — component ablations, extended baselines, hyperparameter sweeps, a selection-oracle comparison, and a first-order theory (Appendix B) showing that bidirectional attention induces drift and that KL bounds attention-output change. The main quantitative claims are, however, carried by Polestar-Commit's suffix pre-commit behavior, whose underlying premise is validated only on a narrow configuration. The paper is therefore substantial and publishable in principle, but the throughput headline requires additional validation before the claims can be fully credited.

major comments (3)
  1. [§3.2, §4.2, Algorithm 1 lines 35–39] Table 4 shows suffix-block commit adds +0.19 TPF and raises TPS from 82.14 to 87.57, so the headline 3.67 TPF depends on pre-committing suffix tokens that have never been denoised. The evidence is two LLaDA-8B/GSM8K trajectories (Fig. 3b/c) plus a qualitative HumanEval analogue (§D.9); Appendix B Propositions 1–2 do not establish that drift spikes coincide with logit convergence. The ablations expose fragility: α=2.5 gives 73.4% accuracy, h=0 gives 76.91%, and suffix threshold 0.85 gives 76.4% (Fig. 9b–d). Please validate post-commit accuracy of suffix-committed tokens across model families and code benchmarks, or soften the parallelism claim.
  2. [§5.2, Table 1] The paper claims "a new state of the art on the accuracy–throughput Pareto frontier" without defining or demonstrating a frontier. Several Table 1 cells sit below baseline accuracy — e.g., Dream GSM8K-256: 72.40 vs. 75.85; Dream HumanEval-256: 57.69 vs. 58.80; LLaDA MATH-512: 34.85 vs. 36.60. Since Pareto dominance requires at least an explicit trade-off front, please provide a formal dominance check over the reported operating points or state the acceptable accuracy-degradation budget; otherwise the headline overstates the tables.
  3. [§5.3, Fig. 6(a)] Fig. 6(a) is the only direct evidence that KL-drift outperforms cosine/uncertainty/random selection for cache-refresh targeting, but it is a single accuracy number on LLaDA-8B/GSM8K with no seeds or error bars. Polestar-Cache's contribution is precisely this selector, so the same oracle test should be reported on Dream-7B and HumanEval, or at least with multiple seeds. Without this, the 86% selection accuracy could be configuration-specific and does not yet support the cross-model claim.
minor comments (5)
  1. [Figure 1 vs. Table 1] Figure 1 labels Polestar with TPF=3.54, while Table 1 and Table 4 report TPF=3.67 for the same GSM8K/LLaDA-8B setting. Please clarify whether Figure 1 uses a different configuration or an outdated number.
  2. [Algorithm 1 line 5] The condition b%2 == 0 with b starting at 0 means full-sequence context is used at block 0 and every even block, but the text says "every alternate block entry." Clarify whether "alternate" means every other block starting from the first, and ensure the algorithm matches the prose.
  3. [Equation (4) and Algorithm 1 line 20] Equation (4) defines drift over attention distributions A^(t), but Algorithm 1 line 20 measures KL between proxy attention distributions P. Define the relationship between A and P explicitly, since the proxy is a centroid-level approximation.
  4. [Table 4] The row label "Fast-dLLM" with annotation "(+0.00%)" is confusing because Fast-dLLM's accuracy (77.88) is below the baseline (79.30). Clarify that this row is the reference point for the component ablation, not an improvement over baseline.
  5. [§5.2 and Appendix D.2] Most main accuracy tables lack confidence intervals or error bars; only MBPP reports them. Adding error bars to the key GSM8K/ HumanEval tables would make the small accuracy differences (e.g., 78.33 vs. 77.58) easier to interpret.

Circularity Check

0 steps flagged

No significant circularity: the drift signal is defined independently and validated against external oracles; the main risks are empirical generalizability of the commitment premise, not definitional reduction.

full rationale

The paper's central signal, KL-based token-representation drift (Eq. 4), is defined independently of both of the downstream targets: KV-cache staleness is measured via cosine distance between cached and recomputed KV states (Appendix C.2), and commitment progress is measured via cosine similarity to the final committed logit state (Section 3.2). Neither target is used to define drift, so there is no self-definitional reduction. Appendix B is explicitly a first-order perturbation analysis, not an end-to-end performance guarantee; Proposition 1 shows bidirectional attention generically induces drift under exact recomputation, and Proposition 2 is a standard Pinsker bound on attention-output change. Both are parameter-free and do not assume the conclusion. Polestar-Cache's refresh selector is validated against an explicit recomputation-based staleness oracle (Figure 6a), not against its own output. Polestar-Commit uses a hindsight oracle only in the motivating analysis; the deployed rule combines drift delta with confidence and is not equivalent to the oracle. The main weakness is that the claim 'sharp drift spikes correlate with the onset of convergence' is supported by few trajectories and the operating point is fragile to hyperparameters (e.g., α=2.5 drops GSM8K to 73.4%), but this is a correctness/generalization risk, not a circularity. Self-citations [30,31] are used for implementation components (NVFP4 quantization, CPU offloading, spherical K-means) and are not load-bearing in the derivation of the drift signal or the Pareto claims. Overall, the derivation chain does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 0 invented entities

No invented entities: drift is a measured property of existing attention distributions, not a new force, particle, or dimension. The cost of the approach is entirely in free parameters (9 listed), most selected by ablation on LLaDA-8B/GSM8K — the same configuration that leads the main results table — plus domain assumptions about transfer of drift-behavior across model families and about cluster-proxy fidelity.

free parameters (9)
  • Spherical K-means centroid count K = 8
    Ablated in Fig 7(a); K=8 balances proxy granularity vs overhead; chosen on LLaDA-8B GSM8K.
  • Top-k updated clusters = 4 of 8
    Ablated in Fig 6(c); k=4 gives the best accuracy-TPS tradeoff on GSM8K.
  • Refresh trigger tau_upd (decoded-token count) = 3
    Ablated in Fig 6(d) and D.7; drift spikes align with >=3 tokens decoded per step on LLaDA-8B GSM8K.
  • Drift-gate coefficient alpha = 10
    Ablated in Fig 9(c); balances commit aggressiveness vs accuracy on GSM8K; alpha=2.5 collapses accuracy to 73.4%.
  • Drift history length h = 5
    Ablated in Fig 9(b); h=0 is aggressive (76.91% acc), h=5 balances accuracy and TPF.
  • Local window sizes (prefix, suffix) = (2B, 1B) with B=32
    Ablated in Fig 7(b); 96-token local window around the active block.
  • Block size B = 32
    Ablated in Fig 9(a); B=64 raises TPF/TPS but drops accuracy to 75.1% on GSM8K.
  • Alternate-block full-sequence refresh cadence = every 2nd block entry
    Hand-set schedule in §4.1 ('full-sequence forward pass at every alternate block entry'); not ablated.
  • Suffix commitment confidence threshold = 0.9 (= tau_s)
    Ablated in Fig 9(d); 0.85 drops accuracy to 76.4%, 0.95 loses TPF.
axioms (6)
  • standard math Softmax attention with full support gives a_is > 0 for visible positions in bidirectional attention, so unmasking one position perturbs all others to first order.
    Used in Proposition 1 (Appendix B) to show bidirectional attention induces drift; this structural fact is what the drift signal rests on.
  • domain assumption LayerNorm, residual connections, FFN, and multi-layer propagation can be omitted from the drift derivation.
    Appendix B states these are 'omitted from the derivation' and would only propagate the perturbation further; the measured drift in the real model includes them, so the theory is a lower-bound-style argument, not a quantitative guarantee.
  • domain assumption Sharp KL-drift events mark the onset of logit convergence toward final committed tokens, across model families, tasks, and suffix positions.
    Core of Polestar-Commit (§3.2, Fig 3b/c); established empirically on LLaDA-8B/GSM8K, assumed to transfer to Dream-7B, LLaDA-1.5, LLaDA-V, MATH, HumanEval, MBPP, ParallelBench.
  • domain assumption KL drift of attention distributions measured on 8 cluster centroids faithfully proxies token-level staleness and commitment readiness.
    The cache-update path (§4.1) measures drift on proxy centroids and refreshes whole clusters; fidelity is validated in Fig 6(a) only for the top-25% stale-position selection task on LLaDA-8B.
  • domain assumption Benchmark and harness correctness (lm-eval, GSM8K/MATH/HumanEval/MBPP/ParallelBench/MathVista/MathVerse).
    Standard evaluation infrastructure (§5.1, D.1); no independent audit provided.
  • domain assumption Reported baseline accuracies and TPS for Fast-dLLM/Elastic-Cache/Dynamic-dLLM/d2Cache/EntropyCache/DAWN/KLASS reflect their standard tuned configurations.
    Comparisons in Tables 1-3, 6-9 rely on externally implemented baselines without released code, so per-configuration fairness is not independently verifiable.

pith-pipeline@v1.3.0-alltime-deepseek · 25379 in / 26595 out tokens · 267149 ms · 2026-08-02T14:41:34.147349+00:00 · methodology

0 comments
read the original abstract

The inference efficiency of diffusion large language models (dLLMs) is constrained by two challenges: bidirectional attention precludes efficient KV-cache reuse, while increasing decoding parallelism with static confidence thresholds can compromise generation quality. We observe that both challenges arise from a shared phenomenon: as tokens are decoded, their contextual integration through bidirectional attention causes token representations to drift (evolve) across decoding steps. This insight motivates Polestar, a training-free inference framework that uses token representation drift as a unified signal to jointly address both challenges. Polestar comprises two components: Polestar-Cache, which identifies stale KV-cache positions via drift and performs sparse KV-cache refreshes to enable efficient reuse, and Polestar-Commit, which detects sharp drift events to reliably identify commit-ready tokens. Across mathematics and coding benchmarks on several dLLM families, Polestar sets a new state of the art on the accuracy-throughput Pareto frontier, achieving up to 10.73% accuracy improvement, up to 3.7x higher throughput, and high decoding parallelism of 3.67 tokens per forward pass over existing baselines.

Figures

Figures reproduced from arXiv: 2607.14107 by Akshat Ramachandran, Mingyu Lee, Souvik Kundu, Tushar Krishna.

Figure 1
Figure 1. Figure 1: Accuracy-throughput (TPS) trade-off on GSM8K using LLaDA-8B￾Instruct, with each method subscripted by its achieved tokens per forward pass (TPF). The inherent sequential dependency of autoregressive large language models (AR-LLMs) [11, 30, 34] limits decoding parallelism and often leads to substantial in￾ference performance degradation, particularly for long￾form generation [41, 31]. Recently, masked diffu… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of dLLM inference strategies. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Motivating analyses for Polestar: (a) KV-cache error concentrates on high-drift tokens near [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Heatmap depicting drift across token positions and decod￾ing steps, with unmasked tokens at every step shown in red. Setup. We use LLaDA-8B-Instruct [27] with generation length of 256 with B = 32 [36], for input prompts from GSM8K [10]. The baseline performs full-sequence forward passes with a TPF of 1 at each decoding step without cache reuse. We first demonstrate that token drift, as quantified in Equa￾t… view at source ↗
Figure 5
Figure 5. Figure 5: Overview of Polestar. System optimizations omitted for brevity. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Polestar ablations on LLaDA-8B-Instruct, GSM8K, generation length 256 and block size [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Number of centroids K. (b) (Prefix,Suffix) local-window size Ωb. # of centroids K. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Multi-token decoding induces local representation drift. (a) Local-window drift grouped by [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Extended hyperparameter ablation. (a) Block size [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Additional HumanEval motivation analysis for Polestar: (a) KV-cache error concentrates [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: HumanEval drift heatmap depicting measured drift across token positions and decoding [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Timing breakdown of Polestar execution. (a) Unoptimized Polestar, (b) Optimized Polestar [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: GPU-memory traces comparison during decoding on LLaDA-8B-Instruct with 5-shot [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 21 linked inside Pith

  1. [1]

    T., Yang, Z., Qi, Z., Han, J., Sahoo, S

    Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Han, J., Sahoo, S. S., and Kuleshov, V . Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    D., Ho, J., Tarlow, D., and Van Den Berg, R

    Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces.Advances in neural information processing systems, 34:17981–17993, 2021

    2021

  3. [3]

    Program synthesis with large language models, 2021

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  4. [4]

    Learning to parallel: Accelerating diffusion large language models via learnable parallel decoding

    Bao, W., Chen, Z., Xu, D., and Shang, Y . Learning to parallel: Accelerating diffusion large language models via learnable parallel decoding. InThe Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=bFJ8Sdr224

    2026

  5. [5]

    F., Am- manamanchi, P

    Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Am- manamanchi, P. S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde, J. Z., Foster, C., Hsu, J., Jaiswal, M., Lee, W. Y ., Li, H., Lovering, C., Muennighoff, N., Pavlick, E., Phang, J., Skowron, A., Tan, S., Tang, X., Wang, K. A., Winata, G. I., Yvon...

    Pith/arXiv arXiv 2024

  6. [6]

    Beyond confidence: Adaptive and coherent decoding for diffusion language models, 2025

    Chen, K., Liu, Z., Tao, X., Liu, H., Fu, X., Zhang, S., Tu, D., Kong, L., Liu, R., and Li, H. Beyond confidence: Adaptive and coherent decoding for diffusion language models, 2025. URL https://arxiv.org/abs/2512.02044

    arXiv 2025

  7. [7]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., C...

    Pith/arXiv arXiv 2021

  8. [8]

    H., and Chen, Y

    Chen, X., Huang, S., Guo, C., Wei, C., He, Y ., Zhang, J., Li, H. H., and Chen, Y . Dpad: Efficient diffusion language models with suffix dropout, 2025. URL https://arxiv.org/abs/2508. 14148

    2025

  9. [9]

    Entropycache: Decoded token entropy guided kv caching for diffusion language models.arXiv preprint arXiv:2603.18489, 2026

    Cheong, M., Son, D., Lim, W., and Yoo, S. Entropycache: Decoded token entropy guided kv caching for diffusion language models.arXiv preprint arXiv:2603.18489, 2026

    arXiv 2026

  10. [10]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 10

    Pith/arXiv arXiv 2024

  12. [12]

    Measuring mathematical problem solving with the math dataset, 2021

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset, 2021. URL https: //arxiv.org/abs/2103.03874

    Pith/arXiv arXiv 2021

  13. [13]

    W., Shao, Y

    Hooper, C., Zhao, S., Manolache, L., Kim, S., Mahoney, M. W., Shao, Y . S., Keutzer, K., and Gholami, A. Multipole attention for efficient long context reasoning.arXiv preprint arXiv:2506.13059, 2025

    arXiv 2025

  14. [14]

    S., sun Seo, J., Zhang, Z., and Gupta, U

    Hu, Z., Meng, J., Akhauri, Y ., Abdelfattah, M. S., sun Seo, J., Zhang, Z., and Gupta, U. Flashdlm: Accelerating diffusion language model inference via efficient kv caching and guided diffusion, 2025. URLhttps://arxiv.org/abs/2505.21467

    arXiv 2025

  15. [15]

    V ., and Grover, A

    Israel, D., den Broeck, G. V ., and Grover, A. Accelerating diffusion llms via adaptive parallel decoding, 2025. URLhttps://arxiv.org/abs/2506.00413

    arXiv 2025

  16. [16]

    d2cache: Accelerating diffusion-based llms via dual adaptive caching.arXiv preprint arXiv:2509.23094, 2025

    Jiang, Y ., Cai, Y ., Luo, X., Fu, J., Wang, J., Liu, C., and Yang, X. d2cache: Accelerating diffusion-based llms via dual adaptive caching.arXiv preprint arXiv:2509.23094, 2025

    arXiv 2025

  17. [17]

    I., Cho, N

    Kang, W., Galim, K., Oh, S., Lee, M., Zeng, Y ., Zhang, S., Hooper, C., Hu, Y ., Koo, H. I., Cho, N. I., et al. Parallelbench: Understanding the trade-offs of parallel decoding in diffusion llms. arXiv preprint arXiv:2510.04767, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    H., Hong, S., Jung, H., Park, Y ., and Yun, S.-Y

    Kim, S. H., Hong, S., Jung, H., Park, Y ., and Yun, S.-Y . Klass: Kl-guided fast inference in masked diffusion models.arXiv preprint arXiv:2511.05664, 2025

    arXiv 2025

  19. [19]

    Diffusion language models know the answer before decoding, 2025

    Li, P., Zhou, Y ., Muhtar, D., Yin, L., Yan, S., Shen, L., Liang, Y ., V osoughi, S., and Liu, S. Diffusion language models know the answer before decoding, 2025. URL https://arxiv. org/abs/2508.19982

    Pith/arXiv arXiv 2025

  20. [20]

    Tidar: Think in diffusion, talk in autoregression, 2025

    Liu, J., Dong, X., Ye, Z., Mehta, R., Fu, Y ., Singh, V ., Kautz, J., Zhang, C., and Molchanov, P. Tidar: Think in diffusion, talk in autoregression, 2025. URL https://arxiv.org/abs/ 2511.08923

    arXiv 2025

  21. [21]

    dllm- cache: Accelerating diffusion large language models with adaptive caching.arXiv preprint arXiv:2506.06295, 2025

    Liu, Z., Yang, Y ., Zhang, Y ., Chen, J., Zou, C., Wei, Q., Wang, S., and Zhang, L. dllm- cache: Accelerating diffusion large language models with adaptive caching.arXiv preprint arXiv:2506.06295, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    M., Karashima, Y ., Wang, Z., Fujiki, D., and Fan, H

    Lu, G., Chen, H. M., Karashima, Y ., Wang, Z., Fujiki, D., and Fan, H. Adablock-dllm: Semantic- aware diffusion llm inference via adaptive block size.arXiv preprint arXiv:2509.26432, 2025

    arXiv 2025

  23. [23]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URLhttps://arxiv.org/abs/2310.02255

    Pith/arXiv arXiv 2024

  24. [24]

    Dawn: Dependency-aware fast inference for diffusion llms.arXiv preprint arXiv:2602.06953, 2026

    Luo, L., Shi, Z., Luo, J., Wang, Z., Ren, S., Wang, W., and Zhang, T. Dawn: Dependency-aware fast inference for diffusion llms.arXiv preprint arXiv:2602.06953, 2026

    arXiv 2026

  25. [25]

    dkv-cache: The cache for diffusion language models

    Ma, X., Yu, R., Fang, G., and Wang, X. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Attention is all you need for kv cache in diffusion llms.arXiv preprint arXiv:2510.14973, 2025

    Nguyen-Tri, Q., Ranjan, M., and Shen, Z. Attention is all you need for kv cache in diffusion llms.arXiv preprint arXiv:2510.14973, 2025

    arXiv 2025

  27. [27]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y ., Wen, J.-R., and Li, C. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Blockspec: Blockwise speculative decoding for diffusion LLMs, 2026

    Pan, T., Gong, B., Guang, M., Yong, H., Jiang, T., Li, Y ., Cao, Z., and Long, K. Blockspec: Blockwise speculative decoding for diffusion LLMs, 2026. URL https://openreview.net/ forum?id=hmAviop5rm

    2026

  29. [29]

    Hierarchy decoding: A training-free parallel decoding strategy for diffusion large language models

    Qi, X., Du, L., Zhang, X., Wei, L., Jin, T., and Zheng, D. Hierarchy decoding: A training-free parallel decoding strategy for diffusion large language models. InThe Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum? id=ZsIQUjQtdW. under review. 11

    2026

  30. [30]

    Microscopiq: Accelerating foundational models through outlier-aware microscaling quantization

    Ramachandran, A., Kundu, S., and Krishna, T. Microscopiq: Accelerating foundational models through outlier-aware microscaling quantization. InProceedings of the 52nd Annual International Symposium on Computer Architecture, pp. 1193–1209, 2025

    2025

  31. [31]

    Thinkv: Thought-adaptive kv cache compression for efficient reasoning models.arXiv preprint arXiv:2510.01290, 2025

    Ramachandran, A., Neseem, M., Sakr, C., Venkatesan, R., Khailany, B., and Krishna, T. Thinkv: Thought-adaptive kv cache compression for efficient reasoning models.arXiv preprint arXiv:2510.01290, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    N., Wang, Z., Akella, A., and Kundu, S

    Shen, J., Sarkar, G., Ro, Y ., Sridhar, S. N., Wang, Z., Akella, A., and Kundu, S. Improving the throughput of diffusion-based large language models via a training-free confidence-aware calibration.ACL, 2026

    2026

  33. [33]

    Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction, 2025

    Song, Y ., Liu, X., Li, R., Liu, Z., Huang, Z., Guo, Q., He, Z., and Qiu, X. Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction, 2025. URL https://arxiv.org/ abs/2508.02558

    arXiv 2025

  34. [34]

    B., McPherson, S., Sridhar, S

    Tian, J., Azizi, S., Zhao, Y ., Potraghloo, E. B., McPherson, S., Sridhar, S. N., Wang, Z., Zhang, Z., Pedram, M., and Kundu, S. Skipkv: Selective skipping of kv generation and storage for efficient inference with large reasoning models.MLSys, 2026

    2026

  35. [35]

    Accelerating diffusion large language models with slowfast sampling: The three golden principles, 2025

    Wei, Q., Zhang, Y ., Liu, Z., Liu, D., and Zhang, L. Accelerating diffusion large language models with slowfast sampling: The three golden principles, 2025. URL https://arxiv.org/abs/ 2506.10848

    arXiv 2025

  36. [36]

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025

    Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., and Xie, E. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Dynamic-dLLM: Dynamic cache-budget and adaptive parallel decoding for training-free acceleration of diffusion LLM

    Wu, T., Sun, X., Jiao, Y ., Li, Y ., Chen, Y ., Cao, Y .-H., Hu, Y .-Q., and Tian, Z. Dynamic-dLLM: Dynamic cache-budget and adaptive parallel decoding for training-free acceleration of diffusion LLM. InThe Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=SdnkB5pGbq

    2026

  38. [38]

    Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Llada-v: Large language diffusion models with visual instruction tuning, 2025

    You, Z., Nie, S., Zhang, X., Hu, J., Zhou, J., Lu, Z., Wen, J.-R., and Li, C. Llada-v: Large language diffusion models with visual instruction tuning, 2025. URL https://arxiv.org/ abs/2505.16933

    Pith/arXiv arXiv 2025

  40. [40]

    Dimple: Discrete diffusion multimodal large language model with parallel decoding, 2025

    Yu, R., Ma, X., and Wang, X. Dimple: Discrete diffusion multimodal large language model with parallel decoding, 2025. URLhttps://arxiv.org/abs/2505.16990

    Pith/arXiv arXiv 2025

  41. [41]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025

    Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y ., Wang, L., Xiao, Z., et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?, 2024

    Zhang, R., Jiang, D., Zhang, Y ., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W., Gao, P., and Li, H. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?, 2024. URLhttps://arxiv.org/abs/2403.14624

    Pith/arXiv arXiv 2024

  43. [43]

    Llada 1.5: Variance-reduced preference optimization for large language diffusion models,

    Zhu, F., Wang, R., Nie, S., Zhang, X., Wu, C., Hu, J., Zhou, J., Chen, J., Lin, Y ., Wen, J.-R., and Li, C. Llada 1.5: Variance-reduced preference optimization for large language diffusion models,

  44. [2025]

    12 Appendix A Extended Related Works 14 B Theoretical Justification of Representation Drift 14 C Supplementary Details on Polestar 16 C.1 Polestar Algorithm

    URLhttps://arxiv.org/abs/2505.19223. 12 Appendix A Extended Related Works 14 B Theoretical Justification of Representation Drift 14 C Supplementary Details on Polestar 16 C.1 Polestar Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 C.2 KV-Cache Staleness Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16...

    Pith/arXiv arXiv