Pith. sign in

REVIEW 4 major objections 4 minor 168 references

DiffusionGemma Technical Report

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DiffusionGemma claims that discrete diffusion, obtained by fine-tuning instead of pretraining, can decode about 20 tokens per forward pass and reach roughly 1,500 tokens per second on a single H100, establishing a new Pareto frontier for sp

desk verdict A well-engineered open-weights diffusion LLM with a plausible but over-stated Pareto claim — the speed is measured, the loop-free assumption is not. read the letter →

arxiv 2608.00146 v1 pith:XMEY7REV submitted 2026-07-31 cs.CL cs.AI

DiffusionGemma Team: Adrien Ali Taïga , James Assiene , Daniele Calandriello , Rahma Chaabouni , João Gante , Tamara von Glehn , Nate Keating , Chris Knutsen
show 35 more authors
Martin Kukla Tianlin Liu Ivan Lobov Ofir Nabati João Gabriel Oliveira Nicolas Perez-Nieves Nastasia Prutianova Bobak Shahriari Jean Tarbouriech Pavel Tyletski Çağlar Ünlü Cindy Wu Glenn Cameron Jerome Connor Sertan Girgin Maarten Grootendorst Alon Levkovitch Eliya Nachmani Omar Sanseviero Piotr Stanczyk Quentin Berthet Andrew Campbell Clément Crepy Valentin De Bortoli Arnaud Doucet Romuald Elie Alexandre Galashov Klaus Greff Alexis Jacq David Ruhe Yu-Han Wu Sebastian Flennerhag Brendan O'Donoghue George Scrivener Shantanu Thakoor
This is my paper · ORCID
classification cs.CLcs.AI
keywords discretediffusiontextlanguagemodelblock-autoregressivedecodingentropy-boundedsamplingadaptivestoppingsamplerdistillationreinforcementlearninginferencespeed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiffusionGemma is a bid to move text generation from token-at-a-time decoding to parallel block refinement without sacrificing the intelligence of a modern model. The authors claim that fine-tuning an existing 25B-parameter mixture-of-experts autoregressive model rather than pretraining a diffusion model from scratch produces a discrete diffusion model that emits about 20 tokens per forward pass and reaches roughly 1,500 output tokens per second on a single H100 at batch size one. That is several times faster than the same model served autoregressively, even with speculative decoding, with a modest benchmark-quality penalty and retained multimodal, long-context, and thinking capabilities. A sympathetic reader should read this as the first open-weight evidence that text diffusion can be a practical serving alternative to autoregression in the low-concurrency regime.

What carries the argument

The mechanism that carries the argument is block-autoregressive discrete diffusion. The model encodes the prompt and already-written text with causal attention into a key-value cache, then iteratively denoises a 256-token canvas of uniformly random tokens using bidirectional attention, self-conditioning on its own previous predictions; each finished canvas is appended to the cache. Two named components matter: the entropy-bounded sampler accepts the lowest-entropy tokens and re-noises the rest, capping mutual information at a budget of 0.1, and adaptive stopping ends denoising when mean entropy falls below 0.005 and two consecutive argmax predictions agree. Together these turn the number of

What would settle it

Take a set of open-ended prompts, run DiffusionGemma with its default sampler, and log the step at which adaptive stopping fires together with the final output. If any substantial fraction of outputs show a coherent prefix that collapses into repetition while mean entropy falls below 0.005, the speed figure is measuring loop collapse rather than convergence. A complementary check is to rerun the same prompts with the maximum 48 denoising steps: wherever the 48-step output is clearly better than the default-stop output, the effective-step claim is not holding for that input.

Watch

Extended reading notes

Core claim

The central claim is that a text diffusion model can be obtained by fine-tuning an existing autoregressive checkpoint, and that the resulting model establishes a new Pareto frontier between generation speed and capability. The report argues this is achieved by keeping a causal encoder for context and key-value cache appends, denoising 256-token canvases with bidirectional attention, and then running a second training stage that combines sampler distillation with reinforcement learning, simultaneously raising reward and compressing the denoising trajectory to about 12 effective steps. The empirical assertion is concrete: roughly 20 tokens per forward pass and about 1,500 tokens per second on

Load-bearing premise

The strongest speed and quality numbers rest on one premise: the second training stage has made few-step generation safe, so the adaptive stop fires because the canvas has converged rather than because the model fell into a repetitive loop; and the report itself admits occasional token stuttering still occurs, so that premise is only approximately true.

Editorial extensions

If this is right

  • At batch size 1 on an H100, DiffusionGemma reaches about 1,500 tokens per second, versus 204 for the same model autoregressively and 303 with multi-token-prediction speculative decoding, roughly 7x and 5x faster.
  • The same weights can still decode left-to-right, with quality close to the original autoregressive model, so requests could be routed between diffusion and AR modes by latency and task difficulty.
  • The two-stage recipe uses less than 10% of the starting autoregressive model's training token budget, so converting an existing checkpoint into a diffusion model is cheap relative to pretraining.
  • Structured or highly constrained outputs such as JSON, code edits, or Sudoku grids can converge in two or three denoising steps, and even sequentially dependent outputs take only about seven steps for 40 tokens.
  • The joint reinforcement-learning and sampler-distillation stage also makes the model produce roughly half as many tokens as the supervised-fine-tuning checkpoint, which compounds the speed gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the dual diffusion/AR capability suggests a serving system could route short, latency-critical generations to diffusion and difficult or open-ended reasoning to autoregressive mode, or interleave both within one response; the paper does not implement such a router.
  • Editorial extension: the fixed entropy thresholds are global, but the report's own step-count plots show domain-dependent convergence, so a likely improvement is learned or per-prompt stopping thresholds, which the paper does not explore.
  • Editorial extension: the single-request speed advantage comes partly from shifting memory-bound work to compute-bound work, so the same recipe should benefit most on hardware with high compute-to-bandwidth ratios; the paper notes this but does not quantify it across GPUs.
  • Editorial extension: because the sampler re-noises high-entropy tokens, the model has a built-in opportunity for constrained decoding: a schema or grammar could force low-entropy structure and let the sampler explore only the uncertain slots; the paper demonstrates the phenomenon qualitatively without building such an interface.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. DiffusionGemma converts the pretrained Gemma 4 26B A4B mixture-of-experts language model into a discrete diffusion language model via a two-stage pipeline: supervised fine-tuning to teach bidirectional 256-token canvas denoising, followed by an online sampler-distillation and reinforcement-learning stage (SD·RL) that jointly improves quality and compresses the denoising trajectory. The resulting model generates text in block-autoregressive fashion, with an entropy-bounded sampler, temperature annealing, and adaptive stopping. The report claims roughly 20 tokens per forward pass (TPF) and about 1,500 output tokens per second on a single H100, establishing a new Pareto frontier for speed-versus-quality, while retaining AR decoding, thinking mode, multimodal inputs, and long-context support. The paper includes detailed architecture descriptions, training details, latency breakdowns, benchmark tables, and qualitative examples of bidirectional reasoning and self-correction.

Significance. If the central claims hold, this is a significant systems contribution: it demonstrates that a discrete diffusion model fine-tuned from an AR MoE backbone can deliver large low-batch decoding speedups while remaining competitive on reasoning and coding benchmarks, and the open-weights release with HuggingFace/vLLM reference implementations would provide a strong reproducible baseline for the community. The paper's strengths include measured hardware timings, a transparent per-kernel breakdown, a clear two-stage training recipe, and explicit acknowledgment of known limitations. However, the headline Pareto and speed claims rest on the entropy-bounded adaptive-stopping heuristic not being fooled by degenerate repetitive loops. The paper's own evidence shows that this exact failure mode occurs in the SFT checkpoint, and Section 10 concedes that the final model still exhibits occasional stuttering and repetitive loops. Because TPF/TPS are defined through adaptive-stopping trajectories, the empirical robustness of the stopping trigger is load-bearing and must be established before the central claim can be accepted. The comparison to Mercury 2 also rests on a black-box API speed estimate

major comments (4)
  1. [§5, §10, Algorithm 1, Eq. (10)] The headline TPF/TPS figures are computed along adaptive-stopping trajectories, and the stopping condition (Algorithm 1: mean entropy ≤ e_stop and stable argmax) can be triggered by the repetitive-loop degeneration documented for the SFT checkpoint in Section 5 and Appendix B (Figures 16–17). Section 10 concedes that the final SD·RL checkpoint still exhibits "occasional token stuttering... repetitive loops." The manuscript does not report the per-canvas rate of loop-induced early stops on the final checkpoint, nor the correlation between early stopping and correctness. Since Eq. (10) defines TPF using the adaptive-stopping trajectory, a non-negligible loop rate would inflate TPF/TPS while deflating quality. This is load-bearing for the abstract's "~20 TPF / ~1,500 TPS" and the Pareto claim. Please report loop/degeneration rates on the final checkpoint across the evaluation suite and show
  2. [Figure 1, §7, Table 3] The "new Pareto frontier" claim is established on a two-benchmark average (GPQA-Diamond and LiveCodeBench-v6) for both quality and speed, while the full evaluation suite contains 19 benchmarks. A two-task average is a thin basis for a frontier claim, especially because both tasks are reasoning/coding oriented and may favor thinking mode. In addition, Table 3 reports no error bars or seed variance; Figure 9 mentions 3-seed averages only for the SD·RL frontier. Please report the Pareto comparison over the full suite or provide a principled justification for the restricted subset, and include variance estimates or significance tests for the headline quality and speed comparisons.
  3. [Appendix E, §7] The speed comparison to Mercury 2 is based on a black-box API latency estimate using NNLS on Eq. (14). The appendix explicitly states that corrupted outputs exhausting the 50k-token maximum produce outliers that give a "slightly favorable bias" to the speed estimate, yet the paper uses this estimate for the "roughly 2.5× speedup over Mercury 2" claim and for Figure 1. Please report the estimates with outliers excluded (for example, capping output tokens at a plausible limit) and state how the comparison changes; otherwise the 2.5× speedup claim is not robust.
  4. [Abstract, §3, §10] The paper claims that DiffusionGemma "retains the starting model's support for ... long contexts," but no long-context evaluation is presented anywhere in the manuscript. Given the block-autoregressive KV-cache mechanism is a novel architectural modification, this claim needs empirical support, for example a long-context QA or summarization experiment at increasing input lengths, before it can be presented as a retained capability.
minor comments (4)
  1. [Figure 1 and Table 3 caption] The speed measurements for LLaDA 2.1 Flash (8× B200) and Nemotron 14B (1× H100, bfloat16) are made on different hardware than the DiffusionGemma H100 FP8 measurements, yet they are plotted on the same axes without a normalization caveat. Please add a clear note that cross-hardware TPS comparisons are indicative only.
  2. [§6, Eq. (12), Table 4] Eq. (12) defines TPS using a single t_fwd (13.56 ms for 4096-input/1024-output tokens), while Table 4 reports per-benchmark TPS that vary with context length. Please clarify whether the headline "~1,500 TPS" uses the fixed t_fwd or the per-benchmark estimates, and report the distribution of t_fwd across benchmarks.
  3. [Section 10, Table 3] The MMMU-Pro thinking score (54.3) dropping below the no-think score (66.0) is attributed to missing closing thought tags. This is a significant qualitative effect and should be discussed in the main experimental section, not only in the limitations list, so that readers do not misinterpret the thinking-mode benchmark as a capability limitation.
  4. [General] The first paragraph of the PDF/marked-up text contains missing spaces between words ("WeintroduceDiffusionGemma..."). Please ensure the arXiv source formatting is fixed in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's central claims are empirical measurements.

full rationale

DiffusionGemma is an empirical systems report. The headline numbers (TPF≈20, TPS≈1500) are measured values: TPF is defined in Eq. 10 as Total Tokens/(Total Denoising Steps + K − 1), and each quantity is reported per benchmark in Table 4; Eq. 12 then converts TPF to TPS using a measured t_fwd. No parameter is fitted to the headline speed and then re-presented as a prediction. The adaptive-stopping heuristic (Algorithm 1) and hyperparameters (b=0.1, e_stop=0.005, N=48) are engineering choices, and the paper explicitly shows that without SD·RL the SFT checkpoint degenerates into loops that collapse the entropy signal, which is a benchmark/deployment validity concern rather than an equation-level circularity. Self-citations are present (e.g., the block-AR provenance note and interpretability citations) but none is load-bearing: the block-AR strategy is also credited to independent concurrent work (Wu et al. 2025, Arriola et al. 2025, Deschenaux and Gulcehre 2026a), and the interpretability citations support a non-essential remark. The residual-loop limitation in Section 10 could affect the robustness of the Pareto claim, but that is an unverified assumption about deployment, not a reduction of the derivation to its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The self-conditioning MLP is an architectural module adopted from prior self-conditioning methods; it is not a new entity. Free parameters are the tuned sampler hyperparameters and canvas size, which are not derived and directly influence the reported efficiency metrics. The central derivation uses standard discrete diffusion results from the cited literature.

free parameters (5)
  • Canvas length C = 256 tokens
    Architectural choice for block-AR decoding; determines parallelization and per-forward cost; not derived from first principles.
  • Entropy budget b = 0.1
    Sampler acceptance threshold controlling token commitment; tuned for quality/latency; central to TPF.
  • Adaptive stopping threshold e_stop = 0.005
    Entropy threshold to trigger early stopping; tuned; directly determines the average number of denoising steps.
  • Temperature schedule endpoints tau_max/tau_min = 0.8 / 0.4
    Linear annealing schedule for softmax over denoising time; tuned; affects sampling diversity and convergence behavior.
  • Maximum denoising steps N = 48
    Inference-time budget; average effective ~12 via adaptive stopping; chosen to balance quality and latency.
assumptions (4)
  • standard math Continuous-time Markov chain / discrete flow matching theory (Campbell et al., Gat et al.) correctly describes the denoising process, and the neural network can learn the posterior ℙ(X0|Xt).
    Invoked throughout Section 2; the whole training objective (Eq. 11) rests on this.
  • domain assumption Warm-starting from the post-trained Gemma 4 26B A4B checkpoint transfers capabilities (reasoning, multimodal, long context) into the diffusion model.
    Section 3: inheritance of features is a stated motivation; no ablation shows how much capability comes from initialization vs. diffusion training.
  • domain assumption Entropy-bounded sampling with adaptive stopping reliably signals convergence (low entropy = high-quality tokens) across tasks, not degenerate loops.
    Section 3.3; the paper shows this fails for the SFT model at N=48 (Figures 16-17) and assumes SD·RL fixes it robustly across the eval suite.
  • domain assumption SD·RL online teacher trajectories serve as high-quality references for distillation, and joint reward + entropy minimization does not collapse diversity.
    Section 5; no analysis of diversity collapse beyond conciseness observed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffusionGemma Technical Report." pith.science (2026). https://pith.science/paper/XMEY7REV

@misc{pith2026260800146,
  author       = {Pith},
  title        = {Pith review of: DiffusionGemma Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMEY7REV}},
  note         = {Machine review of arXiv:2608.00146}
}
read the original abstract

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

168 extracted references · 25 linked inside Pith

  1. [1]

    Mastering the game of

    Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others , journal=. Mastering the game of. 2016 , publisher=

    2016

  2. [2]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 technical report , author=. arXiv preprint arXiv:2503.19786 , year=

    arXiv

  3. [3]

    Beyond autoregression: Fast

    Deschenaux, Justin and Gulcehre, Caglar , booktitle=iclr, year=. Beyond autoregression: Fast

  4. [4]

    Song, Yang and Dhariwal, Prafulla and Chen, Mark and Sutskever, Ilya , title =

  5. [5]

    Progressive Distillation for Fast Sampling of Diffusion Models , author=

  6. [6]

    Learnable Sampler Distillation for Discrete Diffusion Models , author=

  7. [7]

    Liu, Jie and Liu, Gongye and Liang, Jiajun and Li, Yangguang and Liu, Jiaheng and Wang, Xintao and Wan, Pengfei and Zhang, Di and Ouyang, Wanli , booktitle=neurips, year=

  8. [8]

    Zheng, Kaiwen and Chen, Huayu and Ye, Haotian and Wang, Haoxiang and Zhang, Qinsheng and Jiang, Kai and Su, Hang and Ermon, Stefano and Zhu, Jun and Liu, Ming-Yu , booktitle=icml, year=

Show all 168 references
  1. [9]

    Training diffusion models with reinforcement learning , author=

  2. [10]

    d1: Scaling reasoning in diffusion large language models via reinforcement learning , author=

  3. [11]

    Reinforcement learning with discrete diffusion policies for combinatorial action spaces , author=

  4. [12]

    Accelerated sampling from masked diffusion models via entropy bounded unmasking , author=

  5. [13]

    Discrete flow matching , author=

  6. [14]

    2026 , howpublished=

    2026

  7. [15]

    Zadouri, Ted and Hoehnerbach, Markus and Shah, Jay and Thakkar, Vijay and Dao, Tri , booktitle =

  8. [16]

    Language models are few-shot learners , author =

  9. [17]

    arXiv preprint arXiv:2312.11805 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open models based on. arXiv preprint arXiv:2403.08295 , year=

    arXiv

  11. [19]

    Deep unsupervised learning using nonequilibrium thermodynamics , author=

  12. [20]

    Denoising diffusion probabilistic models , author=

  13. [21]

    Li, Xiang Lisa and Thickstun, John and Gulrajani, Ishaan and Liang, Percy and Hashimoto, Tatsunori B , booktitle=neurips, year=

  14. [22]

    arXiv preprint arXiv:2211.04236 , year=

    Self-conditioned Embedding Diffusion for Text Generation , author=. arXiv preprint arXiv:2211.04236 , year=

    arXiv

  15. [23]

    2022 , primaryClass=

    Continuous diffusion for categorical data , author=. 2022 , primaryClass=

    2022

  16. [24]

    arXiv preprint arXiv:2506.02070 , year =

    Holderrieth, Peter and Erives, Ezra , title =. arXiv preprint arXiv:2506.02070 , year =

  17. [25]

    PeerJ Computer Science , year=

    Diffusion models in text generation: a survey , author =. PeerJ Computer Science , year=

  18. [26]

    Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall , author=

  19. [27]

    Shabalin, Alexander and Meshchaninov, Viacheslav and Chimbulatov, Egor and Lapikov, Vladislav and Kim, Roman and Bartosh, Grigory and Vetrov, Dmitry , booktitle=aaai, year=

  20. [28]

    Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling , author=

  21. [29]

    Hu, Keya and Qiu, Linlu and Lu, Yiyang and Zhao, Hanhong and Li, Tianhong and Kim, Yoon and Andreas, Jacob and He, Kaiming , journal=

  22. [30]

    Journal of Machine Learning Research , year =

    Lingxiao Zhao and Xueying Ding and Lijun Yu and Leman Akoglu , title =. Journal of Machine Learning Research , year =

  23. [31]

    Structured denoising diffusion models in discrete state-spaces , author=

  24. [32]

    Simple and Effective Masked Diffusion Language Models , author=

  25. [33]

    Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution , author=

  26. [34]

    2026 , howpublished =

    2026

  27. [35]

    2026 , note =

    Gemma 4 Model Card , url =. 2026 , note =

    2026

  28. [36]

    arXiv preprint arXiv:2607.02770 , year=

    Gemma 4 technical report , author=. arXiv preprint arXiv:2607.02770 , year=

    arXiv

  29. [37]

    A Continuous Time Framework for Discrete Denoising Models , author=

  30. [38]

    Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design , author=

  31. [39]

    Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective , author=

  32. [40]

    Mask-Predict: Parallel Decoding of Conditional Masked Language Models , author=

  33. [41]

    arXiv preprint arXiv:2602.23225 , year=

    Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? , author=. arXiv preprint arXiv:2602.23225 , year=

  34. [42]

    Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning , author=

  35. [43]

    The Curious Case of Neural Text Degeneration , author=

  36. [44]

    Zhang, Shimao and Bao, Yu and Huang, Shujian , journal=

  37. [45]

    How Transparent is

    Engels, Joshua and McDougall, Callum and Chughtai, Bilal and Kramar, Janos and Rajamanoharan, Senthoran and Wu, Cindy and Conmy, Arthur and Chen, Asic Q and Tarbouriech, Jean and Ma, Min and O'Donoghue, Brendan and de Oliveira, Jo. How Transparent is. arXiv preprint arXiv:2606...

    arXiv

  38. [46]

    Fast inference from transformers via speculative decoding , author=

  39. [47]

    Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

    Xia, Heming and Yang, Zhe and Dong, Qingxiu and Wang, Peiyi and Li, Yongqi and Ge, Tao and Liu, Tianyu and Li, Wenjie and Sui, Zhifang. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. Findings of the Association for Compu...

    2024

  40. [48]

    2025 , booktitle =

    Zhu, Kan and Gao, Yufei and Zhao, Yilong and Zhao, Liangyu and Zuo, Gefei and Gu, Yile and Xie, Dedong and Tang, Tian and Xu, Qinyu and Ye, Zihao and Kamahori, Keisuke and Lin, Chien-Yu and Wang, Ziren and Wang, Stephanie and Krishnamurthy, Arvind and Kasikci, Baris , title =....

    2025

  41. [49]

    Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua , booktitle=emnlp, year=

  42. [50]

    Zhu, Ruidong and Jiang, Ziheng and Jin, Chao and Wu, Peng and Stuardo, Cesar A. and Wang, Dongyang and Zhang, Xinlei and Zhou, Huaping and Wei, Haoran and Cheng, Yang and Xiao, Jianzhe and Zhang, Xinyi and Liu, Lingjun and Lin, Haibin and Chang, Li-Wen and Ye, Jianxi and Yu, X...

    2025

  43. [51]

    Quest: Query-Aware Sparsity for Efficient Long-Context

    Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , booktitle=icml, year=. Quest: Query-Aware Sparsity for Efficient Long-Context

  44. [52]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  45. [53]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  47. [55]

    Language models are multilingual chain-of-thought reasoners , author=

  48. [56]

    Terry Yue Zhuo and Vu Minh Chien and Jenny Chim and Han Hu and Wenhao Yu and Ratnadira Widyasari and Imam Nur Bani Yusuf and Haolan Zhan and Junda He and Indraneil Paul and Simon Brunner and Chen GONG and James Hoang and Armel Randy Zebaze and Xiaoheng Hong and Wen-Ding Li and...

  49. [57]

    Bowman , booktitle=colm, year=

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , booktitle=colm, year=

  50. [58]

    Tsoukalas, George and Lee, Jasper and Jennings, John and Xin, Jimmy and Ding, Michelle and Jennings, Michael and Thakur, Amitayush and Chaudhuri, Swarat , booktitle=neurips, year=

  51. [59]

    On leakage of code generation evaluation datasets , author=

  52. [60]

    Jain, Naman and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and Solar-Lezama, Armando and Sen, Koushik and Stoica, Ion , booktitle=iclr, year=

  53. [61]

    Yue, Xiang and Zheng, Tianyu and Ni, Yuansheng and Wang, Yubo and Zhang, Kai and Tong, Shengbang and Sun, Yuxuan and Yu, Botao and Zhang, Ge and Sun, Huan and Su, Yu and Chen, Wenhu and Neubig, Graham , booktitle=acl, year=

  54. [62]

    Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with

    Jasper Dekoninck and Nikola Jovanovi. Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with. 3rd AI for Math Workshop at the International Conference on Machine Learning (ICML) , year=

  55. [63]

    Attention is All you Need , year =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser,. Attention is All you Need , year =

  56. [64]

    Gemini Diffusion , year =

  57. [65]

    Accelerating Gemma 4: faster inference with multi-token prediction drafters , year =

  58. [66]

    arXiv preprint arXiv:2508.02193 , year=

    Seed diffusion: A large-scale diffusion language model with high-speed inference , author=. arXiv preprint arXiv:2508.02193 , year=

    arXiv

  59. [67]

    2026 , note =

    Yonggan Fu and Lexington Whalen and Abhinav Garg and Chengyue Wu and Maksim Khadkevich and Nicolai Oswald and Enze Xie and Daniel Egert and Sharath Turuvekere Sreenivas and Shizhe Diao and Chenhan Yu and Ye Yu and Weijia Chen and Sajad Norouzi and Jingyu Liu and Shiyi Lan and ...

    2026

  60. [68]

    arXiv preprint arXiv:2506.17298 , year=

    arXiv

  61. [69]

    Large language diffusion models , author=

  62. [70]

    Bie, Tiwei and Cao, Maosong and Chen, Kun and Du, Lun and Gong, Mingliang and Gong, Zhuochen and Gu, Yanmei and Hu, Jiaqi and Huang, Zenan and Lan, Zhenzhong and others , journal=

  63. [71]

    Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang , booktitle=neurips, year=

  64. [72]

    Chen, Jian and Liang, Yesheng and Liu, Zhijian , booktitle=icml, year=

  65. [73]

    Cheng, Xin and Yu, Xingkai and Shao, Chenze and Li, Jiashi and Xiong, Yunfan and Qian, Yi and Zhu, Jiaqi and Ma, Shirong and Zhang, Xiaokang and Ye, Jiasheng and Chen, Qinyu and Deng, Chengqi and Yu, Jiping and Dai, Damai and Zhang, Zhengyan and Wei, Yixuan and Tan, Yixuan and...

  66. [74]

    Liu, Jingyu and Dong, Xin and Ye, Zhifan and Mehta, Rishabh and Fu, Yonggan and Singh, Vartika and Zhang, Ce and Molchanov, Pavlo , booktitle =

  67. [75]

    Journal of Machine Learning Research , volume=

    A neural probabilistic language model , author=. Journal of Machine Learning Research , volume=

  68. [76]

    Interspeech , volume=

    Recurrent neural network based language model , author=. Interspeech , volume=

  69. [77]

    arXiv preprint arXiv:1308.0850 , year=

    Generating sequences with recurrent neural networks , author=. arXiv preprint arXiv:1308.0850 , year=

    arXiv

  70. [78]

    Sequence to sequence learning with neural networks , author=

  71. [79]

    Non-autoregressive neural machine translation , author=

  72. [80]

    arXiv preprint arXiv:2302.01318 , year=

    Accelerating Large Language Model Decoding with Speculative Sampling , author=. arXiv preprint arXiv:2302.01318 , year=

    arXiv

  73. [81]

    and Chen, Deming and Dao, Tri , title =

    Cai, Tianle and Li, Yuhong and Geng, Zhengyang and Peng, Hongwu and Lee, Jason D. and Chen, Deming and Dao, Tri , title =

  74. [82]

    arXiv preprint arXiv:2412.19437 , year=

    DeepSeek-V3 Technical Report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  75. [83]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    arXiv

  76. [84]

    Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models , author=

  77. [85]

    Argmax flows and multinomial diffusion: Learning categorical distributions , author=

  78. [86]

    2026 , note =

    Crepy, Clement and De Bortoli, Valentin and Galashov, Alexandre and Greff, Klaus and Korshunova, Ira , title =. 2026 , note =

    2026

  79. [87]

    Score-Based Generative Modeling through Stochastic Differential Equations , author=

  80. [88]

    The Diffusion Duality, Chapter

    Deschenaux, Justin and Gulcehre, Caglar and Sahoo, Subham Sekhar , booktitle=iclr, year=. The Diffusion Duality, Chapter

  81. [89]

    Abramovich, Talor and Ashkenazi, Maor and Putterman, Izzy and Chislett, Benjamin and Mitra, Tiyasa and Rouhani, Bita Darvish and Zilberstein, Ran and Geifman, Yonatan , booktitle=icml, year=

  82. [90]

    The First Open Source Diffusion Audio ASR Model , year =

  83. [91]

    arXiv preprint arXiv:2607.01436 , year=

    Discrete Diffusion Language Models for Interactive Radiology Report Drafting , author=. arXiv preprint arXiv:2607.01436 , year=

    arXiv

  84. [92]

    Neither Parallel Nor Sequential: How

    Asaria, Ali and Salomone, Tony and Gandhi, Deep , journal=. Neither Parallel Nor Sequential: How

  85. [93]

    Blockwise Parallel Decoding for Deep Autoregressive Models , author=

  86. [94]

    and Zhang, Hao and Stoica, Ion , booktitle=

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle=. Efficient Memory Management for Large Language Model Serving with

  87. [95]

    2024 , publisher =

    Zhao, Yao and Xie, Zhitian and Liang, Chen and Zhuang, Chenyi and Gu, Jinjie , title =. 2024 , publisher =

    2024

  88. [96]

    Proceedings of Machine Learning and Systems (MLSys) , volume=

    Efficiently Scaling Transformer Inference , author=. Proceedings of Machine Learning and Systems (MLSys) , volume=

  89. [97]

    Dao, Tri and Fu, Daniel and Ermon, Stefano and Rudra, Atri and R

  90. [98]

    Lepikhin, Dmitry and Lee, HyoukJoong and Xu, Yuanzhong and Chen, Dehao and Firat, Orhan and Huang, Yanping and Krikun, Maxim and Shazeer, Noam and Chen, Zhifeng , booktitle=iclr, year=

  91. [99]

    Gale, Trevor and Narayanan, Deepak and Young, Cliff and Zaharia, Matei , booktitle =

  92. [100]

    Reid, Machel and Marrese-Taylor, Edison and Matsuo, Yutaka , booktitle=

  93. [101]

    Zelikman, Eric and Harik, Georges and Shao, Yijia and Jayasiri, Varuna and Haber, Nick and Goodman, Noah D , booktitle=colm, year=. Quiet-

  94. [102]

    Wallace, Bram and Dang, Meihua and Rafailov, Rafael and Zhou, Linqi and Lou, Aaron and Purushwalkam, Senthil and Ermon, Stefano and Xiong, Caiming and Joty, Shafiq and Naik, Nikhil , title =

  95. [103]

    Fan, Ying and Watkins, Olivia and Du, Yuhao and Liu, Hao and Moon, Kuang-Huei and Boutilier, Craig and Abbeel, Pieter and Lin, Bogdan and Lee, Honglak , booktitle=neurips, volume=

  96. [104]

    Directly Fine-Tuning Diffusion Models on Differentiable Rewards , author=

  97. [105]

    Dong, Hanze and Xiong, Wei and Goyal, Deepanshu and Pan, Rui and Diao, Shizhe and Zhang, Jipeng and Shum, Kashun and Zhang, Tong , journal=

  98. [106]

    arXiv preprint arXiv:2302.12192 , year=

    Aligning Text-to-Image Models using Human Feedback , author=. arXiv preprint arXiv:2302.12192 , year=

    arXiv

  99. [107]

    Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference , author=

  100. [108]

    Adversarial Diffusion Distillation , author=

  101. [109]

    Liu, Xingchao and Zhang, Xiwen and Ma, Jianzhu and Peng, Jian and Liu, Qiang , booktitle=iclr, year=

  102. [110]

    One-Step Image Translation with Text-to-Image Models , author=

  103. [111]

    Flow Matching for Generative Modeling , author=

  104. [112]

    Denoising Diffusion Implicit Models , author=

  105. [113]

    Generative Modeling by Estimating Gradients of the Data Distribution , author=

  106. [114]

    Maximum Likelihood Training of Score-Based Diffusion Models , author=

  107. [115]

    Meng, Chenlin and He, Yutong and Song, Yang and Song, Jiaming and Wu, Jiajun and Zhu, Jun-Yan and Ermon, Stefano , booktitle=iclr, year=

  108. [116]

    Liu, Enshu and Ning, Xuefei and Wang, Yu and Lin, Zinan , booktitle=iclr, year=

  109. [117]

    arXiv preprint arXiv:2607.00773 , year=

    Accelerating Discrete Diffusion Models with Parallel-In-Time Sampling , author=. arXiv preprint arXiv:2607.00773 , year=

    arXiv

  110. [118]

    Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms , author=

  111. [119]

    arXiv preprint arXiv:2406.11473 , year=

    Promises, Outlooks and Challenges of Diffusion Language Modeling , author=. arXiv preprint arXiv:2406.11473 , year=

    arXiv

  112. [120]

    Wu, Chengyue and Zhang, Hao and Xue, Shuchen and Diao, Shizhe and Fu, Yonggan and Liu, Zhijian and Molchanov, Pavlo and Luo, Ping and Han, Song and Xie, Enze , journal=

  113. [121]

    Deschenaux, Justin and Gulcehre, Caglar , booktitle=iclr, year=

  114. [122]

    Step-unrolled Denoising Autoencoders for Text Generation , author=

  115. [123]

    Predictor-Corrector Sampling for Discrete Diffusion Models , author=

  116. [124]

    Autoregressive Diffusion Models , author=

  117. [125]

    Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding , author =

  118. [126]

    NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , year=

    Classifier-Free Diffusion Guidance , author=. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , year=

    2021

  119. [127]

    Variational Diffusion Models , author=

  120. [128]

    Chen, Jingye and Huang, Yupan and Lv, Tengchao and Cui, Lei and Chen, Qifeng and Wei, Furu , booktitle=neurips, volume=

  121. [129]

    Video Diffusion Models , author=

  122. [130]

    Journal of Machine Learning Research , volume=

    Cascaded Diffusion Models for High Fidelity Image Generation , author=. Journal of Machine Learning Research , volume=

  123. [131]

    ACM SIGGRAPH 2022 Conference Proceedings , year=

    Palette: Image-to-Image Diffusion Models , author=. ACM SIGGRAPH 2022 Conference Proceedings , year=

    2022

  124. [132]

    arXiv preprint arXiv:2605.11125 , year=

    Language Modeling with Hyperspherical Flows , author=. arXiv preprint arXiv:2605.11125 , year=

    arXiv

  125. [133]

    Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data , author=

  126. [134]

    Arrows of Time for Large Language Models , author=

  127. [135]

    The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More , author=

  128. [136]

    Reverse that number!

    Zhang-Li, Daniel and Lin, Nianyi and Yu, Jifan and Zhang, Zheyuan and Yao, Zijun and Zhang, Xiaokang and Hou, Lei and Zhang, Jing and Li, Juanzi , journal=. Reverse that number!

  129. [137]

    Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction , author=

  130. [138]

    Diffusion of Thought: Chain-of-Thought Reasoning in Diffusion Language Models , author=

  131. [139]

    DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models , author=

  132. [140]

    Pynadath, Patrick and Shi, Jiaxin and Zhang, Ruqi , booktitle=icml, year=

  133. [141]

    Manifold Preserving Guided Diffusion , author=

  134. [142]

    arXiv preprint arXiv:2602.16813 , year=

    Flow Map Language Models: One-step Language Modeling via Continuous Denoising , author=. arXiv preprint arXiv:2602.16813 , year=

    arXiv

  135. [143]

    arXiv preprint arXiv:2604.09784 , year=

    Discrete Flow Maps , author=. arXiv preprint arXiv:2604.09784 , year=

    arXiv

  136. [144]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    arXiv

  137. [145]

    Discrete Diffusion Language Model for Efficient Text Summarization , author=

  138. [146]

    Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling , author=

  139. [147]

    Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete

    Hoogeboom, Emiel and Ruhe, David and Heek, Jonathan and Mensink, Thomas and Salimans, Tim , journal=. Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete. 2026 , doi=

    2026

  140. [148]

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , booktitle=naacl, year=

  141. [149]

    Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin , journal=

  142. [150]

    Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas , journal=

  143. [151]

    Le and Christopher D

    Kevin Clark and Minh-Thang Luong and Quoc V. Le and Christopher D. Manning , booktitle = iclr, year =

  144. [152]

    Edward J Hu and yelong shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=iclr, year=

  145. [153]

    Rajbhandari, Samyam and Li, Conglong and Yao, Zhewei and Zhang, Minjia and Aminabadi, Reza Yazdani and Awan, Ammar Ahmad and Rasley, Jeff and He, Yuxiong , booktitle=icml, year=

  146. [154]

    Toward Efficient Inference for Mixture of Experts , author=

  147. [155]

    Compressive Transformers for Long-Range Sequence Modelling , author=

  148. [156]

    Han, Xiaochuang and Finkelstein, Sachar and Sharma, Shikhar and Chen, Yilun and He, He , booktitle=acl, year=

  149. [157]

    Diffusion

    De Bortoli, Valentin and Thornton, James and Heng, Jeremy and Doucet, Arnaud , booktitle=neurips, volume=. Diffusion

  150. [158]

    arXiv preprint arXiv:2401.17181 , year=

    Transfer learning for text diffusion models , author=. arXiv preprint arXiv:2401.17181 , year=

    arXiv

  151. [159]

    Scaling Diffusion Language Models via Adaptation from Autoregressive Models , author=

  152. [160]

    2026 , month =

    Introducing. 2026 , month =

    2026

  153. [161]

    Quan, Shanghaoran and Yang, Jiaxi and Yu, Bowen and Zheng, Bo and Liu, Dayiheng and Yang, An and Ren, Xuancheng and Gao, Bofei and Miao, Yibo and Feng, Yunlong and others , journal=

  154. [162]

    Challenging

    Suzgun, Mirac and Scales, Nathan and Sch. Challenging. Findings of the Association for Computational Linguistics: ACL 2023 , year=

    2023

  155. [163]

    2024 , howpublished=

    Multilingual Massive Multitask Language Understanding (. 2024 , howpublished=

    2024

  156. [164]

    arXiv preprint arXiv:2406.12045 , year=

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year=

    arXiv

  157. [165]

    Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and Li, Tianle and Ku, Max and Wang, Kai and Zhuang, Alex and Fan, Rongqi and Yue, Xiang and Chen, Wenhu , book...

  158. [166]

    , title =

    Chang, Huiwen and Zhang, Han and Jiang, Lu and Liu, Ce and Freeman, William T. , title =

  159. [167]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. Journal of Machine Learning Research , year =

  160. [168]

    Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Veselin and Zettlemoyer, Luke , title =

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.