pith. sign in

arxiv: 2606.29215 · v2 · pith:NHCKQL3Ynew · submitted 2026-06-28 · 💻 cs.LG · cs.CL

Multi-Block Diffusion Language Models

Yijie Jin , Jiajun Xu , Yuxuan Liu , Chenkai Xu , Yi Tu , Jiajun Li , Dandan Tu , Xiaohui Yan
show 3 more authors
Kai Yu Pengfei Liu Zhijie Deng
This is my paper

Pith reviewed 2026-07-01 07:01 UTC · model grok-4.3

classification 💻 cs.LG cs.CL
keywords Block DiffusionMulti-Block DiffusionDiffusion Language ModelsTeacher ForcingKV CacheParallel DecodingText Generation
0
0 comments X

The pith

Post-training block diffusion LMs on bounded noise groups with randomized schedulers enables multi-block inference that raises average tokens per forward pass from 3.47 to 6.19 while lifting accuracy from 79.95% to 81.03%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Block diffusion language models already support KV caching and flexible lengths under single-block diffusion. Extending them to multi-block diffusion requires the model to handle a running set of consecutive blocks that carry heterogeneous noise levels at inference time. Existing training regimes either show only one noisy block or multiple noisy blocks under fixed visibility patterns, so the learned distribution does not match the states seen at multi-block decode time. The paper closes the gap by post-training with multi-block teacher forcing: each training example presents a clean prefix followed by a bounded group of blocks whose noise levels are drawn from randomized schedulers. The resulting models decode several blocks in parallel, and an accompanying block-buffer decoder reuses prefix caches without changing input shapes.

Core claim

Multi-Block Diffusion Language Models are obtained by post-training existing BD-LMs with Multi-block Teacher Forcing. MultiTF supplies training states that consist of a clean prefix plus a bounded noise-group whose individual blocks receive independent noise schedules; these states are deliberately close to the heterogeneous slot-wise noise patterns that arise when a running set of blocks is decoded concurrently. Combined with a Block Buffer decoding algorithm that preserves KV-cache reuse and static input shapes, the post-trained models achieve higher tokens-per-forward-pass while preserving or improving benchmark accuracy.

What carries the argument

Multi-block Teacher Forcing (MultiTF): training on bounded noise-groups conditioned on clean prefixes using randomized noise-schedulers.

If this is right

  • MBD-LLaDA2-Mini raises average TPF from 3.47 to 6.19 and accuracy from 79.95 percent to 81.03 percent.
  • When further combined with DMax, the same model reaches average TPF of 9.34 with only a 1.02 percent accuracy drop on math and code tasks.
  • The Block Buffer decoder converts the extra parallelism into wall-clock speed-up while keeping input shapes and prefix-cache reuse unchanged.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same post-training recipe could be applied to any diffusion language model that already supports single-block teacher forcing.
  • Further increases in the size of the noise-group or in the number of concurrent blocks would test how far the randomized-scheduler matching can be pushed before accuracy degrades.
  • Because the method only modifies the training distribution and adds a decoding buffer, it can be combined with other inference-time accelerations such as speculative decoding or quantization.

Load-bearing premise

Training states produced by bounded noise-groups and randomized schedulers are close enough to the heterogeneous noise patterns encountered at multi-block inference time that the post-trained weights transfer without large distribution shift.

What would settle it

Measure accuracy and tokens-per-forward-pass on the same math and code benchmarks when the post-trained model is run with actual multi-block inference using a running window of four or more blocks; a drop larger than 2-3 percent would indicate the training-inference mismatch was not bridged.

read the original abstract

Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation. A natural next step is to extend them from Single-Block Diffusion (SingleBD) to Multi-Block Diffusion (MultiBD), where a running-set of consecutive blocks is decoded concurrently for inter-block parallelism. However, existing BD-LMs are mostly trained under teacher forcing, where the model observes only one noisy block conditioned on a clean prefix. While the recent diffusion forcing strategy introduces visibility among multiple noisy blocks, its training states still differ from MultiBD inference, where decoding operates on a bounded running-set with heterogeneous slot-wise noise patterns. To bridge this gap, we propose Multi-Block Diffusion Language Models (MBD-LMs), obtained by post-training BD-LMs with Multi-block Teacher Forcing (MultiTF). MultiTF integrates teacher forcing and diffusion forcing by training on bounded noise-groups conditioned on clean prefixes, with randomized noise-schedulers that better match MultiBD inference states. To make MultiBD practically executable, we further introduce an optimized decoding algorithm based on the Block Buffer mechanism that preserves prefix-cache reuse, keeps input shapes static, and translates increased decoding parallelism into wall-clock acceleration. Empirically, MBD-LLaDA2-Mini increases average Tokens Per Forward pass (TPF) from 3.47 to 6.19 and improves average accuracy from 79.95% to 81.03%; when combined with DMax, MBD-LLaDA2-Mini-DMax reaches an average TPF of 9.34 with only a 1.02% accuracy drop on math and code benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes Multi-Block Diffusion Language Models (MBD-LMs) by post-training existing Block Diffusion LMs with Multi-block Teacher Forcing (MultiTF), which trains on bounded noise-groups conditioned on clean prefixes using randomized noise-schedulers to better align training states with Multi-Block Diffusion (MultiBD) inference. It further introduces an optimized Block Buffer decoding algorithm that preserves prefix-cache reuse and static input shapes. Empirically, MBD-LLaDA2-Mini raises average Tokens Per Forward pass (TPF) from 3.47 to 6.19 while lifting accuracy from 79.95% to 81.03%; combining with DMax yields 9.34 TPF at a 1.02% accuracy cost on math and code benchmarks.

Significance. If the training-to-inference transfer holds, the approach could meaningfully increase decoding parallelism and wall-clock throughput for diffusion language models on math and code tasks with only marginal accuracy impact. The work supplies concrete before-and-after TPF and accuracy figures on named benchmarks and builds directly on prior BD-LM and diffusion-forcing baselines.

major comments (2)
  1. [Abstract] Abstract: the central claim that randomized noise-schedulers in MultiTF produce states sufficiently close to MultiBD inference (heterogeneous slot-wise noise patterns determined by the Block Buffer trajectory) is load-bearing for the reported TPF gains (3.47→6.19 and 9.34), yet the manuscript provides no distribution comparison, moment matching, or ablation that removes randomization to test this assumption.
  2. [Abstract] Abstract: the accuracy and TPF improvements are reported as single point estimates without error bars, number of random seeds, or dataset-split details, so it is impossible to assess whether the 1.02% accuracy drop or the 0.08% accuracy gain are statistically distinguishable from noise.
minor comments (1)
  1. The abstract does not separate the contribution of the randomized schedulers from that of the Block Buffer algorithm, making it harder to attribute the TPF gains.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and will incorporate revisions to provide stronger empirical support and statistical reporting.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that randomized noise-schedulers in MultiTF produce states sufficiently close to MultiBD inference (heterogeneous slot-wise noise patterns determined by the Block Buffer trajectory) is load-bearing for the reported TPF gains (3.47→6.19 and 9.34), yet the manuscript provides no distribution comparison, moment matching, or ablation that removes randomization to test this assumption.

    Authors: We agree that the alignment between MultiTF training states and MultiBD inference states is central to the claimed gains, and that direct validation would strengthen the work. The randomized noise-schedulers were designed to produce heterogeneous slot-wise noise patterns that approximate the Block Buffer trajectory, but the current manuscript does not include distribution comparisons, moment matching, or an ablation removing randomization. We will add these elements in the revision, including an ablation study and quantitative comparison of noise distributions between training and inference. revision: yes

  2. Referee: [Abstract] Abstract: the accuracy and TPF improvements are reported as single point estimates without error bars, number of random seeds, or dataset-split details, so it is impossible to assess whether the 1.02% accuracy drop or the 0.08% accuracy gain are statistically distinguishable from noise.

    Authors: We acknowledge that single-point estimates without variability measures limit assessment of statistical significance. The reported figures were obtained from single runs with fixed seeds for reproducibility. We will revise the manuscript to report means and standard deviations over multiple random seeds (at least three), specify the number of seeds, and clarify the dataset splits and evaluation protocol used for the math and code benchmarks. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper presents MBD-LMs via post-training with MultiTF (bounded noise-groups + randomized schedulers) and a Block Buffer decoding algorithm. All reported gains (TPF 3.47→6.19, accuracy 79.95%→81.03%, and DMax variant) are framed as direct empirical measurements against prior BD-LM baselines. No equations, fitted parameters, or predictions are shown that reduce these outcomes to definitions internal to the paper. The similarity assumption between MultiTF training states and MultiBD inference is stated but does not enter any derivation that would make results tautological. The chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The abstract introduces no new free parameters, invented entities, or non-standard axioms; it relies on the standard diffusion modeling assumption that teacher-forcing on noisy blocks can be extended to bounded multi-block groups.

axioms (2)
  • domain assumption Diffusion language models can be trained by conditioning on clean prefixes while observing noisy blocks.
    This is the baseline teacher-forcing setup referenced as the starting point for BD-LMs.
  • domain assumption Randomized noise-schedulers on bounded noise-groups produce training states close enough to heterogeneous multi-block inference states for post-training to transfer.
    This is the explicit bridging claim for MultiTF.

pith-pipeline@v0.9.1-grok · 5858 in / 1227 out tokens · 38228 ms · 2026-07-01T07:01:08.709660+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 22 canonical work pages · 11 internal anchors

  1. [1]

    2022 , eprint=

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , author=. 2022 , eprint=

    2022

  2. [6]

    International Conference on Learning Representations (ICLR) , year =

    Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  3. [14]

    arXiv preprint arXiv:2512.15596 , year =

    Corrective Diffusion Language Models , author =. arXiv preprint arXiv:2512.15596 , year =

    work page arXiv

  4. [20]

    Evaluating Large Language Models Trained on Code

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [21]

    Program Synthesis with Large Language Models

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [22]

    Advances in Neural Information Processing Systems , volume=

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation , author=. Advances in Neural Information Processing Systems , volume=

  7. [25]

    2025 , howpublished =

    Python Code Dataset 500k , author =. 2025 , howpublished =

    2025

  8. [26]

    Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

    Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2503.09573. Oral Presentation

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [27]

    LLaDA2.0: Scaling Up Diffusion Language Models to 100B

    Tiwei Bie, Zenan Huang, Chongxuan Li, et al. Llada2.0: Scaling up diffusion language models to 100b. arXiv preprint arXiv:2512.15745, 2025. URL https://arxiv.org/abs/2512.15745

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [28]

    Llada2.1: Speeding up text diffusion via token editing

    Tiwei Bie et al. Llada2.1: Speeding up text diffusion via token editing. arXiv preprint arXiv:2602.08676, 2026. URL https://arxiv.org/abs/2602.08676

    work page arXiv 2026

  11. [29]

    Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation

    Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El-Haddad, C \'e line Hudelot, and Pierre Colombo. When does reasoning matter? a controlled study of reasoning's contribution to model performance. arXiv preprint arXiv:2509.22193, 2025. URL https://arxiv.org/abs/2509.22193

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [30]

    dparallel: Learnable parallel decoding for dllms

    Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. dparallel: Learnable parallel decoding for dllms. arXiv preprint arXiv:2509.26488, 2025. URL https://arxiv.org/abs/2509.26488

    work page arXiv 2025

  13. [31]

    DMax: Aggressive Parallel Decoding for dLLMs

    Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. Dmax: Aggressive parallel decoding for dllms. arXiv preprint arXiv:2604.08302, 2026. URL https://arxiv.org/abs/2604.08302

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  14. [32]

    Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation

    Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and Bowen Zhou. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. arXiv preprint arXiv:2510.06303, 2025. URL https://arxiv.org/abs/2510.06303

    work page arXiv 2025

  15. [33]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  16. [34]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  17. [35]

    Lightningrl: Breaking the accuracy--parallelism trade-off of block-wise dllms via reinforcement learning

    Yanzhe Hu, Yijie Jin, Pengfei Liu, Kai Yu, and Zhijie Deng. Lightningrl: Breaking the accuracy--parallelism trade-off of block-wise dllms via reinforcement learning. arXiv preprint arXiv:2603.13319, 2026. URL https://arxiv.org/abs/2603.13319

    work page arXiv 2026

  18. [36]

    Python code dataset 500k

    jtatman . Python code dataset 500k. Hugging Face dataset, 2025. URL https://huggingface.co/datasets/jtatman/python-code-dataset-500k

    2025

  19. [37]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36: 0 21558--21572, 2023

    2023

  20. [38]

    Adablock-dllm: Semantic-aware diffusion llm inference via adaptive block size

    Guanxi Lu, Hao Mark Chen, Yuto Karashima, Zhican Wang, Daichi Fujiki, and Hongxiang Fan. Adablock-dllm: Semantic-aware diffusion llm inference via adaptive block size. arXiv preprint arXiv:2509.26432, 2026. URL https://arxiv.org/abs/2509.26432

    work page arXiv 2026

  21. [39]

    Veomni: Scaling any modality model training with model-centric distributed recipe zoo

    Qianli Ma, Yaowei Zheng, Zhelun Shi, Zhongkai Zhao, Bin Jia, Ziyue Huang, Zhiqi Lin, Youjie Li, Jiacheng Yang, Yanghua Peng, Zhi Zhang, and Xin Liu. Veomni: Scaling any modality model training with model-centric distributed recipe zoo. arXiv preprint arXiv:2508.02317, 2025. URL https://arxiv.org/abs/2508.02317

    work page arXiv 2025

  22. [40]

    Large Language Diffusion Models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025. URL https://arxiv.org/abs/2502.09992

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [41]

    d3llm: Ultra-fast diffusion llm using pseudo-trajectory distillation

    Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, and Hao Zhang. d3llm: Ultra-fast diffusion llm using pseudo-trajectory distillation. arXiv preprint arXiv:2601.07568, 2026. URL https://arxiv.org/abs/2601.07568

    work page arXiv 2026

  24. [42]

    Chiu, Alexander Rush, and Volodymyr Kuleshov

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07524, 2024. URL https://arxiv.org/abs/2406.07524

    work page arXiv 2024

  25. [43]

    Diffusion llms can do faster-than-ar inference via discrete diffusion forcing

    Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09192, 2025. URL https://arxiv.org/abs/2508.09192

    work page arXiv 2025

  26. [44]

    Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

    Chengyue Wu et al. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025. URL https://arxiv.org/abs/2505.22618

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [45]

    Lopa: Scaling dllm inference via lookahead parallel decoding

    Chenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, and Zhijie Deng. Lopa: Scaling dllm inference via lookahead parallel decoding. arXiv preprint arXiv:2512.16229, 2025. URL https://arxiv.org/abs/2512.16229

    work page arXiv 2025

  28. [46]

    Dream 7B: Diffusion Large Language Models

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. URL https://arxiv.org/abs/2508.15487

    work page internal anchor Pith review Pith/arXiv arXiv 2025