Pith. sign in

REVIEW 1 major objections 5 minor 38 references

Training a diffusion language model with extra rollouts from a stronger external policy—shortened to the target's generation length and rewarded separately from the model's own rollouts—improves zero-shot reasoning, with best-of-4 Sudoku ac

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 22:14 UTC pith:AW4GZPZD

load-bearing objection The Sudoku result is real and the framework is worth knowing; the Countdown/MATH500 gains are softer than the abstract claims and largely depend on best-checkpoint selection. the 1 major comments →

arxiv 2608.01717 v1 pith:AW4GZPZD submitted 2026-08-03 cs.LG

Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models

Wonseok Lee , Jimyeong Kim , Jungmin Ko , Wonjong Rhee This is my paper
classification cs.LG
keywords diffusion language modelsexternal rolloutsreinforcement learningverifiable rewardsrollout length controlsource-specific reward processingzero-shot evaluationreasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement learning for diffusion language models usually trains on rollouts that the model being improved generates itself; when the model is weak, successful rollouts are rare and the reward signal is sparse. This paper tries to establish that a better route is to mix in rollouts from a stronger external policy, first shortening them to roughly the target model's generation length, and then scoring the two sources' rewards separately so the external reward does not unbalance the target model's own baseline. The proposed method, ERILS, is tested under zero-shot evaluation on Sudoku, Countdown, and MATH500 and improves multi-sample accuracy on all three, most sharply on Sudoku (98.4% best-of-4 vs 40.3% for the strongest baseline). The paper's central point is that how external rollouts are constructed and how their rewards are processed are design decisions that determine whether the external data helps at all.

Core claim

ERILS builds each training group from four on-policy rollouts of the target dLLM and two length-controlled rollouts from a fixed, stronger autoregressive policy. Rollout Length Control brings external completions near the target's generation length, by prompting for a shorter response or by rewriting an initial long completion. Source-Specific Reward Processing centres on-policy rewards only within the on-policy subgroup, while external rollouts are weighted by their raw verifier reward, and the two contributions are averaged in one mixed-rollout gradient. The paper claims this avoids the training collapse caused by jointly centering a high-reward external group with a low-reward on-policy g

What carries the argument

The load-bearing update is the mixed-rollout gradient g_ERILS, in which on-policy rollouts contribute their group-relative advantage and external rollouts contribute their raw verifier reward, both passed through the same diffusion-compatible surrogate objective (a stand-in for the intractable sequence log-likelihood). Two mechanisms carry the argument: Rollout Length Control, which forces external completions toward the target generation length by instruction or rewriting, and Source-Specific Reward Processing, which decouples the on-policy baseline from external rewards. The paper explains the collapse under joint processing as the external-versus-on-policy reward gap making on-policy adva

Load-bearing premise

The load-bearing premise is that a target diffusion model can be improved by applying its policy-gradient surrogate to completions produced by a different, stronger model, with each external completion weighted by its raw verifier reward and no correction for the difference between the two policies.

What would settle it

Train ERILS on a task where the external policy's completions are correct and length-controlled but the target dLLM's own sampled completions are semantically very distant (e.g., a verifier that only checks the final boxed answer while the reasoning is templated). If deterministic accuracy still rises to around 90%, the raw reward signal alone is carrying the update; if it falls when the external reasoning is scrambled, the content of the external rollouts is load-bearing. Either outcome would test the claim that external rollouts can be incorporated through length control plus source-specific

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Sudoku best-of-4 completion accuracy improves from 40.3% (strongest reproduced on-policy baseline) to 98.4%, and best-of-1 from 25.2% to 91.2%.
  • Deterministic single-completion Sudoku accuracy stays around 90% at generation lengths 128, 256, and 512, while the strongest baseline remains below 30%.
  • Countdown Pass@4 reaches 87.5% versus 76.6% for SPG, and MATH500 reports the highest Pass@k among all compared methods.
  • Length-controlled external rollouts outperform uncontrolled ones, and source-specific reward processing prevents the training collapse observed with joint reward processing.
  • ERILS also beats supervised fine-tuning on the same external rollouts under matched external-rollout usage and matched training steps, and beats external-only training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reported gains hold, external rollouts could serve as a general sparse-reward remedy for dLLM reasoning: any strong model can supply training completions before the target model can solve the task itself.
  • The paper uses a fixed external policy and no importance correction, so the update implicitly assumes external completions are representative; a natural variant would anneal or reweight external data as the target improves, or apply an off-policy correction when one becomes tractable for masked diffusion.
  • Because dLLMs decode from a fixed token budget, length control may matter more fundamentally than prompt engineering suggests; a formal version might normalize rewards by length or constrain generation, which could remove the need for per-task rewrite instructions.
  • The Sudoku result may reflect transferred structure (compact reasoning plus a fixed 16-digit answer format) rather than reward alone; testing whether ERILS transfers across structured-output tasks would separate reward-driven learning from format learning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes ERILS, a method for integrating rollouts from a stronger external autoregressive policy into reinforcement learning (RL) for diffusion large language models (dLLMs). ERILS combines on-policy rollouts from the target dLLM with length-controlled external rollouts, and processes rewards from the two sources separately to avoid the instability of joint reward centering. The target dLLM is updated with a mixed-rollout surrogate objective following SPG. Experiments on Sudoku, Countdown, and MATH500 report large gains on Sudoku and modest gains on Countdown and MATH500 in multi-sample evaluation, with ablations showing the importance of length control and source-specific reward processing.

Significance. If the reported results hold, the paper makes a useful empirical contribution to dLLM RL: it identifies two practical obstacles to off-policy data integration (rollout-length mismatch and cross-source reward coupling), proposes simple, well-motivated fixes, and provides a dramatic Sudoku improvement. The manuscript is unusually transparent: it includes detailed external-rollout construction prompts, filtering statistics, cost analysis, qualitative examples, and a self-critical limitations section. The component analyses (Fig. 4, 5), the comparison with SFT (Fig. 6, 7), and the reward-dynamics analysis (Fig. 10, 12) are valuable. However, the breadth of the headline claim is undermined by the checkpoint-selection protocol and the absence of a validation split, as shown by the paper's own fixed-checkpoint appendix.

major comments (1)
  1. [Section IV-B (Eq. 10) and Appendix I (Limitations)] External rollouts are generated by a fixed Qwen policy and used in the SPG surrogate with A_ext = raw verifier reward, with no importance-ratio or distribution-correction term. As the authors acknowledge in Appendix I, this is an off-policy update without a likelihood correction, so Eq. (10) is not an unbiased estimator of the target policy's gradient. The empirical stability on the three tasks is encouraging, but the manuscript should either provide a supporting analysis (e.g., bounded bias, diagnostics comparing the surrogate gradient on external vs. on-policy rollouts) or explicitly frame ERILS as a heuristic whose justification is entirely empirical, rather than presenting Eq. (10) as the optimization objective without further discussion.
minor comments (5)
  1. [Table 1, Reference [11]] The table lists both 'UniGRPO [11]' and 'SPG [11]' with the same reference. This is confusing; please clarify the relationship between UniGRPO and SPG, or correct the citation.
  2. [Abstract / throughout] 'MA TH500' appears with a spurious space in the abstract and several places in the text; unify as MATH500.
  3. [Appendix G-A (Sudoku example)] The length-controlled example before rewriting has an internal inconsistency: the reasoning states 'Column 2: ... so (4,2)=1' after already assigning (3,2)=2 and (4,2)=?; the text reads as if the generated solution is self-contradictory. This may be a transcription error in the qualitative example, but it undermines the illustrative purpose.
  4. [Section IV-A and Appendix B] The length-control target {target_total_tokens}=256 and the 500-token filtering threshold are free hyperparameters. The component analysis compares length-controlled vs. uncontrolled rollouts but does not vary the target length. At least a brief sensitivity discussion would help.
  5. [Fig. 12] The word 'collapsed' is placed above the reward, advantage, and length panels for joint reward processing, which is ambiguous; consider labeling the panels more explicitly (e.g., 'training collapsed at step ~500').

Circularity Check

0 steps flagged

No significant circularity: ERILS's claimed gains are empirical and not forced by construction or self-citation.

full rationale

The paper's central derivation is not circular. External rollouts are generated once by a fixed stronger autoregressive policy (Qwen3-30B-A3B-Instruct-2507) and are filtered only by completion length, not by reward or correctness (Appendix B-C). The mixed-rollout gradient in Eq. 10 directly uses verifier rewards and the SPG surrogate; nothing in this update is fitted to the reported evaluation numbers. The claim that external rollouts receive higher verifier rewards is measured (Fig. 11), not assumed. Length control and source-specific reward processing are justified by ablations that can and do fail (e.g., joint reward processing collapses in Fig. 5), which is an empirical rather than circular argument. The paper does not rely on self-citations: the dLLM RL baselines (d1, wd1, SPG) and the SPG surrogate are by other author groups, and the released code is external evidence. The acknowledged limitation in Appendix I about the missing importance-ratio correction is a theoretical robustness concern, not a circularity. The best-checkpoint reporting protocol (Section V-A, Appendix D) raises a reproducibility concern, but it does not make the derivation equivalent to its inputs. Overall, the reported improvements are empirical outcomes that could in principle have gone the other way, so there is no circular step to flag.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The method introduces no new theoretical entities, particles, or forces. It depends on the SPG surrogate from prior work, on verifier correctness, and on the external policy being stronger. The main hand-chosen design parameters are the length-control target, the external-rollout group size, and the 500-token filter.

free parameters (3)
  • Length-control target ({target_total_tokens}) = 256
    Set equal to training generation length L_gen; chosen by hand as a practical reference, not fitted to task performance (Appendix B-A).
  • External rollouts per group G_ext = 2
    Selected from sensitivity analysis over {1,2,3,4}; performance is comparable across values, so 2 was chosen as default (Section V-D2).
  • External rollout length filter threshold = 500 tokens
    Rollouts longer than 500 tokens are treated as unusable for length-controlled external rollouts; the threshold is chosen by hand (Appendix B-C).
axioms (4)
  • standard math SPG surrogate S_theta provides a valid, low-bias estimator of the target dLLM's policy gradient
    The paper instantiates S_theta with SPG [11] for both on-policy and external rollouts (Section V-A), relying on SPG's derivation.
  • domain assumption The task verifiers give correct and calibrated rewards
    Verifier implementations are adopted unchanged from SPG [11] (Section V-A, Appendix C-B); correctness of these verifiers is assumed.
  • domain assumption The external policy Qwen3-30B-A3B generates rollouts with higher expected reward than the target's initial on-policy rollouts
    Empirically confirmed in Fig. 11, but the method's benefit depends on this generally; a weaker advisor would not provide the described gains.
  • domain assumption Length-control procedures preserve the correctness of external rollouts
    Supported by Table 5 for Sudoku two-stage rewriting; for Countdown/MATH500 the direct instruction's quality preservation is assumed.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models." pith.science (2026). https://pith.science/paper/AW4GZPZD

@misc{pith2026260801717,
  author       = {Pith},
  title        = {Pith review of: Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AW4GZPZD}},
  note         = {Machine review of arXiv:2608.01717}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent reinforcement learning methods for diffusion large language models (dLLMs) commonly rely on on-policy rollouts generated by the target dLLM itself. When successful on-policy rollouts are scarce, however, on-policy training may receive little positive reward and make only limited progress. To mitigate this problem, we explore incorporating higher-reward rollouts generated by a stronger external policy alongside on-policy rollouts from the target dLLM. However, directly incorporating these external rollouts introduces two practical challenges: differences in rollout length and instability when jointly processing rewards from on-policy and external rollouts. To address these challenges, we propose External Rollout Integration with Length Control and Source-Specific Processing (ERILS), which controls external-rollout length and processes the rewards of on-policy and external rollouts separately. Experiments on Sudoku, Countdown, and MATH500 under zero-shot evaluation show that ERILS improves multi-sample performance across all three tasks, with the largest gains on Sudoku. On Sudoku, ERILS achieves 98.4% best-of-4 completion accuracy, compared with 40.3% for the strongest baseline. ERILS also maintains approximately 90% deterministic single-completion accuracy on Sudoku across generation lengths of 128, 256, and 512 tokens. Our component analysis further shows that length-controlled external rollouts are more effective than uncontrolled external rollouts, and that source-specific reward processing avoids the training collapse observed with joint reward processing. These results show that rollout construction and reward processing are important design dimensions when integrating external rollouts into dLLM reinforcement learning.

Figures

Figures reproduced from arXiv: 2608.01717 by Jimyeong Kim, Jungmin Ko, Wonjong Rhee, Wonseok Lee.

Figure 1
Figure 1. Figure 1: FIGURE 1 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FIGURE 2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: FIGURE 4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FIGURE 5 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: FIGURE 8 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: FIGURE 9 [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: FIGURE 10 [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: FIGURE 11 [PITH_FULL_IMAGE:figures/full_fig_p009_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: reveals distinct on-policy training dynamics un￾der the two reward-processing strategies. Under joint reward processing, the on-policy rewards remain low while the ex￾ternal rollouts maintain high verifier rewards. This reward gap places the on-policy rollouts below the mixed-group reward mean, resulting in consistently negative on-policy advantages. Over the same period, the average on-policy com￾pletion… view at source ↗
Figure 13
Figure 13. Figure 13: FIGURE 13 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: illustrates an improvement in final candidate con￾struction. SPG degenerates into a long sequence of repeated digits and fails to produce a valid 16-digit candidate. In contrast, ERILS produces a complete candidate that matches the reference board and satisfies the required output format. Although the intermediate reasoning contains inconsistent statements, the final candidate is correct; this example the… view at source ↗
Figure 15
Figure 15. Figure 15: FIGURE 15 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: FIGURE 16 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗
Figure 20
Figure 20. Figure 20: shows a case in which both methods return an expression evaluating to 51 instead of the target value 55. ERILS explicitly recognizes that the intermediate result is too low but still returns the same expression without continuing the search. 3) MATH500: Sign Error after Completing the Square [PITH_FULL_IMAGE:figures/full_fig_p021_20.png] view at source ↗
Figure 19
Figure 19. Figure 19: shows another Sudoku failure in which both meth￾ods produce incorrect candidates. ERILS produces a com￾plete candidate but incorrectly concludes that all Sudoku constraints are satisfied, despite a duplicate digit in the first column. 2) Countdown: Accepting a Detected Near Miss [PITH_FULL_IMAGE:figures/full_fig_p021_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 20 canonical work pages

  1. [1]

    S. S. Sahoo, M. Arriola, Y . Schiff, A. Gokaslan, E. M. Marroquin, J. T. Chiu, A. M. Rush, and V . Kuleshov, ‘‘Simple and effective masked diffu- sion language models,’’ inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  2. [2]

    A. Lou, C. Meng, and S. Ermon, ‘‘Discrete diffusion modeling by esti- mating the ratios of the data distribution,’’ inInternational Conference on Machine Learning. PMLR, 2024, pp. 32 819–32 848

    work page 2024

  3. [3]

    J. Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias, ‘‘Simplified and gener- alized masked diffusion for discrete data,’’Advances in neural information processing systems, vol. 37, pp. 103 131–103 167, 2024

    work page 2024

  4. [4]

    S. Gong, S. Agarwal, Y . Zhang, J. Y e, L. Zheng, M. Li, C. An, P . Zhao, W. Bi, J. Hanet al., ‘‘Scaling diffusion language models via adaptation from autoregressive models,’’ inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 5046–5073

    work page 2025

  5. [5]

    S. Nie, F. Zhu, Z. Y ou, X. Zhang, J. Ou, J. Hu, J. Zhou, Y . Lin, J.-R. Wen, and C. Li, ‘‘Large language diffusion models,’’Advances in Neural Information Processing Systems, vol. 38, pp. 50 608–50 646, 2026

    work page 2026

  6. [6]

    F. Zhu, R. Wang, S. Nie, X. Zhang, C. Wu, J. Zhou, Y . Lin, J.-R. Wen, and C. Li, ‘‘Llada 1.5: V ariance-reduced preference optimization for large language diffusion models,’’ inProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), 2026, pp. 11 425–11 460

    work page 2026

  7. [7]

    J. Y e, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong, ‘‘Dream 7b: Diffusion large language models,’’arXiv preprint arXiv:2508.15487, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    S. Zhao, D. Gupta, Q. Zheng, and A. Grover, ‘‘d1: Scaling reasoning in diffusion large language models via reinforcement learning,’’Advances in Neural Information Processing Systems, vol. 38, pp. 56 729–56 762, 2026

    work page 2026

  9. [9]

    X. Tang, R. Dolga, S. Y oon, and I. Bogunovic, ‘‘wd1: Weighted policy optimization for reasoning in diffusion language models,’’arXiv preprint arXiv:2507.08838, 2025

    arXiv 2025

  10. [10]

    Rojas, J

    K. Rojas, J. Lin, K. Rasul, A. Schneider, Y . Nevmyvaka, M. Tao, and W. Deng, ‘‘Improving reasoning for diffusion language models via group diffusion policy optimization,’’arXiv preprint arXiv:2510.08554, 2025

    arXiv 2025

  11. [11]

    C. Wang, P . Rashidinejad, D. Su, S. Jiang, S. Wang, S. Zhao, C. Zhou, S. Z. Shen, F. Chen, T. Jaakkolaet al., ‘‘Spg: Sandwiched policy gradient for masked diffusion language models,’’arXiv preprint arXiv:2510.09541, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Lambert, J

    N. Lambert, J. Morrison, V . Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V . Miranda, A. Liu, N. Dziri, S. Lyuet al., ‘‘Tulu 3: Pushing frontiers in open language model post-training,’’arXiv preprint arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Z. Shao, P . Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., ‘‘Deepseekmath: Pushing the limits of mathematical reasoning in open language models,’’arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    D. Guo, D. Y ang, H. Zhang, J. Song, P . Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Biet al., ‘‘Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,’’arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    J. Y an, Y . Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y . Cheng, and Y . Zhang, ‘‘Learning to reason under off-policy guidance,’’Advances in Neural In- formation Processing Systems, vol. 38, pp. 117 157–117 186, 2026

    work page 2026

  16. [16]

    Arel, ‘‘Arel’s sudoku generator,’’ https://www.ocf.berkeley.edu/ arel/su- doku/main.html, 2025, accessed: 2025-04-08

    work page 2025

  17. [17]

    J. Pan, J. Zhang, X. Wang, L. Y uan, H. Peng, and A. Suhr, ‘‘Tinyzero,’’ https://github.com/Jiayi-Pan/TinyZero, 2025, accessed: 2025-01-24

    work page 2025

  18. [18]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, ‘‘Let’s verify step by step,’’ in International Conference on Learning Representations, vol. 2024, 2024, pp. 39 578–39 601

    work page 2024

  19. [19]

    S. Nie, F. Zhu, C. Du, T. Pang, Q. Liu, G. Zeng, M. Lin, and C. Li, ‘‘Scaling up masked diffusion models on text,’’ inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 82 974–82 997

    work page 2025

  20. [20]

    C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P . Luo, S. Han, and E. Xie, ‘‘Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding,’’arXiv preprint arXiv:2505.22618, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    X. Ma, R. Y u, G. Fang, and X. Wang, ‘‘dkv-cache: The cache for diffusion language models,’’arXiv preprint arXiv:2505.15781, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Z. Liu, Y . Y ang, Y . Zhang, J. Chen, C. Zou, Q. Wei, S. Wang, Y . Zhu, and L. Zhang, ‘‘dllm-cache: Accelerating diffusion large language models with adaptive caching,’’ inF orty-third International Conference on Machine Learning, 2025

    work page 2025

  23. [23]

    Z. Hu, J. Meng, Y . Akhauri, M. S. Abdelfattah, J.-s. Seo, Z. Zhang, and U. Gupta, ‘‘Accelerating diffusion language model inference via efficient kv caching and guided diffusion,’’arXiv e-prints, pp. arXiv–2505, 2025

    work page 2025

  24. [24]

    X. Liu, Y . Song, Z. Liu, Z. Huang, Q. Guo, Z. He, and X. Qiu, ‘‘Longllada: Unlocking long context capabilities in diffusion llms,’’ inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 38, 2026, pp. 32 186–32 194

    work page 2026

  25. [25]

    Arriola, A

    M. Arriola, A. Gokaslan, J. Chiu, Z. Y ang, Z. Qi, J. Han, S. Sahoo, and V . Kuleshov, ‘‘Block diffusion: Interpolating between autoregressive and diffusion language models,’’ inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 50 726–50 753

    work page 2025

  26. [26]

    B. Sun, Y . Cai, M.-H. Y ang, and Y . Wang, ‘‘Blockwise sft for diffusion language models: Reconciling bidirectional attention and autoregressive decoding,’’arXiv preprint arXiv:2508.19529, 2025

    arXiv 2025

  27. [27]

    X. Wang, C. Xu, Y . Jin, J. Jin, H. Zhang, and Z. Deng, ‘‘Diffusion llms can do faster-than-ar inference via discrete diffusion forcing,’’arXiv preprint arXiv:2508.09192, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    J. Y e, S. Gong, L. Chen, L. Zheng, J. Gao, H. Shi, C. Wu, X. Jiang, Z. Li, W. Biet al., ‘‘Diffusion of thought: Chain-of-thought reasoning in diffusion language models,’’Advances in Neural Information Processing Systems, vol. 37, pp. 105 345–105 374, 2024

    work page 2024

  29. [29]

    S. Gong, R. Zhang, H. Zheng, J. Gu, N. Jaitly, L. Kong, and Y . Zhang, ‘‘Diffucoder: Understanding and improving masked diffusion models for code generation,’’arXiv preprint arXiv:2506.20639, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Huang, Z

    Z. Huang, Z. Chen, Z. Wang, T. Li, and G.-J. Qi, ‘‘Reinforcing the diffusion chain of lateral thought with diffusion language models,’’arXiv preprint arXiv:2505.10446, 2025

    arXiv 2025

  31. [31]

    H. Zhao, D. Liang, W. Tang, D. Y ao, and N. Kallus, ‘‘Diffpo: Training diffusion llms to reason fast and furious via reinforcement learning,’’arXiv preprint arXiv:2510.02212, 2025

    arXiv 2025

  32. [32]

    S. Zhao, M. Liu, J. Huang, M. Liu, C. Wang, B. Liu, Y . Tian, G. Pang, S. Bell, A. Groveret al., ‘‘Inpainting-guided policy optimization for diffu- sion large language models,’’arXiv preprint arXiv:2509.10396, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Z. Liu, C. Chen, W. Li, P . Qi, T. Pang, C. Du, W. S. Lee, and M. Lin, ‘‘Un- derstanding r1-zero-like training: A critical perspective,’’arXiv preprint arXiv:2503.20783, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Team, ‘‘Qwen3 technical report,’’ 2025

    Q. Team, ‘‘Qwen3 technical report,’’ 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  35. [35]

    M. Chen, J. Tworek, H. Jun, Q. Y uan, H. P . D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., ‘‘Evaluating large language models trained on code,’’arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  36. [36]

    Z. Chen, R. Lu, A. Zhao, Z. Wang, Y . Y ue, S. Song, and G. Huang, ‘‘Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?’’Advances in Neural Information Processing Systems, vol. 38, pp. 57 654–57 689, 2026. PREPRINT, 2026 23 Wonseok Leeet al.: Beyond On-Policy Exploration: Integrating External Policy Rollouts f...

    work page 2026

  37. [2012]

    24 PREPRINT, 2026

    His general research interests are in the fields of machine learning and representation learning. 24 PREPRINT, 2026

    work page 2026

  38. [2023]

    degree in the Interdisciplinary Program in Artificial Intelli- gence at Seoul National University, Seoul, Korea

    He is currently pursuing his Ph.D. degree in the Interdisciplinary Program in Artificial Intelli- gence at Seoul National University, Seoul, Korea. His current research interests include multimodal large language models and large language models. JIMYEONG KIMreceived the B.S. degree in Mathematics and Computer Science from Sungkyunkwan University, Suwon, ...

    work page 2021

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.