Pith. sign in

REVIEW 4 major objections 5 minor 33 cited by

The paper argues that reinforcement learning for diffusion language models should be applied to the model's own decoding trajectory rather than to randomly masked sequences, and demonstrates that this trajectory-aware objective produces str

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 22:50 UTC pith:OWBKCNRA

load-bearing objection TraceRL is a genuinely useful idea—optimizing RL along the model's own decoding trace—with solid empirical gains, but the long-CoT flagship needs an off-threshold eval and the MATH500 inconsistency fixed. the 4 major comments →

arxiv 2509.06949 v1 pith:OWBKCNRA submitted 2025-09-08 cs.CL

Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

Yinjie Wang , Ling Yang , Bowen Li , Ye Tian , Ke Shen , Mengdi Wang This is my paper
classification cs.CL
keywords diffusion language modelsreinforcement learningtrajectory-aware RLmasked diffusionchain-of-thought reasoningblock diffusionvalue modelreasoning benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Diffusion language models generate text by unmasking tokens in parallel steps, but existing post-training methods train the policy on randomly masked views of the output, which do not match how the model actually decodes. TraceRL instead treats the decoding process as a trajectory—each batch of tokens unmasked at a step becomes a training timestep—and applies a PPO-style policy update over that trace, with a diffusion-based value model providing step-wise baselines. On math and coding tasks, models trained this way (the TraDo series) improve steadily and outperform autoregressive models of comparable or larger size, including the first long chain-of-thought diffusion model. The paper concludes that aligning the reinforcement learning objective with the inference trajectory is the key to effective post-training of diffusion language models.

Core claim

TraceRL's central claim is that the standard random-masking objective used to post-train masked diffusion language models is misaligned with the confidence-based, left-to-right decoding those models actually perform, and that optimizing the policy over its own preferred inference trace fixes the mismatch. The method collects rollouts as traces, shrinks each trace by grouping neighboring steps to cut compute, and uses a diffusion-based value model with token-wise and step-wise returns and generalized advantage estimation to stabilize training. It applies to both full-attention and block-attention architectures; block-attention models slice the trace into per-block training units for parallel

What carries the argument

The object that carries the argument is the preferred inference trajectory, the ordered sequence of token sets that the policy itself unmasks during parallel decoding, written as τ = (τ(1), …, τ(|τ|)). TraceRL converts this trajectory into a reinforcement learning objective: advantages are computed per trace step, optionally aggregated by a shrinkage parameter s to reduce forward passes, then assigned to the tokens unmasked at that step, and the policy is updated with a clipped importance ratio and a KL penalty. A diffusion-based value model predicts token-wise values conditioned on the prefix, producing step-wise GAE baselines for variance reduction. For block-attention models, the trace is

Load-bearing premise

The argument rests on assuming that the model's own confidence-based unmasking order is the right sequence to reward, and on measuring gains with the same decoding threshold used during training—if either assumption fails, the advantage over random-mask RL could shrink.

What would settle it

Run TraceRL against a control where each rollout's unmasked token sets are randomly permuted (same rewards, same policy), or evaluate TraDo at dynamic-sampling thresholds other than T=0.9; if the accuracy gap over random-masking RL vanishes or inverts, the trajectory alignment claim is not what drives the results.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Random-mask-based post-training for diffusion language models can be replaced by trajectory-conditioned RL, so the rollouts themselves supply the training signal and no mask augmentation is needed.
  • Diffusion language models can match or exceed the math and coding reasoning of 7B-scale autoregressive models at 4B/8B scale, and can produce long chain-of-thought outputs, a capability previously limited to autoregressive models.
  • Block-diffusion models can be retrained to larger block sizes, giving users a speed-versus-accuracy knob without losing performance.
  • The diffusion value model stabilizes training curves and supports process-level rewards, enabling finer credit assignment than a single terminal verifiable reward.
  • The released open-source framework makes TraceRL reproducible on full-attention and block-attention models with accelerated KV-cache inference for both RL and deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If trajectory alignment is the operative mechanism, a direct test would be to shuffle the order of unmasked token sets within each rollout while keeping rewards and policy identical; the accuracy gain over random-mask RL should shrink or disappear if the ordering is what matters.
  • The paper evaluates with the same dynamic-sampling threshold (T=0.9) used during RL rollouts; evaluating across other thresholds would separate genuine reasoning gains from calibration to a specific decoding rule.
  • The same trajectory-aware objective could generalize to any iterative decoder with a defined token-ordering, including block-wise autoregressive hybrids, so TraceRL may become a standard post-training recipe beyond masked diffusion language models.
  • A diffusion value model trained on step-level process rewards could enable trace-level credit assignment for long reasoning chains, an extension the paper leaves for future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs), which optimizes the policy over the model's own inference trace (the sequence of unmasking steps) rather than over random masking objectives. A diffusion-based value model is introduced to provide step-wise advantages and reduce training variance. The authors apply TraceRL to both full-attention and block-attention DLMs, yielding the TraDo series: TraDo-4B/8B-Instruct and the first long-CoT DLM, TraDo-8B-Thinking. They report consistent improvements over SDAR baselines on math and coding benchmarks, with the instruction models surpassing strong autoregressive baselines such as Qwen2.5-7B-Instruct on several tasks. The paper also releases an open-source framework for training and deploying DLMs, including KV-cache acceleration and multiple RL/SFT methods. A theoretical proposition (Proposition 1) derives token-wise returns and advantages from step-wise GAE recursions; the derivation is self-contained and appears correct.

Significance. If the results hold, TraceRL is a valuable contribution to post-training of diffusion language models: it explicitly aligns the RL objective with the inference trajectory, which is a plausible and largely unexplored principle for DLMs. The value model formulation is clean and the open-source framework is a practical asset for reproducibility. The instruction-model gains (e.g., TraDo-8B-Instruct improving over SDAR-8B-Chat by +4.2 static / +4.8 dynamic on MATH500) are consistent under both static and dynamic sampling, which lends credibility to the core method. However, the flagship long-CoT claim rests on a model evaluated only under the same dynamic sampling protocol used in training, and there is an unresolved numeric inconsistency in the reported MATH500 score. These issues need to be addressed before the headline claims can be fully accepted.

major comments (4)
  1. [Section 5.3 / Table 2 / Figure 1] There is an unresolved numeric inconsistency for the flagship long-CoT model. Section 5.3 states TraDo-8B-Thinking achieves 85.8% on MATH500, whereas Table 2 and Figure 1 report 87.4%. The abstract's 18.1% relative gain over Qwen2.5-7B-Instruct (74.0%) corresponds to 87.4%, not 85.8%. Please correct whichever value is wrong and ensure all occurrences, including the abstract and Figure 1, are consistent.
  2. [Section B.3 / Section 5.1.2] The long-CoT model is evaluated only with dynamic sampling at T=0.9, the same strategy used during RL rollouts (Section 5.1.2). This train/eval alignment is a deliberate feature of TraceRL (Section 3.2), but it means the reported gains for TraDo-8B-Thinking could in part reflect specialization to this exact decoding protocol rather than general reasoning ability. Please add evaluations under static sampling and at least one alternative threshold (e.g., T=0.8 or T=0.95) on MATH500 and LiveCodeBench-V2, or provide a principled argument why dynamic-only evaluation is the right protocol. The instruction models already report static and dynamic numbers, which mitigates the concern there; the long-CoT model currently does not.
  3. [Section 5.2 / Table 2] The sentence 'TraDo-4B-Instruct outperforms strong autoregressive baselines such as Qwen2.5-7B-Instruct across all math tasks' is contradicted by Table 2 under dynamic sampling on MATH500: TraDo-4B-Instruct scores 71.8 vs Qwen2.5-7B-Instruct's 74.0. Please qualify the claim by sampling mode (static) or identify the specific setting in which 'all math tasks' holds.
  4. [Section B.5] The RL training retains only tasks with policy accuracy between 0.2 and 0.8. This accuracy-based filtering, applied on the policy being trained, may bias the training distribution toward tasks of intermediate difficulty. Please report how many of the 8K MATH and 6K coding tasks are filtered, and discuss (or ablate) the effect on the reported benchmark improvements. Without this, the generalization claims are harder to interpret.
minor comments (5)
  1. [Table 2 caption and row] The caption says 'The long-CoT model TraDo-8B-Instruct here is evaluated by dynamic sampling' but the row is TraDo-8B-Thinking. Also the row has five numbers while the table has five metrics with Static/Dynamic pairs; clarify which sampling mode(s) apply.
  2. [Section 5.9 / Table 4] 'Acceleration' is defined as response length divided by total sampling steps; this is an average tokens-per-step measure, not an acceleration ratio. Rename to 'tokens per step' or define acceleration relative to a baseline.
  3. [Introduction] Typo: 'emergyed' should be 'emerged'.
  4. [Table 1] Define 'trace semi-ar' and 'fully random' in the table or text; the current labels are unclear without reading Section B.1.
  5. [Related Work / Experiments] The paper cites d1 (Zhao et al., 2025), a diffusion reasoning model trained with RL, but does not compare against it. A row in Table 2 or a discussion would strengthen the long-CoT and RL claims.

Circularity Check

0 steps flagged

No significant circularity: TraceRL's RL objective and value-model derivation are self-contained; the train/eval decoding match is a modeling choice, not a circular reduction.

full rationale

The paper's central derivation is self-contained. Section 4.2 defines step-wise returns and GAE advantages from the stated recursions, and Proposition 1 derives token-wise forms without fitting any constant to the target benchmarks. Equation (3) is a standard clipped-importance PPO objective applied to decoding traces; no parameter is fitted to the reported accuracy gains. Section 3.2's 'preferred inference traces' are the policy's own static-sampling trajectories, and Table 1 reports an empirical comparison of training objectives, not a definitional equivalence: the claim that this trace is 'optimal' is an empirical observation, not an input. The use of the same dynamic sampling threshold (T=0.9) during RL rollouts and evaluation (Sections 5.1.2 and B.3) is a legitimate experimental-design concern that could inflate apparent gains, especially for the long-CoT model, but it is not a mathematical circularity because the model could plausibly fail under that protocol; moreover, static-sampling results are reported for the instruction models. The internal inconsistency between 85.8 (Section 5.3) and 87.4 (Table 2) for MATH500 is a correctness/reporting risk, not a circular step. Self-citations (e.g., MMaDA, ReasonFlow) are used for baselines or related work, not to supply the load-bearing argument, and no uniqueness theorem or ansatz is smuggled in via citation. Thus the derivation chain is not circular.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The only new component is a diffusion value network, a conventional RL baseline estimator, not an invented physical or natural entity. The ledger records the hand-chosen thresholds and hyperparameters that shape the empirical results.

free parameters (5)
  • Task retention accuracy window = [0.2, 0.8]
    Only RL training tasks where the policy's accuracy lies in [0.2, 0.8] are kept (Appendix B.5); thresholds are chosen by hand and can inflate measured gains.
  • Shrinkage parameter s = 8 (full-attention coding)
    Groups every s neighboring trace steps to cut forward passes; set to 8 in the full-attention experiment, chosen by the authors.
  • GAE/discount parameters gamma, lambda = 1.0, 1.0
    Set as default in Section 5.1.2; not fitted but are hyperparameters of the advantage estimator.
  • KL coefficient beta and PPO clip epsilon = 0.01, 0.2
    Standard PPO hyperparameters chosen by hand.
  • Dynamic sampling threshold T = 0.9 (block models)
    Unmasking threshold used in rollouts and evaluation; affects trajectory shape and results.
axioms (4)
  • standard math The masked diffusion training objective in Eq. (2) is a valid ELBO for the data likelihood.
    Taken from cited prior work (Ou et al., Sahoo et al., Shi et al.) and used as the starting point for the RL derivation.
  • domain assumption Confidence-based static/dynamic unmasking approximates the model's preferred inference trajectory.
    Sections 2.4 and 3.2 define preferred traces via these decoding rules; the paper does not compare against alternative trajectory definitions.
  • domain assumption Verifiable rewards (binary math correctness, unit-test pass rate) are unbiased targets for RL.
    Reward is treated as ground truth without reward-model calibration.
  • domain assumption Block-attention slicing preserves the per-token loss when forwarding each block once.
    Section 4.3 asserts each slice can be forwarded once using block attention with no loss in objective fidelity.

pith-pipeline@v1.3.0-alltime-deepseek · 21844 in / 14004 out tokens · 138839 ms · 2026-08-04T22:50:43.874740+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models." pith.science (2026). https://pith.science/paper/OWBKCNRA

@misc{pith2026250906949,
  author       = {Pith},
  title        = {Pith review of: Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWBKCNRA}},
  note         = {Machine review of arXiv:2509.06949}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We propose TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that incorporates preferred inference trajectory into post-training, and is applicable across different architectures. Equipped with a diffusion-based value model that enhances training stability, we demonstrate improved reasoning performance on complex math and coding tasks. Besides, it can also be applied to adapt block-specific models to larger blocks, which improves sampling flexibility. Employing TraceRL, we derive a series of state-of-the-art diffusion language models, namely TraDo. Although smaller than 7B-scale AR models, TraDo-4B-Instruct still consistently outperforms them across complex math reasoning tasks. TraDo-8B-Instruct achieves relative accuracy improvements of 6.1% over Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct on mathematical reasoning benchmarks. Through curriculum learning, we also derive the first long-CoT DLM, outperforming Qwen2.5-7B-Instruct on MATH500 with an 18.1% relative accuracy gain. To facilitate reproducible research and practical applications, we release a comprehensive open-source framework for building, training, and deploying diffusion LLMs across diverse architectures. The framework integrates accelerated KV-cache techniques and inference engines for both inference and reinforcement learning, and includes implementations of various supervised fine-tuning and RL methods for mathematics, coding, and general tasks. Code and Models: https://github.com/Gen-Verse/dLLM-RL

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 33 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

    cs.LG 2026-06 conditional novelty 7.0

    HERALD overlaps CPU-side sparse KV-cache selection with GPU-side block-denoising to hide offloading latency, achieving up to 2.28–2.47x decode throughput over dense GPU serving at 5–10% KV budgets.

  2. Learning from the Self-future: On-policy Self-distillation for dLLMs

    cs.CL 2026-06 unverdicted novelty 7.0

    d-OPSD reframes on-policy self-distillation for dLLMs via suffix conditioning from self-generated answers and step-level supervision, outperforming RLVR and SFT on reasoning benchmarks with ~10% of the optimization steps.

  3. Unified Energy for Invariant and Independent Decoding in Diffusion Language Models

    cs.CL 2026-06 unverdicted novelty 7.0

    The paper introduces Uni-E, a unified energy for DLMs that accounts for model capacity, dependency and invariance, can be computed exactly, and corrects distribution shifts from dependency and invariance.

  4. Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models

    cs.LG 2026-05 conditional novelty 7.0

    TraFL applies trajectory flow balancing to post-train diffusion language models, preventing mode collapse and delivering consistent gains on reasoning tasks that hold under increased sampling.

  5. Relative Score Policy Optimization for Diffusion Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    RSPO interprets reward advantages as targets for relative log-ratios in dLLMs, calibrating noisy estimates to stabilize RLVR training and achieve strong gains on planning tasks with competitive math reasoning performance.

  6. TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM

    cs.CL 2026-05 unverdicted novelty 7.0

    TAD improves the accuracy-parallelism trade-off in diffusion LLMs via temporal-aware self-distillation that applies hard labels to soon-to-be-decoded tokens and soft supervision to future tokens.

  7. MemDLM: Memory-Enhanced DLM Training

    cs.CL 2026-03 unverdicted novelty 7.0

    MemDLM embeds a simulated denoising trajectory into DLM training via bi-level optimization, creating a parametric memory that improves convergence and long-context performance even when the memory is dropped at test time.

  8. Improving Sampling for Masked Diffusion Models via Information Gain

    cs.CL 2026-02 unverdicted novelty 7.0

    Info-Gain Sampler improves MDM decoding by using bidirectional information gain to reduce cumulative uncertainty, outperforming greedy samplers on reasoning accuracy and creative writing tasks.

  9. DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels

    cs.LG 2026-02 conditional novelty 7.0

    DICE, a family of diffusion LLMs trained with a bi-phase infilling-to-generation RL curriculum, achieves state-of-the-art CUDA kernel generation at data and token budgets far below prior specialized autoregressive models.

  10. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

    cs.CL 2026-01 conditional novelty 7.0

    Arbitrary-order generation lowers the reasoning Pass@k of diffusion LLMs; constraining RL to autoregressive order (JustGRPO) improves accuracy and preserves parallel decoding.

  11. From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

    cs.LG 2026-07 conditional novelty 6.0

    Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...

  12. AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters

    cs.LG 2026-07 conditional novelty 6.0

    AdaFlash improves diffusion-drafter speculative decoding with on-policy reverse-KL distillation and an online adaptive length head, sustaining acceleration under high concurrency.

  13. Trace-Based On-Policy Distillation for Masked Diffusion Language Models

    cs.CL 2026-07 conditional novelty 6.0

    TOPD distills a stronger diffusion LLM into a weaker one on the student's own denoising trajectories and matches a reward-trained 4B model on MATH500 with 4x fewer rollouts.

  14. Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters

    cs.AI 2026-07 unverdicted novelty 6.0

    Accept-Until-Fail training improves average accepted block length in speculative decoding from 2.40 to 2.61 by limiting cross-entropy support to the drafter's first predicted failure point.

  15. Diffusion-GR2: Diffusion Generative Reasoning Re-ranker

    cs.IR 2026-07 unverdicted novelty 6.0

    Diffusion-GR2 converts an AR reasoning re-ranker to block-diffusion via CFT, OPD, and RL stages, recovering near-parity accuracy on Amazon Beauty with 2.4-3.5x decode speedup.

  16. Diffusion-GR2: Diffusion Generative Reasoning Re-ranker

    cs.IR 2026-07 conditional novelty 6.0

    CFT + on-policy distillation + RL converts an AR reasoning re-ranker into a block-diffusion model that recovers near-AR accuracy at 2.4–3.5× decode throughput on Amazon Beauty.

  17. Diffusion-GR2: Diffusion Generative Reasoning Re-ranker

    cs.IR 2026-07 conditional novelty 6.0

    CFT plus on-policy distillation and RL converts an AR reasoning re-ranker into a block-diffusion model that matches AR accuracy at 2.4–3.5× higher throughput on Amazon Beauty.

  18. HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

    cs.LG 2026-06 unverdicted novelty 6.0

    HERALD enables near-lossless accuracy at 5-10% KV budget for block dLLMs by amortizing top-k selection across denoising steps and overlapping CPU-GPU retrieval, yielding up to 2.47x higher throughput than GPU-only inference.

  19. VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination

    cs.CL 2026-06 unverdicted novelty 6.0

    VoidPadding decouples padding from termination in MDLMs via a new [VOID] token, delivering +17.84 average benchmark points and 55.7% fewer decoding steps on Dream-7B-Instruct.

  20. Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    AGDO improves dLLM reasoning performance by determining denoising order and emphasizing tokens based on attention-derived dependencies rather than random masking.

  21. Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    PAPO improves reasoning performance in diffusion LLMs by converting sparse terminal rewards into dense step-wise credit and replaying real high-uncertainty trajectories, reporting gains up to 42.2% on Countdown.

  22. Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

    cs.CL 2026-06 unverdicted novelty 6.0

    On-policy distillation from a frozen autoregressive teacher to a bidirectional student eliminates train-inference mismatch and enables data-efficient ARLM-to-DLM conversion.

  23. GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

    cs.LG 2026-05 unverdicted novelty 6.0

    GDSD reduces RL for dLLMs to likelihood-free self-distillation via a normalization-free logit-matching objective, outperforming ELBO methods with more stable training on LLaDA-8B and Dream-7B.

  24. Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

    cs.CL 2026-05 unverdicted novelty 6.0

    Fast-dDrive is a block-diffusion VLA that reports SOTA accuracy on WOD-E2E and nuScenes driving benchmarks together with 12x throughput over autoregressive baselines via section scaffolds and test-time averaging.

  25. Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models

    cs.AI 2026-05 unverdicted novelty 6.0

    Proposes HT-GRPO with sketch-then-paint staged updates, prompt-conditioned importance ratios, and hierarchical credit assignment for dMLLMs, reporting gains on GenEval and DPG plus quality metrics.

  26. Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization

    cs.LG 2025-10 conditional novelty 6.0

    DMPO improves diffusion LLM reasoning by matching the model's distribution to the reward-tilted target via importance-weighted denoising cross-entropy plus weight-baseline subtraction, beating GRPO-style baselines on ...

  27. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    cs.LG 2026-07 conditional novelty 5.5

    SNLP reduces encrypted Transformer nonlinear depth from L sequential stages to (L−N)+K, cutting symbolic bootstraps ~2.65× with lower error amplification than sequential inference.

  28. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    cs.LG 2026-07 conditional novelty 5.0

    SNLP reduces symbolic FHE bootstraps from 53 to 20 on a 0.5B model with +1.2% PPL degradation and lower polynomial-error amplification than sequential inference.

  29. Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Models

    cs.CL 2026-06 unverdicted novelty 5.0

    DIA is a training-free method that dynamically adjusts anchor positions in diffusion LLMs to improve format compliance and accuracy on reasoning benchmarks like GSM8K and MATH.

  30. Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

    cs.CL 2026-05 unverdicted novelty 5.0

    Fast-dDrive is a block-diffusion VLA that reports SOTA ADE on WOD-E2E, 0.32 m L2 on nuScenes, and 12x throughput over AR baselines via section scaffolds and test-time rollout averaging.

  31. FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion

    cs.CV 2026-02 conditional novelty 5.0

    Block-external attention is stable across diffusion steps, so caching and reusing it while recomputing only block-internal attention speeds up long-context generation with limited quality loss.

  32. T$^\star$: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learning

    cs.CL 2026-01 conditional novelty 5.0

    A progressive RL-based curriculum, T⋆, doubles block sizes in masked diffusion LMs while keeping or improving math reasoning accuracy.

  33. Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generation

    cs.SE 2026-05 unverdicted novelty 4.0

    Static checking rewards and moderate AST-based hints improve diffusion RL performance for code generation, with effectiveness varying by task difficulty across HumanEval, MBPP, and LiveCodeBench.

Reference graph

Works this paper leans on

39 extracted references · 5 canonical work pages · cited by 28 Pith papers

  1. [1]

    Arriola, A

    M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573,

    Pith/arXiv arXiv

  2. [5]

    Accessed: 2024-07-24. A. Graves, R. K. Srivastava, T. Atkinson, and F. Gomez. Bayesian flow networks.arXiv preprint arXiv:2308.07037,

    Pith/arXiv arXiv 2024

  3. [7]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    D.Hendrycks, C.Burns,S.Kadavath, A.Arora,S.Basart, E.Tang, D.Song,andJ.Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  4. [9]

    Hosseini, X

    A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-star: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457,

    Pith/arXiv arXiv

  5. [10]

    14 Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H.-Y. Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025a. Z. Hu, J. Meng, Y. Akhauri, M. S. Abdelfattah, J.-s. Seo, Z. Zhang, a...

    Pith/arXiv arXiv

  6. [11]

    URLhttps: //github.com/huggingface/open-r1. S. Jaghouar, J. M. Ong, M. Basra, F. Obeid, J. Straube, M. Keiblinger, E. Bakouch, L. Atkins, M. Panahi, C. Goddard, et al. Intellect-1 technical report.arXiv preprint arXiv:2412.01152,

    Pith/arXiv arXiv

  7. [12]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974,

    Pith/arXiv arXiv

  8. [13]

    Jiang, J

    P. Jiang, J. Lin, L. Cao, R. Tian, S. Kang, Z. Wang, J. Sun, and J. Han. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.arXiv preprint arXiv:2503.00223,

    Pith/arXiv arXiv

  9. [14]

    B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Pith/arXiv arXiv

  10. [15]

    J. Kim, K. Shah, V. Kontonis, S. Kakade, and S. Chen. Train for the worst, plan for the best: Under- standing token ordering in masked diffusions.arXiv preprint arXiv:2502.06768,

    Pith/arXiv arXiv

  11. [16]

    Mercury: Ultra-fast language models based on diffusion.arXiv preprint arXiv:2506.17298,

    I.Labs, S.Khanna, S.Kharbanda, S.Li, H.Varma, E.Wang, S.Birnbaum, Z.Luo, Y.Miraoui, A.Palrecha, et al. Mercury: Ultra-fast language models based on diffusion.arXiv preprint arXiv:2506.17298,

    Pith/arXiv arXiv

  12. [18]

    Z. Liu, Y. Yang, Y. Zhang, J. Chen, C. Zou, Q. Wei, S. Wang, and L. Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching.arXiv preprint arXiv:2506.06295,

    Pith/arXiv arXiv

  13. [19]

    Notion Blog. X. Ma, R. Yu, G. Fang, and X. Wang. dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781,

    Pith/arXiv arXiv

  14. [20]

    American invitational mathematics examination (aime) 2024: Aime i and aime ii.https://artofproblemsolving

    15 Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models Mathematical Association of America, American Mathematics Competitions. American invitational mathematics examination (aime) 2024: Aime i and aime ii.https://artofproblemsolving. com/wiki/index.php/AIME_Problems_and_Solutions,

    work page 2024

  15. [21]

    Competition problems used as an evaluation dataset; original problems by MAA AMC. S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen, and C. Li. Large language diffusion models.arXiv preprint arXiv:2502.09992,

    Pith/arXiv arXiv

  16. [22]

    J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data.arXiv preprint arXiv:2406.03736,

    Pith/arXiv arXiv

  17. [23]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  18. [25]

    Sui, Y.-N

    Y. Sui, Y.-N. Chuang, G. Wang, J. Zhang, T. Zhang, J. Yuan, H. Liu, A. Wen, S. Zhong, H. Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419,

    Pith/arXiv arXiv

  19. [26]

    K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    Pith/arXiv arXiv

  20. [27]

    X. Wang, C. Xu, Y. Jin, J. Jin, H. Zhang, and Z. Deng. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025a. Y. Wang, L. Yang, Y. Tian, K. Shen, and M. Wang. Co-evolving llm coder and unit tester via reinforce- ment learning.arXiv preprint arXiv:2506.03136, 2025b. J. Wei, X. Wang, D. Schuurma...

    Pith/arXiv arXiv

  21. [28]

    White, S

    C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Naidu, et al. Livebench: A challenging, contamination-free llm benchmark.arXiv preprint arXiv:2406.19314,

    Pith/arXiv arXiv

  22. [29]

    C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie. Fast-dllm: Training- free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618,

    Pith/arXiv arXiv

  23. [30]

    16 Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models A

    URLhttps://hkunlp.github.io/blog/2025/dream-coder. 16 Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024a. C. Yang, H. J. Kang, J. Shi, and D. Lo. Acecode: A reinforc...

    Pith/arXiv arXiv 2025

  24. [31]

    URLhttps: //hkunlp.github.io/blog/2025/dream. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025a. R. Yu, X. Ma, and X. Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding.arXiv pre...

    Pith/arXiv arXiv 2025

  25. [32]

    S. Zhao, D. Gupta, Q. Zheng, and A. Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning.arXiv preprint arXiv:2504.12216,

    Pith/arXiv arXiv

  26. [33]

    Zheng, Y

    K. Zheng, Y. Chen, H. Mao, M.-Y. Liu, J. Zhu, and Q. Zhang. Masked diffusion models are se- cretly time-agnostic masked models and exploit inaccurate categorical sampling.arXiv preprint arXiv:2409.02908,

    Pith/arXiv arXiv

  27. [34]

    F. Zhu, R. Wang, S. Nie, X. Zhang, C. Wu, J. Hu, J. Zhou, J. Chen, Y. Lin, J.-R. Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223,

    Pith/arXiv arXiv

  28. [35]

    J. Zou, L. Yang, J. Gu, J. Qiu, K. Shen, J. He, and M. Wang. Reasonflux-prm: Trajectory-aware prms for long chain-of-thought reasoning in llms.arXiv preprint arXiv:2506.18896,

    arXiv

  29. [36]

    thinking

    17 Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models A. Theoretical Results Proposition 1(Token-wise return and advantage from step-wise recursions).Let a trajectory𝜏 be partitioned into trace steps𝜏(1),...,𝜏(|𝜏|) , and let𝑡𝑗 denote the unique step index with𝑗∈𝜏(𝑡 𝑗). For token-wise rewards𝑟𝑗 and token-wise values𝑉old 𝑗 ...

    work page 2025

  30. [37]

    With dynamic sampling, we use a threshold ofT= 0.9 and 𝑡𝑜𝑝-𝑘= 0(i.e., all tokens are kept)

    For the SDAR and TraDo instruction models, we keep the pretrained block size of 4, a response limit of 2000, and a temperature of 1.0. With dynamic sampling, we use a threshold ofT= 0.9 and 𝑡𝑜𝑝-𝑘= 0(i.e., all tokens are kept). For static sampling, we set𝑡𝑜𝑝-𝑘= 1, following (Cheng et al., 2025). For the long-CoT model TraDo-8B-Thinking, we set the response...

    work page 2000

  31. [39]

    By default, we use the𝑘= 3estimator for KL

    (Hu et al., 2025a). By default, we use the𝑘= 3estimator for KL. For math tasks, we use binary outcomes as verifiable rewards and retain only those tasks with accuracy between 0.2 and 0.8 for training (Yu et al., 2025a). For coding tasks, we use as the reward the proportion of unit tests passed by the generated solutions. To accelerate training, we use 64 ...

    work page 2025

  32. [1024]

    We employ static decoding (one token per step) to enhance sampling quality (Gong et al., 2025), using the KV-cache

    During each step, we sample 56 problems, with 8 responses generated for each problem. We employ static decoding (one token per step) to enhance sampling quality (Gong et al., 2025), using the KV-cache. For the SDAR models, we use the default block size of 4, dynamic decoding with thresholdT= 0.9, 𝑡𝑜𝑝-𝑘= 0, temperature 1.0, and𝑡𝑜𝑝-𝑝= 1.0(also applied durin...

    work page 2025

  33. [2019]

    Y. Song, Z. Zhang, C. Luo, P. Gao, F. Xia, H. Luo, Z. Li, Y. Yang, H. Yu, X. Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference.arXiv preprint arXiv:2508.02193,

    Pith/arXiv arXiv

  34. [2020]

    F. Hong, G. Yu, Y. Ye, H. Huang, H. Zheng, Y. Zhang, Y. Wang, and J. Yao. Wide-in, narrow-out: Revokable decoding for efficient and effective dllms.arXiv preprint arXiv:2507.18578,

    arXiv

  35. [2021]

    Dieleman, L

    S. Dieleman, L. Sartran, A. Roshannai, N. Savinov, Y. Ganin, P. H. Richemond, A. Doucet, R. Strudel, C. Dyer, C. Durkan, et al. Continuous diffusion for categorical data.arXiv preprint arXiv:2211.15089,

    Pith/arXiv arXiv

  36. [2022]

    URLhttps://www.science.org/ doi/10.1126/science.abq1158

    doi: 10.1126/science.abq1158. URLhttps://www.science.org/ doi/10.1126/science.abq1158. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations,

    doi:10.1126/science.abq1158

  37. [2023]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  38. [2024]

    S. Gong, R. Zhang, H. Zheng, J. Gu, N. Jaitly, L. Kong, and Y. Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation.arXiv preprint arXiv:2506.20639,

    Pith/arXiv arXiv

  39. [2025]

    URL https://github.com/JetAstra/SDAR. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv