Pith. sign in

REVIEW 5 major objections 3 minor 35 references

A compute-aware evaluation protocol claims that reported gains from remasking strategies in masked diffusion language models are largely artifacts of uncontrolled compute, metric, and stochasticity choices, with temperature alone explaining

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:10 UTC pith:F6K6DN7V

load-bearing objection Useful protocol and honest limitations, but the two headline claims — the compute-matched reversal and the 0.296 interaction — rest on an NFE accounting that contradicts the paper's own Table 3. the 5 major comments →

arxiv 2607.24763 v1 pith:F6K6DN7V submitted 2026-06-04 cs.AI

CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models

Yash Shah , Abhijit Chakraborty , Vivek Gupta This is my paper
classification cs.AI
keywords masked diffusion language modelsremasking strategiescompute-aware evaluationnumber of function evaluationsstochastic unmaskingMAUVEevaluation protocoltemperature sensitivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that current evaluations of masked diffusion language models conflate algorithmic improvements with hidden evaluation choices. It introduces CaRE, a protocol that jointly controls actual number of function evaluations, multiple metrics, and stochasticity. Using this protocol, the authors find that temperature explains the majority of MAUVE variance, compute-matched comparisons reverse several published strategy rankings, and informed remasking interacts harmfully with stochastic unmasking. The central claim is that this interaction—where high-entropy remasking reduces MAUVE by 0.296 at 256 steps and unmask_temp=0.25—is undetectable without controlling all three confounds simultaneously.

Core claim

CaRE's central finding is that informed remasking and stochastic unmasking are in consistent tension. Under joint control of compute, metric, and stochasticity, high-entropy remasking collapses MAUVE by 0.296 relative to no remasking at 256 steps and unmask_temp=0.25 (p=0.020), while temperature alone explains 91% of MAUVE variance (η²=0.91) with a significant strategy×temperature interaction (p=0.002, η²=0.47). The authors also show that compute-matched comparisons reverse several published strategy rankings on both unconditional generation and code generation.

What carries the argument

The central object is the CaRE protocol, a compute-aware evaluation framework built on automatic NFE tracking, multi-metric reporting (PPL, MAUVE, Self-BLEU, Distinct-3), and unmask_temp as a required stochasticity parameter with sweep support. The analysis machinery is a three-way ANOVA separating strategy, step budget, and temperature, with paired t-tests over seeds; token-stability heatmaps and a dose-response ablation identify persistent token churn as the mechanism behind the collapse.

Load-bearing premise

The compute-matched reversal assumes that the no-remasking baseline can be run at actual NFE values of 256 and 512, even though the paper states that none requires exactly 128 forward passes regardless of any nominal step count above 128.

What would settle it

Run the no-remasking strategy at a genuine actual NFE of 256 and 512 without increasing generation length or reintroducing masks—if those runs cannot be realized with gen=128, then Tables 4 and 11 do not represent matched compute. Alternatively, re-run the 256-step, unmask_temp=0.25 comparison with a remasking fraction below 5%; the dose-response prediction is that MAUVE should rise toward the none baseline, while a non-monotonic result would undercut the churn mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, previously published remasking strategy rankings are not trustworthy unless actual NFE, temperature, and metrics are matched.
  • Stochasticity is not a minor tuning detail: changing unmask_temp from 0.0 to 0.1 shifts MAUVE by roughly 0.3, exceeding the gap between any two strategies at a fixed temperature.
  • High-entropy remasking should not be combined with stochastic unmasking at higher compute budgets, since it degrades distributional coverage while preserving surface diversity.
  • The interaction direction holds across twelve open-weight MDLMs of varying scale and architecture, suggesting the finding is not model-specific.
  • Log-likelihood benchmarks like HellaSwag and BBH are stable across the manipulated sampling settings, confirming the effects are confined to sampling behavior rather than parametric knowledge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If temperature truly explains 91% of MAUVE variance, then the field's default practice of comparing strategies at a single temperature is systematically misleading; a temperature sweep should become a required component of any remasking claim.
  • The dose-response ablation suggests a churn threshold below which remasking is benign; a practical extension would be an adaptive remasking strategy that stops re-masking once token stability exceeds a bound, potentially recovering the benefits of informed remasking without the collapse.
  • The same interaction may appear in continuous-state or flow-based diffusion language models, but with a different compute currency; developing an analogy to NFE for those families could be a direct test of whether the tension is a general property of iterative denoising.
  • A testable extension is to predict the MAUVE gap from token-stability statistics alone: if the 3.2× reduction in stable token-step cells is causal, then strategies with equal stability should show equal MAUVE regardless of their remasking policy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper introduces CaRE, a compute-aware evaluation protocol for masked diffusion language models that standardizes actual NFE, enforces multi-metric reporting, and explicitly controls stochasticity. Applied to seven remasking strategies on LLaDA-8B-Base and Dream-7B-Base, the paper claims three findings: temperature explains the majority of MAUVE variance (η²=0.91); compute-matched comparisons reverse several published strategy rankings; and informed remasking (high_entropy) and stochastic unmasking are in tension, with high_entropy reducing MAUVE by 0.296 at 256 nominal steps and unmask_temp=0.25 (p=0.020). The paper also releases a seven-point protocol, an implementation, and a leaderboard covering twelve open-weight MDLMs.

Significance. If the empirical claims were supported, CaRE would be a valuable contribution to MDLM evaluation, analogous to Clean-FID or SacreBLEU. The release of the protocol, implementation, and leaderboard is a concrete strength, as is the explicit attention to NFE tracking, multi-metric reporting, and stochasticity. However, the central findings rest on NFE accounting that contradicts the paper's own compute model and on an interaction test that is not compute-matched. These are load-bearing issues, so the conclusions as stated are not currently established. The framework idea is worthwhile, but the validation presented here does not support the advertised claims.

major comments (5)
  1. [§4.1, Tables 3, 4, 11] Table 3 states that the none strategy 'requires exactly 128 forward passes regardless of any nominal step count above 128.' Yet Table 4 reports none at NFE=256 and NFE=512, and Table 11 reports none* adj. at NFE=437. The footnote about running strategies at different nominal step counts cannot resolve this: with L=128 generated tokens and no remasking, all positions are unmasked after 128 forward passes, so additional nominal steps cannot consume further compute. No mechanism is provided for these rows. The compute-matched reversal, one of the paper's two central contributions, is therefore based on unrealizable configurations under the paper's own NFE model.
  2. [§4.4, Tables 6 and 7] The headline interaction test compares none@256 (actual NFE=128, per Table 3) with high_entropy@256 (actual NFE=437). This violates CaRE's own requirement to equate actual NFE. The 0.296 MAUVE gap and p=0.020 cannot be attributed to a strategy×stochasticity interaction; it may reflect the 3.4× compute advantage rather than the remasking policy. The deterministic compute-matched data in Table 4 show much smaller gaps (0.012 to 0.114), so the stochastic interaction at matched NFE is unestablished. A matched comparison at NFE=128 at t=0.25 is needed; under the paper's NFE model, none cannot be evaluated at NFE>128, so the 256-step claim as formulated cannot be repaired by adding rows.
  3. [Tables 4 and 11] The compute-matched reversal is also statistically weak. Table 4 reports only deterministic t=0.0 data, and while none entries are 3-seed means, random and high_entropy rows are single-seed; no tests or confidence intervals accompany the claim that none dominates at every budget. In Table 11, the compute-matched block shows none* adj. 437 vs. high_entropy 437 differing by only 0.3 percentage points, far below the stated binomial SE of approximately 4.7pp. The nominal-vs-matched contrast is also comparable to that noise. The table therefore does not demonstrate a robust reversal.
  4. [§4.7, Tables 12 and 21] The leaderboard's generalization claim that the interaction direction holds across architectures and scales is not based on a like-for-like protocol. LLaDA-MoE is measured at t=0.1 rather than t=0.25; Dream uses an 'entropy' strategy rather than high_entropy; and PPL is unavailable for Dream-7B-Base. The gap also varies widely, from 0.034 to 0.296. Thus the cross-model comparison mixes different configurations, weakening the claim of a consistent interaction.
  5. [§4.4, Table 8] The text says 'temperature explains the majority of MAUVE variance (η²=0.91)' and later 'strategy×temperature is significant (η²=0.47)'. Table 8 reports partial eta-squared values. Partial η² values do not sum to total variance and can be interpreted differently from proportions of total variance; strategy also has partial η²=0.80. The wording is misleading unless the authors clarify that these are partial effects and provide, if possible, the proportion of total variance attributable to each factor.
minor comments (3)
  1. [Abstract/Introduction] The abstract says 'We present CaRE 2' but the title and rest of the paper use 'CaRE'. Inconsistent naming should be fixed.
  2. [Figure 1] The figure contains the typo 'MOUVE' in the metric list; should be 'MAUVE'.
  3. [Appendix A] The appendix notes that the remaining five strategies are not re-evaluated at the stochastic headline configuration. This is an important limitation and should be stated earlier in the main text, since the abstract's '7 remasking strategies' could imply all were tested under the full protocol.

Circularity Check

0 steps flagged

No significant circularity: the reported gaps and interactions are empirical measurements, not reductions to their own inputs.

full rationale

The paper's central claims—the MAUVE gap of 0.296 at 256 steps/t=0.25, the three-way ANOVA result (η²=0.91 for temperature; strategy×temperature η²=0.47), and the leaderboard interaction direction—are all computed from experimental outputs under a stated protocol. None of these quantities is a fitted parameter later relabeled as a prediction, and no equation in the paper defines the outcome in terms of the input. The NFE accounting is given as a stated characteristic of the strategies (Table 3: none 'requires exactly 128 forward passes regardless of any nominal step count above 128'), not derived from MAUVE or from the reversal claim. The fact that Tables 4 and 11 list none at NFE 256/512/437 despite that stated cap is an internal inconsistency and a serious validity concern, but it is not a circular reduction: the reversal claim is unsupported rather than true-by-construction. There is no load-bearing self-citation chain: the cited prior work is external (metric choices, benchmarks, sampler conventions), and the paper does not invoke an author-owned uniqueness theorem or an unverified ansatz to force its conclusion. Hand-chosen thresholds (ASCII≥0.85, remask fraction 15%, temperature grid) are arbitrary choices, but they were not fit to the target MAUVE gap. The paper's own statistical caveat ('larger seed counts would strengthen marginal claims') is an uncertainty disclosure, not an admission of circularity. The derivation chain is therefore self-contained in the circularity sense; the main risks are correctness of the compute-matched rows and post hoc selection of the headline cell, not definitional equivalence.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central results rest on hand-set thresholds (language filter, remask fraction, MAUVE settings) and an NFE convention that is internally inconsistent. No new theoretical entities are introduced.

free parameters (4)
  • language filter thresholds = ASCII>=0.85, CJK<=0.02
    Hand-chosen thresholds determine which samples are analyzed and could affect MAUVE and other metrics; no sensitivity analysis is provided.
  • high_entropy remask fraction = 0.15 (default); ablation uses 0.05, 0.30
    The fraction of positions remasked per step is a chosen parameter that directly controls the 'churn' driving the interaction.
  • MAUVE K and backbone = K=500, GPT-2-XL
    MAUVE computation parameters are chosen and influence absolute values; the paper acknowledges saturation concerns.
  • seed count = 3 seeds for headline tests
    The statistical power is based on 3 seeds, which yields t(2) tests; single-seed cells appear in Tables 4, 10, and the leaderboard.
axioms (4)
  • standard math The paired t-test and ANOVA assume normality and independence of the 3-seed means.
    Standard statistical assumptions; with only 3 seeds, these are fragile but methodologically unremarkable.
  • domain assumption MAUVE with a GPT-2-XL backbone is a reliable distributional metric for MDLM outputs.
    The paper relies on MAUVE as the primary metric for the interaction finding; the saturation caveat is acknowledged but not resolved.
  • domain assumption The number of forward passes (NFE) accurately represents compute and is comparable across strategies.
    The paper's whole framework treats NFE as the compute measure; this is stated but not independently validated beyond citing prior practice.
  • ad hoc to paper The 'none' strategy's unmasking schedule stops after L forward passes, so NFE is L regardless of nominal steps.
    This is the specific assumption behind Table 3, but it is contradicted by Table 4 where none appears at NFE>L.

pith-pipeline@v1.3.0-alltime-deepseek · 17151 in / 11489 out tokens · 114401 ms · 2026-08-02T12:10:43.560694+00:00 · methodology

0 comments
read the original abstract

Masked diffusion language models (MDLMs) are advancing rapidly, yet the evaluation standards needed to reliably interpret their progress have not kept pace. Despite MDLMs becoming competitive with autoregressive language models, seven recent remasking papers evaluate under incompatible settings, varying nominal step counts, metrics, and sampling temperatures without jointly controlling these factors, rendering their strategy rankings largely incomparable and leaving open whether reported gains reflect algorithmic improvements or evaluation artifacts. We present CaRE, a compute-aware evaluation framework that audits MDLM remasking strategies by standardizing actual number of function evaluations (NFE), enforcing multi-metric reporting, and explicitly controlling stochasticity. Applied to 7 remasking strategies across LLaDA-8B-Base and Dream-7B-Base at 4 stochasticity levels and 3 step budgets on OpenWebText and LM1B, CaRE reveals that: (i) temperature explains the majority of MAUVE variance, (ii) compute-matched comparisons reverse several published strategy rankings, and (iii) informed remasking and stochastic unmasking are in tension, with high-entropy remasking reducing MAUVE by 0.296 at 256 steps at unmask_temp=0.25 (p=0.020). A CaRE leaderboard covering 12 open-weight MDLMs (150M to 8B parameters) shows that this interaction direction holds across architectures and scales. These findings demonstrate that current MDLM evaluations can systematically conflate algorithmic improvements with hidden choices of compute and stochasticity. We release the evaluation protocol, implementation, and leaderboard to ensure future remasking claims are reproducible and comparable.

Figures

Figures reproduced from arXiv: 2607.24763 by Abhijit Chakraborty, Vivek Gupta, Yash Shah.

Figure 1
Figure 1. Figure 1: The evaluation gap: seven remasking papers evaluate under different conditions, producing con￾tradictory rankings. Three confounds (compute, metric, stochasticity) each independently reverse conclusions. Controlling all three jointly reveals their interaction (p=0.020; none vs. high_entropy, LLaDA-8B-Base, OWT, 256 steps, t=0.25, 3 seeds, paired t-test). The evaluation crisis. In the last six months, seven… view at source ↗
Figure 2
Figure 2. Figure 2: LLaDA-8B-Base, OWT. MAUVE vs PPL across step budgets (64/128/256) and tempera [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: MAUVE vs actual NFE across stochasticity levels (LLaDA-8B-Base, OWT, English-filtered). At t=0.0 all strategies cluster low (<0.7). At t ≥ 0.1, none rises to 0.9+; random shows moderate improvement (0.8–0.98 depending on temperature); high_entropy collapses at high NFE despite moderate gains at lower compute. 4.4 The Interaction That Only Joint Control Reveals While none maintains MAUVE >0.92 at all temper… view at source ↗
Figure 4
Figure 4. Figure 4: Token stability heatmap (LLaDA-8B-Base, 128 steps, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: LLaDA-8B-Base, OWT. Generative frontier overlay. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Stochasticity interaction, LLaDA-8B-Base, OWT, 256 steps. (a) MAUVE: stochastic [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Token replacement rate per denoising step (LLaDA-8B-Base, [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Diversity at 256 steps (LLaDA-8B-Base, OWT). (a) Self-BLEU: [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Polar area charts: deterministic vs stochastic (LLaDA-8B-Base, OWT, 128 steps, 3-seed [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Dream-7B MAUVE vs PPL Pareto. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 15 linked inside Pith

  1. [1]

    Where-to-unmask: Ground- truth-guided unmasking order learning for masked diffusion language models.arXiv preprint arXiv:2602.09501, 2026

    Hikaru Asano, Tadashi Kozuno, Kuniaki Saito, and Yukino Baba. Where-to-unmask: Ground- truth-guided unmasking order learning for masked diffusion language models.arXiv preprint arXiv:2602.09501, 2026

    arXiv 2026

  2. [2]

    Prism: Efficient test-time scaling via hierarchical search and self-verification for discrete diffusion language models.arXiv preprint arXiv:2602.01842, 2026

    Jinbin Bai, Yixuan Li, Yuchen Zhu, Yi Xin, Qingyu Shi, Aosong Feng, Xiaohong Liu, Molei Tao, Jianru Xue, Xiangtai Li, et al. Prism: Efficient test-time scaling via hierarchical search and self-verification for discrete diffusion language models.arXiv preprint arXiv:2602.01842, 2026

    Pith/arXiv arXiv 2026

  3. [3]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315–11325, 2022

    2022

  4. [4]

    Brants, Phillip Todd Koehn, and Tony Robinson

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, T. Brants, Phillip Todd Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. InInterspeech, 2013. URLhttps://api.semanticscholar.org/CorpusID: 14136307

    2013

  5. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    A framework for few-shot lan- guage model evaluation.Zenodo, 10256836, 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, et al. A framework for few-shot lan- guage model evaluation.Zenodo, 10256836, 2023. URLhttps://zenodo.org/records/ 10256836

    2023

  7. [7]

    Openwebtext corpus.http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

  8. [8]

    De- mystifying maskgit sampler and beyond: Adaptive order selection in masked diffusion.arXiv preprint arXiv:2510.04525, 2025

    Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. De- mystifying maskgit sampler and beyond: Adaptive order selection in masked diffusion.arXiv preprint arXiv:2510.04525, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Mdpo: Overcoming the training- inference divide of masked diffusion language models.arXiv preprint arXiv:2508.13148, 2025

    Haoyu He, Katrin Renz, Yong Cao, and Andreas Geiger. Mdpo: Overcoming the training- inference divide of masked diffusion language models.arXiv preprint arXiv:2508.13148, 2025

    arXiv 2025

  10. [10]

    The curious case of neural text degeneration.ArXiv, abs/1904.09751, 2019

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.ArXiv, abs/1904.09751, 2019. URLhttps://api.semanticscholar. org/CorpusID:127986954

    Pith/arXiv arXiv 1904

  11. [11]

    Don’t settle too early: Self- reflective remasking for diffusion language models.arXiv preprint arXiv:2509.23653, 2025

    Zemin Huang, Yuhang Wang, Zhiyang Chen, and Guo-Jun Qi. Don’t settle too early: Self- reflective remasking for diffusion language models.arXiv preprint arXiv:2509.23653, 2025

    arXiv 2025

  12. [12]

    Fine-tuning masked diffusion for provable self-correction.arXiv preprint arXiv:2510.01384, 2025

    Jaeyeon Kim, Seunggeun Kim, Taekyun Lee, David Z Pan, Hyeji Kim, Sham Kakade, and Sitan Chen. Fine-tuning masked diffusion for provable self-correction.arXiv preprint arXiv:2510.01384, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Stop training for the worst: Progressive unmasking accelerates masked diffusion training.arXiv preprint arXiv:2602.10314, 2026

    Jaeyeon Kim, Jonathan Geuter, David Alvarez-Melis, Sham Kakade, and Sitan Chen. Stop training for the worst: Progressive unmasking accelerates masked diffusion training.arXiv preprint arXiv:2602.10314, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    A diversity- promoting objective function for neural conversation models

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. A diversity- promoting objective function for neural conversation models. InProceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 110–119, 2016. 10

    2016

  15. [15]

    Holistic evaluation of language models.arXiv preprint arXiv:2211.09110, 2022

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models.arXiv preprint arXiv:2211.09110, 2022

    Pith/arXiv arXiv 2022

  16. [16]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in neural information processing systems, 35:5775–5787, 2022

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in neural information processing systems, 35:5775–5787, 2022

    2022

  17. [17]

    Unmaskfork: Test-time scaling for masked diffusion via deter- ministic action branching.arXiv preprint arXiv:2602.04344, 2026

    Kou Misaki and Takuya Akiba. Unmaskfork: Test-time scaling for masked diffusion via deter- ministic action branching.arXiv preprint arXiv:2602.04344, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data.arXiv preprint arXiv:2406.03736, 2024

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data.arXiv preprint arXiv:2406.03736, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    On aliased resizing and surprising subtleties in gan evaluation

    Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11410–11420, 2022

    2022

  21. [21]

    How efficient are diffusion language models? a critical examination of efficiency evaluation practices.arXiv preprint arXiv:2510.18480, 2025

    Han Peng, Peiyu Liu, Zican Dong, Daixuan Cheng, Junyi Li, Yiru Tang, Shuo Wang, and Wayne Xin Zhao. How efficient are diffusion language models? a critical examination of efficiency evaluation practices.arXiv preprint arXiv:2510.18480, 2025

    arXiv 2025

  22. [22]

    Mauve: Measuring the gap between neural text and human text using divergence frontiers.Advances in Neural Information Processing Systems, 34:4816– 4828, 2021

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers.Advances in Neural Information Processing Systems, 34:4816– 4828, 2021

    2021

  23. [23]

    A call for clarity in reporting bleu scores

    Matt Post. A call for clarity in reporting bleu scores. InProceedings of the third conference on machine translation: Research papers, pages 186–191, 2018

    2018

  24. [24]

    Generative frontiers: Why evaluation matters for diffusion language models.arXiv preprint arXiv:2604.02718, 2026

    Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. Generative frontiers: Why evaluation matters for diffusion language models.arXiv preprint arXiv:2604.02718, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    d3llm: Ultra-fast diffusion llm using pseudo-trajectory distillation.arXiv preprint arXiv:2601.07568, 2026

    Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, and Hao Zhang. d3llm: Ultra-fast diffusion llm using pseudo-trajectory distillation.arXiv preprint arXiv:2601.07568, 2026

    arXiv 2026

  26. [26]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Pith/arXiv arXiv 2010

  27. [27]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Compu- tational Linguistics: ACL 2023, pages 13003–13051, 2023

    2023

  28. [28]

    Remasking discrete diffusion models with inference-time scaling.arXiv preprint arXiv:2503.00307, 2025

    Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and V olodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling.arXiv preprint arXiv:2503.00307, 2025

    arXiv 2025

  29. [29]

    Dare: Diffusion large language models alignment and reinforcement executor.arXiv preprint arXiv:2604.04215, 2026

    Jingyi Yang, Yuxian Jiang, Xuhao Hu, Shuang Cheng, Biqing Qi, and Jing Shao. Dare: Diffusion large language models alignment and reinforcement executor.arXiv preprint arXiv:2604.04215, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Ling- peng Kong. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025. 11

    Pith/arXiv arXiv 2025

  31. [31]

    Hellaswag: Can a machine really finish your sentence? InAnnual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InAnnual Meeting of the Association for Computational Linguistics, 2019. URLhttps://api.semanticscholar.org/CorpusID:159041722

    2019

  32. [32]

    Core: Context-robust remasking for diffusion language models.arXiv preprint arXiv:2602.04096, 2026

    Kevin Zhai, Sabbir Mollah, Zhenyi Wang, and Mubarak Shah. Core: Context-robust remasking for diffusion language models.arXiv preprint arXiv:2602.04096, 2026

    arXiv 2026

  33. [33]

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling.ArXiv, abs/2409.02908, 2024

    Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Mingying Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling.ArXiv, abs/2409.02908, 2024. URLhttps://api.semanticscholar. org/CorpusID:272397565

    Pith/arXiv arXiv 2024

  34. [34]

    dllm: Simple diffusion lan- guage modeling.arXiv preprint arXiv:2602.22661, 2026

    Zhanhui Zhou, Lingjie Chen, Hanghang Tong, and Dawn Song. dllm: Simple diffusion lan- guage modeling.arXiv preprint arXiv:2602.22661, 2026

    arXiv 2026

  35. [35]

    ""Re-mask the top-k highest-entropy positions per step

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. InThe 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097–1100, 2018. Broader Impact CaREis an evaluation framework rather than a new model or capability, so its ...

    2018