Pith. sign in

REVIEW 4 major objections 5 minor 67 references

AURORA-LM generates text by denoising a high-capacity continuous latent, and reports quality that matches or beats discrete-token language models on free generation and summarization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 03:17 UTC pith:VE7JNUQG

load-bearing objection Coherent and thoroughly ablated method whose central contribution stands, but the headline matched-protocol superiority claims rest on a broken Cola-DLM evaluation: WinoGrande at 5.7% for a 1.8B model is a format mismatch, not a capability gap. the 4 major comments →

arxiv 2608.02602 v1 pith:VE7JNUQG submitted 2026-08-03 cs.CL

AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

Jiajun Liang , Yucheng Liao , Yukang Cao , Jiazhe Wei , Ken Li , Wende Tan , Jiankun Zhang , ZY Cui
show 7 more authors
Jingkang Yang Liucheng Guo Shiqi Yang B. Yang Caifeng Shan Ziwei Liu Chenyang Si
This is my paper
classification cs.CL
keywords continuous latent diffusionflow matchingtext generationautoencoderblock-causalself-trajectory consistencynoise scheduleOpenWebText
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to show that language can be generated directly in a continuous latent space without sacrificing token-level fidelity. It argues that previous continuous language models either reused embedding spaces not built for joint generation and decoding, or compressed autoencoded latents to make diffusion easier at the cost of exact word recovery. AURORA-LM instead keeps a high-capacity, prefix-aligned latent built by a query-based encoder–decoder, and trains a block-causal diffusion transformer to model that latent's distribution directly via flow matching. On OpenWebText free generation and XSum summarization, the authors report the strongest scores among evaluated continuous and diffusion-based language models, and a 1B-parameter version outperforms a larger publicly released latent-diffusion language model on nine benchmarks under a matched protocol. A sympathetic reader would care because it suggests continuous spaces can serve as a general interface for text, not just images, video, and audio.

Core claim

The central claim is that the tension between a latent that is easy to generate and a latent that decodes accurately is not a necessary trade-off. AURORA-LM resolves it by making the representation the fixed point: the autoencoder is trained, frozen, and never compressed; the diffusion model adapts instead. Three specific mechanisms carry this: a low-rank bottleneck on the noisy-input pathway only (so the full-width clean target is preserved), a noise-level distribution calibrated to the latent width (wider latents get more high-noise training mass), and self-trajectory consistency, which aligns clean-latent predictions before and after one Euler step along the sampling trajectory. With thes

What carries the argument

The Query-based Encoder-Decoder builds a prefix-aligned latent: position i reads tokens up to ⌈iL/N⌉, so latent prefixes mirror text prefixes. The Block-causal Diffusion Transformer then learns this latent's distribution with flow matching, factorizing the prior into blocks and denoising positions within a block in parallel. The load-bearing mechanisms are the low-rank noisy-input bottleneck (projection with intermediate width Db, applied only to the corrupted state while the output head predicts the full D-dimensional clean latent), the width-calibrated noise schedule (tan-d with increasing d for wider latents), and self-trajectory consistency (a loss aligning clean-latent predictions at co

Load-bearing premise

The headline scaling result against the larger public latent-diffusion model rests on the assumption that the shared nine-task benchmark protocol reads both models' outputs fairly; if the answer matcher misreads the baseline's output format, as the 5.7% WinoGrande score (chance is 50%) suggests, the comparison collapses.

What would settle it

Re-run the nine-task benchmark on the released baseline checkpoint using a permissive answer matcher that accepts any normalized paraphrase or the model's own decoding preferences, and compare with AURORA-LM-L under the same matcher; if the baseline's WinoGrande score jumps from 5.7% to near 50% while AURORA-LM's scores remain flat, the matched-protocol claim is falsified. Alternatively, train a 130M AURORA-LM with the width-calibrated noise schedule replaced by a fixed schedule: if MAUVE does not decline when latent width increases, the width-calibration claim is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Continuous-latent generation becomes a viable alternative to next-token prediction, so language modeling is no longer necessarily token-only.
  • The recipe — full-width prediction target, restricted noisy-input pathway, width-calibrated noise, and trajectory consistency — can be evaluated independently of the autoencoder, since the encoder is frozen before the denoiser is trained.
  • Block-causal factorization offers a tunable middle ground: block size trades sequential reasoning against parallel generation, and small blocks can be chosen when quality matters most.
  • Scaling a latent-diffusion denoiser to 1B parameters keeps improving benchmark performance, suggesting that continuous latent language models benefit from scale similarly to discrete ones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark comparison against the larger latent-diffusion model is the most protocol-sensitive result: the baseline scores 5.7% on WinoGrande where random choice is 50%, which is far below any plausible model ability gap and suggests the shared answer matcher may misread its output style — so the 'surpasses a larger model' headline may owe more to protocol than to raw capability.
  • If self-trajectory consistency is a generic trajectory-level regularizer, it should transfer to other few-step generative settings, including image and audio latent diffusion, where consistency along solver trajectories is also beneficial.
  • The width-calibrated noise schedule yields a testable monotonicity prediction: the optimal high-noise mass (Pr(σ>0.7)) should increase with latent dimensionality for any autoencoder-based diffusion model.
  • Freezing the autoencoder before training the denoiser means the same latent space could be shared across downstream tasks or modalities, potentially simplifying multimodal generation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. AURORA-LM is a two-stage continuous-latent diffusion language model. A query-based encoder–decoder maps text into a fixed-length, causally ordered latent sequence while retaining enough capacity for exact token decoding; a block-causal flow-matching denoiser then models this latent distribution. The paper’s main technical contributions are the low-rank noisy-input bottleneck with a full-width clean-latent prediction target, width-calibrated high-noise training allocation, and a self-trajectory consistency regularizer. The authors report state-of-the-art results among evaluated diffusion/continuous models on OpenWebText free generation and XSum summarization, and claim that a 1B model outperforms the larger publicly released Cola-DLM on a nine-task benchmark suite under a matched protocol. The internal derivations and ablations are coherent and extensive, but the external comparison claims depend on baseline evaluation protocols that are not yet convincingly fair.

Significance. If the comparative results survive a fair evaluation harness, this paper is significant: it demonstrates that a high-capacity, decoder-facing continuous latent need not be compressed for diffusion to model it, and it offers a transferable recipe (input-pathway bottleneck, width-calibrated noise, trajectory consistency) for latent diffusion text generation. The manuscript is also strong on transparency: it provides code, detailed training configurations, seed-level inference sweeps, and thorough controlled ablations (Tables 11–16; Figs. 5–7). The internal algebra is consistent—the Euler update in Eq. (13) follows from the linear interpolant in Eq. (8), and Eq. (25) correctly reproduces m0.7 = 0.795 for d = 7 in Table 14. However, the paper’s headline comparative claims are currently undercut by apparent evaluation artifacts, especially the Cola-DLM WinoGrande score of 5.7% on a two-choice task. The significance is therefore conditional on baseline fairness being established.

major comments (4)
  1. [§4.3, Table 4, A.2.4/A.2.9] The matched-protocol claim is load-bearing for the scaling headline, and it is undermined by Cola-DLM’s 5.7% on WinoGrande, a two-choice task where chance is 50%. This is far below chance and characteristic of output-format mismatch, not capability. The shared first-newline truncation and answer matcher (A.2.9) may misread Cola-DLM’s generations, which were produced with release defaults but constrained by AURORA-LM’s prompt and parser. Please report Cola-DLM under its native decoding and answer-extraction pipeline, or demonstrate that the shared matcher does not discard or mislabel its answers. Without a repaired protocol, the abstract’s claim of ‘surpassing a larger publicly released latent-diffusion language model’ is not established.
  2. [§4.2.1, Table 2, A.2.5] The final OpenWebText configuration for AURORA-LM was selected by a Gen-PPL/MAUVE sweep over the same held-out OpenWebText distribution used in the final comparison (Table 7), while baselines were not swept under an equivalent protocol. This conflates a genuine method advantage with test-time selection. The comparison should use a validation split held out from the final evaluation, or report all configurations (including baseline inference sweeps) with the same selection procedure. This is needed before Table 2 can support the claim of ‘strongest performance among evaluated’ models.
  3. [§4.2.2, Table 3, A.2.7] The XSum comparison in Table 3 is not matched, despite the paper’s ‘matched data and evaluation protocols’ claim. The footnote states that all † values were collected by Hu et al. [23], so AURORA-LM-S is compared against published numbers, not against baselines run through the same sampler, scorer, and truncation pipeline. The paper should either reproduce these baselines in-house or explicitly qualify the XSum claim as ‘outperforms published ELF-reported numbers.’ As written, the table’s mixed provenance does not support the strong comparative conclusion.
  4. [§4.2.1, Table 2, A.2.7] The ELF-B reproduction reports MAUVE 0.229 with Gen-PPL 24.11, far outside the range of every other baseline (0.668–0.851) and far below ELF’s own published results. Since ELF-B is the closest continuous-flow baseline, a MAUVE this low strongly suggests an under-tuned or incompatible reproduction harness (e.g., tokenizer or sampler mismatch) rather than a genuine model deficiency. A broken ELF-B reproduction inflates AURORA-LM’s relative strength. Please provide ELF-B’s published OWT numbers, the exact sample-generation settings used, and a sanity-check experiment showing the reproduction is faithful.
minor comments (5)
  1. [Table 5 vs §3.2] The architecture table uses ‘Latent length M’ while the text defines the latent sequence length as N = round(cL). Please align the notation (M is used nowhere else in the method section).
  2. [Table 3] The row ‘E2D2†’ is not described or cited anywhere else in the paper; if it is a distinct baseline, it needs a reference and a short description. The following row ‘SeqDiffuSeq [64]’ is also not discussed in the related-work section.
  3. [Figure 1] The caption says radial values are normalized per metric, but the normalization formula is not given. State whether it is min–max scaling, z-scoring, or something else, so the reader can interpret the relative sizes.
  4. [A.2.4] The text says Cola-DLM is evaluated with ‘release defaults’ and then specifies per-sample inference-noise base seed 66. A fixed seed is a choice, not a release default; clarify whether this seed is from the Cola-DLM repository and report sensitivity to the seed.
  5. [Table 9] The AR row lists ‘Sampling steps 1024 tokens,’ which is confusing: for an autoregressive model the number of sampling steps is the number of tokens generated. Please relabel the column header or the entry to distinguish sequence length from scheduler steps.

Circularity Check

0 steps flagged

No significant circularity: the architecture and objectives are independently ablated, with no load-bearing self-citation or definitional reduction.

full rationale

AURORA-LM's central derivation chain is self-contained. The autoencoder is trained with token-level cross-entropy (Eq. 4) and then frozen; the denoiser is trained with flow matching (Eq. 11) and self-trajectory consistency (Eq. 14), both anchored to external clean-latent targets or EMA predictions rather than to the final evaluation metric. The design choices (low-rank noisy-input bottleneck, clean-latent prediction, width-calibrated noise allocation, block-causal generation) are each validated by controlled ablations in Sec. 4.1 with MAUVE against held-out OpenWebText references, not assumed from a self-citation. The paper cites JiT, ELF, BD3-LMs, COSMOS and Cola-DLM as external prior work; none of these are self-citations by the present authors, and no uniqueness theorem or prior-work ansatz is used as the load-bearing justification. The 'noise calibration to latent width' is an empirical sweep (Fig. 6b), not an identity: MAUVE varies with the tan-d/logit-normal schedule and is reported as a chosen operating point. Similarly, 'self-trajectory consistency' is an auxiliary loss whose effect is shown by turning it off/on (Fig. 7a), so it is not a renamed version of the evaluation metric. The only related concerns are benchmark-selection and evaluation-fairness risks, not circularity: the final OWT sampling configuration (64-step SDE-DPM++, SC-CFG 4) was selected via a Gen-PPL/MAUVE sweep on the same evaluation distribution (Table 7), and the matched-protocol comparison with Cola-DLM shows a suspiciously low WinoGrande score for Cola-DLM (5.7% vs chance 50%, Table 4), suggesting a possible format or matcher mismatch. These issues affect the strength of comparative claims but do not make any prediction equivalent to an input by construction.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The central claim rests on internal sweeps (d, Db, Q, guidance) and three unproven external anchors: the quality of the frozen decoder, the stability of MAUVE at 1,000 samples, and the fairness of the shared benchmark matcher. The last is the costliest: Table 4's Cola-DLM WinoGrande 5.7% suggests the protocol itself is a hidden coefficient in the scaling result.

free parameters (8)
  • tan-d noise schedule shift d = 7 (system); 5 (ablations)
    Selected by MAUVE sweep d∈{1,3,5,7} per latent width (Fig. 5b, 6b, Table 11); governs high-noise mass m0.7≈0.795.
  • Noisy-input bottleneck width Db = 128
    Swept {32,128,256,512,1024}; 128 gives highest mean MAUVE (Fig. 6a, Table 13).
  • Block size Q = 16
    Swept {4,8,16,32,64}; 16 chosen as balance point between MAUVE 0.816 and sequential stages (Fig. 7b).
  • Latent width D = 1024 (AURORA-LM-L); 768 (AURORA-LM-S)
    D=1024 best in width sweep (Fig. 5b); Table 5 lists S at D=768 without explanation, so S uses a width never ablated.
  • Self-trajectory consistency weight λct and budget S = 1.0 / 16
    Set by hand with linear warmup (A.1.4); S fixed to the 16-step schedule used at inference.
  • Self-conditioning probability psc = 0.6 (S) / 0.5 (L)
    Chosen by hand in A.2.3, not swept.
  • Inference guidance scales = SC-CFG 4 (OWT); CFG 2.0 (XSum); CFG 3.0 (benchmark)
    Selected by sweeps on the respective evaluation sets (Tables 7-8), i.e., tuned on the test distribution.
  • Autoencoder dropouts px / pz = 0.3 / 0.6
    Taken from COSMOS [37] without a dedicated sweep.
axioms (6)
  • standard math Linear-interpolant flow matching path x_t = (1−t)x_0 + tε (Eqs. 1, 8) is an adequate generative target for latents
    Used throughout §3.1/§3.3; standard flow matching (Lipman et al. 2023, Liu et al. 2023).
  • domain assumption Block-causal factorization pθ(α)=∏_b pθ(α(b)|α(<b)) (Eq. 7) approximates the true joint latent distribution
    §3.3; generation quality depends on this approximation; Q-sweep (Fig. 7b) shows strong sensitivity (MAUVE 0.91 at Q=4 vs 0.63 at Q=64).
  • domain assumption Frozen query-decoder pψ recovers tokens from sampled latents reliably
    §3.2/§3.5; token recovery at σ=0 is shown only qualitatively in Fig. 5a, never quantified for the final AURORA-LM-S/L configs; the whole pipeline is bounded by this decoder.
  • domain assumption MAUVE/Gen-PPL on 1,000 samples per seed is a stable quality proxy
    §4.1 protocol; Table 7 shows ±0.02 MAUVE across seeds, so five seeds mitigate but do not eliminate instability.
  • domain assumption Shared answer matcher and prompt format treat AURORA-LM-L and Cola-DLM fairly
    A.2.9 / Table 4; contradicted by Cola-DLM WinoGrande 5.7% vs chance 50%.
  • ad hoc to paper Wider latents require stronger high-noise allocation (d=7)
    Empirical trend from Fig. 5b/6b used to justify the final tan-d shift; not derived, and would need re-derivation for widths not in the sweep (e.g., D=768).

pith-pipeline@v1.3.0-daily-deepseek · 33796 in / 22008 out tokens · 775962 ms · 2026-08-04T03:17:14.048902+00:00 · methodology

0 comments
read the original abstract

Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for joint generation and decoding, or compress autoencoded latents to ease diffusion, sacrificing token-level fidelity. Instead of simplifying the representation to suit the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly. We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its distribution. A Query-based Encoder-Decoder organizes text into a high-capacity, prefix-aligned latent sequence, and a Block-causal Diffusion Transformer learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. Because such a latent is harder for diffusion to model, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1B parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 1 canonical work pages

  1. [1]

    Albergo and Eric Vanden-Eijnden

    Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. InInter- national Conference on Learning Representations (ICLR), 2023

    2023

  2. [2]

    Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov

    Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  3. [3]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  4. [4]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  5. [5]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Analog bits: Generating discrete data using diffusion models with self-conditioning

    Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  7. [7]

    Langflow: Continuous diffusion rivals discrete in language modeling.arXiv preprint arXiv:2604.11748, 2026

    Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, and Ge Liu. Langflow: Continuous diffusion rivals discrete in language modeling.arXiv preprint arXiv:2604.11748, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  9. [9]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, et al. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machi...

    2024

  12. [12]

    Openwebtext corpus.https://skylion007.github.io/OpenWebTextCorpus/, 2019

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.https://skylion007.github.io/OpenWebTextCorpus/, 2019

    2019

  13. [13]

    Diffuseq: Sequence to sequence text generation with diffusion models

    Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  14. [14]

    Hashimoto

    Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood-based diffusion language models. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  15. [15]

    Continuous latent diffusion language model.arXiv preprint arXiv:2605.06548, 2026

    Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, Rui Zhu, Qiushan Guo, Feng Wang, Tao Yang, Heng- shuang Zhao, Guoqiang Wei, and Yan Zeng. Continuous latent diffusion language model.arXiv preprint arXiv:2605.06548, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  17. [17]

    Query-key normalization for transformers

    Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.findings-emnlp.379

    doi:10.18653/v1/2020.findings-emnlp.379 2020

  18. [18]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  20. [20]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models.arXiv preprint arXiv:2204.03458, 2022. 20

    Pith/arXiv arXiv 2022

  21. [21]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Milli- can, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sif...

    Pith/arXiv arXiv 2022

  22. [22]

    Simple diffusion: End-to-end diffusion for high resolution images

    Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. InInternational Conference on Machine Learning (ICML), 2023

    2023

  23. [23]

    Elf: Embedded language flows.arXiv preprint arXiv:2605.10938, 2026

    Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, and Kaiming He. Elf: Embedded language flows.arXiv preprint arXiv:2605.10938, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    Textldm: Language modeling with continuous latent diffusion.arXiv preprint arXiv:2605.07748, 2026

    Jiaxiu Jiang, Jingjing Ren, Wenbo Li, Bo Wang, Haoze Sun, Yijun Yang, Jianhui Liu, Yanbing Zhang, Shenghe Zheng, Yuan Zhang, Haoyang Huang, Nan Duan, and Wangmeng Zuo. Textldm: Language modeling with continuous latent diffusion.arXiv preprint arXiv:2605.07748, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  26. [26]

    Diffwave: A versatile diffusion model for audio synthesis

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  27. [27]

    RACE: Large-scale ReAding compre- hension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding compre- hension dataset from examinations. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017

    2017

  28. [28]

    Back to basics: Let denoising generative models denoise.arXiv preprint arXiv:2511.13720, 2025

    Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise.arXiv preprint arXiv:2511.13720, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Hashimoto

    Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-lm improves controllable text generation. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  30. [30]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summarization Branches Out, pages 74–81, Barcelona, Spain, 2004. Association for Computational Linguistics. URLhttps://aclanthology. org/W04-1013/

    2004

  31. [31]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  32. [32]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  33. [33]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 32819–32848, 2024

    2024

  34. [34]

    Weinberger

    Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q. Weinberger. Latent diffusion for language generation. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  35. [35]

    DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models.arXiv preprint arXiv:2211.01095, 2022

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models.arXiv preprint arXiv:2211.01095, 2022

    Pith/arXiv arXiv 2022

  36. [36]

    Latent consistency models: Synthesizing high-resolution images with few-step inference.arXiv preprint arXiv:2310.04378, 2023

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference.arXiv preprint arXiv:2310.04378, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Cosmos: Compressed and smooth latent space for text diffusion modeling

    Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, and Dmitry Vetrov. Cosmos: Compressed and smooth latent space for text diffusion modeling. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  38. [38]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018

    2018

  39. [39]

    A corpus and cloze evaluation for deeper understanding of commonsense stories

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Push- meet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. 21 InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, 2016

    2016

  40. [40]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, 2018

    2018

  41. [41]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    MAUVE: Measuring the gap between neural text and human text using divergence frontiers

    Krishna Pillutla, Swabha Swayamditta, Luke Zettlemoyer, and Zaid Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. InAdvances in Neural Information Processing Systems, 2021

    2021

  44. [44]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI technical report, 2019

    2019

  45. [45]

    SQuAD: 100,000+ questions for ma- chine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for ma- chine comprehension of text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016

    2016

  46. [46]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  47. [47]

    Chiu, Alexan- der M

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexan- der M. Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  48. [48]

    Chiu, and Volodymyr Kuleshov

    Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T. Chiu, and Volodymyr Kuleshov. The diffusion duality. InInternational Conference on Machine Learning (ICML), 2025

    2025

  49. [49]

    WinoGrande: An adversarial wino- grad schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial wino- grad schema challenge at scale. InProceedings of the 34th AAAI Conference on Artificial Intelligence, 2020

    2020

  50. [50]

    SocialIQa: Commonsensereasoning about social interactions

    MaartenSap, HannahRashkin, DerekChen, RonanLeBras, andYejinChoi. SocialIQa: Commonsensereasoning about social interactions. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019

    2019

  51. [51]

    Alexander Shabalin, Viacheslav Meshchaninov, Egor Chimbulatov, Vladislav Lapikov, Roman Kim, Grigory Bartosh, Dmitry Molchanov, Sergey Markov, and Dmitry Vetrov. Tencdm: Understanding the properties of the diffusion model in the space of language model encodings.Proceedings of the AAAI Conference on Artificial Intelligence, 39(23):25110–25118, 2025

    2025

  52. [52]

    GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  53. [53]

    Codar: Continuous diffusion language models are more powerful than you think.arXiv preprint arXiv:2603.02547, 2026

    Junzhe Shen, Jieru Zhao, Ziwei He, and Zhouhan Lin. Codar: Continuous diffusion language models are more powerful than you think.arXiv preprint arXiv:2603.02547, 2026

    arXiv 2026

  54. [54]

    Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

    Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  55. [55]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. InInternational Conference on Machine Learning (ICML), 2023

    2023

  56. [56]

    Seed diffusion: A large-scale diffusion language model with high-speed inference.arXiv preprint arXiv:2508.02193, 2025

    Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang, Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu, and Hao Zhou. Seed diffusion: A large-scale diffusion language model with high-speed inference.arXiv prepri...

    Pith/arXiv arXiv 2025

  57. [57]

    RoFormer: Enhanced trans- former with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced trans- former with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021. 22

    Pith/arXiv arXiv 2021

  58. [58]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  59. [59]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  60. [60]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Pith/arXiv arXiv 2024

  61. [61]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  62. [62]

    Continuous diffusion scales competitively with discrete diffusion for language.arXiv preprint arXiv:2605.18530, 2026

    Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, and John Thickstun. Continuous diffusion scales competitively with discrete diffusion for language.arXiv preprint arXiv:2605.18530, 2026

    Pith/arXiv arXiv 2026

  63. [63]

    Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Pith/arXiv arXiv 2025

  64. [64]

    Text diffusion model with encoder- decoder transformers for sequence-to-sequence generation

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Text diffusion model with encoder- decoder transformers for sequence-to-sequence generation. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 22–39, Mexico City,...

    work page doi:10.18653/v1/2024.naacl-long.2 2024

  65. [65]

    HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  66. [66]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  67. [67]

    backCreating

    Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning.arXiv preprint arXiv:2504.12216, 2025. 23 Appendix A Additional Method and Experimental Details Thisappendixsupplementsthemaintextwithimplementationdetails, evaluationprotocols, completecontrolled- study results...

    Pith/arXiv arXiv 2025