Pith. sign in

REVIEW 5 major objections 4 minor 4 cited by

Anchored Diffusion Language Model

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Masked diffusion language models close the gap with autoregressive models by anchoring important tokens first.

desk verdict A real empirical idea with a fixable theory flaw and some overclaimed headlines. read the letter →

arxiv 2505.18456 v1 pith:H6T4MY4M submitted 2025-05-24 cs.CL cs.LG

Litu Rout , Constantine Caramanis , Sanjay Shakkottai This is my paper · ORCID
classification cs.CLcs.LG
keywords diffusionlanguagemodelsanchortokensANELBOmaskedtokenpredictionMAUVEsamplecomplexitychain-of-thoughtlikelihoodmodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked diffusion language models generate in parallel and see the whole sequence, but they have consistently trailed autoregressive models in both likelihood and sample quality. The paper argues that the gap comes from early masking of important tokens: when content-bearing, low-frequency words are erased, the denoiser lacks the context needed to reconstruct the sentence. It proposes the Anchored Diffusion Language Model, which splits denoising into two stages—an anchor network predicts distributions over important tokens, then a denoiser predicts the remaining missing tokens conditioned on those anchored predictions—and trains both with a single anchored evidence lower bound (ANELBO) objective. If correct, the method would mean diffusion models need not trade generation quality for parallel, bidirectional decoding: they can match autoregressive perplexity and, with a remasking sampler at many steps, beat autoregressive models on MAUVE.

What carries the argument

The load-bearing object is the Anchored Negative Evidence Lower Bound (ANELBO), a variational objective that appends a weighted anchor-prediction term to the standard masked-diffusion denoising loss. The architecture is a two-stage reverse process: an anchor network $y_\varphi$ predicts distributions over important tokens from the masked context, and a denoiser $x_\psi(y_\varphi(\cdot))$ predicts the likelihoods of missing tokens conditioned on those anchored outputs. The anchor loss is supervised only on tokens whose relative frequency is below a threshold $\tau$, with a weight $\gamma$; at inference the anchored predictions are projected into the denoiser's embedding space, keeping the pipeline differentiable. In a simplified directed-graphical-model analysis, conditioning on a fixed anchor set of size $d$ reduces the sample complexity of maximum-likelihood estimation from $O(K^L)$ to $O(LK^{d+1})$, which is the paper's theoretical reason to expect anchoring to help.

What would settle it

Train ADLM with anchor positions chosen uniformly at random instead of by low frequency, holding architecture, loss strength, and compute fixed: if random anchors reproduce the reported OpenWebText perplexity and MAUVE gains, the frequency-based importance criterion is not what carries the result. A second test would construct a corpus where low-frequency tokens are mostly noise and high-frequency content words carry the gist; under the paper's hypothesis this corpus should show anchoring underperform its own two-stage variant anchored on content words.

Watch

Extended reading notes

Core claim

The central claim is that a masked diffusion language model fails mainly because important tokens are masked away early, and that this failure is fixable by predicting the important tokens first and denoising conditionally on them. In the method, an anchor network $y_\varphi$ maps the partially masked sequence to a mixture distribution over important tokens at every position, and a shallower denoiser $x_\psi$ predicts all missing-token likelihoods given $y_\varphi$. Both are trained jointly by the ANELBO objective, with important tokens selected by the frequency criterion $\mu(x_l)\le\tau$. On OpenWebText at 524B tokens the paper reports test perplexity 20.14 versus 22.98 for MDLM and 17.26 for an autoregressive transformer; with a remasking sampler at 4096 steps it reports MAUVE 0.791 versus 0.760 for autoregressive decoding, and it reports state-of-the-art zero-shot perplexity among diffusion models on six of seven benchmarks. The same anchoring idea is then applied to autoregressive models, improving next-token perplexity on OpenWebText and raising accuracy on GSM8K and logical-reasoning tasks.

Load-bearing premise

The load-bearing premise is that low-frequency tokens are the important tokens worth anchoring on; the paper itself, in its limitation section, concedes this proxy may not generalize. The theoretical sample-complexity result further assumes anchor sets are fixed and small ($|\pi_l|\le d$), which the transformer-based implementation does not enforce.

Editorial extensions

If this is right

  • Perplexity of masked diffusion models drops by roughly 10–25 percent on LM1B and OpenWebText at matched token budgets, narrowing the gap to autoregressive baselines without any autoregressive component.
  • With enough sampling steps and the remasking sampler, a diffusion language model reports a higher MAUVE score than an autoregressive baseline, which the paper states as the first such result for human-like text quality.
  • The two-stage architecture alone, without the anchor loss, improves the MDLM baseline on OpenWebText from 23.17 to 21.79 perplexity at 262B tokens, indicating that the decomposition itself contributes beyond the auxiliary loss.
  • If the sample-complexity result transfers to trained transformers, anchoring should make masked-diffusion training noticeably more sample-efficient than uniform masking.
  • Anchoring also improves autoregressive models: an anchored autoregressive variant reaches 15.86 test perplexity versus 17.26 for plain AR at 524B tokens, and anchored chain-of-thought raises GSM8K accuracy from 42.9 to 45.2 percent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test left implicit by the paper: train ADLM with anchor positions chosen at random instead of by low frequency, holding architecture, loss strength, and compute fixed; if the gains persist, the mechanism is the two-stage decomposition rather than the importance criterion.
  • The graphical-model analysis assumes a fixed anchor set of bounded size, which the transformer implementation does not enforce; on small synthetic tree-structured distributions where anchors are known exactly, one could test whether the promised exponential sample-complexity reduction materializes in practical gradient training.
  • The ACoT result suggests anchor tokens act as a lightweight look-ahead mechanism; comparing ACoT against stronger base models and longer reasoning traces could reveal whether anchoring remains useful as left-to-right context grows.
  • Because the denoiser uses half the layers of the anchor network, ADLM effectively allocates capacity to a salience model; the same division of labor might transfer to retrieval-augmented generation, where an anchor module selects the few entities the generator should condition on.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes Anchored Diffusion Language Model (ADLM), a two-stage masked diffusion language model in which an anchor network first predicts distributions over tokens deemed important by a frequency criterion, and a denoiser network then predicts masked-token likelihoods conditioned on those anchored predictions. The authors derive an ANELBO training objective, present a graphical-model argument for reduced sample complexity, and report improved perplexity on LM1B and OpenWebText, strong zero-shot perplexities, and MAUVE scores that exceed an autoregressive baseline at high sampling budgets. They also extend the anchoring idea to autoregressive next-token prediction and to chain-of-thought reasoning. The report focuses on the strength of the empirical attribution, the validity of the ANELBO derivation, and the accuracy of the headline comparisons.

Significance. If established, the two-stage anchor idea is a useful and simple contribution: it is architecture-agnostic, and several matched-token-budget comparisons consistently favor ADLM over prior diffusion language models (e.g., OWT 20.14 vs 22.98 at 524B tokens, Table 1b). The paper is unusually complete in reporting ablations over gamma and tau, zero-shot results on seven benchmarks, and qualitative samples, and the main claims are falsifiable. However, the central attribution of the gains to anchoring is not controlled for parameter count, the ANELBO proof has a circular step, and the headline gains and the MAUVE claim are stated without the budget caveats under which they hold. These issues are local and fixable, so the work is worth pursuing in revision.

major comments (5)
  1. [Appendix A.1, Theorem 4.1] The proof of Theorem 4.1 evaluates the KL term using the sigma-parameterized transition (6) as the q distribution, rather than the true forward posterior (2). In Case 1, the paper sets q(Zs=m|Zt != m) = sigma_t and p_theta(Zs=m|Zt != m) = sigma_t, so the KL vanishes by construction. With the actual forward posterior (2), an unmasked token is deterministic, and the unmasked KL is -log(1 - sigma_t), which is not part of the stated LANELBO. The theorem is therefore not proved as written; at most, the appendix proves a bound for a different reverse process in which q and p_theta are chosen to be equal on unmasked tokens.
  2. [Section 5.1.1, Tables 1(b) and 7, Section C.1.3] The paper's central attribution of the empirical gains to anchoring is not controlled for model capacity. ADLM is 293M parameters (Table 7) while the MDLM and ReMDM baselines are 170M parameters. The ablation ADLM* with gamma = 0 keeps the same two-network 293M architecture and already reduces OWT perplexity from 23.17 to 21.79 at 262B tokens, showing that the added architecture itself contributes substantially. A matched-parameter MDLM (or a matched-parameter ADLM variant) is needed before the remaining matched-token-budget gains at 524B tokens (20.14 vs 22.98, Table 1b) and the MAUVE gains in Table 3 can be attributed to the anchor loss. The sentence in Section 5.1.1 claiming that anchoring is 'not just adding extra capacity' is not supported by the reported ablations.
  3. [Abstract and Introduction vs Table 1(a)] The headline improvements '9.54% over MDLM' and '25.4% over SEDD' are not matched-budget comparisons. At the same 33B-token budget, ADLM achieves 26.40 vs MDLM's 27.04, a 2.4% gain; the 9.54% figure compares ADLM at 65B tokens with MDLM at 33B, and the 25.4% figure similarly compares ADLM at 65B with SEDD at 33B. These percentages should be restated as matched-budget numbers, or the different training budgets should be explicitly disclosed wherever the percentages appear.
  4. [Table 3 and Section 5.1.1] The claim of being the first DLM to surpass an AR model in MAUVE is only supported at T = 2048 and T = 4096, while the AR baseline is reported at T = 1024. At the matched budget T = 1024, ADLM's MAUVE is 0.699, below the AR baseline's 0.760. The 2-4x sampling budget should be stated next to the claim, and the comparison should either be reported at equal T or presented as a cost-quality curve.
  5. [Section 4.2, Assumption 4.3, Proposition 4.4] Proposition 4.4 is a CPT parameter-counting result that assumes fixed anchor sets pi_l with |pi_l| <= d. ADLM's transformer denoiser conditions on the entire sequence through attention and uses a learned y_phi that is not a fixed d-subset, so the O(L K^(d+1)) sample-complexity bound does not apply to the architecture being trained. The claim that anchoring gives 'exponential reductions in sample complexity' for ADLM overstates the theory; the proposition should be explicitly scoped to a tabular abstraction, not the neural ADLM model.
minor comments (4)
  1. [Throughout the appendix] There are several typos, including 'autregresive' in the Section C.1.1 header and 'prvoides' in Section C.2.5; these should be corrected in a final pass.
  2. [Equations (7) and (9)] The anchor loss LAnchor in Eq. (9) is defined with a sum over i = 0..T, while the bound in Theorem 4.1 and Eq. (7) sums over i = 1..T; the index range and the definition of the boundary step should be reconciled.
  3. [Section C.1.2] The important caveat that diffusion perplexities are upper bounds while autoregressive perplexities are exact is placed only in the appendix; it should also appear in the main text wherever ADLM perplexities are compared directly with AR perplexities.
  4. [Section 6, Limitation] The frequency-based anchor criterion mu(x_l) <= tau is explicitly acknowledged as a proxy that 'may not generalize.' Given that this criterion is used for all of the diffusion-language-model experiments, the limitation should be stated earlier in the paper, not only in the conclusion, and the sensitivity analysis in Table 6 should be supplemented for LM1B and for the downstream tasks.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ANELBO objective is a standard variational bound, the sample-complexity result is an explicit CPT-counting argument under stated assumptions, and the empirical comparisons are not fitted to the evaluation metrics.

full rationale

The paper's derivation chain is self-contained. The ANELBO objective (Eq. 7, Theorem 4.1) is obtained by the standard NELBO decomposition for absorbing discrete diffusion, with an added KL regularizer that supervises the anchor network; the bound does not presuppose the claimed likelihood improvement, it only upper-bounds the negative log-likelihood by a training objective. The sample-complexity result (Proposition 4.4 and Appendix A.3) is a textbook CPT parameter-counting argument under the explicit Assumption 4.3 that anchor sets are fixed and of bounded size; it does not smuggle in the empirical conclusion, though it admittedly abstracts away the transformer's actual conditioning structure. The ablation ADLM* with gamma = 0 (Table 1b) is a genuine control separating the two-stage architecture from the anchor loss, and the final ADLM includes the anchor loss, so the attribution of the gain to anchoring is not circular. Hyperparameters gamma and tau are selected on an OWT validation set and then reported on a held-out split, which is standard tuning rather than a fitted prediction. The only self-citation is Kwon et al. (2024), co-authored by Caramanis, in Appendix A.3.3; it is used as a general reference for convergence analysis and is not load-bearing because the monotonic-improvement proof is given in full. The paper's own limitations (Section 6: frequency-based importance proxy may not generalize; Appendix C.1.3: DLM perplexities are upper bounds while AR perplexities are exact) are acknowledged caveats that affect interpretation but do not constitute circular reasoning. No equation or result in the paper reduces to its inputs by construction, and no prediction is a renamed fitted parameter. Therefore, no circularity is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central method rests on two tuned hyperparameters (τ and γ), a frequency-based importance heuristic that is not theoretically justified, an architectural assumption about how anchor logits enter the denoiser, and a toy-DAG assumption that does not match the transformer. No new physical or external entities are introduced.

free parameters (2)
  • anchor threshold τ = 5
    Chosen by ablation on OWT (78B tokens, Table 6b); trade-off between too few and too many anchors.
  • anchor strength γ = 3e-3
    Chosen by ablation on OWT (78B tokens, Table 6a); balances anchor loss vs denoising loss.
assumptions (4)
  • standard math The forward process is the absorbing (masking) diffusion with a predefined log-linear schedule α_t.
    Inherited from D3PM/MDLM (Eq. 1); used as the noising process throughout.
  • ad hoc to paper Token importance is captured by relative frequency µ(x_l) ≤ τ.
    The central heuristic for defining anchor tokens; the paper's limitation section concedes it may not generalize.
  • domain assumption The denoiser can condition on anchor logits through a learned linear projection into the embedding space.
    Architectural assumption (Appendix C.1.3); no guarantee that this projection preserves anchor information.
  • domain assumption Assumption 4.3: each conditional is categorical, parameterized by CPTs, and anchor sets are fixed with |π_l| ≤ d.
    Used for the sample-complexity Proposition 4.4; does not hold for the transformer architecture with attention over all tokens, as the paper acknowledges in A.3.2.
invented entities (1)
  • [ANT] anchor tokens
    purpose: Inserted tokens (ACoT) or implicit anchor distributions (ADLM) that are predicted before the rest of the sequence is decoded.
    Model-specific construct; no external falsifiable handle outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Anchored Diffusion Language Model." pith.science (2026). https://pith.science/paper/H6T4MY4M

@misc{pith2026250518456,
  author       = {Pith},
  title        = {Pith review of: Anchored Diffusion Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H6T4MY4M}},
  note         = {Machine review of arXiv:2505.18456}
}
read the original abstract

Diffusion Language Models (DLMs) promise parallel generation and bidirectional context, yet they underperform autoregressive (AR) models in both likelihood modeling and generated text quality. We identify that this performance gap arises when important tokens (e.g., key words or low-frequency words that anchor a sentence) are masked early in the forward process, limiting contextual information for accurate reconstruction. To address this, we introduce the Anchored Diffusion Language Model (ADLM), a novel two-stage framework that first predicts distributions over important tokens via an anchor network, and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-art performance in zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Theoretically, we derive an Anchored Negative Evidence Lower Bound (ANELBO) objective and show that anchoring improves sample complexity and likelihood modeling. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thought approaches

Figures

Figures reproduced from arXiv: 2505.18456 by the authors.

Figure 2
Figure 2. Training loss and validation PPL versus number of iterations on OWT. We train both MDLM (Sahoo et al., 2024) and our ADLM model for 2M iterations (524B tokens). As discussed in §4, anchoring improves the sample complexity during training, resulting in faster convergence and lower validation perplexity. While the anchor loss is part of the training objective, we only visualize the NELBO here for a direct comparison w… view at source ↗
Figure 3
Figure 3. Training of standard autoregres￾sive (AR) models. A neural network is trained to predict the next token using causal attention (left￾to-right context). All tokens contribute equally to the training loss, and the model treats the se￾quence uniformly without structural guidance. Anchor Network playing fighting a 64 349 318 1760 326 64 329 a cat is playing with a dog a the lion Large Language Model [PITH_FULL_IMAGE:fi… view at source ↗
Figure 5
Figure 5. Multi-stage training pipeline for Anchored Chain-of-Thought (ACoT). Here, [BOA] and [EOA] denote the beginning and end of anchors, respectively. Many reasoning traces contain redundant information, increasing entropy and making the reasoning process harder to learn. By supervising the model through a small set of important tokens extracted from the reasoning trace, ACoT encourages more structured intermediate comput… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...

  2. Fine-Tuning Masked Diffusion for Provable Self-Correction

    cs.LG 2025-10 conditional novelty 6.0 of 10

    PRISM fine-tunes any masked diffusion model with a binary-cross-entropy loss so its new head provably estimates per-token quality p(x_i=y_i|y⊕m_i) and can remask low-quality tokens at inference.

  3. Any-Order Flexible Length Masked Diffusion

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, ma...

  4. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Reference graph

Works this paper leans on

62 extracted references · 25 canonical work pages · cited by 4 Pith papers

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1: 0 1, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf

    work page 2024

  2. [2]

    Block diffusion: Interpolating between autoregressive and diffusion language models

    Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tyEyYT267x

    2025

  3. [3]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=h7-XixPCAL

    work page 2021

  4. [4]

    A neural probabilistic language model

    Yoshua Bengio, R \'e jean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3 0 (Feb): 0 1137--1155, 2003

    work page 2003

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020 a . URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967...

    work page 1901

  6. [6]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp.\ 1877--1901, 2020 b

    work page 1901

  7. [7]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=DmT862YAieY

    work page 2022

  8. [8]

    One billion word benchmark for measuring progress in statistical language modeling

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013

    arXiv 2013

Show all 62 references
  1. [9]

    What does bert look at? an analysis of bert’s attention

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp.\ 276. Association for Computational Ling...

    2019

  2. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [11]

    A discourse-aware attention model for abstractive summarization of long documents

    Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association f...

    2018

  4. [12]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 2978--2988, 2019

    2019

  5. [13]

    Maximum likelihood from incomplete data via the em algorithm

    Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39 0 (1): 0 1--22, 1977

    1977

  6. [14]

    From explicit cot to implicit cot: Learning to internalize cot step by step

    Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838, 2024

    2024 arXiv

  7. [15]

    Hierarchical neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 889--898, Melbourne, Australia, July 2018. Association for Computational Lin...

    2018 doi

  8. [16]

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=GTDKo3Sv9p

    2024

  9. [17]

    Openwebtext corpus

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  10. [18]

    Think before you speak: Training language models with pause tokens

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/for...

    2024

  11. [19]

    Likelihood-based diffusion language models

    Ishaan Gulrajani and Tatsunori Hashimoto. Likelihood-based diffusion language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=e2MCL6hObn

    2023

  12. [20]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  13. [21]

    Diffusionbert: Improving generative masked language models with diffusion models

    Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuan-Jing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2023

  14. [22]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 6840--6851. Curran Associates, Inc., 2020. URL http...

    2020

  15. [23]

    Interpolated estimation of markov source parameters from sparse data

    Frederick Jelinek. Interpolated estimation of markov source parameters from sparse data. In Proc. Workshop on Pattern Recognition in Practice, 1980, 1980

    1980

  16. [24]

    An introduction to variational methods for graphical models

    Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37: 0 183--233, 1999

    1999

  17. [25]

    Generalization through memorization: Nearest neighbor language models

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HklBjCEKvH

    2020

  18. [26]

    Probabilistic graphical models: principles and techniques

    Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT Press, 2009

    2009

  19. [27]

    Global optimality of the em algorithm for mixtures of two-component linear regressions

    Jeongyeol Kwon, Wei Qian, Yudong Chen, Constantine Caramanis, Damek Davis, and Nhat Ho. Global optimality of the em algorithm for mixtures of two-component linear regressions. IEEE Transactions on Information Theory, 70 0 (9): 0 6519--6546, 2024. doi:10.1109/TIT.2024.3435522

    2024

  20. [28]

    Visualizing and understanding neural models in nlp

    Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 681--691, 2016

    2016

  21. [29]

    Diffusion- LM improves controllable text generation

    Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022 a . U...

    2022

  22. [30]

    Diffusion- LM improves controllable text generation

    Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022 b . U...

    2022

  23. [31]

    Assessing the ability of LSTM s to learn syntax-sensitive dependencies

    Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTM s to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4: 0 521--535, 2016. doi:10.1162/tacl_a_00115. URL https://aclanthology.org/Q16-1037/

    2016 doi

  24. [32]

    Think while you generate: Discrete diffusion with planned denoising

    Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stark, Yilun Xu, Tommi Jaakkola, and Rafael Gomez-Bombarelli. Think while you generate: Discrete diffusion with planned denoising. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview....

    2025

  25. [33]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=CNicRIVIPA

    2024

  26. [34]

    Building a large annotated corpus of english: The penn treebank

    Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19 0 (2): 0 313--330, 1993

    1993

  27. [35]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe

    2017

  28. [36]

    A view of the em algorithm that justifies incremental, sparse, and other variants

    Radford M Neal and Geoffrey E Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pp.\ 355--368. Springer, 1998

    1998

  29. [37]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=WNvvwK0tut

    2025

  30. [38]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025 b

    2025 arXiv

  31. [39]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://o...

    2025

  32. [40]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meet...

    2016

  33. [41]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 4195--4205, 2023

    2023

  34. [42]

    MAUVE : Measuring the gap between neural text and human text using divergence frontiers

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE : Measuring the gap between neural text and human text using divergence frontiers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), A...

    2021

  35. [43]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  36. [44]

    Reasoning to learn from latent thoughts

    Yangjun Ruan, Neil Band, Chris J Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts. arXiv preprint arXiv:2503.18866, 2025

    2025

  37. [45]

    Simple and effective masked diffusion language models

    Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing ...

    2024

  38. [46]

    Language models are greedy reasoners: A systematic formal analysis of chain-of-thought

    Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=qFVVBzXxR2V

    2023

  39. [47]

    Codi: Compressing chain-of-thought into continuous space via self-distillation

    Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025

    2025 arXiv

  40. [48]

    Simplified and generalized masked diffusion for discrete data

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=xcqSOfHt4g

    2024

  41. [49]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of ...

    2015

  42. [50]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  43. [51]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp.\ 3319--3328. PMLR, 2017

    2017

  44. [52]

    Omninet: Omnidirectional representations from transformers

    Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip M Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning, pp.\ 10193--10202. PMLR, 2021

    2021

  45. [53]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. URL https://arxiv.org/pdf...

    2023 arXiv

  46. [54]

    BERT rediscovers the classical NLP pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4593--4601, Florence, Italy,...

    2019 doi

  47. [55]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. URL https...

    2023 arXiv

  48. [56]

    Glauber generative model: Discrete diffusion models via binary classification

    Harshit Varma, Dheeraj Mysore Nagaraj, and Karthikeyan Shanmugam. Glauber generative model: Discrete diffusion models via binary classification. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=HyjIEf90Tn

    2025

  49. [57]

    u tte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Sch \

    Dimitri von R \"u tte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Sch \"o lkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.04482, 2025. URL https://arxiv.org/pdf/2503.04482

    2025 arXiv

  50. [58]

    Bert has a mouth, and it must speak: Bert as a markov random field language model

    Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094, 2019

    1902 arXiv

  51. [60]

    Remasking discrete diffusion models with inference-time scaling

    Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307, 2025 b

    2025

  52. [61]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in N...

    2022

  53. [62]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015

    2015

  54. [63]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.