Discrete Diffusion Models for Language Generation

Ashen Weligalle

REVIEW 3 major objections 6 minor 19 references

Discrete Diffusion Models for Language Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This thesis establishes that discrete diffusion models can generate language in parallel, but on WikiText-103 they trail autoregressive models in compression and stability while showing a modest speed advantage.

desk verdict The thesis's only pro-diffusion result—speed—turns out to be a transcription of NLL, so it confirms the known quality gap but provides no evidence for the parallel-decoding advantage. read the letter →

arxiv 2507.07050 v1 pith:VA4OZ52D submitted 2025-07-02 cs.CL cs.LGstat.ML

Ashen Weligalle This is my paper

classification cs.CLcs.LGstat.ML

keywords discretediffusionD3PMautoregressivelanguagemodelsparalleldecodingBitsPerTokenWikiText-103seedsensitivitynon-autoregressivegeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis asks whether discrete diffusion models, which generate text by iteratively unmasking corrupted tokens in parallel, can compete with autoregressive language models that generate one token at a time. Training both on WikiText-103, the author finds that the best D3PM run reaches a Bits Per Token (BPT) of 5.72 but the mean across three seeds is 8.05, while the autoregressive model averages 4.59 and GPT-2 averages 4.25. The diffusion model's advantage is speed: up to 3.97 batches per second against 3.18 for the custom autoregressive model, though both trail GPT-2's 6.74. The conclusion is that parallel diffusion generation is feasible and potentially faster, but in its current form it is less stable and produces worse compression and fluency than autoregressive models.

What carries the argument

The central object is the absorbing-state D3PM. A forward process pushes each token toward a special mask state through transition matrices $Q_i = (1-\beta_i)I + \beta_i \mathbf{1}\mathbf{e}_m^{\top}$, and a learned denoiser $x_\theta(x_t, t)$ estimates the clean token, which is inserted into the reverse posterior $q(x_s | x_t, x_0)$ to unmask tokens in parallel over $T$ diffusion steps. That parallel unmasking is what gives D3PM its speed profile, while the noise schedule and initialization determine quality and stability. The comparison machinery is the set of metrics, Bits Per Token, Negative Log-Likelihood, Perplexity, and batch processing speed, applied to 100,000 generated tokens per model.

What would settle it

Re-run one D3PM seed and the autoregressive model on WikiText-103 with the same loader batch size, for example 4, and the same 100,000 generated tokens, reporting both BPT and batches per second; if the D3PM speed advantage disappears or its best-run BPT moves outside the 5.0 to 6.5 range while the autoregressive model stays near 4.6, the claimed trade-off is configuration-dependent.

Watch

Extended reading notes

Core claim

Taken on its own terms, the paper establishes that a Discrete Denoising Diffusion Probabilistic Model (D3PM) can be trained to generate text on WikiText-103, but only one of three seeded runs converges to competitive behavior; that run gives BPT 5.72, NLL 3.97, and perplexity 52.78, while the other two runs diverge to BPT above 9. The same evaluation shows the custom autoregressive model averaging BPT 4.60 and GPT-2 averaging 4.25, with GPT-2 reaching perplexity near 19. The author reads this as a trade-off: autoregressive models remain better at fluent, context-aware generation and compression, while D3PM offers genuinely parallel decoding and a modest speed advantage over the custom autoregressive pipeline, making it a candidate for masked completion and structured reconstruction tasks rather than open-ended generation.

Load-bearing premise

The central claim rests on the assumption that the comparison is fair, meaning identical training configurations, batch size, and 100,000-token evaluation across models, so that any observed difference reflects architecture rather than protocol.

Editorial extensions

If this is right

If the result holds, autoregressive models remain the safer default for fluent open-ended generation, because both BPT and perplexity are lower.
Diffusion-based generation becomes a practical option in settings where parallel throughput matters more than peak quality, provided a converged seed or stabilized training is used.
D3PM's seed sensitivity implies that a single run is not a reliable performance estimate; multiple seeds are needed before comparing paradigms.
For masked-token completion, sequence reordering, and denoising tasks, the parallel reverse process may be a better fit than sequential autoregressive decoding.
The speed advantage over a from-scratch autoregressive model, but not over GPT-2, delimits where non-autoregressive generation currently competes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An unstated consequence of the parallel architecture is that its speed advantage should grow with sequence length, so a follow-up measuring batches per second at several lengths would test the trade-off more directly.
The paper positions D3PM for masked completion and reconstruction; a direct benchmark on text infilling or sentence reordering would be a natural test of that positioning.
If additional seeds cluster near the best run, the mean BPT of 8.05 would fall and the quality gap to autoregressive models would narrow; if they spread, seed instability is the more durable finding.
The same metrics and 100,000-token protocol could be reused to benchmark other non-autoregressive generators, making the comparison reproducible beyond the two model families tested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the only result in this thesis that favors diffusion—higher batch speed—is unsupported because the reported speed values are identical to the reported NLL values in the same rows. That looks like a copy-paste error, not a measurement.

The quality comparison is plausible: the AR model and GPT-2 beat D3PM on BPT/NLL/PPL, and D3PM's best seed (2000) is far better than the other two. The thesis is honest about its limitations: it admits training was heavy, only three D3PM seeds were run, and some runs diverged. It also uses a single unified framework (MDLM) for both model families, which is the right way to control for pipeline differences. The theory chapter on D3PM is readable and correctly cites Austin et al. and MDLM. For a licentiate thesis, the structure is fine.

But the central speed claim breaks down. In Table 4.2, D3PM seed 1000: NLL 6.2693, speed 6.2690; seed 2000: NLL 3.9661, speed 3.9661; seed 3000: NLL 6.5199, speed 6.5199. The AR row in Table 4.1 shows the same pattern (NLL 3.1833, speed 3.1832). GPT-2's speed is different from its NLL, so this is not a systemic formatting artifact—the D3PM and AR speed numbers were apparently never independently collected. That removes the thesis's only positive result for diffusion.

Secondary issues: Section 4 says batch size 4 for all models, but the D3PM command in Appendix 7.5 sets loader.batch_size=8. Table 5.2 uses "D3PM (Best / Mean)" and cherry-picks seed 2000; the mean BPT is 8.06, much worse than AR's 4.60. No error bars or standard deviations for speed are given. And the comparison itself is not new: Austin et al. already applied D3PM to text, and MDLM already compared AR vs diffusion. The one thing that could have been new—a careful speed benchmark under identical inference conditions—is missing.

Who is this for? A reader who wants a beginner-level, worked example of training D3PM on WikiText-103, with honest reporting of instability, will get some value. But as a research paper it does not hold up: the main claim in the abstract is unsupported, and the protocol inconsistencies are checkable.

I would desk-reject in current form. If the authors re-run the speed measurements under the same batch size, report all seeds with variance, and stop cherry-picking the best seed, a short benchmark note might be worth a second look. But that would be a different paper.

Referee Report

3 major / 6 minor

Summary. This licentiate thesis compares a Discrete Denoising Diffusion Probabilistic Model (D3PM), implemented with the MDLM codebase, against a small autoregressive (AR) model and a fine-tuned GPT-2 on WikiText-103. It reports Bits Per Token (BPT), Negative Log-Likelihood (NLL), perplexity, and batch processing speed across a small number of seeds. The central claims are that AR achieves lower BPT (mean 4.59) than D3PM (mean 8.05, best 5.72), while D3PM offers higher processing speed (up to 3.97 batches/s), indicating potential for parallel generation. The thesis also contains an introduction to discrete diffusion theory and a discussion of limitations.

Significance. If the reported results held, the paper would provide a modest but useful empirical datapoint on the quality/speed trade-off between autoregressive and discrete diffusion language models. The negative quality result (AR better BPT than D3PM) is plausible and consistent with the broader literature, and the public code repository is a strength. However, the only result favoring D3PM—the speed advantage—is not supported by the reported data: the 'Speed' column for AR and D3PM exactly reproduces the NLL values in the same tables. Since the abstract and conclusions hinge on D3PM's speed potential, the manuscript's main positive claim is currently unverified.

major comments (3)

[Tables 4.1 and 4.2; Abstract] In Tables 4.1 and 4.2, the reported 'Speed (Batch/s)' values for the AR and D3PM models coincide numerically with the NLL values in the same rows (for example, D3PM seed 2000: speed 3.9661 vs NLL 3.9661; seed 1000: speed 6.2690 vs NLL 6.2693; seed 3000: speed 6.5199 vs NLL 6.5199; AR seed 1: speed 3.1832 vs NLL 3.1833). An independent throughput measurement would not be expected to track a log-likelihood loss to several decimal places. This strongly suggests that the speed was not measured independently or that the wrong quantity was copied into the Speed column. Because the abstract's favorable claim for D3PM is precisely its 'higher processing speed, reaching up to 3.97 batches per sec.', the speed advantage is unsupported and must be re-measured under controlled conditions before any comparison can be made.
[Section 4 and Appendix 7.5] The evaluation protocol is not consistent between models. Section 4 states that all models were trained and evaluated with batch size 4, but the D3PM training command in Appendix 7.5 sets loader.batch_size=8 and only loader.eval_batch_size=4. Because throughput in batches per second depends directly on batch size, this discrepancy invalidates the claimed 'consistent conditions' premise and could by itself explain speed differences. The comparison must be repeated with identical batch sizes in both training and evaluation.
[Abstract, Section 4.2, and Section 5.1] The speed claim is internally inconsistent. Even taking Table 4.2 at face value, the maximum reported D3PM speed is 6.5199 batches/s (seed 3000), and seed 1000 reports 6.2690 batches/s, both larger than the 'up to 3.97 batches per sec.' stated in the abstract; the 3.97 value is simply the speed of the best-BPT seed, not a maximum. Moreover, Section 5.1 states that 'in terms of inference speed, both AR and D3PM models demonstrated comparable performance,' which contradicts the abstract's assertion of a D3PM speed advantage. The narrative needs to be aligned with the actual, re-measured numbers.

minor comments (6)

[Section 3.3.3 and Section 4] Section 3.3.3 describes 'generation speed (measured in tokens per second),' while Section 4 and the tables report 'Speed (Batch/s)'; the units must be defined consistently throughout the thesis.
[Table 4.1] Table 4.1 is malformed: the AR row contains values ('4.5976512 4.6028 3.1904 24.2989 3.1904') that do not align with the column headers and appear to merge results from two seeds into one row; the table must be reformatted so each column contains a single metric.
[Section 5.1] Section 5.1 contains unresolved '??' placeholders (for example, 'Figures ?? and 7.6'); these need to be replaced with actual cross-references.
[Appendix 7.4] Appendix 7.4 includes raw terminal output and TensorFlow warning messages; this material should be replaced with clean, formatted results and sample generations.
[Section 2.1.1] The BPC and BPT formulas use a leading negative sign and a fraction in a manner that is difficult to parse; using display math with clear notation would improve readability.
[Section 3.3.2 and Bibliography] The MDLM package is credited to Ho et al. [10] in Section 3.3.2, but the correct reference for MDLM is Sahoo et al. [16]; please verify and correct the citation.

Circularity Check

1 steps flagged · score 4.0 of 10

The thesis is an empirical benchmark with no derivation-chain circularity, but the reported D3PM speed advantage is compromised because the 'Speed' values equal the NLL values in the same rows, so the central speed claim is not independently supported.

other [Abstract; Tables 4.1, 4.2, and 5.2]
"Results show ... D3PM achieves higher processing speed, reaching up to 3.97 batches per sec., indicating potential for parallel generation. Table 4.2, seed 2000 row: '5.7219 3.9661 52.7819 3.9661'."

The abstract's central claim of a D3PM speed advantage of 3.97 batches/s is taken from the seed-2000 row of Table 4.2, where the 'Speed (Batch/s)' value is 3.9661, exactly equal to the NLL value 3.9661 in the same row. The same coincidence appears for D3PM seeds 1000 and 3000 (Speed 6.2690 vs NLL 6.2693; Speed 6.5199 vs NLL 6.5199) and for the AR model in Table 4.1 (Speed 3.1832 vs NLL 3.1833). Thus the speed numbers are not independent throughput measurements; the reported speed result reduces to the reported NLL value by transcription, and the paper's main positive claim for D3PM rests on that equivalence.

full rationale

The thesis is an empirical comparison rather than a derivation: the theory sections reproduce standard D3PM/MDLM equations from Austin et al. and Sahoo et al., and no new first-principles result is claimed, so there is no circular derivation chain to walk. The MDLM framework attributions in Appendix 7.3 are ordinary tool citations, not load-bearing self-citations. The notable circularity-adjacent issue is the speed metric: in Tables 4.1, 4.2, and 5.2, the 'Speed (Batch/s)' values for AR and D3PM equal the NLL values in the same rows, while GPT-2's speed values differ from its NLL, indicating that throughput was not measured independently for the former models. Because the abstract's headline claim that D3PM reaches 3.97 batches/s is exactly the seed-2000 NLL value, the speed advantage is an unsupported, reported-metric equivalence rather than a real measurement. That is primarily a data-integrity and correctness problem, but because the claimed speed result is equivalent to the NLL result by construction in the table, it warrants a partial circularity flag and a score of 4 rather than 0. Additional protocol inconsistencies, such as Section 4 stating batch size 4 while the Appendix 7.5 command sets loader.batch_size=8 and loader.eval_batch_size=4, further weaken the empirical comparison but are not themselves circular reasoning.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central comparison rests on standard diffusion assumptions imported from Austin et al. and MDLM, on the validity of the metrics, and on the claim that conditions were identical. The thesis introduces no new entities and fits no free parameters in the usual statistical sense; its weaknesses are measurement and protocol inconsistencies rather than hidden fitted constants.

free parameters (3)

Diffusion horizon T and sampling steps = 1000
Set in the training command in Appendix 7.5; not fitted to data, but all D3PM results are conditional on this schedule.
Noise/learning-rate parameter beta = 0.0003
Fixed for all D3PM runs near Figure 4.8; not optimized, and the symbol is confusingly shared with the transition-matrix beta in Section 2.5.
Model size small = small / about 124M parameters
Chosen through the MDLM framework; the exact architecture parameterization is not fully reported, making cross-model comparability hard to verify.

assumptions (4)

domain assumption The forward corruption process is Markov, so q(xs|xt) equals q(xs|xt, x0).
Invoked in Section 2.5.2 as the basis for the reverse posterior used to train D3PM; the implementation must actually satisfy this property.
domain assumption Masking with an absorbing MASK state and transition matrix Qi = (1 - beta_i)I + beta_i times 1 e_m^T is a valid discrete corruption process.
Imported from Austin et al. in Section 2.5.1; the thesis relies on it without testing alternative discrete noise schedules.
domain assumption The same evaluation harness yields comparable BPT, NLL, and PPL across AR and D3PM.
The thesis assumes metrics from the MDLM loop are architecture-independent and that the AR and D3PM pipelines are consistent; the garbled speed/NLL columns and differing batch sizes weaken this assumption.
ad hoc to paper All models were trained and evaluated under identical conditions, including batch size 4 and 100,000 generated tokens.
Stated in Sections 1 and 4; the D3PM command in Appendix 7.5 uses loader.batch_size=8, so this assumption is central and contested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Discrete Diffusion Models for Language Generation." pith.science (2026). https://pith.science/paper/VA4OZ52D

@misc{pith2026250707050,
  author       = {Pith},
  title        = {Pith review of: Discrete Diffusion Models for Language Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VA4OZ52D}},
  note         = {Machine review of arXiv:2507.07050}
}

read the original abstract

Diffusion models have emerged as a powerful class of generative models, achieving state-of-the-art results in continuous data domains such as image and video generation. Their core mechanism involves a forward diffusion process that gradually transforms structured data into a Gaussian-like distribution, followed by a learned reverse process to reconstruct the data. While successful in continuous modalities, applying this framework to discrete data-particularly natural language-remains challenging due to token dependency complexities and the lack of a defined generation order.This thesis investigates the feasibility and performance of discrete diffusion models for natural language generation. Specifically, we evaluate the Discrete Denoising Diffusion Probabilistic Model (D3PM) and compare it with traditional autoregressive (AR) language models. To assess generative performance, we use Bits Per Token (BPT), Negative Log-Likelihood (NLL), Perplexity (PPL), and Batch Processing Speed. Results show the best-performing D3PM model achieves a BPT of 5.72, with a mean of 8.05. The AR model outperforms in compression with a lower mean BPT of 4.59, but D3PM achieves higher processing speed, reaching up to 3.97 batches per sec., indicating potential for parallel generation.All evaluations were conducted under consistent conditions-generating 100,000 tokens per model with a fixed batch size of four-for fair comparison. This research presents a detailed analysis of diffusion-based vs. autoregressive models, highlighting trade-offs in generative quality and efficiency. Findings emphasize both the promise and limitations of diffusion models for discrete data, supporting future work in non-autoregressive language generation.

Figures

Figures reproduced from arXiv: 2507.07050 by the authors.

**Figure 2.1.** Illustration of the token-by-token generation process in an autore [PITH_FULL_IMAGE:figures/full_fig_p022_2_1.png] view at source ↗

**Figure 2.2.** Illustration of the D3PM forward process. [PITH_FULL_IMAGE:figures/full_fig_p025_2_2.png] view at source ↗

**Figure 2.3.** Using a trained D3PM absorbing model for LM1B to (top) gener [PITH_FULL_IMAGE:figures/full_fig_p027_2_3.png] view at source ↗

Figures from the paper (19 more)

**Figure 3.1.** Figure 3.1: Sample spectrum from the WikiText-103 dataset. [PITH_FULL_IMAGE:figures/full_fig_p032_3_1.png]

**Figure 3.2.** Figure 3.2: Illustration of the tokenization process used in this project. [PITH_FULL_IMAGE:figures/full_fig_p034_3_2.png]

**Figure 3.3.** Figure 3.3: Illustration of the methodology in this project. [PITH_FULL_IMAGE:figures/full_fig_p035_3_3.png]

**Figure 4.1.** Figure 4.1: AR model training loss (seeds 1 and 12). [PITH_FULL_IMAGE:figures/full_fig_p039_4_1.png]

**Figure 4.2.** Figure 4.2: GPT-2 model training loss (seeds 1000 and 2000). [PITH_FULL_IMAGE:figures/full_fig_p040_4_2.png]

**Figure 4.3.** Figure 4.3: Learning rate schedule for AR model. exhibits irregularity and overlap in step progression. This discrepancy arises from an earlier test run conducted with the AR model using seed 1 to validate the HPC configuration. The subsequent continuation of training under the …

**Figure 4.4.** Figure 4.4: Learning rate schedule for GPT-2 model [PITH_FULL_IMAGE:figures/full_fig_p041_4_4.png]

**Figure 4.5.** Figure 4.5: Global step progression in AR model training. [PITH_FULL_IMAGE:figures/full_fig_p041_4_5.png]

**Figure 4.6.** Figure 4.6: Global step progression in GPT-2 model training. [PITH_FULL_IMAGE:figures/full_fig_p042_4_6.png]

**Figure 4.7.** Figure 4.7: D3PM training loss over global steps for different seeds. [PITH_FULL_IMAGE:figures/full_fig_p043_4_7.png]

**Figure 4.8.** Figure 4.8: Learning rate schedule followed by D3PM across different train [PITH_FULL_IMAGE:figures/full_fig_p043_4_8.png]

**Figure 4.** Figure 4: presents the progression of global training steps for each [PITH_FULL_IMAGE:figures/full_fig_p043_4.png]

**Figure 4.9.** Figure 4.9: Global step progression during D3PM training. [PITH_FULL_IMAGE:figures/full_fig_p044_4_9.png]

**Figure 7.1.** Figure 7.1: Transformer architecture used in autoregressive [PITH_FULL_IMAGE:figures/full_fig_p059_7_1.png]

**Figure 7.2.** Figure 7.2: AR model test results (seed = 1) [PITH_FULL_IMAGE:figures/full_fig_p063_7_2.png]

**Figure 7.3.** Figure 7.3: AR model generation output (seed = 12) [PITH_FULL_IMAGE:figures/full_fig_p063_7_3.png]

**Figure 7.4.** Figure 7.4: Autoregressive model test data results when seed= 1. [PITH_FULL_IMAGE:figures/full_fig_p063_7_4.png]

**Figure 7.5.** Figure 7.5: GPT2 model outputs when seed = 1000 [PITH_FULL_IMAGE:figures/full_fig_p064_7_5.png]

**Figure 7.6.** Figure 7.6: GPT2 model outputs when seed = 2000. E0000 00:00:1747486989.533439 295628 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2025-05-17 15:03:09.542551: I tensorflow/core/platfor…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages

[1]

Structured denoising diffusion models in discrete state-spaces

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981--17993, 2021

work page 2021
[2]

A survey on prompting techniques in llms

Prabin Bhandari. A survey on prompting techniques in llms. arXiv preprint arXiv:2312.03740 , 2023

arXiv 2023
[3]

Better Character Language Modeling Through Morphology

Terra Blevins and Luke Zettlemoyer. Better character language modeling through morphology. arXiv preprint arXiv:1906.01037 , 2019

work page Pith review arXiv 1906
[4]

Pixelsnail: An improved autoregressive generative model

Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning , pages 864--872. PMLR, 2018

work page 2018
[5]

What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771 , 2024

Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771 , 2024

arXiv 2024
[6]

Tokenization

Gregory Grefenstette. Tokenization. In Syntactic Wordclass Tagging , pages 117--133. Springer, 1999

work page 1999
[7]

Towards tokenization evaluation

Benoît Habert, Gilles Adda, Martine Adda-Decker, Philippe Boula de Mareüil, Silvana Ferrari, Olivier Ferret, Gabriel Illouz, and P Paraubeck. Towards tokenization evaluation. In LREC , pages 427--432, 1998

work page 1998
[8]

Symbolic word clustering for medium-size corpora

Benoît Habert, Elie Naulleau, and Adeline Nazarenko. Symbolic word clustering for medium-size corpora. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics , 1996

work page 1996

Show all 19 references

[9]

Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives

Marcel Hirt, Domenico Campolo, Victoria Leong, and Juan-Pablo Ortega. Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives. arXiv preprint arXiv:2309.00380 , 2023

2023 arXiv
[10]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840--6851, 2020

2020
[11]

Perplexity—a measure of the difficulty of speech recognition tasks

Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America , 62(S1):S63--S63, 1977

1977
[12]

Discrete diffusion modeling by estimating the ratios of the data distribution

Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834 , 2023

2023 arXiv
[13]

Pointer sentinel mixture models, 2016

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

2016
[14]

Medt: Using multimodal encoding-decoding network as in transformer for multimodal sentiment analysis

Qingfu Qi, Liyuan Lin, Rui Zhang, and Chengrong Xue. Medt: Using multimodal encoding-decoding network as in transformer for multimodal sentiment analysis. IEEE Access , 10:28750--28759, 2022

2022
[15]

Fast generation for convolutional autoregressive models

Prajit Ramachandran, Tom Le Paine, Pooya Khorrami, Mohammad Babaeizadeh, Shiyu Chang, Yang Zhang, Mark A Hasegawa-Johnson, Roy H Campbell, and Thomas S Huang. Fast generation for convolutional autoregressive models. arXiv preprint arXiv:1704.06001 , 2017

2017 arXiv
[16]

Simple and effective masked diffusion language models

Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems , 37:130136--130184, 2024

2024
[17]

Simplified and generalized masked diffusion for discrete data

Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in Neural Information Processing Systems , 37:103131--103167, 2024

2024
[18]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017

2017
[19]

Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts

Haofei Yu, Zhengyang Qi, Lawrence Jang, Ruslan Salakhutdinov, Louis-Philippe Morency, and Paul Pu Liang. Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts. arXiv preprint arXiv:2311.09580 , 2023

2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Structured denoising diffusion models in discrete state-spaces

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981--17993, 2021

work page 2021

[2] [2]

A survey on prompting techniques in llms

Prabin Bhandari. A survey on prompting techniques in llms. arXiv preprint arXiv:2312.03740 , 2023

arXiv 2023

[3] [3]

Better Character Language Modeling Through Morphology

Terra Blevins and Luke Zettlemoyer. Better character language modeling through morphology. arXiv preprint arXiv:1906.01037 , 2019

work page Pith review arXiv 1906

[4] [4]

Pixelsnail: An improved autoregressive generative model

Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning , pages 864--872. PMLR, 2018

work page 2018

[5] [5]

What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771 , 2024

Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771 , 2024

arXiv 2024

[6] [6]

Tokenization

Gregory Grefenstette. Tokenization. In Syntactic Wordclass Tagging , pages 117--133. Springer, 1999

work page 1999

[7] [7]

Towards tokenization evaluation

Benoît Habert, Gilles Adda, Martine Adda-Decker, Philippe Boula de Mareüil, Silvana Ferrari, Olivier Ferret, Gabriel Illouz, and P Paraubeck. Towards tokenization evaluation. In LREC , pages 427--432, 1998

work page 1998

[8] [8]

Symbolic word clustering for medium-size corpora

Benoît Habert, Elie Naulleau, and Adeline Nazarenko. Symbolic word clustering for medium-size corpora. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics , 1996

work page 1996