Tractable Transformers for Flexible Conditional Generation

Anji Liu; Dayuan Zhao; Guy Van den Broeck; Mathias Niepert; Xuejie Liu; Yitao Liang

REVIEW 4 major objections 6 minor 13 references

Tractable Transformers for Flexible Conditional Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tracformer claims that a sparse encoder with exponentially growing scopes closes the conditional-generation gap in non-autoregressive language models.

desk verdict Solid architecture paper with a real empirical observation, but the headline comparison against diffusion baselines rests on an unvalidated ELBO bound and a 'consistently outperforms' claim contradicted by its own tables. read the letter →

arxiv 2502.07616 v2 pith:NJZG3X2Q submitted 2025-02-11 cs.CL cs.LG

Anji Liu , Xuejie Liu , Dayuan Zhao , Mathias Niepert , Yitao Liang , Guy Van den Broeck This is my paper · ORCID

classification cs.CLcs.LG

keywords non-autoregressivegenerationconditionalquerygeneralizationmulti-scopesparseattentiontransformerencoder-decoderdiscretediffusionlanguagemodelsperplexitytextinfilling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Non-autoregressive language models are attractive for conditional generation because they can in principle answer queries that condition on arbitrary subsets of tokens, but the paper argues that strong unconditional modeling does not transfer to that setting. It shows that state-of-the-art discrete diffusion models, which match or beat GPT-2 on unconditional perplexity, produce conditional likelihoods that depend on which order or mask pattern the query uses, and that they get worse as more context is supplied. The paper proposes Tractable Transformers (Tracformer), an encoder-decoder in which the encoder builds features over exponentially growing local scopes and the decoder routes only context-consistent features to each token prediction. The paper claims Tracformer consistently outperforms BERT, BART, and current diffusion language models on conditional generation tasks, including zero-shot mask-ratio and span-length generalization, while remaining competitive on unconditional perplexity.

What carries the argument

The central object is the multi-scope sparse encoder, whose feature at position $t$ in layer $l$ has variable scope $\phi^l_t = \{t': 0 \le t-t' < 2^l\}$ for the prefix encoder (and its mirror for the suffix encoder). The encoder uses sparse attention masks that let each position attend to only a small set of boundary tokens per layer (at most $N_{\max}$), giving $O(L \cdot T \cdot N_{\max})$ encoder cost. The decoder is a cross-attention-only Transformer whose masks restrict each decoder embedding to encoder features consistent with the given context, enabling conditional predictions in either contextual-AR or arbitrary-context mode. This combination is what the paper credits for generalization to unseen conditional queries.

What would settle it

Use a stochastic likelihood estimator (for example, annealed importance sampling) to approximate the true conditional log-likelihood of SEDD and MDLM on the mask ranges of Table 2; if the estimated perplexities fall within the noise of Tracformer's numbers, the claim that Tracformer consistently beats these diffusion models would not be supported.

Watch

Extended reading notes

Core claim

Tracformer is a non-autoregressive Transformer built around the idea that each encoder feature should have a variable scope—the set of input tokens that can influence it—whose size doubles per layer: $\phi^l_t = \{t': 0 \le t-t' < 2^l\}$ for the prefix encoder and its mirror for the suffix encoder. This multi-scope sparse attention is the design choice that lets early layers produce local features invariant to distant mask tokens while later layers carry global context, and the decoder's cross-attention masks ensure each output token attends only to encoder features whose scopes lie inside the observed context. The paper's central empirical claim is that this design generalizes to conditional queries unseen during training—different mask ratios, span-length distributions, and arbitrary-context sets—producing lower conditional perplexity than BERT, BART, SEDD, and MDLM on text benchmarks, including zero-shot evaluations, while remaining competitive on unconditional perplexity despite a smaller model size.

Load-bearing premise

The comparison to diffusion baselines assumes the ELBO-based conditional perplexity is close to their true conditional likelihood; if that bound is loose, Tracformer's reported advantage over SEDD and MDLM could be inflated.

Editorial extensions

If this is right

On fixed mask ranges (Table 2), Tracformer reports lower zero-shot conditional perplexity than SEDD and MDLM on both WikiText-103 and 1BW.
Under ratio- and span-generalization tests, Tracformer's conditional perplexity degrades more gracefully than BART, BERT, and diffusion baselines as mask ratios or span-length distributions shift away from the training distribution.
Tracformer reaches competitive or better unconditional perplexity than a larger GPT-2 and diffusion models, showing that robustness to conditional queries does not require sacrificing unconditional modeling quality.
Because the encoder and decoder attention are sparse, the architecture scales to long sequences more cheaply than full-attention Transformers, with linear encoder cost and exponentially sparse cross-attention.
The arbitrary-context variant is a close match to one-step diffusion training, so Tracformer is a natural backbone for future diffusion language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The comparison to diffusion baselines rests on an ELBO proxy for their conditional perplexity; the paper does not establish how tight that bound is, so a portion of the reported gap could be a metric artifact. A tighter estimator would reveal whether the advantage is real.
Tracformer is only evaluated at GPT-2-base scale and at 1024-token sequences; whether the exponential-scope construction scales to larger models and longer contexts is untested.
The same sparse multi-scope attention masks could be installed in other NAR backbones, such as masked diffusion or BERT-style encoders, as a drop-in replacement for dense attention, which would test whether the benefit is architectural rather than tied to the full Tracformer training recipe.
Applications the paper motivates but does not evaluate, such as DNA imputation, protein infilling, and code editing, are exactly the arbitrary-context tasks where a query-generalizing NAR model should pay off; the paper's text results leave open whether the benefit transfers to those domains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does two useful things. First, it documents a real failure mode: discrete diffusion LMs like SEDD give order-dependent likelihoods, and their strong unconditional perplexity does not carry over to conditional queries. That is a clean, reproducible observation with the chain-rule consistency check and the order-sweep histogram. Second, the proposed Tracformer architecture—sparse multi-scope encoder with exponential scope growth plus a cross-attention decoder—is a simple, sensible answer to the query-generalization problem, and the controlled comparisons against BART and BERT on CAR/AC tasks are convincing. The sparse attention is a variation of known ideas, but the application to NAR conditional generation and the systematic evaluation of mask-ratio and span-length generalization is a real contribution.

The soft spots are mostly around the diffusion baseline comparison. Section 6 computes conditional PPL for SEDD and MDLM from a conditional NELBO (Appendix G) because exact likelihood is intractable. That is legitimate, but the bound's tightness is never checked. If the ELBO is loose, the baselines' true perplexities are lower and Tracformer's margin shrinks. The paper's own tables show at least one row (WikiText103, [0.2,0.3]&[0.4,0.6]) where MDLM beats Tracformer, so the text's claim of "consistently outperforms" overstates the evidence. The abstract's "state-of-the-art" is hedged, but the body should be more careful. There are also no error bars or multiple seeds, which matters for a comparison with margins this narrow in places.

That said, the central argument holds up. The architecture does improve conditional generalization relative to standard NAR backbones, and the query-generalization problem is real. The ELBO issue weakens the diffusion comparison but does not undermine the core finding. The conditional NELBO derivation looks correct under the stated assumptions.

Who is this for? Anyone working on NAR text generation, discrete diffusion, or controllable generation. It deserves a serious referee; the ELBO caveat and overclaim are fixable in revision, not fatal. I would engage with it.

Referee Report

4 major / 6 minor

Summary. The paper proposes Tractable Transformers (Tracformer), an encoder-decoder architecture with a sparse multi-scope encoder and a cross-attention decoder, designed for non-autoregressive (NAR) conditional text generation. It argues that strong unconditional performance of NAR diffusion models does not transfer to conditional generation because these models fail to generalize to conditional queries unseen during training, and it presents a local-plus-global feature design intended to improve such generalization. The paper reports experiments on WikiText103, LAMBADA, One Billion Words, and OpenWebText, comparing Tracformer against BERT, BART, SEDD, and MDLM, and it includes a derivation of a conditional NELBO for masked diffusion language models in Appendix G.

Significance. If the empirical claims hold, the paper makes a useful contribution by identifying query generalization as a distinct failure mode of NAR models and by proposing a concrete architectural remedy with released code and extensive experiments. The derivation of a conditional NELBO for MDLM in Appendix G is a useful technical extension of prior work. However, the central comparison against diffusion baselines is weakened by the use of an ELBO upper bound for baseline perplexity, the absence of error bars, and several overstatements in the text that are contradicted by the paper's own tables and figures.

major comments (4)

[Section 6.2, Table 2 and Figure 6] The text states that Tracformer 'consistently outperforms' SEDD and MDLM across all mask ranges in Table 2, but the WikiText103 row for mask range [0.2,0.3]&[0.4,0.6] shows MDLM at 22.97 versus Tracformer at 24.31, and Figure 6 shows MDLM with lower conditional perplexity at low mask ratios on WikiText103. The claim of consistent superiority should be revised to describe the actual pattern, and the analysis should discuss in which regimes Tracformer's advantage appears and where it does not.
[Section 6 and Appendix G] The central comparison in Table 2 and Figure 6 computes baseline conditional perplexity from the conditional NELBO derived in Appendix G, which is an upper bound on the true negative log-likelihood, while Tracformer's CAR perplexity is an exact autoregressive likelihood. The paper never quantifies the tightness of this bound, so the reported margins may exaggerate Tracformer's advantage over the diffusion baselines. Additionally, the derivation in Appendix G is for MDLM's masked diffusion and relies on the 'Carry Over Masking' property; the paper does not justify that this property and the resulting NELBO formula apply to SEDD, whose training objective estimates score ratios rather than a masked-token classifier. The evaluation should include a tightness check (for example, importance-sampling estimates of the true likelihood on short sequences) or should otherwise temper the claim that Tracformer beats state-of-the-art diffusion language models.
[Section 6, all tables and figures] No error bars or multiple-seed results are reported. Several headline differences are small (for example, Table 1 MAUVE 0.960 vs 0.951 for the [0.25,0.75] mask range, and Figure 5 shows small gaps near the training mask ratio), and without variance estimates it is not possible to assess whether the claimed consistent improvements are statistically meaningful. The paper should report standard errors or results from multiple training runs for at least the central comparisons in Table 2 and Figure 6.
[Section 6.2, Table 4] The text claims that Tracformer 'remains highly competitive, achieving results comparable to or better than larger models' in unconditional perplexity, but Table 4 shows that on PTB Tracformer's 166.10 is worse than all baselines (GPT-2 138.43, SEDD 114.24, MDLM 96.40), and Tracformer is also worse than GPT-2 on WikiText103 (43.27 vs 41.60), WikiText2 (43.82 vs 42.32), and LAMBADA (58.10 vs 45.04). The text should be corrected to state the actual pattern or the claim should be qualified to the datasets and models where it holds.

minor comments (6)

[Section 4.1] The claim that 'only O(T) 1s in every attention mask' are needed is followed by the introduction of Nmax, which actually yields O(T·Nmax) nonzeros per mask; the text should state the dependence on Nmax consistently.
[Appendix E] The per-layer decoder cost is written as O(T·2^l), but for the mask in Eq. (5) the number of attended keys per query is roughly T/2^{L-l+1}, which decreases with layer index; the final O(T^2) total is correct, but the intermediate expression is misleading.
[Appendix B.3] The main text says that conditional PPL results for LAMBADA and 1BW are deferred to Appendix B.3, but the appendix reports only MAUVE and BERT-score infilling results for those datasets; the promised PPL results appear to be missing.
[Section 7] The Limitations section does not acknowledge that the diffusion baselines in the main comparison are evaluated with an upper-bound NELBO, which is a key uncertainty in the paper's central claim.
[Appendix B.2] The model name 'BERT-bidder' appears in the text and should be 'BERT-bidir' for consistency with the rest of the paper.
[Title and Section 4] The term 'Tractable' in the title is not defined or justified; if it refers to tractable inference, the paper does not discuss what tractable operations are supported beyond likelihood computation in CAR mode.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the architecture and training objectives are new, the conditional-NELBO baseline evaluation is an independent extension of prior work, and the only self-citation is non-load-bearing.

full rationale

I walked the paper's derivation chain. Tracformer's contribution is an architectural proposal (Section 4) with training objectives defined directly as negative log-likelihood losses in Equations (7) and (8); no parameter is fitted to the evaluation quantities and then reported as a prediction. The conditional perplexity results in Section 6 are computed on held-out masking ratios and span distributions, which are not used to fit the model. The conditional NELBO derivation in Appendix G is an extension of the unconditional NELBO of Sahoo et al. (2024), not of the present authors' work: it conditions by clamping the observed tokens xS into the denoising network (Equation 15) and applies Jensen's inequality to obtain an upper bound on the conditional negative log-likelihood. The final continuous-time expression in Equation (22) is not definitionally equal to Tracformer's objective in Equation (7); it is an independent estimator for the diffusion baselines. The only self-citation is to Liu et al. (2025) (Discrete Copula Diffusion) in Related Work, where it is cited merely as an example of combining diffusion models with other generative models; it is not load-bearing for any architectural or empirical claim. The ELBO-tightness concern raised for the diffusion baseline perplexities is a measurement/fairness risk, not a circularity, because the ELBO is an upper bound computed from the baselines' own models and does not reduce Tracformer's reported advantage to its own inputs. Accordingly, no circular step was identified.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The free parameters are standard architecture hyperparameters. The key assumptions are domain assumptions about the baseline diffusion models and the validity of ELBO as a comparison metric. The conditional NELBO derivation is rigorous under these assumptions, but the ELBO validity assumption is the most fragile.

free parameters (2)

Nmax (max attended tokens in encoder) = 16 (small-scale), 32 (large-scale)
Integer hyperparameter controlling the sparsity of the multi-scope attention. Chosen by hand to balance efficiency and performance, not fitted to the target metric.
Number of encoder/decoder layers L = 10
Architecture hyperparameter. Must be at least log2(T) to achieve full context, but specific value is a design choice.

assumptions (3)

domain assumption The MDLM model satisfies the 'Carry Over Masking' property, i.e., x_theta(z_t, t) = z_t when z_t is the mask token.
Invoked in Appendix G.2 to derive the conditional NELBO for MDLM baselines. The property is stated in the original MDLM paper, but not re-verified here.
domain assumption The forward noising and denoising processes for diffusion models factorize independently across tokens.
Used in Appendix G.2 to decompose the conditional NELBO into per-token losses. This is a standard assumption in discrete diffusion models.
domain assumption ELBO (negative evidence lower bound) is a valid proxy for true conditional log-likelihood when comparing models.
Implicit in Section 6 when computing conditional perplexity for diffusion baselines. The tightness of the bound is not assessed, and this assumption is load-bearing for the comparative claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tractable Transformers for Flexible Conditional Generation." pith.science (2026). https://pith.science/paper/NJZG3X2Q

@misc{pith2026250207616,
  author       = {Pith},
  title        = {Pith review of: Tractable Transformers for Flexible Conditional Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NJZG3X2Q}},
  note         = {Machine review of arXiv:2502.07616}
}

read the original abstract

Non-autoregressive (NAR) generative models are valuable because they can handle diverse conditional generation tasks in a more principled way than their autoregressive (AR) counterparts, which are constrained by sequential dependency requirements. Recent advancements in NAR models, such as diffusion language models, have demonstrated superior performance in unconditional generation compared to AR models (e.g., GPTs) of similar sizes. However, such improvements do not always lead to improved conditional generation performance. We show that a key reason for this gap is the difficulty in generalizing to conditional probability queries (i.e., the set of unknown variables) unseen during training. As a result, strong unconditional generation performance does not guarantee high-quality conditional generation. This paper proposes Tractable Transformers (Tracformer), a Transformer-based generative model that is more robust to different conditional generation tasks. Unlike existing models that rely solely on global contextual features derived from full inputs, Tracformers incorporate a sparse Transformer encoder to capture both local and global contextual information. This information is routed through a decoder for conditional generation. Empirical results demonstrate that Tracformers achieve state-of-the-art conditional generation performance on text modeling compared to recent diffusion and AR model baselines.

Figures

Figures reproduced from arXiv: 2502.07616 by the authors.

**Figure 1.** Zero-shot conditional perplexity (↓) of AR (GPT-2 (Radford et al., 2019)) and NAR models (SEDD (Lou et al., 2023), MDLM (Sahoo et al., 2024)) models conditioned on different proportions of prefix given sequences of length 128. Existing NAR models perform worse than GPT-2 despite having comparable unconditional perplexity on sequences of length 1024. Their performance gets worse when provided with more context. 1.… view at source ↗

**Figure 2.** Different conditional queries do not respect the same joint distribution in NAR models. (a) The log-likelihoods (LLs) of the sequence “The cat is cute” differ depending on whether SEDD is queried in the forward order or the reverse order. (b) Histogram of the difference between the highest and the lowest LLs of 2,000 length-5 sequences from WikiText103 when queried in all possible orders. While the FFN is applied in… view at source ↗

**Figure 3.** Tractable Transformers (Tracformers) use an encoder to learn feature embeddings (i.e., h enc,l t ) with context (denoted as scope on the left) sizes ranging from 1 to 2 L , where L is the number of layers. The encoder features are then fed to a decoder for conditional generation. Each decoder layer contains a feed-forward neural network and a cross-attention layer to collect information from corresponsing encoder fe… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Tracformers for contextual AR (CAR) and arbitrary-context (AC) generation. In both cases, we use a prefix encoder and a suffix encoder to capture prefix and suffix information for every token, respectively. Model parameters of the two encoders are shared. The decoder r…

**Figure 5.** Figure 5: Validation conditional perplexity (↓) of CAR/AC generation tasks with varying masking strategies, evaluated on the WikiText103 validation set with a sequence length of 1024. (a) and (b) show CAR generation with span lengths sampled from Geometric(µ = 50) and Geometric(…

**Figure 6.** Figure 6: Conditional perplexity under varied masking strategies, evaluating ratio-generalization and span-generalization. We evaluate two [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Conditional perplexity (PPL) evaluated on the WikiText-103 validation set. The span length follows a [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 6 canonical work pages

[1]

⟨1, xℓ θ(zℓ t , t)⟩ = 1

The output of xℓ θ(zℓ t , t) is a probability distribution over all categories, i.e. ⟨1, xℓ θ(zℓ t , t)⟩ = 1

work page
[2]

Zero Masking Probabilities

“Zero Masking Probabilities” i.e. ⟨xℓ θ(zℓ t , t), m⟩ = 0

work page
[3]

Carry Over Unmasking

“Carry Over Unmasking” i.e. xℓ θ(zℓ t , t) =xℓ when zℓ t = xℓ. which all happens to be properties that MDLM models share, and therefore also holds for xℓ θ([zF −S t , xS], t). Thus, by substituting xθ(zt, t) with xℓ θ([zF −S t , xS], t), we can apply the results from the single-token case and derive the following equation from Eqn. 42 of the original pape...

work page
[4]

Bart: Denois- ing sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension

1 Lewis, M., Liu, Y ., Goyal, N., et al. Bart: Denois- ing sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2020. 2, 6, 7 Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-lm improves controllable text generation. Advances in Neural Inform...

arXiv 1910
[6]

Discrete diffusion lan- guage modeling by estimating the ratios of the data distri- bution

1 Lou, A., Meng, C., and Ermon, S. Discrete diffusion lan- guage modeling by estimating the ratios of the data distri- bution. arXiv preprint arXiv:2310.16834, 2023. 1, 2, 3, 6, 8, 12, 16 Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022. 2, 3, 6 Nguyen, E., P...

arXiv 2023
[7]

Chatgpt (gpt-4), 2023

2 OpenAI. Chatgpt (gpt-4), 2023. URL https://chat. openai.com. 2 Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. 6 Pillutla, K., Swayamdipta, S., Zellers, R., Thick-...

arXiv 2023
[128]

new beginning

The prompt texts are bolded and in blue: track-two diplomacy between U.S. and Soviet officials and nuclear scientists, which helped lead to the Compre- hensive Test Ban Treaty. MacArthur grantees also helped develop real alternatives to the IAEA, such as providing uranium to New Zealand. The centerpiece was the launch of a satellite called WUSA.\n\nFinanc...

work page 2024
[2018]

The llama 3 herd of models

2, 3, 6, 7 9 Tractable Transformers Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 1 Fu, D. Y ., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with sta...

arXiv 2024

Show all 13 references

[2019]

1, 2, 8 10 Tractable Transformers Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1–67, 20...

2020 arXiv
[2020]

B., Mann, B., Ryder, N., et al

7 Brown, T. B., Mann, B., Ryder, N., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020. 1, 2 Campbell, A., Benton, J., De Bortoli, V ., Rainforth, T., Deli- giannidis, G., and Doucet, A. A continuous time frame-...

1901 arXiv
[2023]

D., Ho, J., Tarlow, D., and Van Den Berg, R

1 Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021. 3, 6, 8 Bi, B., Li, C., Wu, C., Yan, M., Wang, W., Huang, S., Huang, ...

2021 arXiv
[2024]

Discrete copula diffusion

1 Liu, A., Broadrick, O., Niepert, M., and Van den Broeck, G. Discrete copula diffusion. In The Thirteenth International Conference on Learning Representations, 2025. 6 Liu, F., Fu, Z., Li, G., Jin, Z., Liu, H., Hao, Y ., and Zhang, L. Non-autoregressive line-level code comple...

2025
[2025]

very difficult interplay of life and conflict

6 Zhang, T., Kishore, V ., Wu, F., Weinberger, K. Q., and Artzi, Y . BERTScore: Evaluating text generation with BERT. In International Conference on Learning Rep- resentations, 2020. URL https://openreview.net/ forum?id=SkeHuCVFDr. 8 11 Tractable Transformers A. Additional Det...

2020

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

⟨1, xℓ θ(zℓ t , t)⟩ = 1

The output of xℓ θ(zℓ t , t) is a probability distribution over all categories, i.e. ⟨1, xℓ θ(zℓ t , t)⟩ = 1

work page

[2] [2]

Zero Masking Probabilities

“Zero Masking Probabilities” i.e. ⟨xℓ θ(zℓ t , t), m⟩ = 0

work page

[3] [3]

Carry Over Unmasking

“Carry Over Unmasking” i.e. xℓ θ(zℓ t , t) =xℓ when zℓ t = xℓ. which all happens to be properties that MDLM models share, and therefore also holds for xℓ θ([zF −S t , xS], t). Thus, by substituting xθ(zt, t) with xℓ θ([zF −S t , xS], t), we can apply the results from the single-token case and derive the following equation from Eqn. 42 of the original pape...

work page

[4] [4]

Bart: Denois- ing sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension

1 Lewis, M., Liu, Y ., Goyal, N., et al. Bart: Denois- ing sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2020. 2, 6, 7 Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-lm improves controllable text generation. Advances in Neural Inform...

arXiv 1910

[5] [6]

Discrete diffusion lan- guage modeling by estimating the ratios of the data distri- bution

1 Lou, A., Meng, C., and Ermon, S. Discrete diffusion lan- guage modeling by estimating the ratios of the data distri- bution. arXiv preprint arXiv:2310.16834, 2023. 1, 2, 3, 6, 8, 12, 16 Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022. 2, 3, 6 Nguyen, E., P...

arXiv 2023

[6] [7]

Chatgpt (gpt-4), 2023

2 OpenAI. Chatgpt (gpt-4), 2023. URL https://chat. openai.com. 2 Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. 6 Pillutla, K., Swayamdipta, S., Zellers, R., Thick-...

arXiv 2023

[7] [128]

new beginning

The prompt texts are bolded and in blue: track-two diplomacy between U.S. and Soviet officials and nuclear scientists, which helped lead to the Compre- hensive Test Ban Treaty. MacArthur grantees also helped develop real alternatives to the IAEA, such as providing uranium to New Zealand. The centerpiece was the launch of a satellite called WUSA.\n\nFinanc...

work page 2024

[8] [2018]

The llama 3 herd of models

2, 3, 6, 7 9 Tractable Transformers Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 1 Fu, D. Y ., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with sta...

arXiv 2024