Pith. sign in

REVIEW 10 cited by

GIVT: Generative Infinite-Vocabulary Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.02116 v4 pith:LVT4BEDV submitted 2023-12-04 cs.CV

Michael Tschannen , Cian Eastwood , Fabian Mentzer This is my paper · ORCID
classification cs.CV
keywords givttransformerslatentmodelsequencesdiscretegenerationgenerative
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

We introduce Generative Infinite-Vocabulary Transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT, where transformers are used to model the discrete latent sequences of a VQ-VAE, we use GIVT to model the unquantized real-valued latent sequences of a $\beta$-VAE. In class-conditional image generation GIVT outperforms VQ-GAN (and improved variants thereof) as well as MaskGIT, and achieves performance competitive with recent latent diffusion models. Finally, we obtain strong results outside of image generation when applying GIVT to panoptic segmentation and depth estimation with a VAE variant of the UViM framework.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Estimating Musical Surprisal in Audio

    cs.SD 2025-01 conditional novelty 7.0 of 10

    An autoregressive model over compressed audio latents yields an information-content measure of musical surprisal that predicts EEG responses to songs and correlates with musical complexity features.

  2. XYZFlow:Scaling Multi dimensional Shortcut Flows for Efficient Generative Modeling

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Conditioning each patch's denoising on the full trajectories of earlier patches lets XYZFlow generate ImageNet images with FID 1.22 to 1.63 in only 2 to 5 steps per patch, at 7.2 to 8.5x teacher speedups.

  3. Enhancing next token prediction based pre-training for jet foundation models

    hep-ph 2025-12 conditional novelty 6.0 of 10

    Using continuous particle features as input and combining next-token with masked-token pre-training markedly improves classification accuracy of the OmniJet jet foundation model without visibly hurting its generative quality.

  4. Transition Matching: Scalable and Flexible Generative Modeling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Transition Matching unifies flow matching and continuous autoregressive generation as discrete-time Markov processes, with three variants that improve text-to-image quality and speed.

  5. DiSA: Diffusion Step Annealing in Autoregressive Image Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Autoregressive image diffusion models can use far fewer denoising steps for later tokens without losing quality, yielding 1.4-2.5x speedup from step annealing and up to 10x when combined with fewer autoregressive steps.

  6. Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Token-based language-style vision models (VAR, LlamaGen) tolerate quantization better than diffusion models, and a custom TopKLD distillation loss pushes their low-bit scaling roughly one precision level higher.

  7. RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A recurrent token-prediction model that adds noise during quantization and generates images by predicting discrete codes over ten steps, reaching FID 2.56 on ImageNet 256x256.

  8. Continuous Autoregressive Models with Noise Augmentation Avoid Error Accumulation

    cs.LG 2024-11 conditional novelty 5.0 of 10

    Injecting random noise into input embeddings during training lets purely autoregressive models generate continuous audio embeddings without quality degradation over long sequences.

  9. MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    An empirical testbed shows that allocating more transformer layers to autoregressive conditioning improves FID under low inference budgets, while block size should scale with image resolution.

  10. Self-control: A Better Conditional Mechanism for Masked Autoregressive Model

    cs.CV 2024-12 reject novelty 3.0 of 10

    The paper proposes a self-attention conditioning mechanism for continuous masked autoregressive image generation, but reports no experiments to support it.

Pith tools