Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multidimensional byte pair encoding shortens visual token sequences and makes autoregressive transformers converge faster to better FID scores.

desk verdict Useful generalization of BPE to 2D/3D token grids with consistent evidence of improved autoregressive generation; the core idea holds, but the paper needs to fix an anchor-consistency gap and report variance. read the letter →

arxiv 2411.10281 v1 pith:HOI2SLEZ submitted 2024-11-15 cs.CV cs.LG

Tim Elsner , Paula Usinger , Julius Nehring-Wirxel , Gregor Kobsik , Victor Czech , Yanjiang He , Isaak Lim , Leif Kobbelt This is my paper · ORCID
classification cs.CVcs.LG
keywords bytepairencodingvisualtokenisationautoregressivegenerationsequencecompressionimagetransformersvoxelgridsVQ-VAEcodebookcollapse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that visual data generation improves when token sequences are shortened by a multidimensional extension of Byte Pair Encoding before a transformer ever sees them. The method, MDBPE, counts frequent neighbouring token constellations in horizontal and vertical directions, merges the most frequent constellation into a new token, and repeats until a vocabulary budget is reached; because the merge records anchor points and offset vectors, it is lossless and can form non-convex token shapes. The result is sequences that are shorter and whose information is more evenly distributed, condensing empty regions into single tokens. The paper reports that autoregressive transformers trained on these sequences converge earlier and reach better FID scores on several image datasets, and that the same recipe transfers to 3D voxel grids and signed-distance-function lattices. The cost of the extra dimension is a factor of two in preprocessing time compared with 1D BPE, which the paper argues is negligible relative to transformer training.

What carries the argument

The load-bearing object is the constellation count: instead of counting adjacent token pairs along one axis as 1D BPE does, MDBPE slides a pairwise mask horizontally and vertically and counts triples of token class, neighbour class, and offset vector, where the anchor point of a token is the leftmost pixel in its uppermost row and the offset vector records the alignment between two anchors. Each pixel also carries a unique instance ID, so that when a constellation is chosen for merging, every pixel in both tokens can be re-labelled with a new token class and one shared ID; the final sequence is produced by scanning top-to-bottom, left-to-right and emitting each unique ID once at its anchor. This machinery is what lets the algorithm compress in two or three dimensions, form non-convex merged tokens, remain invertible, and keep the extra cost at roughly twice the pair count of 1D BPE.

What would settle it

Train the same transformer on sequences shortened by the least frequent token constellations instead of the most frequent ones, keeping sequence length and vocabulary size matched; if FID still improves over the uncompressed baseline, then shortness alone drives the effect and the frequency-based merge rule is not essential. Alternatively, match sequence length across two different token-shape vocabularies: if the FID gap persists, token shape carries part of the benefit.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that shortening visual token sequences by merging frequent constellations of neighbouring tokens makes them easier for transformers to process, not harder, even though the vocabulary grows. MDBPE is a lossless preprocessing add-on to existing discrete tokenisations: given a grid of VQ-VAE or VQGAN codes, greyscale values, or quantised colours, it repeatedly replaces the most frequent horizontally or vertically adjacent token pair with a fresh token, using anchor points and offset vectors to tell different alignments of the same pair apart. The extracted sequence writes each token once at its anchor position, so the sequence is shorter and each token can cover a larger area. Across MNIST, CIFAR-10, SVHN, CelebA, and ImageNet, the paper finds that transformers on these condensed sequences converge earlier, reach lower test loss and better FID, and can generate $32^{3}$ voxel grids that would not fit in consumer GPU memory without compression. A lossy extension that clusters the codebook into fewer representative tokens further amplifies compression, and extra positional encodings describing token area and next-token position improve learning.

Load-bearing premise

The paper's main load-bearing premise is that merging the most frequent token constellations, chosen by global training-set frequency counts, leaves exactly the information a transformer needs for generation; this frequency heuristic is empirical, and the claim that anchor-point extraction stays losslessly invertible for arbitrary non-convex token shapes is asserted rather than proved.

Editorial extensions

If this is right

  • Autoregressive image generation models trained on MDBPE-compressed sequences converge earlier and reach better final FID scores than the same transformers trained on uncompressed VQ-VAE, VQGAN, or colour-quantised sequences on MNIST, CIFAR-10, SVHN, and CelebA.
  • Because attention cost scales quadratically with sequence length, shorter sequences translate directly into lower FLOPs, lower memory, and larger batch sizes, so the same consumer GPU can train models on compressed sequences faster.
  • The 3D extension compresses voxel and SDF token grids strongly enough, for example to 47% of original length on 8^3 SDF grids, to bring 32^3 voxel autoregressive generation within consumer GPU memory, where uncompressed attention would not fit.
  • Lossy codebook collapse, snapping a VQGAN codebook of 2048 entries down to far fewer representatives, roughly preserves visual fidelity while shortening sequences further, so it can be used as a cheap speed-up for prototyping and training.
  • Diversity is not sacrificed: the paper reports train and test losses that stay close even under strong compression, and nearest-neighbour checks on CelebA show the model is not simply memorising training images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The frequency-only merge rule is the main unproved choice; a controlled comparison against entropy-based or difficulty-weighted merge criteria would show whether the gains come from shortening itself or from the specific shapes frequency selection produces.
  • The paper's evidence is on standalone autoregressive transformers; whether the same shortening helps multimodal LLMs that consume visual tokens is an extension the authors suggest but do not test.
  • If the shortness effect is the active ingredient, a minimal test would be to hold sequence length fixed and vary only token shapes, or hold shapes fixed and vary sequence length, to separate the two factors.
  • The lossy codebook collapse suggests that posterior clustering of a trained discrete codebook can substitute for expensive retraining of a smaller-vocabulary VQGAN, a recipe that could transfer to other discrete generative pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Multidimensional Byte Pair Encoding (MDBPE), a tokenization preprocessing step that extends BPE from 1D text to 2D image and 3D voxel grids. The algorithm counts frequent pairs of adjacent tokens together with their relative alignment (constellation vector), merges the most frequent constellation into a new token, and outputs a condensed sequence of token classes at anchor points. The paper also proposes two positional encoding modifications (next-token position and integrated spatial coverage) and a lossy codebook-collapse variant. Experiments on MNIST, CIFAR-10, SVHN, CelebA, and ShapeNet report improved FID, Coverage, and MMD for autoregressive transformers compared to vanilla transformers, pixel-level BPE, and the Octree Transformer.

Significance. MDBPE is a simple and potentially compelling add-on to existing visual tokenizers: it is agnostic to the base quantizer, is claimed to be lossless, and, according to Appendix A, increases preprocessing cost by only a constant factor. The authors provide code and evaluate across multiple datasets and modalities, which is a strength. If the losslessness and the attribution of the improvements are rigorously established, the method could be a broadly useful preprocessing technique for visual autoregressive models.

major comments (4)
  1. [Sec. 4.2 / Algorithm 1] The paper claims that MDBPE is a 'lossless preprocessing step' (Abstract, Sec. 4), but it never defines the inverse mapping from a condensed token sequence back to the original grid. Algorithm 1 stores only token classes and unique IDs; it does not maintain a dictionary that maps each newly introduced token class to its spatial footprint (the set of covered pixels relative to the anchor). Without such a dictionary, a generated sequence cannot be rendered into an image, and the losslessness claim cannot be verified. Furthermore, the definition of an anchor as 'the leftmost pixel in the uppermost row' (Sec. 4.1) is stated as a property of a token, but Algorithm 1 does not update any stored anchor when two tokens are merged (lines 23-28). The paper should clarify whether the anchor is a stored field or recomputed on the fly; a merge of a token with anchor (1,1) and a vertical token with anchor (2,0) (adjacent via the pixel at (2,1)) yields a union whose row-major first pixel is (2,0), not the left constituent's anchor (1,1). This ambiguity directly affects the sequence-extraction rule in Sec. 4.2 and the next-token positional encoding in Sec. 4.3.
  2. [Sec. 5.3 / Table 1 / Fig. 14] The comparison to Pixel-Level BPE (Razzhigaev et al.) is confounded. The MDBPE pipeline includes novel positional encodings introduced in Sec. 4.3 (Next Token Encoding and Integrated Positional Encoding), while the 1D BPE baseline appears not to use them. Since Fig. 7 shows that these encodings reduce test loss on their own, the FID gaps in Table 1 (e.g., 36.53 vs 33.93 on CIFAR-10 VQ-VAE) cannot be attributed solely to multidimensional compression. The authors should ablate the new encodings for both the 1D BPE baseline and MDBPE, or add a 1D BPE baseline with the same encodings, to isolate the effect of multidimensionality.
  3. [Tables 1, 2, 4, 5] All reported FID, Coverage, and MMD values are single-run point estimates without error bars or multiple seeds. Several improvements are small (e.g., CelebA VQ-VAE: 52.15 to 50.87 in Table 1; CIFAR-10 VQ-VAE: 36.53 to 33.93 in the same table), so the claimed gains might not be statistically significant. Please provide means and standard deviations over at least three independent training runs for the main comparisons.
  4. [Sec. 4.4 / Table 5 / Fig. 10] The training-speed boost is partly achieved by pruning the 5% of sequences with the longest length ('lastly, we further boost training speed by pruning the dataset from the 5% with the longest sequence length'). Removing the longest sequences changes the training set and reduces the effective padding length. The paper does not state whether the vanilla and 1D BPE baselines are trained on the same pruned data; if they are not, the comparisons in Table 5 and Fig. 10 are not controlled. The authors should either apply identical pruning to all methods or report the results without pruning.
minor comments (5)
  1. [Tables 1 and 5] The label 'Compression' is used with opposite conventions: in Table 1, lower values correspond to shorter sequences (e.g., 59% for MDBPE vs 65% for 1D BPE), but in Table 5 the header 'Compression↓' with 100% for the 'Full' codebook implies that lower is better as well. This should be clarified consistently, e.g., by stating explicitly that the numbers denote the remaining sequence length as a percentage of the original.
  2. [Table 2 and Sec. 5.2] Several resolutions are typeset ambiguously as '282', '322', '162', '323'; these should be written as 28x28, 32x32, 16x16, and 32x32x32 for readability.
  3. [Sec. 4.3, Fig. 5] The notation for positional encodings is inconsistent: the text uses 'positional encoding(i)' in the IPE formula, while Fig. 5 uses 'Pe' and 'P_e'. Use a single symbol throughout.
  4. [Footnote 1] The paper states that the full codebase, including the faster C++ implementation, will be released, but only a small MNIST demo is currently linked. Please include the full code at submission time so that the reported results can be reproduced.
  5. [Sec. 5.1] The sentence 'VQ-VAEs are often regularised to avoid unused codebook entries' is followed by a list of citations, but the exact form of regularisation used in the experiments (e.g., codebook reset or commitment loss) is not described. Please specify the regularisation method in Appendix F.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MDBPE is an empirical preprocessing heuristic evaluated against external FID benchmarks; the paper's self-citations are only related work and baselines.

full rationale

The central claim—that frequency-based multidimensional merging shortens token sequences and that transformers trained on these sequences reach better FID—is not obtained by fitting a model to the FID target or by defining the output in terms of the input. Algorithm 1 selects constellations using only training-set co-occurrence counts; test-set compression is measured on unseen data, and generation quality is measured by FID against external test sets. The extra positional encodings (IPE/NPE) are additional inputs, not quantities derived from the quantities they predict. The paper's self-citations, [18] (Quantised Global Autoencoder) and [27] (Octree Transformer), appear as related work or comparison baselines and are not invoked to justify losslessness, compression, or the reported FID improvements. The lossless claim itself is asserted without the formal invariant that would prove anchor consistency after merges; that is a correctness gap, and the anchor-update omission in Algorithm 1 is a genuine implementation concern, but it is not circularity: an unproven or even false guarantee is not the same as a result that equals its own input by construction. Likewise, the Appendix E admission that SDF results are not published and the limited scope of the VQGAN-collapse experiments reduce the strength of the empirical claims without making them circular. No self-definitional step, fitted-input-as-prediction step, self-citation chain, or renaming of a known result was found.

Assumptions & free parameters 4 free parameters · 3 assumptions · 3 invented entities

The central method introduces no physical parameters, but the algorithm's effectiveness depends on the frequency-counting heuristic, the lossless claim, and the empirical hypothesis that shorter sequences are better. The main free parameters are the extra-token budget, the codebook-collapse K, and two training heuristics (learning rate, sequence pruning). The invented entities are algorithmic constructs with no external-evidence requirement.

free parameters (4)
  • Number of extra tokens (vocabulary size increase) = 256, 512, 1024, 2048 (chosen per dataset)
    Hyperparameter swept in Fig. 11, with an optimum at 1024 extra tokens for CIFAR-10. It directly controls the sequence-length reduction and the reported FID gains.
  • Codebook collapse cluster count K = 16, 32, 64, 128 (lossy variant)
    Lossy compression strength is chosen experimentally in Tab. 5 and Fig. 16; smaller K gives stronger compression at the cost of higher FID.
  • Learning rate for compressed sequences = 0.0002 (vs 0.001 for baseline)
    The paper sets a lower learning rate for compressed sequences, claiming 'a slight boost in performance'. The extent to which this biases the FID comparison is not quantified.
  • Pruning fraction of longest training sequences = 5%
    The 5% longest sequences are pruned to reduce padding overhead; this is a heuristic that may also remove hard examples and improve reported FID.
assumptions (3)
  • domain assumption Global frequency of token constellations on the training set is a sufficient criterion for selecting which pairs to merge so that the compressed sequence preserves information useful for generation.
    Algorithm 1 uses argmax of counts to pick the next constellation. The paper provides no theoretical justification or comparison to alternative merging criteria; the assumption is supported only by empirical FID results on a few datasets.
  • domain assumption The anchor-point extraction (write each unique token once at its anchor) yields a lossless representation of the original grid.
    The paper asserts 'This is a lossless preprocessing step' but does not prove that the sequence of anchor tokens plus the fixed token shapes uniquely reconstructs every possible grid, especially for non-rectangular merged tokens.
  • domain assumption Shorter token sequences are easier for transformers to optimise in visual generation.
    The core hypothesis is motivated by NLP BPE. The paper tests it empirically, but the underlying mechanism (fewer positions, more uniform information density) is not formally established.
invented entities (3)
  • Extra tokens (merged constellations) independent evidence
    purpose: Represent frequent local patterns as single vocabulary entries, shortening sequences and increasing information density.
    The extra tokens are produced by the algorithm and directly observed in the compressed sequences; their utility is demonstrated in generation evaluations.
  • Anchor point coordinate system
    purpose: Provide a canonical reference point for each token and define the constellation vector between pairs.
    A purely algorithmic device introduced to enable multidimensional pair counting; it has no external falsifiable handle.
  • Integrated positional encoding (IPE)
    purpose: Inform the transformer about the spatial area covered by each variable-shape token.
    An architectural component proposed by the paper, similar in spirit to Mip-NeRF's integrated encoding but applied to discrete token grids.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation." pith.science (2026). https://pith.science/paper/HOI2SLEZ

@misc{pith2026241110281,
  author       = {Pith},
  title        = {Pith review of: Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOI2SLEZ}},
  note         = {Machine review of arXiv:2411.10281}
}
read the original abstract

In language processing, transformers benefit greatly from text being condensed. This is achieved through a larger vocabulary that captures word fragments instead of plain characters. This is often done with Byte Pair Encoding. In the context of images, tokenisation of visual data is usually limited to regular grids obtained from quantisation methods, without global content awareness. Our work improves tokenisation of visual data by bringing Byte Pair Encoding from 1D to multiple dimensions, as a complementary add-on to existing compression. We achieve this through counting constellations of token pairs and replacing the most frequent token pair with a newly introduced token. The multidimensionality only increases the computation time by a factor of 2 for images, making it applicable even to large datasets like ImageNet within minutes on consumer hardware. This is a lossless preprocessing step. Our evaluation shows improved training and inference performance of transformers on visual data achieved by compressing frequent constellations of tokens: The resulting sequences are shorter, with more uniformly distributed information content, e.g. condensing empty regions in an image into single tokens. As our experiments show, these condensed sequences are easier to process. We additionally introduce a strategy to amplify this compression further by clustering the vocabulary.

Figures

Figures reproduced from arXiv: 2411.10281 by the authors.

Figure 1
Figure 1. Our algorithm compresses visual data in order to make [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 4
Figure 4. For predicting the next token at a position (red cross), we can reroll to￾kens that exceed the boundary (orange) or overlap with existing tokens (blue). Algorithm 1 Multidimensional Byte Pair Encoding 1: class ← class of the token a pixel belongs to 2: id ← unique ID of the token a pixel belongs to 3: while |vocabulary| < K do 4: count ← 0 5: for all images do 6: used ← {} 7: for all grid points g in the image do 8:… view at source ↗
Figure 5
Figure 5. In addition to giving a positional encoding of the token [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (10 more)
Figure 6
Figure 6. Figure 6: Exemplary compression factors across different datasets [ [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 9
Figure 9. Figure 9: Loss plot comparing lossless variant to lossy (codebook collapse for K = 64). Collapsed trains for 14 epochs in the same time lossless trains for 10 (shorter sequences allow larger batches). tra tokens, and refer to the impact on the sequence length as compression: A c…
Figure 10
Figure 10. Figure 10: Comparing generation quality of ours with [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Generation performance boost from introducing extra [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Upper row: Generated image, lower row: Closest in Inception[ [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 14
Figure 14. Figure 14: Development of FID values when applying MDBPE [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 13
Figure 13. Figure 13: Generation over time on quantised CelebA at 32-by-32 [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 16
Figure 16. Figure 16: Impact of the number of tokens used, i.e. how the code￾book collapse barely impacts quality. 322 pixels and CelebA at 1282 pixels, and a VQGAN for CelebA with latent size 162 , K = 2048, and 1282 pix￾els. Note that we always use the same VQ-VAE/VQGAN as baseline for a…
Figure 15
Figure 15. Figure 15: Generated voxel grids from ShapeNet [8] at a reso￾lution of 323 (planes, rifles) and 8 3 (chairs) with our approach. Without compression, attention computation on 32768 voxels for 323 would far exceed the memory of consumer GPUs. F. VQ-VAE Settings for Compression We …
Figure 17
Figure 17. Figure 17: Different numbers of codebook entries K and their impact on the resulting image quality produced by a VQGAN [19] with initially 2048 codebook entries. Note how dropping from 2048 to 128 tokens barely makes a visible difference. 15 [PITH_FULL_IMAGE:figures/full_fig_p0…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Byte Pair Encoding for Efficient Time Series Forecasting

    cs.LG 2025-05 conditional novelty 7.0 of 10

    A byte-pair-encoding tokenizer that converts repeated temporal motifs into single tokens improves zero-shot forecasting accuracy and speed over sample-wise and patch-based methods.

Reference graph

Works this paper leans on

69 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Learning representations and generative models for 3D point clouds

    Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3D point clouds. In Proceedings of the 35th In- ternational Conference on Machine Learning, pages 40–49. PMLR, 2018. 8, 13

    work page 2018

  2. [2]

    Efficient string matching: an aid to bibliographic search

    Alfred V Aho and Margaret J Corasick. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340, 1975. 2

    work page 1975

  3. [3]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. 2, 12

    arXiv 2014

  4. [4]

    Neural machine translation by jointly learning to align and translate, 2016

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016. 2

    work page 2016

  5. [5]

    Mip-nerf: A multiscale representation for anti-aliasing neu- ral radiance fields

    Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neu- ral radiance fields. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 5855–5864,

  6. [6]

    Viewpoint textual inversion: Discovering scene representa- tions and 3d view control in 2d diffusion models, 2024

    James Burgess, Kuan-Chieh Wang, and Serena Yeung-Levy. Viewpoint textual inversion: Discovering scene representa- tions and 3d view control in 2d diffusion models, 2024. 5

    work page 2024

  7. [7]

    Efficient-vqgan: To- wards high-resolution image generation with efficient vision transformers

    Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang. Efficient-vqgan: To- wards high-resolution image generation with efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7368–7377, 2023. 2

    2023

  8. [8]

    Chang, Thomas A

    Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR, abs/1512.03012, 2015. 7, 8, 14

    arXiv 2015

Show all 69 references
  1. [9]

    PixelSNAIL: An improved autoregressive genera- tive model

    XI Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixelSNAIL: An improved autoregressive genera- tive model. In Proceedings of the 35th International Con- ference on Machine Learning, pages 864–872. PMLR, 2018. 8

    2018

  2. [10]

    Up- gpt: Universal diffusion model for person image generation, editing and pose transfer, 2023

    Soon Yau Cheong, Armin Mustafa, and Andrew Gilbert. Up- gpt: Universal diffusion model for person image generation, editing and pose transfer, 2023. 1

    2023

  3. [11]

    Emu: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 1

    2023 arXiv

  4. [12]

    The road less scheduled, 2024

    Aaron Defazio, Xingyu, Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled, 2024. 12

    2024

  5. [13]

    T-free: Tokenizer- free generative llms via sparse representations for memory- efficient embeddings

    Bj ¨orn Deiseroth, Manuel Brack, Patrick Schramowski, Kris- tian Kersting, and Samuel Weinbach. T-free: Tokenizer- free generative llms via sparse representations for memory- efficient embeddings. arXiv preprint arXiv:2406.19223 ,

    arXiv

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 6, 7, 12, 14

    2009

  7. [15]

    The mnist database of handwritten digit images for machine learning research

    Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Maga- zine, 29(6):141–142, 2012. 6, 7, 13

    2012

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2010 arXiv

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  10. [18]

    Quantised global autoencoder: A holistic approach to representing visual data

    Tim Elsner, Paula Usinger, Victor Czech, Gregor Kobsik, Yanjiang He, Isaak Lim, and Leif Kobbelt. Quantised global autoencoder: A holistic approach to representing visual data. arXiv preprint arXiv:2407.11913, 2024. 2, 3, 6

    2024 arXiv

  11. [19]

    Tam- ing transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Tam- ing transformers for high-resolution image synthesis. CoRR, abs/2012.09841, 2020. 2, 5, 6, 7, 8, 12, 14, 15

    2012 arXiv

  12. [20]

    Ccedit: Creative and controllable video editing via diffu- sion models

    Ruoyu Feng, Wenming Weng, Yanhui Wang, Yuhui Yuan, Jianmin Bao, Chong Luo, Zhibo Chen, and Baining Guo. Ccedit: Creative and controllable video editing via diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6712– 67...

    2024

  13. [21]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2014. 3

    2014

  14. [22]

    Vec- tor quantized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vec- tor quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 10696–10706, 2022. 2

    2022

  15. [23]

    Gans trained by a two time-scale update rule converge to a nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, G ¨unter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR, abs/1706.08500, 2017. 5, 12

    2017 arXiv

  16. [24]

    Reducing the dimensionality of data with neural networks

    Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006. 2

    2006

  17. [25]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In Advances in Neural Infor- mation Processing Systems, pages 6840–6851. Curran Asso- ciates, Inc., 2020. 3

    2020

  18. [26]

    Towards accurate image coding: Improved au- toregressive image generation with dynamic vector quantiza- tion (supplementary material)

    Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yong- dong Zhang. Towards accurate image coding: Improved au- toregressive image generation with dynamic vector quantiza- tion (supplementary material). 3 9

  19. [27]

    Octree trans- former: Autoregressive 3d shape generation on hierarchi- cally structured sequences

    Moritz Ibing, Gregor Kobsik, and Leif Kobbelt. Octree trans- former: Autoregressive 3d shape generation on hierarchi- cally structured sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2698–2707, 2023. 3, 7, 8

    2023

  20. [28]

    Edibert, a generative model for image editing

    Thibaut Issenhuth, Ugo Tanielian, J ´er´emie Mary, and David Picard. Edibert, a generative model for image editing. arXiv preprint arXiv:2111.15264, 2021. 1

    2021 arXiv

  21. [29]

    One-shot learning meets depth diffusion in multi-object videos, 2024

    Anisha Jain. One-shot learning meets depth diffusion in multi-object videos, 2024. 5

    2024

  22. [30]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  23. [31]

    Lookupvit: Compressing visual information to a limited number of tokens

    Rajat Koner, Gagan Jain, Prateek Jain, V olker Tresp, and Su- joy Paul. Lookupvit: Compressing visual information to a limited number of tokens. arXiv preprint arXiv:2407.12753,

    arXiv

  24. [32]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 5, 6, 8, 13

    2009

  25. [33]

    Sentencepiece: A sim- ple and language independent subword tokenizer and detok- enizer for neural text processing, 2018

    Taku Kudo and John Richardson. Sentencepiece: A sim- ple and language independent subword tokenizer and detok- enizer for neural text processing, 2018. 2

    2018

  26. [34]

    Openassistant conversa- tions – democratizing large language model alignment, 2023

    Andreas K ¨opf, Yannic Kilcher, Dimitri von R ¨utte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Rich ´ard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen,...

    2023

  27. [35]

    Robust training of vector quantized bottleneck models

    Adrian Ła ´ncucki, Jan Chorowski, Guillaume Sanchez, Ri- card Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khu- rana, Tanel Alum¨ae, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN) ,...

    2020

  28. [36]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022. 2, 5

    2022

  29. [37]

    Flow matching for generative mod- eling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. arXiv preprint arXiv:2210.02747, 2022. 3

    2022 arXiv

  30. [38]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015. 5, 6, 7, 8, 12, 13, 14

    2015

  31. [39]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 12

    2019

  32. [40]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learn- ing, page 4. Granada, 2011. 6, 7, 12

    2011

  33. [41]

    Gpt-4 technical report,

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. Gpt-4 technical report,

  34. [42]

    Tokenisation and sentence segmentation

    David D Palmer. Tokenisation and sentence segmentation. Handbook of natural language processing , pages 11–35,

  35. [43]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  36. [44]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660,

  37. [45]

    Improving language understanding with unsuper- vised learning

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding with unsuper- vised learning. 2018. 5

    2018

  38. [46]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 5

    2019

  39. [47]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092,

    arXiv

  40. [48]

    A novel sampling scheme for text-and image-conditional im- age synthesis in quantized latent spaces

    Dominic Rampas, Pablo Pernias, and Marc Aubreville. A novel sampling scheme for text-and image-conditional im- age synthesis in quantized latent spaces. arXiv preprint arXiv:2211.07292, 2022. 2

    2022 arXiv

  41. [49]

    Gener- ating diverse high-fidelity images with vq-vae-2, 2019

    Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Gener- ating diverse high-fidelity images with vq-vae-2, 2019. 2

    2019

  42. [50]

    Pixel-level BPE for auto-regressive image generation

    Anton Razzhigaev, Anton V oronov, Andrey Kaznacheev, Andrey Kuznetsov, Denis Dimitrov, and Alexander Panchenko. Pixel-level BPE for auto-regressive image generation. In Proceedings of the First Workshop on Performance and Interpretability Evaluations of Multi- modal, Multipurp...

    2022

  43. [51]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.04597, 2015. 2

    2015 arXiv

  44. [52]

    Learning internal representations by error prop- agation, parallel distributed processing, explorations in the microstructure of cognition, ed

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error prop- agation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclel- land. vol. 1. 1986. Biometrika, 71(599-607):...

    1986

  45. [53]

    Learning factorial codes by pre- dictability minimization

    J ¨urgen Schmidhuber. Learning factorial codes by pre- dictability minimization. Neural Computation , 4(6):863– 879, 1992. 3

    1992

  46. [54]

    Neural machine translation of rare words with subword units.CoRR, abs/1508.07909, 2015

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units.CoRR, abs/1508.07909, 2015. 2, 3

    2015 arXiv

  47. [55]

    Bivdiff: A training-free framework for general-purpose video synthesis via bridging image and video diffusion models, 2024

    Fengyuan Shi, Jiaxi Gu, Hang Xu, Songcen Xu, Wei Zhang, and Limin Wang. Bivdiff: A training-free framework for general-purpose video synthesis via bridging image and video diffusion models, 2024. 1 10

    2024

  48. [56]

    Byte pair encoding: A text compression scheme that accelerates pattern matching

    Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. Byte pair encoding: A text compression scheme that accelerates pattern matching. 1999. 2, 3

    1999

  49. [57]

    Linear-time wordpiece tokenization

    Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. Linear-time wordpiece tokenization. CoRR, abs/2012.15524, 2020. 2

    2012 arXiv

  50. [58]

    Discrete acoustic space for an efficient sampling in neural text-to-speech

    Marek Strong, Jonas Rohnke, Antonio Bonafonte, Mateusz Łajszczak, and Trevor Wood. Discrete acoustic space for an efficient sampling in neural text-to-speech. arXiv preprint arXiv:2110.12539, 2021. 2, 6

    2021 arXiv

  51. [59]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021. 8

    2021 arXiv

  52. [60]

    Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. 13, 14

    2014 arXiv

  53. [61]

    Neural discrete representation learning

    A ¨aron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. CoRR, abs/1711.00937, 2017. 2, 5, 6, 12, 14

    2017 arXiv

  54. [62]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. 2

    2023

  55. [63]

    Vector quantized wasserstein auto-encoder, 2023

    Tung-Long Vuong, Trung Le, He Zhao, Chuanxia Zheng, Mehrtash Harandi, Jianfei Cai, and Dinh Phung. Vector quantized wasserstein auto-encoder, 2023. 2, 6

    2023

  56. [64]

    Mambabyte: Token-free selective state space model

    Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint arXiv:2401.13660, 2024. 2

    2024 arXiv

  57. [65]

    Vector quantization-based regularization for autoencoders

    Hanwei Wu and Markus Flierl. Vector quantization-based regularization for autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 6380– 6387, 2020. 2, 6

    2020

  58. [66]

    Videogpt: Video generation using vq-vae and trans- formers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 2, 5

    2021 arXiv

  59. [67]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406.07550, 2024. 3

    2024 arXiv

  60. [68]

    Online clustered code- book

    Chuanxia Zheng and Andrea Vedaldi. Online clustered code- book. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 22798–22807, 2023. 2, 6

    2023

  61. [69]

    Designing a better asymmetric vqgan for stablediffusion

    Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, and Gang Hua. Designing a better asymmetric vqgan for stablediffusion. arXiv preprint arXiv:2306.04632, 2023. 2 11 Appendix A. Computational Complexity The complexity of Byte Pair Encoding is de...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.