Pith. sign in

REVIEW 4 major objections 6 minor 34 references

MambaVideo for Discrete Video Tokenization with Channel-Split Quantization

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A video tokenizer built from Mamba layers and channel-split quantization outperforms prior discrete tokenizers on reconstruction and autoregressive generation benchmarks.

desk verdict Solid tokenizer paper with a genuinely simple channel-split idea; the capacity proof has an off-by-one error and baseline fairness needs scrutiny, but the empirical case is strong enough to referee. read the letter →

arxiv 2507.04559 v1 pith:4LAOCZZ7 submitted 2025-07-06 cs.CV

Dawit Mureja Argaw , Xian Liu , Joon Son Chung , Ming-Yu Liu , Fitsum Reda This is my paper
classification cs.CV
keywords discretevideotokenizationMambastatespacemodelsfinitescalarquantizationchannel-splitautoregressivegenerationreconstructionlook-upfree
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Discrete video tokenization is the step that turns raw video into the token sequences autoregressive generators consume, and this paper argues that both halves of that step can be improved at once. Architecturally, it proposes replacing Transformer attention with Mamba layers inside a hierarchical encoder-decoder that downsamples and upsamples through cascaded 3D-convolution patchify and topixel modules with residual skip connections. On the quantization side, it introduces channel-split quantization: the encoded latent is split along the channel dimension into $K$ groups, each group is quantized separately with finite-scalar or look-up-free quantization, and the $K$ tokens per pixel are treated as an ordered sequence, so the effective codebook grows beyond $2^{NK}$ while the total number of tokens stays fixed. The authors report that the resulting tokenizer beats both causal 3D-convolution and Transformer baselines on reconstruction, and that it lowers FVD when plugged into an autoregressive video generator. If correct, this gives video generation a tokenizer that is more expressive per token without lengthening the sequence the generative model must handle.

What carries the argument

The load-bearing object is channel-split quantization. Given a base quantizer such as FSQ that would use $c$ channels with codebook size $2^N$, the encoder is asked to produce a latent with $c \cdot K$ channels; the latent is split along the channel dimension into $K$ groups, each group is quantized independently, and the quantized groups are concatenated for the decoder. Each spatial-temporal location is therefore represented by an ordered sequence of $K$ tokens rather than one token, and the paper proves this behaves like a single codebook of size greater than $2^{NK}$, far above the $2^N$ of the base quantizer. To keep the total token count identical to plain FSQ/LFQ, the encoder's spatio-temporal compression rate is increased by a factor of $K$. The supporting architecture is a hierarchical Mamba encoder-decoder: cascaded patchify and topixel modules use 3D convolutions, encoder blocks are linked by token pooling residual connections and decoder blocks by token interpolation, and spatial and temporal attention are Mamba layers rather than Transformers, so no positional encoding is needed and sequence-length cost stays linear.

What would settle it

Re-run the Table 1 comparison with several random seeds per method and check whether the 2.81 dB average PSNR margin of the best configuration over the causal 3D-convolution baseline exceeds the seed-to-seed spread; if the margins fall inside training noise, the state-of-the-art claim would not survive. A cheaper check is to train the Mamba tokenizer with plain FSQ and with CS-FSQ at the same token count and see whether the 1.41 dB gain is reproducible under matched compute.

Watch

Extended reading notes

Core claim

The central claim is that a discrete video tokenizer built from hierarchical Mamba encoding-decoding and channel-split quantization reconstructs video better than established discrete tokenizers at the same token budget. In the paper's experiments, the best configuration, the Mamba tokenizer with CS-FSQ at $8\times8\times8$ compression, two splits, and codebook size $2^{16}$, averages 32.45 dB PSNR on Xiph-2K and DAVIS, which is 2.81 dB above Magvit-v2 and 4.1 dB above CViViT, despite applying twice the temporal compression. The same tokenizer, when used as the front end of VideoGPT for unconditional generation, achieves FVD of 55.4 on SkyTimelapse and 266.2 on UCF-101, beating all baselines tested. The paper also claims the channel-split mechanism itself is responsible for much of the gain: CS-FSQ improves over plain FSQ by 1.41 dB on the Mamba tokenizer, and CS-LFQ improves over LFQ by 1.13 dB on Magvit-v2, with the same number of tokens.

Load-bearing premise

The load-bearing premise is that the baseline tokenizers were trained to comparable strength: the paper reports single runs in which every model was trained under the same settings following official implementations, so if the causal 3D-convolution or Transformer baselines were not pushed to similar quality, the reported margins could shrink or disappear.

Editorial extensions

If this is right

  • The tokenizer keeps the sequence length the generative model must handle unchanged while giving each token position a $K$-token representation, so autoregressive generators get more expressiveness per position at no added sequence cost.
  • It confirms that non-learnable quantization (FSQ/LFQ) can beat codebook-based VQ when paired with a strong encoder-decoder, which points toward simpler training without codebook losses or codebook search.
  • Because Mamba attention does not rely on positional embeddings, the tokenizer is designed to tokenize resolutions not seen in training, a limitation the paper attributes to Transformer-based tokenizers.
  • When the same pretrained tokenizer is dropped into VideoGPT, the lower FVD numbers indicate the reconstruction gains carry over to actual generated video, not just to reconstruction metrics.
  • The paper's own analysis shows the benefit of additional splits plateaus at very high compression rates, since the compression side of the trade-off eventually dominates the quantization capacity gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's effective-codebook argument is a capacity bound, not a usage guarantee; an immediate test is measuring how many of the possible combinations actually occur in a trained model, since underutilized combinations would make the practical gain smaller than the bound.
  • The same channel-split trade-off should transfer to image and audio tokenizers, since it only assumes a latent with a channel dimension and a spatial or temporal grid to compress; the paper's plateau at $K=4$ suggests the benefit will shrink in settings where compression dominates quality.
  • A natural follow-up the paper does not run is a residual scheme across channel splits, quantizing one split and using its residual to inform the next split, which could combine the expressiveness of channel-split quantization with the signal refinement of residual quantization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MambaVideo, a discrete video tokenizer with a hierarchical Mamba-based encoder-decoder, and channel-split quantization (CS-LFQ/CS-FSQ). The encoder downsamples through cascaded 3D-convolution patchify and Mamba attention blocks, with token-pooling skip connections; the decoder mirrors this with topixel upsampling and interpolation. Channel-split quantization divides the latent channels into K groups, quantizes each group independently, and compensates by increasing spatio-temporal compression so the total token count is unchanged. The authors report state-of-the-art reconstruction results on Xiph-2K and DAVIS, a downstream gain when the tokenizer is plugged into VideoGPT for generation on SkyTimelapse and UCF-101, and ablations for hierarchy, attention type, residual connections, and patchify/topixel modules.

Significance. The two proposed ingredients are simple and plausibly useful: Mamba layers avoid positional-embedding extrapolation issues in sequence-based video tokenizers, and channel-split quantization is an architecture-agnostic way to reshape the latent representation while fixing token count. The paper contains extensive ablations for each design choice and validates the tokenizer in a real autoregressive-generation pipeline, which is a strength. However, the central SOTA claim depends on baseline training fairness and single-run numbers, and the Sec. 3.3 capacity proof contains both an arithmetic error and a conceptual conflation of per-pixel and per-sequence capacity. With a corrected proof and a fully specified baseline protocol, the contribution would be solid; as written, the theoretical motivation and the headline comparison need revision.

major comments (4)
  1. [§3.3] The proof is not correct as written. Since each qi is an integer between 0 and 2^N - 1, the maximum of f is (2^N - 1)(1 + 2^N + ... + 2^{N(K-1)}) = 2^{NK} - 1, so the effective per-pixel codebook size is exactly 2^{NK}, not greater than 2^{NK}. More importantly, the per-pixel comparison conflates levels: after increasing the spatio-temporal compression by K, the number of spatial positions is divided by K, so the total number of possible token sequences is (2^N)^P in both naive FSQ/LFQ and CS-FSQ/CS-LFQ. The claim that channel-split quantization 'effectively increases representation capacity' is therefore not supported by the counting argument; the empirical gains in Tables 1 and 2 should be attributed to a different latent geometry or decoder inductive bias rather than to a larger effective codebook.
  2. [§4.1] The statement 'All models are trained under the same settings as ours, strictly adhering to their official implementations' is internally inconsistent: if the baseline models use the authors' settings, they are not strictly following official implementations; if they follow official implementations, the settings are not the same. No baseline hyperparameters (loss coefficients, GAN activation step, resolution, number of iterations) are reported. Since the headline claim is the 2.81 dB improvement over Magvit-v2 in Table 1, the paper must report the exact training recipe for each baseline or use official checkpoints; otherwise the SOTA claim is not verifiable.
  3. [Tables 1, 2, 4, 5] All reconstruction and FVD numbers are single-run. Given that several reported margins are small relative to typical training noise (e.g., the Transformer variants in Table 3b span roughly 0.3-0.5 dB), the comparisons may not be stable. The authors should report at least two runs or error bars for the main quantitative claims, especially the headline SOTA margins.
  4. [Table 5 vs. Table 1] The same named configuration 'Ours + CS-FSQ' at an 8×8×8 compression rate reports PSNR 32.54/32.36 in Table 1 but 34.47/34.34 in Table 5, with no explanation. If the two tables use different numbers of splits K, this must be stated explicitly; as written, the discrepancy prevents the reader from reproducing the appendix comparison and casts doubt on the consistency of the experimental reporting.
minor comments (6)
  1. [Table 1] The codebook-size column renders inconsistently (e.g., '2 16' and '216'); use a uniform superscript notation such as 2^16 throughout.
  2. [Figure 1] The label '(c)' is used for both the encoder and decoder diagrams; relabel one of them to avoid confusion.
  3. [Table 3] The header 'ixel layer' should be 'Pixel layer'.
  4. [§2.2] The decoder indexing is unclear: l=1 is said to denote the last block, but the decoder level indices are not defined separately from the encoder levels, making the token-interpolation description hard to follow.
  5. [§4.2] There is a typesetting issue in 'Fr´echet Video Distance' where the accented character renders incorrectly.
  6. [§2.1] Please use consistent terminology for the patchify embedding: the text says '3D convolution-based embedding layer' in some places and 'patchify module' in others.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the capacity claim is a self-contained combinatorial count and the SOTA claim is benchmarked against external baselines; the §3.3 arithmetic slip is a correctness issue, not circularity.

full rationale

The paper's central derivation chain is not circular. CS-FSQ's capacity gain is established in Sec. 3.3 by counting the joint states of K independently quantized splits; this is a definitional combinatorial fact rather than a fitted or back-fed result, and the empirical comparisons in Tables 1-5 are measured against external baselines (VideoGPT, CViViT, OmniTokenizer, Magvit-v2) with no self-citation used as load-bearing evidence. The 'maintaining token count' property is obtained by construction (scaling compression by K) and is explicitly stated as such, not disguised as a prediction. No fitted parameter is renamed as a prediction: hyperparameters (entropy/commitment coefficients 0.1/0.25, GAN activation at 200K, FSQ levels [8,8,8,5,5,5]) are taken from prior work and applied uniformly. The only notable defect is in Sec. 3.3's proof, where the maximum per-split token id is written as 2^N whereas it is 2^N−1, so the effective codebook size is exactly 2^NK rather than >2^NK; this is a correctness/arithmetic slip that does not invalidate the qualitative claim and is not a circularity. The baseline-fairness statement in Sec. 4.1 ('All models are trained under the same settings as ours, strictly adhering to their official implementations') is ambiguous and single-run results carry noise, but that is an experimental-validity risk, not a circular derivation. Under the hard rules, no specific Eq.-to-Eq. reduction or fitted-input-renamed-as-prediction can be exhibited, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim relies mainly on standard hyperparameters (FSQ levels, K, loss weights) and assumptions about benchmark transfer and baseline fairness. No free parameters are fitted to the target metrics; the theoretical capacity argument is independent of the experimental fits. The one arithmetic error in the proof weakens but does not destroy the qualitative claim.

free parameters (3)
  • FSQ levels L = [8, 8, 8, 5, 5, 5]
    Hand-chosen in Sec. 4.1 for all FSQ-based quantizers; defines the codebook of size 2^16 and affects reconstruction performance.
  • Number of channel splits K = 2 (main results; 1 and 4 in ablations)
    Hyperparameter of channel-split quantization (Sec. 3.2, Sec. 4.1); trades spatial-temporal compression for per-token capacity.
  • LFQ loss weights = entropy 0.1, commitment 0.25
    Hand-chosen in Sec. 4.1 for LFQ-based tokenizers, following prior work [29]; affects training stability and codebook usage.
assumptions (5)
  • ad hoc to paper A codebook of size 2^N has maximum token ID 2^N
    Used in the Sec. 3.3 proof to derive an effective codebook size '> 2^{NK}'; strictly the maximum is 2^N - 1, making the stated inequality incorrect by one.
  • domain assumption Reconstruction metrics on Xiph-2K and DAVIS are a valid proxy for tokenizer utility
    The paper evaluates tokenizers mainly via PSNR/SSIM/LPIPS on two datasets (Sec. 4.1) and assumes improvements transfer to generation; it partially validates via Sec. 4.2.
  • domain assumption Baselines trained under the same protocol faithfully represent each method
    Sec. 4.1 states all models are trained under the same settings as ours, strictly adhering to official implementations; the fairness of this comparison is unverified.
  • domain assumption Mamba layers, lacking positional embeddings, generalize to unseen resolutions
    Stated in Sec. 2.1 as a motivation, but no experiment evaluates tokenization at resolutions unseen during training.
  • domain assumption The 'best non-overlapping' mapping f from K tokens to one ID measures joint capacity
    Sec. 3.3 assumes this mapping is a valid proxy for representational power; it is a modeling choice that supports the theoretical claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MambaVideo for Discrete Video Tokenization with Channel-Split Quantization." pith.science (2026). https://pith.science/paper/4LAOCZZ7

@misc{pith2026250704559,
  author       = {Pith},
  title        = {Pith review of: MambaVideo for Discrete Video Tokenization with Channel-Split Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4LAOCZZ7}},
  note         = {Machine review of arXiv:2507.04559}
}
read the original abstract

Discrete video tokenization is essential for efficient autoregressive generative modeling due to the high dimensionality of video data. This work introduces a state-of-the-art discrete video tokenizer with two key contributions. First, we propose a novel Mamba-based encoder-decoder architecture that overcomes the limitations of previous sequencebased tokenizers. Second, we introduce a new quantization scheme, channel-split quantization, which significantly enhances the representational power of quantized latents while preserving the token count. Our model sets a new state-of-the-art, outperforming both causal 3D convolutionbased and Transformer-based approaches across multiple datasets. Experimental results further demonstrate its robustness as a tokenizer for autoregressive video generation.

Figures

Figures reproduced from arXiv: 2507.04559 by the authors.

Figure 1
Figure 1. Architecture Overview: (a) The encoder network for CViViT [25], a state-of-the-art Transformer-based tokenizer. (b) The encoder network for Magvit-v2 [29], a state-of-the-art causal 3D convolution-based tokenizer (c) The encoder and decoder architecture of the proposed Mamba-based tokenizer. Each model is designed with an 8 × 8 × 8 spatio-temporal compression rate. Spatial and Temporal Attention The output tokens of… view at source ↗
Figure 2
Figure 2. Qualitative analysis of our tokenizer compared with the best-performing baselines on the video reconstruction task [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Qualitative analysis of videos generated by VideoGPT, enabled by our tokenizer, i.e. Ours (CS-FSQ) + VideoGPT. eling arising from VideoGPT’s lower spatio-temporal com￾pression rate. In contrast, both Magvit-v2 and our tokenizer operate on sequences that are 4× shorter, resulting in sig￾nificantly improved performance. Notably, VideoGPT en￾abled by our Mamba-based tokenizer, i.e. Ours (CS-FSQ) + VideoGPT, achieves th… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Quantitative analysis of ablation experiments on the video reconstruction task. Token Pooling/Interpolation We investigate the benefits of adding residual connections within the encoder blocks (using token pooling) and decoder blocks (using token inter￾polation) as dis…
Figure 5
Figure 5. Figure 5: Qualitative analysis of residual and channel-split with Magvit-v2 and our proposed tokenizer [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 13 canonical work pages

  1. [1]

    S-HR-VQVAE: Sequential Hierarchical Residual Learning Vector Quantized Variational Autoencoder for Video Prediction

    Mohammad Adiban, Kalin Stefanov, Sabato Marco Sinis- calchi, and Giampiero Salvi. S-hr-vqvae: Sequential hierar- chical residual learning vector quantized variational autoen- coder for video prediction.arXiv preprint arXiv:2307.06701,

    work page Pith review arXiv

  2. [2]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021. 5

    2021

  3. [3]

    Transformers are ssms: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: General- ized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 1, 3, 7

    arXiv 2024

  4. [4]

    High fidelity neural audio compression

    Alexandre D ´efossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022. 10

    arXiv 2022

  5. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1

    arXiv 2010

  6. [6]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 1, 2

    work page 2021

  7. [7]

    Vector quantization

    Robert Gray. Vector quantization. IEEE Assp Magazine, 1 (2):4–29, 1984. 1, 4, 5

    work page 1984

  8. [8]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 1, 3

    arXiv 2023

Show all 34 references
  1. [9]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023. 1

    2023 arXiv

  2. [10]

    Towards accurate image coding: Improved au- toregressive image generation with dynamic vector quantiza- tion

    Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yong- dong Zhang. Towards accurate image coding: Improved au- toregressive image generation with dynamic vector quantiza- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 22596–22605,

  3. [11]

    Image-to-image translation with conditional adver- sarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134,

  4. [12]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  5. [13]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022. 10

    2022

  6. [14]

    Finite scalar quantization: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023. 2, 4, 5, 8

    2023 arXiv

  7. [15]

    Softmax splatting for video frame interpolation

    Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. InIEEE Conference on Computer Vision and Pattern Recognition, 2020. 2, 5, 7

    2020

  8. [16]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 2, 5, 7

    2017 arXiv

  9. [17]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. 3, 7

    2021 arXiv

  10. [18]

    Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elho- seiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3626–3636, 2022. 6

    2022

  11. [19]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  12. [20]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  13. [21]

    Visual autoregressive modeling: Scalable im- age generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable im- age generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024. 10

    2024 arXiv

  14. [22]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  15. [23]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 1, 2

    2017

  16. [24]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 1, 3, 7

    2017

  17. [25]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  18. [26]

    Omnitokenizer: A joint image-video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zux- uan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.09399, 2024. 1, 2, 3, 5

    2024 arXiv

  19. [27]

    Videogpt: Video generation using vq-vae and trans- formers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 1, 2, 5, 6

    2021 arXiv

  20. [28]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021. 1, 5

    2021 arXiv

  21. [29]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 1, 2, 3, 4, 5...

    arXiv

  22. [30]

    Video probabilistic diffusion models in projected latent space, 2023

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space, 2023. 6

    2023

  23. [31]

    Dtvnet: Dynamic time-lapse video generation via single still image

    Jiangning Zhang, Chao Xu, Liang Liu, Mengmeng Wang, Xia Wu, Yong Liu, and Yunliang Jiang. Dtvnet: Dynamic time-lapse video generation via single still image. In Eu- ropean Conference on Computer Vision , pages 300–315. Springer, 2020. 2, 6

    2020

  24. [32]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 5

    2018

  25. [33]

    Image and video tokenization with binary spherical quantization

    Yue Zhao, Yuanjun Xiong, and Philipp Kr ¨ahenb¨uhl. Image and video tokenization with binary spherical quantization. arXiv preprint arXiv:2406.07548, 2024. 2, 3

    2024 arXiv

  26. [34]

    Appendix Here, we present additional experimental analysis that com- plements our findings in the main paper. Table 5. Experimental comparison between channel-split and residual quantization Method Xiph-2K DA VIS PSNR↑ LPIPS↓ PSNR↑ LPIPS↓ Magvit-v2 + Res. FSQ 30.60 0.187 30.08...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.