Pith. sign in

REVIEW 3 major objections 3 minor 72 references

Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction

T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A video tokenizer that learns to reconstruct patches from sampled space-time coordinates can encode a 128-frame video in 1,280 tokens, where prior tokenizers need 6,144–8,192 tokens for comparable quality.

desk verdict CoordTok is a genuinely useful tokenizer design, but the abstract's 'similar reconstruction quality' claim is contradicted by its own rFVD numbers. read the letter →

arxiv 2411.14762 v4 pith:SLISYALV submitted 2024-11-22 cs.CV cs.AIcs.LG

Huiwon Jang , Sihyun Yu , Jinwoo Shin , Pieter Abbeel , Younggyo Seo This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords videotokenizationtriplanerepresentationcoordinate-basedreconstructionlonggenerationlatentdiffusiontransformertemporalcoherenceUCF-101tokenefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoordTok is a video tokenizer built on the claim that training directly on long video clips unlocks temporal coherence, letting one 128-frame, 128x128 video be encoded into 1,280 tokens while earlier tokenizers need 6,144 or 8,192 tokens for comparable reconstruction. The design that makes long-clip training affordable is a decoder that reconstructs only patches at randomly sampled $(x,y,t)$ coordinates from factorized triplane latents, instead of reconstructing every frame at once. If the claim holds, video tokenization stops being the bottleneck that forces generative models to work in short clips, and long-video generation can be done in a single pass.

What carries the argument

The load-bearing mechanism is coordinate-based patch reconstruction over factorized triplane latents. The encoder maps a video to three 2D planes (content plane $z_{xy}$ plus two motion planes $z_{yt}$, $z_{xt}$) rather than a 3D latent; the decoder samples $N$ patch coordinates $(i,j,k)$, reads each coordinate from the planes via bilinear interpolation, lets the sampled features attend to each other through self-attention, and projects them to the RGB patches. Because only 3% of patches are reconstructed per training step, training cost no longer grows linearly with video length, which is what makes 128-frame training tractable.

What would settle it

Train a strong baseline such as PVDM-AE or TATS-AE from scratch on 128-frame clips, with compute comparable to CoordTok's, and evaluate reconstructed 128-frame videos; if the 6,144–8,192-token gap shrinks or disappears, the efficiency claim comes from long-video training rather than from the coordinate-based decoder. Alternatively, train CoordTok on 16-frame clips and evaluate on 128-frame videos: if it then needs as many tokens as the baselines, the paper's central attribution to temporal coherence fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a video tokenizer trained end-to-end on 128-frame clips can exploit temporal coherence to encode long videos far more compactly than tokenizers trained on 16-frame clips and stitched together. CoordTok encodes an input video into three 2D latent planes, $z_{xy}$, $z_{yt}$, $z_{xt}$, and trains a transformer decoder to map sampled space-time coordinates to the pixels of the corresponding patches using bilinear interpolation of the planes. On UCF-101, 1,280 tokens reconstruct a 128-frame video at rFVD 102.9, a quality PVDM-AE cannot reach with 1,152 tokens (rFVD > 1,000) and that other baselines reach only with 6,144–8,192 tokens. A flow-based diffusion transformer trained on these 1,280-token latents generates 128 frames at once with FVD 369.3, outperforming prior baselines, and does so faster than iterative approaches.

Load-bearing premise

The comparison assumes that training CoordTok from scratch on 128-frame clips against baselines trained on 16-frame clips or released weights, evaluated by stitching independent reconstructions, is a fair measure of token efficiency: if the baselines are under-trained or unfairly penalized by stitching, the token-count advantage is not established.

Editorial extensions

If this is right

  • Video tokenizers can be trained on clips of 128 frames or longer without memory and FLOP costs growing linearly with clip length, because only sampled coordinates are reconstructed.
  • Long videos can be represented with roughly a fifth to a sixth of the tokens earlier tokenizers need at comparable quality, shrinking the sequence length seen by generative models.
  • A diffusion transformer can generate 128 frames in one pass rather than iteratively, with lower synthesis time and competitive memory.
  • Because token count and reconstruction quality are only loosely coupled above a threshold, generative quality can improve even when reconstruction quality is unchanged.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's logic is that training on 256- or 512-frame clips should keep training cost nearly flat while further exploiting temporal redundancy, since only sampled coordinates are decoded; this is the authors' stated scaling ambition but not demonstrated here.
  • The strong correlation between reconstruction error and video dynamics (-0.87) suggests the triplane content/motion split is the bottleneck for fast motion, so an adaptive scheme that allocates more temporal planes to dynamic clips could recover quality where CoordTok currently struggles.
  • The FVD gain from 1,280 over 3,072 tokens, despite near-equal reconstruction quality, implies that generative-model gains come partly from sequence-length reduction; this predicts similar gains for other latent diffusion transformers trained on CoordTok-style latents.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces CoordTok, a video tokenizer that encodes a video into factorized triplane representations and reconstructs only patches at randomly sampled (x,y,t) coordinates during training, allowing direct training on 128-frame clips at reduced memory cost. On UCF-101, CoordTok encodes a 128-frame 128x128 video into 1280 tokens and reports strong reconstruction metrics, and a downstream diffusion transformer (CoordTok-SiT-L/2) generates 128 frames at once with FVD 369.3, outperforming published baselines. The central claim is that prior tokenizers need 6144 or 8192 tokens to reach similar reconstruction quality, demonstrating the benefit of long-video training.

Significance. The method is a clean and plausible contribution to video tokenization: it adapts coordinate-based decoding from 3D generation to video tokenizer training, and the reported GPU-memory and FLOPs savings are concrete. The generation results (FVD 369.3 on 128-frame UCF-101) are competitive and the illustration of temporal-consistency advantages over stitched baselines is suggestive. However, the headline token-efficiency claim is currently supported only on some metrics and is confounded by the comparison protocol, so the significance is diminished until those issues are addressed.

major comments (3)
  1. [Abstract and Section 3.2, Table 1, Figure 4] The claim that 'baselines need 6144 or 8192 tokens to achieve similar reconstruction quality' is not supported on the paper's own rFVD metric. In Table 1, PVDM-AE with 6144 tokens achieves rFVD 66.5 and OmniTokenizer-CV with 8192 tokens achieves rFVD 49.5, both substantially better (lower) than CoordTok's 102.9 at 1280 tokens. The paper does not define 'similar reconstruction quality,' and Figure 4 plots CoordTok against baseline curves at smaller token counts, which visually obscures the crossover that Table 1 shows on rFVD. Because token efficiency is the headline contribution, this overstatement is load-bearing. The abstract should be revised to qualify which metrics support the claim (e.g., PSNR/LPIPS) or to compare at matched token budgets.
  2. [Section 3.2, experimental setup] The baseline comparison confounds model architecture with training clip length. Baselines are trained on 16-frame clips and evaluated by stitching independent reconstructions, while CoordTok is trained on 128-frame clips. The paper states that this demonstrates the benefit of long-video training, but no experiment holds training length fixed: there is no CoordTok variant trained on 16-frame clips, nor a 3D VAE baseline trained on 128-frame clips. Without such a control, the observed token efficiency could be attributed to the longer training context rather than to the coordinate-based decoder design. Please add a controlled comparison or substantially temper the causal claim.
  3. [Table 1 and Section 3.2, 'similar reconstruction quality'] The multi-metric results disagree in direction, so the phrase 'similar reconstruction quality' is too strong without designating a primary metric. On PSNR CoordTok (28.6) slightly exceeds OmniTokenizer-CV (28.3); on LPIPS it is better (0.066 vs 0.081); on SSIM it is worse (0.892 vs 0.913); and on rFVD it is much worse (102.9 vs 49.5). The paper should either define a single primary fidelity metric or present a trade-off analysis that makes the metric-dependent conclusion explicit rather than asserting an unqualified similarity.
minor comments (3)
  1. [Section 3.2, Results paragraph] The sentence 'PVDM-AE achieves >1000 rFVD score when using 1152 tokens' selects the most favorable comparison point for CoordTok; the reader should also be reminded that PVDM-AE at 6144 tokens achieves rFVD 66.5, much better than CoordTok's 102.9.
  2. [Figure 4] The x-axis labels (1024, 1280, 2560, 6144, 8192) are visually crowded and the CoordTok point is not labelled with its rFVD value; consider a log-scale x-axis and explicit annotation of the 1280-token comparison.
  3. [Section 2.2 and Table 4] The relationship between the sampling ratio (e.g., 3%) and the number N of sampled coordinates (1024 for main training, 4096 for fine-tuning) is not stated in the main text; please clarify this explicitly.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: token-efficiency and generation claims are measured on held-out UCF-101 clips; same-group citations (PVDM, VideoNVP, ElasticTok) serve as experimental controls or metric sources, not as premises that force the result.

full rationale

CoordTok is an empirical architecture paper, and I could not exhibit any step where a 'prediction' reduces by construction to a fitted parameter or to a self-citation chain. The central token-efficiency claim ('CoordTok can encode a 128-frame video with 128x128 resolution into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar reconstruction quality') is a measured comparison on held-out UCF-101 clips reported in Table 1, evaluated with external metrics (rFVD, PSNR, LPIPS, SSIM); the token counts are architectural choices, not fitted outputs, and the reconstruction numbers are not defined in terms of the claim. The skeptical observation that Table 1 contradicts the abstract on rFVD (CoordTok 1280 tokens: rFVD 102.9, vs. PVDM-AE 6144 tokens: 66.5 and OmniTokenizer-CV 8192 tokens: 49.5) is a genuine internal-consistency and metric-selection concern that a reviewer should raise, but it is not circularity: no equation equates the conclusion with its input. Self-citations exist and are moderate: PVDM [66] and VideoNVP [22] (same-group authors Sihyun Yu and Jinwoo Shin) supply the triplane representation and coordinate-to-RGB formulation, DIGAN [65] appears as a generation baseline, and ElasticTok [60] (co-author Pieter Abbeel) supplies the Sobel frequency metric in Figure 7b. None is load-bearing in the forbidden sense: PVDM-AE is the central control for the decoder-design claim, but it is retrained from scratch on UCF-101 and compared on external metrics; the triplane idea is also attributed to EG3D [4]; the frequency-metric correlation in Figure 7b is computed from data for both CoordTok and baselines rather than imported as an assumed theorem; and no uniqueness argument is invoked to forbid alternatives. The paper also candidly flags its own limitation of degraded quality on dynamic videos (Section 5, Figure 7a), which is consistent with honest reporting. The asymmetric evaluation setup (baselines stitched from 16-frame clips, CoordTok trained on 128 frames) is a fairness risk that the paper discloses in Section 3.2, but fairness of a benchmark is not circularity of a derivation. Verdict: the derivation is self-contained; the score of 1 reflects the presence of several same-group citations that are not load-bearing rather than any reduction of the result to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger reflects the hand-chosen architectural and training choices that are ablated in the paper but not derived from any principle. The main free parameters are the triplane sizes and channel width, the coordinate sampling ratio, and the two-stage loss schedule. The axioms are the dataset and evaluation assumptions that support the broad claim, plus the unvalidated train/eval decoding generalization.

free parameters (4)
  • Triplane plane dimensions and channel width = 16x16 (z_xy), 16x32 (z_yt, z_xt), channel width 8
    Chosen by ablations (Figure 6) to balance token count (1280) and reconstruction quality; directly sets the headline token count. No theoretical derivation is given for these sizes.
  • Training patch sampling ratio = 3.125% (N=1024 of 32,768 patches) for main training; 12.5% (N=4096) for LPIPS fine-tuning
    Selected based on Table 4; the paper reports that sampling only 3% of video patches can achieve strong performance. The ratio is a hand-tuned hyperparameter affecting training cost and the claimed efficiency.
  • Fine-tuning protocol = 50k iterations with equal-weighted l2+LPIPS loss after 1M l2 iterations
    Determined by the authors' experiments (Table 9); improves rFVD from 186.3 to 102.9 while lowering PSNR from 30.3 to 28.6. The chosen balance is a free choice that shapes the reported metrics.
  • Model capacity (CoordTok-L) = Encoder 8 layers, cross-self 24, decoder 24, hidden dim 1024, 16 heads (Table 5)
    Selected after a scaling study (Figure 6a); larger model improves PSNR and rFVD. The final model size is a free choice that affects all results.
assumptions (4)
  • domain assumption UCF-101 is a sufficient testbed for the general claim that CoordTok efficiently tokenizes long videos
    All experiments and ablations are on a single dataset at 128x128 resolution; the paper generalizes in the abstract to long videos without multi-dataset validation.
  • domain assumption The baseline tokenizers are adequately trained and fairly configured
    Section 3.2 states baselines are trained from scratch on UCF-101 or use official weights, but training compute, convergence, and hyperparameter tuning effort are not reported; the comparison assumes parity.
  • domain assumption Evaluation metrics (rFVD, PSNR, LPIPS, SSIM) faithfully measure reconstruction quality for the token-efficiency comparison
    The paper uses rFVD as the headline efficiency metric in Figure 4, but Table 1 shows high-token baselines beat CoordTok on rFVD, so the metric choice is consequential and not justified.
  • ad hoc to paper The decoder generalizes from training on random coordinate subsets to dense full-video decoding at evaluation
    Appendix B.1 describes passing all patch coordinates at once; the paper does not analyze or ablate this train/eval distribution shift.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction." pith.science (2026). https://pith.science/paper/SLISYALV

@misc{pith2026241114762,
  author       = {Pith},
  title        = {Pith review of: Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SLISYALV}},
  note         = {Machine review of arXiv:2411.14762}
}
abstract

Efficient tokenization of videos remains a challenge in training vision models that can process long videos. One promising direction is to develop a tokenizer that can encode long video clips, as it would enable the tokenizer to leverage the temporal coherence of videos better for tokenization. However, training existing tokenizers on long videos often incurs a huge training cost as they are trained to reconstruct all the frames at once. In this paper, we introduce CoordTok, a video tokenizer that learns a mapping from coordinate-based representations to the corresponding patches of input videos, inspired by recent advances in 3D generative models. In particular, CoordTok encodes a video into factorized triplane representations and reconstructs patches that correspond to randomly sampled $(x,y,t)$ coordinates. This allows for training large tokenizer models directly on long videos without requiring excessive training resources. Our experiments show that CoordTok can drastically reduce the number of tokens for encoding long video clips. For instance, CoordTok can encode a 128-frame video with 128$\times$128 resolution into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar reconstruction quality. We further show that this efficient video tokenization enables memory-efficient training of a diffusion transformer that can generate 128 frames at once.

Figures

Figures reproduced from arXiv: 2411.14762 by the authors.

Figure 1
Figure 1. Limitation of existing video tokenizers. (a) Existing video tokenizers [11, 52, 66] are often not scalable to long videos because of excessive memory and computational demands. This is because they are trained to reconstruct all video frames at once, i.e., a giant 3D array of pixels, which incurs a huge computation and memory burden in training especially when trained on long videos. For instance, PVDM-AE [66] becom… view at source ↗
Figure 2
Figure 2. Overview of CoordTok. We design our encoder to encode a video x into factorized triplane representations z = [z xy , z yt , z xt] which can efficiently represent the video with three 2D latent planes. Given the triplane representations z, our decoder learns a mapping from (x, y, t) coordinates to RGB pixels within the corresponding patches. In particular, we extract coordinate-based representations of N sampled coor… view at source ↗
Figure 3
Figure 3. 128-frame, 128×128 resolution video reconstruction results from CoordTok (Ours) and baselines [52, 66] trained on the UCF-101 dataset [42]. For each frame, we visualize the ground-truth (GT) and reconstructed pixels within the region highlighted in the red box, where CoordTok achieves noticeably better reconstruction quality than other baselines. 1000 1200 TATS-AE PVDM-AE LARP OmniTokenizer-CV CoordTok (Ours) 1024 1… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: CoordTok can efficiently encode long videos. rFVD scores of video tokenizers, evaluated on 128-frame videos, with respect to the token size. ↓ indicates lower values are better. frames for fine-tuning. This enables the tokenizer to com￾pute and minimize LPIPS loss, whi…
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Analysis on the effect of (a) model size, (b) spatial dimensions of triplane representations, and (c) temporal dimensions of triplane representations. For our main experiments, we use CoordTok-L with triplane representations of 16×16 spatial dimensions and 32 temporal …
Figure 7
Figure 7. Figure 7: Analysis on the effect of (a) triplane representations and (b) coordinate-based representations. (a) We measure the Pearson correlation r between the reconstruction quality and a dynamics metric that measures how dynamic each video is. A video with a larger dynamics ma…
Figure 8
Figure 8. Figure 8: Illustration of factorized triplane representations z = [z xy , z yt , z xt] of CoordTok trained on the UCF-101 dataset [42]. We note that z xy captures the global content in the video across time, e.g., layout and appearance of the scene or object, and z yt , z xt cap…
Figure 9
Figure 9. Figure 9: FLOPs when training video tokenizers on 128 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: rFVD scores of video tokenizers, evaluated on 16- [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Additional 128-frame, 128×128 resolution video reconstruction results from CoordTok (Ours) trained on the UCF-101 dataset [42]. For each frame, we visualize the ground-truth (GT) and reconstructed pixels from CoordTok. 𝑡𝑡 = 0 𝑡𝑡 = 16 𝑡𝑡 = 32 𝑡𝑡 = 48 𝑡𝑡 = 64 𝑡𝑡 = 80 𝑡𝑡…
Figure 12
Figure 12. Figure 12: Unconditional 128-frame, 128×128 resolution video generation results from CoordTok-SiT-L/2 trained on 128-frame videos from the UCF-101 dataset [42]. 5 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Unconditional 128-frame, 128×128 resolution video generation results from CoordTok-SiT-L/2 trained on 128-frame videos from the UCF-101 dataset [42]. 6 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 56 canonical work pages

  1. [1]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In IEEE Conference on Computer Vi- sion and Pattern Recognition, 2023. 1, 8

    work page 2023

  2. [2]

    Video generation models as world simulators.Ope- nAI Blog, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators.Ope- nAI Blog, 2024. 1, 8

    work page 2024

  3. [3]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition ,

    work page

  4. [4]

    Efficient geometry-aware 3d generative adversarial networks

    Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. InIEEE Conference on Computer Vision and Pattern Recognition ,

    work page

  5. [5]

    MaskGIT: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2022. 5, 7, 8, 2, 3

    work page 2022

  6. [6]

    VideoINR: Learning video implicit neural represen- tation for continuous space-time super-resolution

    Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. VideoINR: Learning video implicit neural represen- tation for continuous space-time super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition ,

    work page

  7. [7]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInternational Con- ference on Learning Representations, 2021. 4

    work page 2021

  8. [8]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition ,

    work page

Show all 72 references
  1. [9]

    Cosmos world foundation model platform for physical ai

    NVIDIA et al. Cosmos world foundation model platform for physical ai. arXiv preprint, 2025. 5

    2025

  2. [10]

    Latte: Latent diffusion transformer for video generation

    Xin Ma et al. Latte: Latent diffusion transformer for video generation. arXiv preprint, 2024. 5, 6, 3

    2024

  3. [11]

    Long video generation with time-agnostic vqgan and time- sensitive transformer

    Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time- sensitive transformer. In European Conference on Computer Vision, 2022. 1, 2, 4, 5, 7, 8, 3

    2022

  4. [12]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014. 3

    2014

  5. [13]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, 2024. 8

    2024

  6. [14]

    A technical overview of av1

    Jingning Han, Bohan Li, Debargha Mukherjee, Ching-Han Chiang, Adrian Grange, Cheng Chen, Hui Su, Sarah Parker, Sai Deng, Urvang Joshi, et al. A technical overview of av1. Proceedings of the IEEE, 109(9):1435–1462, 2021. 8

    2021

  7. [15]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  8. [16]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In Advances in Neural Informa- tion Processing Systems, 2020. 8

    2020

  9. [17]

    CogVideo: Large-scale pretraining for text-to- video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. CogVideo: Large-scale pretraining for text-to- video generation via transformers. In International Confer- ence on Learning Representations, 2023. 8

    2023

  10. [18]

    LRM: Large reconstruction model for single image to 3D

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3D. In International Conference on Learning Representa- tions, 2024. 1, 2

    2024

  11. [19]

    Shap-e: Generat- ing conditional 3d implicit functions

    Heewoo Jun and Alex Nichol. Shap-e: Generat- ing conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023. 1

    2023 arXiv

  12. [20]

    Analyzing and improving the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2020. 5, 3

    2020

  13. [21]

    Hybrid video diffusion models with 2d triplane and 3d wavelet rep- resentation

    Kihong Kim, Haneol Lee, Jihye Park, Seyeon Kim, Kwanghee Lee, Seungryong Kim, and Jaejun Yoo. Hybrid video diffusion models with 2d triplane and 3d wavelet rep- resentation. In European Conference on Computer Vision ,

  14. [22]

    Scal- able neural video representations with learnable positional features

    Subin Kim, Sihyun Yu, Jaeho Lee, and Jinwoo Shin. Scal- able neural video representations with learnable positional features. In Advances in Neural Information Processing Sys- tems, 2022. 1, 2

    2022

  15. [23]

    9 Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. 9 Videopoet: A large language model for zero-shot video gen- eration. In International Conference on Machine Learning ,

  16. [24]

    One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimiza- tion

    Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimiza- tion. In Advances in Neural Information Processing Systems,

  17. [25]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2018. 4, 1

    2018

  18. [26]

    Vdt: General-purpose video diffusion transformers via mask modeling

    Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General-purpose video diffusion transformers via mask modeling. In International Conference on Learning Representations, 2024. 8

    2024

  19. [27]

    SiT: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. SiT: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Com- puter Vision, 2024. 2, 5, 6, 1

    2024

  20. [28]

    Detlev Marpe, Thomas Wiegand, and Gary J Sullivan. The h. 264/mpeg4 advanced video coding standard and its appli- cations. IEEE communications magazine, 2006. 1

    2006

  21. [29]

    GTA: A geometry-aware attention mechanism for multi-view transformers

    Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. GTA: A geometry-aware attention mechanism for multi-view transformers. In International Conference on Learning Representations, 2024. 1

    2024

  22. [30]

    A technical overview of vp9—the latest open- source video codec

    Debargha Mukherjee, Jingning Han, Jim Bankoski, Ronald Bultje, Adrian Grange, John Koleszar, Paul Wilkins, and Yaowu Xu. A technical overview of vp9—the latest open- source video codec. SMPTE Motion Imaging Journal, 2015. 1, 8

    2015

  23. [31]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In IEEE International Conference on Computer Vision, 2023. 2

    2023

  24. [32]

    Searching for activation functions

    Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017. 2

    2017 arXiv

  25. [33]

    Gen- erating diverse high-fidelity images with VQ-V AE-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gen- erating diverse high-fidelity images with VQ-V AE-2. InAd- vances in Neural Information Processing Systems, 2019. 8

    2019

  26. [34]

    Karel Rijkse. H. 263: Video coding for low-bit-rate commu- nication. IEEE Communications magazine, 1996. 1

    1996

  27. [35]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022. 8

    2022

  28. [36]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. 2

    2015

  29. [37]

    Make-a-video: Text-to-video genera- tion without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video genera- tion without text-video data. In International Conference on Learning Representations, 2023. 8

    2023

  30. [38]

    Implicit neural representa- tions with periodic activation functions

    Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representa- tions with periodic activation functions. In Advances in Neu- ral Information Processing Systems, 2020. 3

    2020

  31. [39]

    StyleGAN-V: A continuous video generator with the price, image quality and perks of StyleGAN2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elho- seiny. StyleGAN-V: A continuous video generator with the price, image quality and perks of StyleGAN2. In IEEE Con- ference on Computer Vision and Pattern Recognition, 2022. 4, 5, 6, 2, 3

    2022

  32. [40]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Confer- ence on Machine Learning, 2015. 8

    2015

  33. [41]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 5

    2021

  34. [42]

    UCF101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  35. [43]

    Overview of the high efficiency video coding (hevc) standard

    Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology, 2012. 1

    2012

  36. [44]

    High efficiency video coding (hevc)

    Vivienne Sze, Madhukar Budagavi, and Gary J Sullivan. High efficiency video coding (hevc). In Integrated cir- cuit and systems, algorithms and architectures , page 40. Springer, 2014. 8

    2014

  37. [45]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 1

    2024 arXiv

  38. [46]

    A good image generator is what you need for high-resolution video synthe- sis

    Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthe- sis. In International Conference on Learning Representa- tions, 2021. 5, 6, 3

    2021

  39. [47]

    MoCoGAN: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 5, 6, 3

    2018

  40. [48]

    FVD: A new metric for video generation, 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation, 2019. 4, 2

    2019

  41. [49]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017. 8

    2017

  42. [50]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. 2

    2017

  43. [51]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  44. [52]

    LARP: Tokenizing videos with a learned autoregressive generative prior

    Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. LARP: Tokenizing videos with a learned autoregressive generative prior. arXiv preprint arXiv:2410.21264, 2024. 1, 2, 4, 5, 8, 3 10

    2024 arXiv

  45. [53]

    OmniTokenizer: A joint image- video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. OmniTokenizer: A joint image- video tokenizer for visual generation. In Advances in Neural Information Processing Systems, 2024. 5, 8, 2, 3

    2024

  46. [54]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 1, 2, 8

    2024 arXiv

  47. [55]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing,

  48. [56]

    Overview of the h

    Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/avc video coding stan- dard. IEEE Transactions on circuits and systems for video technology, 13(7):560–576, 2003. 8

    2003

  49. [57]

    Group normalization

    Yuxin Wu and Kaiming He. Group normalization. In Euro- pean Conference on Computer Vision, 2018. 2

    2018

  50. [58]

    Videogpt: Video generation using vq-vae and trans- formers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 8

    2021 arXiv

  51. [59]

    Temporally consistent transformers for video gen- eration

    Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. Temporally consistent transformers for video gen- eration. In International Conference on Machine Learning ,

  52. [60]

    ElasticTok: Adap- tive tokenization for image and video

    Wilson Yan, Matei Zaharia, V olodymyr Mnih, Pieter Abbeel, Aleksandra Faust, and Hao Liu. ElasticTok: Adap- tive tokenization for image and video. arXiv preprint arXiv:2410.08368, 2024. 7, 8, 2

    2024 arXiv

  53. [61]

    CogvideoX: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. CogvideoX: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 8

    2024 arXiv

  54. [62]

    Towards end-to-end generative model- ing of long videos with memory-efficient bidirectional trans- formers

    Jaehoon Yoo, Semin Kim, Doyup Lee, Chiheon Kim, and Seunghoon Hong. Towards end-to-end generative model- ing of long videos with memory-efficient bidirectional trans- formers. In IEEE Conference on Computer Vision and Pat- tern Recognition, 2023. 8

    2023

  55. [63]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In IEEE Conference on Com- puter Vision and Pattern Recognition , 2023. 1, 2, 4, 5, 8, 3

    2023

  56. [64]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. In In- ternational Conference on Learning Representation...

    2024

  57. [65]

    Generating videos with dynamics-aware implicit generative adversarial net- works

    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial net- works. In International Conference on Learning Represen- tations, 2022. 5, 6, 3

    2022

  58. [66]

    Video probabilistic diffusion models in projected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In IEEE Conference on Computer Vision and Pattern Recognition, 2023. 1, 2, 4, 5, 6, 8, 3

    2023

  59. [67]

    Efficient video diffusion models via content-frame motion-latent decomposition

    Sihyun Yu, Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, and Anima Anandkumar. Efficient video diffusion models via content-frame motion-latent decomposition. In Interna- tional Conference on Learning Representations, 2024. 1, 8

    2024

  60. [68]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 3, 4, 2

    2018

  61. [69]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 1

    2024 arXiv

  62. [70]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 8 11 Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction Supplementa...

    2022 arXiv

  63. [71]

    Architecture We use the same structure as SiT, except that our patch embedding and final projection layers are im- plemented separately for each plane

    We train the model for 600K iterations and we update an EMA model with a momentum parameter 0.9999. Architecture We use the same structure as SiT, except that our patch embedding and final projection layers are im- plemented separately for each plane. To train the uncon- ditio...

  64. [72]

    for 50K iterations after training CoordTok with ℓ2 loss for 1M iterations. We find that fine-tuning phase improves the perceptual quality ( i.e., rFVD score: 188.3 → 102.9, and LPIPS score: 0.141 → 0.066), but degrades the pixel- level reconstruction quality ( i.e., PSNR: 30.3...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.