Pith. sign in

REVIEW 3 major objections 5 minor 58 references

SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SweetTok claims that a video tokenizer which decouples appearance from motion can compress a video to 1,280 tokens while improving reconstruction fidelity by 42.8% and generation quality by 15.1% on UCF-101.

desk verdict Genuinely new decoupled spatial/temporal video tokenizer with supportive ablations, but the headline gains are confounded by a token-count mismatch and K-600 results are worse—worth refereeing with a demand for matched-token experiments. read the letter →

arxiv 2412.10443 v3 pith:3HKUS3LM submitted 2024-12-11 cs.CV cs.AI

Zhentao Tan , Ben Xue , Jian Jia , Junhao Wang , Wencai Ye , Shaoyun Shi , Mingjie Sun , Wenjin Wu
show 2 more authors
Quan Chen Peng Jiang
This is my paper · ORCID
classification cs.CVcs.AI
keywords videotokenizationvectorquantizationspatial-temporaldecouplinglanguagecodebookreconstructiongenerationfew-shotactionrecognitiondiscretevisualtokens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a video tokenizer can compress aggressively and still reconstruct better if it separates appearance from motion instead of flattening all video patches into one sequence. SweetTok uses a Decoupled Query AutoEncoder: spatial tokens are learned from the first frame, temporal tokens from frame-wise residuals, and the video is rebuilt by tiling the decoded first frame and adding residual information back. On UCF-101 at 256x256, the paper reports an rFVD of 20.46 with 1,280 tokens, a 42.8% improvement over the 1,024-token LARP baseline, and a gFVD of 84 versus 99 in class-conditional generation. Because the codebook is built from language words split by part of speech, the same tokens also feed few-shot image and video recognition through a language model, so the paper's larger claim is that one compact discrete representation can serve generation and understanding simultaneously.

What carries the argument

The load-bearing object is the Decoupled Query AutoEncoder (DQAE), a pair of transformer branches in which learnable query tokens cross-attend into patch features: the spatial branch reads only the first frame's patches and the temporal branch reads only the frame-wise residual patches. Reconstruction is staged in the same order, so the model never has to decode appearance and motion from a single mixed sequence. The second load-bearing object is the Motion-enhanced Language Codebook (MLC), a quantizer whose codebook entries are text embeddings from a pretrained vision-language model, projected into the visual latent space by a graph convolution network and split by part of speech: spatial queries search nouns and adjectives, temporal queries search verbs and adverbs. These two mechanisms carry the argument because the first sets the compression structure and the second sets what information survives quantization, and together they make the token sequence both compact and semantically addressable.

What would settle it

Run a trained SweetTok on UCF-101 but move the spatial reference from frame 1 to a later frame, say frame 9, treating the original frames as residuals around that new anchor; if rFVD stays near 20.46, the first-frame residual decomposition is not the mechanism driving the reported gains, whereas a large degradation would confirm that the choice of anchor is load-bearing.

Watch

Extended reading notes

Core claim

SweetTok's central claim is that the previous query-based tokenization loses fidelity because it couples spatial and temporal compression in one flattened sequence, forcing the model to learn intertwined redundancies at once. The paper's answer is a decoupled autoencoder with two branches: one compresses only the first frame into 256 spatial queries, the other compresses frame-wise residuals into 1,024 temporal queries, and reconstruction is ordered so the decoded first frame is tiled and then refined by the temporal decoder. A second claim is that the quantizer gains from being motion-aware: a language codebook partitioned into nouns and adjectives for spatial tokens and verbs and adverbs for temporal tokens lowers rFVD from 24.80 to 20.46 in their ablation, whereas a flat language codebook only reaches 24.80. The paper concludes that decoupling plus motion-sensitive semantic quantization delivers both higher compression and higher reconstruction fidelity than vanilla patch tokenizers or query-based tokenizers, and that the resulting tokens carry semantic content usable by language models.

Load-bearing premise

The load-bearing premise is that the first frame contains essentially all static appearance and every later frame can be represented as that first frame plus frame-wise residuals; if a later frame introduces appearance the first frame lacks, or if first-frame reconstruction errors are large, those errors are replicated into all frames before the temporal decoder can correct them.

Editorial extensions

If this is right

  • With 1,280 tokens, SweetTok reports rFVD 20.46 on UCF-101 versus 35.15 for LARP at 1,024 tokens, so a decoupled design can improve fidelity while using a similar or smaller token budget.
  • Class-conditional video generation improves with the same tokens: gFVD 84 versus 99 for LARP at comparable generator size, and 65 when the generator is scaled to 1.9B parameters.
  • The spatial branch transfers to images: fine-tuning on ImageNet gives rFID 0.37 at 1,024 tokens, so the decoupling benefit is not video-specific.
  • Because tokens map to words, few-shot classification works with a language model: 90.8% on miniImageNet and 90.1% on UCF-101 under the paper's protocols.
  • The verb and adverb half of the codebook is what closes most of the remaining gap, meaning motion semantics are a distinct and recoverable component of video compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper leaves open is reference-frame choice: using a middle frame or a scene-change frame as the spatial anchor could reduce first-frame error propagation in long videos.
  • The decoupled branches suggest modular training: one could upgrade the temporal branch or pretrain it on larger motion datasets without retraining the spatial branch, which the paper does not explore.
  • The paper's limitation section says the codebook is unsupervised and lacks visual-question-answering-grade semantic alignment; adding contrastive learning between video features and codebook text is a direct next step it names.
  • If the part-of-speech split is the active ingredient, then synthetic captions or action labels mined from datasets without captions should produce a similar gain, which would separate the split itself from caption availability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SweetTok, a video tokenizer that splits compression into a spatial branch handling the first frame and a temporal branch handling frame-wise residuals, using decoupled query-based autoencoders (DQAE). A motion-enhanced language codebook (MLC) quantizes spatial tokens with nouns/adjectives and temporal tokens with verbs/adverbs. The authors report a UCF-101 reconstruction rFVD of 20.46 at 1,280 tokens, a 15.1% gFVD improvement over LARP in class-conditional generation, strong image reconstruction results after fine-tuning on ImageNet, and few-shot visual classification results through an LLM-based prompt pipeline.

Significance. If the headline gains survive a matched token-budget comparison, SweetTok's decoupled query design is a genuinely useful step toward compact, high-fidelity video discretization, and the language-codebook interface for few-shot understanding is an appealing additional property. The paper provides an internal ablation chain (Tables 4 and 5) and compares against external baselines (OmniTok, LARP, MAGVIT), which makes the claims testable rather than circular. The weaknesses are experimental: the main comparisons confound architecture with token count, and the central residual decomposition is not stress-tested for error propagation. With additional matched-token experiments and first-frame error analysis, the work could become a solid contribution to the video tokenization literature.

major comments (3)
  1. [§4.2, Table 1, Table 4] The headline improvements are computed across different token budgets: SweetTok uses 1,280 tokens (256 spatial + 1,024 temporal) while LARP-L uses 1,024 tokens in Table 1, and the ablation in Table 4 compares 'Vanilla Query-based (LARP)' at 1,024 tokens with 'Decoupled Query-based (DQAE)' at 1,280 tokens. Since rFVD and gFVD are sensitive to token count, the 42.8% (or 41.8% when computed from Table 4 as (35.15-20.46)/35.15) and 15.1% improvements cannot be attributed to the decoupled query design without a matched-token-count experiment. Please add a DQAE row at 1,024 total tokens (e.g., 256 spatial + 768 temporal) and/or a LARP row at 1,280 tokens, and report gFVD for the same matched conditions.
  2. [§3.2.2, Eqs. (4)-(7)] The residual decomposition assumes that the first frame contains essentially all static spatial content and that frame-wise residuals to the first frame capture all motion information. Because Eq. (7) tiles the decoded first frame T times before adding residuals, any reconstruction error in frame 1 is replicated into all frames before the temporal decoder can correct it. The paper reports no first-frame reconstruction error versus full-video error, no sensitivity to the choice of reference frame (e.g., middle frame), and no comparison against encoding residuals from later frames. Without such measurements, the error-propagation risk of this design is unquantified; please provide these analyses or justify the first-frame choice more directly.
  3. [§4.4, Tables 4 and 5] The ablation chain is internally inconsistent: Table 4 reports rFVD 35.15 for 'Vanilla Query-based (LARP)' while Table 5 reports rFVD 29.45 for 'Baseline (w/o LC)'. It is unclear whether the Table 5 baseline is the same LARP-style model, a DQAE without any language codebook, or something else, and Table 5 does not list token counts. Since the MLC contribution is a main contribution, please reconcile the baselines across the two tables and report token counts and architecture for every ablation row.
minor comments (5)
  1. [Abstract, §4.2] The abstract's 42.8% improvement does not match the precise numbers in Table 4: (35.15 - 20.46)/35.15 = 41.8%. Please unify the percentage claim with the reported rFVD values.
  2. [Table 1, §4.2] On Kinetics-600, SweetTok's rFVD of 25 is worse than LARP-L's 23, although the text says the results are 'comparable'. Please acknowledge this explicitly and discuss the discrepancy, since the main text elsewhere emphasizes improvements.
  3. [§3.2.3, §4.5] The description of the vocabulary extraction and the graph construction is thin: please specify how candidate vocabularies are extracted from video captions, how the 5-token co-occurrence window is used to build graph edges, and why 21 tokens are selected for the few-shot prompt.
  4. [Throughout] There are several typos and inconsistent names: 'pre-traiend' (Introduction), 'reconsturct' (§3.2.2), 'LlammaGen' (Table 3 text), 'similary' (§4.4), and 'grean' (supplementary Figure 9 caption). Please proofread the manuscript.
  5. [Supplementary, Table 9] The training hyperparameters are useful, but the paper does not state whether model weights or code will be released; please include an availability statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SweetTok's reconstruction and generation claims are benchmarked against external baselines and supported by an internal ablation chain, with no self-citation or construction-level reduction.

full rationale

SweetTok's derivation chain is self-contained against external benchmarks. The DQAE equations (4)-(8) define spatial tokens from the first frame and temporal tokens from frame-wise residuals, and the reconstruction is supervised by an external reconstruction loss (L2, LPIPS, VQ, GAN) against ground-truth video; no predicted quantity is defined as its own input. The MLC codebook is initialized from CLIP text embeddings of caption vocabularies, but reconstruction fidelity is measured by rFVD on video frames, and the reported gains are against external baselines (LARP, OmniTok, MAGVIT, etc.) rather than against a fitted version of the same claim. The ablation in Table 4 varies architecture and token count simultaneously (DQAE at 1,280 vs LARP at 1,024), so the headline 42.8% and 15.1% improvements may partly reflect the larger token budget; that is a comparison-fairness and correctness issue, not circularity, because the numbers are not forced by construction and no self-citation carries the argument. No load-bearing self-citations or imported uniqueness theorems appear; all citations are to independent prior work. Accordingly, no circular step can be exhibited with the required specificity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The claims rest on one empirical architecture plus standard learning machinery; the load-bearing assumptions are the first-frame residual decomposition and the sufficiency of a caption-derived CLIP codebook after a learned projection, neither of which is separately proven. The free parameters are hand-chosen capacities (256 spatial and 1,024 temporal tokens, patch sizes, vocabulary threshold, 21 selected tokens for the few-shot prompt) that define the compression ratio and the evaluation pipeline. No physical or mathematical entities are invented; the MLC codebook is an architectural artifact with testable handles in the ablations.

free parameters (5)
  • Spatial query count Lspatial = 256
    Hand-chosen in Section 3.2.2; it directly sets the compression ratio (256 tokens from 1,024 spatial patches) and is load-bearing for the 'compact tokenization' claim.
  • Temporal query count Ltemporal = 1024
    Hand-chosen in Section 3.2.2; 1,024 temporal tokens from 4x32x32 residual patches; the resulting total of 1,280 tokens is the basis for comparing with LARP's 1,024 tokens.
  • Patch kernels (pt, ph, pw) = (4, 8, 8)
    Patchify settings in Section 3.2.1 producing vs of shape 1x32x32 and vt of shape 4x32x32; chosen by hand, determines the token layout.
  • Vocabulary frequency threshold = 5
    Section 4.1: words with frequency below 5 are excluded, yielding codebooks of 10,481 spatial and 11,139 temporal entries; the threshold shapes quantization granularity.
  • Top-k token selection for few-shot prompt = 21
    Section 4.5: the top 21 tokens by CLIP similarity are formed into the LLM prompt; the reported few-shot accuracies depend on this choice.
assumptions (6)
  • domain assumption The first frame concentrates the spatial information of the video, and frame-wise residuals to that first frame capture the remaining temporal information.
    Invoked in Section 3.2.2 and Figure 2c; load-bearing because the decoded first frame is tiled T times and errors propagate into every reconstructed frame.
  • domain assumption Frozen CLIP text embeddings of dataset caption vocabularies, after a learned GCN projection, form codebook points that align with visual features under the VQ-VAE losses.
    Section 3.2.3, Equations 9-11; the paper relies on this alignment for quantized tokens to carry semantic meaning; support comes only indirectly from few-shot results, and the limitation section concedes precise semantic alignment fails.
  • domain assumption The reconstruction losses (L2, LPIPS, VQ commitment, GAN) suffice for a tokenizer that transfers to autoregressive generation and few-shot understanding.
    Section 3.2.2 final paragraph; this is the standard VQ-VAE objective assumption, not separately tested in this paper beyond ablations of architecture and codebook.
  • domain assumption rFVD and gFVD numbers quoted from prior papers are comparable under the same protocol.
    Sections 4.1-4.2 and Table 1: several rows are taken from other papers and some are at 128x128 resolution, marked only by gray highlighting that is not verifiable in plain text.
  • domain assumption Dataset captions used for vocabulary extraction cover the semantics needed for reconstruction and understanding.
    Section 3.2.3: 'we first extract candidate vocabularies of the whole dataset from video captions'; the codebook is dataset-derived, which can inflate metrics on that same dataset.
  • standard math Standard transformer, attention, GCN, EMA, and VQ machinery behave as assumed in Equations 2-7.
    Self-attention, cross-attention, graph convolution, and EMA codebook updates are background techniques from cited works, assumed to work without proof here.
invented entities (1)
  • Motion-enhanced Language Codebook (MLC) with POS-split quantization independent evidence
    purpose: A codebook of CLIP embeddings of nouns and adjectives for spatial tokens and verbs and adverbs for temporal tokens, intended to carry appearance and motion semantics during compression.
    This is an architectural artifact, not a physical entity. It has falsifiable handles outside the construct itself: the reconstruction ablations (rFVD 29.45 to 20.46 in Table 5) and the few-shot results in Table 6, so independent_evidence is true in the sense that the design is testable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization." pith.science (2026). https://pith.science/paper/3HKUS3LM

@misc{pith2026241210443,
  author       = {Pith},
  title        = {Pith review of: SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3HKUS3LM}},
  note         = {Machine review of arXiv:2412.10443}
}
read the original abstract

This paper presents the \textbf{S}emantic-a\textbf{W}ar\textbf{E} spatial-t\textbf{E}mporal \textbf{T}okenizer (SweetTok), a novel video tokenizer to overcome the limitations in current video tokenization methods for compacted yet effective discretization. Unlike previous approaches that process flattened local visual patches via direct discretization or adaptive query tokenization, SweetTok proposes a decoupling framework, compressing visual inputs through distinct spatial and temporal queries via \textbf{D}ecoupled \textbf{Q}uery \textbf{A}uto\textbf{E}ncoder (DQAE). This design allows SweetTok to efficiently compress video token count while achieving superior fidelity by capturing essential information across spatial and temporal dimensions. Furthermore, we design a \textbf{M}otion-enhanced \textbf{L}anguage \textbf{C}odebook (MLC) tailored for spatial and temporal compression to address the differences in semantic representation between appearance and motion information. SweetTok significantly improves video reconstruction results by \textbf{42.8\%} w.r.t rFVD on UCF-101 dataset. With a better token compression strategy, it also boosts downstream video generation results by \textbf{15.1\%} w.r.t gFVD. Additionally, the compressed decoupled tokens are imbued with semantic information, enabling few-shot recognition capabilities powered by LLMs in downstream applications.

Figures

Figures reproduced from arXiv: 2412.10443 by the authors.

Figure 1
Figure 1. Illustration of our framework. We build a compact vi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline overview. (a) Vanilla video tokenizers directly quantize flattened video patches. (b) Vanilla query-based tokenizers compress flattend video patches into adaptive queries. (c) SweetTok proposes decoupled query-based autoencoder (DQAE, §3.2.2). The spatial encoder quantizes the first frame’s patch embeddings, while the temporal encoder quantizes residual between consecutive frames. The spatial decoder recons… view at source ↗
Figure 3
Figure 3. The semantics of spatial-temporal “words”. The atten [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Video reconstruction result on UCF-101 dataset. We also visualize the reconstruction and GT error map, where brighter areas [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Class-conditional video generation result on UCF-101 dataset. The action class label of each row is: [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of image reconstruction results. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison of the reconstruction results of OmniTokenizer and SweetTok on UCF-101 dataset, where ”Diff” represents the pixel [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Comparison of the reconstruction results of OmniTokenizer and SweetTok on K-600 dataset, where ”Diff” represents the pixel [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Comparison of the reconstruction results of TiTok, OmniTokenizer, and SweetTok on ImageNet-1K dataset. Differences are [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Semantic words visualization for UCF-101. The visualization is based on few shot video action recognition tasks. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 39 canonical work pages

  1. [1]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 8

    arXiv 2023

  2. [2]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 3

    arXiv 2005

  3. [3]

    End- to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-end object detection with transformers. In ECCV, pages 213–229. Springer, 2020. 2

    2020

  4. [4]

    A short note about kinetics-

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-

  5. [5]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, pages 11315–11325, 2022. 1, 2, 6

    work page 2022

  6. [6]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. JMLR, 24(240):1–113, 2023. 3

    work page 2023

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 2, 5

    work page 2009

  8. [8]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional trans- formers for language understanding. NAACL, 2018. 2, 3

    work page 2018

Show all 58 references
  1. [9]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. ICLR, 2020. 1, 2

    2020

  2. [10]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Taming transformers for high-resolution image synthesis. In CVPR,

  3. [11]

    Long video generation with time-agnostic vqgan and time- sensitive transformer

    Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time- sensitive transformer. In ECCV, pages 102–118. Springer,

  4. [12]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. NeurIPS, 30, 2017. 5

    2017

  5. [13]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 6

    2022 arXiv

  6. [14]

    Video-lavit: Unified video- language pre-training with decoupled visual-motional tok- enization

    Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, and Yadong Mu. Video-lavit: Unified video- language pre-training with decoupled visual-motional tok- enization. In ICML, pages 22185–22209, 2024. 1, 2, 6

    2024

  7. [15]

    Unified language-vision pre- training in llm with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jian- chao Tan, Yadong Mu, et al. Unified language-vision pre- training in llm with dynamic discrete visual tokenization. In ICLR, 2024. 2

    2024

  8. [16]

    The kinetics hu- man action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  9. [17]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. ICLR,

  10. [18]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 6

    2014 arXiv

  11. [19]

    Semi-supervised classifi- cation with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classifi- cation with graph convolutional networks. ICLR, 2016. 3

    2016

  12. [20]

    Few shot activ- ity recognition using variational inference

    Neeraj Kumar and Siddhansh Narang. Few shot activ- ity recognition using variational inference. arXiv preprint arXiv:2108.08990, 2021. 8

    2021 arXiv

  13. [21]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, pages 11523–11532, 2022. 7

    2022

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023. 2, 4

    2023

  15. [23]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. EMNLP, 2023. 1, 2

    2023

  16. [24]

    Language quan- tized autoencoders: Towards unsupervised text-image align- ment

    Hao Liu, Wilson Yan, and Pieter Abbeel. Language quan- tized autoencoders: Towards unsupervised text-image align- ment. NeurIPS, 36, 2023. 2, 3, 5

    2023

  17. [25]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024. 7, 1

    2024 arXiv

  18. [26]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 2

    2019

  19. [27]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763, 2021. 3, 5, 8

    2021

  20. [28]

    Gen- erating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gen- erating diverse high-fidelity images with vq-vae-2. NeurIPS, 32, 2019. 2

    2019

  21. [29]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684– 10695, 2022. 3

    2022

  22. [30]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  23. [31]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 7, 1

    2024 arXiv

  24. [32]

    Generative pretraining in multi- modality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multi- modality. ICLR, 2024. 1, 2

    2024

  25. [33]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  26. [34]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Es- lami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeurIPS, 34:200– 212, 2021. 5

    2021

  27. [35]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5

    2018 arXiv

  28. [36]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017. 1, 2, 5

    2017

  29. [37]

    Attention is all you need

    A Vaswani. Attention is all you need. NeurIPS, 2017. 2

    2017

  30. [38]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2023. 1, 2

    2023

  31. [39]

    Larp: Tokenizing videos with a learned autoregressive generative prior

    Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior. arXiv preprint arXiv:2410.21264, 2024. 1, 2, 3, 4, 6, 7, 8

    2024 arXiv

  32. [40]

    Omnivid: A generative framework for universal video understanding

    Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Omnivid: A generative framework for universal video understanding. In CVPR, pages 18209–18220, 2024. 1, 2

    2024

  33. [41]

    Omnitokenizer: A joint image- video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image- video tokenizer for visual generation. NeurIPS, 2024. 1, 2, 3, 4, 5, 6, 7, 8

    2024

  34. [42]

    Internvideo2: Scaling video foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. ECCV, 2024. 1, 2

    2024

  35. [43]

    De-diffusion makes text a strong cross- modal interface

    Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, and Jiahui Yu. De-diffusion makes text a strong cross- modal interface. In CVPR, pages 13492–13503, 2024. 3

    2024

  36. [44]

    Videogpt: Video generation using vq-vae and trans- formers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 7

    2021 arXiv

  37. [45]

    Towards end-to-end generative model- ing of long videos with memory-efficient bidirectional trans- formers

    Jaehoon Yoo, Semin Kim, Doyup Lee, Chiheon Kim, and Seunghoon Hong. Towards end-to-end generative model- ing of long videos with memory-efficient bidirectional trans- formers. In CVPR, pages 22888–22897, 2023. 1

    2023

  38. [46]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. ICLR, 2022. 2, 7

    2022

  39. [47]

    Scaling autoregres- sive models for content-rich text-to-image generation.ICLR,

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation.ICLR,

  40. [48]

    MAGVIT: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. MAGVIT: Masked generative video transformer. In CVPR, 2023. 1, 2, 3, 6, 7

    2023

  41. [49]

    Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms

    Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolf- gang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. NeurIPS, 36, 2023. 2, 3, 5, 8

    2023

  42. [50]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. ICLR,

  43. [51]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. NeurIPS, 2024. 1, 2, 4, 7, 8

    2024

  44. [52]

    Codebook transfer with part-of-speech for vector-quantized image modeling

    Baoquan Zhang, Huaibin Wang, Chuyao Luo, Xutao Li, Guotao Liang, Yunming Ye, Xiaochen Qi, and Yao He. Codebook transfer with part-of-speech for vector-quantized image modeling. In CVPR, pages 7757–7766, 2024. 2, 3, 5

    2024

  45. [53]

    Few-shot action recog- nition with permutation-invariant attention

    Hongguang Zhang, Li Zhang, Xiaojuan Qi, Hongdong Li, Philip HS Torr, and Piotr Koniusz. Few-shot action recog- nition with permutation-invariant attention. In ECCV, pages 525–542. Springer, 2020. 5, 8

    2020

  46. [54]

    Beyond text: Frozen large language models in visual signal comprehension

    Lei Zhu, Fangyun Wei, and Yanye Lu. Beyond text: Frozen large language models in visual signal comprehension. In CVPR, pages 27047–27057, 2024. 3, 8 SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization Supplementary Material Notations Explanatio...

    2024

  47. [56]

    Model Implementation Details Visual Tokenizer

    Experimental Settings 6.1. Model Implementation Details Visual Tokenizer. The tokenizer is composed of an en- coder E, decoder D, and latent quantizer Q. The tokenizer takes a video clip of 17 consecutive frames with a resolution of 256 × 256 with the elements normalized to[−0...

  48. [57]

    More evaluation metrics We assess SweetTok using additional metrics: PSNR, SSIM, and LPIPS

    Additional Results 7.1. More evaluation metrics We assess SweetTok using additional metrics: PSNR, SSIM, and LPIPS. As shown in Table 8, SweetTok outper- forms all baselines on both video and image datasets, fur- ther validating the superiority of our model design. 7.2. More V...

  49. [58]

    Limitations Our tokenizer is not suitable for tasks requiring precise semantic understanding, like VQA, because the MLC is trained in an unsupervised manner. Without additional con- straints, such as contrastive learning between image fea- tures from Qwen-VLM and text embeddin...

  50. [600]

    arXiv preprint arXiv:1808.01340, 2018. 5

    2018 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.