arxiv: 2604.12887 · v1 · submitted 2026-04-14 · 💻 cs.CV · cs.LG

Recognition: unknown

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

Andrei Atanov , Jesse Allardice , Roman Bachmann , O\u{g}uzhan Fatih Kar , R Devon Hjelm , David Griffiths , Peter Fu , Afshin Dehghan

show 1 more author

Amir Zamir

Authors on Pith no claims yet

Pith reviewed 2026-05-10 15:41 UTC · model grok-4.3

classification 💻 cs.CV cs.LG

keywords video tokenizationcoarse-to-fine representationflexible-length tokensvideo generationefficient traininggenerative decoder3D grid tokenslong video modeling

0 comments

The pith

VideoFlexTok represents videos as variable-length sequences of tokens ordered from coarse semantics to fine details.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a tokenizer that encodes each video into a flexible sequence of tokens rather than a fixed 3D grid. Early tokens in the sequence capture high-level information such as overall semantics and motion, while later tokens supply additional low-level details. A generative flow decoder reconstructs the video from any prefix of this sequence. If correct, this structure lets downstream generative models allocate tokens according to a video's complexity instead of always processing every spatial and temporal location, which reduces training cost and supports longer videos under the same token limit. Experiments show that this approach achieves similar generation quality to standard grid tokenizers while using a model five times smaller.

Core claim

VideoFlexTok encodes videos as variable-length token sequences structured in a coarse-to-fine manner where the initial tokens emergently capture abstract information such as semantics and motion and subsequent tokens add fine-grained details, with a generative flow decoder that enables realistic reconstructions from any number of tokens without further supervision.

What carries the argument

Variable-length coarse-to-fine token sequence with generative flow decoder supporting reconstruction from arbitrary prefixes.

If this is right

Downstream class-to-video and text-to-video models require fewer parameters to reach comparable gFVD and ViCLIP scores.
The same token budget can encode videos with more frames than fixed-grid baselines allow.
Token count per video can be chosen at inference time to match task requirements without retraining the tokenizer.
Training runs become more efficient because models no longer predict every low-level detail uniformly across all videos.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Generation pipelines could adjust token usage on the fly according to content complexity to trade quality for speed.
The same tokenizer might reduce compute for other sequential visual tasks such as video prediction or editing.
Longer-form video generation becomes practical by scaling token count proportionally to duration rather than to total pixels.
Visual inspection of partial token sequences could serve as an unsupervised probe of semantic structure in videos.

Load-bearing premise

The tokens will automatically organize into a hierarchy that places semantics and motion first without explicit training signals for that ordering.

What would settle it

Reconstruction quality fails to improve steadily as additional tokens are included beyond the first few, or early tokens show no semantic content when decoded and visualized independently.

Figures

Figures reproduced from arXiv: 2604.12887 by Afshin Dehghan, Amir Zamir, Andrei Atanov, David Griffiths, Jesse Allardice, O\u{g}uzhan Fatih Kar, Peter Fu, R Devon Hjelm, Roman Bachmann.

**Figure 1.** Figure 1: VideoFlexTok represents videos with a flexible-length coarse-to-fine sequence of tokens. Top: Compared to the common 3D grid tokenizers, which can adjust the token sequence length only by reducing the video length, VideoFlexTok can represent the same-length video with a varying number of tokens corresponding to different levels of detail – with just a few tokens emergently capturing abstract information, s… view at source ↗

**Figure 2.** Figure 2: VideoFlexTok reconstructions from a variable number of tokens. We find that just a few VideoFlexTok tokens capture information such as the semantic identities (e.g., a woman in the right example), scene geometry (the “arch”), camera motion (moving forward), and object motion (rotation). 1. Introduction Video modeling1 is computationally expensive, primarily triggered by the high dimensionality of the raw p… view at source ↗

**Figure 3.** Figure 3: VideoFlexTok overview. The encoder takes the spatiotemporal VAE video latents, interleaves them with learnable register tokens across the time dimension, and passes them through the Transformer with a time-causal attention pattern. This results in a 2D representation with the temporal and coarse-to-fine dimensions. Nested dropout randomly drops a random number of last register tokens along the 2nd dimensio… view at source ↗

**Figure 4.** Figure 4: Probing the first VideoFlexTok tokens. We design the following probing experiment to analyze the information contained in the first VideoFlexTok tokens. Given a source video, we keep only one or two tokens per latent frame and make an isolated change to its first frame, e.g., changing an orange to an apple, using Nano Banana (Google, 2025). We then condition the decoder on both the original tokens and the … view at source ↗

**Figure 5.** Figure 5: Flexible-length autoregressive text-to-video generation. A text-to-video generative model using VideoFlexTok tokens can generate token sequences of varying length for a given conditioning. All token budgets lead to plausible generations, with 2-4 tokens/frame capturing the overall scene details and motion described in the text conditioning well (e.g., the balloon movement), while generating more tokens can… view at source ↗

**Figure 6.** Figure 6: Compute-efficient AR training with VideoFlexTok. We show how the fidelity (top) and alignment (bottom) metrics change across three complementary scaling axes. Scaling the model size (left). We show how the fidelity (top, gFVD) and alignment (bottom, Classification Score) metrics change as we scale the size of the class-to-video autoregressive model. Using VideoFlexTok maintains good fidelity across a wider… view at source ↗

**Figure 7.** Figure 7: Flexible-length generation. We measure fidelity (top, gFVD), and alignment (bottom, classification score and ViCLIP similarity, see Section 4.1) for VideoFlexTok and 3D grid tokenizers on class-to-video (left) and text-to-video (right) tasks. Using much fewer tokens, VideoFlexTok maintains fidelity comparable to or better than the 3D tokenizer, while achieving higher alignment, i.e., better solving the co… view at source ↗

**Figure 8.** Figure 8: Long text-to-video generation. We show an exemplar generation of a 10-second 81-frame video using only 672 tokens (32 tokens per frame) [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

**Figure 9.** Figure 9: REPA (Yu et al., 2025) loss ablation. We compare tokenizers trained with and without the REPA loss on the classto-video downstream task. We find that REPA inductive bias loss improves both the fidelity of the generated samples and the alignment with the class conditioning. A. Overview video We provide an overview video of our submission in overview.mp4. B. Additional qualitative results In Figures 13 to 1… view at source ↗

**Figure 11.** Figure 11: Hierarchical vs. raster-order generation. We compare the VideoFlexTok and 3D grid tokenizers’ performance at the same sequence length (1280 tokens). We find that hierarchical generation with VideoFlexTok leads to 1) better alignment (ViCLIP score) and 2) much better fidelity (gFVD) when not using classifier-free guidance. using the same number of tokens per frame N = 256 for both. First, we find that Vi… view at source ↗

**Figure 12.** Figure 12: Text-to-video inference cost analysis. We compare the inference cost of various configurations of the number of AR-generated tokens and the number of VideoFlexTok flow decoder steps. For each AR model size and the number of generated tokens, we perform {1, 2, 5, 10, 20, 40} denoising steps and plot the corresponding lines. We find that for all considered AR sizes and inference budgets, the best performanc… view at source ↗

**Figure 13.** Figure 13: VideoFlexTok reconstruction example. From top to bottom, each row corresponds to a video reconstructed using 1, 2, 4, . . . , 256 tokens. The last row shows the original video. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗

**Figure 14.** Figure 14: VideoFlexTok reconstruction example. From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, . . . , 256 tokens. The last row shows the original video. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗

**Figure 15.** Figure 15: VideoFlexTok reconstruction example. From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, . . . , 256 tokens. The last row shows the original video. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗

**Figure 16.** Figure 16: VideoFlexTok reconstruction example. From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, . . . , 256 tokens. The last row shows the original video. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png] view at source ↗

read the original abstract

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

VideoFlexTok's variable token lengths with a flow decoder give real efficiency gains for video generation, but the emergent coarse-to-fine claim still needs direct checks.

read the letter

The main point is a tokenizer that turns videos into variable-length token sequences ordered coarse-to-fine, with a generative flow decoder that reconstructs from any prefix length. This replaces fixed 3D grids and lets the token budget scale with video complexity instead of always paying for full detail upfront. They report matching gFVD and ViCLIP scores on class- and text-to-video tasks with a 1.1B model versus 5.2B for the grid baseline, plus 81-frame 10-second videos at 672 tokens, 8x fewer than the comparison setup. That efficiency angle is the clearest practical win and rests on concrete model-size and token-count comparisons rather than just theory. The flow decoder itself is a straightforward addition that makes variable-length training feasible without extra supervision for each count. The soft spot is the hierarchy itself. The paper states the first tokens capture semantics and motion emergently, yet the abstract and stress-test note give no prefix-only reconstruction curves, no linear probes on early tokens, and no ablation that shows later tokens only add fine detail rather than fix coherence. Without those, the smaller-model advantage could trace more to the decoder's flexibility than to any reliable ordering in the tokens. If the full experiments include attention rollouts or progressive masking results, that would close the gap; otherwise the central efficiency story stays partly inferential. This is aimed at groups working on scalable video generators who already fight token budgets. It has enough of a working method and reported numbers to go to referees, who will probably push for the missing hierarchy diagnostics but will still find the efficiency results worth checking.

Referee Report

3 major / 3 minor

Summary. The paper introduces VideoFlexTok, a video tokenization method that encodes videos as variable-length sequences of tokens structured in a coarse-to-fine hierarchy. The first tokens are claimed to emergently capture abstract semantics and motion, with later tokens adding fine-grained details; a generative flow decoder enables realistic reconstruction from arbitrary token prefixes. This design is evaluated on class- and text-to-video generation tasks, reporting efficiency gains such as comparable gFVD and ViCLIP scores using a 1.1B model versus a 5.2B baseline, plus support for longer videos (e.g., 81 frames with 672 tokens, 8x fewer than 3D grid baselines).

Significance. If the emergent coarse-to-fine property holds and enables the reported efficiency without decoder compensation, the work could meaningfully improve scalability of video generative models by allowing adaptive token budgets and reducing the need to model low-level details uniformly. The 5x model size reduction while maintaining quality metrics would be a notable practical advance for training large video models.

major comments (3)

[Abstract and method description] The central efficiency claim (comparable quality with 1.1B vs 5.2B models and long-video capability at 672 tokens) rests on the assertion that tokens emergently organize into a coarse-to-fine hierarchy without additional supervision. However, the manuscript provides no mechanism (e.g., ordering loss, progressive masking) and no verification experiments such as prefix reconstruction curves, attention rollout on early tokens, or linear probes showing semantic/motion capture in the first tokens. This is load-bearing because if the ordering does not emerge consistently, the variable-length training reduces to standard compression and the smaller-model advantage may not hold.
[Experiments section] §4 (experiments): The reported results on gFVD and ViCLIP scores for the 1.1B model lack details on experimental setup, including whether the 5.2B baseline uses the identical tokenizer or a standard 3D grid, the number of training runs or seeds, error bars, and ablations isolating the effect of variable token count versus the flow decoder. Without these, it is difficult to confirm that gains are attributable to the coarse-to-fine structure rather than other factors.
[Method section] The generative flow decoder is presented as enabling reconstruction from any token count, but the manuscript does not specify its architecture, training objective, or how it differs from standard decoders in a way that would support the hierarchy claim. This invented component requires more technical detail to evaluate its contribution independently of the tokenizer.

minor comments (3)

[Abstract] The abstract states that the hierarchy occurs 'emergently' but the full text should clarify any implicit biases in the training procedure (e.g., loss weighting or masking) that might encourage ordering, even if not explicitly designed.
[Method] Notation for variable token count per video and the flow decoder's conditioning on prefix length could be formalized with an equation or pseudocode for clarity.
[Related work] Add references to prior work on hierarchical or variable-length tokenization in images/videos and flow-based generative models to better situate the contribution.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate the revisions planned for the updated manuscript.

read point-by-point responses

Referee: [Abstract and method description] The central efficiency claim (comparable quality with 1.1B vs 5.2B models and long-video capability at 672 tokens) rests on the assertion that tokens emergently organize into a coarse-to-fine hierarchy without additional supervision. However, the manuscript provides no mechanism (e.g., ordering loss, progressive masking) and no verification experiments such as prefix reconstruction curves, attention rollout on early tokens, or linear probes showing semantic/motion capture in the first tokens. This is load-bearing because if the ordering does not emerge consistently, the variable-length training reduces to standard compression and the smaller-model advantage may not hold.

Authors: We appreciate this observation. The coarse-to-fine hierarchy emerges naturally from training with variable-length sequences and the generative flow decoder, which must produce realistic outputs from any prefix length, thereby pressuring early tokens to capture essential semantics and motion. No explicit ordering loss is used, as the emergence is a consequence of the reconstruction objective. We agree that verification experiments are valuable and will include prefix reconstruction quality curves, attention rollout visualizations, and linear probe results on early tokens in the revised manuscript to substantiate the claim. revision: yes
Referee: [Experiments section] §4 (experiments): The reported results on gFVD and ViCLIP scores for the 1.1B model lack details on experimental setup, including whether the 5.2B baseline uses the identical tokenizer or a standard 3D grid, the number of training runs or seeds, error bars, and ablations isolating the effect of variable token count versus the flow decoder. Without these, it is difficult to confirm that gains are attributable to the coarse-to-fine structure rather than other factors.

Authors: We concur that additional experimental details and controls are needed. The 5.2B baseline employs a standard 3D grid tokenizer rather than our method. In the revision, we will specify the experimental setup in full, including the number of training runs and random seeds, report error bars on the metrics, and provide ablations that isolate the variable token count from the flow decoder's contribution. These additions will clarify that the efficiency improvements arise from the proposed tokenization. revision: yes
Referee: [Method section] The generative flow decoder is presented as enabling reconstruction from any token count, but the manuscript does not specify its architecture, training objective, or how it differs from standard decoders in a way that would support the hierarchy claim. This invented component requires more technical detail to evaluate its contribution independently of the tokenizer.

Authors: We acknowledge the manuscript's brevity on this component. The generative flow decoder is a flow-based model trained to maximize the likelihood of the video given variable token prefixes. Its architecture consists of a series of invertible transformations conditioned on the token sequence. Unlike standard decoders that assume a fixed input length, it supports progressive conditioning, which underpins the hierarchy by enabling high-fidelity reconstruction from short prefixes. We will provide a detailed description of the architecture, objective, and differences in the revised method section. revision: yes

Circularity Check

0 steps flagged

No significant circularity; architecture and empirical results are self-contained

full rationale

The paper defines VideoFlexTok via a new variable-length tokenization scheme whose coarse-to-fine organization is presented as an emergent training outcome rather than a quantity fitted or defined in terms of the target metrics. Efficiency claims rest on direct experimental comparisons (1.1B vs 5.2B models, gFVD/ViCLIP scores, token budgets for long videos) with no load-bearing step that reduces by the paper's own equations or self-citations to previously fitted inputs. No self-definitional loops, fitted-input predictions, or uniqueness theorems imported from prior author work appear in the derivation chain; the representation structure is an architectural choice whose downstream benefits are measured externally.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 1 invented entities

The central claim depends on the new tokenization structure and flow decoder; several typical deep-learning hyperparameters are implicitly present but unspecified in the abstract.

free parameters (1)

token count per video
Variable and chosen per downstream task; specific experimental value of 672 tokens for 81-frame videos is demonstrated.

axioms (1)

domain assumption Videos admit a hierarchical coarse-to-fine representation where initial tokens capture semantics and motion
Invoked in the design of the token sequence structure.

invented entities (1)

generative flow decoder no independent evidence
purpose: Enables realistic video reconstruction from any number of tokens in the coarse-to-fine sequence
New component introduced to support variable-length tokenization.

pith-pipeline@v0.9.0 · 5626 in / 1403 out tokens · 47562 ms · 2026-05-10T15:41:35.876303+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · 1 internal anchor

[1]

DINOv2: Learning Robust Visual Features without Supervision

Accessed: 2025-02-14. OpenAI. Sora 2, 2025. URL https://openai.com/ index/sora-2/. Accessed: 2025-02-14. Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023. Peebles, W. an...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[2]

with both temporal and spatial compression. • As the REPA (Yu et al., 2025) head, we use a Trans- former with time-causal attention mimicking the de- coder design, which we found to perform better in terms of both reconstruction and downstream genera- tion performance in our early explorations. • We introduce an additional decoder fine-tuning stage where ...

work page arXiv 2025