{"id":"c6462480-a99f-4e9d-a8c2-04b8f01b939c","arxiv_id":"2501.05442","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Progressive growing of a video tokenizer from 4x to 8x and 16x temporal compression yields higher reconstruction quality and efficient diffusion training than direct training.","lead":"This paper presents a way to train video tokenizers that compress time by up to 16x while keeping good reconstruction quality, by progressively building on a well-trained 4x model. This matters because it can make video generation models much faster and able to produce longer videos with the same compute budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed advantage of progressive growing over direct training is not isolated from the base-model modifications: Tables 2 and 4 compare progressive ProMAG to directly trained vanilla MagViT-v2, not to a directly trained ProMAG.","rationale":"I read the paper in good faith: the method is plausible, the ablations in Table 4 show that residual/AdaNorm additions help on top of naive progressive training, and the efficiency analysis is internally coherent. However, the central claim that progressive growing outperforms direct full-model training is not cleanly established, because the direct baseline is vanilla MagViT-v2 rather than a directly trained ProMAG. ProMAG differs from that baseline by three architectural modifications, one of which (removing mean subtraction from group normalization) is explicitly described as necessary to avoid spot artifacts during progressive growing. Thus the improvement in Tables 2 and 4 could plausibly stem from these modifications instead of from the progressive-growing strategy. This is a missing control rather than an internal inconsistency, and a single from-scratch ProMAG training run would settle it. The reader's conditional verdict remains appropriate, so I recommend no change to the verdict category, but I would add this specific control as a condition for acceptance.","tokens_in":18247,"tokens_out":18416,"duration_ms":175105,"concrete_test":"Run a controlled experiment: train ProMAG-8x and ProMAG-16x from scratch (all three Section 3.1 modifications, no stagewise growth) for the same total iterations, losses, data, and evaluation protocol as the direct MagViT-v2 baselines in Table 2 on MCL-JCV and DAVIS. If from-scratch ProMAG matches the progressive ProMAG within noise, or beats it, the progressive-growing claim is unsupported; if it collapses to the vanilla baseline level, the claim is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 introduces three modifications to MagViT-v2 to form ProMAG: freezing a pretrained 2D image encoder for the first frame, discarding the first frame in decoder upsampling, and removing mean subtraction from group normalization. Section 4.1 then supports the central claim by comparing ProMAG-8x/16x to 'MagViT-v2' trained directly at the target compression. Table 4's progressive-training ablation is also relative to this vanilla baseline; there is no row for the full ProMAG architecture trained from scratch at 8x or 16x. Because the custom norm alone is reported to remove the spot artifacts that otherwise plague progressive training, and image-init changes the optimization trajectory, the observed gap between progressive ProMAG and direct vanilla MagViT-v2 could be explained by the architectural modifications rather than by bootstrapping or AdaNorm. This is load-bearing because the abstract and contributions claim that directly training a full high-compression model is the wrong strategy. If a from-scratch ProMAG also reached the 'ProMAG (w/o residuals & AdaNorm)' numbers in Table 4 or better, the specific progressive-growing contribution would largely disappear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ProMAG, a video tokenizer that extends MagViT-v2 to 8× and 16× temporal compression by progressively growing from a pretrained 4× model. The method trains a 4× base model first, then freezes its encoder/decoder blocks, inserts additional bottleneck downsampling/upsampling blocks, and uses AdaNorm to condition the intermediate latents on key-frame embeddings from temporally subsampled video. A residual path combines the key-frame and learned residual information. The paper reports that this progressive-growing approach achieves significantly better reconstruction quality than directly training MagViT-v2 at the same high compression ratios, and the resulting 16× latent space can train a DiT for text-to-video generation at quality comparable to 4× latents while using far fewer tokens. It also introduces three modifications to the base MagViT-v2 architecture (frozen 2D image encoder initialization, discarding the first frame in decoder upsampling, and a group-norm variant without mean subtraction) and a layer-wise spatial tiling method for high-resolution decoding.","tokens_in":18537,"tokens_out":6775,"duration_ms":62607,"significance":"If the central claim is confirmed, this is a practically valuable contribution to efficient latent video diffusion, since temporal compression beyond 4× without increasing latent channels could significantly reduce the token budget for diffusion transformers. The paper is well motivated by the subsampling observation in Figure 2, and it contains several good experimental practices: an ablation in Table 4 separating the full method from plain progressive training, a comparison against external frame interpolation in Table 5, a training-efficiency table in the appendix, and a downstream text-to-video validation that checks the latent space is actually usable by a DiT. The main weakness is experimental: the advantage of progressive growing over direct training is not isolated from the base-model modifications, because the direct-training baseline is vanilla MagViT-v2 rather than a from-scratch ProMAG. This is a fixable omission, but it currently leaves the core novelty insufficiently supported.","major_comments":[{"comment":"The central claim that progressive growing outperforms direct high-compression training is not isolated from the architectural modifications introduced in Section 3.1. Table 4 compares directly trained \"MagViT-v2\" with \"ProMAG (w/o residuals &AdaNorm)\", but the latter already includes the frozen 2D image encoder, the first-frame-discarding upsampling, and the custom group norm. The observed PSNR gain of 30.35 vs. 28.51 could be due entirely to these modifications rather than to progressive training. Since the abstract and contributions state that directly training the full high-compression model is the wrong strategy, the manuscript must include a from-scratch ProMAG baseline (same architecture, trained directly at 8× and 16× with comparable compute and iterations) in Table 4 and ideally also in Table 2. Without that row, the conclusion that progressive growing is beneficial is not supported.","section":"§4.1, Tables 2 and 4"},{"comment":"All reconstruction metrics are reported as point estimates without standard errors, confidence intervals, or significance tests. Several key comparisons are close, such as the LPIPS values of 6.53 vs. 6.49 for ProMAG and MagViT-v2 at z=8 in Table 1, and the PSNR gain of about 1.7 dB in Table 4 is the main evidence for the method's advantage. The abstract and Section 4.1 repeatedly use the word \"significantly\" without statistical support. The authors should report variance across benchmark clips or multiple training runs, and preferably a paired test, to justify the strength of the claims.","section":"§4.1, Tables 1–5"},{"comment":"The contribution of the individual base-model modifications to the high-compression setting is not quantified. In particular, the paper states that removing mean subtraction from group normalization resolves the spot artifacts (Figure 3), but it does not evaluate whether this modification also improves direct training at 8× or 16× compression. If the custom norm and image-init changes improve direct training, then part of the gap in Table 4 is attributable to Section 3.1 rather than to progressive growing. An ablation of the base modifications on a directly trained ProMAG model would help separate these factors and strengthen the interpretation of the progressive-growing results.","section":"§3.1 and Table 4"}],"minor_comments":[{"comment":"The notation is inconsistent: the superscript * appears on some z variables, and the AdaNorm output is not clearly defined. Please define all symbols explicitly and clarify the dimensions and roles of zkey, zinter, and the final latent z.","section":"§3.2, Eq. (2)–(3)"},{"comment":"Please specify which pretrained 2D image model is used for the frozen image encoder and whether its architecture matches the image-encoding portion of MagViT-v2; this detail is needed for reproducibility.","section":"§3.1"},{"comment":"The red-bordered rectangle marking the artifact region is not conspicuous. A thicker or brighter border would make the qualitative comparison easier to see.","section":"Figure 5"},{"comment":"The dataset name is misspelled as \"MCL-MJC\"; it should be \"MCL-JCV\".","section":"§5.1, Table 4 caption"},{"comment":"The claim of being \"the first to achieve high-quality reconstruction with a 16× temporal compression model\" is strong; please qualify it with \"to our knowledge\" and state the precise comparison regime (continuous tokenizer, fixed latent channels, reconstruction benchmarks) so the novelty claim is verifiable.","section":"§1 and §4.1"},{"comment":"The layer-wise spatial tiling technique is described only qualitatively with Figure 5. Since this tiling is used for high-resolution decoding, please provide a quantitative comparison (e.g., PSNR/LPIPS with and without tiling) or state clearly whether the reported benchmark numbers use the tiling procedure for all methods consistently.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the missing from-scratch ProMAG baseline at 8× and 16× compression. If the authors can add that row and show that progressive growing still provides a meaningful improvement over direct training of the same architecture, the paper would be acceptable. The comparison to a reimplemented MagViT-v2 is understandable given the lack of an official release, but the paper should also be explicit about any potential differences from the original implementation. The downstream T2V experiment is a useful sanity check, though it is limited to internal data and a relatively small training budget."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Progressive growing of a video tokenizer to 8x/16x temporal compression gives real gains over a from-scratch baseline, but the paper never compares against a directly-trained version of its own modified architecture, so the attribution of those gains is under-determined. The paper reports a practical recipe for training video tokenizers at 8x and 16x temporal compression by progressively growing a well-trained 4x model: freeze the old blocks, insert bottleneck down/up-sampling blocks, and use AdaNorm to condition the new blocks on key-frame encodings from the frozen encoder. The headline result - reconstruction at 8x/16x that beats a from-scratch MagViT-v2 by wide margins, plus a 16x latent space that trains a DiT to roughly 4x-level quality at 2.5x the video length per token budget - looks real. If reproducible, that is a material efficiency win for latent video diffusion. What is new: the first credible 16x temporal compression tokenizer, and a specific progressive design (key-frame residuals plus AdaNorm cross-level conditioning) that the paper partially decomposes in Table 4. The progressive-training control shows residuals/AdaNorm add about 1.7 PSNR on top of plain progressive training - a proper ablation. Comparing against 4x-encode plus EMA-VFI interpolation is the right baseline for their motivating observation. The appendix is honest about the 17-frame-chunking jump artifacts and the overlap-blending patch. Soft spots, in proportion. The stress-test concern is accurate: every head-to-head against direct training uses vanilla MagViT-v2, never a from-scratch ProMAG with the same architecture modifications. The frozen-image-init and the no-mean-subtraction norm could plausibly help direct training at high compression too, and since that control is missing, the Table 2 gains cannot be cleanly attributed between architecture and progressive strategy. The specific claim that directly training the full model is the wrong strategy is only demonstrated for an unmodified base. This is a genuine experimental gap and I would want that run before endorsing the attribution. Secondary, mostly minor: no error bars anywhere, and some Table 1 comparisons are within noise distance; MagViT-v2 is a reimplementation; internal training data and no code release cap confidence. These are typical for this subfield, but they are why independent verification matters. Who it is for: practitioners building or using video tokenizers for latent diffusion. It deserves a serious referee. I would push for conditional acceptance: require the from-scratch ProMAG baseline at 8x and 16x, and code or at least released checkpoints.","headline":"Progressive growing of a video tokenizer to 8x/16x temporal compression gives real gains over a from-scratch baseline, but the paper never compares against a directly-trained version of its own modified architecture, so the attribution of those gains is under-determined.","tokens_in":728,"tokens_out":1530,"would_cite":true,"duration_ms":64234,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Progressive growing of a 4× video tokenizer yields better 8× and 16× temporal compression than training the high-compression model directly, and the 16× latent space trains a diffusion model to match 4× generation quality with far fewer…","keywords":["video tokenizer","temporal compression","progressive growing","latent video diffusion","reconstruction quality","text-to-video generation","residual embedding","AdaNorm"],"falsifier":"Train the identical architecture at 8× or 16× temporal compression from scratch with the same data and iteration budget, then compare PSNR, LPIPS, and FVD on the same public benchmark; if the from-scratch model matches or beats ProMAG, the progressive-growing mechanism is not what carries the result. A more targeted check is to replace the frozen 4× encoder with a randomly initialized or lightly trained encoder while keeping the key-frame and AdaNorm pathway intact; if reconstruction quality stays high, the paper's core premise about reusing a well-trained lower-compression representation is unnecessary.","tokens_in":24,"feed_emoji":"🎬","tokens_out":6838,"duration_ms":105630,"temperature":0.7,"pith_summary":"This paper argues that the right way to reach high temporal compression in a video tokenizer is to grow an already-trained low-compression model rather than train the whole high-compression model from scratch. The key observation is that a 4× tokenizer fed a temporally subsampled, low-framerate video reconstructs that video more accurately than a 16× tokenizer reconstructs the original full-framerate video. ProMAG therefore keeps a frozen 4× encoder and decoder, inserts extra bottleneck downsampling and upsampling blocks, and trains only those blocks to capture the residual information needed for in-between frames. On standard reconstruction benchmarks, this progressive growing beats directly trained baselines at 8× and 16× compression, and the resulting 16× latent space trains a video diffusion model to match 4× generation quality while using about 2.5× fewer tokens per step. The paper also claims to be the first to reach high-quality reconstruction at 16× temporal compression.","feed_headline":"Bootstrapping a 4× tokenizer yields sharp 16× video latents","feed_subtitle":"Grow the model in stages instead of training from scratch, and 16× latents cut diffusion cost 2.5× without quality loss.","key_machinery":"The mechanism is progressive model growing with key-frame and residual embeddings. Stage I trains a base 4× tokenizer; Stage II (and Stage III) freeze the pretrained encoder and decoder blocks, insert extra bottleneck downsampling and upsampling blocks, and subsample the input by 2 to obtain key-frame embeddings through the frozen encoder. An adaptive group-normalization (AdaNorm) layer conditions the full-video intermediate features on those key-frame embeddings, the new bottleneck blocks compress the conditioned features into a residual embedding, and a 1×1×1 convolution combines the key-frame and residual latents. This forces the newly added blocks to learn only the in-between frame information, turning high compression into a guided interpolation problem rather than a from-scratch reconstruction problem.","core_discovery":"The central claim is that a video tokenizer supporting 8× or 16× temporal compression can be obtained by bootstrapping a well-trained 4× temporal-compression model: freeze the pretrained encoder and decoder blocks, insert additional 2× bottleneck downsampling and upsampling layers, and train only the new layers plus a 1×1×1 bottleneck to learn the residual information that the frozen 4× representation does not already carry. A cross-level feature-mixing scheme conditions the intermediate encoder features on key-frame embeddings obtained by encoding a temporally subsampled version of the input with the frozen encoder, and the final latent is a linear combination of the key-frame and residual embeddings. The paper reports that this scheme outperforms directly extending and training MagViT-v2 at 8× and 16× compression on both reconstruction metrics and the representational quality for downstream text-to-video diffusion, and that the 16× latent space produces generation quality comparable to 4× with substantially fewer tokens.","pith_inferences":["An implication the paper leaves implicit is that the same progressive recipe may transfer to other bottlenecks, such as spatial compression or mixed spatial-temporal compression, whenever a well-trained lower-compression stage already reconstructs subsampled versions of the input well.","The method's encoder cost is higher than the baseline because it performs two forward passes (one on the full video and one on the subsampled video); the paper reports about 1.5× encoder time, so the total efficiency gain depends on how often encoding is amortized against diffusion training and generation.","A testable extension is to apply the AdaNorm-conditioned residual grow to other base tokenizers and to measure whether the reconstruction gain survives when the frozen base model was trained on a different data distribution; the paper's results are from one internal training set, so transferability is not yet established.","The frame-chunking and overlap-blending workaround for videos longer than 17 frames suggests that a truly streaming or temporally autoregressive decoder could be a natural next step for making high-compression latents seamless on long footage."],"forward_implications":["At fixed latent channel dimension, 8× and 16× temporal compression become usable for video generation, so a given token budget produces much longer videos: 340 frames with 16× latents instead of 136 with 4× latents at the same denoising cost.","Training a high-compression tokenizer by progressive growing is roughly 2.7× cheaper in cumulative GPU time than training the same model directly, because the frozen blocks need no gradients and no discriminator is used during the growth stages.","The compact 16× latent space is compatible with standard DiT-based text-to-video training, yielding generation quality that matches 4× latents on VBench while cutting per-timestep cost by about 2.5×.","The reconstruction gain is not simply stage-wise training: ablations show that including the key-frame and AdaNorm residual pathway adds a further large improvement over progressive training without those components.","A layer-wise spatial tiling trick for the decoder removes the artifacts that arise when decoding high-resolution videos in tiles, which is needed to make the compressed model practical at larger resolutions."],"supporting_citations":[{"why":"Supplies the base MagViT-v2 tokenizer architecture and continuous-token training paradigm that ProMAG modifies and extends.","marker":"[47]"},{"why":"Provides the progressive growing idea from GAN image generation that the paper adapts to video tokenizer compression blocks.","marker":"[20]"},{"why":"Contributes the observation that training many downsampling and upsampling layers at once is difficult, which motivates growing the model in stages.","marker":"[8]"},{"why":"Supplies the discriminator design from Stable Diffusion's VAE, converted to 3D, for training the base 4× tokenizer.","marker":"[35]"},{"why":"Provides the VBench benchmark and its 16 quality dimensions used to evaluate text-to-video generation quality.","marker":"[19]"},{"why":"Supplies the CogVideoX-style diffusion transformer architecture and recipe that the text-to-video experiments build on.","marker":"[46]"},{"why":"Provides the external frame interpolation method used as the baseline for comparing progressive growing against subsampled encoding plus interpolation.","marker":"[48]"},{"why":"Supplies the MCL-JCV benchmark used for reconstruction quality comparisons.","marker":"[43]"},{"why":"Supplies the DAVIS 2019 benchmark used for reconstruction quality comparisons.","marker":"[7]"}],"fun_headline_variants":["Bootstrapping 4x tokenizer yields 16x video latents","Progressive tokenizer growth beats direct 16x training","Stepwise tokenizer bootstrapping yields 16x video latents"],"cache_read_input_tokens":21248,"weakest_assumption_plain":"The method's gains rest on the premise that a frozen, well-trained 4× encoder and decoder remain informative and compatible when additional bottleneck compression blocks are inserted, so that the new blocks only need to learn residual in-between-frame information; this is an empirical premise validated only on the authors' internal training data and a single base architecture.","fun_headline_variants_meta":{"raw":{"variants":["Bootstrapping 4x tokenizer yields 16x video latents","Progressive tokenizer growth beats direct 16x training","Stepwise tokenizer bootstrapping yields 16x video latents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00119,"raw_usage":{"total_tokens":4921,"prompt_tokens":969,"completion_tokens":3952,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":3892}},"tokens_in":585,"tokens_out":3952,"duration_ms":27533,"temperature":1.0,"reasoning_tokens":3892,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:11:59.544901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical architecture at 8× or 16× temporal compression from scratch with the same data and iteration budget, then compare PSNR, LPIPS, and FVD on the same public benchmark; if the from-scratch model matches or beats ProMAG, the progressive-growing mechanism is not what carries the result. A more targeted check is to replace the frozen 4× encoder with a randomly initialized or lightly trained encoder while keeping the key-frame and AdaNorm pathway intact; if reconstruction quality stays high, the paper's core premise about reusing a well-trained lower-compression representation is unnecessary.","supporting_citations":[{"cited_title":"Progressive growing of gans for improved quality, stability, and variation","cited_arxiv_id":null,"evidence_quote":"Provides the progressive growing idea from GAN image generation that the paper adapts to video tokenizer compression blocks."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the discriminator design from Stable Diffusion's VAE, converted to 3D, for training the base 4× tokenizer."},{"cited_title":"Vbench: Comprehensive benchmark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"Provides the VBench benchmark and its 16 quality dimensions used to evaluate text-to-video generation quality."},{"cited_title":"Extracting motion and appearance via inter-frame attention for efficient video frame interpolation","cited_arxiv_id":null,"evidence_quote":"Provides the external frame interpolation method used as the baseline for comparing progressive growing against subsampled encoding plus interpolation."},{"cited_title":"Mcl-jcv: a jnd-based h","cited_arxiv_id":null,"evidence_quote":"Supplies the MCL-JCV benchmark used for reconstruction quality comparisons."}],"review_version":1}