{"id":"f042d212-f7ca-4599-8e19-6f8c67a98e9c","arxiv_id":"2411.14762","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CoordTok encodes a 128-frame video into three 2D triplane latents (1280 tokens total) and reconstructs randomly sampled patch coordinates, enabling efficient long-video tokenization and 128-frame generation.","lead":"CoordTok is a video tokenizer that turns a 128-frame clip into 1280 tokens by storing it as three 2D planes and reconstructing only randomly sampled patches during training. This lets large tokenizers train directly on long videos, and the authors use it to train a diffusion transformer that generates 128 frames at once.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the abstract: baselines at 6144/8192 tokens have better rFVD (66.5/49.5) than CoordTok at 1280 (102.9), so 'similar reconstruction quality' does not hold on the paper's own primary metric.","rationale":"In good faith, the paper presents a genuinely useful tokenizer design: the coordinate-sampled decoder is a plausible way to train on long videos with bounded memory, and the generation results (FVD 369.3) are competitive. The central claim, however, is the token-efficiency comparison stated in the abstract. My concern is narrower and more direct than the fairness asymmetry emphasized in the reader's weakest_assumption: the paper's own Table 1 shows that the two baselines cited in the abstract achieve better rFVD at 6144/8192 tokens than CoordTok achieves at 1280 tokens. Since rFVD is the metric used in Figure 4, which is the paper's evidence for token efficiency, the abstract's phrase 'similar reconstruction quality' is not supported by the paper's own numbers. This is an internal inconsistency, not a disagreement with field consensus. The reader did flag the rFVD problem in the rationale, but their designated weakest assumption was the training-setup asymmetry; I see the direct metric contradiction as the more load-bearing issue. A conditional verdict remains appropriate: the method and its smaller-token-count improvements appear real, but the headline claim needs to be reworded or supported by a metric where the comparison holds. Thus I do not move the verdict; I keep the reader's CONDITIONAL, with the condition being a corrected, metric-aware statement of the token-efficiency claim.","tokens_in":18950,"tokens_out":3894,"duration_ms":35458,"concrete_test":"Recompute rFVD on the same 10,000 128-frame UCF-101 clips for CoordTok-1280, PVDM-AE-6144, and OmniTokenizer-CV-8192. If the baseline rFVDs remain near 66.5 and 49.5 while CoordTok is 102.9, the abstract's 'similar reconstruction quality' claim is false on rFVD, and the paper must be revised to restrict the claim to equal-token-count comparisons or to PSNR/LPIPS with an explicit tolerance. As a secondary check, inspect whether 'similar' is defined anywhere with a quantitative bound; currently no such definition appears.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—'baselines need 6144 or 8192 tokens to achieve similar reconstruction quality'—is contradicted by the paper's own Table 1 on the rFVD metric. CoordTok at 1280 tokens obtains rFVD 102.9, while PVDM-AE at 6144 tokens and OmniTokenizer-CV at 8192 tokens obtain 66.5 and 49.5, respectively (lower is better). On the metric plotted in Figure 4, the baselines do not just match CoordTok; they substantially outperform it at the token counts the abstract says they 'need' merely to reach similar quality. The claim is only defensible on PSNR/LPIPS, and even there SSIM is mixed (0.892 for CoordTok vs. 0.913 for OmniTokenizer-CV, with lower rFVD for the baseline). Because the headline contribution is token efficiency, this internal inconsistency is load-bearing: it is not a question of external baseline training fairness, but of the paper's own numbers failing to support the stated comparison. The paper never defines 'similar reconstruction quality,' and Figure 4 plots CoordTok's 102.9 point against baseline curves at smaller token counts, visually implying a crossover that Table 1 shows does not occur on rFVD. The token-efficiency story may still hold when comparing at equal token counts or on perceptual metrics, but as written the abstract overstates the result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoordTok, a video tokenizer that encodes a video into factorized triplane representations and reconstructs only patches at randomly sampled (x,y,t) coordinates during training, allowing direct training on 128-frame clips at reduced memory cost. On UCF-101, CoordTok encodes a 128-frame 128x128 video into 1280 tokens and reports strong reconstruction metrics, and a downstream diffusion transformer (CoordTok-SiT-L/2) generates 128 frames at once with FVD 369.3, outperforming published baselines. The central claim is that prior tokenizers need 6144 or 8192 tokens to reach similar reconstruction quality, demonstrating the benefit of long-video training.","tokens_in":19281,"tokens_out":5798,"duration_ms":50944,"significance":"The method is a clean and plausible contribution to video tokenization: it adapts coordinate-based decoding from 3D generation to video tokenizer training, and the reported GPU-memory and FLOPs savings are concrete. The generation results (FVD 369.3 on 128-frame UCF-101) are competitive and the illustration of temporal-consistency advantages over stitched baselines is suggestive. However, the headline token-efficiency claim is currently supported only on some metrics and is confounded by the comparison protocol, so the significance is diminished until those issues are addressed.","major_comments":[{"comment":"The claim that 'baselines need 6144 or 8192 tokens to achieve similar reconstruction quality' is not supported on the paper's own rFVD metric. In Table 1, PVDM-AE with 6144 tokens achieves rFVD 66.5 and OmniTokenizer-CV with 8192 tokens achieves rFVD 49.5, both substantially better (lower) than CoordTok's 102.9 at 1280 tokens. The paper does not define 'similar reconstruction quality,' and Figure 4 plots CoordTok against baseline curves at smaller token counts, which visually obscures the crossover that Table 1 shows on rFVD. Because token efficiency is the headline contribution, this overstatement is load-bearing. The abstract should be revised to qualify which metrics support the claim (e.g., PSNR/LPIPS) or to compare at matched token budgets.","section":"Abstract and Section 3.2, Table 1, Figure 4"},{"comment":"The baseline comparison confounds model architecture with training clip length. Baselines are trained on 16-frame clips and evaluated by stitching independent reconstructions, while CoordTok is trained on 128-frame clips. The paper states that this demonstrates the benefit of long-video training, but no experiment holds training length fixed: there is no CoordTok variant trained on 16-frame clips, nor a 3D VAE baseline trained on 128-frame clips. Without such a control, the observed token efficiency could be attributed to the longer training context rather than to the coordinate-based decoder design. Please add a controlled comparison or substantially temper the causal claim.","section":"Section 3.2, experimental setup"},{"comment":"The multi-metric results disagree in direction, so the phrase 'similar reconstruction quality' is too strong without designating a primary metric. On PSNR CoordTok (28.6) slightly exceeds OmniTokenizer-CV (28.3); on LPIPS it is better (0.066 vs 0.081); on SSIM it is worse (0.892 vs 0.913); and on rFVD it is much worse (102.9 vs 49.5). The paper should either define a single primary fidelity metric or present a trade-off analysis that makes the metric-dependent conclusion explicit rather than asserting an unqualified similarity.","section":"Table 1 and Section 3.2, 'similar reconstruction quality'"}],"minor_comments":[{"comment":"The sentence 'PVDM-AE achieves >1000 rFVD score when using 1152 tokens' selects the most favorable comparison point for CoordTok; the reader should also be reminded that PVDM-AE at 6144 tokens achieves rFVD 66.5, much better than CoordTok's 102.9.","section":"Section 3.2, Results paragraph"},{"comment":"The x-axis labels (1024, 1280, 2560, 6144, 8192) are visually crowded and the CoordTok point is not labelled with its rFVD value; consider a log-scale x-axis and explicit annotation of the 1280-token comparison.","section":"Figure 4"},{"comment":"The relationship between the sampling ratio (e.g., 3%) and the number N of sampled coordinates (1024 for main training, 4096 for fine-tuning) is not stated in the main text; please clarify this explicitly.","section":"Section 2.2 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty is moderate: triplane representations and coordinate-based reconstruction are established in 3D generation, and the main new element is applying them to video tokenizer training. The rFVD inconsistency in Table 1 is a serious internal contradiction with the abstract, and the comparison confound (training length vs architecture) needs to be addressed. However, the method and generation results are promising, and the issues are fixable with a revised claim and additional controlled experiments or discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoordTok is a real contribution: a video tokenizer that combines triplane latents with coordinate-based patch reconstruction, trained on 128-frame clips. The design is clean, the memory/FLOPs curves are compelling, and the ablations (sampling ratio, triplane dimensions, LPIPS fine-tuning) are thorough. The 128-frame generation result (FVD 369.3, faster than baselines) is concrete evidence that few-token long-video latents can feed a diffusion transformer. The authors also honestly flag that the tokenizer struggles with dynamic videos.\n\nThe problem is the abstract's headline claim. It says baselines need 6144 or 8192 tokens 'to achieve similar reconstruction quality.' Table 1 contradicts that on rFVD: CoordTok at 1280 tokens gets 102.9, while PVDM-AE at 6144 gets 66.5 and OmniTokenizer-CV at 8192 gets 49.5. On rFVD, the baselines don't need those tokens to match CoordTok; they need them to beat it. The claim holds on PSNR and LPIPS (CoordTok is best among the UCF-trained, non-Cosmos methods) but not SSIM or rFVD. The paper never defines 'similar reconstruction quality,' and Figure 4 plots rFVD without showing the high-token baseline points, which visually implies a crossover that doesn't exist. This is load-bearing, because token efficiency is the whole point. It is fixable: report all four metrics at matched token counts, and either restrict the claim to PSNR/LPIPS or explain the rFVD tradeoff.\n\nThe second soft spot is the causal story. The paper says training on long videos exploits temporal coherence and that's why CoordTok is more token-efficient. But the comparison confounds clip length with architecture and training objective. Baselines are short-clip models stitched at evaluation; there's no CoordTok trained on 16-frame clips. The token savings could come from the coordinate decoder objective alone. A 16-frame vs 128-frame ablation of CoordTok would settle it.\n\nThe baselines are reasonable and mostly trained on UCF-101; the stitch-based evaluation is a practical compromise, though it may penalize temporal consistency. The citation pattern is fine; the prior triplane/INR work is relevant.\n\nI'd take this paper. The core design works and the experiments are mostly solid. The overclaim is real but not fatal. Ask for a revision that tones down the abstract, adds matched-token-count tables, and includes the long-vs-short training ablation. A serious referee can get this into good shape.","headline":"CoordTok is a genuinely useful tokenizer design, but the abstract's 'similar reconstruction quality' claim is contradicted by its own rFVD numbers.","tokens_in":19812,"tokens_out":4027,"would_cite":true,"duration_ms":37816,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video tokenizer that learns to reconstruct patches from sampled space-time coordinates can encode a 128-frame video in 1,280 tokens, where prior tokenizers need 6,144–8,192 tokens for comparable quality.","keywords":["video tokenization","triplane representation","coordinate-based reconstruction","long video generation","latent diffusion transformer","temporal coherence","UCF-101","token efficiency"],"falsifier":"Train a strong baseline such as PVDM-AE or TATS-AE from scratch on 128-frame clips, with compute comparable to CoordTok's, and evaluate reconstructed 128-frame videos; if the 6,144–8,192-token gap shrinks or disappears, the efficiency claim comes from long-video training rather than from the coordinate-based decoder. Alternatively, train CoordTok on 16-frame clips and evaluate on 128-frame videos: if it then needs as many tokens as the baselines, the paper's central attribution to temporal coherence fails.","tokens_in":18753,"feed_emoji":"🎞️","tokens_out":4747,"duration_ms":42984,"temperature":0.7,"pith_summary":"CoordTok is a video tokenizer built on the claim that training directly on long video clips unlocks temporal coherence, letting one 128-frame, 128x128 video be encoded into 1,280 tokens while earlier tokenizers need 6,144 or 8,192 tokens for comparable reconstruction. The design that makes long-clip training affordable is a decoder that reconstructs only patches at randomly sampled $(x,y,t)$ coordinates from factorized triplane latents, instead of reconstructing every frame at once. If the claim holds, video tokenization stops being the bottleneck that forces generative models to work in short clips, and long-video generation can be done in a single pass.","feed_headline":"One tokenizer fits a 128-frame video in 1,280 tokens","feed_subtitle":"Existing tokenizers need 6,144–8,192 tokens for similar quality, making one-pass long-video generation practical.","key_machinery":"The load-bearing mechanism is coordinate-based patch reconstruction over factorized triplane latents. The encoder maps a video to three 2D planes (content plane $z_{xy}$ plus two motion planes $z_{yt}$, $z_{xt}$) rather than a 3D latent; the decoder samples $N$ patch coordinates $(i,j,k)$, reads each coordinate from the planes via bilinear interpolation, lets the sampled features attend to each other through self-attention, and projects them to the RGB patches. Because only 3% of patches are reconstructed per training step, training cost no longer grows linearly with video length, which is what makes 128-frame training tractable.","core_discovery":"The paper's central discovery is that a video tokenizer trained end-to-end on 128-frame clips can exploit temporal coherence to encode long videos far more compactly than tokenizers trained on 16-frame clips and stitched together. CoordTok encodes an input video into three 2D latent planes, $z_{xy}$, $z_{yt}$, $z_{xt}$, and trains a transformer decoder to map sampled space-time coordinates to the pixels of the corresponding patches using bilinear interpolation of the planes. On UCF-101, 1,280 tokens reconstruct a 128-frame video at rFVD 102.9, a quality PVDM-AE cannot reach with 1,152 tokens (rFVD > 1,000) and that other baselines reach only with 6,144–8,192 tokens. A flow-based diffusion transformer trained on these 1,280-token latents generates 128 frames at once with FVD 369.3, outperforming prior baselines, and does so faster than iterative approaches.","pith_inferences":["A testable extension of the paper's logic is that training on 256- or 512-frame clips should keep training cost nearly flat while further exploiting temporal redundancy, since only sampled coordinates are decoded; this is the authors' stated scaling ambition but not demonstrated here.","The strong correlation between reconstruction error and video dynamics (-0.87) suggests the triplane content/motion split is the bottleneck for fast motion, so an adaptive scheme that allocates more temporal planes to dynamic clips could recover quality where CoordTok currently struggles.","The FVD gain from 1,280 over 3,072 tokens, despite near-equal reconstruction quality, implies that generative-model gains come partly from sequence-length reduction; this predicts similar gains for other latent diffusion transformers trained on CoordTok-style latents."],"forward_implications":["Video tokenizers can be trained on clips of 128 frames or longer without memory and FLOP costs growing linearly with clip length, because only sampled coordinates are reconstructed.","Long videos can be represented with roughly a fifth to a sixth of the tokens earlier tokenizers need at comparable quality, shrinking the sequence length seen by generative models.","A diffusion transformer can generate 128 frames in one pass rather than iteratively, with lower synthesis time and competitive memory.","Because token count and reconstruction quality are only loosely coupled above a threshold, generative quality can improve even when reconstruction quality is unchanged."],"supporting_citations":[{"why":"PVDM-AE shares the factorized triplane latent structure with CoordTok, isolating the decoder design and serving as the 6,144-token baseline.","marker":"[66]"},{"why":"TATS-AE is a 3D-latent video tokenizer trained on 16 frames, used as a baseline for token-count comparison.","marker":"[11]"},{"why":"LARP is a 1D autoregressive tokenizer baseline compared at both 8,192 and 1,024 tokens.","marker":"[52]"},{"why":"OmniTokenizer provides discrete and continuous baseline tokenizers compared at 8,192 and 1,024 tokens.","marker":"[53]"},{"why":"MAGVIT supplies the rFVD, PSNR, LPIPS, and SSIM evaluation setup for reconstruction quality.","marker":"[63]"},{"why":"SiT is the scalable interpolant transformer used as the generation backbone for CoordTok-SiT-L/2.","marker":"[27]"},{"why":"This work's coordinate-querying triplane idea is the source of CoordTok's decoder formulation.","marker":"[22]"},{"why":"The memory-efficient learnable-embeddings-to-triplane encoder design is adopted for CoordTok's encoder.","marker":"[18]"}],"fun_headline_variants":["CoordTok: 1,280 tokens for 128-frame videos","128-frame video tokenized in just 1,280 tokens","Tokenize long videos with 5x fewer tokens","Long-video tokenizer needs only 1,280 tokens","5x fewer tokens for long video encoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that training CoordTok from scratch on 128-frame clips against baselines trained on 16-frame clips or released weights, evaluated by stitching independent reconstructions, is a fair measure of token efficiency: if the baselines are under-trained or unfairly penalized by stitching, the token-count advantage is not established.","fun_headline_variants_meta":{"raw":{"variants":["CoordTok: 1,280 tokens for 128-frame videos","128-frame video tokenized in just 1,280 tokens","Tokenize long videos with 5x fewer tokens","Long-video tokenizer needs only 1,280 tokens","5x fewer tokens for long video encoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001137,"raw_usage":{"total_tokens":4743,"prompt_tokens":989,"completion_tokens":3754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":3673}},"tokens_in":605,"tokens_out":3754,"duration_ms":26741,"temperature":1.0,"reasoning_tokens":3673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:56:21.624461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a strong baseline such as PVDM-AE or TATS-AE from scratch on 128-frame clips, with compute comparable to CoordTok's, and evaluate reconstructed 128-frame videos; if the 6,144–8,192-token gap shrinks or disappears, the efficiency claim comes from long-video training rather than from the coordinate-based decoder. Alternatively, train CoordTok on 16-frame clips and evaluate on 128-frame videos: if it then needs as many tokens as the baselines, the paper's central attribution to temporal coherence fails.","supporting_citations":[{"cited_title":"Video probabilistic diffusion models in projected latent space","cited_arxiv_id":null,"evidence_quote":"PVDM-AE shares the factorized triplane latent structure with CoordTok, isolating the decoder design and serving as the 6,144-token baseline."},{"cited_title":"Long video generation with time-agnostic vqgan and time- sensitive transformer","cited_arxiv_id":null,"evidence_quote":"TATS-AE is a 3D-latent video tokenizer trained on 16 frames, used as a baseline for token-count comparison."},{"cited_title":"OmniTokenizer: A joint image- video tokenizer for visual generation","cited_arxiv_id":null,"evidence_quote":"OmniTokenizer provides discrete and continuous baseline tokenizers compared at 8,192 and 1,024 tokens."},{"cited_title":"Magvit: Masked generative video transformer","cited_arxiv_id":null,"evidence_quote":"MAGVIT supplies the rFVD, PSNR, LPIPS, and SSIM evaluation setup for reconstruction quality."},{"cited_title":"SiT: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers","cited_arxiv_id":null,"evidence_quote":"SiT is the scalable interpolant transformer used as the generation backbone for CoordTok-SiT-L/2."},{"cited_title":"Scal- able neural video representations with learnable positional features","cited_arxiv_id":null,"evidence_quote":"This work's coordinate-querying triplane idea is the source of CoordTok's decoder formulation."},{"cited_title":"LRM: Large reconstruction model for single image to 3D","cited_arxiv_id":null,"evidence_quote":"The memory-efficient learnable-embeddings-to-triplane encoder design is adopted for CoordTok's encoder."}],"review_version":1}