{"id":"96238bc2-2a3e-4fca-b139-5c7d19129280","arxiv_id":"2412.10443","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SweetTok compresses 17-frame 256x256 videos into 1,280 tokens via decoupled spatial-temporal query autoencoding and a parts-of-speech language codebook, reporting an rFVD of 20.46 on UCF-101 versus 35.15 for LARP.","lead":"This paper introduces a video tokenizer that compresses videos into a small set of spatial and temporal tokens with separate query compressors and a language-derived codebook. It reports large gains in video reconstruction and generation quality on UCF-101 over recent tokenizers, plus few-shot action recognition from the tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 42.8% rFVD and 15.1% gFVD gains compare SweetTok at 1,280 tokens against LARP at 1,024 tokens; no matched-token-count experiment is reported, so the headline improvements may be partly a token-budget effect.","rationale":"I read SweetTok as an empirical architecture claim: decoupled spatial/temporal query compression plus a language codebook improves video reconstruction and generation. The internal ablations (Tables 4-5) do show that decoupling and MLC help, and I do not dispute that the method achieves the reported numbers. My concern is that the headline comparisons do not isolate the architecture from token budget. The reader's weakest assumption about first-frame residual error propagation is a legitimate design risk, but it is a robustness/failure-mode concern rather than a direct challenge to the reported comparison; the paper could still be correct even if first-frame errors propagate, because it reports aggregate rFVD that may be dominated by frames where the first frame is reconstructed well. The token-count confound, by contrast, directly undercuts the central 42.8%/15.1% claims: if a matched-token LARP or a 1,024-token SweetTok closes the gap, the stated contribution loses its quantitative basis. This is a cheap, decisive experiment. Given that the concern is empirically addressable and not a demonstrated refutation, the reader's CONDITIONAL verdict is appropriate; I would make the matched-token-count comparison an explicit acceptance condition. I also note the MLC vocabulary is mined from captions of the evaluation dataset and reported without error bars, which reinforces the need for the conditional verdict.","tokens_in":15141,"tokens_out":13572,"duration_ms":149938,"concrete_test":"Run a matched-token-count comparison on UCF-101: train or fine-tune LARP with 1,280 tokens and/or train SweetTok with 1,024 total tokens (e.g., set Ltemporal to 768 while keeping Lspatial at 256), then recompute rFVD and gFVD under the same training recipe. If the SweetTok advantage shrinks by more than half (below roughly 7 rFVD points or 7 gFVD points) at matched token counts, the headline gains are substantially token-budget effects rather than evidence for DQAE/MLC. Report a small token-count sweep (e.g., 1,024 and 1,280) for both methods.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SweetTok's central numerical claims are computed across different token budgets. Table 1 reports rFVD 20.46 for SweetTok at 1,280 tokens (256 spatial + 1,024 temporal, Section 3.2.2) versus rFVD 35.15 for LARP-L at 1,024 tokens. Table 2 reports gFVD 84 for SweetTok at 1,280 tokens with a 650M generator versus gFVD 99 for LARP-L at 1,024 tokens with a 632M generator. The paper calls these token counts 'comparable' or 'similar,' but 1,280 is 25% larger than 1,024, and reconstruction/generation FVD is sensitive to token count. The same confound appears in the key ablation: Table 4 compares 'Vanilla Query-based (LARP)' at 1,024 tokens with 'Decoupled Query-based (DQAE)' at 1,280 tokens, so architecture and token budget change simultaneously. There is no reported row with DQAE at 1,024 total tokens or LARP at 1,280 tokens. Consequently, the abstract's 42.8% and 15.1% improvements cannot yet be attributed to the decoupled query design; they may be partly due to the extra 256 tokens.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SweetTok, a video tokenizer that splits compression into a spatial branch handling the first frame and a temporal branch handling frame-wise residuals, using decoupled query-based autoencoders (DQAE). A motion-enhanced language codebook (MLC) quantizes spatial tokens with nouns/adjectives and temporal tokens with verbs/adverbs. The authors report a UCF-101 reconstruction rFVD of 20.46 at 1,280 tokens, a 15.1% gFVD improvement over LARP in class-conditional generation, strong image reconstruction results after fine-tuning on ImageNet, and few-shot visual classification results through an LLM-based prompt pipeline.","tokens_in":15318,"tokens_out":5478,"duration_ms":48748,"significance":"If the headline gains survive a matched token-budget comparison, SweetTok's decoupled query design is a genuinely useful step toward compact, high-fidelity video discretization, and the language-codebook interface for few-shot understanding is an appealing additional property. The paper provides an internal ablation chain (Tables 4 and 5) and compares against external baselines (OmniTok, LARP, MAGVIT), which makes the claims testable rather than circular. The weaknesses are experimental: the main comparisons confound architecture with token count, and the central residual decomposition is not stress-tested for error propagation. With additional matched-token experiments and first-frame error analysis, the work could become a solid contribution to the video tokenization literature.","major_comments":[{"comment":"The headline improvements are computed across different token budgets: SweetTok uses 1,280 tokens (256 spatial + 1,024 temporal) while LARP-L uses 1,024 tokens in Table 1, and the ablation in Table 4 compares 'Vanilla Query-based (LARP)' at 1,024 tokens with 'Decoupled Query-based (DQAE)' at 1,280 tokens. Since rFVD and gFVD are sensitive to token count, the 42.8% (or 41.8% when computed from Table 4 as (35.15-20.46)/35.15) and 15.1% improvements cannot be attributed to the decoupled query design without a matched-token-count experiment. Please add a DQAE row at 1,024 total tokens (e.g., 256 spatial + 768 temporal) and/or a LARP row at 1,280 tokens, and report gFVD for the same matched conditions.","section":"§4.2, Table 1, Table 4"},{"comment":"The residual decomposition assumes that the first frame contains essentially all static spatial content and that frame-wise residuals to the first frame capture all motion information. Because Eq. (7) tiles the decoded first frame T times before adding residuals, any reconstruction error in frame 1 is replicated into all frames before the temporal decoder can correct it. The paper reports no first-frame reconstruction error versus full-video error, no sensitivity to the choice of reference frame (e.g., middle frame), and no comparison against encoding residuals from later frames. Without such measurements, the error-propagation risk of this design is unquantified; please provide these analyses or justify the first-frame choice more directly.","section":"§3.2.2, Eqs. (4)-(7)"},{"comment":"The ablation chain is internally inconsistent: Table 4 reports rFVD 35.15 for 'Vanilla Query-based (LARP)' while Table 5 reports rFVD 29.45 for 'Baseline (w/o LC)'. It is unclear whether the Table 5 baseline is the same LARP-style model, a DQAE without any language codebook, or something else, and Table 5 does not list token counts. Since the MLC contribution is a main contribution, please reconcile the baselines across the two tables and report token counts and architecture for every ablation row.","section":"§4.4, Tables 4 and 5"}],"minor_comments":[{"comment":"The abstract's 42.8% improvement does not match the precise numbers in Table 4: (35.15 - 20.46)/35.15 = 41.8%. Please unify the percentage claim with the reported rFVD values.","section":"Abstract, §4.2"},{"comment":"On Kinetics-600, SweetTok's rFVD of 25 is worse than LARP-L's 23, although the text says the results are 'comparable'. Please acknowledge this explicitly and discuss the discrepancy, since the main text elsewhere emphasizes improvements.","section":"Table 1, §4.2"},{"comment":"The description of the vocabulary extraction and the graph construction is thin: please specify how candidate vocabularies are extracted from video captions, how the 5-token co-occurrence window is used to build graph edges, and why 21 tokens are selected for the few-shot prompt.","section":"§3.2.3, §4.5"},{"comment":"There are several typos and inconsistent names: 'pre-traiend' (Introduction), 'reconsturct' (§3.2.2), 'LlammaGen' (Table 3 text), 'similary' (§4.4), and 'grean' (supplementary Figure 9 caption). Please proofread the manuscript.","section":"Throughout"},{"comment":"The training hyperparameters are useful, but the paper does not state whether model weights or code will be released; please include an availability statement.","section":"Supplementary, Table 9"}],"recommendation":"major_revision","confidential_remarks":"The paper's main baselines are outside the author group and the internal ablations are informative. The primary issue is experimental design: the headline numbers compare different token budgets, which is directly fixable with additional matched-token rows. The second issue is the unmeasured error propagation in the first-frame residual design. A revision that adds the missing experiments and reconciles Tables 4 and 5 would make the paper publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper is not a rehash: it separates video compression into first-frame spatial queries and residual temporal queries, and splits the language codebook by part of speech (nouns/adjectives for appearance, verbs/adverbs for motion). That combination is new for video tokenizers, and the ablations give it real support. Table 4 shows DQAE at 1,280 tokens drops rFVD from 35.15 to 20.46 relative to LARP-style flattening at 1,024; Table 5 shows MLC adds another 4.34 points. Those internal comparisons are the paper's best evidence.\n\nThe soft spots are mostly around the headline claims. The 42.8% improvement in the abstract is computed against LARP at 1,024 tokens while SweetTok uses 1,280—25% more. There is no row with matched token counts, so we don't know how much of the gain is the architecture and how much is extra budget. The same confound appears in Table 4. On K-600, SweetTok's rFVD is 25 versus LARP-L's 23, so the 'superior fidelity' story is not uniform. Also, the abstract's 42.8% is 41.8% by the paper's own table; minor, but sloppy. The residual decomposition—first frame plus frame-wise differences—assumes the first frame carries all static content; errors there propagate to every reconstructed frame. The paper doesn't measure first-frame reconstruction error or test sensitivity to which frame is the reference. The MLC vocabulary is extracted from captions of the evaluation dataset itself, which is a mild selection bias. And no code is released.\n\nThe limitations section is honest: they concede the tokens don't support precise semantics like VQA. That tempers the 'language-aligned' marketing.\n\nNet: the architecture is worth taking seriously, and the internal ablations are the right kind of evidence. But the central quantitative claim is not precisely supported as stated. A referee should ask for a matched-token comparison (DQAE at 1,024 and/or LARP at 1,280), K-600 details, and first-frame error analysis. These are addressable. I'd send it to review, conditional on those experiments.","headline":"Genuinely new decoupled spatial/temporal video tokenizer with supportive ablations, but the headline gains are confounded by a token-count mismatch and K-600 results are worse—worth refereeing with a demand for matched-token experiments.","tokens_in":16042,"tokens_out":2509,"would_cite":true,"duration_ms":24295,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SweetTok claims that a video tokenizer which decouples appearance from motion can compress a video to 1,280 tokens while improving reconstruction fidelity by 42.8% and generation quality by 15.1% on UCF-101.","keywords":["video tokenization","vector quantization","spatial-temporal decoupling","language codebook","video reconstruction","video generation","few-shot action recognition","discrete visual tokens"],"falsifier":"Run a trained SweetTok on UCF-101 but move the spatial reference from frame 1 to a later frame, say frame 9, treating the original frames as residuals around that new anchor; if rFVD stays near 20.46, the first-frame residual decomposition is not the mechanism driving the reported gains, whereas a large degradation would confirm that the choice of anchor is load-bearing.","tokens_in":14789,"feed_emoji":"🎬","tokens_out":9088,"duration_ms":86942,"temperature":0.7,"pith_summary":"This paper sets out to show that a video tokenizer can compress aggressively and still reconstruct better if it separates appearance from motion instead of flattening all video patches into one sequence. SweetTok uses a Decoupled Query AutoEncoder: spatial tokens are learned from the first frame, temporal tokens from frame-wise residuals, and the video is rebuilt by tiling the decoded first frame and adding residual information back. On UCF-101 at 256x256, the paper reports an rFVD of 20.46 with 1,280 tokens, a 42.8% improvement over the 1,024-token LARP baseline, and a gFVD of 84 versus 99 in class-conditional generation. Because the codebook is built from language words split by part of speech, the same tokens also feed few-shot image and video recognition through a language model, so the paper's larger claim is that one compact discrete representation can serve generation and understanding simultaneously.","feed_headline":"One tokenizer gets 4x fewer tokens and 43% better video fidelity","feed_subtitle":"Appearance and motion get separate codebooks, so one compact token set improves both reconstruction and generation.","key_machinery":"The load-bearing object is the Decoupled Query AutoEncoder (DQAE), a pair of transformer branches in which learnable query tokens cross-attend into patch features: the spatial branch reads only the first frame's patches and the temporal branch reads only the frame-wise residual patches. Reconstruction is staged in the same order, so the model never has to decode appearance and motion from a single mixed sequence. The second load-bearing object is the Motion-enhanced Language Codebook (MLC), a quantizer whose codebook entries are text embeddings from a pretrained vision-language model, projected into the visual latent space by a graph convolution network and split by part of speech: spatial queries search nouns and adjectives, temporal queries search verbs and adverbs. These two mechanisms carry the argument because the first sets the compression structure and the second sets what information survives quantization, and together they make the token sequence both compact and semantically addressable.","core_discovery":"SweetTok's central claim is that the previous query-based tokenization loses fidelity because it couples spatial and temporal compression in one flattened sequence, forcing the model to learn intertwined redundancies at once. The paper's answer is a decoupled autoencoder with two branches: one compresses only the first frame into 256 spatial queries, the other compresses frame-wise residuals into 1,024 temporal queries, and reconstruction is ordered so the decoded first frame is tiled and then refined by the temporal decoder. A second claim is that the quantizer gains from being motion-aware: a language codebook partitioned into nouns and adjectives for spatial tokens and verbs and adverbs for temporal tokens lowers rFVD from 24.80 to 20.46 in their ablation, whereas a flat language codebook only reaches 24.80. The paper concludes that decoupling plus motion-sensitive semantic quantization delivers both higher compression and higher reconstruction fidelity than vanilla patch tokenizers or query-based tokenizers, and that the resulting tokens carry semantic content usable by language models.","pith_inferences":["A natural test the paper leaves open is reference-frame choice: using a middle frame or a scene-change frame as the spatial anchor could reduce first-frame error propagation in long videos.","The decoupled branches suggest modular training: one could upgrade the temporal branch or pretrain it on larger motion datasets without retraining the spatial branch, which the paper does not explore.","The paper's limitation section says the codebook is unsupervised and lacks visual-question-answering-grade semantic alignment; adding contrastive learning between video features and codebook text is a direct next step it names.","If the part-of-speech split is the active ingredient, then synthetic captions or action labels mined from datasets without captions should produce a similar gain, which would separate the split itself from caption availability."],"forward_implications":["With 1,280 tokens, SweetTok reports rFVD 20.46 on UCF-101 versus 35.15 for LARP at 1,024 tokens, so a decoupled design can improve fidelity while using a similar or smaller token budget.","Class-conditional video generation improves with the same tokens: gFVD 84 versus 99 for LARP at comparable generator size, and 65 when the generator is scaled to 1.9B parameters.","The spatial branch transfers to images: fine-tuning on ImageNet gives rFID 0.37 at 1,024 tokens, so the decoupling benefit is not video-specific.","Because tokens map to words, few-shot classification works with a language model: 90.8% on miniImageNet and 90.1% on UCF-101 under the paper's protocols.","The verb and adverb half of the codebook is what closes most of the remaining gap, meaning motion semantics are a distinct and recoverable component of video compression."],"supporting_citations":[{"why":"LARP is the flattened query-based tokenizer baseline; SweetTok's 42.8% rFVD and 15.1% gFVD improvements are measured against it.","marker":"[39]"},{"why":"OmniTok is the vanilla video tokenizer baseline and supplies the UCF-101 and K-600 training and evaluation protocol SweetTok follows.","marker":"[41]"},{"why":"TiTok contributes the 1D sequence token compression idea and the proxy-code training trick that DQAE's query design and training schedule adapt.","marker":"[51]"},{"why":"The cross-attention adaptive query module from this work is the basis of DQAE's learnable spatial and temporal queries.","marker":"[22]"},{"why":"LQAE introduces replacing a learned visual codebook with frozen word embeddings, the idea MLC extends to video.","marker":"[24]"},{"why":"This work supplies the language-codebook and frozen-LLM evaluation pipeline used for few-shot classification with SweetTok tokens.","marker":"[49]"},{"why":"This work transfers part-of-speech codebooks for image modeling, the direct precedent for MLC's noun and adjective versus verb and adverb split.","marker":"[52]"},{"why":"VideoGPT is the autoregressive generator used to measure downstream class-conditional video generation with SweetTok tokens.","marker":"[44]"}],"fun_headline_variants":["Decoupled spatial-temporal tokenizer cuts video tokens, boosts fidelity 43%","Separate codebooks for appearance and motion yield 43% better video tokens","SweetTok: decoupled queries compress video tokens 4x with 43% fidelity gain","Video tokenizer with decoupled spatial-temporal queries improves fidelity by 43%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the first frame contains essentially all static appearance and every later frame can be represented as that first frame plus frame-wise residuals; if a later frame introduces appearance the first frame lacks, or if first-frame reconstruction errors are large, those errors are replicated into all frames before the temporal decoder can correct them.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled spatial-temporal tokenizer cuts video tokens, boosts fidelity 43%","Separate codebooks for appearance and motion yield 43% better video tokens","SweetTok: decoupled queries compress video tokens 4x with 43% fidelity gain","Video tokenizer with decoupled spatial-temporal queries improves fidelity by 43%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001115,"raw_usage":{"total_tokens":4674,"prompt_tokens":1005,"completion_tokens":3669,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":3583}},"tokens_in":621,"tokens_out":3669,"duration_ms":26631,"temperature":1.0,"reasoning_tokens":3583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:53:06.268226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a trained SweetTok on UCF-101 but move the spatial reference from frame 1 to a later frame, say frame 9, treating the original frames as residuals around that new anchor; if rFVD stays near 20.46, the first-frame residual decomposition is not the mechanism driving the reported gains, whereas a large degradation would confirm that the choice of anchor is load-bearing.","supporting_citations":[{"cited_title":"Omnitokenizer: A joint image- video tokenizer for visual generation","cited_arxiv_id":null,"evidence_quote":"OmniTok is the vanilla video tokenizer baseline and supplies the UCF-101 and K-600 training and evaluation protocol SweetTok follows."},{"cited_title":"An image is worth 32 tokens for reconstruction and generation","cited_arxiv_id":null,"evidence_quote":"TiTok contributes the 1D sequence token compression idea and the proxy-code training trick that DQAE's query design and training schedule adapt."},{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"The cross-attention adaptive query module from this work is the basis of DQAE's learnable spatial and temporal queries."},{"cited_title":"Language quan- tized autoencoders: Towards unsupervised text-image align- ment","cited_arxiv_id":null,"evidence_quote":"LQAE introduces replacing a learned visual codebook with frozen word embeddings, the idea MLC extends to video."},{"cited_title":"Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms","cited_arxiv_id":null,"evidence_quote":"This work supplies the language-codebook and frozen-LLM evaluation pipeline used for few-shot classification with SweetTok tokens."},{"cited_title":"Codebook transfer with part-of-speech for vector-quantized image modeling","cited_arxiv_id":null,"evidence_quote":"This work transfers part-of-speech codebooks for image modeling, the direct precedent for MLC's noun and adjective versus verb and adverb split."}],"review_version":1}