{"id":"2974ed97-2a5a-4c6a-9633-dd6a57e5801a","arxiv_id":"2608.05798","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"KVAE introduces image, video, and full-band audio tokenizers whose reconstruction and downstream generation quality is competitive with, and often better than, current open-source tokenizers in head-to-head tests.","lead":"This paper releases a family of open-source autoencoder tokenizers for video, image, and audio, including a full-band 48 kHz audio model with a 50 Hz latent. The authors report that these tokenizers match or beat several frontier open-source tokenizers on reconstruction and text-to-media generation, and they share training details and ablations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The controlled tokenizer swap is not neutral across latent geometries: a fixed downstream DiT sees widely different token counts and channel dims, so the reported cross-tokenizer rankings may be pipeline-specific.","rationale":"I read the paper as an empirical engineering report whose central claim is that the released KVAE tokenizers are competitive drop-in components for latent diffusion. That claim is much stronger than saying the tokenizers work in the authors' pipeline, and the only evidence for the general form is the controlled tokenizer swap. I looked for the point where that swap could break. The specific failure is quantitative: a fixed DiT operated at a 50 Hz latent frame rate processes roughly 4.6x more tokens per second than one operated at SAME-L's 10.8 Hz, and attention cost scales with sequence length. Without a statement of how batch composition was equalized, the comparison does not isolate the latent space. This is an internal-control concern, not a disagreement with field consensus, and the authors' own Sec. 6.7 qualification that the 64-channel audio optimum is for a 0.6B generator makes the scale dependence explicit. I credit the released code, weights, and external reconstruction benchmarks as real evidence, but the headline generation claim should not be accepted as a general property until ranking robustness across generator scale and token-rate normalization is demonstrated. Because the reader already chose CONDITIONAL, my read does not move the verdict; it sharpens the condition that should be tested.","tokens_in":26149,"tokens_out":7014,"duration_ms":75769,"concrete_test":"Re-run the Sec. 6.6 audio tokenizer swap with a second downstream generator scale (for example a 2B DiT) and with per-second token budget normalized across tokenizers (for example patch or stride higher-rate latents to a common roughly 12.5 Hz frame rate), keeping data, captions, optimizer steps, and batch seconds fixed. If KVAE-Audio's advantage over MMAudio and SAME-L on AudioCaps FAD/CLAP and side-by-side win rates persists, the ranking is robust; if it shrinks or reverses, the headline comparison is pipeline-dependent and should be narrowed to the specific generator scale used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that KVAE matches or surpasses frontier tokenizers rests on the tokenizer-swap evaluations in Sec. 4.2 and Sec. 6.6. Those comparisons require the assertion that everything downstream of the tokenizer is fixed by construction, so differences in generation quality are attributable to the latent space. That premise fails once tokenizers differ in latent frame rate and channel count. In the audio swap, the same 0.6B DiT is trained on MMAudio (40 channels, 43.07 Hz), KVAE-Audio (64 channels, 50 Hz), DAC-VAE MovieGen (128 channels, 25 Hz), and SAME-L (256 channels, about 10.8 Hz). For one second of audio the transformer sees roughly 43, 50, 25, or 11 tokens, so per-example attention cost and effective capacity differ by up to about 4.6x. The paper does not state how batch size or sequence length were equalized, for example seconds-per-batch versus tokens-per-batch. The visual protocol balances spatial tokens by changing patch size, but channel counts still differ (16 vs 64) and only one 2B generator scale is used. The paper itself shows the optimal audio channel count depends on generator scale in Sec. 6.7, so the reported ranking may be a property of the 0.6B or 2B pipeline rather than of the tokenizer. The broad drop-in claim is therefore not established by the presented experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces KVAE, a family of continuous-latent tokenizers for image, video, and audio, designed for text-conditioned latent diffusion. For video, it presents two causal 3D tokenizers (4x8x8/16ch and 4x16x16/64ch); for image, a 2D tokenizer (8x8/32ch); for audio, a 48 kHz full-band waveform tokenizer with a 50 Hz/64-channel latent. The paper reports reconstruction metrics (PSNR, LPIPS, SSIM, PESQ, spectral distances) and generation evaluations (FID, CLIP, CLAP, FAD, side-by-side preference) against open tokenizers from Wan, HunyuanVideo, FLUX, MovieGen, StableAudio/SAME, and MMAudio, claiming these results 'match or surpass' the baselines. It also proposes a correlation-decay-slope (CDS) diffusability diagnostic and describes ablations on channel count, normalization, decoder width, attention, and perceptual losses. The final section concludes that the released tokenizers are competitive drop-in components for latent diffusion systems.","tokens_in":26412,"tokens_out":5022,"duration_ms":49743,"significance":"If the findings hold, the KVAE tokenizers are a useful open contribution: they are released with code and weights, the comparisons are carried out under a fixed downstream generator with external baselines, and the paper is unusually candid about caveats (e.g., the 0.6B-scale dependence of the audio channel optimum in Sec. 6.7, and the structural insensitivity of FAD backbones to the band above 16 kHz in Sec. 6.6). The controlled tokenizer-swap protocol is a valuable methodological template, and the explicit distinction between reconstruction fidelity, diffusability, and generation quality is a strength. However, the broad competitive claim is only partially supported by the presented evidence: the tokenizer swaps are not neutral to latent frame rate, channel count, and generator scale, and the central audio generation comparison lacks uncertainty quantification and specification of the alignment regularizer. The manuscript is therefore a solid technical report whose central claim needs either additional experiments or careful reframing.","major_comments":[{"comment":"The audio tokenizer-swap comparison of Table 4 does not isolate the latent space as claimed. The fixed 0.6B DiT is trained on MMAudio (40 channels, 43.07 Hz), KVAE-Audio (64 channels, 50 Hz), DAC-VAE MovieGen (128 channels, 25 Hz), and SAME-L (256 channels, about 10.8 Hz), so per second of audio the transformer processes approximately 43, 50, 25, or 11 tokens, respectively. The paper does not state how batch size or sequence length were equalized (e.g., seconds-per-batch versus tokens-per-batch), and attention cost and effective model capacity differ by up to a factor of about 4.6x across condition. Section 6.7 itself acknowledges that the optimal channel count is a property of the generator scale, so the reported ranking may be a property of the 0.6B pipeline rather than of the tokenizer itself. Please either re-run the comparison with equalized token budgets or at multiple generator scales, or restrict the claim to 'competitive under the Kandinsky-5 0.6B pipeline'.","section":"Sec. 6.6 and Sec. 6.7"},{"comment":"The audio model's training objective includes an alignment regularizer Lalign against a frozen audio foundation model F, but the identity of F and the form of Lalign are explicitly deferred to 'the dedicated publication' (Sec. 6.3 and Sec. 6.4). Since this term is part of the objective used to train the released checkpoints, the method is not reproducible from the manuscript and the contribution of the alignment term to the reported generation quality cannot be independently assessed. Please provide at least a precise specification of F and Lalign (or a complete pseudo-code description) in an appendix, or remove the alignment term from the reported model and retrain/re-evaluate without it.","section":"Sec. 6.3 and Sec. 6.4"},{"comment":"The headline comparative claims are based on point estimates without error bars, confidence intervals, or significance tests. For example, in Table 4 (Song Describer), the MMAudio baseline has a higher CLAP score (0.356 vs 0.339) and a lower FAD-PANNs (5.412 vs 7.971), and in Table 4 (LibriSpeech) DAC-VAE MovieGen has a higher CLAP (0.413 vs 0.389); in Table 3(c) EARS, the text calls a 0.31 dB SI-SDR deficit 'within noise' without defining the noise level. Similarly, the side-by-side evaluations in Sec. 4.2-4.4 and Sec. 6.6 report win rates and 'preferred over all three baselines on all three criteria' without stating the number of annotators, the number of prompts, the inter-annotator agreement, or the statistical test used. These omissions make it impossible to distinguish genuine differences from noise, especially for the subjective claims that the text says 'carry the main weight of the comparison.' Please add uncertainty quantification and the experimental protocol details, or temper the claims accordingly.","section":"Tables 3 and 4, Sec. 4.2-4.4"},{"comment":"The same neutrality concern applies to the visual tokenizer swaps. Section 4.2 states that patch size is adjusted to balance the number of spatial tokens, but the compared tokenizers still differ in channel count (e.g., 16 vs 64 for the 4x8x8 vs 4x16x16 comparison) and in temporal compression, and all image/video generation comparisons use a single 2B generator. The paper's own Sec. 5.2 ablation shows that channel count interacts with convergence and final quality, so the observed ranking may be specific to the Kandinsky-5 2B pipeline at the evaluated resolutions. To support the abstract's broad 'matches or surpasses frontier opensource tokenizers' claim, the visual comparisons should either include at least one additional generator scale or be explicitly framed as pipeline-specific evidence.","section":"Sec. 4.2 and Sec. 4.3"}],"minor_comments":[{"comment":"The OmniDoc-TokenBench table lists FID values (KVAE 1.74 vs FLUX.1-dev 0.554 and FLUX.2-dev 0.73) but the text only claims superiority on PSNR, SSIM, and NED; please define what FID is measuring in this reconstruction context and explain why the KVAE value is worse if the comparison is meant to support the 'surpassing' claim.","section":"Sec. 4.4, Table 2"},{"comment":"The CDS analysis, while honestly caveated, is presented as 'crucial' for model selection despite being based on a single in-sample cross-sectional correlation (r=0.906 over 14 configurations) and one joint-training trajectory; please add an explicit out-of-sample test or clearly label CDS as a heuristic rather than a validated selection criterion.","section":"Sec. 5.1"},{"comment":"There are several typos and formatting inconsistencies: 'KV AE' is sometimes written 'KV AE' and sometimes 'KV AE-...' in text, 'charachteristic' appears in Sec. 3.1, 'HunyaunVideo' in Sec. 4.1, and the reference list contains a placeholder '[89] VERIFY author list' that must be resolved before publication.","section":"Throughout"},{"comment":"The crop-length schedule is presented as a design choice but the specific schedule (e.g., how many steps at 0.38 s, how the length increases to 5 s, and how the batch size is adjusted) is not reported; since the paper emphasizes sharing training details, please provide the exact schedule or a reference to the repository where it is defined.","section":"Sec. 6.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a technical report-style contribution from the Kandinsky Lab with a broad 'matches or surpasses frontier opensource tokenizers' claim. The strongest evidence is the controlled tokenizer-swap methodology, but the neutrality of that control is undermined by differences in token counts, channel counts, and generator scale, and the audio alignment regularizer is withheld. These issues are fixable within the manuscript's scope by adding experiments at matched token budgets or multiple scales, fully specifying the training objective, and reporting uncertainty quantification for the comparative metrics. I would recommend major revision rather than rejection, provided the authors either provide the additional evidence or carefully narrow the central claim to the specific pipelines evaluated. The paper would also benefit from a clearer separation between 'reconstruction quality', 'diffusability diagnostics', and 'generation quality under a specific generator', since the current abstract and conclusion overstate the generality of the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful engineering report with released weights and code, and the real advance is KVAE-Audio, a full-band 48 kHz continuous waveform tokenizer at 50 Hz with 64 channels. The controlled tokenizer-swap experiments are a genuine improvement over comparing reconstruction numbers alone, and the paper is candid about several limits: FAD backbones ignore the band above 16 kHz, aesthetic predictors are not independent, and Sec. 6.7 concedes the optimal channel count depends on generator scale. That last concession matters.\n\nThe main soft spot is the neutrality claim in Sec. 6.6. The fixed 0.6B DiT sees roughly 43, 50, 25, or 11 tokens per second across the audio baselines, and 16 vs. 64 channels in video, so 'everything downstream fixed' is not really posterior to token count and geometry. The paper never says how sequence length or batch size were equalized. The authors' own Sec. 6.7 shows the optimal channel count grows with generator scale, which directly undercuts the abstract's broad drop-in statement. The audio alignment target and the exact form of L_align are withheld, deferred to a dedicated publication—a reproducibility gap given the claim to share training details. No error bars or significance tests appear anywhere. The CDS screening statistic is based on an in-sample correlation across 14 configs; the paper explicitly says it doesn't establish out-of-sample prediction, so it's fine as an internal heuristic but not a validated predictor.\n\nWho gains from this: anyone building or selecting tokenizers for latent diffusion, especially for full-band audio. The released artifacts and the swap methodology are checkable, and the paper deserves a serious referee even though the central claim is overreaching. A referee should ask for disclosure of the alignment details, variance estimates or significance tests, and a tempered abstract. I'd cite it for the audio tokenizer and the evaluation methodology.","headline":"Useful engineering report with released models, but the 'matches or surpasses' headline overreaches: the tokenizer-swap comparisons are pipeline-specific and the audio alignment details are withheld.","tokens_in":27073,"tokens_out":3323,"would_cite":true,"duration_ms":35323,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KVAE claims its continuous-latent tokenizers for image, video, and audio match or surpass six frontier open-source tokenizers on reconstruction and generation metrics.","keywords":["latent diffusion","tokenizer","variational autoencoder","text-to-image generation","text-to-video generation","text-to-audio generation","diffusability","correlation decay slope"],"falsifier":"Train the same downstream generator at a very different scale (for example, an 8B DiT) on KVAE-4x16x16 versus HunyuanVideo-1.5, or on 128-channel versus 64-channel audio latents, under the same data and steps: the paper's own analysis predicts these rankings can shift with generator capacity, so a reversal at another scale would show the claimed superiority is conditional on the generator rather than intrinsic to the tokenizers.","tokens_in":25901,"feed_emoji":"🎨","tokens_out":10285,"duration_ms":87759,"temperature":0.7,"pith_summary":"The paper presents KVAE, a family of continuous-latent tokenizers for audio, image, and video, all built for text-conditioned latent diffusion. It claims that these tokenizers match or surpass the released tokenizers of six frontier open-source systems—Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio, and MMAudio—on both reconstruction and generation quality, measured objectively (FID, CLIP, CLAP, Fréchet distance, PESQ) and by human side-by-side preference. The models span a 48 kHz full-band audio tokenizer with a 50 Hz latent, two causal video tokenizers at 4x8x8 and 4x16x16 compression, and an 8x8 image tokenizer. The authors also share training details, a model-selection method built on a correlation-decay statistic, and ablations of design choices.","feed_headline":"Match or beat six rival tokenizers across image, video, audio","feed_subtitle":"KVAE's continuous latents cut training cost and lift generation quality in diffusion pipelines for all three modalities.","key_machinery":"The load-bearing design choices are a continuous Gaussian bottleneck in every tokenizer (no vector quantization), an attention-free causal Conv3D stack for video with spatial RMSNorm and an asymmetric decoder, and, for audio, a DAC-derived convolutional backbone re-strided to [2,3,4,5,8] to reach 960x temporal compression, a single self-attention block at the 50 Hz bottleneck, and an alignment regularizer that pulls the latent toward a frozen audio foundation model. The paper's selection mechanism is the correlation decay slope (CDS), the negative slope of the cosine-similarity-versus-distance fit over the latent grid, used to screen candidates cheaply; final choices are always confirmed by training a generation model on top of the frozen tokenizer.","core_discovery":"At its core the paper claims that a tokenizer's value for latent diffusion lies less in reconstruction fidelity than in 'diffusability'—how readily a diffusion model can learn to denoise its latent space—and that this property can be engineered and screened. Under a controlled protocol, a fixed downstream generator (Kandinsky-5's 2B DiT for image and video, a 0.6B DiT for audio) is trained on each tokenizer with data, captions, and steps held constant, so that quality differences are attributed to the latent space alone. With that protocol, KVAE-4x16x16 surpasses HunyuanVideo-1.5's tokenizer in text-to-video generation and is preferred in side-by-side evaluation; KVAE-Audio with 64 channels at 50 Hz is preferred over all three audio baselines on all three judged criteria; and KVAE-2D-2.0 leads its FLUX baselines on semantic quality at equal steps. The paper introduces the correlation decay slope (CDS), a spatial cosine-similarity decay measure, as a cheap screening statistic that correlated strongly (r=0.906) with subjective visual quality across 14 image-tokenizer configurations.","pith_inferences":["At larger generator scales, the reported rankings may shift: the paper itself ties the 64-channel audio optimum to the 0.6B generator, so a wider audio latent could win with a bigger DiT.","The headline 'matches or surpasses' is established with one in-house generator family; on different diffusion backbones the ranking may compress or reverse, so the claim is most credible for pipelines close to Kandinsky-5's.","The CDS screening result is an in-sample correlation (n=14) and, as the paper notes, does not establish out-of-sample prediction; pre-registering CDS thresholds and testing them on new configurations would settle its value.","Two stated gaps limit the audio result's reproducibility: the frozen alignment model and loss are deferred to another publication, and the FAD backbones run at 16–32 kHz, so the objective metrics cannot confirm the advertised full-band generation."],"forward_implications":["KVAE's released checkpoints can be dropped into existing latent-diffusion pipelines as open-source components, with the reported results suggesting they would not degrade and often improve generation quality.","Higher compression ratios, such as 4x16x16 with 64 channels, bring faster convergence in the tested 2B image and video generator, which translates to shorter training runs and lower compute cost.","A full-band 48 kHz audio tokenizer with a 50 Hz latent removes the need for a vocoder and makes joint text-to-video-and-audio generation possible from a single continuous latent space.","The optimal number of latent channels is not intrinsic: the paper finds 64 channels best for audio at a 0.6B generator scale while 64 channels improves video, showing the choice must be made jointly with compression factor and downstream model size.","CDS or similar latent diagnostics can screen tokenizer candidates before the costly step of training a full diffusion generator, as long as the correlation is re-checked on new configurations."],"supporting_citations":[{"why":"Supplies the downstream generator (Kandinsky-5, 2B DiT) used in every image and video generation comparison; the audio comparison re-uses its DiT backbone.","marker":"[4]"},{"why":"Source of the attention-free Conv3D architecture, sequence-length scaling schedule, and encoder-decoder asymmetry the video tokenizers adapt.","marker":"[112]"},{"why":"The DAC-style convolutional backbone, Snake activation, and spectral discriminators the audio tokenizer inherits.","marker":"[52]"},{"why":"Wan-2.2 VAE, one of the six baselines compared for reconstruction and generation.","marker":"[99]"},{"why":"HunyuanVideo-1.5 VAE, the main video-generation baseline it is compared against.","marker":"[104]"},{"why":"MovieGen provides both a waveform DAC-VAE audio baseline and the prompt set used for video generation evaluation.","marker":"[69]"},{"why":"MMAudio's mel-spectrogram VAE, the primary audio baseline and the most common tokenizer under open text-to-audio-video systems.","marker":"[16]"},{"why":"SAME-L, the Stable Audio 3 autoencoder used as the third audio baseline.","marker":"[68]"},{"why":"Introduces the 'diffusability' criterion that motivates the design and selection of latent spaces beyond reconstruction quality.","marker":"[82]"},{"why":"Documents the reconstruction-generation dilemma used to frame channel-count trade-offs in both audio and video.","marker":"[113]"}],"fun_headline_variants":["KVAE tokenizers match or beat six rivals across modalities","Diffusability-first KVAE tops video and audio baselines","CDS metric predicts tokenizer quality with r=0.906"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on the assumption that the fixed in-house generator used for the tokenizer swap—Kandinsky-5's 2B DiT for image and video, a 0.6B DiT for audio—is neutral across different latent geometries (patch size, channel count, frame rate), so that every measured difference is attributable to the tokenizer rather than to interactions with the pipeline.","fun_headline_variants_meta":{"raw":{"variants":["KVAE tokenizers match or beat six rivals across modalities","Diffusability-first KVAE tops video and audio baselines","CDS metric predicts tokenizer quality with r=0.906"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1454,"prompt_tokens":1072,"completion_tokens":382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":324}},"tokens_in":688,"tokens_out":382,"duration_ms":4357,"temperature":1.0,"reasoning_tokens":324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:19:43.336487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same downstream generator at a very different scale (for example, an 8B DiT) on KVAE-4x16x16 versus HunyuanVideo-1.5, or on 128-channel versus 64-channel audio latents, under the same data and steps: the paper's own analysis predicts these rankings can shift with generator capacity, so a reversal at another scale would show the claimed superiority is conditional on the generator rather than intrinsic to the tokenizers.","supporting_citations":[{"cited_title":"Cogvideox: Text-to-video diffusion models with an expert transformer, 2025","cited_arxiv_id":null,"evidence_quote":"Source of the attention-free Conv3D architecture, sequence-length scaling schedule, and encoder-decoder asymmetry the video tokenizers adapt."},{"cited_title":"High-fidelity audio compression with improved RVQGAN","cited_arxiv_id":null,"evidence_quote":"The DAC-style convolutional backbone, Snake activation, and spectral discriminators the audio tokenizer inherits."},{"cited_title":"Wan-2.2 anouncement","cited_arxiv_id":null,"evidence_quote":"Wan-2.2 VAE, one of the six baselines compared for reconstruction and generation."},{"cited_title":"Hunyuanvideo 1.5 technical report, 2025","cited_arxiv_id":null,"evidence_quote":"HunyuanVideo-1.5 VAE, the main video-generation baseline it is compared against."},{"cited_title":"Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du","cited_arxiv_id":null,"evidence_quote":"MovieGen provides both a waveform DAC-VAE audio baseline and the prompt set used for video generation evaluation."},{"cited_title":"Parker, Zach Evans, CJ Carr, Zack Zukowski, Josiah Taylor, Matthew Rice, and Jordi Pons","cited_arxiv_id":null,"evidence_quote":"SAME-L, the Stable Audio 3 autoencoder used as the third audio baseline."},{"cited_title":"Improving the diffusability of autoencoders, 2025","cited_arxiv_id":null,"evidence_quote":"Introduces the 'diffusability' criterion that motivates the design and selection of latent spaces beyond reconstruction quality."}],"review_version":1}