{"id":"85f214e2-9d29-4526-a0cd-d8ac561e76e4","arxiv_id":"2505.13439","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"VTBench evaluates visual tokenizers in isolation across reconstruction, detail, and text tasks, and finds discrete tokenizers lag continuous VAEs.","lead":"This paper introduces VTBench, a benchmark that measures how well image tokenizers preserve pictures, detail, and text when converting images into the discrete tokens used by autoregressive image models. It finds that the continuous encoders used in diffusion models still beat all current discrete tokenizers on these tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 mixes input resolution with quantization: continuous VAEs are run at 1024×1024 while most discrete VTs run at 256–512, so the headline gap may reflect resolution/capacity differences rather than discrete-vs-continuous quality.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing issue: running each tokenizer at its native resolution means the continuous VAEs are evaluated at 1024×1024 while most discrete VTs are evaluated at 256×256 or 512×512, so resolution handling and representational budget are entangled with quantization type. I agree with this concern and with the CONDITIONAL verdict. I did not raise a separate objection about the GPT-4o-generated ground truth because, while human verification would improve dataset quality, the resolution mismatch is more directly tied to the paper's central quantitative comparison. The paper does deserve credit for releasing code and data, and the qualitative observations are consistent with the comparison, but the headline numeric gap is not yet isolated from a strong confound. A matched-resolution or matched-token-budget experiment would settle whether the conclusion is about discrete tokenization per se or about fixed-size, low-resolution discrete VTs. Until that check is run, the benchmark is valuable as a diagnostic tool but the strongest quantitative claim should be read as conditional.","tokens_in":24879,"tokens_out":3943,"duration_ms":41637,"concrete_test":"Recompute Table 1's ImageNet and High Resolution columns under a matched information budget. Use the same 50,000 ImageNet center crops at 256×256; for SD3.5L and FLUX.1, encode with their pretrained VAEs at 256×256 (latent grid roughly 32×32) and for discrete VTs use their native 256×256 settings. Then repeat at 1024×1024 only for models that accept it. Report PSNR, SSIM, LPIPS, and FID for each. If the continuous VAE still keeps a lead of the same magnitude (e.g., >3 dB PSNR), the resolution confound is secondary. If the gap narrows substantially or reverses, the paper's headline claim should be restated as 'continuous VAEs are more resolution-flexible and higher-capacity at high resolution' rather than 'discrete quantization is intrinsically worse.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that discrete visual tokenizers fall significantly behind continuous VAEs rests on Table 1, but the comparison is not resolution-matched. In the ImageNet subtask, each model is evaluated at its native input size (Section 3.1): SD3.5L and FLUX.1 run at 1024×1024, while most discrete VTs run at 256×256, 384×384, or 512×512. This changes both the input pixel resolution and the latent/token grid size: a FLUX VAE at 1024 encodes a 128×128 latent grid, while a 256×256 VQ tokenizer produces a much smaller token grid. The High Resolution and Varying Resolution subtasks amplify the confound: discrete models are forced to process 1024×1024 or arbitrary-resolution inputs that their training pipelines were never designed for, and several rows are simply missing (Titok, VAR, BSQ-ViT). Those failures may be resolution-generalization failures, not evidence about the intrinsic representational capacity of discrete quantization. The paper explicitly frames the finding as a tokenizer-quality gap (Abstract, Section 5), but the experimental design cannot separate quantization scheme from input-resolution handling and representational budget. This is the load-bearing weakness: if the gap shrinks or reverses when input resolution and token budget are matched, the headline conclusion must be reframed as a resolution-flexibility difference, not a fundamental discrete-vs-continuous deficiency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VTBench, a benchmark for evaluating visual tokenizers (VTs) in autoregressive image generation. It proposes three tasks—image reconstruction (ImageNet, high-resolution, varying-resolution), detail preservation, and text preservation (including multilingual Chinese, Hindi, Japanese, Korean)—with metrics PSNR, SSIM, LPIPS, FID, CER, and WER. The authors evaluate a broad selection of open-source discrete VTs (VQ, LFQ, BSQ, RVQ, RBSQ) plus continuous VAEs from SD3.5L and FLUX.1, and provide GPT-4o as an undisclosed reference. The central claim is that current discrete VTs fall significantly behind continuous VAEs in reconstruction quality, detail preservation, and text fidelity, and that this gap is a key bottleneck for AR image generation. The paper also includes speculative architectural analysis of GPT-4o and releases code and data.","tokens_in":25269,"tokens_out":6464,"duration_ms":59246,"significance":"The paper addresses an important and under-served evaluation gap: visual tokenizers are typically assessed only indirectly through end-to-end generation quality. VTBench is a focused, inference-only benchmark with no fitted parameters, and the public release of code and data is a concrete asset for the community. The qualitative demonstrations (e.g., Figure 5) are visually striking and support the intuition that discrete tokenizers lose fine-grained detail and text legibility. If the quantitative conclusions were fully controlled, the benchmark would provide a useful diagnostic tool for improving AR image generation pipelines. However, as detailed in the major comments, the headline comparison conflates quantization scheme with input resolution and representational budget, and the 100-image settings lack statistical rigor. The existence of Infinity-d64, a discrete tokenizer that at 1024x1024 nearly matches SD3.5L on PSNR/SSIM, shows that the claimed gap is not intrinsic to discrete versus continuous quantization and that the conclusion needs refinement.","major_comments":[{"comment":"The headline comparison between discrete VTs and continuous VAEs is not resolution-matched. In the ImageNet subtask, SD3.5L and FLUX.1 are evaluated at 1024x1024, whereas most discrete VTs are evaluated at 256x256 or 512x512; in the High Resolution subtask, discrete VTs are forced to process 1024x1024 inputs for which they were not trained. This confounds quantization scheme with input resolution and representational budget, since a 1024x1024 input yields a much larger latent/token grid than a 256x256 input even at the same downsampling factor. The confound is visible in Table 1: Infinity-d64, a discrete RBSQ tokenizer evaluated at 1024x1024, achieves PSNR 36.0 on ImageNet and 37.6 on High Resolution, close to SD3.5L (38.8 and 38.5) and much closer to FLUX.1 than the 256x256 discrete models are. The paper should either add matched-resolution and matched-budget experiments across all tokenizers (e.g., at 256, 512, and 1024 with comparable latent dimensions) or explicitly reframe the conclusion as a resolution-flexibility limitation rather than a fundamental discrete-vs-continuous quality gap.","section":"Section 3.1, Table 1"},{"comment":"Text-preservation metrics are computed by comparing OCR outputs of the original and reconstructed images using Gemma 3, rather than comparing reconstructed-image OCR to the known ground-truth strings that were used to synthesize the images. This makes the reference itself subject to OCR errors, which can be considerable for stylized poster text and small abstract fonts, and it conflates tokenizer ability with OCR quality. Since the ground-truth text is available by construction (Section 3.3, Appendix B), the evaluation should compare reconstructed images' OCR directly to those ground-truth strings, or at minimum report OCR accuracy on the original images as a sanity check.","section":"Section 3.4, Appendix C"},{"comment":"All non-ImageNet evaluation settings use only 100 images per condition, and no variance estimates, confidence intervals, or significance tests are reported. Given the large image-to-image variability in reconstruction metrics, close rankings (e.g., OpenMagViT2 versus LlamaGen ds8 in Table 1) may not be stable, and the word 'significantly' in the abstract and Section 4.3 is not statistically supported. I recommend reporting bootstrapped confidence intervals or per-image standard errors for every 100-image subtask, and an analysis of ranking stability as a function of sample size.","section":"Sections 3.1-3.3, Tables 2-3"}],"minor_comments":[{"comment":"A typo: 'across a wide range of task' should be 'across a wide range of tasks'.","section":"Introduction"},{"comment":"A typo: 'caluation' should be 'calculation'; also 'detialed' in Section 4 should be 'detailed'.","section":"Section 3.4"},{"comment":"Model naming is inconsistent: 'Titok' vs 'TiTok', 'V AR' vs 'VAR', and 'BSQ-VIT' vs 'BSQ-ViT'. Please standardize.","section":"Throughout"},{"comment":"For the Varying Resolution subtask, the paper does not specify how output images are handled when a discrete VT cannot produce the requested resolution (e.g., if the model outputs a fixed size or a different aspect ratio). Please state the preprocessing and any resizing performed before computing PSNR/SSIM/LPIPS/FID.","section":"Section 3.1"},{"comment":"The analysis of GPT-4o's internal architecture is based on qualitative, anecdotal examples and is phrased as speculation (e.g., 'we hypothesize'), but later conclusions in Section 5 read as established findings ('GPT-4o may employ an autoregressive generation mechanism'). Please clearly mark this section as untested speculation and soften the language so it does not appear to be an empirical result of the benchmark.","section":"Section 5, Appendix E"},{"comment":"There are duplicate references (Janus-Pro appears as both [4] and [5]) and inconsistent formatting between 'arXiv preprint' and 'CoRR' entries; please unify.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a useful community contribution, and the core direction is sound. The main issue is the resolution/budget confound in Table 1, which, if not addressed, would make the central claim over-stated. The 100-image error-bar omission and the OCR-reference issue are also important but readily fixable. The GPT-4o speculation should likely be trimmed or clearly labeled as a discussion point, since it does not add to the benchmark's validity. I see no evidence of circularity or misrepresentation of prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you follow autoregressive image generation. VTBench isolates visual tokenizers from downstream generation models, with three tasks — reconstruction, detail preservation, text preservation — plus multilingual text renders. The code and data are public. That part is genuinely useful: no one had put together a standalone tokenizer benchmark with this scope, and the release makes it a concrete tool for the community.\n\nThe headline result — discrete tokenizers fall well behind continuous VAEs — is less new than the framing implies. The paper cites prior work (Yu et al., others) that already found this. What's new is the measurement framework, not the conclusion.\n\nThe soft spot is real, though. Table 1 compares continuous VAEs at 1024×1024 with most discrete VTs at 256–512 even in the ImageNet task, and the High Resolution and Varying Resolution tasks mostly test resolution generalization, not quantization quality. The paper frames the gap as a tokenizer-quality deficiency, but the design can't separate that from input-size handling and representational budget. Missing rows for several discrete models (Titok, VAR, BSQ-ViT) in the high-res task underscore that this is partly a resolution-flexibility test. That doesn't mean the conclusion is false — the gap is probably real — but as written it's overclaimed. Also, the 100-image subsets have no error bars, and the ground truth is GPT-4o-synthesized without human verification. OCR-based CER/WER with Gemma 3 is reasonable, but the text datasets are synthetic and small.\n\nThe GPT-4o architecture analysis in the appendix is clearly labeled as hypothesis, and it reads that way. It's speculative but not harmful — just don't let it distract from the benchmark itself.\n\nIf you work on tokenizer design, the benchmark is worth citing and using. If you're looking for a definitive ranking of discrete vs continuous, wait for a revision that either matches resolutions or rephrases the claim as a resolution-flexibility result. My recommendation for an editor: send it to peer review. The benchmark itself is a solid contribution, and the flaws are fixable with reframing and a few controlled comparisons.","headline":"A useful new benchmark for isolating visual tokenizer quality, but the headline discrete-vs-continuous gap is partly a resolution mismatch and should be reframed.","tokens_in":25647,"tokens_out":2262,"would_cite":false,"duration_ms":20949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VTBench shows that discrete visual tokenizers used in autoregressive image generation reconstruct images, fine detail, and text substantially worse than continuous VAEs.","keywords":["visual tokenizer","autoregressive image generation","image reconstruction","detail preservation","text preservation","vector quantization","continuous VAE","benchmark"],"falsifier":"Run each discrete tokenizer at 1024x1024 with the input downscaled to the tokenizer's native resolution and the reconstruction upscaled back, and also with matched token counts per image; if discrete tokenizers then close the PSNR and SSIM gap to continuous VAEs, the claim that discrete quantization itself is the bottleneck would be undercut. A second check: if a discrete tokenizer trained with OCR-aware losses reaches near-zero CER and WER on the VTBench text set, the text-preservation gap is not intrinsic to discretization.","tokens_in":24677,"feed_emoji":"🖼️","tokens_out":7308,"duration_ms":66324,"temperature":0.7,"pith_summary":"Autoregressive image models depend on a visual tokenizer that turns pixels into discrete tokens, and the paper argues this component sets the ceiling on what the generation model can produce. VTBench is designed to test that component in isolation, rather than the end-to-end generator, across three tasks: reconstruction, detail preservation, and text preservation. Measured this way, current discrete tokenizers fall clearly behind the continuous variational autoencoders used by diffusion models, with the largest failures at high resolution, variable resolution, and embedded text in non-Latin scripts. If this stands, tokenizer quality is a primary bottleneck for autoregressive image generation, and the benchmark gives the field a shared way to measure progress.","feed_headline":"Discrete image tokenizers lag continuous VAEs on all tests","feed_subtitle":"VTBench isolates tokenizer quality and finds drops in detail, text, and high-resolution fidelity.","key_machinery":"The central object is the visual tokenizer treated as a standalone encoder-quantizer-decoder module, probed by reconstruction rather than by end-to-end generation. VTBench's load-bearing machinery is the task decomposition, covering ImageNet reconstruction, high-resolution and varying-resolution inputs, texture-rich detail images, and easy, hard, and multilingual text images, together with the metric set PSNR, SSIM, LPIPS, FID, CER, and WER, which makes quantization loss visible separately from the generative model.","core_discovery":"On the paper's own terms, the discovery is that the bottleneck in current autoregressive image generation sits in the discrete visual tokenizer, not the downstream model. Across VQ, LFQ, BSQ, RVQ, and RBSQ tokenizers taken from Chameleon, LlamaGen, MaskBiT, TiTok, VAR, OpenMagViT2, Janus Pro, FlowMo, and Infinity, VTBench's reconstruction numbers show lower PSNR and SSIM, higher LPIPS and FID, and far worse OCR-based character and word error rates than continuous VAEs such as SD3.5L and FLUX.1. The best discrete tokenizer, Infinity-d64, narrows the image-quality gap but still loses on text and resolution flexibility. The paper also reports that GPT-4o, when asked to copy an image with unreadable blurred text, outputs clean readable text, which it interprets as evidence that GPT-4o's tokenizer or backbone operates on semantic content and language priors rather than raw pixels.","pith_inferences":["The paper leaves implicit that its native-resolution protocol conflates two effects: the discrete tokenizer's quantization error and its inability to process off-spec resolutions; a token-count-matched comparison at the same resolution would isolate the quantization scheme.","Reconstruction-only scoring may underrate tokenizers designed to feed a language model: a tokenizer that discards low-level noise while the language model fills in plausible content could score poorly on PSNR yet generate more useful images, as the GPT-4o restoration behavior suggests.","A direct testable extension is to train a discrete tokenizer with OCR-aware and high-frequency objectives on VTBench-like data and measure whether end-to-end autoregressive generation closes the gap to diffusion models."],"forward_implications":["Every autoregressive generator built on a current discrete visual tokenizer inherits a reconstruction ceiling below diffusion models, so tokenizer research should move from end-to-end FID comparisons to VTBench-style isolated evaluation.","Resolution flexibility is a required property: because most discrete tokenizers fail on 1024x1024 or mixed-resolution inputs, autoregressive image generation cannot handle real-world images without resizing or cropping preprocessing.","Text rendering in autoregressive models will stay unreliable for documents, signage, and interfaces until tokenizers preserve symbolic detail, which blocks OCR and document-generation use cases.","The GPT-4o experiments suggest that a competitive autoregressive pipeline may need a tokenizer that encodes semantics and supports language-conditioned editing, not just pixel-level fidelity."],"supporting_citations":[{"why":"introduces VQ-VAE, the discrete tokenization paradigm most of the benchmarked tokenizers instantiate","marker":"[43]"},{"why":"supplies the claim that the tokenizer is key to visual generation and defines the LFQ tokenizer evaluated in VTBench","marker":"[52]"},{"why":"provides the continuous VAE used as one of the main continuous baselines","marker":"[35]"},{"why":"provides the FLUX.1 continuous VAE baseline that scores highest on reconstruction and text preservation","marker":"[20]"},{"why":"introduces VAR's residual next-scale tokenizer, a baseline and the architecture the paper suspects GPT-4o resembles","marker":"[41]"},{"why":"introduces Infinity's BSQ-based tokenizer, the strongest discrete tokenizer in the benchmark","marker":"[14]"},{"why":"defines binary spherical quantization, the scheme behind BSQ-ViT and Infinity","marker":"[56]"},{"why":"provides the ImageNet validation set used for the canonical reconstruction subtask","marker":"[36]"},{"why":"Gemma 3 is the OCR model whose outputs define the CER and WER text-preservation metrics","marker":"[19]"},{"why":"is the GPT-4o image generation reference used to synthesize benchmark images and to probe the model's tokenizer behavior","marker":"[30]"}],"fun_headline_variants":["Discrete tokenizers fail where VAEs shine: text and detail","VTBench highlights tokenizer gap in autoregressive image gen","Tokenizers, not models, cap autoregressive image quality","Discrete VTs trail VAEs on fidelity, text, and detail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that running each tokenizer at its native resolution is a fair test, so the large gaps at high and varying resolutions may reflect resolution handling rather than the quantization scheme alone.","fun_headline_variants_meta":{"raw":{"variants":["Discrete tokenizers fail where VAEs shine: text and detail","VTBench highlights tokenizer gap in autoregressive image gen","Tokenizers, not models, cap autoregressive image quality","Discrete VTs trail VAEs on fidelity, text, and detail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3064,"prompt_tokens":1004,"completion_tokens":2060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1988}},"tokens_in":620,"tokens_out":2060,"duration_ms":13548,"temperature":1.0,"reasoning_tokens":1988,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:13:02.768567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each discrete tokenizer at 1024x1024 with the input downscaled to the tokenizer's native resolution and the reconstruction upscaled back, and also with matched token counts per image; if discrete tokenizers then close the PSNR and SSIM gap to continuous VAEs, the claim that discrete quantization itself is the bottleneck would be undercut. A second check: if a discrete tokenizer trained with OCR-aware losses reaches near-zero CER and WER on the VTBench text set, the text-preservation gap is not intrinsic to discretization.","supporting_citations":[{"cited_title":"High- resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"provides the continuous VAE used as one of the main continuous baselines"},{"cited_title":"Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A","cited_arxiv_id":null,"evidence_quote":"supplies the claim that the tokenizer is key to visual generation and defines the LFQ tokenizer evaluated in VTBench"},{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"introduces VQ-VAE, the discrete tokenization paradigm most of the benchmarked tokenizers instantiate"},{"cited_title":"Visual autoregressive modeling: Scalable image generation via next-scale prediction","cited_arxiv_id":null,"evidence_quote":"introduces VAR's residual next-scale tokenizer, a baseline and the architecture the paper suspects GPT-4o resembles"},{"cited_title":"Bernstein, Alexander C","cited_arxiv_id":null,"evidence_quote":"provides the ImageNet validation set used for the canonical reconstruction subtask"}],"review_version":1}