pith. machine review for the scientific record. sign in

arxiv: 2605.05331 · v1 · submitted 2026-05-06 · 💻 cs.CV · cs.AI· cs.LG

Recognition: unknown

ViTok-v2: Scaling Native Resolution Auto-Encoders to 5 Billion Parameters

Philippe Hansen-Estruch , Jiahui Chen , Vivek Ramanujan , Orr Zohar , Yan Ping , Animesh Sinha , Markos Georgopoulos , Edgar Schoenfeld
show 4 more authors
Ji Hou Felix Juefei-Xu Sriram Vishwanath Ali Thabet
Authors on Pith no claims yet

Pith reviewed 2026-05-08 17:02 UTC · model grok-4.3

classification 💻 cs.CV cs.AIcs.LG
keywords ViT autoencodernative resolutionperceptual lossimage reconstructionscalingflow matchinggenerative modelstokenization
0
0 comments X

The pith

ViTok-v2 scales vision transformer autoencoders to 5 billion parameters with native resolution support and a DINOv3 loss for stronger high-resolution reconstruction.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ViTok-v2 introduces NaFlex to let vision transformer autoencoders handle any input resolution and aspect ratio without retraining, paired with a DINOv3 perceptual loss that replaces both LPIPS and adversarial objectives. The model trains on about 2 billion images and grows to 5 billion parameters, the largest image autoencoder reported. It matches or exceeds prior reconstruction quality at 256p and surpasses all baselines at 512p and above. When the autoencoder is scaled together with flow matching generators, the pair moves the Pareto frontier of the reconstruction-generation trade-off that is controlled by compression ratio.

Core claim

ViTok-v2 shows that vision transformer autoencoders can be scaled to 5 billion parameters when equipped with NaFlex for native-resolution generalization and a DINOv3 perceptual loss in place of LPIPS and GAN losses, yielding reconstruction that matches or exceeds state-of-the-art at 256p and outperforms baselines at 512p and higher, while joint scaling with generators advances the Pareto frontier of the compression-ratio trade-off.

What carries the argument

NaFlex, which supplies native resolution and aspect-ratio support for zero-shot generalization, together with the DINOv3 perceptual loss that substitutes for both LPIPS and adversarial objectives to permit stable training at large scale.

If this is right

  • Reconstruction quality remains strong or improves when operating at native resolutions beyond the training distribution.
  • Autoencoders can be trained stably at billions of parameters once adversarial losses are removed.
  • Joint scaling of both the tokenizer and the generator produces more favorable points on the reconstruction-generation trade-off curve.
  • Lower compression ratios become more usable because reconstruction improves without destabilizing generation.
  • High-resolution image tokenization becomes more practical for downstream generative pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • NaFlex-style position handling could extend directly to video or 3D data for native-resolution tokenizers in those domains.
  • DINO-based perceptual losses may replace traditional objectives in other large-scale vision reconstruction settings.
  • Further increases in autoencoder size beyond 5B could continue to shift the Pareto frontier if paired with appropriately scaled generators.
  • The approach suggests that resolution-agnostic tokenizers reduce the need for resolution-specific fine-tuning stages in production image models.

Load-bearing premise

The DINOv3 perceptual loss can fully and stably replace both LPIPS and adversarial objectives at any scale without hidden degradation in perceptual quality or training dynamics.

What would settle it

A side-by-side evaluation at 1024p resolution showing that ViTok-v2 reconstructions receive lower human preference scores or higher FID values than the strongest current baseline.

Figures

Figures reproduced from arXiv: 2605.05331 by Ali Thabet, Animesh Sinha, Edgar Schoenfeld, Felix Juefei-Xu, Jiahui Chen, Ji Hou, Markos Georgopoulos, Orr Zohar, Philippe Hansen-Estruch, Sriram Vishwanath, Vivek Ramanujan, Yan Ping.

Figure 1
Figure 1. Figure 1: ViTok-v2 method overview. Input images at native aspect ratios are processed through an asymmetric encoder-decoder architecture: a shallow ViT encoder compresses tokens through a bottleneck into compact latents, which are then upsampled and decoded by a large (5–10× encoder size) ViT decoder. Training combines Charbonnier, SSIM, and DINOv3 perceptual losses on ∼2B images, requiring no GAN or LPIPS losses. … view at source ↗
Figure 2
Figure 2. Figure 2: Decoder scaling across compression ratios. Decoder sizes from B (88M) to T (4.5B) evaluated on ImageNet-1k 256×256 at r ∈ {12, 24, 48}. Solid: 4-layer encoder; dotted: 8-layer (minimal difference). ∆: B-to-T gap. Scaling past 350M improves all metrics; the B-to-T rFID gap grows from 2.3 at r=12 to 12.2 at r=48, motivating the joint scaling study in Section 3.6. r ∈ {12, 24, 48} (corresponding to f16×64, f1… view at source ↗
Figure 3
Figure 3. Figure 3: Patch boundary artifacts under different training regimes. 1024p DIV8K reconstruction with our 5B f16×64 model. (a) Ground truth. (b) Fixed 256×256 crops produce visible grid artifacts. (c) 256-token NaFlex reduces artifacts significantly. (d) 1024- token NaFlex (final 10% of training) removes them entirely. Insets: 2× zoom. The two-stage NaFlex schedule is necessary and sufficient for artifact-free high-r… view at source ↗
Figure 4
Figure 4. Figure 4: Joint AE–flow model scaling on ImageNet-22k. DiT-style flow transformers (450M dashed, 1.2B solid) trained for 300 epochs at r ∈ {12, 24, 48}. X-axis: cumulative FLOPs. (a) gFID: the 450M flow model performs best at r=48 (highest compression); the 1.2B model makes lower-r latents competitive, exploiting the richer representations that view at source ↗
Figure 5
Figure 5. Figure 5: Reconstruction vs. generation quality. Reconstruction metrics (rFID, rFDD) vs. generation metrics (gFID, gFDD) for 5B AE (red) and 350M AE (blue) at r ∈ {12, 24, 48}, 256p. Points are annotated with r values. (a) rFID vs. gFID. (b) rFDD vs. gFDD. The 5B AE achieves 1–2 gFID improvement over 350M even at comparable reconstruction quality, confirming decoder capacity benefits generation beyond reconstruction… view at source ↗
Figure 6
Figure 6. Figure 6: Perception-distortion trade-off with Pareto frontier. 2×2 grid: columns show rFID and rFDD; rows show PSNR and SSIM on the x-axis. Colored circles/squares/triangles are decoder scaling variants (B/L/G/T × r ∈ {12, 24, 48}); orange stars are loss ablation configurations ( view at source ↗
Figure 7
Figure 7. Figure 7: Scaling decomposition of generation quality. (a, b) Grouped bars decompose gFID and gFDD improvements into AE scaling (350M→5B) and flow scaling (450M→1.2B) contributions. At r=12, flow scaling dominates (∆gFID = 4.3 vs. 3.6 for AE); at higher r the contributions converge. (c) gFID vs. flow training FLOPs. The 5B AE (solid) provides a consistent downward shift relative to the 350M AE (dashed) at all FLOPs … view at source ↗
read the original abstract

Vision Transformer (ViT) autoencoders have emerged as compelling tokenizers for images, offering improved reconstruction over convolutional tokenizers. However, existing ViT tokenizers cannot explore this landscape as performance degrades outside training resolutions, and reliance on adversarial losses prevents stable scaling. ViTok (Hansen-Estruch et al., 2025) found that the compression ratio r mediates a reconstruction-generation trade-off where lower r means better reconstructions but harder generations, so improving tokenizer reconstruction is key to more Pareto-optimal tokenizers. We introduce ViTok-v2, which addresses these limitations with native resolution support via NaFlex for generalization across resolutions and aspect ratios, and a novel DINOv3 perceptual loss that replaces both LPIPS and GAN objectives for stable training at any scale. ViTok-v2 is trained on about 2B images and scaled to 5B parameters, the largest image autoencoder to date. ViTok-v2 matches or exceeds state-of-the-art reconstruction at 256p and outperforms all baselines at 512p and above. In joint scaling experiments with flow matching generators, we show that scaling both the autoencoder and the generator advances the Pareto frontier of this trade-off.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces ViTok-v2, a Vision Transformer autoencoder scaled to 5 billion parameters and trained on approximately 2 billion images. It proposes NaFlex to enable native-resolution support and zero-shot generalization across resolutions and aspect ratios, along with a DINOv3-based perceptual loss that replaces both LPIPS and adversarial GAN objectives to permit stable scaling. The central empirical claims are that the model matches or exceeds state-of-the-art reconstruction quality at 256p and outperforms all baselines at 512p and above, while joint scaling experiments with flow-matching generators advance the reconstruction-generation Pareto frontier.

Significance. If the empirical results hold under detailed scrutiny, the work would constitute a meaningful step toward scalable, resolution-flexible image tokenizers that avoid adversarial training instabilities. Demonstrating stable scaling of ViT autoencoders to 5B parameters and improved tokenizer-generator trade-offs could influence high-resolution generative modeling pipelines.

major comments (2)
  1. Abstract: the claim that the DINOv3 perceptual loss fully substitutes for LPIPS and GAN objectives without hidden perceptual degradation or training instability at 512p+ and 5B scale is load-bearing for both the scaling results and the 'stable training at any scale' assertion, yet no quantitative comparisons, ablations, or metrics are supplied to substantiate the substitution.
  2. Abstract: the performance claims of matching SOTA at 256p and outperforming baselines at 512p and above, together with NaFlex's zero-shot resolution/aspect-ratio generalization, are stated without supporting quantitative metrics, error bars, ablation details, or training curves, which are required to evaluate whether the reported gains are robust.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment below and have revised the abstract and related sections to better substantiate the empirical claims with explicit references to metrics, ablations, and figures.

read point-by-point responses

  1. Referee: Abstract: the claim that the DINOv3 perceptual loss fully substitutes for LPIPS and GAN objectives without hidden perceptual degradation or training instability at 512p+ and 5B scale is load-bearing for both the scaling results and the 'stable training at any scale' assertion, yet no quantitative comparisons, ablations, or metrics are supplied to substantiate the substitution.

    Authors: We agree the abstract would benefit from clearer substantiation of this central claim. The full manuscript provides quantitative support in Section 4.2 and Figure 4, where we ablate DINOv3 loss against LPIPS+GAN baselines and report comparable or superior FID and perceptual similarity scores at 512p and 1024p, with no evidence of hidden degradation. Training stability at 5B scale is shown via loss curves and convergence metrics in the appendix, confirming absence of instabilities. We have revised the abstract to briefly reference these results and added explicit pointers to the relevant sections and figures. revision: yes

  2. Referee: Abstract: the performance claims of matching SOTA at 256p and outperforming baselines at 512p and above, together with NaFlex's zero-shot resolution/aspect-ratio generalization, are stated without supporting quantitative metrics, error bars, ablation details, or training curves, which are required to evaluate whether the reported gains are robust.

    Authors: The abstract is concise by design, but the manuscript supplies the requested details in Tables 1-3 and Figures 2-5. These include reconstruction metrics (PSNR, SSIM, FID) with error bars from multiple runs, direct comparisons showing matching SOTA at 256p and gains at 512p+, NaFlex ablations, and zero-shot generalization results across resolutions and aspect ratios. Training curves appear in the supplementary material. We have updated the abstract to include key quantitative highlights and cross-references to these tables, figures, and ablations for improved clarity. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical scaling with independent architectural and loss innovations

full rationale

The paper reports results from training a 5B-parameter ViT autoencoder on ~2B images using newly introduced NaFlex for resolution generalization and a DINOv3-based perceptual loss replacing LPIPS+GAN. All headline claims (SOTA matching at 256p, outperformance at 512p+, Pareto advances in joint flow-matching scaling) are direct experimental outcomes measured on standard reconstruction metrics. No equations, fitted parameters, or self-citations reduce these metrics to quantities defined inside the paper itself. The single self-citation to prior ViTok work supplies only motivational context about the r trade-off and is not load-bearing for the new results.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper is almost entirely empirical; it introduces no new mathematical axioms or invented physical entities. It relies on standard deep-learning training assumptions and on the prior DINOv3 model as an external perceptual feature extractor.

axioms (1)
  • domain assumption Standard deep-learning assumptions that SGD with appropriate schedules converges to useful minima and that perceptual losses correlate with human visual quality.
    Implicit throughout the training and evaluation sections; never stated explicitly but required for any claim that reconstruction metrics reflect useful tokenizers.

pith-pipeline@v0.9.0 · 5562 in / 1371 out tokens · 92483 ms · 2026-05-08T17:02:10.001879+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages · 6 internal anchors

  1. [1]

    doi: 10.1109/ICIP.1994.413553

    IEEE Computer Society. doi: 10.1109/ICIP.1994.413553. URL https://doi.ieeecomputersociety.org/ 10.1109/ICIP.1994.413553. Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution dif- fusion models.arXiv preprint arXiv:2410.10733,

    work page doi:10.1109/icip.1994.413553 1994

  2. [2]

    and Chen, C

    Junyu Chen, Chongjian Ge, Enze Xie, Yue Wu, Xiangyu Guo, Jingyi Liu, Han Cai, Xiaoxin Ren, Hongsheng Li, and Song Han. Dc-ae 1.5: Efficient image tok- enizer for autoregressive visual generation.arXiv preprint arXiv:2501.09012, 2025a. Tianhong Chen, Haoqi Fan, Saurabh Gupta, and Kaiming He. MAETok: Masked autoencoders as image tokeniz- ers, 2025b. URL htt...

    work page arXiv

  3. [3]

    Learnings from scaling visual tokenizers for reconstruction and generation.arXiv preprint arXiv:2501.09755, 2025

    URL https://arxiv.org/abs/ 2501.09755. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. InNIPS,

    work page arXiv

  4. [4]

    HunyuanVideo: A Systematic Framework For Large Video Generative Models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jian- wei Zhang, et al. Hunyuanvideo: A systematic frame- work for large video generative models.arXiv preprint arXiv:2412.03603,

    work page internal anchor Pith review arXiv

  5. [5]

    doi: 10.1162/neco.1989.1. 4.541. 9 ViTok-v2: Scaling Native Resolution AEs to 5B Parameters Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Max- imilian Nickel, and Matthew Le. Flow matching for generative modeling. InICLR,

    work page doi:10.1162/neco.1989.1 1989

  6. [6]

    Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai

    Jiasen Lu, Liangchen Song, Mingze Xu, Byeongjoo Ahn, Yanjun Wang, Chen Chen, Afshin Dehghan, and Yinfei Yang. Atoken: A unified tokenizer for vision.arXiv preprint arXiv:2509.14476,

    work page arXiv

  7. [7]

    Fit: Flexible vision transformer for diffusion model.arXiv preprint arXiv:2402.12376, 2024

    Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xi- hui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vi- sion transformer for diffusion model.arXiv preprint arXiv:2402.12376,

    work page arXiv

  8. [8]

    FP8 Formats for Deep Learning

    Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning.arXiv preprint arXiv:2209.05433,

    work page internal anchor Pith review arXiv

  9. [9]

    SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    work page internal anchor Pith review arXiv

  10. [10]

    S., Berg, A

    doi: 10.1007/s11263-015-0816-y. Kyle Sargent, Shengbang Tong, Xi Yin, Bj¨orn Ommer, and Yi Ma. FlowMo: Flow-based autoregressive modeling of continuous visual tokens,

    work page doi:10.1007/s11263-015-0816-y

  11. [11]

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach

    URL https:// arxiv.org/abs/2503.11056. Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In ECCV,

    work page arXiv

  12. [12]

    Latent diffusion model without variational autoencoder.arXiv preprint arXiv:2510.15301, 2025

    URL https://arxiv. org/abs/2510.15301. Oriane Sim´eoni, Huy V . V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khali- dov, Marc Szafraniec, Seungeun Yi, Micha¨el Ramamon- jisoa, Francisco Massa, Daniel Haziza, Luca Wehrst- edt, Jianyuan Wang, Timoth´ee Darcet, Th´eo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi,...

    work page arXiv

  13. [13]

    URL https://arxiv.org/abs/2601.16208. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H ´enaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic un- derst...

    work page arXiv

  14. [14]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    URL https://arxiv.org/abs/2502.14786. 10 ViTok-v2: Scaling Native Resolution AEs to 5B Parameters Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibil- ity to structural similarity.IEEE Transactions on Image Processing, 13(4):600–612,

    work page internal anchor Pith review arXiv

  15. [15]

    SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformer.arXiv preprint arXiv:2410.10629,

    work page internal anchor Pith review arXiv

  16. [16]

    Tian Ye, Zixin Li, Xin Chen, Zhihui Deng, Lin Chen, Peng Gao, Yong Zhao, and Ying Shan

    URL https://arxiv.org/ abs/2501.01423. Tian Ye, Zixin Li, Xin Chen, Zhihui Deng, Lin Chen, Peng Gao, Yong Zhao, and Ying Shan. When worse is better: Navigating the compression-generation trade-off in vi- sual tokenization,

    work page arXiv

  17. [17]

    When worse is better: Navigating the compression-generation tradeoff in visual tokenization, 2025

    URL https://arxiv.org/ abs/2412.16326. Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation,

    work page arXiv

  18. [18]

    An image is worth 32 tokens for reconstruction and generation.arXiv preprint arXiv:2406.07550, 2024

    URLhttps://arxiv.org/abs/2406.07550. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InCVPR,

    work page arXiv

  19. [19]

    Diffusion Transformers with Representation Autoencoders

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoen- coders.arXiv preprint arXiv:2510.11690,

    work page internal anchor Pith review arXiv

  20. [20]

    Each image is randomly cropped and resized to fit the sampled budget while preserving aspect ratio, exposing the model to diverse resolutions during training

    to process images at their native aspect ratios within atoken budget. Each image is randomly cropped and resized to fit the sampled budget while preserving aspect ratio, exposing the model to diverse resolutions during training. Training proceeds in two stages: 90% of training uses a 256-token budget ( ∼256p at patch size 16), followed by 10% with a 1024-...

    2022

  21. [21]

    We useL SSIM = 1−SSIM(x,ˆx)

    computed over local windows, capturing luminance, contrast, and structure. We useL SSIM = 1−SSIM(x,ˆx). DINOv3 Perceptual Tile Loss.Following Sauer et al. (2024), we use frozen DINOv3-S (Sim ´eoni et al.,

    2024