Pith. sign in

REVIEW 3 major objections 6 minor 87 references

Large-scale text-to-video generation pre-trains a single feed-forward model that matches or beats specialized systems on depth, normals, camera pose, referring segmentation and 3D keypoints.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 00:55 UTC pith:OLVZJOSN

load-bearing objection Solid multi-task video perception system with real data efficiency and OOD generalization; the 'generative objective is the catalyst' claim is over-sold by an uncontrolled scale comparison. the 3 major comments →

arxiv 2607.09024 v1 pith:OLVZJOSN submitted 2026-07-10 cs.CV cs.AI

Video Generation Models are General-Purpose Vision Learners

Letian Wang , Chuhan Zhang , Rishabh Kabra , Jasper Uijlings , Steven Waslander , Andrew Zisserman , Joao Carreira , Kaiming He
show 4 more authors
Misha Andriluka Eduard Gabriel Bazavan Andrei Zanfir Cristian Sminchisescu
This is my paper
classification cs.CV cs.AI
keywords video generationvision foundation modelsdiffusion modelsmulti-task perceptiondepth estimationtext-to-videofeed-forward adaptationsynthetic data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper contends that the missing catalyst for general-purpose computer vision is large-scale text-to-video generation. Taking a pre-trained video diffusion backbone and converting it into a single-step feed-forward network called GenCeption yields one architecture that performs many perception tasks simply by changing a text prompt. On standard benchmarks the model reaches or exceeds dedicated systems such as DepthAnything3, SAM3, D4RT and VGGT-Ω while using far less fine-tuning data, and it generalizes from synthetic human-only videos to real multi-object scenes and to animals and robots. A sympathetic reader would care because this suggests video generation already encodes the spatiotemporal priors and language alignment needed for a true vision foundation model, playing the same role next-token prediction played for language.

Core claim

Large-scale text-to-video generative pre-training supplies the spatiotemporal world priors, vision-language alignment and scalability that computer vision has lacked. After multi-task post-training on mostly synthetic data, the resulting single feed-forward model GenCeption matches or surpasses specialized state-of-the-art systems across geometry, segmentation and pose tasks, often with 7×–500× less fine-tuning data, and exhibits zero-shot transfer from synthetic humans to real footage and out-of-distribution categories.

What carries the argument

GenCeption: a pre-trained video diffusion transformer is recast as a single-pass feed-forward feature extractor by feeding clean video latents at timestep zero, negating the predicted velocity, and steering all dense and sparse outputs via a text prompt (plus optional learnable tokens), under one shared L2 loss in RGB or coordinate space.

Load-bearing premise

The claim that video generation itself is the superior pre-training recipe rests on comparing a commercial-scale generator against far smaller alternative backbones under matched fine-tuning data, without isolating the objective from pre-training scale and data volume.

What would settle it

Train an identically sized and data-matched masked-autoencoder or JEPA backbone on the same video corpus used for the generative model, then fine-tune both under identical multi-task conditions; if the generative backbone no longer wins on the depth and geometry suite, the causal claim that generation is the foundational path collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single video-generation backbone can replace families of task-specific vision models once lightly post-trained.
  • Synthetic data alone becomes sufficient for competitive real-world perception when the backbone already contains strong world priors.
  • Further scaling of video generators should continue to lift multi-task perception without architectural redesign.
  • Text instructions become the natural interface for adding new visual tasks, analogous to language models.
  • Emergent generalization to unseen categories implies the pre-trained features already encode broad physical structure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same backbone may already contain usable priors for robotics control or physics simulation without further generative training.
  • If the generative objective is truly causal, future self-supervised video models that only predict rather than generate may under-perform even at matched scale.
  • Friction observed when jointly training sparse coordinate outputs suggests pre-training objectives may need redesign to natively support regression tokens.
  • Extreme data efficiency invites re-examination of whether commercial video generators have already absorbed most of the geometric supervision that specialist models re-learn from scratch.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that large-scale text-to-video generation is a strong pre-training paradigm for general-purpose computer vision, supplying spatiotemporal priors, vision-language alignment, and scalability. It introduces GenCeption: a feed-forward multi-task perception model obtained by adapting a pretrained video diffusion DiT (WAN 2.1) via single-step inference (t=0, negated Rectified-Flow velocity), a unified 3-channel RGB representation for dense tasks (including a spatially partitioned “Rothko” raymap for camera pose), learnable tokens for sparse 2D/3D keypoints, and multi-task fine-tuning predominantly on synthetic human-centric video. Empirically, the model matches or exceeds specialized SOTA systems (DepthAnything3, SAM3, D4RT, VGGT-Ω, Sapiens, David, Genmo, Lotus-2, etc.) on depth, normals, camera pose, expression-referring segmentation and 3D keypoints, shows data efficiency (claimed 7×–500× less fine-tuning data), preliminary scaling, and emergent sim-to-real / OOD category generalization from human-only synthetic training. Ablations compare the generative backbone to V-JEPA and VideoMAE under matched fine-tuning data and examine layer transfer and specialist vs. generalist training.

Significance. If the core empirical picture holds, the work is a substantial contribution to vision foundation models: it demonstrates that a single text-steered, feed-forward architecture built on a video generative backbone can compete with heavily specialized systems across dense and sparse video perception tasks while using far less task-specific data and exhibiting useful zero-shot transfer. The unified RGB-space formulation, single L2 loss, and explicit feed-forward reformulation of a multi-step DiT are clean engineering choices that lower the barrier to multi-task scaling. The reported sim-to-real and OOD generalization from purely synthetic human videos is particularly interesting and, if robust, strengthens the case that generative video pretraining encodes reusable physical-world priors. These strengths remain valuable even if the strongest causal claim (generative diffusion objective as the decisive catalyst) requires tempering.

major comments (3)
  1. [Table 2, §4.5, Abstract, §1] Table 2 and §4.5 claim that the WAN 2.1 generative backbone “significantly outperforms” the largest available V-JEPA-H (0.6B) and VideoMAE V2-G (1B) under matched fine-tuning data (7.5K synthetic videos) and that “the generative diffusion objective itself—rather than just the dataset or model scale—is essential.” The comparison does not control pretraining data volume, compute, or architectural maturity; commercial text-to-video models are trained on orders-of-magnitude more video-text pairs than the academic self-supervised baselines. Without a matched-scale (or at least matched-data) ablation of objective, the leap from “WAN works well after fine-tuning” to the abstract/§1 claim that video generation is “the foundational path” / “equivalent catalyst” to next-token prediction remains an uncontrolled inference. Either supply a controlled isolation experiment or substantially revise the c
  2. [Table 1, §4.5 (Joint training vs. task-specific training)] Table 1 shows that joint multi-task training (the “generalist”) degrades 3D human keypoint estimation severely relative to the specialist and produces mixed or mildly negative effects on several dense geometry metrics (depth AbsRel, camera pose ATE/RPE on some sets). The authors correctly note architectural tension between continuous pixel-space pretraining and the added learnable tokens, yet the paper’s central narrative of a “unified backbone, head, and loss” that “promotes a paradigm shift” is undercut by this degradation. The manuscript should quantify the trade-off more carefully, report whether additional data or staged training recovers the sparse-task performance, and avoid overstating architectural unity when sparse tasks currently require a non-native token pathway that harms other modalities.
  3. [Figure 2, Table 2, Abstract] The data-efficiency claim (comparable performance to D4RT / VGGT-Ω with 7×–500× less training data) is central to the abstract and Figure 2. Table 2 shows that the 14B WAN model trained on ~1.23M frames reaches competitive AbsRel numbers, but the specialist baselines were trained on heterogeneous real+synthetic mixtures whose exact frame counts, domain coverage, and multi-view geometry supervision differ substantially. A clearer apples-to-apples accounting of supervision type (metric vs. relative depth, multi-view consistency, camera labels) and a sensitivity analysis to the synthetic data mixture would make the efficiency claim more rigorous and less dependent on cross-paper aggregation.
minor comments (6)
  1. [§3.2] §3.2: the decision to fix t=0 and negate the Rectified-Flow velocity is presented as empirically beneficial, but no ablation against alternative single-step formulations (predicting x0 directly, intermediate-layer features, or a small number of denoising steps) is reported. A short table would strengthen the design choice.
  2. [§3.5] §3.5: the nonlinear depth mapping d′=clip(α log(d+1),0,1) introduces a free parameter α whose value and sensitivity are not stated. Please report the chosen α and a brief sensitivity check.
  3. [Figure 5, §3.3] Figure 5 (“Rothko” raymap): the spatial partitioning of 6-D ray data into 3 channels is clever but underspecified (exact region sizes, normalization of origins vs. directions, handling of varying focal lengths). A short formal definition would aid reproducibility.
  4. [Table 1] Table 1 contains several “∼” and missing entries; clarifying which numbers were re-run under identical protocols versus taken from original papers would improve fairness of the SOTA comparison.
  5. [Figure 1, Abstract] Minor typographical issues: “Methdology” in Figure 1 caption; inconsistent spacing around “7× to 500×”; “outperforms alternative pretraining paradigms (e.g., V-JEPA, and Video MAE)” has a stray comma.
  6. [§2.3] Related work (§2.3) correctly positions concurrent efforts ([69], Vision Banana). A short explicit statement of what is uniquely quantified here (feed-forward multi-task numbers on standard benchmarks) versus qualitative/training-free prompting would help readers.

Circularity Check

0 steps flagged

Empirical paper with external-benchmark evaluations; only a non-load-bearing self-citation of the authors' prior THFM work, no derivation that reduces by construction to its inputs.

full rationale

The paper's central claims are empirical: a video-diffusion backbone (WAN 2.1) is fine-tuned into a feed-forward multi-task model and evaluated on held-out real-world benchmarks (Sintel, KITTI, ETH3D, Hi4D, EMDB, Ref-DAVIS, MeViS, etc.) that are never used for training (except limited real data for referring segmentation). Depth normalization and the log mapping are data-representation choices that enable a unified L2 loss; they do not redefine the reported AbsRel or other standard metrics, which remain scale-invariant comparisons against external ground truth. No equation equates a 'prediction' to a fitted constant by construction. The sole self-citation (THFM [66]) is presented merely as prior human-centric work being generalized; it supplies neither a uniqueness theorem, an ansatz, nor any load-bearing premise for the multi-task results or the generative-pretraining thesis. The uncontrolled scale comparison versus V-JEPA/VideoMAE is a validity concern, not circularity. The derivation chain is therefore self-contained against external data.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim rests on three pillars: (1) that a commercial video diffusion backbone already encodes the needed spatiotemporal and vision-language priors, (2) that a minimal single-step, unified-RGB post-training recipe can extract them for many tasks, and (3) that synthetic human-centric data plus a few public sets suffice for real-world and OOD transfer. Free parameters are the usual training hyperparameters and the depth-mapping constant; axioms are domain assumptions about generative pretraining and transfer; invented entities are the system name, the Rothko raymap layout, and the sparse-token interface.

free parameters (5)
  • Adam learning rate = 5e-5
    Set to 5e-5 with 250-step linear warmup; controls convergence of the 15k-step fine-tune.
  • depth nonlinear mapping alpha
    α in d′ = clip(α log(d+1), 0, 1) chosen to map median-normalized depth into [0,1] RGB; directly affects depth supervision scale.
  • training steps and batch size = 15000 steps, batch 64
    15 000 steps, batch 64 on 256 TPUs; determines how far the backbone is adapted from the generative prior.
  • data mixture ratios across tasks and synthetic sources
    Balances multi-task L2 losses; paper states balancing is done via mixture ratios but does not publish exact weights.
  • gradient clip / drop thresholds
    Described as vital for stability; exact numeric thresholds not given.
axioms (5)
  • domain assumption Large-scale text-to-video generative pretraining internalizes 4D spatiotemporal priors, 3D geometry, physics and vision-language alignment sufficient for downstream perception.
    Stated as the core premise in §1 and §3.1; not independently proven, only supported by transfer experiments.
  • ad hoc to paper Conditioning the DiT at t=0 and negating the Rectified-Flow velocity yields the most informative features for single-step perception.
    §3.2 design choice; justified empirically by faster convergence, not derived from first principles.
  • ad hoc to paper All dense vision modalities can be losslessly (or adequately) represented in 3-channel RGB video space, including 6-D camera raymaps via spatial partitioning.
    §3.3 “Rothko” raymap and RGB unification; enables single decoder/loss but is a representational assumption.
  • domain assumption Synthetic human-centric video with Blender ground truth transfers to real-world footage and to non-human categories without real multi-task labels.
    Training recipe §3.4–3.5 and emergent-behavior claims §4.6; load-bearing for the data-efficiency and generalization narrative.
  • domain assumption A single shared L2 loss in latent/output space is sufficient for multi-task optimization once data formats are harmonized.
    §3.5; mirrors LLM practice but is an unproven transfer of that principle to continuous vision outputs.
invented entities (3)
  • GenCeption no independent evidence
    purpose: Name for the feed-forward multi-task perception system built on a video DiT backbone.
    System-level construct; evaluated only inside this paper’s experiments.
  • Rothko raymap no independent evidence
    purpose: Spatial layout that packs 6-channel camera ray origin/direction into 3 RGB channels for the unified decoder.
    Novel data-format invention (§3.3, Fig. 5); no external validation outside the paper.
  • Learnable sparse tokens with temporal position interpolation no independent evidence
    purpose: Interface that lets the same DiT emit per-frame 2D/3D keypoints without extra attention layers.
    Architectural add-on (§3.3); paper notes it disrupts pretrained attention and hurts joint training.

pith-pipeline@v1.1.0-grok45 · 24578 in / 4122 out tokens · 49962 ms · 2026-07-13T00:55:06.168868+00:00 · methodology

0 comments
read the original abstract

Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models. What, then, is the equivalent catalyst needed to achieve a general-purpose model in computer vision? In this paper, we contend that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision, providing the necessary spatiotemporal priors, vision-language alignment, and scalability required for general visual intelligence. We introduce GenCeption, which leverages a pre-trained video generative diffusion backbone to define a feed-forward perception model, capable of performing various vision tasks steered by text instructions. Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks, including depth, surface normal, and camera pose estimation, expression-referring segmentation, and 3D keypoint prediction, often matching or surpassing specialized models (e.g. DepthAnything3, SAM3, D4RT, VGGT-Omega, Sapiens, David, Genmo, and Lotus-2). Furthermore, the video generative pretrained backbone outperforms alternative pretraining paradigms (e.g., V-JEPA, and Video MAE) under comparable settings. Importantly, GenCeption exhibits preliminary data and model scaling properties along with exceptional data efficiency, where it achieves comparable performance with leading models like D4RT and VGGT-Omega with 7 to 500 less training data. Finally, GenCeption also exhibits intriguing emergent behaviors: a model trained exclusively on synthetic human videos generalizes to real-world footage and out-of-distribution object categories (e.g., animals and robots). These findings suggest that video generation is not merely a synthesis tool, but a foundational path toward generalist vision intelligence for the physical world. Project page: https://genception.github.io

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

87 extracted references · 30 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2506.07280 (2025) 4

    Acuaviva, P., Davtyan, A., Hassan, M., Stapf, S., Rahimi, A., Alahi, A., Favaro, P.: From genera- tion to generalization: Emergent few-shot learning in video diffusion models. arXiv preprint arXiv:2506.07280 (2025) 4

    Pith/arXiv arXiv 2025

  2. [2]

    Advances in Neural Information Processing Systems37, 61872–61911 (2024) 3

    Bachmann, R., Kar, O.F., Mizrahi, D., Garjani, A., Gao, M., Griffiths, D., Hu, J., Dehghan, A., Zamir, A.: 4m-21: An any-to-any vision model for tens of tasks and modalities. Advances in Neural Information Processing Systems37, 61872–61911 (2024) 3

    2024

  3. [3]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Bagchi, A., Bao, Z., Wang, Y.X., Tokmakov, P., Hebert, M.: Refereverything: Towards segmenting everything we can speak of in videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23221–23231 (2025) 4, 11

    2025

  4. [4]

    arXiv preprint arXiv:2404.08471 (2024) 2, 13

    Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., Ballas, N.: Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471 (2024) 2, 13

    Pith/arXiv arXiv 2024

  5. [5]

    org/abs/2112.128678

    Bazavan, E.G., Zanfir, A., Zanfir, M., Freeman, W.T., Sukthankar, R., Sminchisescu, C.: Hspace: Synthetic parametric humans animated in complex environments (2022),https://arxiv. org/abs/2112.128678

    Pith/arXiv arXiv 2022

  6. [6]

    arXiv preprint arXiv:2504.11427 (2025) 4, 11, 12

    Bin, Y., Hu, W., Wang, H., Chen, X., Wang, B.: Normalcrafter: Learning temporally consistent normals from video diffusion priors. arXiv preprint arXiv:2504.11427 (2025) 4, 11, 12

    Pith/arXiv arXiv 2025

  7. [7]

    Blender Online Community: Blender - a 3d modeling and rendering package (2025),https: //www.blender.org9

    2025

  8. [8]

    Advances in neural information processing systems33, 1877–1901 (2020) 3

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020) 3

    1901

  9. [9]

    In: European conference on computer vision

    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: European conference on computer vision. pp. 611–625. Springer (2012) 11

    2012

  10. [10]

    arXiv preprint arXiv:2511.16719 (2025) 1, 2, 3, 11, 12

    Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., et al.: Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719 (2025) 1, 2, 3, 11, 12

    Pith/arXiv arXiv 2025

  11. [11]

    Carnegie Mellon University: CMU Graphics Lab Motion Capture Database.http://mocap.cs .cmu.edu/(2001), database created with funding from NSF EIA-0196217 8 14 Video Generation Models are General-Purpose Vision Learners

    2001

  12. [12]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021) 4

    2021

  13. [13]

    URL https://arxiv

    Chen, S., Wong, S., Chen, L., Tian, Y.: Extending context window of large language models via positional interpolation, 2023. URL https://arxiv. org/abs/2306.15595 7

    Pith/arXiv arXiv 2023

  14. [14]

    In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186 (2019) 3

    2019

  15. [15]

    Ding, H., Liu, C., He, S., Jiang, X., Loy, C.C.: Mevis: A large-scale benchmark for video segmentationwithmotionexpressions.In: ProceedingsoftheIEEE/CVFinternationalconference on computer vision. pp. 2694–2703 (2023) 9, 11

    2023

  16. [16]

    In: Forty-first international conference on machine learning (2024) 6

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Forty-first international conference on machine learning (2024) 6

    2024

  17. [17]

    In: European Conference on Computer Vision (ECCV) (2024) 4

    Fu, X., Yin, W., Hu, M., Wang, K., Ma, Y., Tan, P., Shen, S., Lin, D., Long, X.: Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In: European Conference on Computer Vision (ECCV) (2024) 4

    2024

  18. [18]

    arXiv preprint arXiv:2604.20329 (2026) 5, 9

    Gabeur, V., Long, S., Peng, S., Voigtlaender, P., Sun, S., Bao, Y., Truong, K., Wang, Z., Zhou, W., Barron, J.T., et al.: Image generators are generalist vision learners. arXiv preprint arXiv:2604.20329 (2026) 5, 9

    Pith/arXiv arXiv 2026

  19. [19]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4340–4349 (2016) 9

    2016

  20. [20]

    In: Proceedings of the Winter Conference on Applications of Computer Vision

    Garcia, G.M., Zeid, K.A., Schmidt, C., De Geus, D., Hermans, A., Leibe, B.: Fine-tuning image- conditional diffusion models is easier than you think. In: Proceedings of the Winter Conference on Applications of Computer Vision. pp. 753–762 (2025) 11

    2025

  21. [21]

    The international journal of robotics research32(11), 1231–1237 (2013) 11

    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. The international journal of robotics research32(11), 1231–1237 (2013) 11

    2013

  22. [22]

    arXiv preprint arXiv:2512.01030 (2025) 3, 11, 12

    He, J., Li, H., Sheng, M., Chen, Y.C.: Lotus-2: Advancing geometric dense prediction with powerful image generative model. arXiv preprint arXiv:2512.01030 (2025) 3, 11, 12

    Pith/arXiv arXiv 2025

  23. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009 (2022) 3

    2022

  24. [24]

    org/abs/2006.112396

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models (2020),https://arxiv. org/abs/2006.112396

    Pith/arXiv arXiv 2020

  25. [25]

    Hu, W., Gao, X., Li, X., Zhao, S., Cun, X., Zhang, Y., Quan, L., Shan, Y.: Depthcrafter: Generating consistent long depth sequences for open-world videos (2024),https://arxiv.org/abs/ 2409.020954

    Pith/arXiv arXiv 2024

  26. [26]

    arXiv preprint arXiv:2512.07831 (2025) 3

    Huang,J.,Zhang,Y.,He,X.,Gao,Y.,Cen,Z.,Xia,B.,Zhou,Y.,Tao,X.,Wan,P.,Jia,J.: Unityvideo: Unified multi-modal multi-task learning for enhancing world-aware video generation. arXiv preprint arXiv:2512.07831 (2025) 3

    arXiv 2025

  27. [27]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Huang, P.H., Matzen, K., Kopf, J., Ahuja, N., Huang, J.B.: Deepmvs: Learning multi-view stereopsis. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2821–2830 (2018) 9 15 Video Generation Models are General-Purpose Vision Learners

    2018

  28. [28]

    arXiv preprint arXiv:2504.07961 (2025) 4, 7

    Jiang, Z., Zheng, C., Laina, I., Larlus, D., Vedaldi, A.: Geo4d: Leveraging video generators for geometric 4d scene reconstruction. arXiv preprint arXiv:2504.07961 (2025) 4, 7

    Pith/arXiv arXiv 2025

  29. [29]

    In: International Conference on Computer Vision (ICCV) (2023) 11

    Kaufmann, M., Song, J., Guo, C., Shen, K., Jiang, T., Tang, C., Zárate, J.J., Hilliges, O.: EMDB: TheElectromagneticDatabase of Global3D HumanPose andShapein theWild. In: International Conference on Computer Vision (ICCV) (2023) 11

    2023

  30. [30]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Repurposing diffusion- based image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9492–9502 (June 2024) 4

    2024

  31. [31]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Ke, Z., Sun, J., Li, K., Yan, Q., Lau, R.W.: Modnet: Real-time trimap-free portrait matting via objective decomposition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1140–1147 (2022) 11

    2022

  32. [32]

    arXiv preprint arXiv:2509.13414 (2025) 11

    Keetha, N., Müller, N., Schönberger, J., Porzi, L., Zhang, Y., Fischer, T., Knapitsch, A., Zauss, D., Weber, E., Antunes, N., et al.: Mapanything: Universal feed-forward metric 3d reconstruction. arXiv preprint arXiv:2509.13414 (2025) 11

    Pith/arXiv arXiv 2025

  33. [33]

    In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G

    Khirodkar, R., Bagautdinov, T., Martinez, J., Zhaoen, S., James, A., Selednik, P., Anderson, S., Saito, S.: Sapiens: Foundation for human vision models. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision – ECCV 2024. pp. 206–228. Springer Nature Switzerland, Cham (2025) 2, 3, 11

    2024

  34. [34]

    In: Asian conference on computer vision

    Khoreva, A., Rohrbach, A., Schiele, B.: Video object segmentation with language referring expressions. In: Asian conference on computer vision. pp. 123–141. Springer (2018) 11

    2018

  35. [35]

    arXiv preprint arXiv:1412.6980 (2014) 9

    Kingma, D.P.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 9

    Pith/arXiv arXiv 2014

  36. [36]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023) 1, 3

    2023

  37. [37]

    Kuang, Z., Zhang, T., Zhang, K., Tan, H., Bi, S., Hu, Y., Xu, Z., Hasan, M., Wetzstein, G., Luan, F.: Buffer anytime: Zero-shot video depth and normal from image priors (2024),https: //arxiv.org/abs/2411.172494

    Pith/arXiv arXiv 2024

  38. [38]

    arXiv preprint arXiv:2505.01425 (2025) 2, 3, 11, 12

    Li, J., Cao, J., Zhang, H., Rempe, D., Kautz, J., Iqbal, U., Yuan, Y.: Genmo: Generative models for human motion synthesis. arXiv preprint arXiv:2505.01425 (2025) 2, 3, 11, 12

    Pith/arXiv arXiv 2025

  39. [39]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, T., He, K.: Back to basics: Let denoising generative models denoise. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 36115–36125 (2026) 6

    2026

  40. [40]

    Liang, R., Gojcic, Z., Ling, H., Munkberg, J., Hasselgren, J., Lin, Z.H., Gao, J., Keller, A., Vijaykumar, N., Fidler, S., Wang, Z.: Diffusionrenderer: Neural inverse and forward rendering with video diffusion models (2025),https://arxiv.org/abs/2501.185904

    Pith/arXiv arXiv 2025

  41. [41]

    arXiv preprint arXiv:2511.10647 (2025) 1, 2, 3, 11, 12

    Lin, H., Chen, S., Liew, J., Chen, D.Y., Li, Z., Shi, G., Feng, J., Kang, B.: Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647 (2025) 1, 2, 3, 11, 12

    Pith/arXiv arXiv 2025

  42. [42]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lin, S., Ryabtsev, A., Sengupta, S., Curless, B.L., Seitz, S.M., Kemelmacher-Shlizerman, I.: Real-time high-resolution background matting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8762–8771 (2021) 11

    2021

  43. [43]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Lin, S., Yang, L., Saleemi, I., Sengupta, S.: Robust high-resolution video matting with temporal guidance. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 238–247 (2022) 11 16 Video Generation Models are General-Purpose Vision Learners

    2022

  44. [44]

    arXiv preprint arXiv:2210.02747 (2022) 6

    Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022) 6

    Pith/arXiv arXiv 2022

  45. [45]

    arXiv preprint arXiv:2209.03003 (2022) 6

    Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 (2022) 6

    Pith/arXiv arXiv 2022

  46. [46]

    Martinez, J., Kim, E., Romero, J., Bagautdinov, T., Saito, S., Yu, S.I., Anderson, S., Zollhöfer, M., Wang, T.L., Bai, S., et al.: Codec avatar studio: Paired human captures for complete, driveable, andgeneralizableavatars.AdvancesinNeuralInformationProcessingSystems37, 83008–83023 (2024) 11

    2024

  47. [47]

    Advances in Neural Information Processing Systems36, 58363– 58408 (2023) 3

    Mizrahi, D., Bachmann, R., Kar, O., Yeo, T., Gao, M., Dehghan, A., Zamir, A.: 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems36, 58363– 58408 (2023) 3

    2023

  48. [48]

    arXiv preprint arXiv:2304.07193 (2023) 4

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023) 4

    Pith/arXiv arXiv 2023

  49. [49]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2536–2544 (2016) 3

    2016

  50. [50]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 724–732 (2016) 12

    2016

  51. [51]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language su- pervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 4

    2021

  52. [52]

    IEEE transactions on pattern analysis and machine intelligence44(3), 1623–1637 (2020) 11

    Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence44(3), 1623–1637 (2020) 11

    2020

  53. [53]

    arXiv preprint arXiv:2408.00714 (2024) 1, 3

    Ravi,N.,Gabeur,V.,Hu,Y.T.,Hu,R.,Ryali,C.,Ma,T.,Khedr,H.,Rädle,R.,Rolland,C.,Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024) 1, 3

    Pith/arXiv arXiv 2024

  54. [54]

    Renderpeople: Renderpeople: 3d people for renderings.,https://renderpeople.com 8, 9

  55. [55]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR). pp. 10684–10695 (2022) 4

    2022

  56. [56]

    Saleh, F., Aliakbarian, S., Hewitt, C., Petikam, L., Xiao-Xian, Criminisi, A., Cashman, T.J., Baltrušaitis, T.: DAViD: Data-efficient and accurate vision models from synthetic data (2025), https://arxiv.org/abs/2507.153652, 3, 11

    Pith/arXiv arXiv 2025

  57. [57]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Schops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3260–3269 (2017) 11

    2017

  58. [58]

    In: SIGGRAPH Asia Conference Proceedings (2024) 11 17 Video Generation Models are General-Purpose Vision Learners

    Shen, Z., Pi, H., Xia, Y., Cen, Z., Peng, S., Hu, Z., Bao, H., Hu, R., Zhou, X.: World-grounded hu- man motion recovery via gravity-view coordinates. In: SIGGRAPH Asia Conference Proceedings (2024) 11 17 Video Generation Models are General-Purpose Vision Learners

    2024

  59. [59]

    arXiv preprint arXiv:2508.10104 (2025) 4

    Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025) 4

    Pith/arXiv arXiv 2025

  60. [60]

    In: International Conference on Learning Representations (2021) 6

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: International Conference on Learning Representations (2021) 6

    2021

  61. [61]

    Advances in neural information processing systems35, 10078–10093 (2022) 2, 4, 13

    Tong, Z., Song, Y., Wang, J., Wang, L.: Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems35, 10078–10093 (2022) 2, 4, 13

    2022

  62. [62]

    arXiv preprint arXiv:2508.21809 (2025) 11

    Uijlings, J., Zhou, X., Gu, X., Nagrani, A., Arnab, A., Fathi, A., Ross, D., Schmid, C.: Vocap: Video object captioning and segmentation from any prompt. arXiv preprint arXiv:2508.21809 (2025) 11

    Pith/arXiv arXiv 2025

  63. [63]

    arXiv preprint arXiv:2503.20314 (2025) 9

    Wan, T.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025) 9

    Pith/arXiv arXiv 2025

  64. [64]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: Vggt: Visual geome- try grounded transformer. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 5294–5306 (2025) 11

    2025

  65. [65]

    arXiv preprint arXiv:2605.15195 (2026) 2, 3, 11, 12

    Wang, J., Chen, M., Zhang, S., Karaev, N., Schönberger, J., Labatut, P., Bojanowski, P., Novotny, D., Vedaldi, A., Rupprecht, C.: Vggt-𝑜𝑚𝑒𝑔𝑎. arXiv preprint arXiv:2605.15195 (2026) 2, 3, 11, 12

    Pith/arXiv arXiv 2026

  66. [66]

    arXiv preprintarXiv:2603.25892(2026) 4

    Wang, L., Zanfir, A., Bazavan, E.G., Andriluka, M., Sminchisescu, C.: Thfm: A unified video foundationmodelfor4dhuman perceptionandbeyond. arXiv preprintarXiv:2603.25892(2026) 4

    arXiv 2026

  67. [67]

    In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Wang, W., Zhu, D., Wang, X., Hu, Y., Qiu, Y., Wang, C., Hu, Y., Kapoor, A., Scherer, S.: Tartanair: A dataset to push the limits of visual slam. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4909–4916. IEEE (2020) 9

    2020

  68. [68]

    In: European Conference on Computer Vision

    Wang, Y., Wang, Z., Liu, L., Daniilidis, K.: Tram: Global trajectory and motion of 3d humans from in-the-wild videos. In: European Conference on Computer Vision. pp. 467–487. Springer (2024) 11, 12

    2024

  69. [69]

    arXiv preprint arXiv:2509.20328 (2025) 4, 5

    Wiedemer, T., Li, Y., Vicol, P., Gu, S.S., Matarese, N., Swersky, K., Kim, B., Jaini, P., Geirhos, R.: Video models are zero-shot learners and reasoners. arXiv preprint arXiv:2509.20328 (2025) 4, 5

    Pith/arXiv arXiv 2025

  70. [70]

    In: CVPR (2022) 11

    Wu, J., Jiang, Y., Sun, P., Yuan, Z., Luo, P.: Language as queries for referring video object segmentation. In: CVPR (2022) 11

    2022

  71. [71]

    Xu, G., Ge, Y., Liu, M., Fan, C., Xie, K., Zhao, Z., Chen, H., Shen, C.: What matters when repur- posing diffusion models for general dense perception tasks? arXiv preprint arXiv:2403.06090 (2024) 4

    Pith/arXiv arXiv 2024

  72. [72]

    In: Proceedings of the European conference on computer vision (ECCV)

    Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., Huang, T.: Youtube-vos: Sequence-to-sequence video object segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 585–601 (2018) 9

    2018

  73. [73]

    In: European Conference on Computer Vision

    Yan, C., Wang, H., Yan, S., Jiang, X., Hu, Y., Kang, G., Xie, W., Gavves, E.: Visa: Reasoning video object segmentation via large language models. In: European Conference on Computer Vision. pp. 98–115. Springer (2024) 11

    2024

  74. [74]

    Yang, H., Huang, D., Yin, W., Shen, C., Liu, H., He, X., Lin, B., Ouyang, W., He, T.: Depth any video with scalable synthetic data (2025),https://arxiv.org/abs/2410.108154 18 Video Generation Models are General-Purpose Vision Learners

    Pith/arXiv arXiv 2025

  75. [75]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., Zhao, H.: Depth anything: Unleashing the power of large-scale unlabeled data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10371–10381 (2024) 3

    2024

  76. [76]

    Advances in Neural Information Processing Systems37, 21875–21911 (2024) 1, 3, 11

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems37, 21875–21911 (2024) 1, 3, 11

    2024

  77. [77]

    Advances in Neural Information Processing Systems37, 21875–21911 (2025) 1

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems37, 21875–21911 (2025) 1

    2025

  78. [78]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yang, P., Zhou, S., Zhao, J., Tao, Q., Loy, C.C.: Matanyone: Stable video matting with consis- tent memory propagation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 7299–7308 (2025) 11

    2025

  79. [79]

    In: Computer Vision and Pattern Recognition (CVPR) (2023) 11

    Yin, Y., Guo, C., Kaufmann, M., Zarate, J., Song, J., Hilliges, O.: Hi4d: 4d instance segmentation of close human interaction. In: Computer Vision and Pattern Recognition (CVPR) (2023) 11

    2023

  80. [80]

    In: European conference on computer vision

    Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: European conference on computer vision. pp. 69–85. Springer (2016) 9

    2016

Showing first 80 references.