Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen video diffusion model learns new vision tasks from 3–30 examples, including abstract reasoning on ARC-AGI.

desk verdict A solid few-shot adapter paper whose headline 'emergent reasoning' claim is weaker than the paper's own best evidence; the ARC number is not load-bearing until contamination and checkpoint selection are addressed. read the letter →

arxiv 2506.07280 v2 pith:5FB4L56Z submitted 2025-06-08 cs.CV cs.AI

Pablo Acuaviva , Aram Davtyan , Mariam Hassan , Sebastian Stapf , Ahmad Rahimi , Alexandre Alahi , Paolo Favaro This is my paper
classification cs.CVcs.AI
keywords videodiffusionmodelsfew-shotlearningLoRAfine-tuningvisualreasoningARC-AGIgeneralistvisiontransferlatentunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that video diffusion models, trained only to generate coherent video, internalize structured visual knowledge that can be re-directed to new tasks with minimal supervision. It proposes turning each input–output image pair into a short transition video and fine-tuning LoRA adapters on a frozen image-to-video model. With only 3 to 30 such examples, the model generalizes to unseen inputs across segmentation, pose estimation, style transfer, and abstract reasoning. The authors read this as evidence that VDMs possess latent visual understanding rather than being mere generative engines.

What carries the argument

The load-bearing mechanism is the transition-video representation: each task pair $(x_i, y_i)$ is turned into an $F$-frame sequence $v = [x_i, v_2, \dots, v_{F-1}, y_i]$ by an interpolation function $\varphi$, and a frozen image-to-video diffusion model is adapted by LoRA on these sequences. At inference the model is conditioned on a new input $x$ and the final generated frame $\hat{y}$ is taken as the prediction. The interpolation choice matters: a discrete sharp transition is used for ARC-AGI reasoning tasks, while quadratic interpolation works best for segmentation and pose.

What would settle it

Audit CogVideoX1.5's training data for ARC-like grid puzzles, or run the identical LoRA transition-video procedure on a VDM pretrained only on natural videos: if that model still scores near 16.75% on ARC-AGI, the result does not depend on grid-like exposure, while if it collapses to near zero the paper's key premise fails.

Watch

Extended reading notes

Core claim

The central claim is that a pre-trained video diffusion model, CogVideoX1.5, can be repurposed for a wide range of visual tasks by fine-tuning lightweight LoRA weights on a handful of input–output pairs rendered as transition videos. On the public ARC-AGI evaluation set, this procedure reaches 16.75% accuracy under the two-attempt protocol, a level the paper compares favorably against several large language models, and on ConceptARC it averages 33.12% with three attempts per test input. The authors conclude that successful adaptation from so few examples reveals latent visual understanding already present in the pretrained VDM, and that video generation provides a powerful inductive bias for unifying perception, generation, and reasoning.

Load-bearing premise

The central claim rests on the assertion that CogVideoX1.5 was not pretrained on data resembling ARC-AGI; if its training corpus contained grid-like puzzles, the 16.75% accuracy could come from distribution overlap rather than latent visual understanding.

Editorial extensions

If this is right

  • A single frozen video diffusion model can be repurposed for many visual tasks by swapping lightweight LoRA weights instead of retraining the full model per task.
  • The few-shot regime of 3–30 examples suggests the pretrained prior does most of the work, which is evidence for transferable visual knowledge inside VDMs.
  • Scaling helps: CogVideoX1.5 (5B) outperforms LTX-Video (2B) at every training-set size, paralleling the scaling behavior seen in language models.
  • Video-based reasoning is a viable new route for ARC-style abstract visual reasoning, distinct from direct LLM prompting on grids.
  • Video diffusion models could serve as the backbone of future visual foundation models that adapt rapidly to novel tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the latent-understanding interpretation is correct, the interpolation trajectory itself acts as a task prior: smooth trajectories for continuous transformations and sharp transitions for discrete reasoning, suggesting that learning the interpolation jointly with the task could improve results.
  • The ARC-AGI result would be sharpened by an audit of CogVideoX1.5's pretraining data; without that audit, the 16.75% accuracy could reflect distribution overlap with grid-like puzzles rather than general visual reasoning.
  • Composable LoRA modules, which the paper names as future work, would let a frozen VDM stack skills such as segmentation followed by style transfer without retraining.
  • A direct comparison against an image diffusion model trained on the same transition videos would isolate whether temporal coherence, rather than the generative prior alone, supplies the few-shot generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a few-shot fine-tuning framework that repurposes a frozen image-to-video diffusion model (CogVideoX1.5, with LTX-Video for comparison) for diverse vision tasks. Each task is encoded as a set of input-output image pairs rendered as short transition videos, and LoRA adapters are trained on these videos while the base model remains frozen. At inference the final frame of the generated video is taken as the prediction. Experiments cover geometric transformations, style transfer, jigsaw puzzles, colorization, inpainting, binary segmentation, pose estimation, TinyImageNet grid classification, and the ARC-AGI and ConceptARC reasoning benchmarks. The authors report strong generalization from 3 to 30 training examples and argue that this success reveals latent visual understanding in video diffusion models, positioning VDMs as adaptable visual learners and potential vision foundation models.

Significance. If the central claim holds, the framework is a valuable probe of the representations learned by video diffusion models and a promising step toward using VDMs as generalist vision backbones. The paper has clear strengths: the task-to-video reformulation is simple and unified, the LoRA-based adaptation is parameter-efficient, the ablation study over interpolation methods, LoRA modules, and rank is useful, and the appendix provides detailed hyperparameters, compute costs, and an unusually candid discussion of failure modes such as underspecification and color shifts. The comparison across two VDMs and the inclusion of public reasoning benchmarks are also positive. However, the central interpretation depends on several load-bearing empirical premises that are not yet adequately supported: the absence of ARC-like data in CogVideoX pretraining, the validity of the proposed Match Rate metric, and the fairness of the checkpoint-selection procedure for ARC and ConceptARC. These issues do not invalidate the framework but they do affect whether the paper can support its broad claims about emergent latent understanding.

major comments (4)
  1. [Section 4.3.1] The claim that CogVideoX1.5 "has not been pre-trained on any data resembling ARC-AGI" is stated without evidence, yet it is load-bearing for contribution 3 and for the conclusion that VDMs possess latent visual understanding. CogVideoX is trained on large-scale, undisclosed internet video, and ARC-AGI's simple colored grids are exactly the kind of visual content that can appear in screenshots, animations, or synthetic video. If ARC-like or visually similar reasoning tasks were present in pretraining, the 16.75% accuracy could reflect distribution overlap rather than a general latent reasoning ability. Please provide a contamination audit, report any known pretraining-data composition, or reframe the claim to avoid asserting absence of overlap; a concrete control would be to test on newly generated ARC-like tasks that cannot have been in the pretraining corpus.
  2. [Appendix A.4.2] The ARC-AGI and ConceptARC evaluation selects validation predictions from checkpoints at which the model's prediction on the training set is correct, and otherwise uses the final available predictions. This protocol uses training-set labels to choose among checkpoints, which can inflate the reported accuracy relative to a fixed evaluation schedule and makes the numbers in Table 3 and Figure 9 difficult to interpret as a genuine few-shot generalization result. Please report results under a fixed checkpoint or averaged over multiple checkpoints, and quantify the variance induced by the selection procedure.
  3. [Appendix A.4.1] The Match Rate metric is introduced as "well correlated with pose estimation quality," but no evidence for this correlation is provided, and the metric explicitly does not penalize spurious or excessive predictions. The threshold of 1.5 times the average inter-head distance, with a default of 20 pixels when no heads are available, is ad hoc. Because Match Rate is used in Tables 1 and 2 to support conclusions about pose estimation performance, please validate it against a standard metric such as PCK or OKS on a subset of runs, and either incorporate a false-positive penalty or justify why the visual verification described in the appendix is sufficient.
  4. [Section 6] The conclusion that VDMs "possess latent visual understanding that can be unlocked through minimal supervision" goes beyond what the current experiments demonstrate. Few-shot success on held-out inputs shows that the LoRA adapters can learn a mapping from a handful of examples, but it does not by itself establish that the knowledge was already latent in the frozen VDM rather than learned by the adapters from the provided examples. To support the latent-knowledge interpretation, please include a baseline in which LoRA is trained on the same few examples but with a model that lacks the VDM's pretraining (for example, a randomly initialized or weakly pretrained encoder), or an analysis of how performance changes as pretraining quality varies.
minor comments (5)
  1. [Table 1] The table footnote states that some rows are duplicated because they correspond to the same run, but the duplicated entries are not visually marked; please indicate them with a symbol or a note in the table itself to avoid confusion.
  2. [Section 4.2] The sentence "we perform several runs and report the average results" does not state the number of runs or the variance; please include the number of seeds and standard deviations in Table 1 and Table 2, at least for the n=3 and n=5 settings where variability is expected to be highest.
  3. [Appendix A.4.2] The statement that only one ARC-AGI case required manual grid-size handling and that "visual inspection confirmed" correctness is not verifiable from the text; please provide a reproducible rule for such cases or exclude them from the reported accuracy.
  4. [Section 3.3] The inference procedure in step 3 is written as "Apply denoising for t=T,...,1 to vt", but the notation for the noise scheduling is inconsistent with Eq. (1); please align the notation so that v_T denotes the fully noised sample and v_0 the denoised output.
  5. [References] Reference [16] is listed as "Yoav HaCohen et al." while the in-text citation uses "LTX-Video"; please ensure the reference list entry matches the author order of the actual paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core results are held-out evaluations after LoRA fine-tuning, and the interpretive 'latent understanding' claim is an abductive inference rather than a definitional reduction.

full rationale

The paper's central derivation is an empirical pipeline: encode (x,y) pairs as transition videos, fine-tune LoRA adapters on a frozen video diffusion model, then generate a video from a held-out input and read the last frame as the prediction. The reported numbers on COCO segmentation/pose, ARC-AGI, ConceptARC, and TinyImageNet are all measured on validation inputs that were not in the LoRA training set, so they are not forced by construction. The interpolation function in Eq. (5) and its variants are manually chosen design choices, not fitted parameters, and their effect is systematically ablated in Table 1. No fitted parameter is simultaneously reported as a prediction. The only self-citations (e.g., [24] in Related Work) are not load-bearing. The interpretive step in Section 3.1, 'The key intuition is that successful adaptation under such constraints implies the model already encodes the relevant knowledge,' is an abductive argument from few-shot generalization to prior knowledge, not a definitional equivalence: the paper does not define 'latent visual understanding' as equivalent to 'few-shot adaptability.' The unverified assertion in Section 4.3.1 that CogVideoX1.5 'has not been pre-trained on any data resembling ARC-AGI' is a missing-evidence and contamination risk for the reasoning interpretation, but it is an external correctness concern rather than circularity: the 16.75% ARC-AGI figure is a measured held-out result, and the absence of a data audit does not make the result equivalent to its own inputs. Overall, the derivation chain is self-contained with respect to the specific circularity patterns considered, so the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on domain assumptions about pretraining data and evaluation metrics rather than on fitted equations. The only hand-set numeric constants are the pose matching threshold and default head distance in the custom metric; these affect supporting results but not the method's equations. No new physical entities are introduced.

free parameters (2)
  • Match Rate distance threshold = 1.5 times average inter-head distance
    Hand-chosen threshold in Appendix A.4.1 to decide whether a predicted body-part centroid matches ground truth; affects pose estimation scores and is not validated against standard metrics.
  • Default average inter-head distance = 20 pixels
    Used when no heads are detected in the annotation; another hand-set constant in the custom pose metric.
assumptions (3)
  • domain assumption CogVideoX1.5 pretraining data contains no ARC-AGI-like examples
    Stated in Section 4.3.1 without verification; the ARC result is interpreted as emergent reasoning only under this premise.
  • ad hoc to paper Match Rate is well correlated with pose estimation quality
    Claimed in Appendix A.4.1 with no comparison to standard pose metrics (PCK, OKS); the metric is introduced by the authors for feasibility.
  • domain assumption Interpolated transitions from input to output align with the VDM's inductive biases
    The framework in Section 3.1 assumes task input-output pairs can be represented as frames of a plausible video; this is justified only by empirical results, not by analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models." pith.science (2026). https://pith.science/paper/5FB4L56Z

@misc{pith2026250607280,
  author       = {Pith},
  title        = {Pith review of: From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5FB4L56Z}},
  note         = {Machine review of arXiv:2506.07280}
}
read the original abstract

Video Diffusion Models (VDMs) have emerged as powerful generative tools, capable of synthesizing high-quality spatiotemporal content. Yet, their potential goes far beyond mere video generation. We argue that the training dynamics of VDMs, driven by the need to model coherent sequences, naturally pushes them to internalize structured representations and an implicit understanding of the visual world. To probe the extent of this internal knowledge, we introduce a few-shot fine-tuning framework that repurposes VDMs for new tasks using only a handful of examples. Our method transforms each task into a visual transition, enabling the training of LoRA weights on short input-output sequences without altering the generative interface of a frozen VDM. Despite minimal supervision, the model exhibits strong generalization across diverse tasks, from low-level vision (for example, segmentation and pose estimation) to high-level reasoning (for example, on ARC-AGI). These results reframe VDMs as more than generative engines. They are adaptable visual learners with the potential to serve as the backbone for future foundation models in vision.

Figures

Figures reproduced from arXiv: 2506.07280 by the authors.

Figure 1
Figure 1. Proposed framework. Given a task encoded as input-target image pairs (dashed gray box), [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Examples of learned transformations applied to a new image (a) in 1-shot setting (b), (c), [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Style transfer in a 1-shot setting, applied to a new image (second row). [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Qualitative performance on different vision tasks with an increasing number of training [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Grid classification on TinyImageNet: (a) Training examples, (b) Confusion matrix with [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of different interpolation methods for pose estimation: (a) Interpolation [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Examples of solved tasks from ARC. Training samples for each task (first three rows), [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: (a) Samples where the best model prediction failed on a single grid point, (b) Accuracy [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Accuracy in ConceptARC with different number of tries. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Match Rate calculation. Left: original image. Top row: model prediction and estimated [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Additional qualitative results for Style Transfer. Training sample on first row, generated [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Predictions for pose estimation (left) and binary segmentation (right) for training with [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Progression of results on the Jigsaw task under varying numbers of training samples. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Progression of results on the Colorization task under varying numbers of training samples. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Progression of results on the Inpainting task under varying numbers of training samples. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Validation results on the TinyImageNet grid classification task with [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Additional examples of solved tasks from ARC-AGI. Training samples for each task (first [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Additional examples of solved tasks from ARC-AGI. Training samples for each task (first [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Additional examples of solved tasks from ARC-AGI. Training samples for each task (first [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Left: a training sample used in the inpainting task (one of several masked-region variations). [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Examples from the binary segmentation task (validation), where the model, trained to [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Examples for Segmentation to Image with out-of-distribution validation samples. [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: Validation examples for depth estimation. [PITH_FULL_IMAGE:figures/full_fig_p026_23.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video Generation Models are General-Purpose Vision Learners

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A video-diffusion backbone fine-tuned as a single-step multi-task perceiver matches or beats specialists on depth, normals, pose and segmentation, with high data efficiency and sim-to-real transfer.

Reference graph

Works this paper leans on

66 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    The surprising effectiveness of test-time training for few-shot learning

    Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning. arXiv preprint arXiv:2411.07279, 2025

    arXiv 2025

  2. [2]

    Sequential modeling enables scalable learning for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22861–22872, 2024

    2024

  3. [3]

    Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros. Visual prompting via image inpainting.CoRR, abs/2209.00647, 2022

    arXiv 2022

  4. [4]

    How I got a record 53.6% on ARC-AGI-Pub using Sonnet 3.5.1

    Jeremy Berman. How I got a record 53.6% on ARC-AGI-Pub using Sonnet 3.5.1. https:// jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi , 2024. Ac- cessed: 2025-05-12

    work page 2024

  5. [5]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.CoRR, abs/2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets.CoRR, abs/2311.15127, 2023

    arXiv 2023

  6. [6]

    On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    arXiv 1911

  7. [7]

    Arc prize 2024: Technical report.arXiv preprint arXiv:2412.04604, 2024

    Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report.arXiv preprint arXiv:2412.04604, 2024

    arXiv 2024

  8. [8]

    Studying Image Diffusion Features for Zero-Shot Video Object Segmentation

    Thanos Delatolas, Vicky Kalogeiton, and Dim P Papadopoulos. Studying image diffusion features for zero-shot video object segmentation.arXiv preprint arXiv:2504.05468, 2025

    work page Pith review arXiv 2025

Show all 66 references
  1. [9]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, and Zhifang Sui. A survey on in-context learning. In EMNLP, pages 1107–1128, 2024

    2024

  2. [10]

    Movie gen: A cast of media foundation models.CoRR, abs/2410.13720, 2024

    Adam Polyak et al. Movie gen: A cast of media foundation models.CoRR, abs/2410.13720, 2024

    2024 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948, January 2025

    DeepSeek-AI et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948, January 2025. 10

    2025 arXiv

  4. [12]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.CoRR, abs/2403.05530, 2024

    Machel Reid et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.CoRR, abs/2403.05530, 2024

    2024 arXiv

  5. [13]

    Cosmos world foundation model platform for physical ai.CoRR, abs/2501.03575, January 2025

    Niket Agarwal et al. Cosmos world foundation model platform for physical ai.CoRR, abs/2501.03575, January 2025

    2025 arXiv

  6. [14]

    Phenaki: Variable length video generation from open domain textual description.CoRR, abs/2210.02399, 2022

    Ruben Villegas et al. Phenaki: Variable length video generation from open domain textual description.CoRR, abs/2210.02399, 2022

    2022 arXiv

  7. [15]

    Brown et al

    Tom B. Brown et al. Language models are few-shot learners.CoRR, abs/2005.14165, 2020

    2005 arXiv

  8. [16]

    Ltx-video: Realtime video latent diffusion.CoRR, abs/2501.00103, January 2025

    Yoav HaCohen et al. Ltx-video: Realtime video latent diffusion.CoRR, abs/2501.00103, January 2025

    2025 arXiv

  9. [17]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang et al. Cogvideox: Text-to-video diffusion models with an expert transformer. CoRR, abs/2408.06072, 2024

    2024 arXiv

  10. [18]

    Richard P. Feynman. Feynman’s office; the last blackboards.Physics Today, 42(2):88–88, February 1989

    1989

  11. [19]

    The free-energy principle: a unified brain theory?Nature reviews neuroscience, 11(2):127–138, 2010

    Karl Friston. The free-energy principle: a unified brain theory?Nature reviews neuroscience, 11(2):127–138, 2010

    2010

  12. [20]

    Instructdiffusion: A generalist modeling interface for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling interface for vision tasks. InProceedings of the IEEE/CVF Conference on computer vision and pattern recogniti...

    2024

  13. [21]

    Few-shot diffusion models.CoRR, abs/2205.15463, 2022

    Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models.CoRR, abs/2205.15463, 2022

    2022 arXiv

  14. [22]

    Google DeepMind. Veo 2. https://deepmind.google/technologies/veo/veo-2/, De- cember 2024. Accessed: 2025-05-12

    2024

  15. [23]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    2024

  16. [24]

    Gem: A generalizable ego-vision mul- timodal world model for fine-grained ego-motion, object dynamics, and scene composition control.CVPR, 2025

    Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Pedro M B Rezende, Yasaman Haghighi, David Brüggemann, Isinsu Katircioglu, Lin Zhang, Xiaoran Chen, Suman Saha, Marco Cannici, Elie Aljalbout, Botao Ye, Xi Wang, Aram Davtyan, Mathieu Salzmann, Davide Scaramuzza, Marc Pollefeys, Pa...

    2025

  17. [25]

    Conceptattention: Diffusion transformers learn highly interpretable features.CoRR, abs/2502.04320, February 2025

    Alec Helbling, Tuna Han Salih Meral, Benjamin Hoover, Pinar Yanardag, and Duen Horng Chau. Conceptattention: Diffusion transformers learn highly interpretable features.CoRR, abs/2502.04320, February 2025

    2025 arXiv

  18. [26]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  19. [27]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers.CoRR, abs/2205.15868, 2022

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.CoRR, abs/2205.15868, 2022

    2022 arXiv

  20. [28]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  21. [29]

    Lo- rahub: Efficient cross-task generalization via dynamic lora composition.arXiv preprint arXiv:2307.13269, 2024

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition.arXiv preprint arXiv:2307.13269, 2024. 11

    2024 arXiv

  22. [30]

    World and human action models towards gameplay ideation.Nature, 638(8051):656–663, 2025

    Anssi Kanervisto, Dave Bignell, Linda Yilin Wen, Martin Grayson, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Tabish Rashid, Tim Pearce, Yuhan Cao, et al. World and human action models towards gameplay ideation.Nature, 638(8051):656–663, 2025

    2025

  23. [31]

    Flux.1-dev

    Black Forest Labs. Flux.1-dev. https://huggingface.co/black-forest-labs/FLUX. 1-dev, 2025

    2025

  24. [32]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context.CoRR, abs/1405.0312, 2014

    2014 arXiv

  25. [33]

    Realgeneral: Unifying visual generation via temporal in-context learning with video models.arXiv preprint arXiv:2503.10406, 2025

    Yijing Lin, Mengqi Huang, Shuhan Zhuang, and Zhendong Mao. Realgeneral: Unifying visual generation via temporal in-context learning with video models.arXiv preprint arXiv:2503.10406, 2025

    2025 arXiv

  26. [34]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.CoRR, abs/2205.05638, 2022

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.CoRR, abs/2205.05638, 2022

    2022 arXiv

  27. [35]

    Ada-adapter:fast few-shot style personlization of diffusion model with pre-trained image encoder.CoRR, abs/2407.05552, 2024

    Jia Liu, Changlin Li, Qirui Sun, Jiahui Ming, Chen Fang, Jue Wang, Bing Zeng, and Shuaicheng Liu. Ada-adapter:fast few-shot style personlization of diffusion model with pre-trained image encoder.CoRR, abs/2407.05552, 2024

    2024 arXiv

  28. [36]

    Tiny imagenet

    mnmoustafa and Mohammed Ali. Tiny imagenet. https://kaggle.com/competitions/ tiny-imagenet, 2017. Kaggle

    2017

  29. [37]

    The conceptarc benchmark: Evaluating understanding and generalization in the arc domain.Trans

    Arsenii Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain.Trans. Mach. Learn. Res., 2023, 2023

    2023

  30. [38]

    Riff: Learning to rephrase inputs for few-shot fine-tuning of language models.arXiv preprint arXiv:2403.02271, 2024

    Saeed Najafi and Alona Fyshe. Riff: Learning to rephrase inputs for few-shot fine-tuning of language models.arXiv preprint arXiv:2403.02271, 2024

    2024 arXiv

  31. [39]

    Dia: A tts model capable of generating ultra-realistic dialogue in one pass

    Nari Labs. Dia: A tts model capable of generating ultra-realistic dialogue in one pass. https: //github.com/nari-labs/dia, 2025. Accessed: 2025-05-12

    2025

  32. [40]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. InECCV, 2012

    2012

  33. [41]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  34. [42]

    MIT Press, 2022

    Thomas Parr, Giovanni Pezzulo, and Karl J Friston.Active inference: the free energy principle in mind, brain, and behavior. MIT Press, 2022

    2022

  35. [43]

    Kakade, Eran Malach, and Samy Jelassi

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham M. Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks.CoRR, abs/2410.13025, 2024

    2024 arXiv

  36. [44]

    Worldsimbench: Towards video generation models as world simulators.CoRR, abs/2410.18072, 2024

    Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, Lei Bai, Wanli Ouyang, and Ruimao Zhang. Worldsimbench: Towards video generation models as world simulators.CoRR, abs/2410.18072, 2024

    arXiv

  37. [45]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. InInternational Conference on Computer Vision (ICCV) 2021, 2021

    2021

  38. [46]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  39. [47]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.CoRR, abs/2208.12242, 2022

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.CoRR, abs/2208.12242, 2022. 12

    2022 arXiv

  40. [48]

    Crossing the uncanny valley of voice

    Sesame. Crossing the uncanny valley of voice. https://www.sesame.com/research/ crossing_the_uncanny_valley_of_voice, n.d. Accessed: 2025-05-12

    2025

  41. [49]

    Emer- gent correspondence from image diffusion.Advances in Neural Information Processing Systems, 36:1363–1389, 2023

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion.Advances in Neural Information Processing Systems, 36:1363–1389, 2023

    2023

  42. [50]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNIPS, pages 6000–6010, 2017

    2017

  43. [51]

    Zero-shot video semantic segmentation based on pre- trained diffusion models.CoRR, abs/2405.16947, 2024

    Qian Wang, Abdelrahman Eldesokey, Mohit Mendiratta, Fangneng Zhan, Adam Kortylewski, Christian Theobalt, and Peter Wonka. Zero-shot video semantic segmentation based on pre- trained diffusion models.CoRR, abs/2405.16947, 2024

    2024 arXiv

  44. [52]

    Images speak in images: A generalist painter for in-context visual learning

    Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830–6839, 2023

    2023

  45. [53]

    In-context learning unlocked for diffusion models.Advances in Neural Information Processing Systems, 36:8542–8562, 2023

    Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou, et al. In-context learning unlocked for diffusion models.Advances in Neural Information Processing Systems, 36:8542–8562, 2023

    2023

  46. [54]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models.Co...

    2022 arXiv

  47. [55]

    Omnigen: Unified image generation.CoRR, abs/2409.11340, 2024

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shut- ing Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation.CoRR, abs/2409.11340, 2024

    2024 arXiv

  48. [56]

    What matters when repurposing diffusion models for general dense perception tasks?The Thirteenth International Conference on Learning Representations (ICLR), 2025

    Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. What matters when repurposing diffusion models for general dense perception tasks?The Thirteenth International Conference on Learning Representations (ICLR), 2025

    2025

  49. [57]

    Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, André Barreto, Pieter Abbeel, and Dale Schuurmans

    Sherry Yang, Jacob C. Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, André Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. CoRR, abs/2402.17139, 2024

    2024 arXiv

  50. [58]

    Open-sora: Democratizing efficient video production for all

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404, 2024

    2024 arXiv

  51. [59]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017

    2017

  52. [60]

    Semantic understanding of scenes through the ade20k dataset.International Journal of Computer Vision, 127(3):302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset.International Journal of Computer Vision, 127(3):302–321, 2019. 13 Appendix A Experimental Details In this section, we...

    2019

  53. [61]

    This is impractical, as it requires 17 visually distinct colors

    Color-based encoding:Assigning a unique color to each keypoint and decoding them from the RGB image. This is impractical, as it requires 17 visually distinct colors. Given that our fine-tuning process may induce color shifts, maintaining this separation is unreliable. Moreover...

  54. [62]

    However, this increases inference costs by nearly a factor of 20, which is prohibitive for our setup where inference speed is already a bottleneck

    Per-keypoint prediction:Generating a separate output for each keypoint and aggregating them afterward. However, this increases inference costs by nearly a factor of 20, which is prohibitive for our setup where inference speed is already a bottleneck. Importantly, our goal is n...

  55. [63]

    Given a predicted pose, we use the color channels to segment the figure into four main components: head, torso, arms, and legs

  56. [64]

    For each connected component, we compute its centroid

  57. [65]

    We determine the best possible matching between predicted centroids and the corresponding centroids derived from annotated data for each body part

  58. [66]

    segment animals

    A match is considered valid if the Euclidean distance between the predicted and annotated centroids is below a fixed multiple (1.5 in our experiments) of the average inter-head distance, computed from annotated keypoints. If no heads are available, we use a default threshold o...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.