Pith. sign in

REVIEW 3 major objections 4 minor 52 references

Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that adding a cross-frame representation alignment loss to LoRA fine-tuning of video diffusion models improves both visual fidelity and cross-frame semantic coherence over vanilla fine-tuning and per-frame REPA.

desk verdict A cheap and useful fine-tuning trick, but the paper's causal story is undercut by a missing matched-regularization control. read the letter →

arxiv 2506.09229 v2 pith:6SBCXXEL submitted 2025-06-10 cs.CV

Sungwon Hwang , Hyojin Jang , Kinam Kim , Minho Park , Jaegul Choo This is my paper
classification cs.CV
keywords cross-framerepresentationalignmentvideodiffusionmodelsfine-tuningLoRADINOv2semanticconsistencydistillationregularizationnovelviewsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning large video diffusion models on small attribute-specific datasets is unstable: models trained with only a score-matching loss converge slowly and often break object identity across frames. The paper claims that a regularization term pulling each frame's internal representation toward the pretrained visual features of its neighboring frames fixes much of this. Its cross-frame alignment loss adds an exponentially weighted similarity between a frame's projected hidden state and the clean-frame features of frames at distance $d$, on top of the usual per-frame alignment. Trained for 3,000 iterations on a single A100, CREPA improves VBench consistency metrics, FVD/IS, novel-view-synthesis quality, and user preference over vanilla fine-tuning and over the per-frame REPA baseline across seven datasets.

What carries the argument

The load-bearing mechanism is a distillation regularizer applied to a diffusion-transformer encoder layer located by linear probing: a small MLP $h_\phi$ projects the noisy hidden state $h^f_t$ into the feature space of DINOv2, and the loss pulls that projection toward the current frame's clean feature and, with weight $e^{-|k-f|/\tau}$, toward features of frames at distances $\pm d$ ($d=1$, $\tau=1$). The exponential decay ensures nearby frames dominate, so the target is the local direction of the video's feature trajectory rather than a global average. This turns the fine-tuning objective into $L = L_{\text{score}} + \lambda L_{\text{align}}$ with $\lambda=0.5$ for CogVideoX and $\lambda=1$ for Hunyuan.

What would settle it

Train the same video diffusion model under CREPA with adjacency $d=0$, $d=1$, and $d=3$ while holding all else fixed. If $d=3$ does not increase CKNNA similarity to adjacent frames over $d=1$, or if subject consistency on VBench decreases with larger $d$, then the exponential neighbor weighting is not tracking a semantic manifold. A second direct test: replace the adjacent-frame term with a single temporally averaged DINOv2 feature; if that matches or exceeds CREPA, the benefit is smoothing rather than semantic alignment.

Watch

Extended reading notes

Core claim

The central discovery is that per-frame representation alignment (REPA*) under-constrains video diffusion fine-tuning: a noisy hidden state can match its own frame's clean feature while sitting at an arbitrary distance from adjacent frames' features, and this freedom produces temporally inconsistent generations. CREPA resolves the under-constraint by maximizing $\text{sim}(\bar{y}_f, h_\phi(h^f_t)) + \sum_{k\in\{f\pm d\}} e^{-|k-f|/\tau}\, \text{sim}(\bar{y}_k, h_\phi(h^f_t))$ for each frame $f$, with $\bar{y}_k = E(x^k_0)$ extracted by a pretrained DINOv2 encoder from clean neighboring frames. In the authors' experiments this raises CKNNA similarity to adjacent frames while maintaining current-frame alignment, and translates into better subject/background consistency and lower FVD (281.2 vs 291.4 for REPA* and 305.5 for vanilla on their DL3DV setup) on CogVideoX-5B and Hunyuan Video.

Load-bearing premise

The load-bearing premise is that the DINOv2 features of nearby clean frames form a temporally coherent manifold, so pulling a noisy frame's hidden state toward them improves semantic tracking; if those features are not coherent across adjacent frames, the extra term would just average frames together and smooth away distinct content.

Editorial extensions

If this is right

  • Applied to any DiT-based video diffusion model, CREPA needs only the chosen encoder layer and a small MLP, so it adds no inference-time cost and works with existing LoRA pipelines.
  • With 3,000 iterations (9-13 hours on one A100) it enables attribute-specific fine-tuning within a user-level budget, which the paper demonstrates on cartoon style, physical interactions, and static 3D scenes.
  • Because it improves frame-to-frame semantic coherence, downstream tasks that consume generated video—camera-pose estimation and 3D reconstruction with COLMAP and 3DGS—also improve, as shown by PSNR/SSIM/LPIPS on novel view synthesis.
  • The regularization is attribute-agnostic: the same $d=1$, $\tau=1$ setting works across seven datasets, suggesting it can be bundled into fine-tuning toolkits as a default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the DINOv2 feature trajectory is the right temporal manifold, CREPA should also help during pre-training of video foundation models, not just fine-tuning; the authors note this as future work.
  • The exponential weighting suggests a clean experiment the paper does not run: as $\tau \to 0$ the objective becomes local frame matching, and as $\tau \to \infty$ it becomes a uniform temporal average; performance should peak at intermediate $\tau$ if the manifold story holds, and would reveal whether the gains are from smoothing or semantic tracking.
  • Since layer search is done once per model and shared, CREPA can be composed with other fine-tuning regularizers, such as long-context tuning or attention modifications, as an orthogonal consistency term.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Cross-frame Representation Alignment (CREPA), a regularization objective for fine-tuning video diffusion models (VDMs) with parameter-efficient methods such as LoRA. Starting from REPA*, a direct adaptation of the image-domain REPA objective that aligns each frame's hidden state with its own DINOv2 feature, the authors add terms that align the hidden state of a frame with the pretrained features of adjacent frames, weighted by an exponential decay in temporal distance. The loss is combined with the standard score-matching objective. The method is evaluated on CogVideoX-5B and Hunyuan Video over seven datasets (cartoon style, physical interactions, 3D scenes, movie scenes) using VBench, FVD/IS, novel view synthesis, and a user study, with comparisons against vanilla fine-tuning and REPA*. The paper reports consistent improvements, especially on semantic-consistency metrics.

Significance. If the central claim holds, CREPA is a simple and practical regularizer that improves both visual fidelity and cross-frame semantic coherence in VDM fine-tuning, with a single-GPU training budget and no architectural changes. The paper's strengths include a clearly specified objective (Eq. 6), experiments across two large models and multiple datasets, and the inclusion of both automatic metrics and human evaluation. The proposed method is easy to implement on top of existing REPA-style losses, which could make it immediately useful. However, the paper's mechanistic claim—that the cross-frame terms specifically, rather than simply a stronger overall alignment signal, are responsible for the gains—is not yet supported by the evidence. The main empirical results lack error bars and significance tests, and the diagnostic in Fig. 2 measures exactly the quantity the loss maximizes, so it cannot independently validate the mechanism. These issues make the contribution promising but not yet fully established.

major comments (3)
  1. [Sec. 4.2, Eq. (6) and Sec. 5.1 (implementation details)] The cross-frame terms are not compared against a matched-strength REPA* baseline. With d=1 and tau=1, the total weight on the two neighbor terms is 2e^{-1} ≈ 0.736 times the current-frame term, so the overall alignment weight is about 1.736 times that of Eq. (4). Since the effective temporal adjacency after 4× latent compression is only a few frames, adjacent-frame DINOv2 features may be near-duplicates of the current-frame feature; if so, CREPA reduces to REPA* with a larger lambda. The paper does not report the similarity between y_f and y_k for k=f±d on its training data, nor does it include a REPA* run with lambda scaled by 1.736 (or a normalized version of L_align). Without this control, the improvements in Tables 1–3 and Fig. 25 do not establish that cross-frame information, rather than stronger alignment regularization, drives the gains. This is load-bearing for the paper's central claim.
  2. [Sec. 4.2, Fig. 2, and Appendix B] The empirical observation in Fig. 2 measures CKNNA between hidden states and pretrained features of current and adjacent frames, which is precisely the quantity the CREPA loss (Eq. 6) maximizes. An increase in this metric is expected by construction and therefore cannot serve as independent evidence that the method improves temporal semantic structure. The paper should provide a diagnostic that is not directly optimized, such as the redundancy of adjacent-frame features (e.g., cosine similarity between y_f and y_{f±d}), or an ablation in which the neighbor terms are replaced by current-frame features with a total weight matched to CREPA.
  3. [Sec. 5.2, Tables 1–3, and Appendix E] All quantitative claims are reported without error bars, confidence intervals, or significance tests. Many VBench differences are small (e.g., 0.5145 vs 0.5207 for Aesthetic Quality on Hunyuan Video, Table 1), and the FVD/IS evaluation is limited to a single dataset (DL3DV) with one I2V model. The user study uses 20 samples per criterion with no reported inter-annotator agreement or statistical test. Since the paper's central claim is empirical, the absence of variance estimates makes it difficult to assess whether the observed improvements are reliable beyond the specific checkpoint and seed used.
minor comments (4)
  1. [Sec. 5.2, Quantitative Results] The text says 'as shown in Table 4' but the VBench results for the main comparison are in Table 1; Table 4 in the appendix reports layer-search ablations. The reference should be corrected.
  2. [Fig. 1 caption] There is a typo: 'beter' should be 'better'.
  3. [Appendix E] The descriptions of Text–Video Alignment and Visual Quality are duplicated; criterion (1) says it reflects frame-level clarity and artifacts, which is the same wording as criterion (2). This should be clarified.
  4. [Sec. A] The linear probing dataset is constructed from all fine-tuning datasets used in the paper, which may leak information about the target datasets into the layer-selection procedure. The authors should clarify whether the linear probes are trained on held-out videos or only on the training splits, and whether this selection affects the reported results.

Circularity Check

1 steps flagged · score 3.0 of 10

Fig. 2 measures exactly what Eq. (6) maximizes, so that mechanistic evidence is circular; the headline benchmarks are external and keep the paper's central claim largely independent.

  1. self definitional [Sec. 4.2, Eq. (6) and Fig. 2]
    "To do so, we define a simple yet novel regularization objective for alignment: Lalign(ϕ) := −E[ Σ_f sim(ȳ_f, h_ϕ(h^f_t)) + Σ_{k∈K} e^{−|k−f|/τ} sim(ȳ_k, h_ϕ(h^f_t)) ], where K={f−d, f+d} ... We first quantify the degree of cross-frame representation alignment ... CREPA yields higher similarity between the current hidden states and adjacent pretrained features, as shown in Fig. 2-(a),(b)."

    The CKNNA diagnostic in Fig. 2 measures essentially the quantity that the CREPA loss in Eq. (6) directly maximizes: the similarity between current-frame hidden states h^f_t and adjacent-frame DINOv2 features ȳ_{f±d}. Since the loss contains those adjacent-frame terms with positive weights e^{−|k−f|/τ}, a CREPA-trained model must exhibit higher such similarity than a REPA*-trained model, so observing this increase is expected by construction. The passage therefore cannot independently validate the hypothesis that cross-frame regularization improves semantic consistency. This circularity is confined to the mechanistic motivation; the paper's headline results on VBench, FVD/IS, NVS, and the user study use external metrics and provide independent support for the central claim.

full rationale

The paper's central claim — that CREPA improves visual fidelity and cross-frame semantic coherence when fine-tuning VDMs — is supported by external benchmarks (VBench, FVD/IS, novel-view synthesis, and a user study) that are not part of the training objective. These evaluations are self-contained against the method's inputs, and they are not defined in terms of the CREPA loss. The one clear circular element is Fig. 2: the paper presents a CKNNA-based diagnostic showing that CREPA increases alignment to adjacent-frame DINOv2 features. But that is precisely the quantity Eq. (6) maximizes, so the result is a direct consequence of the loss rather than an independent confirmation of the mechanism. No other pattern of circularity is present. REPA* is a straightforward re-implementation of an existing external method and is used as a baseline, not as a self-citation carrying the paper's argument. The use of DINOv2-g as the pretrained encoder follows REPA but is an external, openly available model; no uniqueness theorem or author-generated theorem is invoked to force the design. The choice of hidden layer is made by linear probing plus VBench evaluation, which is an external criterion. A possible experimental weakness — that with d=1 and τ=1 the cross-frame terms approximately re-weight the REPA loss when adjacent DINOv2 features are near-duplicates — is a missing control or model-selection concern, not a definitional reduction that the paper itself asserts. Because the central claim survives on independent external benchmarks, and only the supporting diagnostic is circular, an overall score of 3 reflects a low-to-moderate circularity burden.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method adds a new loss term with several hand-set hyperparameters (λ, d, τ, layer index) and depends on the assumption that DINOv2 features of clean frames form a reliable temporal target for noisy hidden states. No new physical entities or conceptual objects are introduced.

free parameters (5)
  • λ (alignment weight) = 0.5 for CogVideoX-5B, 1 for Hunyuan Video
    Chosen empirically based on REPA* performance (Sec 5.1); balances score loss and alignment loss.
  • d (adjacency window) = 1
    Hand-set; with 4x temporal VAE compression this covers up to 4 adjacent latent frames (Sec 5.1).
  • τ (temperature) = 1
    Hand-set; controls exponential decay of cross-frame weights in Eq. (6).
  • hidden layer index = 8th layer for CogVideoX, 10th layer for Hunyuan
    Selected via linear probing and VBench evaluation on REPA* models (Appendix A).
  • LoRA rank/alpha/learning rate = not reported
    Implementation details omit LoRA rank, alpha, and optimizer settings, which are required to reproduce the fine-tuning.
assumptions (4)
  • domain assumption DiT blocks form a denoising autoencoder with an encoder and decoder split
    Motivated by DDAE [47] and REPA [50]; used to justify distilling the encoder hidden states (Sec 4.1).
  • domain assumption DINOv2 features of clean frames are semantically meaningful and temporally coherent enough to serve as distillation targets
    The pretrained image encoder is used to define both per-frame and cross-frame targets; if adjacent-frame features are noisy or inconsistent, CREPA's objective is misguided (Sec 4.2).
  • ad hoc to paper Aligning hidden states to adjacent-frame features improves semantic consistency without collapsing representations to a trivial constant
    This is the central heuristic behind Eq. (6); the paper provides empirical evidence but no theoretical justification.
  • domain assumption The selected encoder layer's hidden states capture the semantic content needed for alignment
    Layer choice is based on linear probing accuracy (Appendix A), but the proxy is not verified against downstream video quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models." pith.science (2026). https://pith.science/paper/6SBCXXEL

@misc{pith2026250609229,
  author       = {Pith},
  title        = {Pith review of: Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SBCXXEL}},
  note         = {Machine review of arXiv:2506.09229}
}
read the original abstract

Fine-tuning Video Diffusion Models (VDMs) at the user level to generate videos that reflect specific attributes of training data presents notable challenges, yet remains underexplored despite its practical importance. Meanwhile, recent work such as Representation Alignment (REPA) has shown promise in improving the convergence and quality of DiT-based image diffusion models by aligning, or assimilating, its internal hidden states with external pretrained visual features, suggesting its potential for VDM fine-tuning. In this work, we first propose a straightforward adaptation of REPA for VDMs and empirically show that, while effective for convergence, it is suboptimal in preserving semantic consistency across frames. To address this limitation, we introduce Cross-frame Representation Alignment (CREPA), a novel regularization technique that aligns hidden states of a frame with external features from neighboring frames. Empirical evaluations on large-scale VDMs, including CogVideoX-5B and Hunyuan Video, demonstrate that CREPA improves both visual fidelity and cross-frame semantic coherence when fine-tuned with parameter-efficient methods such as LoRA. We further validate CREPA across diverse datasets with varying attributes, confirming its broad applicability.

Figures

Figures reproduced from arXiv: 2506.09229 by the authors.

Figure 1
Figure 1. Videos generated by CogVideoX-5B [48] fine-tuned on the Disney [45] dataset. Each model is fine-tuned with: no regularization (Vanilla), REPA* (our implementation of REPA [50] to video diffusion models), and CREPA (ours). Our model yields beter text reflectivity and semantic consistency across frames compared to the baselines. Abstract Fine-tuning Video Diffusion Models (VDMs) at the user level to generate videos th… view at source ↗
Figure 2
Figure 2. CKNNA [21] between hidden states of a frame to pretrained features, or representation alignment, to (a) preceding, (b) future, and (c) current frames. CREPA promotes alignment to adjacent frames, while maintaining or even slightly improving the alignment to the current frame. where ϵθ denotes a neural network trained to predict the noise added at time t, and xt is generated from x0 using the forward noising process.… view at source ↗
Figure 3
Figure 3. Overview of CREPA and comparison to REPA*. By aligning hidden states not only to the current but also to the adjacent pretrained features, CREPA further guides the hidden state representations toward the temporal manifold formed by the sequence of pretrained features. 4.2 Cross-frame Representation Alignment However, due to the inherent nature of DAEs training on noisy inputs and jointly training hθ during fine-tuni… view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Videos generated by Hunyuan Video [23] fine-tuned on Crush [11] dataset. CREPA enhances convergence relative to Vanilla by better learning the physical attribute of the data. Also, CREPA yields better semantic consistency compared to REPA*. Text prompt reported in Appe…
Figure 5
Figure 5. Figure 5: Videos generated by CogVideoX-5B [48] fine-tuned on DL3DV [26] dataset. Compared to REPA*, CREPA yields more semantically consistent objects in video across the frames. Meanwhile, Vanilla yields disappearing road in the background. Text prompt reported in Appendix D. V…
Figure 6
Figure 6. Figure 6: Videos generated by CogVideoX-5B [48] fine-tuned on Tom and Jerry [46] dataset. CREPA improves convergence and semantic consistency over Vanilla and REPA*, respectively. Text prompt reported in Appendix D. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Novel view synthesis on 3D scenes reconstructed with generated videos. Spatially inconsistent videos cause inaccurate camera pose estimation from COLMAP [37], as well as imprecise supervision for rendering loss while training 3DGS [22]. Models FVD ↓ IS ↑ Vanilla 305.54…
Figure 8
Figure 8. Figure 8: Linear probing results on CogVideoX-5B [48] and Hunyuan Video [23]. We regard the layers that precede the higher classification accuracy as the layers of the diffusion encoder. Finding optimal hidden state layers for REPA*. Having identified the diffusion encoder for e…
Figure 9
Figure 9. Figure 9: Videos generated by Hunyuan Video [23] fine-tuned on Crush [11] dataset. REPA* Dataset Vanilla CREPA (ours) [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Videos generated by CogVideoX-5B [48] fine-tuned on Crush [11] dataset. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Videos generated by Hunyuan Video [23] fine-tuned on Cakeify [10] dataset. REPA* Dataset Vanilla CREPA (ours) [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Videos generated by Hunyuan Video [23] fine-tuned on Cakeify [10] dataset. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Videos generated by Hunyuan Video [23] fine-tuned on Cakeify [10] dataset. Dataset Vanilla REPA* CREPA (ours) Dataset [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Videos generated by CogVideoX-5B [48] fine-tuned on Cakeify [10] dataset. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Videos generated by CogVideoX-5B [48] fine-tuned on Cakeify [10] dataset. REPA* Vanilla CREPA (ours) Dataset [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Videos generated by Hunyuan Video [23] fine-tuned on Squish [12] dataset. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: Videos generated by Hunyuan Video [23] fine-tuned on Squish [12] dataset. REPA* Dataset Vanilla CREPA (ours) [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Videos generated by CogVideoX-5B [48] fine-tuned on Squish [12] dataset. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 19
Figure 19. Figure 19: Videos generated by CogVideoX-5B [48] fine-tuned on Disney [45] dataset. REPA* Dataset Vanilla CREPA (ours) [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]
Figure 20
Figure 20. Figure 20: Videos generated by CogVideoX-5B [48] fine-tuned on Disney [45] dataset. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_20.png]
Figure 21
Figure 21. Figure 21: Videos generated by CogVideoX-5B [48] fine-tuned on Tom and Jerry [46] dataset. Vanilla REPA* CREPA (ours) Dataset [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: Videos generated by CogVideoX-5B [48] fine-tuned on DL3DV [26] dataset. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]
Figure 23
Figure 23. Figure 23: Videos generated by Hunyuan Video [23] fine-tuned on Scenes [10] dataset. Dataset Vanilla REPA* CREPA (ours) [PITH_FULL_IMAGE:figures/full_fig_p021_23.png]
Figure 24
Figure 24. Figure 24: Videos generated by Hunyuan Video [23] fine-tuned on Scenes [4] dataset. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_24.png]
Figure 4
Figure 4. Figure 4: • A colorful puzzle ball is being crushed by a large metal cylinder, which flattens the objects as if they were under a hydraulic press [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 9
Figure 9. Figure 9: • A rubber boot is placed on the platform. As the hydraulic press moves down, the boot compresses and wrinkles before bursting at the seams [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 18
Figure 18. Figure 18: • A terracotta pot with a visible crack sits centered on a white surface, bathed in sunlight. Two hands enter the frame, positioning themselves around the pot. The hands then begin to press and mold the pot, the clay beginning to rise from the opening. The clay is res…
Figure 22
Figure 22. Figure 22: • The video explores a traditional Chinese museum, starting with a serene interior featuring a glass display case with golden figurines and a bamboo wall with calligraphy. As the video continues, various scenes show the museum’s historical artifacts, including a lion …
Figure 25
Figure 25. Figure 25: Results of user study conducted over Vanilla, REPA*, and CREPA. CREPA is preferred by human participants over REPA* and Vanilla on all criteria. E User Study To evaluate the quality of the generated videos, we conducted a human evaluation using 20 samples per criterio…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 24 canonical work pages

  1. [1]

    Agarwal, A

    N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y . Chen, Y . Cui, Y . Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    arXiv 2025

  2. [2]

    Baker, I

    B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos.Advances in Neural Information Processing Systems, 35:24639–24654, 2022

    work page 2022

  3. [3]

    X. Bi, J. Lu, B. Liu, X. Cun, Y . Zhang, W. Li, and B. Xiao. Customttt: Motion and appearance customized video generation via test-time training. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1871–1879, 2025

    work page 2025

  4. [4]

    Scenes dataset.https://huggingface.co/datasets/bigdata-pw/scenes, 2025

    bigdata pw. Scenes dataset.https://huggingface.co/datasets/bigdata-pw/scenes, 2025

    work page 2025

  5. [5]

    Blattmann, T

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. English, V . V oleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  6. [6]

    Blattmann, R

    A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563–22575, 2023

    2023

  7. [7]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  8. [8]

    F. Chen, S. Zhao, C. Xu, and L. Lan. Jointtuner: Appearance-motion adaptive joint training for customized video generation.arXiv preprint arXiv:2503.23951, 2025

    arXiv 2025

Show all 52 references
  1. [9]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  2. [10]

    Cakeify smol dataset

    Finetrainers. Cakeify smol dataset. https://huggingface.co/datasets/finetrainers/ cakeify-smol, 2024

    2024

  3. [11]

    Crush smol dataset

    Finetrainers. Crush smol dataset. https://huggingface.co/datasets/finetrainers/ crush-smol, 2024

    2024

  4. [12]

    Squish pika dataset

    Finetrainers. Squish pika dataset. https://huggingface.co/datasets/finetrainers/ squish-pika, 2024

    2024

  5. [13]

    Y . Guo, C. Yang, Z. Yang, Z. Ma, Z. Lin, Z. Yang, D. Lin, and L. Jiang. Long context tuning for video generation.arXiv preprint arXiv:2503.10589, 2025

    2025 arXiv

  6. [14]

    Gururangan, A

    S. Gururangan, A. Marasovi´c, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks.arXiv preprint arXiv:2004.10964, 2020

    2004 arXiv

  7. [15]

    H. He, Y . Xu, Y . Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang. Cameractrl: Enabling camera control for video diffusion models. InThe Thirteenth International Conference on Learning Representations

  8. [16]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  9. [17]

    W. Hong, W. Wang, M. Ding, W. Yu, Q. Lv, Y . Wang, Y . Cheng, S. Huang, J. Ji, Z. Xue, et al. Cogvlm2: Visual language models for image and video understanding.arXiv preprint arXiv:2408.16500, 2024

    2024 arXiv

  10. [18]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  11. [19]

    X. Hu, X. Xu, and Y . Shi. How to efficiently adapt large segmentation model (sam) to medical images. arXiv preprint arXiv:2306.13731, 2023

    2023 arXiv

  12. [20]

    Huang, Y

    Z. Huang, Y . He, J. Yu, F. Zhang, C. Si, Y . Jiang, Y . Zhang, T. Wu, Q. Jin, N. Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024

    2024

  13. [21]

    M. Huh, B. Cheung, T. Wang, and P. Isola. The platonic representation hypothesis.CoRR, 2024. 10

    2024

  14. [22]

    Kerbl, G

    B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph., 42(4):139–1, 2023

    2023

  15. [23]

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  16. [24]

    Kornblith, M

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMLR, 2019

    2019

  17. [25]

    Liang, J

    H. Liang, J. Cao, V . Goel, G. Qian, S. Korolev, D. Terzopoulos, K. N. Plataniotis, S. Tulyakov, and J. Ren. Wonderland: Navigating 3d scenes from a single image.arXiv preprint arXiv:2412.12091, 2024

    2024 arXiv

  18. [26]

    L. Ling, Y . Sheng, Z. Tu, W. Zhao, C. Xin, K. Wan, L. Yu, Q. Guo, Z. Yu, Y . Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160–22169, 2024

    2024

  19. [27]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  20. [28]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  21. [29]

    Y . Liu, X. Cun, X. Liu, X. Wang, Y . Zhang, H. Chen, Y . Liu, T. Zeng, R. Chan, and Y . Shan. Evalcrafter: Benchmarking and evaluating large video generation models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22139–22149, 2024

    2024

  22. [30]

    Z. Lu, H. Yang, D. Xu, B. Li, B. Ivanovic, M. Pavone, and Y . Wang. Lora3d: Low-rank self-calibration of 3d geometric foundation models.arXiv preprint arXiv:2412.07746, 2024

    2024 arXiv

  23. [31]

    N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden-Eijnden, and S. Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. InEuropean Conference on Computer Vision, pages 23–40. Springer, 2024

    2024

  24. [32]

    Mahendran and A

    A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5188–5196, 2015

    2015

  25. [33]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision.Transactions on Machine Learning Research

  26. [34]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  27. [35]

    L. I. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms.Physica D: nonlinear phenomena, 60(1-4):259–268, 1992

    1992

  28. [36]

    N. Ruiz, Y . Li, V . Jampani, Y . Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–22510, 2023

    2023

  29. [37]

    J. L. Schönberger and J.-M. Frahm. Structure-from-motion revisited. InConference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  30. [38]

    V . Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y . Li, and V . Jampani. Ziplora: Any subject in any style by effectively merging loras. InEuropean Conference on Computer Vision, pages 422–438. Springer, 2024

    2024

  31. [39]

    Sohl-Dickstein, E

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. PMLR, 2015

    2015

  32. [40]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  33. [41]

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  34. [42]

    A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025. 11

    2025 arXiv

  35. [43]

    S. Wang, V . Leroy, Y . Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697– 20709, 2024

    2024

  36. [44]

    Z. Wang, L. Zhao, and W. Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7677–7689, 2023

    2023

  37. [45]

    Disney video generation dataset

    Wild-Heart. Disney video generation dataset. https://huggingface.co/datasets/Wild-Heart/ Disney-VideoGeneration-Dataset, 2024

    2024

  38. [46]

    Tom and jerry video generation dataset

    Wild-Heart. Tom and jerry video generation dataset. https://huggingface.co/datasets/ Wild-Heart/Tom-and-Jerry-VideoGeneration-Dataset, 2024

    2024

  39. [47]

    Xiang, H

    W. Xiang, H. Yang, D. Huang, and Y . Wang. Denoising diffusion autoencoders are unified self-supervised learners. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15802– 15812, 2023

    2023

  40. [48]

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.CoRR, 2024

    2024

  41. [49]

    R. A. Yeh, C.-Y . Chen, and A. G. Schwing. Total variation optimization layers for computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12345–12354, 2022

    2022

  42. [50]

    S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie. Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940, 2024

    2024 arXiv

  43. [51]

    Y . Yue, A. Das, F. Engelmann, S. Tang, and J. E. Lenssen. Improving 2d feature representations by 3d-aware fine-tuning. InEuropean Conference on Computer Vision, pages 57–74. Springer, 2024

    2024

  44. [52]

    Zheng, X

    Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y . Zhou, T. Li, and Y . You. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024. 12 Appendix A Locating layers for hidden-state retrieval Linear probing for locating denoising e...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.