Pith. sign in

REVIEW 4 major objections 6 minor 42 references

Driving norms can steer a frozen video world model’s planned path at sampling time, with no retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 09:17 UTC pith:WOBIIJLZ

load-bearing objection Clean positive result that sampling-time energy steers the ego plan of a frozen joint video–trajectory world model, plus an honest negative that the video does not follow; the end-to-end claim is unfinished and the guidance ablations are thin. the 4 major comments →

arxiv 2607.10781 v1 pith:WOBIIJLZ submitted 2026-07-12 cs.CV cs.RO

Is Energy Guidance All You Need? Training-Free Norm Injection for Driving World Models

Xiyan Su , Frank Diermeyer , Markus Lienkamp This is my paper
classification cs.CV cs.RO
keywords driving world modelstraining-free guidanceenergy guidancerectified flowvideo diffusionego trajectory controlsampling-time norms
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large video-diffusion world models for driving look realistic but are hard to control: adding a traffic rule usually means retraining the backbone or baking in hand-built maps and boxes. This paper asks whether that training is required at all. It shows that a frozen rectified-flow model that jointly denoises future video and an ego trajectory can have its planned trajectory steered purely at sampling time by differentiable energy functions that encode driving norms. In a concrete test, an energy that attracts the plan to a counterfactual brake target pulls the guided trajectory to about 1.3 m residual while the unguided plan stays 13.6 m away across 35 clips. The same experiment also shows the current limit: the generated video does not yet follow the steered plan through joint attention, so end-to-end controllable rollouts still need deeper cross-stream coupling.

Core claim

A rectified-flow driving world model that jointly generates future video and a planned ego trajectory can have that planned trajectory steered entirely at sampling time by differentiable energy functions encoding driving norms, without any knowledge-specific retraining of the diffusion backbone. Concretely, energy guidance drives the plan to a counterfactual braking target (mean residual 1.3 m versus 13.6 m unguided) while the video stream remains essentially unmoved under the present architecture.

What carries the argument

Training-free energy guidance on the ego stream: form the one-step clean estimate of the trajectory, take a clipped energy-descent step in clean space, convert it back to a velocity correction, and apply an asymmetric schedule that is off near the start and end of sampling.

Load-bearing premise

The one-step clean trajectory guess, after a simple schedule and clip, is accurate enough for its energy gradient to steer the plan without pushing the sample off the model’s data manifold.

What would settle it

On the same 35 clips and noise seeds, disable the energy gradient while keeping the schedule and clip, or replace the brake energy with a pure random direction of matched magnitude; if the residual to the brake target no longer collapses from ~13.6 m to ~1.3 m, the reported steering is not energy guidance of the prior.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper asks whether a frozen rectified-flow driving world model that jointly generates future video and an ego trajectory can be made norm-compliant without knowledge-specific retraining. It injects differentiable energies only at sampling time via a one-step clean estimate and an affine score–velocity correction (Eqs. 2–3, Appendix A). On an Open-Sora 2.0 MM-DiT backbone post-trained with LoRA, a single MSE energy toward a counterfactual brake target steers the planned trajectory (mean residual 1.3 m vs 13.6 m unguided over 35 clips at λ=40; Figs. 3–5), while per-frame optical flow of the video remains statistically unchanged (Fig. 3 right, Fig. 6). The authors attribute the video failure to bypassed single-stream blocks and weak cross-stream coupling, and outline fixes and multi-norm composition as ongoing work.

Significance. If the trajectory result is genuine energy guidance of the generative prior rather than schedule/clip projection, the work shows that driving norms can be added post-deployment to a large video world model without retraining or layout conditioning—an attractive property for safety-critical simulation and planning. The experimental design (identical noise seeds, guided vs unguided, BEV and optical-flow metrics) is clean, the score–velocity derivation for rectified flow is carefully stated, and the authors release code, the energy library, and the 88M trainable adapters. The negative finding that the video does not follow is itself useful: it isolates cross-stream coupling as a concrete architectural requirement for end-to-end controllable rollouts. The present manuscript, however, demonstrates only a single simple energy and leaves the video stream uncontrolled, so the significance for full world-model controllability remains conditional on the planned architectural fix and on ablations that rule out projection artifacts.

major comments (4)
  1. Section III-B and Appendix A2 introduce an asymmetric λ(t) (zero near t=0 and t=1) and a hard velocity-scale clip on the per-step correction because ˆx0 is unreliable early and under multimodality. The only quantitative support for “energy guidance” is the residual reduction at a single operating point λ=40 (Fig. 3 left). No ablation of clip strength, schedule shape, or λ is reported. Under a sufficiently aggressive clip the update can act as a hard projection onto the brake target regardless of whether the energy gradient is informative or the sample remains on the data manifold. This is load-bearing for the central claim that the frozen prior is being steered by E; without those ablations the observed 1.3 m residual is consistent with both genuine guidance and schedule-plus-clip projection.
  2. Appendix B1 states that the 38 single-stream blocks are bypassed so that video and trajectory can be output simultaneously from the 19 double-stream blocks. Section IV and the Discussion then attribute the video’s failure to follow the steered plan to “cross-stream coupling” and propose re-routing through the single-stream blocks. Because the architecture was deliberately modified to remove the joint single-stream path, the optical-flow null result (Fig. 3 right, Fig. 6) may be an artifact of that modification rather than a general property of double-stream MM-DiT joint self-attention. The paper needs either (i) a controlled comparison with single-stream blocks restored, or (ii) a clearer framing that the current platform is not yet a joint generative world model under guidance.
  3. The abstract and contributions claim “driving norms” and “norm injection,” yet Section III-C and the experiments use only one energy: an MSE attraction to a hand-specified counterfactual brake trajectory (Eq. 4). Collision avoidance, lane keeping, speed limits, and comfort are deferred to “ongoing work” (Section V). A single target-tracking energy does not yet establish that a library of heterogeneous, possibly conflicting norms can be injected training-free; the multi-norm priority-hierarchy proposal remains untested. The empirical claim should be narrowed to what is demonstrated, or at least one additional, non-MSE norm should be shown.
  4. For a driving world model the value proposition is the video, not the trajectory alone (Introduction). The paper correctly reports that the video does not follow, but then the end-to-end controllability claim is left open. Either the architectural fix (re-enable single-stream joint denoising) must be completed and re-evaluated with the same counterfactual probe, or the manuscript should be reframed as a trajectory-steering result on a joint backbone with an explicit negative result on video coupling—rather than as evidence that “energy guidance” yields controllable world-model rollouts.
minor comments (6)
  1. Figure 1 / surrounding text contains the leftover placeholder “GebenSie hier eineFormelein.00”; remove it.
  2. Notation: xego_0 is written with mixed subscripts and the text stream is called both “planning (text) stream” and “Txt Stream”; unify terminology early (Section III-B / Fig. 2).
  3. Classifier-free guidance scale w ∈ [3,5] is used orthogonally to energy guidance (Appendix B4) but is not stated in the main experimental protocol; add it to Section IV for reproducibility.
  4. Fig. 3 caption says “λ=40 (mean ± std)” over 35 clips; report the exact clip selection criterion and how many total samples were discarded as “unusable” (Appendix C mentions “usable”).
  5. Related work: GuideFlow [16] is concurrent and also couples flow matching with energy for ego planning; a one-sentence contrast on training-free vs trained energy would help positioning.
  6. Eq. (3) uses max(t, t_min) in the denominator while the surrounding text discusses 1/t blow-up; define t_min explicitly.

Circularity Check

0 steps flagged

No circularity: empirical steering demonstration measured against an external counterfactual target; guidance formulas transfer standard score-based methods via affine equivalence without self-definition or fitted-as-prediction.

full rationale

The paper is a self-contained empirical methods contribution. The central claim (sampling-time energy guidance steers the ego trajectory of a frozen rectified-flow world model) is demonstrated by comparing guided vs. unguided samples drawn from identical noise against a hand-specified counterfactual brake target that was never seen in training (Eq. 4, Fig. 3 left: 1.3 m residual vs. 13.6 m). Success metrics (L2 distance to target, optical-flow difference) are external and independent of any fitted parameter. The guidance update itself (Eq. 3) and the score–velocity bridge (Appendix A, Eqs. 6–7) are direct transfers of known training-free guidance constructions (FreeDoM, Universal Guidance, Feng et al. flow-matching guidance) applied to the one-step clean estimate of a rectified-flow model; they do not redefine the target or the prior. Hyperparameters (λ schedule, clip, channel weights) modulate step size but do not define the evaluation metric by construction. No self-citations are load-bearing, no uniqueness theorem is imported, and no known empirical pattern is merely renamed. The acknowledged limitation that the video stream fails to follow is likewise measured externally (optical flow ≈ 0) rather than assumed. Consequently the derivation chain contains no circular step.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

Central claims rest on standard flow-matching identities, the usual training-free guidance approximation that replaces the likelihood score by −∇E(ˆx0), and a handful of hand-chosen guidance hyperparameters. No new physical entities are postulated. The architectural diagnosis (bypassed single-stream blocks cause the video lag) is an empirical hypothesis, not an axiom required for the trajectory result.

free parameters (5)
  • guidance scale λ (operating point λ=40)
    Hand-chosen strength of the energy step; results reported at this single operating point across 35 clips.
  • asymmetric guidance schedule λ(t) (zero near t=0 and t=1)
    Ad-hoc schedule to avoid early unreliable ˆx0 and 1/t blow-up; not derived from first principles.
  • per-channel energy weights wxy, wθ, wv
    Relative importance of position, heading, and speed in the MSE energy; chosen by authors.
  • velocity-scale clip on per-step guidance correction
    Fixed multiple of backbone velocity used to keep steps on-manifold; standard practice but free.
  • classifier-free guidance scale w ∈ [3,5]
    Orthogonal CFG on past conditioning; affects video sharpness and is selected at sampling.
axioms (4)
  • standard math Affine equivalence between rectified-flow velocity field and marginal score (Eq. 6), so score-based energy guidance transfers exactly to velocity.
    Invoked in Appendix A1; standard for Gaussian interpolants (Feng et al., Albergo et al.).
  • domain assumption Training-free approximation ∇xt log pt(y|xt) ≈ −∇ˆx0 E(ˆx0) with Jacobian treated as constant.
    Section III-B and Appendix A2; inherited from FreeDoM / DPS-style guidance literature; known to be least accurate early and under multimodality.
  • domain assumption Driving norms can be encoded as differentiable energies on the clean ego trajectory alone (here MSE to a brake target).
    Section III-C; same modeling choice as CTG / Diffusion-Planner, applied here to a joint world model.
  • ad hoc to paper Bypassing the 38 single-stream blocks is the primary reason steered trajectory fails to move video pixels.
    Section IV and V; architectural hypothesis offered as explanation and proposed fix, not yet ablated.

pith-pipeline@v1.1.0-grok45 · 16386 in / 3274 out tokens · 49317 ms · 2026-07-14T09:17:41.999451+00:00 · methodology

0 comments
read the original abstract

Driving world models built on large video-diffusion backbones generate realistic scenes but are hard to control: enforcing a traffic norm typically means retraining the backbone or conditioning it on hand-built layouts. We ask whether controllability requires training at all. Our experiment shows that a rectified-flow driving world model, which jointly generates future video and a planned ego trajectory, can have its planned trajectory steered entirely at sampling time by differentiable energy functions that encode driving norms, without knowledge-specific retraining of the diffusion backbone. Concretely, we demonstrate that a world model built on Open-Sora 2.0 MM-DiT backbone can be steered to brake at a counterfactual target by injecting energy guidance at sampling time. However, we find that the generated video does not yet follow the steered trajectory through the backbone's joint self-attention and identify the cross-stream coupling as a crucial requirement for end-to-end-controllable rollouts.

Figures

Figures reproduced from arXiv: 2607.10781 by Frank Diermeyer, Markus Lienkamp, Xiyan Su.

Figure 1
Figure 1. Figure 1: Overview. A frozen video world model (Open-Sora 2.0 MM￾DiT + LoRA) jointly denoises future video and an ego trajectory conditioned only on observed past without agent or map at training. Driving norms enter at sampling as energies on the clean ego estimate xˆ0 and are applied as a velocity correction to the ego stream. We study the question of whether the generated video is also steered through the backbon… view at source ↗
Figure 2
Figure 2. Figure 2: ), (·) ⋆ is the counterfactual target trajectory, which in this case simulates a slow lead vehicle in front of the ego, T the valid future frames, and wxy, wθ, wv per-channel weights. … Clean Past Frames x4 Noisy Frames x 8 DC-AE Encoder Noisy trajectory Pre-trained OpenSora 2 MM-DiT Backbone Double-stream Blocks x 19 Img tokens (𝒙, 𝒚, 𝜽, 𝒗) ∈ ℝ𝟒𝟎×𝟒 Img Stream Txt Stream DC-AE Decoder Predicted trajectory … view at source ↗
Figure 3
Figure 3. Figure 3: The trajectory steers, the video does not. Across 35 sampled clips at λ=40 (mean ± std). For each clip, we sample the platform twice from identical noise - unguided (factual) and with a counterfactual brake energy (counterfactual) - and compare. Left: L2 distance of each ego plan to the brake target per step; the counterfactual is driven to the target while the factual diverges. Right: per-frame optical-fl… view at source ↗
Figure 4
Figure 4. Figure 4: Per-clip BEV trajectories (1/2). All 35 usable sampled clips at λ=40. GT (black), unguided/factual (grey), counterfactual (orange), brake target (red dashed). The counterfactual plan tracks the braked target across clips [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-clip BEV trajectories (2/2). Continuation of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: One clip, aligned on frames. Top: ego speed - the counterfactual brakes hard while the factual holds speed. Bottom: per-frame optical-flow difference (counterfactual − factual), at zero: the trajectory steers but the video does not. (Plan steps are mapped onto the video-frame axis for alignment.) [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 11 linked inside Pith

  1. [1]

    GAIA-1: A generative world model for autonomous driving,

    A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado, “GAIA-1: A generative world model for autonomous driving,”arXiv preprint arXiv:2309.17080, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Vista: A generalizable driving world model with high fidelity and versatile controllability,

    S. Gao, J. Yang, L. Chen, K. Chitta, Y . Qiu, A. Geiger, J. Zhang, and H. Li, “Vista: A generalizable driving world model with high fidelity and versatile controllability,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  3. [3]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,

    Y . Wang, J. He, L. Fan, H. Li, Y . Chen, and Z. Zhang, “Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  4. [4]

    FreeDoM: Training-free energy-guided conditional diffu- sion model,

    J. Yu, Y . Wang, C. Zhao, B. Ghanem, and J. Zhang, “FreeDoM: Training-free energy-guided conditional diffu- sion model,” inIEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  5. [5]

    Uni- versal guidance for diffusion models,

    A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein, “Uni- versal guidance for diffusion models,”arXiv preprint arXiv:2302.07121, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Diffusion posterior sampling for general noisy inverse problems,

    H. Chung, J. Kim, M. T. McCann, M. L. Klasky, and J. C. Ye, “Diffusion posterior sampling for general noisy inverse problems,” inInternational Conference on Learning Representations (ICLR), 2023

    2023

  7. [7]

    On the guid- ance of flow matching,

    R. Feng, C. Yu, W. Deng, P. Hu, and T. Wu, “On the guid- ance of flow matching,”arXiv preprint arXiv:2502.02150, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Open-sora 2.0: Training a commercial- level video generation model in $200k,

    X. Penget al., “Open-sora 2.0: Training a commercial- level video generation model in $200k,”arXiv preprint arXiv:2503.09642, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    TFG: Unified training-free guidance for diffusion models,

    H. Ye, H. Lin, J. Han, M. Xu, S. Liu, Y . Liang, J. Ma, J. Zou, and S. Ermon, “TFG: Unified training-free guidance for diffusion models,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  10. [10]

    Compositional visual generation with composable dif- fusion models,

    N. Liu, S. Li, Y . Du, A. Torralba, and J. B. Tenenbaum, “Compositional visual generation with composable dif- fusion models,” inEuropean Conference on Computer Vision (ECCV), 2022

    2022

  11. [11]

    Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and MCMC,

    Y . Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Diele- man, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. Grathwohl, “Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and MCMC,” inInternational Conference on Machine Learn- ing (ICML), 2023

    2023

  12. [12]

    Manifold preserving guided diffusion,

    Y . He, N. Murata, C.-H. Lai, Y . Takida, T. Uesaka, D. Kim, W.-H. Liao, Y . Mitsufuji, J. Z. Kolter, R. Salakhutdinov, and S. Ermon, “Manifold preserving guided diffusion,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  13. [13]

    Flow straight and fast: Learning to generate and transfer data with rectified flow,

    X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” inInternational Conference on Learning Representations (ICLR), 2023

    2023

  14. [14]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” in International Conference on Learning Representations (ICLR), 2023

    2023

  15. [15]

    Stochastic interpolants: A unifying framework for flows and diffusions,

    M. S. Albergo, N. M. Boffi, and E. Vanden-Eijnden, “Stochastic interpolants: A unifying framework for flows and diffusions,”arXiv preprint arXiv:2303.08797, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    GuideFlow: Constraint-guided flow matching for planning in end-to-end autonomous driving,

    L. Liu, C. Jia, G. Yu, Z. Song, J. Li, F. Jia, P. Wu, X. Hao, and Y . Luo, “GuideFlow: Constraint-guided flow matching for planning in end-to-end autonomous driving,” arXiv preprint arXiv:2511.18729, 2025

    arXiv 2025

  17. [17]

    Frame guidance: Training-free guidance for frame-level control in video diffusion models,

    S. Jang, T. Ki, J. Jo, J. Yoon, S. Y . Kim, Z. Lin, and S. J. Hwang, “Frame guidance: Training-free guidance for frame-level control in video diffusion models,”arXiv preprint arXiv:2506.07177, 2025

    arXiv 2025

  18. [18]

    MotionCtrl: A unified and flex- ible motion controller for video generation,

    Z. Wang, Z. Yuan, X. Wang, Y . Li, T. Chen, M. Xia, P. Luo, and Y . Shan, “MotionCtrl: A unified and flex- ible motion controller for video generation,” inACM SIGGRAPH 2024 Conference Papers, 2024

    2024

  19. [19]

    DragNUWA: Fine-grained control in video generation by integrating text, image, and trajectory,

    S. Yin, C. Wu, J. Liang, J. Shi, H. Li, G. Ming, and N. Duan, “DragNUWA: Fine-grained control in video generation by integrating text, image, and trajectory,” arXiv preprint arXiv:2308.08089, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Boximator: Generating rich and con- trollable motions for video synthesis,

    J. Wang, Y . Zhang, J. Zou, Y . Zeng, G. Wei, L. Yuan, and H. Li, “Boximator: Generating rich and con- trollable motions for video synthesis,”arXiv preprint arXiv:2402.01566, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    GAIA-2: A controllable multi- view generative world model for autonomous driving,

    L. Russell, A. Hu, L. Bertoni, G. Fedoseev, J. Shotton, E. Arani, and G. Corrado, “GAIA-2: A controllable multi- view generative world model for autonomous driving,” arXiv preprint arXiv:2503.20523, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    DriveDreamer: Towards real-world-driven world models for autonomous driving,

    X. Wang, Z. Zhu, G. Huang, X. Chen, J. Zhu, and J. Lu, “DriveDreamer: Towards real-world-driven world models for autonomous driving,” inEuropean Conference on Computer Vision (ECCV), 2024

    2024

  23. [23]

    Epona: Autoregressive diffusion world model for autonomous driving,

    K. Zhang, Z. Tang, X. Hu, X. Pan, X. Guo, Y . Liu, J. Huang, L. Yuan, Q. Zhang, X.-X. Long, X. Cao, and W. Yin, “Epona: Autoregressive diffusion world model for autonomous driving,” inIEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  24. [24]

    DriveLaW: Unifying planning and video generation in a latent driving world,

    T. Xia, Y . Li, L. Zhou, J. Yao, K. Xiong, H. Sun, B. Wang, K. Ma, G. Chen, H. Ye, W. Liu, and X. Wang, “DriveLaW: Unifying planning and video generation in a latent driving world,”arXiv preprint arXiv:2512.23421, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Guided conditional diffusion for controllable traffic simulation,

    Z. Zhong, D. Rempe, D. Xu, Y . Chen, S. Veer, T. Che, B. Ray, and M. Pavone, “Guided conditional diffusion for controllable traffic simulation,” inIEEE International Conference on Robotics and Automation (ICRA), 2023

    2023

  26. [26]

    Language-guided traffic simulation via scene-level diffusion,

    Z. Zhong, D. Rempe, Y . Chen, B. Ivanovic, Y . Cao, D. Xu, M. Pavone, and B. Ray, “Language-guided traffic simulation via scene-level diffusion,” inConference on Robot Learning (CoRL), 2023

    2023

  27. [27]

    Diffusion- based planning for autonomous driving with flexible guidance,

    Y . Zheng, R. Liang, K. Zheng, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, and J. Liu, “Diffusion- based planning for autonomous driving with flexible guidance,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  28. [28]

    Score-based generative modeling through stochastic differential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Er- mon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” inInternational Conference on Learning Representations (ICLR), 2021

    2021

  29. [29]

    Towards accurate generative models of video: A new metric & challenges,

    T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, “Towards accurate generative models of video: A new metric & challenges,”arXiv preprint arXiv:1812.01717, 2018

    Pith/arXiv arXiv 2018

  30. [30]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations (ICLR), 2022

    2022

  31. [31]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022

    Pith/arXiv arXiv 2022

  32. [32]

    nuScenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for autonomous driving,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. APPENDIX This appendix derives the energy guidance for rectified flow used in Section III (§ Section A), ...

    2020

  33. [33]

    (6) The transformation is affine and invertible at every t∈(0,1) , so any score-based guidance transfers to the velocity field exactly, without score network or retraining

    Score and velocity equivalency:For the Gaussian path, the velocity field and the marginal score st(xt) = ∇xt logp t(xt)are affinely equivalent [7, 15]: vθ(xt, t) =− xt +σ t st(xt) αt , st(xt) =− αt vθ(xt, t) +x t σt . (6) The transformation is affine and invertible at every t∈(0,1) , so any score-based guidance transfers to the velocity field exactly, wit...

  34. [34]

    With the standard training-free approximation (∇xt logp t(y|x t)≈ −∇ xt E( ˆx0)) and the affine transformation Eq

    Guided velocity.:Score-based energy guidance encodes external knowledge as a likelihood pt(y|x t). With the standard training-free approximation (∇xt logp t(y|x t)≈ −∇ xt E( ˆx0)) and the affine transformation Eq. (6), together with the constant Jacobian ∂ ˆx0/∂xt = (1−σ min)I, the guided velocity is a single additive correction in clean-sample space, ˜v(...

  35. [35]

    Because the energy is evaluated on ˆx0 with the predicted velocity detached, the 11B-parameter backbone never enters the autograd graph

    Energy guidance overhead during sampling:The correc- tion needs only ∇ˆx0 E, which we obtain with ordinary automatic differentiation (PyTorch autograd) on the small closed-form energy of the (x, y, θ, v) ego trajectory. Because the energy is evaluated on ˆx0 with the predicted velocity detached, the 11B-parameter backbone never enters the autograd graph. ...

  36. [36]

    The image stream predicts the future videos

    Backbone and Architectural Modifications:The backbone is the Open-Sora 2.0 MM-DiT (DC-AE variant) [ 8]: 19 double- stream blocks and 38 single-stream blocks, hidden size 3072, 24 heads, with separate q/k/v projections. The image stream predicts the future videos. We repurpose the text stream into ego trajectory prediction. Each stream gets its own timeste...

  37. [37]

    Fine-tuning details:The entire backbone is frozen. We attach LoRA adapters [ 30] (rank 16, α= 32 ) to the modulation, attention, and MLP projections of both streams (img_{mod,attn,mlp}, txt_{mod,attn,mlp}, 266 linear layers, 46.7M parameters trained at learning rate 1×10−4). The newly introduced modules (ego encoder, ego output head, ego timestep MLP, pas...

  38. [38]

    The ego term is masked to the valid future frames

    Objective and optimization:We minimize a joint rectified- flow loss L=L video +λ egoLego (λego = 1), each an MSE on the predicted velocity (Section A). The ego term is masked to the valid future frames. Optimization uses AdamW ( β= (0.9,0.98) , weight decay 0, ϵ= 10 −8) with gradient clipping at 1.0 and bf16 autocast. To fit the backbone on a single NVIDI...

  39. [39]

    Classifier-free guidance:The platform is also trained for classifier-free guidance [ 31] over its past conditioning, which sharpens the generated video. With probability 0.15 per sample we drop the observed past by replacing the pooled past-observation vector with a learned null embedding and zeroing the in-context past latent frames together, so the mode...

  40. [40]

    Data and autoencoder:We train on nuScenes v1.0-trainval [32], front camera, at 256×448 , using 12-frame clips at the native sample-data cadence (stride 12, non-overlapping). The frozen DC-AE ( dc-ae-f32t4c128, scaling factor 0.493, 4× temporal / 32× spatial compression) encodes each clip to a (3,8,14) latent, where the first latent frame is the clean past...

  41. [41]

    The checkpoint used here is trained for ∼22k steps on a single GPU, with validation on a held-out tail of 64 clips and best-on-validation checkpointing

    Schedule and checkpoint:Timesteps are sampled logit- normal (in compliance with the Open-Sora 2.0 backbone); the batch size is 8 per GPU. The checkpoint used here is trained for ∼22k steps on a single GPU, with validation on a held-out tail of 64 clips and best-on-validation checkpointing. The platform is trained once. Every result in the main text reuses...

  42. [42]

    We also release the trained model weights with the LoRA adapters and the additionally introduced modules ( 88M trainable parameters)

    Code availability:We release the code required to reproduce the results from the released checkpoint: the training- free energy-guidance sampler, the differentiable energy library and the score-velocity bridge, the counterfactual-brake probe, and the analysis and figure scripts at https://timsu-98.github.io/ energy guidance for driving world models/. We a...

    1938