Pith. sign in

REVIEW 4 major objections 5 minor 66 references

CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that distribution alignment, not model capacity, is the key to adapting pretrained world models to traffic forecasting, and demonstrates it by winning Track 5 with 76.49.

desk verdict A genuinely useful first-place competition recipe with unusually honest ablations, but the alignment-over-capacity claim rests on a 36-clip validation split and a zero-shot baseline is missing. read the letter →

arxiv 2608.07693 v1 pith:TTWXSUZT submitted 2026-08-07 cs.CV cs.AIcs.CL

Quang Minh Dinh , Tuan Kiet Doan This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords trafficvideoforecastingworldmodelsautonomousdrivingLoRAfine-tuningdistributionalignmenttext-conditionedgenerationmedoidselectionAICityChallenge2026
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that adapting a large pretrained world model to a specific traffic-forecasting benchmark is primarily a distribution-alignment problem, not a model-capacity problem. It builds CosmosAlign on the 16B Cosmos3-Nano model with a two-stage LoRA recipe: first rewiring the conditioning-mode distribution so that 70% of gradient steps supervise the five-frame video-conditioned forecasting mode, then re-captioning the training data into the model's native structured prompt format. Paired with a training-free inference step of consensus-based medoid selection and motion-adaptive blending of static regions, the adapted model scores 76.49 on the AI City Challenge 2026 Track 5 leaderboard and ranks first. A sympathetic reader would care because it suggests that strong downstream forecasting can be achieved cheaply by aligning the interface, rather than by scaling the model.

What carries the argument

The carrying mechanism is the two-stage LoRA alignment recipe applied to Cosmos3-Nano, a 16B world foundation model with a Reasoner–Generator architecture. Stage 1 trains low-rank adapters (rank 32, $\alpha=64$) on the Generator attention projections while freezing the base model, reweighting the conditioning-mode distribution $\{k=0,k=1,k=2\}$ from the default $\{0.7,0.2,0.1\}$ to $\{0.1,0.2,0.7\}$ so that 70% of gradient steps supervise the five-frame video-conditioned forecasting mode. Stage 2 continues training on 3,107 WTS windows re-captioned into the model's native JSON temporal-caption format, aligning the prompt distribution. At inference, the payload holds five history frames, the structured prompt, and an extended negative prompt; four samples are drawn with a 35-step UniPC solver at guidance 3, the medoid (minimum mean pairwise distance on a downsampled grid) is selected, and a motion-adaptive blend weights the last observed frame into static regions only.

What would settle it

Re-run the hyperparameter sweeps on a different 36-clip validation split drawn from the same WTS training pool; if the optimal settings (guidance 3, 750 Stage-2 iterations, blending thresholds 3/20/0.9) shift materially or the official test score drops well below 76.49, the small-split representativeness assumption is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that on the WTS-based Track 5 task, the dominant bottleneck in adapting Cosmos3-Nano is distribution mismatch rather than model capacity, and that a two-stage LoRA adaptation—Stage 1 reweights the conditioning-mode distribution from the pretraining default toward the evaluated forecasting mode, and Stage 2 aligns the training captions to the model's native structured prompting format via LLM re-captioning—plus a training-free inference of medoid sample selection and motion-adaptive blending, yields the top official score of 76.49. The paper supports this with ablations: raising the LoRA rank from 32 to 128 degrades four of five validation metrics, longer Stage 1 training shows monotone CLIP-S decay, and longer conditioning histories hurt quality; conversely, guidance tuning to $g=3$, structured prompts, the Stage 2 checkpoint, best-of-$N$ medoid selection, and the motion-adaptive blend each improve the final score on the official test set.

Load-bearing premise

The 36-clip validation set the authors built to mirror the test protocol is representative enough of the 71-clip official test set that the hyperparameters chosen on it (guidance 3, Stage 2 length 750 iterations, blending thresholds $m_{\mathrm{lo}}=3$, $m_{\mathrm{hi}}=20$, $w=0.9$) transfer to the real test.

Editorial extensions

If this is right

  • If the alignment thesis is right, strong traffic-video forecasting can be obtained from a 16B model with low-rank adapters and no capacity increase, making the whole pipeline runnable on a single A100 GPU.
  • The two-stage recipe of conditioning-mode reweighting plus caption-format alignment should transfer to other world models that share a structured prompting interface, likely saving compute on future benchmarks.
  • The training-free medoid selection and motion-adaptive blending improve fidelity without ground truth or auxiliary models, so they can be dropped into other sampling-based generative forecasting pipelines.
  • The ablations imply that in the data-limited regime, memorization and train-test distribution mismatch, not model size, limit forecast quality; the paper's stated next step is to test the recipe on larger variants such as Cosmos3-Super and on other world model families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this alignment-over-capacity claim holds beyond this benchmark, the field's default reflex of scaling up may be misplaced; the first diagnostic when adapting a foundation model should be an analysis of the conditioning-mode and prompt-format gap between pretraining and target.
  • The motion-adaptive blending rule is a test-time constraint that anchors static regions to the last observed frame; this suggests a broader class of physics-aware post-processing that enforces scene-structure priors without retraining.
  • Because the medoid selection is ground-truth-free, it could be applied to online adaptation or self-training loops where no labels exist but consensus across samples is a proxy for reliability.
  • A direct extension: run the same recipe on a non-traffic video-forecasting benchmark (for example, general action prediction) to see whether conditioning-mode reweighting alone reproduces the gains; if it does not, the alignment claim is specific to safety-critical traffic scenes rather than general video generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript presents CosmosAlign, a system for text-conditioned traffic video forecasting built on the frozen Cosmos3-Nano world model. It proposes a two-stage LoRA adaptation (Stage 1 reweights the conditioning-mode distribution on WTS/BDD windows; Stage 2 continues on re-captioned WTS windows), followed by a training-free test-time procedure that selects the medoid of four samples and blends static regions toward the last observed frame. The authors report validation ablations over guidance, adapter rank, prompt format, post-processing, history length, and training length, a test-set progression of submissions, and a final official score of 76.49 ranking first on AI City Challenge 2026 Track 5. They conclude that adapting world models to this task depends primarily on distribution alignment rather than model capacity.

Significance. If the reported result is taken at face value, the paper provides a strong empirical data point that parameter-efficient alignment of a large pretrained world model can win a competitive forecasting benchmark, and the public code and detailed leaderboard progression are useful to the community. The training-free medoid selection and motion-adaptive blending are simple, clearly described, and additive in the reported numbers. I found no circular derivation: the medoid and blending are evaluated on held-out test clips, and the validation set is separate from the test set. However, the broader conceptual claim about distribution alignment versus capacity is not established at the same level of confidence as the benchmark result, because the component attributions rest on a 36-clip validation set and single test runs.

major comments (4)
  1. [Sec. 4.1 and Table 2] Every recipe hyperparameter (guidance g=3, LoRA rank/alpha 32/64, Stage 2 length 750, blending thresholds m_lo=3 and m_hi=20, blending weight w=0.9, and the medoid over seeds 0-3) is selected on the 36-clip validation set using seed 0, and each row of the Table 2 progression is a single official test run. The validation set is described as mirroring the test protocol only in view ratio and horizon distribution, and the validation and test metric scales differ materially (validation PSNR around 23.9 versus test PSNR around 20.1). The paper therefore does not currently establish that the validation-selected choices, or the component gains in Table 2 (notably guidance +1.45 and blending +1.17), transfer to the official test set beyond the single final score. Please add evidence of validation-to-test consistency (e.g., rank correlation across configurations, multiple seeds, or a bootstrap over the 36 clips) and, where possible, repeated test submissions for the key incremental rows.
  2. [Table 1] The 'History 5 frames' row, as displayed, reports PSNR 24.40, LPIPS 0.163, and CLIP-S 28.12, which are better than every fine-tuned configuration in the same table, including the Stage 2 +750 checkpoint (PSNR 24.25, LPIPS 0.166, CLIP-S 27.92). The 'Longer Conditioning History' paragraph interprets the history-length trend as a property of the adapted model, but if '5 frames' is actually the zero-shot Cosmos3-Nano baseline, the table neither labels it nor reports a zero-shot score on the official test set; if it is the adapted model, it contradicts the g=3 and Stage 2 rows. Please clarify this configuration and, if it is the zero-shot baseline, use it to quantify the fine-tuning gain on both validation and test.
  3. [Abstract, Sec. 4.2, Sec. 5] The claim that adaptation 'depends primarily on distribution alignment rather than increased model capacity' is supported only by a rank-32 versus rank-128 comparison within the same 16B backbone, while the authors state in the conclusion that validation on larger variants such as Cosmos3-Super is future work. The reported evidence shows that a larger adapter does not help in this data-limited setting, but it does not support the general comparative claim about capacity. Please either restrict the claim to the data-limited LoRA setting or add a positive control against a larger backbone or a full fine-tune.
  4. [Table 2] The final incremental rows are close to plausible run-to-run noise. The deflickering stack adds +0.035 points (75.35 versus 75.32), and the text attributes +0.32 to best-of-3 despite an intervening deflickering row (75.03 to 75.32); with single submissions per configuration, these margins are not demonstrated to be significant. Please provide repeated official runs or an estimate of test-set noise so the component attributions are credible.
minor comments (5)
  1. [Sec. 3.2] In the 'Stage 1: Forecasting Adaptation' paragraph, 'The conditioning-mode distribution ... are reweighted' should be 'is reweighted'.
  2. [Sec. 4.2] In the 'Stage 2 Fine-Tuning' paragraph, 'best-of-3 selection adds +0.32 on the Stage 2 recipe (75.00 to 75.32)' skips the intermediate deflickering row; please state the exact comparison base.
  3. [Table 2 caption] The phrase 'stack separatedly' appears to be a typo for 'stack separately'.
  4. [References] Reference [24] contains the placeholder 'fill author list from the ICCW 2025 paper, T.' and needs to be completed before publication.
  5. [Sec. 4.1] The statement that 'all configuration sweeps are validated at seed 0' should also clarify whether validation metrics are averaged over the 36 clips or pooled over frames, since this affects the interpretation of the reported FVD.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; benchmark-validated engineering result with one non-load-bearing self-citation.

full rationale

The paper is an empirical systems/benchmark report, not a formal derivation. The load-bearing claim is the first-place score of 76.49 on the AI City Challenge 2026 Track 5 leaderboard, which is verified on an external 71-clip test set by the challenge evaluation server. All design choices — two-stage LoRA, guidance scale, structured prompting, medoid selection, and motion-adaptive blending — are first evaluated on a separate 36-clip validation split and then reported on the official test set. No fitted parameter is renamed as a prediction: test-set ablations are measured outcomes, not forecasts derived from the validation fit. The motion-adaptive blending does use the last observed ground-truth frame x0, but it is explicitly presented and measured as a test-time post-processing step, not as a model prediction, and it is still evaluated against held-out future frames on the official test set. The only self-citation is TrafficVLM [9] in the Related Work section, used as background context; it is not load-bearing for any argument, does not provide a uniqueness theorem, and does not forbid alternative approaches. Concerns about the 36-clip validation split being potentially unrepresentative of the 71-clip test set are a legitimate correctness or generalization risk, but they are not circularity: the validation and test sets are disjoint, and the reported test numbers come from independent evaluation. The central result is therefore self-contained against an external benchmark, and no derivation step reduces to its own input by construction.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central result is an empirical system result, so there is no mathematical derivation to audit. The principal free parameters are the validation-selected hyperparameters listed above. The paper relies on several domain assumptions about Cosmos3-Nano's native interface, the representativeness of its validation split, the fidelity of LLM re-captioning, and the usefulness of the BDD external pool. No new entities are postulated.

free parameters (6)
  • classifier-free guidance scale g = 3
    Selected on the 36-clip validation set from sweep {1,2,3,6}; Table 1 shows a U-shaped quality curve. Single-seed evaluation.
  • motion-adaptive blend thresholds and weight = m_lo=3, m_hi=20, w=0.9, sigma=8
    Hand-chosen on validation to maximize PSNR, SSIM, and LPIPS; the search grid is not reported, and the gain comes partly from copying the last observed frame into static regions.
  • conditioning-mode distribution = {0.1, 0.2, 0.7} for k={0,1,2}
    Reweighted from Cosmos3-Nano's default {0.7,0.2,0.1} to emphasize 5-frame forecasting; no ablation of this exact weight vector is shown.
  • LoRA rank and alpha = r=32, alpha=64
    Chosen after a rank 32 versus rank 128 comparison (Table 1); rank 128 degrades four of five metrics.
  • Stage 2 training length = 750 iterations
    Selected on validation FVD after checking {250,500,750,1000}; PSNR and SSIM plateau after 500, so the selection relies on a single FVD value.
  • number of samples and seeds = 4 samples, seeds 0..3
    Best-of-N ablation tested 3 seeds; adding a fourth seed gave the final gain. The medoid rule uses these four samples.
assumptions (4)
  • domain assumption Cosmos3-Nano's architecture, conditioning-mode distribution, and native structured JSON prompt interface are as described by reference [1].
    Central to both fine-tuning stages and test-time payload design; not independently verified in this paper.
  • domain assumption The 36-clip WTS validation set is representative of the 71-clip official test distribution.
    All hyperparameters and post-processing weights are chosen on this split; Section 4.1 states it mirrors the test protocol exactly.
  • domain assumption Claude Opus 4.8 re-captions are faithful and stable across training and test clips.
    Stage 2 and test prompts depend on the LLM grounding statements in frames and captions; non-determinism or hallucination would break the alignment claim.
  • domain assumption The provided BDD100K external videos improve or at least do not hurt the Stage 1 distribution match.
    Stage 1 mixes 635 WTS multi-view videos with 2,182 BDD dashcam videos; a view-balanced rebalance was tested and did not help, but a no-BDD control is not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting." pith.science (2026). https://pith.science/paper/TTWXSUZT

@misc{pith2026260807693,
  author       = {Pith},
  title        = {Pith review of: CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTWXSUZT}},
  note         = {Machine review of arXiv:2608.07693}
}
read the original abstract

Generative traffic video forecasting aims to synthesize long-horizon, temporally coherent future videos of traffic scenes from a short observation history and textual descriptions. In this paper, we present CosmosAlign, a generative traffic video forecasting framework built upon the pretrained Cosmos3-Nano world foundation model. Our approach is motivated by the observation that successfully adapting large pretrained world models to downstream forecasting tasks depends primarily on distribution alignment rather than increased model capacity. To this end, we propose a two-stage LoRA adaptation strategy that first aligns the conditioning-mode distribution with the target forecasting task, and then aligns the training captions with the model's native structured prompting interface through an LLM-based re-captioning pipeline. During inference, we further improve prediction quality using a fully training-free procedure consisting of consensus-based medoid sample selection and motion-adaptive blending of static scene regions. CosmosAlign achieves a final score of 76.49 on the AI City Challenge 2026 Track 5 benchmark, ranking first on the final leaderboard. Our code is publicly available at https://quangminhdinh.github.io/CosmosAlign/.

Figures

Figures reproduced from arXiv: 2608.07693 by the authors.

Figure 1
Figure 1. Overview of our method. (a) Cosmos3-Nano is adapted in two LoRA stages: Stage 1 fine-tunes the Generator tower on WTS and BDD forecasting windows with the reweighted conditioning-mode distribution, and Stage 2 continues training on the WTS windows only, using the structured temporal-caption format the model natively expects. (b) At test time, each clip is encoded into a payload holding the five history frames, an LL… view at source ↗
Figure 2
Figure 2. System prompt of our caption-generation pipeline. {"temporal_caption": "A broad asphalt driving course is seen from above, lined with black-and-yellow cones and edged by grass strips, distant parked cars, and trees under bright, clear daylight. A man in his twenties in a brown jacket and navy-blue slacks walks across the open pavement with a dark car standing close behind him, both facing the same direction. As time… view at source ↗
Figure 3
Figure 3. Example structured prompt of a test clip (overhead view, horizon N = 87). 3.3 Prompt Construction Structured Prompt Generation. We generate all structured prompts for the 3,107 Stage 2 training windows and the 71 test clips using Claude Opus 4.8 API, which can be replaced by any Visual Language Model (VLM) of equivalent capability. For each video, the VLM receives the camera type, the clip duration, the official ped… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The four sentences appended to the default Cosmos3-Nano negative prompt [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of our final model on held-out clips, ground truth (top) against our prediction (bottom). (a) Overhead view: the pedestrian is advanced along the ground-truth trajectory across the crosswalk, with the static camera and the road markings preserved. (…
Figure 6
Figure 6. Figure 6: Visual conditioning dominates the prompt. The history frames of this validation clip show a dashboard clock; the ground truth (top) cuts to the road scene, whereas our prediction (bottom) carries the glowing digits into the generated road scene and holds them for the w…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 23 canonical work pages

  1. [24]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (2025)

    fill author list from the ICCVW 2025 paper, T.: TrafficInternVL: Understanding traffic scenarios with vision-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (2025)

    work page 2025

  2. [1]

    arXiv preprint arXiv:2606.02800 (2026)

    Agarwal, N., Ali, A., Allen, J., Antolini, M., Aubame, A., Azzolini, A., Bai, J., Bala, M., Balaji, Y., Bapst, J., et al.: Cosmos 3: Omnimodal world models for physical ai. arXiv preprint arXiv:2606.02800 (2026)

    arXiv 2026

  3. [2]

    arXiv preprint arXiv:2501.03575 (2025)

    Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al.: Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575 (2025)

    arXiv 2025

  4. [3]

    arXiv preprint arXiv:2511.00062 (2025)

    Ali, A., Bai, J., Bala, M., Balaji, Y., Blakeman, A., Cai, T., Cao, J., Cao, T., Cha, E., Chao, Y.W., et al.: World simulation with video foundation models for physical ai. arXiv preprint arXiv:2511.00062 (2025)

    arXiv 2025

  5. [4]

    arXiv preprint arXiv:2506.09985 (2025)

    Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Muckley, M., Rizvi, A., Roberts, C., Sinha, K., Zholus, A., et al.: V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985 (2025)

    arXiv 2025

  6. [5]

    arXiv preprint arXiv:2404.08471 (2024)

    Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., Ballas, N.: Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471 (2024)

    arXiv 2024

  7. [6]

    In: Forty-first International Conference on Machine Learning (2024)

    Bruce, J., Dennis, M.D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., et al.: Genie: Generative interactive environments. In: Forty-first International Conference on Machine Learning (2024)

    2024

  8. [7]

    In: 2025 IEEE/CVF International Conference on Computer Vision (ICCV)

    Chen, R., Wu, Z., Liu, Y., Guo, Y., Ni, J., Xia, H., Xia, S.: Unimlvg: Unified framework for multi-view long video generation with comprehensive control capa- bilities for autonomous driving. In: 2025 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 25453–25463. IEEE (2025)

    work page 2025

Show all 66 references
  1. [8]

    comma.ai: commavq.https://github.com/commaai/commavq(2023)

    2023

  2. [9]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops

    Dinh, Q.M., Ho, M.K., Dang, A.Q., Tran, H.P.: TrafficVLM: A controllable visual language model for traffic video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 7134–7143 (2024)

    2024

  3. [10]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) Workshops (2024)

    Duan, Z., Cheng, H., Xu, D., Wu, X., Zhang, X., Ye, X., Xie, Z.: CityLLaVA: Effi- cient fine-tuning for vlms in city scenario. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) Workshops (2024)

    2024

  4. [11]

    arXiv preprint arXiv:1812.00568 (2018)

    Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A., Levine, S.: Visual foresight: Model- based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568 (2018)

    2018 arXiv

  5. [12]

    In: Scott, D., Bel, N., Zong, C

    Eikema, B., Aziz, W.: Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In: Scott, D., Bel, N., Zong, C. (eds.) Proceedings of the 28th International Conference on Computational Linguistics. pp. 4506–4520. International Committee on Comput...

    2020 doi

  6. [13]

    Advances in Neural Information Processing Systems37, 91560–91596 (2024)

    Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., Li, H.: Vista: A generalizable driving world model with high fidelity and versatile controllability. Advances in Neural Information Processing Systems37, 91560–91596 (2024)

    2024

  7. [14]

    arXiv preprint arXiv:1803.101222(3), 440 (2018)

    Ha, D., Schmidhuber, J.: World models. arXiv preprint arXiv:1803.101222(3), 440 (2018)

    2018 arXiv

  8. [15]

    arXiv preprint arXiv:1912.01603 (2019)

    Hafner, D., Lillicrap, T., Ba, J., Norouzi, M.: Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 (2019)

    2019 arXiv

  9. [16]

    In: International conference on machine learning

    Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., Davidson, J.: Learning latent dynamics for planning from pixels. In: International conference on machine learning. pp. 2555–2565. PMLR (2019)

    2019

  10. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hassan, M., Stapf, S., Rahimi, A., Rezende, P., Haghighi, Y., Brüggemann, D., Katircioglu,I.,Zhang,L.,Chen,X.,Saha,S.,etal.:Gem:Ageneralizableego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control. In: Proceedings of the I...

    2025

  11. [18]

    ArXivabs/2104.08718(2021)

    Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference- free evaluation metric for image captioning. ArXivabs/2104.08718(2021)

    2021 arXiv

  12. [19]

    ArXivabs/1706.08500(2017)

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a nash equilibrium. ArXivabs/1706.08500(2017)

    2017 arXiv

  13. [20]

    In: NeurIPS (2022)

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. In: NeurIPS (2022)

    2022

  14. [21]

    URL https://arxiv

    Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., Corrado, G.: Gaia-1: a generative world model for autonomous driving (2023). URL https://arxiv. org/abs/2309.170803

    2023 arXiv

  15. [22]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022)

    2022

  16. [23]

    arXiv preprint arXiv:2412.19505 (2024)

    Hu, X., Yin, W., Jia, M., Deng, J., Guo, X., Zhang, Q., Long, X., Tan, P.: Driv- ingworld: Constructing world model for autonomous driving via video gpt. arXiv preprint arXiv:2412.19505 (2024)

    2024 arXiv

  17. [25]

    In: ECCV

    Kong, Q., Kawana, Y., Saini, R., Kumar, A., Pan, J., Gu, T., Ozao, Y., Opra, B., Anastasiu, D.C., Sato, Y., Kobori, N.: WTS: A pedestrian-centric traffic video dataset for fine-grained spatial-temporal understanding. In: ECCV. pp. 1–18 (2024).https://doi.org/10.1007/978-3-031-...

    2024 doi

  18. [26]

    In: Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT- NAACL 2004

    Kumar, S., Byrne, W.: Minimum Bayes-risk decoding for statistical machine trans- lation. In: Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT- NAACL 2004. pp. 169–176. Association for Co...

    2004

  19. [27]

    2, 2022-06-27

    LeCun, Y., et al.: A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review62(1), 1–62 (2022)

    2022

  20. [28]

    In: International Conference on Machine Learning (ICML) (2023) CosmosAlign for Generative Traffic Video Forecasting 17

    Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International Conference on Machine Learning (ICML) (2023) CosmosAlign for Generative Traffic Video Forecasting 17

    2023

  21. [29]

    arXiv preprint arXiv:2510.16732 (2025)

    Li, X., He, X., Zhang, L., Wu, M., Li, X., Liu, Y.: A comprehensive survey on world models for embodied ai. arXiv preprint arXiv:2510.16732 (2025)

    2025 arXiv

  22. [30]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Advances in Neural Information Processing Systems (NeurIPS) (2023)

    2023

  23. [31]

    In: Forty-first International Conference on Machine Learning (2024)

    Liu, S.Y., Wang, C.Y., Yin, H., Molchanov, P., Wang, Y.C.F., Cheng, K.T., Chen, M.H.: Dora: Weight-decomposed low-rank adaptation. In: Forty-first International Conference on Machine Learning (2024)

    2024

  24. [32]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Liu,W.,Luo,W.,Lian,D.,Gao,S.:Futureframepredictionforanomalydetection– a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6536–6545 (2018)

    2018

  25. [33]

    arXiv preprint arXiv:2312.00438 (2023)

    Ma, Y., Cao, Y., Sun, J., Pavone, M., Xiao, C.: Dolphins: Multimodal language model for driving. arXiv preprint arXiv:2312.00438 (2023)

    2023 arXiv

  26. [34]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    Mao,J.,Li,B.,Ivanovic,B.,Chen,Y.,Wang,Y.,You,Y.,Xiao,C.,Xu,D.,Pavone, M., Wang, Y.: Dreamdrive: Generative 4d scene modeling from street view images. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 367–374. IEEE (2025)

    2025

  27. [35]

    Advances in Neural Information Processing Systems37, 121038–121072 (2024)

    Meng,F.,Wang,Z.,Zhang,M.:Pissa:Principalsingularvaluesandsingularvectors adaptation of large language models. Advances in Neural Information Processing Systems37, 121038–121072 (2024)

    2024

  28. [36]

    Advances in Neural Information Processing Systems 37, 42292–42310 (2024)

    Miles, R., Reddy, P., Elezi, I., Deng, J.: Velora: Memory efficient training using rank-1 sub-token projections. Advances in Neural Information Processing Systems 37, 42292–42310 (2024)

    2024

  29. [37]

    arXiv preprint arXiv:2401.14718 (2024)

    Ming, R., Huang, Z., Wu, J., Ju, Z., Jiang, D., Hu, J., Peng, L., Zhou, S.: A survey on future frame synthesis: Bridging deterministic and generative approaches. arXiv preprint arXiv:2401.14718 (2024)

    2024 arXiv

  30. [38]

    Advances in Neural Information Processing Systems38, 4741–4770 (2026)

    Mousakhan, A., Mittal, S., Galesso, S., Farid, K., Brox, T.: Overcoming challenges of long-horizon prediction in driving world models. Advances in Neural Information Processing Systems38, 4741–4770 (2026)

    2026

  31. [39]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Muhamed, A., Li, O., Woodruff, D., Diab, M., Smith, V.: Grass: Compute effi- cient low-memory llm training with structured sparse gradients. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 14978–15003 (2024)

    2024

  32. [40]

    arXiv preprint arXiv:2505.05086 (2025)

    Nguyen, L.T., Quélennec, A., Nguyen, V.T., Tartaglione, E.: Beyond low-rank decomposition: A shortcut approach for efficient on-device learning. arXiv preprint arXiv:2505.05086 (2025)

    2025 arXiv

  33. [41]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Ni, J., Guo, Y., Liu, Y., Chen, R., Lu, L., Wu, Z.: Maskgwm: A generalizable driv- ing world model with video mask reconstruction. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 22381–22391 (2025)

    2025

  34. [42]

    IEEE Transactions on Pattern Analysis and Machine Intelligence44(6), 2806–2826 (2020)

    Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J.A., Orts- Escolano, S., Garcia-Rodriguez, J., Argyros, A.: A review on deep learning tech- niques for video prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence44(6), 2806–2826 (2020)

    2020

  35. [43]

    In: ICML

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)

    2021

  36. [44]

    arXiv preprint arXiv:2503.20523 (2025)

    Russell, L., Hu, A., Bertoni, L., Fedoseev, G., Shotton, J., Arani, E., Corrado, G.: Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523 (2025)

    2025 arXiv

  37. [45]

    In: Proceedings of the 18 Q

    Shao, H., Hu, Y., Wang, L., Song, G., Waslander, S.L., Liu, Y., Li, H.: Lmdrive: Closed-loop end-to-end driving with large language models. In: Proceedings of the 18 Q. M. Dinh and T. K. Doan IEEE/CVF conference on computer vision and pattern recognition. pp. 15120– 15130 (2024)

    2024

  38. [46]

    In: Conference on Robot Learning

    Shao, H., Wang, L., Chen, R., Li, H., Liu, Y.: Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In: Conference on Robot Learning. pp. 726–737. PMLR (2023)

    2023

  39. [47]

    In: European Conference on Computer Vision (ECCV) (2024)

    Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A., Li, H.: DriveLM: Driving with graph visual question answering. In: European Conference on Computer Vision (ECCV) (2024)

    2024

  40. [48]

    In: ECCV Workshops

    Tang, Z., Wang, S., Anastasiu, D.C., Chang, M.C., et al.: The 10th AI City Chal- lenge. In: ECCV Workshops. Malm"o, Sweden (2026)

    2026

  41. [49]

    ArXivabs/2003.12039(2020),https://api.semanticscholar.org/CorpusID: 214667893

    Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. ArXivabs/2003.12039(2020),https://api.semanticscholar.org/CorpusID: 214667893

    2020 arXiv

  42. [50]

    In: International Conference on Learning Representations Workshop (2019)

    Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Towards accurate generative models of video: A new metric and challenges. In: International Conference on Learning Representations Workshop (2019)

    2019

  43. [51]

    ArXivabs/2503.20314(2025)

    Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., Wang, J., Zhang, J., Zhou, J., Wang, J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Meng, X., Zhang, N., Li, P., Wu, P., Chu, R., Feng, R., Zhang, S., Sun, S., Fang, T., Wang, ...

    2025 arXiv

  44. [52]

    arXiv preprint arXiv:2305.04412 (2023)

    Wang, L., Liu, J., Shao, H., Wang, W., Chen, R., Liu, Y., Waslander, S.L.: Efficient reinforcementlearningforautonomousdrivingwithparameterizedskillsandpriors. arXiv preprint arXiv:2305.04412 (2023)

    2023 arXiv

  45. [53]

    In: European conference on computer vision

    Wang, X., Zhu, Z., Huang, G., Chen, X., Zhu, J., Lu, J.: Drivedreamer: Towards real-world-drive world models for autonomous driving. In: European conference on computer vision. pp. 55–72. Springer (2024)

    2024

  46. [54]

    ArXivabs/2203.11171 (2022)

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E.H., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. ArXivabs/2203.11171 (2022)

    2022 arXiv

  47. [55]

    In: Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing

    Wang, Y., Agarwal, S., Mukherjee, S., Liu, X., Gao, J., Hassan, A., Gao, J.: Adamix: Mixture-of-adaptations for parameter-efficient model tuning. In: Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 5744–5760 (2022)

    2022

  48. [56]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, Y., He, J., Fan, L., Li, H., Chen, Y., Zhang, Z.: Driving into the future: Mul- tiview visual forecasting and planning with world model for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14749–14759 (2024)

    2024

  49. [57]

    arXiv preprint arXiv:2311.05332 (2023)

    Wen, L., Yang, X., Fu, D., Wang, X., Cai, P., Li, X., Ma, T., Li, Y., Xu, L., Shang, D., Li, Z., Sun, L., Li, Y., Xu, Q., Zhao, Z., Wang, B., Liu, Y., Qiao, Y., Shao, J., Chi, C., Zhang, W.: On the road with GPT-4V(ision): Early explorations of visual-language model on autonom...

    2023 arXiv

  50. [58]

    arXiv preprint arXiv:1904.05538 (2019) CosmosAlign for Generative Traffic Video Forecasting 19

    Xie, A., Ebert, F., Levine, S., Finn, C.: Improvisation through physical un- derstanding: Using novel objects as tools with visual foresight. arXiv preprint arXiv:1904.05538 (2019) CosmosAlign for Generative Traffic Video Forecasting 19

    2019 arXiv

  51. [59]

    IEEE Robotics and Automation Letters (2024)

    Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.K.Y., Li, Z., Zhao, H.: DriveGPT4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters (2024)

    2024

  52. [60]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)

    Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Dar- rell, T.: Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)

    2020

  53. [61]

    In: CVPR

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR. pp. 586–595 (2018)

    2018

  54. [62]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X., Wang, X.: Drivedreamer-2: Llm-enhanced world models for diverse driving video generation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 10412–10420 (2025)

    2025

  55. [63]

    arXiv preprint arXiv:2403.03507 (2024)

    Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., Tian, Y.: Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507 (2024)

    2024 arXiv

  56. [64]

    In: Thirty-seventh Conference on Neural Information Processing Systems (2023),https://openreview.net/forum? id=hrkmlPhp1u

    Zhao, W., Bai, L., Rao, Y., Zhou, J., Lu, J.: UniPC: A unified predictor-corrector framework for fast sampling of diffusion models. In: Thirty-seventh Conference on Neural Information Processing Systems (2023),https://openreview.net/forum? id=hrkmlPhp1u

    2023

  57. [65]

    Transactions of the Association for Com- putational Linguistics12, 525–542 (2024)

    Zhou, H., Wan, X., Vulić, I., Korhonen, A.: Autopeft: Automatic configuration search for parameter-efficient fine-tuning. Transactions of the Association for Com- putational Linguistics12, 525–542 (2024)

    2024

  58. [66]

    arXiv preprint arXiv:2601.01528 (2026)

    Zhou, Y., Shao, H., Wang, L., Zong, Z., Li, H., Waslander, S.L.: Drivinggen: A com- prehensive benchmark for generative video world models in autonomous driving. arXiv preprint arXiv:2601.01528 (2026)

    2026

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.