Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Dreamland: Controllable World Creation with Simulator and Generative Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dreamland claims a hybrid simulator-plus-generative pipeline can cut image-fidelity error by over half while keeping per-object control, and backs it with a new paired dataset.

desk verdict Dreamland's pipeline and dataset are solid, but the headline FID/controllability claims are undercut by inconsistent numbers and an unclear baseline evaluation that needs a matched comparison. read the letter →

arxiv 2506.08006 v1 pith:2TLNKXJ5 submitted 2025-06-09 cs.CV

Sicheng Mo , Ziyang Leng , Leon Liu , Weizhen Wang , Honglin He , Bolei Zhou This is my paper · ORCID
classification cs.CV
keywords controllableworldgenerationlayeredabstractionsimulator-to-realtransferdrivingscenediffusionmodelsembodiedAItraininghybridsimulationD3Simdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dreamland is a three-stage pipeline for creating driving scenes that are simultaneously physically controllable and photorealistic: a physics-based simulator first renders the scene, an instruction-following editing model transfers the simulator's layered representation toward real-world image statistics, and a pretrained diffusion model re-renders the scene under a user's text prompt. The paper claims this design beats the prior hybrid system SimGen by 52.3% lower FID and 17.9% better depth controllability while preserving object-level adherence to the simulator's layout. If those numbers hold, Dreamland offers a practical way to generate large volumes of realistic driving imagery with precise control over every vehicle and road element, which is exactly the kind of data needed for training embodied driving agents. The authors also contribute D3Sim, a large dataset of paired simulated and real driving conditions, to train and evaluate such hybrid pipelines.

What carries the argument

Layered World Abstraction (LWA), defined as $W = \bigcup_{i=0}^{N} L_i \odot V_i$, where each layer $L_i$ encodes $K$ condition modalities (e.g., depth, segmentation, instance) and $V_i$ is a visibility mask. The paper composes three layers — traffic participants $L_d$, map layout $L_l$, and editable background $L_b$ — which lets the pipeline preserve simulator-controlled elements while allowing generative freedom elsewhere. The load-bearing steps are the Stage-2 editing model $L_b = \epsilon_e(\hat{W}, V_b, c)$ that converts Sim-LWA into Real-LWA, and the Stage-3 mixed-condition generation that injects the refined conditions into a pretrained diffusion model via a learned projection layer $x' = x + \xi(\bigoplus_{j=0}^{K} E(c_j))$. This design is what decouples 'what the simulator dictates' from 'what the generator invents'.

What would settle it

Re-run the Table 1 comparison using a new depth estimator and a new segmentation model (for example, models released after the paper) and measure the preserved-region si-RMSE and mIoU; if the advantage over SimGen does not persist with both estimators, the reported controllability numbers are tied to the specific evaluation models rather than to the pipeline. A complementary check is to compute the average pixel registration error between the simulator-rendered depth maps and the real nuPlan depth maps on a held-out set of scenarios; if the error exceeds about one pixel per 100 meters, the claimed alignment premise is in doubt.

Watch

Extended reading notes

Core claim

The central claim is that simulator-grounded controllability and large-scale generative realism are not in tension: they can be reconciled by an intermediate 'layered world abstraction' (LWA) that splits a scene into per-object and per-region layers, each carrying pixel-level conditions such as depth and segmentation plus an explicit visibility mask. Dreamland refines the simulator's LWA into a real-world-aligned LWA using an editing model that preserves the traffic-participant and map-layout layers while hallucinating the background, then feeds that refined representation into a pretrained conditional diffusion model by concatenating condition latents with the noise latent through a small trainable projection layer. The paper reports that this yields an FID of 50.58 on the DIVA-Real reference distribution versus 106.02 for the previous hybrid system SimGen, with si-RMSE improving from 0.787 to 0.646 and segmentation mIoU from 0.752 to 0.791, and that the advantage grows when the same real-world-aligned LWA is fed to stronger pretrained generators.

Load-bearing premise

The whole Stage-2 transfer depends on the digital-twin reconstructions of nuPlan scenes being pixel-aligned with the real camera views, so the editing model can learn to preserve simulator object positions while changing appearance; any residual misalignment weakens the reported controllability gains.

Editorial extensions

If this is right

  • If correct, Dreamland lets users specify per-vehicle behavior in the simulator and receive photorealistic frames that keep those positions, making scenario editing as simple as changing the simulator state.
  • The real-world-aligned LWA as an early-aligned intermediate representation means new or stronger pretrained generation models can be swapped into Stage-3 with little or no adaptation, as demonstrated in the paper by the progression SDXL to SD3 to Flux.
  • The pipeline extends to video re-rendering (Dreamland-Video) and to other simulators (zero-shot on MetaUrban), so the same abstraction can power continuous scene generation for agent training and safety-critical scenario synthesis.
  • Downstream VQA fine-tuning improves by 3.9 absolute accuracy on real images, suggesting the generated data transfers to real-world visual understanding tasks for embodied agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension, not explored in the paper, is to use Dreamland's generated frames as on-policy data augmentation during reinforcement learning for driving policies, coupling the simulator's physical rollouts with the generator's visual variety; if the controllability metrics hold, policy behavior should differ measurably from training on raw simulator renders.
  • The principle that early alignment with real-world distributions lowers adaptation cost is a general design claim: it predicts that any future conditional generator, not only diffusion-based ones, can inherit Dreamland's LWA with only a small projection-layer update, a testable claim across model generations.
  • The controllability metrics rely on pretrained depth and segmentation estimators; a stricter test would be whether Dreamland's object bounding boxes reliably match the simulator's object semantics, for example by measuring instance-level IoU on preserved objects, which the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dreamland, a three-stage hybrid pipeline for controllable driving-scene generation that combines a physics-based simulator (MetaDrive) with large-scale pre-trained generative models. The first stage renders scenes in the simulator; the second stage refines a layered world abstraction (LWA) through an instructional editing model to map simulated conditions to real-world distributions (LWA-Sim2Real); the third stage adapts a pre-trained conditional diffusion model (Flux, SDXL, SD3) with a lightweight projection layer for mixed-condition generation. The authors introduce the D3Sim dataset of paired simulator/real conditions built from nuPlan via ScenarioNet, and report quantitative results (FID, si-RMSE, mIoU) against SimGen and other baselines, along with ablations, a video extension, and a downstream VQA experiment. The central claim is that Dreamland outperforms SimGen by 52.3% in FID and 17.9% in si-RMSE while preserving controllability and pretrained world knowledge.

Significance. If the reported results are substantiated, this is a valuable contribution to simulator-conditioned scene generation. The LWA representation and the two-stage refinement (Sim2Real editing followed by condition-following generation) are principled and practical, and the demonstrated plug-and-play compatibility with multiple base generators (SDXL, SD3, Flux) supports scalability. The D3Sim dataset and the video extension are useful community resources, and the downstream VQA experiment addresses an important application. However, the headline quantitative advantage over SimGen is currently undermined by an unmatched evaluation protocol and by internal inconsistencies in the FID numbers across tables. The paper's significance therefore hinges on whether the empirical claims survive a matched, clearly reported evaluation.

major comments (4)
  1. [Section 5 (Evaluation Metrics) and Table 1] The claimed 52.3% FID improvement over SimGen is not supported by a matched evaluation protocol. The main text states that FID is computed by rendering 16K scenes from the D3Sim validation set and comparing against DIVA-Real, while Appendix C.1 says FID is evaluated on 10,000 samples from DIVA-Real unless explicitly specified. It is not stated whether SimGen's 106.02 was computed on the same D3Sim validation scenes and prompts; Table D reports a different SimGen DIVA-Real FID (68.20) under the nuScenes protocol. To support the headline claim, the authors must evaluate both pipelines on the identical generated sample set (same scene records, text prompts, and sample count) and disclose the reference-set size and generated-sample count for every FID entry.
  2. [Tables 1, B, and D] The configuration labeled Dreamland (Flux) in Table 1 has FID 50.58, while Tables B and D report FID 44.61 for Dreamland / Real-LWA Finetuned Flux in the DIVA-Real column. The paper does not explain whether these are different evaluation corpora or different protocols. Without a clear statement of what each table measures, the reader cannot reconcile the numbers or verify the claimed improvements; each table should state the evaluation set, the number of generated samples, and the FID reference set.
  3. [Abstract and Section 1] The paper reports "50.8% improved image quality" in the Abstract, "52.3%" in Section 1, and "52.8%" in the contributions, while Section 5.1 and Table 1 give 52.3%. These inconsistent headline numbers need to be reconciled to the exact values in the tables.
  4. [Appendix C.1 and Table 3] The controllability metrics (si-RMSE, mIoU) are computed "in the preserved region" for Dreamland, but the paper does not state whether the same masked evaluation is applied to SimGen and to all Dreamland variants. If the preserved-region mask is not applied identically, the reported 17.9% si-RMSE improvement and the mIoU gains in Table 3 may reflect the evaluation region rather than the pipeline. The mask protocol should be described and applied uniformly to all compared methods.
minor comments (6)
  1. [Section 3.2, Eq. (5)] The loss notation "E Lb0" is typeset incorrectly; it should be written as an expectation over the target layer, e.g., E_{Lb0}[...].
  2. [Section 5.1] The sentence "Dreamland fine-tuned on Flux matches the visual fidelity of the frozen Flux model" is inconsistent with Table 1, where Dreamland (Flux) has FID 50.58 versus Dreamland (Frozen Flux) at 48.48; the wording should be revised to reflect the actual comparison.
  3. [Section 5.2] The phrase "naively supports" should be "natively supports".
  4. [Section 4] There is a stray period in "Figure. 3" in the text; it should be "Figure 3".
  5. [Table 2] The FID values for Dreamland-Video are reported for "all extracted frames," but the total number of frames or the per-video frame count is not given; this should be reported for reproducibility.
  6. [Section 5.4, Table 4] The row label "MetaVQA-Synthetic" is potentially confusing because the table also has a column named "Synthetic"; consider renaming the row for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified: Dreamland's claimed gains rest on a held-out generation evaluation with standard FID, si-RMSE, and mIoU metrics; the self-citations are to a public benchmark, an infrastructure platform, and a baseline, not to a theorem that forces the result.

full rationale

Dreamland's claimed derivation chain is an empirical pipeline, not an analytic derivation. Stage-2 (Eq. 5) trains an instructional editor to regress the Real-LWA background layer from the Sim-LWA while preserving simulator-provided layers; Stage-3 (Eq. 6) fine-tunes only a projection layer so a pretrained Flux-depth model denoises latents conditioned on the Real-LWA. Evaluation (Sec. 5, Table 1) then measures FID against the fixed DIVA-Real reference and measures controllability by comparing the simulator's depth/segmentation maps in the preserved region with maps re-estimated from generated images. Each training loss and each evaluation metric targets a different quantity, and no fitted parameter is renamed as a prediction. The FID reference and the SimGen baseline originate in the authors' own prior work, but they function as a public benchmark and baseline rather than as a theorem that forces the reported ranking; the paper additionally supports the FID ordering with a human user study (Table C). The D3Sim training and validation sets both derive from nuPlan digital twins, which is a data-distribution overlap worth auditing, but it is not a circular reduction because the validation set is separately filtered, re-prompted, and held out for generation evaluation. The Limitation section acknowledges longer inference time and expensive paired-data annotation, which are practical limitations, not admissions of circularity. Overall, the central claims are self-contained empirical comparisons.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

This is an empirical systems paper, so the ledger is short on theoretical free parameters. The main hand-made choices are the layer decomposition, the validation filtering rule, and resolution settings. The axioms are the heavy reliance on pretrained generative models, the pixel-alignment of digital twins, and the choice of evaluation metrics. LWA and D3Sim are invented artifacts whose independent evidence is currently nil because code and data are not yet public.

free parameters (3)
  • World layer decomposition = three layers (Ld, Ll, Lb)
    The paper chooses to split scenes into traffic participants, map layout, and background layers; the element-wise controllability claim depends on this hand-selected granularity, which is not derived from data or theory.
  • Default generation resolutions = 512 for Stage-2, 1024 for Stage-3
    Section 3.4 states these are set to balance cost and quality; changing them would change FID and controllability numbers.
  • Validation-set filtering rule = exclude occluded and non-visible objects
    Appendix A.3 describes an automated derivation pipeline that filters out occluded and ambiguous objects for the validation set; this hand-made rule directly affects the reported controllability metric.
assumptions (4)
  • domain assumption Pretrained generative models (Flux, ACE++, Cosmos) retain sufficient world knowledge for photorealistic driving scenes when adapted with projection-layer or LoRA fine-tuning.
    The whole pipeline stands on borrowed generative capability; Sections 3.3-3.4 and Appendix B depend on these pretrained checkpoints and their prompts.
  • domain assumption Digital-twin reconstructions of nuPlan scenarios are pixel-aligned with real sensor views after camera calibration, so paired Sim/Real conditions can be trained on.
    This is the load-bearing data premise; stated in Section 4 and Appendix A.1.
  • domain assumption FID computed against DIVA-Real is a valid and preferred image-quality measure for driving generation.
    The paper argues this in Appendix C.4, and all headline quality claims use this reference set rather than nuScenes FID.
  • domain assumption Depth and segmentation estimated by pretrained networks (DepthAnythingV2, SegFormer) can reliably measure controllability.
    si-RMSE and mIoU are computed in the space of these estimator outputs, not against simulator ground truth depth in the generated image; see Appendix C.1.
invented entities (2)
  • Layered World Abstraction (LWA)
    purpose: Intermediate representation that decomposes a scene into per-class world layers with visibility masks, bridging simulator output and generative model conditions.
    LWA is a new design artifact. It makes no falsifiable prediction outside the pipeline and its value is only demonstrated through task metrics; it plays the role of an invented representation rather than a discovered entity.
  • D3Sim dataset
    purpose: Paired synthetic/real driving dataset used to train and evaluate the hybrid pipeline.
    New dataset; not yet released at the time of writing, so its independent utility cannot be checked by outsiders.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dreamland: Controllable World Creation with Simulator and Generative Models." pith.science (2026). https://pith.science/paper/2TLNKXJ5

@misc{pith2026250608006,
  author       = {Pith},
  title        = {Pith review of: Dreamland: Controllable World Creation with Simulator and Generative Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TLNKXJ5}},
  note         = {Machine review of arXiv:2506.08006}
}
read the original abstract

Large-scale video generative models can synthesize diverse and realistic visual content for dynamic world creation, but they often lack element-wise controllability, hindering their use in editing scenes and training embodied AI agents. We propose Dreamland, a hybrid world generation framework combining the granular control of a physics-based simulator and the photorealistic content output of large-scale pretrained generative models. In particular, we design a layered world abstraction that encodes both pixel-level and object-level semantics and geometry as an intermediate representation to bridge the simulator and the generative model. This approach enhances controllability, minimizes adaptation cost through early alignment with real-world distributions, and supports off-the-shelf use of existing and future pretrained generative models. We further construct a D3Sim dataset to facilitate the training and evaluation of hybrid generation pipelines. Experiments demonstrate that Dreamland outperforms existing baselines with 50.8% improved image quality, 17.9% stronger controllability, and has great potential to enhance embodied agent training. Code and data will be made available.

Figures

Figures reproduced from arXiv: 2506.08006 by the authors.

Figure 1
Figure 1. Controllable world creation with Dreamland. It combines a simulator for physically grounded scene generation and a large-scale pretrained generative model for creating a realistic visual world following user-provided text prompts. Abstract Large-scale video generative models can synthesize diverse and realistic visual content for dynamic world creation, but they often lack element-wise controlla￾bility, hindering th… view at source ↗
Figure 2
Figure 2. Illustration of the Dreamland. It is a three-stage pipeline including: (1) Simulation, (2) Layered World Abstraction Simulator-to-Real-World, (3) Mix-Condition Generation. 2 Related Work Controllable Visual Generative Models. Large-scale foundational generative models have been developed rapidly in recent years for image generation [7–12], video generation [13–16], and multimodal generation [20–25]. While those foun… view at source ↗
Figure 4
Figure 4. Rendering diverse appearances align to the simulator’s conditions. Dreamland can simulate the driving scene and render it into high-quality scenes with generative models. 1 2 3 4 5 6 7 8 Number of Object 1.5k 3k 4.5k 835 2165 5138 5135 2156 930 371 129 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Object distribution of the validation dataset. Validation Dataset. The validation dataset is used to evaluate both the visual quality and controllability of the generation pipeline. While the digital twin scenarios provide realistic views and distribution of the real-w…
Figure 5
Figure 5. Figure 5: Qualitative comparison with baseline methods. Dreamland with fine-tuned Flux matches the visual fidelity of the frozen Flux model while improving adherence to simulator conditions. (a) Simulator-conditioned scene editing (b) Safety-critical scenes (c) Zero-shot on Meta…
Figure 6
Figure 6. Figure 6: Extension of Dreamland. Dreamland pipeline is generalized to various downstream tasks. Baselines. Our main baselines include generative models for autonomous driving that take scene layout conditions. SimGen [19] first obtains front-view observation from the MetaDrive …
Figure 7
Figure 7. Figure 7: Qualitative Results. Our Dreamland-Video pipeline can re-render the simulator reference video into realistic and diverse video frames while maintaining the pretrained world knowledge. Visual Quality Controllability Method f-30 FID ↓ f-60 FID ↓ f-90 FID ↓ f-120 FID ↓ si…
Figure 8
Figure 8. Figure 8: Qualitative Comparison of Stage-2 model’s design choices. Image Quality Pipeline Controllability Stage-2 Controllability Method FID ↓ si-RMSE ↓ mIoU ↑ si-RMSE ↓ mIoU ↑ Dreamland w/o editing 63.21 0.713 0.658 0.709 0.649 Dreamland 50.58 0.647 0.791 0.672 0.950 [PITH_FU…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 3D and 4D World Modeling: A Survey

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

Reference graph

Works this paper leans on

76 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Llama 2 team. Llama 2: Open foundation and fine-tuned chat models, 2023. 2

    work page 2023

  2. [2]

    The llama 3 herd of models.ArXiv, abs/2407.21783, 2024

    Abhimanyu Dubey and et al. The llama 3 herd of models.ArXiv, abs/2407.21783, 2024

    arXiv 2024

  3. [3]

    Gemma: Open models based on gemini research and technology, 2024

    Gemma Team. Gemma: Open models based on gemini research and technology, 2024

    work page 2024

  4. [4]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    arXiv 2023

  5. [5]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  6. [6]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 2

    2023

  7. [7]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2022. 2, 3

    work page 2022

  8. [8]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024. 21

    work page 2024

Show all 76 references
  1. [9]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine l...

    2024

  2. [10]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. InInternational Conference on Machine Learning, 2022

    2022

  3. [11]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational Conference on Machine Learning, 2021

    2021

  4. [12]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 2022. 2, 3

    2022 arXiv

  5. [13]

    https://openai.com/index/sora/

    Sora: Creating video from text — openai.com. https://openai.com/index/sora/. [Accessed 30-04- 2025]. 2, 3

    2025

  6. [14]

    Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    2024 arXiv

  7. [15]

    Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    2024 arXiv

  8. [16]

    Latte: Latent diffusion transformer for video generation.arXiv preprint arXiv:2401.03048, 2024

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation.arXiv preprint arXiv:2401.03048, 2024. 2, 3 10

    2024 arXiv

  9. [17]

    Learning visual parkour from generated images

    Alan Yu, Ge Yang, Ran Choi, Yajvan Ravan, John Leonard, and Phillip Isola. Learning visual parkour from generated images. In8th Annual Conference on Robot Learning, 2024. 2, 3

    2024

  10. [18]

    Simworld: A unified benchmark for simulator-conditioned scene generation via world model.arXiv preprint arXiv:2503.13952,

    Xinqing Li, Ruiqi Song, Qingyu Xie, Ye Wu, Nanxin Zeng, and Yunfeng Ai. Simworld: A unified benchmark for simulator-conditioned scene generation via world model.arXiv preprint arXiv:2503.13952,

    arXiv

  11. [19]

    Simgen: Simulator-conditioned driving scene generation.Advances in Neural Information Processing Systems, 37:48838–48874, 2024

    Yunsong Zhou, Michael Simon, Zhenghao Mark Peng, Sicheng Mo, Hongzi Zhu, Minyi Guo, and Bolei Zhou. Simgen: Simulator-conditioned driving scene generation.Advances in Neural Information Processing Systems, 37:48838–48874, 2024. 2, 5, 6, 7, 18, 21, 24

    2024

  12. [20]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024. 3

    2024 arXiv

  13. [21]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  14. [22]

    Llamafusion: Adapting pretrained language models for multimodal generation.arXiv preprint arXiv:2412.15188, 2024

    Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Llamafusion: Adapting pretrained language models for multimodal generation.arXiv preprint arXiv:2412.15188, 2024

    2024 arXiv

  15. [23]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. 2023

    2023

  16. [24]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  17. [25]

    X-fusion: Introducing new modality to frozen large language models, 2025

    Sicheng Mo, Thao Nguyen, Xun Huang, Siddharth Srinivasan Iyer, Yijun Li, Yuchen Liu, Abhishek Tandon, Eli Shechtman, Krishna Kumar Singh, Yong Jae Lee, Bolei Zhou, and Yuheng Li. X-fusion: Introducing new modality to frozen large language models, 2025. 3

    2025

  18. [26]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InInternational Conference on Computer Vision, 2023. 3, 6

    2023

  19. [27]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2023

    2023

  20. [28]

    T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Association for the Advancement of Artificial Intelligence, 2024

    2024

  21. [29]

    HumanSD: A native skeleton-guided diffusion model for human image generation

    Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. HumanSD: A native skeleton-guided diffusion model for human image generation. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2023

    2023

  22. [30]

    Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition

    Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2024

    2024

  23. [31]

    Plug-and-play diffusion features for text- driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. InIEEE/CVF Computer Vision and Pattern Recognition Conference, pages 1921–1930, 2023

    1921

  24. [32]

    Ctrl-x: Controlling structure and appearance for text-to-image generation without guidance.Advances in Neural Information Processing Systems, 37:128911–128939, 2024

    Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, and Bolei Zhou. Ctrl-x: Controlling structure and appearance for text-to-image generation without guidance.Advances in Neural Information Processing Systems, 37:128911–128939, 2024. 3

    2024

  25. [33]

    Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2023. 3

    2023

  26. [34]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.arXiv preprint arxiv:2308.06721, 2023

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.arXiv preprint arxiv:2308.06721, 2023. 11

    2023 arXiv

  27. [35]

    Break-a-scene: Extracting multiple concepts from a single image

    Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from a single image. InACM Special Interest Group on Computer Graphics and Interactive Techniques Asia, 2023

    2023

  28. [36]

    Null-text inversion for editing real images using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2023. 3

    2023

  29. [37]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2023. 3

    2023

  30. [38]

    Ace++: Instruction-based image creation and editing via context-aware content filling.arXiv preprint arXiv:2501.02487, 2025

    Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, and Jingren Zhou. Ace++: Instruction-based image creation and editing via context-aware content filling.arXiv preprint arXiv:2501.02487, 2025. 3, 5, 18

    2025 arXiv

  31. [39]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. InAdvances in Neural Information Processing Systems, 2023. 3

    2023

  32. [40]

    Uni-controlnet: All-in-one control to text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:11127–11150, 2023

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:11127–11150, 2023. 3

    2023

  33. [41]

    Street-view image generation from a bird’s-eye view layout.IEEE Robotics and Automation Letters, 2024

    Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird’s-eye view layout.IEEE Robotics and Automation Letters, 2024. 3, 21

    2024

  34. [42]

    Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout.arXiv preprint arXiv:2308.01661, 2023

    Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout.arXiv preprint arXiv:2308.01661, 2023. 21, 24

    2023 arXiv

  35. [43]

    Magicdrive: Street view generation with diverse 3d geometry control, 2024

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control, 2024. 3

    2024

  36. [44]

    Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control.arXiv preprint arXiv:2411.13807, 2024

    Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control.arXiv preprint arXiv:2411.13807, 2024. 3

    2024 arXiv

  37. [45]

    Drivedreamer: Towards real-world- driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real-world- driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023

    2023 arXiv

  38. [46]

    Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model.arXiv preprint arXiv:2310.07771, 2023

    Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model.arXiv preprint arXiv:2310.07771, 2023. 3, 21, 24

    2023 arXiv

  39. [47]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  40. [48]

    Infinicube: Unbounded and controllable dynamic 3d driving scene generation with world-guided video models.arXiv preprint arXiv:2412.03934, 2024

    Yifan Lu, Xuanchi Ren, Jiawei Yang, Tianchang Shen, Zhangjie Wu, Jun Gao, Yue Wang, Siheng Chen, Mike Chen, Sanja Fidler, et al. Infinicube: Unbounded and controllable dynamic 3d driving scene generation with world-guided video models.arXiv preprint arXiv:2412.03934, 2024. 3

    2024 arXiv

  41. [49]

    Genad: Generative end-to- end autonomous driving

    Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to- end autonomous driving. InEuropean Conference on Computer Vision, pages 87–104. Springer, 2024. 3

    2024

  42. [50]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.17398, 2024

    2024 arXiv

  43. [51]

    Gaia-1: A generative world model for autonomous driving, 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, 2023

    2023

  44. [52]

    Drivegan: Towards a controllable high-quality neural simulation, 2021

    Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation, 2021. 3

    2021

  45. [53]

    Vid2sim: Realistic and interactive simulation from video for urban navigation.arXiv preprint arXiv:2501.06693, 2025

    Ziyang Xie, Zhizheng Liu, Zhenghao Peng, Wayne Wu, and Bolei Zhou. Vid2sim: Realistic and interactive simulation from video for urban navigation.arXiv preprint arXiv:2501.06693, 2025. 3 12

    2025 arXiv

  46. [54]

    Vr-robo: A real-to- sim-to-real framework for visual robot navigation and locomotion.arXiv preprint arXiv:2502.01536, 2025

    Shaoting Zhu, Linzhan Mou, Derun Li, Baijun Ye, Runhan Huang, and Hang Zhao. Vr-robo: A real-to- sim-to-real framework for visual robot navigation and locomotion.arXiv preprint arXiv:2502.01536, 2025. 3

    2025 arXiv

  47. [55]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024. 5, 8, 18, 21

    2024

  48. [56]

    Tan et al

    K. Tan et al. H. Caesar, J. Kabzan. Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. InCVPR ADP3 workshop, 2021. 5, 16

    2021

  49. [57]

    Scenari- onet: Open-source platform for large-scale traffic scenario simulation and modeling.Advances in Neural Information Processing Systems, 2023

    Quanyi Li, Zhenghao Peng, Lan Feng, Zhizheng Liu, Chenda Duan, Wenjie Mo, and Bolei Zhou. Scenari- onet: Open-source platform for large-scale traffic scenario simulation and modeling.Advances in Neural Information Processing Systems, 2023. 5, 16, 24

    2023

  50. [58]

    Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 5, 7, 9, 16, 24

    2022

  51. [59]

    Cosmos world foundation model platform for physical ai, 2025

    NVIDIA, :, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei ...

    2025

  52. [60]

    Metaurban: An embodied ai simulation platform for urban micromobility.International Conference on Learning Representation, 2025

    Wayne Wu, Honglin He, Jack He, Yiran Wang, Chenda Duan, Zhizheng Liu, Quanyi Li, and Bolei Zhou. Metaurban: An embodied ai simulation platform for urban micromobility.International Conference on Learning Representation, 2025. 8

    2025

  53. [61]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations, 2022. 8, 9

    2022

  54. [62]

    Embodied scene understanding for vision language models via metavqa.arXiv preprint arXiv:2501.09167, 2025

    Weizhen Wang, Chenda Duan, Zhenghao Peng, Yuxin Liu, and Bolei Zhou. Embodied scene understanding for vision language models via metavqa.arXiv preprint arXiv:2501.09167, 2025. 9, 24

    2025 arXiv

  55. [63]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Conference on Computer Visi...

    2024

  56. [64]

    Depth anything v2.arXiv:2406.09414, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2.arXiv:2406.09414, 2024. 16, 18

    2024 arXiv

  57. [65]

    Segformer: Simple and efficient design for semantic segmentation with transformers.Advances in neural information processing systems, 34:12077–12090, 2021

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers.Advances in neural information processing systems, 34:12077–12090, 2021. 16, 18

    2021

  58. [66]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023. 16

    2023 arXiv

  59. [67]

    Sam 2: Segment anything in images and videos, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

    2024

  60. [68]

    The llama 3 herd of models, 2024

    Aaron Grattafiori et al. The llama 3 herd of models, 2024. 16

    2024

  61. [69]

    Rehg, and Pinar Yanardag

    Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M. Rehg, and Pinar Yanardag. Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models. InIEEE/CVF Computer Vision and Pattern Recognition Conference, 2024. 18 13

    2024

  62. [70]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017. 18

    2017

  63. [71]

    Depth map prediction from a single image using a multi-scale deep network.Advances in neural information processing systems, 27, 2014

    David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network.Advances in neural information processing systems, 27, 2014. 18

    2014

  64. [72]

    Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023. 21, 24

    2023 arXiv

  65. [73]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027, 2019. 21, 24

    1903 arXiv

  66. [74]

    Street-view image generation from a bird’s-eye view layout.IEEE Robotics and Automation Letters, 2024

    Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird’s-eye view layout.IEEE Robotics and Automation Letters, 2024. 24

    2024

  67. [75]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang,...

    2020

  68. [76]

    Image A is better

    using ScenarioNet [57]. This results in scene records corresponding to more than 20,000 digital twin scenarios of 15-20 seconds in length and up to 10 Hz sample rate. To get the conditions (e.g., depth, segmentation) in the simulation domain (Sim Conditions), we replay the ego...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.