Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OccScene claims that a text-guided diffusion generator and a semantic occupancy predictor can be trained jointly so each improves the other, yielding better generation and perception than either task alone.

desk verdict Real empirical gains, but the paper's own Algorithm 1 doesn't implement the mutual-gradient mechanism that its headline claim and Table VIII depend on. read the letter →

arxiv 2412.11183 v2 pith:52V3PTDQ submitted 2024-12-15 cs.CV

Bohan Li , Xin Jin , Jianan Wang , Yukai Shi , Yasheng Sun , Xiaofeng Wang , Zhuang Ma , Baao Xie
show 3 more authors
Chao Ma Xiaokang Yang Wenjun Zeng
This is my paper · ORCID
classification cs.CV
keywords text-guided3Dscenegenerationsemanticoccupancypredictiondiffusionmodelsmutuallearningcross-taskMambastate-spacemodelcross-viewvideoautonomousdrivingscenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OccScene sets out to show that 3D scene generation and 3D perception need not be separate stages: a text-conditioned latent diffusion model and a semantic occupancy prediction network can be trained in one joint loop, with each task supplying supervision to the other. The generator is conditioned, at every denoising step, on the occupancy predicted from the current noisy image, while the perception network is trained on those same noisy images and on freshly generated scenes. The paper reports that the loop improves both sides: generation FID falls from 47.82 to 15.54 on NYUv2, from 60.55 to 19.86 on SemanticKITTI, and from 16.20 to 11.87 on nuScenes, and the occupancy models gain several mIoU points, for example MonoScene rises from 26.94 to 29.78 on NYUv2. If this mutual-learning effect is real, it would mean generators and perceptual models can bootstrap each other without ground-truth 3D labels at inference time.

What carries the argument

The load-bearing mechanical piece is the Mamba-based Dual Alignment (MDA) module, which projects the predicted occupancy grid and the diffusion latent into spatio-temporal patches and scans them with a bidirectional Mamba state-space block, augmented by a cross-view camera encoding that multiplies deformable-3D-convolution occupancy features with per-view camera parameters. The aligned occupancy features are added back to the latent through a zero-convolution residual connection, preserving the pretrained UNet’s capabilities. It carries the cross-view consistency of generated videos and injects fine-grained semantic and geometric context into the generation process.

What would settle it

Train the perception branch with gradients detached but with the identical noisy-image augmentation and synthetic data pairs; if the attached-gradient mIoU advantage disappears, the mutual-information story is falsified and the gains reduce to data augmentation. A more direct check is to evaluate the occupancy predictor at increasing input-noise levels: if supervision at high noise contributes nothing or hurts, removing high-noise steps from the loss rather than reweighting them would match or beat OccScene.

Watch

Extended reading notes

Core claim

The central claim is that a semantic occupancy predictor and a latent diffusion generator can be coupled in a single denoising loop so that each improves the other. During training, a clean image is VAE-encoded, noise is added, and the noisy latent is both denoised by the UNet and decoded into a noisy image for the perception model, which predicts a semantic occupancy grid; that grid is aligned with the diffusion latent through the Mamba-based Dual Alignment module and used as an additional condition. The total loss is $L = L_{\mathrm{LDM}} + \sqrt{\bar{\alpha}_t} L_p$, so perception supervision is downweighted at high noise levels and emphasized at low noise levels. At inference, the same loop runs: occupancy predicted from the partially denoised image guides the next denoising step, and the final output is an RGB scene together with its occupancy grid. The paper argues this is not “the blind leading the blind” because the perception and generation learners develop different representations of the same scene, and aligning those representations pushes both toward flatter minima and better generalization.

Load-bearing premise

The perception model can extract genuinely useful semantic occupancy from noisy, partially denoised images, and training on that noisy distribution improves its performance on clean real images rather than teaching it generator-specific artifacts.

Editorial extensions

If this is right

  • If the central claim holds, text-only prompts are enough to emit geometrically consistent RGB and occupancy pairs, removing the need for ground-truth 3D labels at inference time.
  • The perception gains generalize across architectures: MonoScene, NDC-Scene, ISO, TPVFormer, and OccFormer all improve when trained inside the loop, suggesting the scheme acts as a plug-and-play boost rather than a quirk of one backbone.
  • Generated RGB-occupancy pairs can train a from-scratch occupancy predictor, so the generator can augment real data in low-annotation regimes, with IoU rising from 18.4 to 21.3 and mIoU from 6.9 to 10.2 on nuScenes.
  • Occupancy-based cross-attention control enables text-guided video editing with stable geometry across views, extending single-view prompt editing to multi-view consistent scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same joint loop may transfer to other dense 3D perception tasks such as depth estimation, BEV segmentation, or panoptic occupancy, wherever a differentiable predictor can be conditioned on partially denoised latents; the paper does not test this.
  • Editorial inference: part of the reported gain could be data augmentation in disguise, coming from noisy images and fresh synthetic pairs rather than true mutual information; the paper’s detached-gradient ablation partially addresses this, but a controlled experiment that holds augmentation fixed while toggling gradient flow would separate the two effects cleanly.
  • Editorial inference: because supervision is weighted by $\sqrt{\bar{\alpha}_t}$, the practical signal comes mostly from late, low-noise steps; a testable extension is to restrict perception supervision to steps beyond a noise threshold and measure whether the mutual benefit saturates or grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. OccScene proposes a unified framework in which a latent diffusion generator and a semantic occupancy predictor are trained jointly, with the generator conditioned on text and on occupancy predicted from noisy images, and the perception model trained on noisy inputs and generated data. The Mamba-based Dual Alignment (MDA) module injects the occupancy condition into the diffusion UNet. The paper reports state-of-the-art generation FID scores on NYUv2 (15.54), SemanticKITTI (19.86), and nuScenes-Occupancy (11.87), together with consistent mIoU gains for several perception baselines (e.g., MonoScene 26.94 to 29.78 on NYUv2 and 11.08 to 14.98 on SemanticKITTI). The central claim is that perception and generation improve each other through a joint diffusion process, rather than using the generator merely as a data augmenter.

Significance. If the mutual-learning mechanism is as claimed, OccScene would be a valuable cross-task framework: it produces paired RGB and semantic-occupancy data from text alone and simultaneously improves a downstream perception model. The paper has clear strengths: extensive experiments on three datasets, consistent gains across multiple perception baselines, ablations for the joint diffusion scheme (Table VII), the gradient-attachment setting (Table VIII), and MDA components (Tables IX and X). The perception evaluation uses held-out real test sets, so the reported improvements are not circular in the sense of fitting the evaluation distribution. However, the central mutual-learning claim is not consistently supported by the algorithm as written, and several comparison protocols are incompletely specified. These issues affect the interpretation of the headline results rather than only their presentation.

major comments (4)
  1. [Algorithm 1 and Eq. (1)] The update rule in Algorithm 1 differentiates the reconstruction loss L_LDM with respect to the generator parameters θ and the occupancy loss L_p with respect to the perception parameters δ. Since X_occ = f_δ(y) enters L_LDM through the conditioning path, a joint training scheme should also contain the cross-term ∇_δ L_LDM, or explicitly state that this gradient is stopped. Neither appears in Algorithm 1, Eq. (1), or the surrounding text. As written, the perception model receives no training signal about whether its occupancy predictions help generation. This is in direct tension with Table VIII, where 'attached gradients' consistently outperform 'detached gradients' under identical data augmentation; that result is exactly the evidence for the mutual-learning mechanism. The authors must specify which mechanism was actually implemented: if ∇_δ L_LDM is included, the algorithm and loss equations are incomplete; if it is not included, Table VIII cannot be attributed to mutual learning and the central claim needs to be revised.
  2. [Section IV-C, Tables II–IV] The perception baseline rows in Tables II–IV appear to be the published scores of MonoScene, NDC-Scene, TPVFormer, OccFormer, and ISO, while the '+ours' rows are trained or fine-tuned with additional synthetic data and noisy-input supervision. The paper does not state whether each baseline was retrained under the identical data schedule, number of iterations, and loss weighting. Without this control, the mIoU gains could reflect extra training data or longer training rather than the mutual-learning mechanism. The authors should specify the exact training protocol for both baseline and '+ours' models, or provide a matched-budget comparison.
  3. [Table I(b)] In Table I(b), the baselines (DriveGAN, DriveDreamer, BEVGen, BEVControl, MagicDrive) are evaluated at 224×400, while OccScene is evaluated at 256×448. FID is not resolution-invariant, and the claimed improvement over MagicDrive (16.20 to 11.87) may be affected by the resolution difference. Please report all methods at a common resolution, or provide a resolution-matched comparison, to make the generation-quality claim clean.
  4. [Section III-D] The theoretical analysis in Section III-D is presented as an explanation of why mutual learning works, but it is a heuristic analogy to flat minima and posterior entropy rather than a derivation. In particular, the claim that 'increasing posterior entropy enables both learners to converge towards a more robust solution' is not supported by any formal argument or by direct measurements such as loss-landscape sharpness. This is not by itself disqualifying, but the paper should either soften the claim or provide the missing evidence, especially since the mutual-learning claim depends on the contested gradient-attachment result.
minor comments (5)
  1. [Figure captions] The captions of Figures 6 and 7 say 'Quantitative comparison', but the figures show qualitative examples; these should read 'Qualitative comparison'.
  2. [Section IV-D, Table X] In the discussion of Table X, the two ablation settings are both called 'w/o MDA-D'; the second occurrence should be 'w/o MDA-T'.
  3. [Algorithm 1] Algorithm 1 does not define the distance/norm used for the two losses; please write the losses explicitly (e.g., with squared L2 norm for the LDM objective and the MonoScene loss for L_p) so the update is unambiguous.
  4. [Section IV-C] The text mentions 'SemanticKITT' instead of 'SemanticKITTI', and 'Nuscene-Occupancy' has inconsistent capitalization.
  5. [Figure 9] The claim that occupancy becomes 'more complete and accurate' as denoising proceeds is supported only by qualitative images; a quantitative mIoU-vs-denoising-step curve would be more convincing.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: held-out real benchmarks anchor both perception and generation claims, though the perception-side synthetic labels are self-generated and the 'attached gradients' mechanism is under-specified.

  1. other [Algorithm 1 (lines 5-7) and Section IV-C 'Training Support for Semantic Occupancy Prediction' (Table VI)]
    "Xocc = fδ(y) ... To produce the data pairs, we generate the same amount of images as the original dataset. Note that the Semantic Occupancy Prediction model of MonoScene [23] is trained from scratch on the synthetic data."

    The synthetic image-occupancy pairs use occupancy labels Xocc = fδ(y), the perception model's own predictions, and the generator is conditioned on that same Xocc. The 'ground truth' in the synthetic pairs is thus a function of the model being improved, not an independent external label. The paper's stated source of additional knowledge (different representations and flatter minima, Section III-D) is therefore self-referential: the loop recycles fδ's outputs as pseudo-labels. Held-out real test evaluation prevents test-set circularity, but the perception-side improvement in the support experiment is, by construction, pseudo-label self-training rather than the claimed mutual-information exchange.

full rationale

The paper's central claims are anchored to external benchmarks: generation fidelity is measured by FID/FVD against real datasets (Tables I, V, XI) and perception gains are measured by mIoU on held-out NYUv2, SemanticKITTI, and nuScenes-Occupancy test/validation sets (Tables II-IV, VI). There is no load-bearing self-citation chain; the cited mutual-learning intuition [27]-[29] and flat-minima results are external and heuristic. The one genuinely self-referential component is the synthetic-pair pipeline: occupancy labels for generated data come from the perception model itself (Xocc = fδ(y)), so the 'new knowledge' for perception is a recycled version of the model's own outputs. This is a limitation and a mild circularity in the framing, but it does not force the reported numbers because the test-time evaluation is independent. A separate correctness concern, not circularity, is that Algorithm 1 shows no ∇δ term from the diffusion reconstruction loss (only ∇δ||X̃occ - Xocc||), while Table VIII attributes gains to 'attached gradients'; if the implementation includes that cross-gradient, the pseudocode omits it, and if it does not, the Table VIII attribution is unsupported. This inconsistency does not make the empirical results circular, but it should be resolved in a revision. Overall circularity is minor.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework adds no new physical entities. It introduces a new architecture (MDA) and a joint training scheme, and depends on the quality of pretrained SD and MonoScene, on three manually chosen loss weights, and on the informal wide-minima hypothesis. These are the main unverified inputs.

free parameters (3)
  • lambda_ce = not reported
    Class weighting loss coefficient in Lp (Eq. 2), chosen by hand without sensitivity analysis.
  • lambda_sem = not reported
    Semantic loss coefficient in Eq. 2.
  • lambda_geo = not reported
    Geometry loss coefficient in Eq. 2.
assumptions (3)
  • domain assumption Pretrained Stable Diffusion and MonoScene provide strong, transferable representations.
    The method inherits all biases and capabilities of the frozen SD UNet and the pre-trained MonoScene perception model; Section III-A and III-B.
  • domain assumption Wide minima / flat valleys lead to better generalization, and mutual learning finds such minima.
    Section III-D explicitly relies on [73,74] and asserts without proof that joint training broadens the minima.
  • ad hoc to paper The occupancy predicted from noisy images is still reliable enough to guide generation.
    Section III-B uses noisy images as perception inputs; the sqrt(alpha_t_bar) weighting is a patch to handle unreliability rather than a guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation." pith.science (2026). https://pith.science/paper/52V3PTDQ

@misc{pith2026241211183,
  author       = {Pith},
  title        = {Pith review of: OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/52V3PTDQ}},
  note         = {Machine review of arXiv:2412.11183}
}
read the original abstract

Recent diffusion models have demonstrated remarkable performance in both 3D scene generation and perception tasks. Nevertheless, existing methods typically separate these two processes, acting as a data augmenter to generate synthetic data for downstream perception tasks. In this work, we propose OccScene, a novel mutual learning paradigm that integrates fine-grained 3D perception and high-quality generation in a unified framework, achieving a cross-task win-win effect. OccScene generates new and consistent 3D realistic scenes only depending on text prompts, guided with semantic occupancy in a joint-training diffusion framework. To align the occupancy with the diffusion latent, a Mamba-based Dual Alignment module is introduced to incorporate fine-grained semantics and geometry as perception priors. Within OccScene, the perception module can be effectively improved with customized and diverse generated scenes, while the perception priors in return enhance the generation performance for mutual benefits. Extensive experiments show that OccScene achieves realistic 3D scene generation in broad indoor and outdoor scenarios, while concurrently boosting the perception models to achieve substantial performance improvements in the 3D perception task of semantic occupancy prediction.

Figures

Figures reproduced from arXiv: 2412.11183 by the authors.

Figure 1
Figure 1. OccScene synthesizes realistic scene generation in RGB and semantic occupancy pairs from customized text prompts. (a): [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The architecture of the proposed Mamba-based Dual Alignment (MDA) module for occupancy-based constraint condition, which is mainly composed of the Cross-view Camera Encoding and the Mamba-based Sequential Feature Encoding. The Cross￾view Camera Encoding incorporates camera parameters P with the semantic occupancy Xocc for camera trajectory awareness, while the Mamba-based Sequential Feature Encoding processes the se… view at source ↗
Figure 4
Figure 4. Visualization results of the heat maps from our proposed [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: Quantitative comparison of scene generation with existing methods. The compared methods of SD [7] and ControlNet [33] [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Quantitative comparison of cross-view generation consistency with existing methods. Our method generates more [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 9
Figure 9. Figure 9: Generation results of different denoising steps. As the [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 11
Figure 11. Figure 11: Comparison of different architecture designs of the Mamba-based Dual Alignment (MDA) module. Although the [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 3D and 4D World Modeling: A Survey

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

Reference graph

Works this paper leans on

90 extracted references · 52 canonical work pages · cited by 1 Pith paper

  1. [1]

    Semantic scene completion using local deep implicit functions on lidar data,

    C. B. Rist, D. Emmerichs, M. Enzweiler, and D. M. Gavrila, “Semantic scene completion using local deep implicit functions on lidar data,” IEEE transactions on pattern analysis and machine intelligence , vol. 44, no. 10, 2021

    2021

  2. [2]

    Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,

    Y . Liao, J. Xie, and A. Geiger, “Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3292–3310, 2022

    2022

  3. [3]

    Anisotropic convolutional neural networks for rgb-d based semantic scene completion,

    J. Li, P. Wang, K. Han, and Y . Liu, “Anisotropic convolutional neural networks for rgb-d based semantic scene completion,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 11, pp. 8125– 8138, 2021

    2021

  4. [4]

    From front to rear: 3d semantic scene completion through planar convolution and attention- based network,

    J. Li, Q. Song, X. Yan, Y . Chen, and R. Huang, “From front to rear: 3d semantic scene completion through planar convolution and attention- based network,” IEEE Transactions on Multimedia , 2023

    2023

  5. [5]

    Stereoscene: Bev-assisted stereo matching empow- ers 3d semantic scene completion,

    B. Li, Y . Sun, X. Jin, W. Zeng, Z. Zhu, X. Wang, Y . Zhang, J. Okae, H. Xiao, and D. Du, “Stereoscene: Bev-assisted stereo matching empow- ers 3d semantic scene completion,” arXiv preprint arXiv:2303.13959 , 2023

    arXiv 2023

  6. [6]

    Diffusion probabilistic models for 3d point cloud generation,

    S. Luo and W. Hu, “Diffusion probabilistic models for 3d point cloud generation,” in CVPR, 2021, pp. 2837–2845

    2021

  7. [7]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10 684–10 695

    2022

  8. [8]

    Conditional diffusion process for inverse halftoning,

    H. Jiang and Y . Mu, “Conditional diffusion process for inverse halftoning,” NeurIPS, vol. 35, pp. 5498–5509, 2022

    2022

Show all 90 references
  1. [9]

    One at a time: Progressive multi-step volumetric probability learning for reliable 3d scene perception,

    B. Li, Y . Sun, J. Dong, Z. Zhu, J. Liu, X. Jin, and W. Zeng, “One at a time: Progressive multi-step volumetric probability learning for reliable 3d scene perception,” in AAAI, 2024

    2024

  2. [10]

    Bevcontrol: Accu- rately controlling street-view elements with multi-perspective consistency via bev sketch layout,

    K. Yang, E. Ma, J. Peng, Q. Guo, D. Lin, and K. Yu, “Bevcontrol: Accu- rately controlling street-view elements with multi-perspective consistency via bev sketch layout,” arXiv preprint arXiv:2308.01661 , 2023

    2023 arXiv

  3. [11]

    Street-view image generation from a bird’s-eye view layout,

    A. Swerdlow, R. Xu, and B. Zhou, “Street-view image generation from a bird’s-eye view layout,” IEEE Robotics and Automation Letters , 2024

    2024

  4. [12]

    Drivedreamer: Towards real-world-driven world models for autonomous driving,

    X. Wang, Z. Zhu, G. Huang, X. Chen, and J. Lu, “Drivedreamer: Towards real-world-driven world models for autonomous driving,” arXiv preprint arXiv:2309.09777, 2023

    2023 arXiv

  5. [13]

    Gan augmentation: Augmenting training data using generative adversarial networks,

    C. Bowles, L. Chen, R. Guerrero, P. Bentley, R. Gunn, A. Hammers, D. A. Dickie, M. V . Hernández, J. Wardlaw, and D. Rueckert, “Gan augmentation: Augmenting training data using generative adversarial networks,” arXiv preprint arXiv:1810.10863 , 2018

    arXiv

  6. [14]

    Integrating geometric control into text-to-image diffusion models for high-quality de- tection data generation via text prompt,

    K. Chen, E. Xie, Z. Chen, L. Hong, Z. Li, and D.-Y . Yeung, “Integrating geometric control into text-to-image diffusion models for high-quality de- tection data generation via text prompt,” arXiv preprint arXiv:2306.04607, 2023

    2023 arXiv

  7. [15]

    Detdiffusion: Synergizing generative and perceptive models for enhanced data generation and perception,

    Y . Wang, R. Gao, K. Chen, K. Zhou, Y . Cai, L. Hong, Z. Li, L. Jiang, D.-Y . Yeung, Q. Xu et al., “Detdiffusion: Synergizing generative and perceptive models for enhanced data generation and perception,” CVPR, 2024

    2024

  8. [16]

    Is synthetic data from generative models ready for image recognition?

    R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi, “Is synthetic data from generative models ready for image recognition?” arXiv preprint arXiv:2210.07574 , 2022

    2022 arXiv

  9. [17]

    Is a prompt and a few samples all you need? using gpt-4 for data augmentation in low-resource classification tasks,

    A. G. Møller, J. A. Dalsgaard, A. Pera, and L. M. Aiello, “Is a prompt and a few samples all you need? using gpt-4 for data augmentation in low-resource classification tasks,” arXiv preprint arXiv:2304.13861 , 2023

    2023 arXiv

  10. [18]

    Open- vocabulary object segmentation with diffusion models,

    Z. Li, Q. Zhou, X. Zhang, Y . Zhang, Y . Wang, and W. Xie, “Open- vocabulary object segmentation with diffusion models,” in ICCV, 2023

    2023

  11. [19]

    Datasetdm: Synthesizing data with perception annotations using diffusion models,

    W. Wu, Y . Zhao, H. Chen, Y . Gu, R. Zhao, Y . He, H. Zhou, M. Z. Shou, and C. Shen, “Datasetdm: Synthesizing data with perception annotations using diffusion models,” in NeurIPS, 2023

    2023

  12. [20]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,

    G. Zhao, X. Wang, Z. Zhu, X. Chen, G. Huang, X. Bao, and X. Wang, “Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,” arXiv preprint arXiv:2403.06845 , 2024

    2024 arXiv

  13. [21]

    Magicdrive: Street view generation with diverse 3d geometry control,

    R. Gao, K. Chen, E. Xie, L. Hong, Z. Li, D.-Y . Yeung, and Q. Xu, “Magicdrive: Street view generation with diverse 3d geometry control,” in ICLR, 2024

    2024

  14. [22]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,

    Y . Wang, J. He, L. Fan, H. Li, Y . Chen, and Z. Zhang, “Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,” in CVPR, 2024

    2024

  15. [23]

    Monoscene: Monocular 3d semantic scene completion,

    A.-Q. Cao and R. de Charette, “Monoscene: Monocular 3d semantic scene completion,” in CVPR, 2022

    2022

  16. [24]

    beta-vae: Learning basic visual concepts with a constrained variational framework,

    I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “beta-vae: Learning basic visual concepts with a constrained variational framework,” in ICLR, 2016

    2016

  17. [25]

    Dif- fusion autoencoders: Toward a meaningful and decodable representation,

    K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, “Dif- fusion autoencoders: Toward a meaningful and decodable representation,” in CVPR, 2022

    2022

  18. [26]

    Closed-loop unsupervised representation disentanglement with beta- vae distillation and diffusion probabilistic feedback,

    X. Jin, B. Li, B. Xie, W. Zhang, J. Liu, Z. Li, T. Yang, and W. Zeng, “Closed-loop unsupervised representation disentanglement with beta- vae distillation and diffusion probabilistic feedback,” arXiv preprint arXiv:2402.02346, 2024

    2024 arXiv

  19. [27]

    Deep mutual learning,

    Y . Zhang, T. Xiang, T. M. Hospedales, and H. Lu, “Deep mutual learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4320–4328

    2018

  20. [28]

    Mutualnet: Adaptive convnet via mutual learning from different model configurations,

    T. Yang, S. Zhu, M. Mendieta, P. Wang, R. Balakrishnan, M. Lee, T. Han, M. Shah, and C. Chen, “Mutualnet: Adaptive convnet via mutual learning from different model configurations,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 1, pp. 811–827, 2021

    2021

  21. [29]

    Spatial-frequency mutual learning for face super-resolution,

    C. Wang, J. Jiang, Z. Zhong, and X. Liu, “Spatial-frequency mutual learning for face super-resolution,” in CVPR, 2023

    2023

  22. [30]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML. PMLR, 2015

    2015

  23. [31]

    Diffustereo: High quality human reconstruction via diffusion-based stereo using sparse cameras,

    R. Shao, Z. Zheng, H. Zhang, J. Sun, and Y . Liu, “Diffustereo: High quality human reconstruction via diffusion-based stereo using sparse cameras,” in ECCV 2022, 2022

    2022

  24. [32]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation,

    K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu, “T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation,” NeurIPS, 2023

    2023

  25. [33]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in ICCV, 2023

    2023

  26. [34]

    Plug-and-play diffusion features for text-driven image-to-image translation,

    N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in CVPR, 2023

    2023

  27. [35]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,

    J. Z. Wu, Y . Ge, X. Wang, S. W. Lei, Y . Gu, Y . Shi, W. Hsu, Y . Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in ICCV, 2023

    2023

  28. [36]

    Video generation models as world simulators,

    T. Brooks, B. Peebles, C. Holmes, W. DePue, Y . Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh, “Video generation models as world simulators,” 2024. [Online]. Available: https: //openai.com/research/video-generation-models-as-world-simulators

    2024

  29. [37]

    Learning universal policies via text-guided video generation,

    Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuur- mans, and P. Abbeel, “Learning universal policies via text-guided video generation,” NeurIPS, 2024

    2024

  30. [38]

    Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation,

    Y . Liu, L. Li, S. Ren, R. Gao, S. Li, S. Chen, X. Sun, and L. Hou, “Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation,” NeurIPS, 2024

    2024

  31. [39]

    Subjectdrive: Scaling generative data in autonomous driving via subject control,

    B. Huang, Y . Wen, Y . Zhao, Y . Hu, Y . Liu, F. Jia, W. Mao, T. Wang, C. Zhang, C. W. Chen et al., “Subjectdrive: Scaling generative data in autonomous driving via subject control,”arXiv preprint arXiv:2403.19438, 2024

    2024 arXiv

  32. [40]

    Mars: An instance-aware, modular and realistic simulator for autonomous driving,

    Z. Wu, T. Liu, L. Luo, Z. Zhong, J. Chen, H. Xiao, C. Hou, H. Lou, Y . Chen, R. Yang, Y . Huang, X. Ye, Z. Yan, Y . Shi, Y . Liao, and H. Zhao, “Mars: An instance-aware, modular and realistic simulator for autonomous driving,” CICAI, 2023

    2023

  33. [41]

    Street gaussians for modeling dynamic urban scenes,

    Y . Yan, H. Lin, C. Zhou, W. Wang, H. Sun, K. Zhan, X. Lang, X. Zhou, and S. Peng, “Street gaussians for modeling dynamic urban scenes,” arXiv preprint arXiv:2401.01339 , 2024

    2024 arXiv

  34. [42]

    Diverse and aligned audio-to-video generation via text-to-video model adaptation,

    G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz, and Y . Adi, “Diverse and aligned audio-to-video generation via text-to-video model adaptation,” in AAAI, 2024

    2024

  35. [43]

    Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering,

    Y . Xiangli, L. Xu, X. Pan, N. Zhao, A. Rao, C. Theobalt, B. Dai, and D. Lin, “Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering,” ECCV, 2022

    2022

  36. [44]

    Street-view image generation from a bird’s-eye view layout,

    A. Swerdlow, R. Xu, and B. Zhou, “Street-view image generation from a bird’s-eye view layout,” 2024

    2024

  37. [45]

    Panacea: Panoramic and controllable video generation for autonomous driving,

    Y . Wen, Y . Zhao, Y . Liu, F. Jia, Y . Wang, C. Luo, C. Zhang, T. Wang, X. Sun, and X. Zhang, “Panacea: Panoramic and controllable video generation for autonomous driving,” 2023

    2023

  38. [46]

    Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion,

    L. Nunes, R. Marcuzzi, B. Mersch, J. Behley, and C. Stachniss, “Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion,” in Proc. SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 15 of the IEEE/CVF Conf. on Computer Vision and Pattern Recog...

    2024

  39. [47]

    Semcity: Semantic scene generation with triplane diffusion,

    J. Lee, S. Lee, C. Jo, W. Im, J. Seon, and S.-E. Yoon, “Semcity: Semantic scene generation with triplane diffusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2024

    2024

  40. [48]

    3d semantic scene completion: A survey,

    L. Roldao, R. De Charette, and A. Verroust-Blondet, “3d semantic scene completion: A survey,” International Journal of Computer Vision , vol. 130, no. 8, 2022

    2022

  41. [49]

    Semantic scene completion from a single depth image,

    S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser, “Semantic scene completion from a single depth image,” in CVPR, 2017

    2017

  42. [50]

    Lmscnet: Lightweight multiscale 3d semantic completion,

    L. Roldao, R. de Charette, and A. Verroust-Blondet, “Lmscnet: Lightweight multiscale 3d semantic completion,” in 3DV, 2020

    2020

  43. [51]

    Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion,

    X. Yan, J. Gao, J. Li, R. Zhang, Z. Li, R. Huang, and S. Cui, “Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion,” in AAAI, 2021

    2021

  44. [52]

    Scpnet: Semantic scene completion on point cloud,

    Z. Xia, Y . Liu, X. Li, X. Zhu, Y . Ma, Y . Li, Y . Hou, and Y . Qiao, “Scpnet: Semantic scene completion on point cloud,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 17 642–17 651

    2023

  45. [53]

    Pasco: Urban 3d panoptic scene completion with uncertainty awareness,

    A.-Q. Cao, A. Dai, and R. de Charette, “Pasco: Urban 3d panoptic scene completion with uncertainty awareness,” in CVPR, 2024

    2024

  46. [54]

    Scenerf: Self-supervised monocular 3d scene reconstruction with radiance fields,

    A.-Q. Cao and R. De Charette, “Scenerf: Self-supervised monocular 3d scene reconstruction with radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 9387–9398

    2023

  47. [55]

    Behind the scenes: Density fields for single view reconstruction,

    F. Wimbauer, N. Yang, C. Rupprecht, and D. Cremers, “Behind the scenes: Density fields for single view reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 9076–9086

    2023

  48. [56]

    Selfocc: Self- supervised vision-based 3d occupancy prediction,

    Y . Huang, W. Zheng, B. Zhang, J. Zhou, and J. Lu, “Selfocc: Self- supervised vision-based 3d occupancy prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 19 946–19 956

    2024

  49. [57]

    Indoor segmentation and support inference from rgbd images

    N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images.” ECCV, 2012

    2012

  50. [58]

    S3cnet: A sparse semantic scene completion network for lidar point clouds,

    R. Cheng, C. Agia, Y . Ren, X. Li, and L. Bingbing, “S3cnet: A sparse semantic scene completion network for lidar point clouds,” in Conference on Robot Learning , 2021

    2021

  51. [59]

    Scfusion: Real-time incremental scene reconstruction with semantic completion,

    S.-C. Wu, K. Tateno, N. Navab, and F. Tombari, “Scfusion: Real-time incremental scene reconstruction with semantic completion,” in 3DV, 2020

    2020

  52. [60]

    V oxformer: Sparse voxel transformer for camera-based 3d semantic scene completion,

    Y . Li, Z. Yu, C. Choy, C. Xiao, J. M. Alvarez, S. Fidler, C. Feng, and A. Anandkumar, “V oxformer: Sparse voxel transformer for camera-based 3d semantic scene completion,” CVPR, 2023

    2023

  53. [61]

    Hierarchical temporal context learning for camera-based semantic scene completion,

    B. Li, J. Deng, W. Zhang, Z. Liang, and D. Du, “Hierarchical temporal context learning for camera-based semantic scene completion,” in ECCV, 2024

    2024

  54. [62]

    Tri-perspective view for vision-based 3d semantic occupancy prediction,

    Y . Huang, W. Zheng, Y . Zhang, J. Zhou, and J. Lu, “Tri-perspective view for vision-based 3d semantic occupancy prediction,” in CVPR, 2023

    2023

  55. [63]

    Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction,

    Y . Zhang, Z. Zhu, and D. Du, “Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction,” ICCV, 2023

    2023

  56. [64]

    Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving,

    Y . Wei, L. Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu, “Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving,” in ICCV, 2023

    2023

  57. [65]

    Image super-resolution via iterative refinement,

    C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022

    2022

  58. [66]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020

    2020

  59. [67]

    Prompt-to-prompt image editing with cross attention control,

    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y . Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” arXiv preprint arXiv:2208.01626 , 2022

    2022 arXiv

  60. [68]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752 , 2023

    2023 arXiv

  61. [69]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,” arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  62. [70]

    Vmamba: Visual state space model,

    Y . Liu, Y . Tian, Y . Zhao, H. Yu, L. Xie, Y . Wang, Q. Ye, and Y . Liu, “Vmamba: Visual state space model,” arXiv preprint arXiv:2401.10166 , 2024

    2024 arXiv

  63. [71]

    Pointmamba: A simple state space model for point cloud analysis,

    D. Liang, X. Zhou, X. Wang, X. Zhu, W. Xu, Z. Zou, X. Ye, and X. Bai, “Pointmamba: A simple state space model for point cloud analysis,” arXiv preprint arXiv:2402.10739, 2024

    2024 arXiv

  64. [72]

    On large-batch training for deep learning: Generalization gap and sharp minima,

    N. S. Keskar, J. Nocedal, P. T. P. Tang, D. Mudigere, and M. Smelyanskiy, “On large-batch training for deep learning: Generalization gap and sharp minima,” in ICLR, 2017

    2017

  65. [73]

    Entropy-sgd: Biasing gradient descent into wide valleys,

    P. Chaudhari, A. Choromanska, S. Soatto, Y . LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina, “Entropy-sgd: Biasing gradient descent into wide valleys,” Journal of Statistical Mechanics: Theory and Experiment , 2019

    2019

  66. [74]

    Understanding deep learning requires rethinking generalization,

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” Communications of the ACM, 2021

    2021

  67. [75]

    Semantickitti: A dataset for semantic scene understanding of lidar sequences,

    J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, “Semantickitti: A dataset for semantic scene understanding of lidar sequences,” in ICCV, 2019

    2019

  68. [76]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in CVPR, 2020

    2020

  69. [77]

    Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception,

    X. Wang, Z. Zhu, W. Xu, Y . Zhang, Y . Wei, X. Chi, Y . Ye, D. Du, J. Lu, and X. Wang, “Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception,” ICCV, 2023

    2023

  70. [78]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” NeurIPS, 2017

    2017

  71. [79]

    Towards accurate generative models of video: A new metric & challenges,

    T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, “Towards accurate generative models of video: A new metric & challenges,” arXiv preprint arXiv:1812.01717 , 2018

    2018 arXiv

  72. [80]

    Drivegan: Towards a controllable high-quality neural simulation,

    S. W. Kim, J. Philion, A. Torralba, and S. Fidler, “Drivegan: Towards a controllable high-quality neural simulation,” in CVPR, 2021

    2021

  73. [81]

    Anisotropic convolutional networks for 3d semantic scene completion,

    J. Li, K. Han, P. Wang, Y . Liu, and X. Yuan, “Anisotropic convolutional networks for 3d semantic scene completion,” in CVPR, 2020

    2020

  74. [82]

    3d sketch-aware semantic scene completion via semi-supervised structure prior,

    X. Chen, K.-Y . Lin, C. Qian, G. Zeng, and H. Li, “3d sketch-aware semantic scene completion via semi-supervised structure prior,” in CVPR, 2020

    2020

  75. [83]

    Ndc- scene: Boost monocular 3d semantic scene completion in normalized device coordinates space,

    J. Yao, C. Li, K. Sun, Y . Cai, H. Li, W. Ouyang, and H. Li, “Ndc- scene: Boost monocular 3d semantic scene completion in normalized device coordinates space,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 9455–9465

    2023

  76. [84]

    Monocular occupancy prediction for scalable indoor scenes,

    H. Yu, Y . Wang, Y . Chen, and Z. Zhang, “Monocular occupancy prediction for scalable indoor scenes,” in European Conference on Computer Vision. Springer, 2024, pp. 38–54

    2024

  77. [85]

    Context and geometry aware voxel transformer for semantic scene completion,

    Z. Yu, R. Zhang, J. Ying, J. Yu, X. Hu, L. Luo, S.-Y . Cao, and H.-L. Shen, “Context and geometry aware voxel transformer for semantic scene completion,” arXiv preprint arXiv:2405.13675 , 2024

    2024 arXiv

  78. [86]

    Diffusion probabilistic models for scene-scale 3d categorical data,

    J. Lee, W. Im, S. Lee, and S.-E. Yoon, “Diffusion probabilistic models for scene-scale 3d categorical data,” arXiv preprint arXiv:2301.00527 , 2023

    2023 arXiv

  79. [87]

    Demystifying mmd gans,

    M. Bi´nkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Demystifying mmd gans,” arXiv preprint arXiv:1801.01401 , 2018

    2018 arXiv

  80. [88]

    Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,

    Y . Li, R. Yu, C. Shahabi, and Y . Liu, “Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,” in ICLR, 2018

    2018

  81. [89]

    A review of recurrent neural network-based methods in computational physiology,

    S. Mao and E. Sejdi ´c, “A review of recurrent neural network-based methods in computational physiology,” IEEE Transactions on Neural Networks and Learning Systems , 2022. Bohan Li (Student Member, IEEE) received the B.E. degree from the School of Control Engineering, Northeas...

    2022

  82. [2021]

    He was a Sr

    He is also the founding Executive Director of the Ningbo Institute of Digital Twin. He was a Sr. Principal Research Manager and a member of the Senior Leadership Team at Microsoft Research Asia, Beijing, from 2014 to 2021, where he led the video analytics research empowering t...

    2014

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.