Pith. sign in

REVIEW 2 major objections 7 minor 1 cited by

$I^{2}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting

T0 review · 2 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read I²-World claims that decoupling 4D occupancy tokenization into intra-scene and inter-scene residual quantizers, and conditioning an encoder-decoder forecaster on a predicted transformation matrix, outperforms prior 4D occupancy world…

desk verdict The tokenizer design is a legitimate efficiency contribution, but the SOTA forecasting claim rests on test-time access to the ground-truth future ego pose, and the paper's own ablation shows the gains vanish without it. read the letter →

arxiv 2507.09144 v2 pith:OOPYC2MM submitted 2025-07-12 cs.CV

Zhimin Liao , Ping Wei , Ruijie Zhang , Shuaijia Chen , Haoxuan Wang , Ziyang Ren This is my paper
classification cs.CV
keywords 4Doccupancyforecastingworldmodelsresidualquantizationintra-intertokenizationautonomousdrivingtransformationmatrixconditioningencoder-decodertransformerzero-shotgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that 4D occupancy forecasting for autonomous driving can be made more accurate and much cheaper by separating spatial compression from temporal compression. Rather than tokenizing each 3D frame alone or tokenizing whole spatiotemporal stacks, I²-World first compresses the current scene with multi-scale residual quantization and then adds residual temporal tokens obtained from historical frames aligned by the ego pose. This keeps the token count close to that of a 3D tokenizer while carrying motion information, which the authors say is why their model outperforms prior occupancy world models by 25.1% mIoU and 36.9% IoU on Occ3D-nuScenes. The forecasting transformer is encoder-decoder instead of decoder-only: it predicts a $4\times4$ transformation matrix from the current scene and planned motion, then conditions the decoder on that matrix to generate the next scene tokens autoregressively. If the claims hold, occupancy world models become practical for latency-critical driving, needing only 2.9 GB of training memory and inferring at 37.0 FPS.

What carries the argument

The load-bearing mechanism is the I²-Scene Tokenizer's two-stream residual quantization. An Intra-Scene Tokenizer applies $S$ scales of multi-scale residual quantization to one shared codebook, and an Inter-Scene Tokenizer keeps $G$ historical feature maps in a memory queue, aligns each to the current frame with an ego-pose transformation matrix $T_{t-g}^{t}$, and quantizes the residual that remains. A single shared codebook keeps spatial and temporal tokens in the same discrete space, and the predicted transformation matrix $T_{t+k+1}^{t+k}$ is injected into the Inter-Decoder, making generation spatially consistent with ego motion. This is what lets the model keep the compactness of 3D tokenizers while carrying temporal dynamics, and it also provides the handle for fine-grained control of generated scenes.

What would settle it

Run the Occ3D-nuScenes forecasting benchmark once with the ground-truth transformation matrix withheld from I²-World, so it must predict its own ego motion as the baselines implicitly do, and once with the same ground-truth matrix supplied to the baselines; if the mIoU gap over DOME and UniScene disappears or falls well below 25.1%, the headline advantage depends on the oracle pose input.

Watch

Extended reading notes

Core claim

The central claim is that dynamic 4D scene forecasting does not require a monolithic 4D tokenizer or a decoder-only autoregressive world model. The I²-Scene Tokenizer decouples tokenization: an intra-scene stream quantizes the current occupancy map at multiple resolutions by residual subtraction against a shared codebook, and an inter-scene stream quantizes the residual left after subtracting temporally aligned historical feature maps, so motion enters the token stream as residual modifications of spatial codes rather than as extra spatial dimensions. The I²-Former then predicts a transformation matrix from the current tokens and an action-plan embedding, and it uses that matrix plus a queue of historical tokens to condition the next-timestep token map. On Occ3D-nuScenes, the occupancy-input variant reports 39.73 mIoU and 49.80 IoU averaged over 1s, 2s, and 3s forecasts, surpassing the previous best results by 25.1% mIoU and 36.9% IoU, and the camera-based variant reports 18.97 mIoU and 28.77 IoU, surpassing prior camera-based baselines by 50.9% mIoU and 40.9% IoU. The same frozen model also transfers zero-shot to Occ3D-Waymo, roughly doubling forecasting mIoU over a copy-paste baseline at both 10 Hz and 2 Hz sampling rates.

Load-bearing premise

The load-bearing premise of the reported gains is that it is fair to give I²-World the ground-truth future ego transformation matrix during evaluation while the comparison methods do not receive that future information; if that oracle pose were removed or given to the baselines too, the 25.1% mIoU margin could shrink.

Editorial extensions

If this is right

  • Averaged over 1s, 2s, and 3s forecasts on Occ3D-nuScenes, the occupancy-input I²-World-O reports 39.73 mIoU and 49.80 IoU, improving on the prior best by 25.1% mIoU and 36.9% IoU.
  • With camera-based occupancy predictions as input, I²-World-STC reports 18.97 mIoU and 28.77 IoU, beating prior camera-based baselines by 50.9% mIoU and 40.9% IoU.
  • The model requires about 2.9 GB of training memory and runs at 37.0 FPS on an RTX 4090, so real-time 4D occupancy forecasting is feasible with much smaller compute than LLM- or diffusion-based world models.
  • Frozen tokenizer and forecaster transfer zero-shot to Occ3D-Waymo, raising forecasting mIoU from 28.34 to 43.73 at 10 Hz and from 17.17 to 36.38 at 2 Hz over a copy-paste baseline.
  • Transformation-matrix conditioning provides two levels of controllability: high-level action commands such as turn left or right, and fine-grained per-step spatial transformation for scenario-specific generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not claimed by the paper, but implied by its setup: the reported margin may partly reflect an information asymmetry, because evaluation gives I²-World the ground-truth future ego transformation matrix while comparison baselines do not receive future information; a symmetric comparison with predicted poses would test how much of the gain is architectural.
  • The residual intra/inter tokenization recipe is not tied to occupancy grids; the same align-and-quantize-the-residual idea could be applied to point cloud, radar, or video sequence tokenizers, where motion is also a sparse residual on top of static structure.
  • Because the transformation matrix is a compact, interpretable control channel, a natural deployment extension is to learn the pose forecast and use it at inference, turning the oracle-conditioned evaluation into a fully closed-loop system that predicts its own future motion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes I2-World, a 4D occupancy forecasting framework that decouples scene tokenization into an intra-scene multi-scale residual quantizer and an inter-scene temporal quantizer, and combines these with an encoder-decoder autoregressive transformer conditioned on a transformation matrix. The authors report state-of-the-art results on Occ3D-nuScenes for 4D occupancy forecasting, with mIoU of 39.73 and IoU of 49.80, along with high efficiency (2.9 GB training memory and 37.0 FPS). They also report zero-shot generalization on Occ3D-Waymo and demonstrate controllable generation via commands and transformation matrices.

Significance. The tokenizer design is interesting and the authors provide a released codebase and extensive ablations. However, the central forecasting claim is not supported because the evaluation protocol supplies the ground-truth future ego transformation matrix at test time, while the comparison baselines do not receive this future information. The paper's own ablation shows that removing this conditioning drops performance to 17.12 mIoU, essentially identical to OccWorld's 17.14 mIoU. Thus the reported 25.1% mIoU improvement is not an unconditional forecasting result; it is a conditional generation result with an oracle. The tokenizer contribution, while potentially valuable for reconstruction, does not by itself lift forecasting performance above existing methods without this leaked future pose.

major comments (2)
  1. [Section 4.1 and Section 3.3, Table 4] The evaluation protocol invalidates the headline claim. Section 4.1 states: 'For evaluation, we employ the ground-truth transformation matrix to guide generation.' Section 3.3 defines T^{t+k+1}_{t+k} as the transformation matrix between consecutive future timesteps, which encodes the future ego motion. At test time, the model is therefore given the answer's ego trajectory. Table 4 shows the effect of this conditioning: without translation or rotation conditioning, mIoU is 17.12; adding translation raises it to 28.74, adding rotation to 20.34, and adding both to 34.25; the full model reaches 39.73. Thus the majority of the reported forecasting gain comes from the ground-truth future pose, not from the proposed tokenizer or architecture. The central claim of state-of-the-art 4D occupancy forecasting is therefore not supported as an unconditional forecasting result.
  2. [Section 4.2, Table 1 vs Table 4] The comparison with baselines is not apples-to-apples. The baselines in Table 1 (OccWorld, DOME, UniScene, DFIT-OccWorld, etc.) do not receive the ground-truth future transformation matrix, whereas I2-World does. The paper's own ablation in Table 4 shows that without this oracle, I2-World achieves 17.12 mIoU and 27.75 IoU, essentially equal to OccWorld's 17.14 mIoU and 26.63 IoU. Therefore, the claimed gains of 25.1% mIoU and 36.9% IoU conflate the model's forecasting ability with the information leaked by the oracle. To support the headline claim, the authors would need to either evaluate without the ground-truth future pose and report those numbers as primary, or give the same oracle information to all baselines and show that I2-World still outperforms them under a fair protocol.
minor comments (7)
  1. [Section 1 and abstract] The phrasing 'nearly requires 2.9 GB of training memory' is awkward; consider 'requires only 2.9 GB' or 'requires approximately 2.9 GB'.
  2. [Section 4.2] The text states that the I2-World-STC variant 'outperforms prior methods by 50.9% (18.97 vs. 12.57) in mIoU', but Table 1 lists no baseline with 12.57 mIoU; the closest STC comparison is DOME-STC at 14.53 mIoU, which would be a 30.6% improvement. Please clarify or correct this number.
  3. [Table 4 caption] The column header 'With' is ambiguous; the caption explains it denotes the Intra-Encoder, but this should be stated explicitly in the caption rather than only in the text.
  4. [Section 3.2, Eqs. (2)-(4)] The symbol B_t is reused for both the original feature map and the residual being updated, which makes the update equations difficult to follow; using distinct symbols (e.g., R_t for the residual) would improve readability.
  5. [Section 5 (Conclusion)] The conclusion says the paper presents a framework for '3D scene generation', but the paper's topic is 4D occupancy forecasting; this wording is inconsistent.
  6. [Section 4.1] In 'the I2Former undergoes 48 epochs of training', there is a missing space between 'I2' and 'Former'.
  7. [Table 1 and Section 4.2] The abstract and Section 4.2 highlight 37.0 FPS, but Table 1 shows 37.04 FPS for I2-World-O and only 4.21 FPS for I2-World-STC; please clarify which configuration the efficiency claim refers to.

Circularity Check

1 steps flagged · score 7.0 of 10

The headline forecasting result is conditioned on the ground-truth future ego transform at test time; without that oracle the model matches OccWorld.

  1. other [Section 4.1, 'Experimental Setup'; Section 3.3, 'I2-Former']
    "For evaluation, we employ the ground-truth transformation matrix to guide generation ... The transformation matrix T^{t+k+1}_{t+k} ∈ R4×4 that maps the scene state from timestep from t + k to t + k + 1 ... the Inter-Decoder utilizes T^{t+k+1}_{t+k} as a spatiotemporal condition to autoregressively predict the subsequent latent token."

    At inference, the model receives T^{t+k+1}_{t+k}, the ground-truth relative ego pose between the current frame and the future frame it must predict. Because the occupancy grid is ego-centric, this transform encodes how the entire scene will shift into the next frame, i.e., a large part of the target future occupancy. The Inter-Decoder is explicitly conditioned on this matrix to produce the next latent token, so the reported forecast is a conditional reconstruction that consumes a component of the answer rather than an independent prediction. Table 4 confirms this: without conditioning the model scores 17.12 mIoU, nearly identical to OccWorld's 17.14, and the SOTA 39.73 mIoU is obtained only after adding ground-truth translation and rotation conditioning.

full rationale

Most of the paper's architecture is self-contained and not circular: the intra/inter tokenizer, residual quantization, codebook losses, and the inter-scene alignment with past ego poses are all internally consistent and are not defined in terms of the future target. The forecasting loss is supervised by the tokenizer's own tokens, which is a normal training setup. The single load-bearing circular step is in the evaluation protocol for the headline forecasting claim. Section 4.1 states that 'For evaluation, we employ the ground-truth transformation matrix to guide generation.' The T matrix is the relative ego pose between the current and the to-be-predicted future frame, and the decoder is conditioned on it to produce the next latent token. Since the occupancy grid is ego-centric, this input substantially specifies the target future scene; the model is not forecasting that component but receiving it. Table 4 makes the dependence quantitative: with no conditioning the model reaches only 17.12 mIoU, essentially OccWorld's 17.14, and the reported SOTA is reached only after adding GT translation and rotation conditioning. Thus the 25.1% mIoU / 36.9% IoU 'state-of-the-art' claim reduces, by construction, to the use of a future oracle. The STCOcc self-citation for the STC variant is not load-bearing in the same way, since STCOcc is a separately published perception model and the tokenizer/forecast design can be evaluated independently of it. Overall, the paper has a genuine tokenization contribution, but the central forecasting comparison is compromised; I assign 7 rather than 8-10 because the tokenizer itself is not circular and the oracle affects only the headline evaluation, not the formal derivation of the model.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central forecasting result rests on several hand-set hyperparameters (codebook size, scales, history length, temporal weights) and on two domain assumptions: occupancy as the scene representation and codebook transfer across scales. The most consequential assumption is the availability of the ground-truth transformation matrix at evaluation, which is an oracle input not provided to comparison methods.

free parameters (7)
  • Codebook size N = 512
    Number of discrete codes in the shared codebook (Section 4.1); generation quality depends on this capacity.
  • Latent dimension C = 128
    Codebook token dimension (Section 4.1); chosen by hand.
  • Number of quantization scales S = not stated
    Intra-scene tokenizer residual scales (Eq. 2); the paper never gives this number, which controls compression ratio.
  • History length G = 4 frames implied (2 s at 2 Hz), not stated
    Number of historical frames used in inter-scene tokenization and I2-Former memory; a free design choice affecting temporal modeling.
  • Temporal loss weights w_k = not stated
    Frame-wise weights in Eq. 7; assigned by hand but not reported.
  • Commitment loss beta = 1
    Default value in Eq. 5 for the VQ loss.
  • Intra-Encoder layers L = 3
    Default depth of the encoder hierarchy (Section 3.3).
assumptions (4)
  • domain assumption Occupancy grids faithfully represent the 3D scene for driving.
    The entire pipeline operates on precomputed occupancy grids from Occ3D; conclusions are limited to that representation.
  • domain assumption The shared codebook trained on intra-scene tokens transfers to inter-scene residual tokens without a direct VQ loss.
    Eq. 4 quantizes inter tokens with the same codebook, but Eq. 5 only supervises intra-scale codebook learning; this assumes cross-scale semantic consistency.
  • ad hoc to paper Ground-truth future ego pose is available at evaluation.
    Section 4.1: 'we employ the ground-truth transformation matrix to guide generation'; this is unique to the proposed evaluation and is not available to baselines.
  • domain assumption Feature-level MSE on token maps is a sufficient learning signal for next-scene generation.
    Eq. 7 supervises generated token maps directly against tokenizer outputs without a perceptual or occupancy-level loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $I^{2}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting." pith.science (2026). https://pith.science/paper/OOPYC2MM

@misc{pith2026250709144,
  author       = {Pith},
  title        = {Pith review of: $I^2$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOPYC2MM}},
  note         = {Machine review of arXiv:2507.09144}
}
abstract

Forecasting the evolution of 3D scenes and generating unseen scenarios via occupancy-based world models offers substantial potential for addressing corner cases in autonomous driving systems. While tokenization has revolutionized image and video generation, efficiently tokenizing complex 3D scenes remains a critical challenge for 3D world models. To address this, we propose $I^{2}$-World, an efficient framework for 4D occupancy forecasting. Our method decouples scene tokenization into intra-scene and inter-scene tokenizers. The intra-scene tokenizer employs a multi-scale residual quantization strategy to hierarchically compress 3D scenes while preserving spatial details. The inter-scene tokenizer residually aggregates temporal dependencies across timesteps. This dual design preserves the compactness of 3D tokenizers while retaining the dynamic expressiveness of 4D tokenizers. Unlike decoder-only GPT-style autoregressive models, $I^{2}$-World adopts an encoder-decoder architecture. The encoder aggregates spatial context from the current scene and predicts a transformation matrix to enable high-level control over scene generation. The decoder, conditioned on this matrix and historical tokens, ensures temporal consistency during generation. Experiments demonstrate that $I^{2}$-World achieves state-of-the-art performance, outperforming existing methods by 25.1\% in mIoU and 36.9\% in IoU for 4D occupancy forecasting while exhibiting exceptional computational efficiency: it requires merely 2.9 GB of training memory and achieves real-time inference at 37.0 FPS. Our code is available on https://github.com/lzzzzzm/II-World.

Figures

Figures reproduced from arXiv: 2507.09144 by the authors.

Figure 1
Figure 1. (a) Our proposed tokenizer versus previous tokenizer methods. We decouple scene tokenization into intra-scene and inter-scene components, which enable our method to maintain the computational efficiency of 3D tokenizers while modeling dynamic information akin to 4D tokenizers. (b) Comparison with the state of the art. Our method significantly outperforms the previous state￾of-the-art method, particularly in 3-second… view at source ↗
Figure 2
Figure 2. The overall architecture of I 2 -World. Our framework consists of two core components: (1) the I 2 -Scene Tokenizer, which dynamically converts 3D scene data into compact tokens to enable efficient representation learning; and (2) the I 2 -Former, an autoregressive transformer conditioned on transformation matrices that iteratively leverages the compact tokens for forecasting. frame and the subsequent frame to mitig… view at source ↗
Figure 3
Figure 3. Qualitative results of 4D occupancy forecasting on Occ3d-nus validation set. occupancy forecasting dataset derived from the nuScenes autonomous driving dataset [6]. Occ3D-nus contains 600 training scenes and 150 validation scenes. The spatial cover￾age of each sample spans [−40, m, 40, m] along the x- and y-axes and [−1, m, 5.4, m] along the z-axis, with a uniform voxel resolution of 0.4 m in all dimensions and a sa… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Demonstration of the controllability of our methods. The first column in each case represents the conditioned frame. Controllable Generation [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 3D and 4D World Modeling: A Survey

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

Reference graph

Works this paper leans on

62 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2

    arXiv 2023

  2. [2]

    Cosmos world foundation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint, 2025. 2

    work page 2025

  3. [3]

    V-jepa 2: Self- supervised video models enable understanding, prediction and planning

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self- supervised video models enable understanding, prediction and planning. arXiv preprint, 2025. 2

    work page 2025

  4. [4]

    Semantickitti: A dataset for semantic scene understanding of lidar sequences

    Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2019. 1

    work page 2019

  5. [5]

    The lov ´asz-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks

    Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lov ´asz-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In CVPR, 2018. 4

    work page 2018

  6. [6]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 6

    work page 2020

  7. [7]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR,

    work page

  8. [8]

    A survey of world models for autonomous driving

    Tuo Feng, Wenguan Wang, and Yi Yang. A survey of world models for autonomous driving. NIPS, 2025. 1

    work page 2025

Show all 62 references
  1. [9]

    Magicdrive: Street view generation with diverse 3d geometry control

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. ICLR, 2024. 2

    2024

  2. [10]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In NIPS, 2024. 1, 2, 3

    2024

  3. [11]

    Dome: Tam- ing diffusion model into high-fidelity controllable occupancy world model

    Songen Gu, Wei Yin, Bu Jin, Xiaoyang Guo, Junming Wang, Haodong Li, Qian Zhang, and Xiaoxiao Long. Dome: Tam- ing diffusion model into high-fidelity controllable occupancy world model. arXiv preprint arXiv:2410.10429, 2024. 1, 2, 3, 4, 6, 7

    2024 arXiv

  4. [12]

    Bevdet: High-performance multi-camera 3d object detection in bird-eye-view

    Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Da- long Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv:2112.11790, 2021. 1

    2021 arXiv

  5. [13]

    Tri-perspective view for vision-based 3d se- mantic occupancy prediction

    Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d se- mantic occupancy prediction. In CVPR, 2023. 1

    2023

  6. [14]

    Differentiable raycasting for self- supervised occupancy forecasting

    Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differentiable raycasting for self- supervised occupancy forecasting. In ECCV, 2022. 2

    2022

  7. [15]

    Point cloud forecasting as a proxy for 4d occupancy forecast- ing

    Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecast- ing. In CVPR, 2023. 2

    2023

  8. [16]

    Pointpillars: Fast encoders for object detection from point clouds

    Alex H Lang, Sourabh V ora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019. 1

    2019

  9. [17]

    Autoregressive image generation using resid- ual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using resid- ual quantization. In CVPR, 2022. 2, 4

    2022

  10. [18]

    Uniscene: Unified occupancy-centric driving scene generation

    Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation. arXiv preprint, 2024. 2, 6

    2024

  11. [19]

    Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo

    Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo. In AAAI,

  12. [20]

    Bevdepth: Acquisition of reliable depth for multi-view 3d object detec- tion

    Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detec- tion. In AAAI, 2023. 1

    2023

  13. [21]

    V oxformer: Sparse voxel transformer for camera- based 3d semantic scene completion

    Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anand- kumar. V oxformer: Sparse voxel transformer for camera- based 3d semantic scene completion. In CVPR, 2023. 1

    2023

  14. [22]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022. 1, 5

    2022

  15. [23]

    Fb-occ: 3d occupancy prediction based on forward-backward view transformation

    Zhiqi Li, Zhiding Yu, David Austin, Mingsheng Fang, Shiyi Lan, Jan Kautz, and Jose M Alvarez. Fb-occ: 3d occupancy prediction based on forward-backward view transformation. arXiv preprint, 2023. 4

    2023

  16. [24]

    Stcocc: Sparse spatial-temporal cascade renova- tion for 3d occupancy and scene flow prediction

    Zhimin Liao, Ping Wei, Shuaijia Chen, Haoxuan Wang, and Ziyang Ren. Stcocc: Sparse spatial-temporal cascade renova- tion for 3d occupancy and scene flow prediction. In CVPR,

  17. [25]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 1, 5

    2017

  18. [26]

    Sparsebev: High-performance sparse 3d object detec- tion from multi-camera videos

    Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High-performance sparse 3d object detec- tion from multi-camera videos. In ICCV, 2023. 1

    2023

  19. [27]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017. 6

    2017 arXiv

  20. [28]

    Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks

    Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In CoRL, 2022. 2

    2022

  21. [29]

    Uniworld: Autonomous driving pre-training via world models

    Chen Min, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai. Uniworld: Autonomous driving pre-training via world models. arXiv preprint, 2023. 2

    2023

  22. [30]

    Driveworld: 4d pre-trained scene understanding via world models for autonomous driving

    Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. In CVPR, 2024. 1, 2

    2024

  23. [31]

    Renderocc: Vision- centric 3d occupancy prediction with 2d rendering supervi- sion

    Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Li Liu, and Shanghang Zhang. Renderocc: Vision- centric 3d occupancy prediction with 2d rendering supervi- sion. arXiv preprint, 2023. 3

    2023

  24. [32]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint, 2022. 2 9

    2022

  25. [33]

    Gener- ating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gener- ating diverse high-fidelity images with vq-vae-2. NIPS, 32,

  26. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 4

    2022

  27. [35]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 2

    2022

  28. [36]

    Pointr- cnn: 3d object proposal generation and detection from point cloud

    Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointr- cnn: 3d object proposal generation and detection from point cloud. In CVPR, 2019. 1

    2019

  29. [37]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. InCVPR, 2020. 6, 7

    2020

  30. [38]

    Vidtok: A versatile and open-source video tokenizer

    Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, and Jiang Bian. Vidtok: A versatile and open-source video tokenizer. arXiv preprint, 2024. 2

    2024

  31. [39]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NIPS, 2025. 2, 4

    2025

  32. [40]

    Occ3d: A large-scale 3d occupancy prediction benchmark for au- tonomous driving

    Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for au- tonomous driving. In NeurIPS, 2023. 1, 5, 6, 7

    2023

  33. [41]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NIPS, 30, 2017. 2, 3, 4

    2017

  34. [42]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 2017. 4, 5

    2017

  35. [43]

    Omnitokenizer: A joint image-video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. NIPS, 2024. 2

    2024

  36. [44]

    Occsora: 4d occupancy generation models as world simulators for autonomous driv- ing

    Lening Wang, Wenzhao Zheng, Yilong Ren, Han Jiang, Zhiy- ong Cui, Haiyang Yu, and Jiwen Lu. Occsora: 4d occupancy generation models as world simulators for autonomous driv- ing. arXiv preprint arXiv:2405.20337, 2024. 1, 2, 3

    2024 arXiv

  37. [45]

    Drivedreamer: Towards real-world- drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- drive world models for autonomous driving. In ECCV, 2024. 2

    2024

  38. [46]

    Occllama: An occupancy- language-action generative world model for autonomous driv- ing

    Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Occllama: An occupancy- language-action generative world model for autonomous driv- ing. arXiv preprint, 2024. 2, 5, 6

    2024

  39. [47]

    Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for sequential pose forecasting

    Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, and Nicholas Rhinehart. Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for sequential pose forecasting. In CoRL, 2021. 2

    2021

  40. [48]

    ivideogpt: Interactive videogpts are scalable world models

    Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. In NIPS, 2024. 2

    2024

  41. [49]

    Occ-llm: Enhancing autonomous driving with occupancy-based large language models

    Tianshuo Xu, Hao Lu, Xu Yan, Yingjie Cai, Bingbing Liu, and Yingcong Chen. Occ-llm: Enhancing autonomous driving with occupancy-based large language models. In ICRA, 2025. 1, 2, 5, 6

    2025

  42. [50]

    Videogpt: Video generation using vq-vae and transform- ers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srini- vas. Videogpt: Video generation using vq-vae and transform- ers. arXiv preprint, 2021. 2

    2021

  43. [51]

    Renderworld: World model with self-supervised 3d label

    Ziyang Yan, Wenzhen Dong, Yihua Shao, Yuhang Lu, Liu Haiyang, Jingwen Liu, Haozhe Wang, Zhe Wang, Yan Wang, Fabio Remondino, et al. Renderworld: World model with self-supervised 3d label. arXiv preprint, 2024. 2

    2024

  44. [52]

    Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective su- pervision

    Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective su- pervision. In CVPR, 2023. 1, 4

    2023

  45. [53]

    Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving.AAAI,

    Yu Yang, Jianbiao Mei, Yukai Ma, Siliang Du, Wenqing Chen, Yijie Qian, Yuxiang Feng, and Yong Liu. Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving.AAAI,

  46. [54]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023. 2

    2023

  47. [55]

    An efficient occupancy world model via decoupled dynamic flow and image-assisted training

    Haiming Zhang, Ying Xue, Xu Yan, Jiacheng Zhang, We- ichao Qiu, Dongfeng Bai, Bingbing Liu, Shuguang Cui, and Zhen Li. An efficient occupancy world model via decoupled dynamic flow and image-assisted training. arXiv preprint arXiv:2412.13772, 2024. 2, 6

    2024 arXiv

  48. [56]

    Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion

    Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. ICLR, 2024. 2

    2024

  49. [57]

    Drivedreamer- 2: Llm-enhanced world models for diverse driving video gen- eration

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer- 2: Llm-enhanced world models for diverse driving video gen- eration. arXiv preprint arXiv:2403.06845, 2024. 2

    2024 arXiv

  50. [58]

    Cv-vae: A compatible video vae for latent generative video models

    Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. NIPS, 2024. 2, 3

    2024

  51. [59]

    Occworld: Learning a 3d occupancy world model for autonomous driving

    Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning a 3d occupancy world model for autonomous driving. In ECCV,

  52. [60]

    Hitvideo: Hierarchical tokenizers for enhancing text-to- video generation with autoregressive large language models

    Ziqin Zhou, Yifan Yang, Yuqing Yang, Tianyu He, Houwen Peng, Kai Qiu, Qi Dai, Lili Qiu, Chong Luo, and Lingqiao Liu. Hitvideo: Hierarchical tokenizers for enhancing text-to- video generation with autoregressive large language models. arXiv preprint, 2025. 2, 3

    2025

  53. [61]

    Scaling the codebook size of vq-gan to 100,000 with a utilization rate of 99%

    Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vq-gan to 100,000 with a utilization rate of 99%. NIPS, 2024. 2

    2024

  54. [62]

    Deformable detr: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021. 5 10

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.