Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fixed-size triplane tokenizer can replace per-camera image patch tokens in end-to-end driving Transformers, cutting token count by up to 72% and inference time by up to 50% without sacrificing planning accuracy.

desk verdict Triplane tokenization for multi-camera driving is a real, well-engineered contribution with credible efficiency gains; the closed-loop claim is statistically thin but the open-loop and runtime evidence carries the paper. read the letter →

arxiv 2506.12251 v2 pith:KUPLCRVS submitted 2025-06-13 cs.CV cs.LGcs.RO

Boris Ivanovic , Cristiano Saltori , Yurong You , Yan Wang , Wenjie Luo , Marco Pavone This is my paper · ORCID
classification cs.CVcs.LGcs.RO
keywords triplanetokenizationmulti-cameraperceptionautoregressivetransformerend-to-enddrivingtokenefficiencyvolumetricrenderingmotionplanningautonomousvehicles
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autoregressive Transformers are becoming end-to-end driving policies, but their token counts grow with the number of cameras and image resolution, which makes real-time deployment hard. This paper proposes encoding all camera images into a single fixed-size triplane, a set of three orthogonal feature planes in 3D space, so the token count no longer depends on how many cameras or pixels are present. Trained only with image-reconstruction losses, the triplane tokenizer feeds an autoregressive model that predicts the ego trajectory. On a 20,000-hour driving dataset and two public evaluation splits, it produces up to 72% fewer tokens, runs up to 50% faster, and matches or slightly improves planning accuracy and closed-loop offroad rates compared with DINOv2 and VQGAN patch tokenizers.

What carries the argument

The triplane: three axis-aligned feature planes $P_{xy}, P_{xz}, P_{yz} \in \mathbb{R}^{S_i \times S_j \times D_f}$, with $96 \times 96 \times 192$ features here, covering 180 meters around the ego vehicle and 45 meters above it, using Mip-NeRF 360-style bilinear grid contraction for the unbounded region. A grid of 3D query points attends to per-camera image features via deformable attention using camera intrinsics and extrinsics, then the queries are averaged along each axis to form the planes. The planes are trained by volumetric rendering: sampled rays query the planes, features are aggregated by elementwise product, an MLP decodes color and density, and LPIPS plus L1 losses compare rendered images to ground truth. For the downstream model, each plane is patchified as in ViT and an MLP maps patches to transformer tokens; optional removal of unused half-planes cuts tokens a further 30 to 40 percent when only front cameras are used.

What would settle it

Continue the paper's resolution sweep with the triplane size fixed: train the same tokenizer at 1152x2048 input and compare open-loop minADE6 and closed-loop offroad rate against the 320x512 model. The paper already shows reconstruction PSNR falling from 29.15 to 28.00 dB when resolution triples, so a further PSNR drop accompanied by a rise in minADE6 would show that the fixed-size triplane has an information limit that invalidates the resolution-agnosticism claim at some point.

Watch

Extended reading notes

Core claim

The central claim is that a fixed-size triplane representation, created by projecting 3D query points into each camera through known intrinsics and extrinsics and attending to image features, can serve as the token source for an autoregressive driving Transformer. Because the triplane's spatial dimensions are constant, token production does not scale with camera number or resolution; instead, all cameras vote into the same 3D feature volume, which is then patchified into tokens. The paper shows this at 1B-parameter scale: with 4 front cameras, its tokenizer emits 45 tokens per image (180 total) versus 160 per image for DINOv2, matches or slightly beats baseline minADE6 at 1, 3, and 5 seconds, and lowers offroad rate in closed-loop simulation. It also shows the same triplane keeps performance when camera count goes from 4 to 7, whereas the patch baseline degrades 18%, and when image resolution is tripled, planning performance is identical.

Load-bearing premise

The load-bearing premise is that a single fixed-size set of three feature planes, created by averaging all cameras' contributions into one 3D grid, still contains the scene information needed for safe motion planning, even when there are more cameras or much higher-resolution images than the triplane was designed for.

Editorial extensions

If this is right

  • A 1B-parameter autoregressive driving model can run with 180 total sensor tokens per timestep instead of 640 for four cameras, and the paper measures up to 50% faster total inference.
  • Because token count no longer grows with camera count, adding cameras (e.g., 7 instead of 4) changes planning accuracy by only 2.5% with the triplane, versus an 18% degradation for DINOv2-based tokenization.
  • The paper's profiling shows a 7B-parameter model with 7 cameras and 4 frames of context can run at 3 Hz with the triplane tokenizer, a configuration that baseline patch tokenization cannot fit at that rate for larger backbones.
  • Resolution can be tripled without retraining the triplane or changing downstream planning performance, although reconstruction PSNR drops from 29.15 to 28.00 dB.
  • Aggressive patchification with patch size (8-8-8) matches the less aggressive (4-6-6) on the Waymo open-loop evaluation, suggesting the triplane's features remain usable for planning even when heavily compressed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the information density of triplane tokens holds, the same fixed-size volumetric tokenizer should transfer to other multi-sensor robotic policies, such as robot arms with multiple cameras or drones with multi-view inputs, where transformer cost currently scales with sensor count.
  • The self-supervised rendering objective means the tokenizer could be pretrained on arbitrary unlabeled driving video before the policy is trained on labeled data; the paper only demonstrates this on its internal dataset, so a public-dataset replication would test how much of the gain is due to pretraining scale.
  • Because patchification is applied after triplane creation and can be changed without retraining, a deployed system could treat token count as a runtime knob, trading speed against planning accuracy per scene or per hardware target.
  • The reported PSNR drop at higher resolution suggests the fixed triplane has a finite information budget; a natural next step is finding where planning accuracy begins to degrade as resolution or camera count keeps increasing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a triplane-based multi-camera tokenization scheme for autoregressive (AR) transformer driving policies. Images are first encoded into per-image features, then a grid of 3D query points attends to those features using camera geometry; the updated queries are averaged along each spatial axis to form three feature planes (XY, XZ, YZ). The triplanes are trained self-supervised via volumetric rendering with LPIPS and L1 losses, and are then patchified into tokens for an AR transformer that predicts future ego trajectories. Experiments on a 20,000-hour internal dataset, nuScenes, Waymo, and a closed-loop neural simulator claim that the method produces up to 72% fewer tokens, enables up to 50% faster inference, achieves open-loop planning accuracy comparable to patch-based baselines, and improves offroad rates in closed-loop simulation.

Significance. If the claims hold, the contribution is significant: decoupling the sensor token count from the number of cameras and image resolution is an important practical step for deploying AR-based driving policies at real-time rates. The paper has clear strengths: the efficiency argument is supported by token-count arithmetic and runtime profiling on a single A100 (Sec. 4.3), open-loop results are reported on both internal data and two public benchmarks (nuScenes and Waymo), and the tokenizer is trained self-supervised without GAN losses. The fixed-size triplane representation is a credible alternative to per-image patchification. However, the two headline comparative claims—'same open-loop planning accuracy' and 'improved offroad rates'—are supported only by point estimates without error bars or statistical tests, and the closed-loop result rests on a single 75-scenario evaluation. These weaknesses limit the strength of the conclusions as currently stated.

major comments (3)
  1. [Sec. 4.5, Table 4] The closed-loop claim of 'improved offroad rates' is not statistically secured. The evaluation uses 75 hand-mined scenarios, one baseline (DINOv2-small), one tokenizer variant (Ours 8-8-8), and reports only aggregate offroad rates at 6s, 12s, and 20s. Offroad rate is a rare-event metric: the reported differences (2.7% vs 1.4% at 6s and 4.0% vs 2.7% at 20s) correspond to roughly one or two scenarios out of 75, and no confidence intervals, per-scenario breakdowns, or significance tests are provided. Because the paper itself cites NAVSIM [10] to note that open-loop metrics do not reliably predict closed-loop behavior, this small, statistically unquantified comparison is load-bearing for the abstract's 'improved offroad rates' claim. The authors should either report uncertainty (e.g., bootstrap confidence intervals or per-scenario rates) or soften the claim to 'comparable offroad rate on this scenario set'.
  2. [Sec. 4.2, Tables 2 and 3] The central claim of 'same open-loop motion planning accuracy' is supported only by single point estimates of minADE6 without error bars, multiple seeds, or statistical comparison. For example, in Table 3 the nuScenes average L2 is 0.73 for DINOv2-small and 0.68 for Ours (8-8-8), and in Table 2 the 5s minADE6 values are 0.69 and 0.72; these differences are plausibly within training noise, but the manuscript does not provide the variance needed to conclude equality or slight superiority. Given that this equality claim appears in the abstract and is used to argue that token reduction does not degrade planning, the authors should report variance (e.g., across seeds) or explicitly state that the comparison is based on a single run and should be interpreted as indicative.
  3. [Sec. 4.4] The resolution-agnosticism claim is validated by reporting that a model retrained at 3.6x resolution achieves 'identical performance' to the 4-6-6 model, but this is again a single point estimate with no error bars, and the reconstruction quality drops from 29.15 dB PSNR (4-camera, 320x512) to 28.00 dB PSNR (576x1024). The planning-equality result is therefore the main evidence that the fixed-size triplane still preserves task-relevant information at higher input resolution, and the manuscript should provide more than one run or otherwise quantify variability before asserting that performance is 'identical'.
minor comments (5)
  1. [Sec. 4.1] Typo: 'reonstruction' should be 'reconstruction'.
  2. [Sec. 3.2] The nonlinear scene parametrization equation is dense; the roles of R_i,x, R_o,x, and S_in,x are described in the text, but the equation would benefit from a short numerical example or a note that the mapping is piecewise linear and continuous at the boundaries.
  3. [Sec. 3.4] The decision to drop discrete FSQ tokens because they 'underperformed' is stated but not quantified or analyzed. Since the paper frames the contribution as 'tokenization' for AR transformers, a reader would benefit from at least a sentence or small table comparing continuous vs. discrete tokens on a representative metric.
  4. [Sec. 4] Training and evaluation details are partly delegated to reference [52], a GTC session URL rather than a peer-reviewed technical description. The AR backbone, the neural simulator, and the closed-loop setup are central to the paper's claims, so the authors should include a more self-contained description of these components in the main text or an appendix.
  5. [Sec. 4.2] The token-count comparison in Table 2 reports 'Tokens per image' for the baselines (160) and for the triplane variants (45, 104), but the triplane token counts are global per timestep and only divided by the number of cameras. The text explains this, but the table header could be clearer (e.g., 'Tokens per camera equivalent').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the triplane tokenizer's savings and driving results are empirical comparisons, not derivations that reduce to their own inputs.

full rationale

The paper's central chain is: (i) images are encoded and aggregated into fixed-size triplanes via geometric attention (Sec. 3.2); (ii) the triplane model is trained with a self-supervised rendering loss (Eq. 2); (iii) triplanes are patchified into tokens and used to train a 1B autoregressive policy by next-trajectory-token prediction; and (iv) the resulting tokenizer-policy systems are evaluated on nuScenes, Waymo Open Dataset, and a closed-loop simulator. None of these steps defines the reported outcome in terms of an input that is being 'predicted.' The 72% token reduction is a direct arithmetic consequence of the stated triplane sizes and patch sizes (e.g., 45 tokens per image versus 160), not a fitted quantity. The open-loop minADE comparisons and the closed-loop offroad rates are measured outcomes on evaluation data; no parameter of the tokenizer or policy is fitted to minADE or offroad rate. The self-citations that exist (NAVSIM [10], TOKEN [40], PARA-Drive [51], STORM [56], the in-house simulator [52]) are used for motivation, evaluation setup, benchmark comparison, or as an evaluation environment, not as a load-bearing proof that the triplane representation must work. The closed-loop result rests on a single baseline and an in-house simulator with no error bars, and the 'same accuracy' claim is statistically underpowered; however, under the hard rules of this pass, those are validity or robustness concerns, not circularity. There is no equation in the paper that reduces a prediction to an input, no parameter fitted to the target metric and renamed as a prediction, and no uniqueness theorem imported from the authors' prior work to force the design. The derivation is therefore self-contained even though the evidential strength of some claims is limited.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method is empirical; it rests mainly on domain assumptions about the sufficiency of fixed-size triplanes and on standard volumetric rendering machinery. No new physical entities are introduced. Manual design choices (grid sizes, scene bounds, loss weights, patch sizes, halfplane reductions) control the token count and therefore the headline efficiency numbers.

free parameters (5)
  • Triplane grid sizes S_x, S_y, S_z and feature dimension D_f = 96 x 96 x 48, D_f = 192
    Set manually in Sec. 4 for all experiments; these directly determine the number of tokens and the representational capacity.
  • Scene parametrization bounds = 180 m around ego, 45 m up, 3 m below; S_in_x = S_in_y = 36; z-axis: 36 cells for [-3,15] m and 12 cells for (15,45] m
    Hand-chosen in Sec. 4 to cover highways and urban streets while keeping grid sizes fixed; affects which parts of the world the triplane can represent.
  • Reconstruction loss weights lambda_LPIPS and lambda_1 = 0.5 and 0.5
    Fixed in Sec. 4 without a sensitivity study; the triplane's learned features depend on this balance.
  • Patch sizes (p_x, p_y, p_z) = (4,6,6) and (8,8,8)
    Ablated in Sec. 4.2; they set the final token count (104 vs 45 tokens per image) and the planning accuracy.
  • Halfplane token reduction = 37.5% fewer tokens for (4,6,6); 40.9% for (8,8,8)
    Applied to all front-camera experiments in Sec. 4.2; it improves efficiency only under the front-camera assumption.
assumptions (6)
  • domain assumption A fixed-size triplane can faithfully encode all information from any number of outward-facing cameras that is needed for motion planning, despite averaging query features along the three spatial axes.
    Sec. 3.2 constructs triplanes by averaging updated query points along each axis; Sec. 4.4 claims camera-count and resolution agnosticism based on this. If averaging discards task-relevant detail, the tokenizer fails.
  • domain assumption Two pixel-wise reconstruction losses (LPIPS and L1) produce tokenizer features that are useful for downstream motion planning.
    Sec. 3.3 trains with only Eq. (2), and the planning results in Sec. 4.2 assume this self-supervised objective transfers to path prediction.
  • standard math Camera intrinsics and extrinsics are known and accurate; 3D-to-2D projections built on them correctly align triplane queries with image content.
    Sec. 3.2 uses per-camera calibration for the deformable attention projections; calibration errors would degrade the triplane features.
  • standard math Volumetric rendering with bilinear triplane feature retrieval and a lightweight MLP decoder is a valid differentiable scene model for training the tokenizer.
    Sec. 3.3 follows NeRF-style ray sampling and aggregation, a standard technique from the cited prior art.
  • ad hoc to paper Continuous triplane tokens can be consumed by an autoregressive transformer trained with next-trajectory-token prediction; discrete FSQ tokens underperformed and were dropped.
    Sec. 3.4 patchifies triplanes into continuous tokens with a one-layer MLP; Sec. 4 reports that discrete tokens underperformed, making the continuous choice a paper-specific design decision.
  • ad hoc to paper For the 4-front-camera setup, the rear halves of the XY and XZ planes contain no useful information and can be removed without loss.
    Sec. 4.2 and Fig. 6 introduce halfplane token reduction, which depends on the specific camera configuration and is not a general property of triplanes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving." pith.science (2026). https://pith.science/paper/KUPLCRVS

@misc{pith2026250612251,
  author       = {Pith},
  title        = {Pith review of: Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KUPLCRVS}},
  note         = {Machine review of arXiv:2506.12251}
}
read the original abstract

Autoregressive Transformers are increasingly being deployed as end-to-end robot and autonomous vehicle (AV) policy architectures, owing to their scalability and potential to leverage internet-scale pretraining for generalization. Accordingly, tokenizing sensor data efficiently is paramount to ensuring the real-time feasibility of such architectures on embedded hardware. To this end, we present an efficient triplane-based multi-camera tokenization strategy that leverages recent advances in 3D neural reconstruction and rendering to produce sensor tokens that are agnostic to the number of input cameras and their resolution, while explicitly accounting for their geometry around an AV. Experiments on a large-scale AV dataset and state-of-the-art neural simulator demonstrate that our approach yields significant savings over current image patch-based tokenization strategies, producing up to 72% fewer tokens, resulting in up to 50% faster policy inference while achieving the same open-loop motion planning accuracy and improved offroad rates in closed-loop driving simulations.

Figures

Figures reproduced from arXiv: 2506.12251 by the authors.

Figure 1
Figure 1. Many Transformer-based AV policies employ patch [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our triplane-based multi-camera tokenization strategy. Images from multiple cameras are first encoded through ImageEnc, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We employ a bilinear triplane resolution, representing [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Triplanes can accurately represent environments observed by outward-facing cameras ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Our triplane-based multi-camera tokenization approach scales much more favorably than baseline approaches as the number of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: If modeling cameras that face a certain direction (e.g., all [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Triplanes can faithfully represent higher resolution data without requiring any changes to triplane sizes. Since these cameras are [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Our triplane model performs similarly with 7 cameras [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: In a challenging closed-loop simulation of a construction scene, the baseline DINOv2-based model ( [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Triplanes can optionally be trained with auxiliary la [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 52 canonical work pages

  1. [10]

    NA VSIM: Data-driven non- reactive autonomous vehicle simulation and benchmarking

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. NA VSIM: Data-driven non- reactive autonomous vehicle simulation and benchmarking. InConf. on Neural Information Processing Systems, 2024. 7

    work page 2024

  2. [1]

    Barron, Ben Mildenhall, Dor Verbin, Pratul P

    Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields.IEEE Conf. on Computer Vision and Pattern Recognition, 2022. 3

    work page 2022

  3. [2]

    RT-1: Robotics transformer for real- world control at scale.arXiv preprint arXiv:2212.06817,

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Ju- lian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav M...

    arXiv

  4. [3]

    RT-2: Vision-language-action mod- els transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakr- ishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnik...

    arXiv 2023

  5. [4]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuScenes: A multi- modal dataset for autonomous driving. InIEEE Conf. on Computer Vision and Pattern Recognition, 2020. 5

    work page 2020

  6. [5]

    HexPlane: A fast representa- tion for dynamic scenes

    Ang Cao and Justin Johnson. HexPlane: A fast representa- tion for dynamic scenes. InIEEE Conf. on Computer Vision and Pattern Recognition, 2023. 2

    work page 2023

  7. [6]

    Chan, Connor Z

    Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. InIEEE Conf. on Computer Vision and Pattern Recognition, 2022. 2

    work page 2022

  8. [7]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image transformer. InIEEE Conf. on Computer Vision and Pattern Recognition, 2022. 3

    work page 2022

Show all 63 references
  1. [8]

    MobileVLM: A fast, reproducible and strong vision language assistant for mobile devices.arXiv preprint arXiv:2312.16886, 2023

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. MobileVLM: A fast, reproducible and strong vision language assistant for mobile devices.arXiv preprint arXiv:2312.16886, 2023. 2

    2023 arXiv

  2. [9]

    Open X-Embodiment: Robotic learning datasets and RT-X models.arXiv preprint arXiv:2310.08864, 2023

    Embodiment Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X models.arXiv preprint arXiv:2310.08864, 2023. 2

    2023 arXiv

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2020

  4. [12]

    Shivam Duggal, Phillip Isola, Antonio Torralba, and William T. Freeman. How many tokens is an image worth? InInt. Conf. on Learning Representations, 2025. 2

    2025

  5. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InIEEE Conf. on Computer Vision and Pattern Recognition, 2021. 1, 2, 3, 4

    2021

  6. [14]

    Spectral image tokenizer.arXiv preprint arXiv:2412.09607,

    Carlos Esteves, Mohammed Suhail, and Ameesh Makadia. Spectral image tokenizer.arXiv preprint arXiv:2412.09607,

    arXiv

  7. [15]

    K- Planes: Explicit radiance fields in space, time, and appear- ance

    Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K- Planes: Explicit radiance fields in space, time, and appear- ance. InIEEE Conf. on Computer Vision and Pattern Recog- nition, 2023. 2

    2023

  8. [16]

    A survey for foundation mod- els in autonomous driving.arXiv preprint arXiv:2402.01105,

    Haoxiang Gao, Zhongruo Wang, Yaqian Li, Kaiwen Long, Ming Yang, and Yiqing Shen. A survey for foundation mod- els in autonomous driving.arXiv preprint arXiv:2402.01105,

    arXiv

  9. [17]

    Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Google. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  10. [18]

    Frankenstein: Gener- ating semantic-compositional 3d scenes in one tri-plane

    Yan Han, Li Yang, Wu Zhennan, Chen Shenzhou, Sun Weix- uan, Shang Taizhang, Liu Weizhe, Chen Tian, Dai Xiaqiang, Ma Chao, Li Hongdong, and Ji Pan. Frankenstein: Gener- ating semantic-compositional 3d scenes in one tri-plane. In Proc. of SIGGRAPH Asia, 2024. 2

    2024

  11. [19]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. InIEEE Conf. on Computer Vision and Pattern Recognition, 2016. 4, 5

    2016

  12. [20]

    Patel, and Fatih Porikli

    Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M. Patel, and Fatih Porikli. Dis- tilling multi-modal large language models for autonomous driving. InIEEE Conf. on Computer Vision and Pattern R...

    2025

  13. [21]

    LRM: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3d. InInt. Conf. on Learning Representations, 2024. 2

    2024

  14. [22]

    GAIA-1: A generative world model for au- tonomous driving.arXiv preprint arXiv:2309.17080, 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. GAIA-1: A generative world model for au- tonomous driving.arXiv preprint arXiv:2309.17080, 2023. 2

    2023 arXiv

  15. [23]

    Planning-oriented autonomous driv- ing

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driv- ing. InIEEE Conf. on Computer Vision and Pattern Reco...

    2023

  16. [24]

    Tri-perspective view for vision-based 3d se- mantic occupancy prediction

    Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d se- mantic occupancy prediction. InIEEE Conf. on Computer Vision and Pattern Recognition, 2023. 2, 3

    2023

  17. [25]

    SelfOcc: Self-supervised vision-based 3d oc- cupancy prediction

    Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu. SelfOcc: Self-supervised vision-based 3d oc- cupancy prediction. InIEEE Conf. on Computer Vision and Pattern Recognition, 2024. 2, 3

    2024

  18. [26]

    EMMA: End-to-end multimodal model for autonomous driving.arXiv preprint arXiv:2410.23262,

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. EMMA: End-to-end multimodal model for autonomous driving.arXiv preprint arXiv:2410.23262,

    arXiv

  19. [27]

    V AD: Vectorized scene representation for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. V AD: Vectorized scene representation for efficient autonomous driving. InIEEE Int. Conf. on Com- puter Vision, 2023. 2, 5

    2023

  20. [28]

    VIMA: General robot manip- ulation with multimodal prompts

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandku- mar, Yuke Zhu, and Linxi Fan. VIMA: General robot manip- ulation with multimodal prompts. InInt. Conf. on Machine Learning, 2023. 2

    2023

  21. [29]

    3d gaussian splatting for real-time radiance field rendering.Proc

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.Proc. of SIGGRAPH, 42(4), 2023. 2

    2023

  22. [30]

    Finite scalar quantization: VQ-V AE made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-V AE made simple. InInt. Conf. on Learning Representations,

  23. [31]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view syn- thesis. InEuropean Conf. on Computer Vision, 2020. 2, 3

    2020

  24. [32]

    Qi, Runzhou Ge, Kratarth Goel, Zoey Yang, Scott Ettinger, Rami Al-Rfou, Dragomir Anguelov, and Yin Zhou

    Norman Mu, Jingwei Ji, Zhenpei Yang, Nate Harada, Hao- tian Tang, Kan Chen, Charles R. Qi, Runzhou Ge, Kratarth Goel, Zoey Yang, Scott Ettinger, Rami Al-Rfou, Dragomir Anguelov, and Yin Zhou. MoST: Multi-modality scene tok- enization for motion prediction. InIEEE Conf. on Comp...

    2024

  25. [33]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael ...

    2024

  26. [34]

    DeepSDF: Learning continuous signed distance functions for shape representa- tion

    Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representa- tion. InIEEE Conf. on Computer Vision and Pattern Recog- nition, 2019. 2

    2019

  27. [35]

    A gener- alist agent.Transactions on Machine Learning Research,

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio G ´omez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Gim ´enez, Yury Sulsky, Jackie Kay, Jost Tobias Sprin- genberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Ed- wards, Nicolas Heess, Yutian Chen, Raia Hadsell, O...

  28. [36]

    Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein

    J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3D neural field generation using triplane diffusion. InIEEE Conf. on Computer Vision and Pattern Recognition, 2023. 2

    2023

  29. [37]

    K. Sohn, H. Lee, and X Yan. Learning structured output representation using deep conditional generative models. In Conf. on Neural Information Processing Systems, 2015. 1, 2

    2015

  30. [38]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Et- tinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhan...

    2020

  31. [39]

    Neural geometric level of detail: Real-time rendering with implicit 3D shapes

    Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacob- son, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. InIEEE Conf. on Computer Vision and Pattern Recognition,

  32. [40]

    Tokenize the world into object-level knowledge to address long-tail events in autonomous driving

    Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. InConf. on Robot Learning, 2024. 5

    2024

  33. [41]

    DriveVLM: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. DriveVLM: The convergence of autonomous driving and large vision-language models. InConf. on Robot Learning, 2024. 2, 6, 7

    2024

  34. [42]

    Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano

    Alex Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. Real- time radiance fields for single-image portrait view synthesis. InProc. of SIGGRAPH, 2023. 2

    2023

  35. [43]

    Rendering every pixel for high-fidelity geometry in 3D gans

    Alex Trevithick, Matthew Chan, Towaki Takikawa, Umar Iqbal, Shalini De Mello, Manmohan Chandraker, Ravi Ra- mamoorthi, and Koki Nagano. Rendering every pixel for high-fidelity geometry in 3D gans. InIEEE Conf. on Com- puter Vision and Pattern Recognition, 2024. 2

    2024

  36. [44]

    Neural discrete representation learning

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. InConf. on Neural Information Processing Systems, 2017. 1, 2

    2017

  37. [45]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InConf. on Neural Information Processing Systems, 2017. 1, 2, 3 10

    2017

  38. [46]

    Scaling laws in patchi- fication: An image is worth 50,176 tokens and more.arXiv preprint arXiv:2502.03738, 2025

    Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Scaling laws in patchi- fication: An image is worth 50,176 tokens and more.arXiv preprint arXiv:2502.03738, 2025. 1

    2025

  39. [47]

    DistillNeRF: Perceiv- ing 3d scenes from single-glance images by distilling neu- ral fields and foundation model features

    Letian Wang, Seung Wook Kim, Jiawei Yang, Cunjun Yu, Boris Ivanovic, Steven Waslander, Yue Wang, Sanja Fidler, Marco Pavone, and Peter Karkus. DistillNeRF: Perceiv- ing 3d scenes from single-glance images by distilling neu- ral fields and foundation model features. InConf. on ...

    2024

  40. [48]

    OmniDrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.arXiv preprint arXiv:2405.01533, 2024

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. OmniDrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.arXiv preprint arXiv:2405.01533, 2024. 2

    2024 arXiv

  41. [49]

    LINGO-1: Exploring natural language for autonomous driving, 2023

    Wayve. LINGO-1: Exploring natural language for autonomous driving, 2023. Available athttps : / / wayve . ai / thinking / lingo - natural - language-autonomous-driving/. 2

    2023

  42. [50]

    LINGO-2: Driving with natural language,

    Wayve. LINGO-2: Driving with natural language,

  43. [51]

    PARA-Drive: Parallelized architecture for real-time autonomous driving

    Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. PARA-Drive: Parallelized architecture for real-time autonomous driving. InIEEE Conf. on Computer Vision and Pattern Recognition, 2024. 2, 5

    2024

  44. [52]

    Accelerate the future of AI-defined vehicles and autonomous driving, 2025

    Xinzhou Wu. Accelerate the future of AI-defined vehicles and autonomous driving, 2025. Available athttps:// www.nvidia.com/en- us/on- demand/session/ gtc25-dd40000/. 4, 8

    2025

  45. [53]

    A survey on occupancy perception for au- tonomous driving: The information fusion perspective.In- formation Fusion, 114:102671, 2025

    Huaiyuan Xu, Junliang Chen, Shiyu Meng, Yi Wang, and Lap-Pui Chau. A survey on occupancy perception for au- tonomous driving: The information fusion perspective.In- formation Fusion, 114:102671, 2025. 2

    2025

  46. [54]

    DriveGPT4: Interpretable end-to-end autonomous driving via large language model.IEEE Robotics and Automation Letters, 2024

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. DriveGPT4: Interpretable end-to-end autonomous driving via large language model.IEEE Robotics and Automation Letters, 2024. 2

    2024

  47. [55]

    ElasticTok: Adaptive tokenization for image and video

    Wilson Yan, V olodymyr Mnih, Aleksandra Faust, Matei Za- haria, Pieter Abbeel, and Hao Liu. ElasticTok: Adaptive tokenization for image and video. InInt. Conf. on Learning Representations, 2025. 2

    2025

  48. [56]

    STORM: Spatio-temporal reconstruction model for large-scale outdoor scenes

    Jiawei Yang, Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Maximilian Igl, Apoorva Sharma, Peter Karkus, Danfei Xu, Boris Ivanovic, Yue Wang, and Marco Pavone. STORM: Spatio-temporal reconstruction model for large-scale outdoor scenes. InInt. Conf. on Learning Repr...

    2025

  49. [57]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. InConf. on Neural Information Processing Sys- tems, 2024. 1

    2024

  50. [58]

    Vector-quantized image modeling with im- proved VQGAN

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with im- proved VQGAN. InInt. Conf. on Learning Representations,

  51. [59]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. InConf. on Neu- ral Information Processing Systems, 2024. 2

    2024

  52. [60]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InIEEE Int. Conf. on Computer Vision, 2023. 2

    2023

  53. [61]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InIEEE Conf. on Computer Vision and Pattern Recognition, 2018. 3

    2018

  54. [62]

    A survey on model compression for large language models

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguis- tics, 12:1556–1577, 2024. 1 11 Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving ...

    2024

  55. [2024]

    Available athttps://wayve.ai/thinking/ lingo-2-driving-with-language/. 2

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.