Pith. sign in

REVIEW 4 major objections 6 minor 96 references

Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A light attention-prediction network lets 3D scene models drop 90% of visual tokens and keep about 97% of performance.

desk verdict Solid efficiency work for 3D MLLMs, but the headline 96.8% retention is measured on the same benchmarks used to train the predictor; the paper needs a transfer experiment before the claim is credible. read the letter →

arxiv 2507.09334 v1 pith:SBX6KTVS submitted 2025-07-12 cs.CV

Wencan Huang , Daizong Liu , Wei Hu This is my paper
classification cs.CV
keywords 3Dmulti-modallargelanguagemodelsvisualtokenpruningglobalattentionpredictionsample-adaptivesceneunderstandinginferenceaccelerationobject-centrictokens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that object-level visual tokens in 3D multi-modal large language models are redundant enough that most can be discarded, and that the target model's own global attention pattern is the right signal for choosing what to keep. It proposes Fast3D, a plug-and-play framework with two parts: a lightweight GAP network trained to predict the model's all-layer attention map, and a sample-adaptive pruning strategy that sets per-layer token budgets from input difficulty. The experiments on the Chat-Scene model across five benchmarks report that pruning 90% of visual tokens keeps 96.87% of original performance, that the GAP network's prediction nearly matches pruning with the model's own oracle attention, and that the added computation is about 0.1% of visual-token FLOPs. A sympathetic reader would care because 3D MLLMs are computationally expensive, and this result suggests a cheap way to accelerate them without retraining.

What carries the argument

The load-bearing object is the GAP network: a lightweight transformer encoder-decoder that fuses, per object, an identifier embedding, a fused 3D/2D semantic embedding, and a spatial-location embedding, and outputs a softmax distribution approximating the target model's object-centric attention map. It is trained with a multi-objective loss combining KL divergence against the aggregated oracle attention map with a pairwise rank-consistency term, so the predicted ordering of tokens matters as much as the exact scores. The second mechanism, SAP, turns predicted scores into per-layer cumulative thresholds $\theta_k$; for each input, the number of tokens kept at layer $k$ is the largest prefix whose cumulative predicted attention stays under $\theta_k$, and a single scaling factor $\alpha$ found by offline binary search matches a FLOPs budget while the thresholds stay sample-adaptive.

What would settle it

Train the GAP network on only two of the five benchmarks and evaluate at 90% visual-token pruning on the remaining three, or apply the trained GAP to a different object-centric 3D MLLM; if the score ratio drops well below the reported 96.87% and far below the oracle-attention result, the claim that predicted global attention generalizes fails.

Watch

Extended reading notes

Core claim

The central discovery is that redundancy survives object-centric structure: even though each 3D visual token is a grounded object rather than an image patch, the token sequence still contains many objects a given question does not need. The paper demonstrates this by aggregating attention over all layers and heads of the frozen Chat-Scene model, combining visual-token self-attention, prompt-to-visual cross-attention, and generated-text-to-visual cross-attention into one object-wise importance score. It then claims that a 159M-parameter GAP network can predict this oracle map well enough that pruning from its predictions is almost as good as pruning from the oracle (96.63% versus 98.28% score ratio at 90% pruning). At 90% average pruning, the full Fast3D system retains 96.87% of Chat-Scene's aggregate benchmark performance, whereas FastV, ToMe, and random pruning fall to roughly 46-48%.

Load-bearing premise

The load-bearing premise is that the GAP network's attention predictions, learned on the same five benchmark datasets used for evaluation, transfer to the scenes and questions the model will actually see; if attention patterns on new distributions differ, the pruning guidance and the 96.87% retention figure do not follow.

Editorial extensions

If this is right

  • At 35%, 65%, and 90% average visual-token pruning, Fast3D retains 99.79%, 99.10%, and 96.87% of Chat-Scene's original benchmark score respectively.
  • Pruning guided by GAP's predicted attention nearly matches the target model's own all-layer oracle attention, so the expensive full inference needed to obtain oracle maps is unnecessary during deployment.
  • SAP extends the FLOPs-reduction level at which performance stays competitive with the original model, from about 40% to over 65% on Scan2Cap and from about 50% to about 80% on SQA3D.
  • Since neither component changes Chat-Scene's parameters, the framework is a plug-and-play accelerator whose GAP overhead is only about 0.1% of the visual-token FLOPs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If attention-map concentration is a genuine difficulty signal, the GAP prediction could be reused as a computational-routing cue beyond pruning, for example to decide when a hard sample should skip pruning entirely.
  • The paper's evidence is limited to one base model and the five benchmarks used for both training and evaluation; a direct test of transfer would train GAP on two benchmarks and prune on the other three, or on a held-out scene dataset.
  • Because 3D MLLMs differ in how they construct visual tokens, some adding auxiliary relation tokens, the framework may need retraining or token-grouping changes when moved to other architectures; this is a testable extension rather than a claim of the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Fast3D, a visual-token pruning framework for object-centric 3D multi-modal large language models. It consists of two components: (i) GAP, a lightweight encoder-decoder network trained to predict the global attention map of a frozen target 3D MLLM (Chat-Scene), and (ii) SAP, a sample-adaptive pruning policy that sets per-layer token budgets via cumulative attention thresholds and an offline binary-search strategy search. Experiments on ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D report that pruning 90% of visual tokens retains 96.87% of the original average score, outperforming random pruning, ToMe, and FastV at high pruning ratios, and that the predicted attention maps approach the global-oracle pruning performance.

Significance. If the results transfer beyond the evaluation setting, Fast3D would be a useful contribution: it is one of the first works to study token pruning for object-centric 3D MLLMs, it provides a tractable way to approximate all-layer attention without full inference, and the code is promised in the paper. The ablations in Tables 3-5 are informative and the multi-source aggregation design is well motivated. However, the evaluation is currently too narrow to support the central plug-and-play claim: a single base model, training and evaluation on the same five benchmarks, no variance estimates, and no held-out distribution or cross-model test. The paper's own target construction (Eq. (5)) includes answer-dependent information, which raises the risk that the strong in-distribution numbers come from dataset-specific associations. These issues are addressable with additional experiments and should be fixed before publication.

major comments (4)
  1. [§4.1 and Table 1] Section 4.1 states that the GAP network undergoes 'unified training on combined datasets generated from all five benchmarks,' and Table 1 evaluates on the same five benchmarks. No held-out scene set, unseen dataset, or second base 3D MLLM is tested. The central claim that Fast3D is a plug-and-play, generalizable pruning framework is therefore unestablished; the 96.87% score ratio at 90% pruning may reflect the predictor memorizing dataset-specific associations between object identifiers, prompt patterns, and attention targets. Please add an unseen-dataset experiment (e.g., a dataset not used in GAP training, or a disjoint scene split) and a different base 3D MLLM (e.g., Chat-3D v2, Robin3D, or 3DGraphLLM) to demonstrate transfer.
  2. [§3.2.1, Eq. (5), and Table 3] Section 3.2.1, Eq. (5) constructs the training target using cross-attention from teacher-forced generated text tokens, which encodes answer-dependent information that is not available at inference time. Table 3 shows that removing a_text lowers the score ratio from 96.63% to 95.12% for the prompt+text variant and that a_text alone gives 90.48%, so this component is important for the headline result. On the training benchmarks this signal can be approximated by learning statistical regularities between prompts, object IDs, and the objects mentioned in answers; the paper provides no evidence that this approximation transfers. Please report the 90% pruning result for a GAP variant trained without a_text, and evaluate it on a held-out distribution, to separate transferable attention prediction from dataset-specific memorization.
  3. [§4.2 and Figure 3] The paper gives only theoretical FLOPs reduction for visual tokens (§4.2, Figure 3) and claims the GAP network's overhead is 'merely 0.1% of the target model's FLOPs related to visual tokens,' but this accounting excludes the GAP's own BERT encoder and decoder and does not report wall-clock time. Since Fast3D is proposed for inference acceleration, please report end-to-end latency on the same hardware for the full pipeline (GAP + pruned Chat-Scene) and total model FLOPs including the 159M-parameter GAP network.
  4. [Table 1 and Table 2] Table 1 reports single-run scores with no error bars or significance tests; the 90%-pruning retention is uneven across tasks (e.g., Scan2Cap CIDEr falls from 76.35 to 69.25, about 90.7%), so the 96.87% average may overstate uniform retention. Additionally, Table 2 reports 'score ratio' for only three benchmarks while Table 1 uses five, and the score-ratio definition is never given in Section 4.1. Please define the metric, report per-task ratios, provide variance over multiple seeds or runs, and include significance tests for the headline comparison.
minor comments (6)
  1. [§4.1] The term 'Score Ratio' is used throughout Tables 1-5 but is never formally defined in Section 4.1; please add a precise definition.
  2. [§3.3, Eq. (7)] In Eq. (7), if the cumulative attention of the highest-scoring token already exceeds the threshold θ_k, the set over which the maximum is taken is empty; please specify the convention in that case (e.g., retain 1 token or 0 tokens).
  3. [Algorithm 1] Algorithm 1 uses binary search on α under the assumption that f(α) is monotonically increasing; please state this assumption explicitly and clarify how the manual initialization of P0 is chosen.
  4. [Figure 3] The y-axis of Figure 3 is labeled only as 'performance'; please specify which metric is used and how it is aggregated across the five benchmarks.
  5. [§3.2.1, Eq. (3)] The causal masking description in Eq. (3) is ambiguous; please clarify that the column-wise average is computed only over the lower-triangular entries and state the indexing conventions.
  6. [References] Reference [66] for Vicuna is incomplete; please include the version and date of the model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GAP network is trained to imitate the target model's attention and evaluated on held-out splits of the same public benchmarks; the reported gains are empirical downstream results, not forced by construction.

full rationale

The claimed derivation chain is not circular. The training target for GAP is defined in Eqs. (3)-(5) as an aggregation of the frozen target model's actual attention (self-attention among visual tokens, cross-attention with prompt, and cross-attention with teacher-forced generated text); Eq. (6) trains the lightweight network to imitate that target, and Eq. (7) uses the predicted scores only to choose which tokens to retain. The downstream score ratios in Tables 1-2 are measured empirically after pruning and are not equal by construction to the GAP loss or to any fitted parameter. The paper trains GAP on "combined datasets generated from all five benchmarks" and evaluates on the same five benchmark families, which is standard in-distribution evaluation on held-out splits; the paper reports official metrics such as Acc@0.25, F1@0.25, and CIDEr@0.5, and the SAP policy is selected on a validation batch via Algorithm 1. No equation makes the evaluation metric a function of the training target in a way that forces the reported 96.87% score ratio. Self-citations in the references are confined to related work and are not load-bearing for the proposed method. The absence of a cross-dataset or cross-base-model experiment is a real generalization risk—the GAP network may learn benchmark-specific associations between prompts, object identifiers, and attention patterns—but that is an empirical limitation, not a circularity, and cannot be scored as such under the hard rules without demonstrating a specific reduction. Verdict: no significant circularity; score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

Fast3D does not postulate new physical entities. It does introduce a trained predictor and adaptive thresholds, which are fitted to the target model's attention and validation data.

free parameters (3)
  • lambda (rank loss weight) = 0.02
    Chosen to balance KL and rank loss in Eq. (6), presumably by validation.
  • Pruning layers: (16, 6, 2) for 35%, 65%, 90% and initial ratios (70%, 80%, 95%) = 16, 6, 2; 70%, 80%, 95%
    These layer choices and ratios are empirically set to match average pruning ratios, affecting all comparisons.
  • alpha scaling in Pruning Strategy Searching (SAP) = found by binary search
    The scaling factor alpha is fit to a validation batch to meet the FLOPs budget, so the strategy is fitted to the data distribution.
assumptions (3)
  • domain assumption Attention concentration correlates with sample difficulty
    Section 3.3 hypothesizes that concentrated predicted attention means easy samples and uniform attention means hard samples; this drives SAP retention counts and is not proven.
  • domain assumption Global attention from all layers is the correct pruning oracle
    The paper assumes that maximizing the average attention over all layers and heads identifies tokens needed for correct answers; Table 2 supports this empirically on their benchmarks.
  • domain assumption Chat-Scene is representative of 3D MLLMs
    The paper uses Chat-Scene as the only target model; transfer to other 3D MLLMs (e.g., Robin3D, 3DGraphLLM) is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding." pith.science (2026). https://pith.science/paper/SBX6KTVS

@misc{pith2026250709334,
  author       = {Pith},
  title        = {Pith review of: Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBX6KTVS}},
  note         = {Machine review of arXiv:2507.09334}
}
read the original abstract

While 3D Multi-modal Large Language Models (MLLMs) demonstrate remarkable scene understanding capabilities, their practical deployment faces critical challenges due to computational inefficiency. The key bottleneck stems from processing excessive object-centric visual tokens required for comprehensive 3D scene representation. Although visual token pruning has shown promise in accelerating 2D MLLMs, its applicability to 3D domains remains largely unexplored due to fundamental disparities in token structures. In this paper, we reveal two critical insights: (1) Significant redundancy exists in object-level 3D token representations, analogous to patch-level redundancy in 2D systems; (2) Global attention patterns exhibit strong predictive power for identifying non-essential tokens in 3D contexts. Building on these observations, we propose Fast3D, a plug-and-play visual token pruning framework for 3D MLLMs featuring two technical innovations: (1) Global Attention Prediction (GAP), where a lightweight neural network learns to predict the global attention distributions of the target model, enabling efficient token importance estimation for precise pruning guidance; (2) Sample-Adaptive visual token Pruning (SAP), which introduces dynamic token budgets through attention-based complexity assessment, automatically adjusting layer-wise pruning ratios based on input characteristics. Both of these two techniques operate without modifying the parameters of the target model. Extensive evaluations across five benchmarks validate the effectiveness of Fast3D, particularly under high visual token pruning ratios. Code is available at https://github.com/wencan25/Fast3D

Figures

Figures reproduced from arXiv: 2507.09334 by the authors.

Figure 1
Figure 1. Visualization of the global attention maps aggre [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of Fast3D. (a) The GAP network learns to predict the aggregated attention maps of the target 3D MLLM, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance-efficiency trade-off curves of Fast3D (GAP + SAP). The x-axis stands for the theoretical FLOPs reduction [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of GAP under different visual token pruning ratios. Object-centric visual tokens are pruned by 70%, 80%, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

96 extracted references · 21 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Dosovitskiy Alexey. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929 (2020)

    arXiv 2020

  3. [3]

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2022. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 19129–19139

    2022

  4. [4]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichten- hofer, and Judy Hoffman. 2022. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461 (2022)

    arXiv 2022

  5. [5]

    Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yansong Tang, Jiwen Lu, and Tao Chen. 2024. MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15710–15719

    2024

  6. [6]

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2024. Hon- eybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13817–13827

    2024

  7. [7]

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. 2020. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision. Springer, 202–221

    2020

  8. [8]

    Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. 2024. Llavolta: Efficient multi-modal models via stage-wise visual context compression. arXiv preprint arXiv:2406.20092 (2024)

    arXiv 2024

Show all 96 references
  1. [9]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2025. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision. Springer, 19–35

    2025

  2. [10]

    Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. 2023. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 17164–17174

    2023

  3. [11]

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. 2024. LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning. InProceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  4. [12]

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems 35 (2022), 20522– 20535

    2022

  5. [13]

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. 2024. Grounded 3D-LLM with Referent Tokens. arXiv preprint arXiv:2405.10370 (2024)

    2024 arXiv

  6. [14]

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. 2021. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 3193–3203

    2021

  7. [15]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 20...

    2023

  8. [16]

    Hengshuo Chu, Xiang Deng, Qi Lv, Xiaoyang Chen, Yinchuan Li, Jianye HAO, and Liqiang Nie. 2025. 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds. In The Thirteenth International Conference on Learning Representations . ht...

    2025

  9. [17]

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. 2023. Mobilevlm: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 (2023)

    2023 arXiv

  10. [18]

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. 2024. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 (2024)

    2024 arXiv

  11. [19]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...

    2019

  12. [20]

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Jürgen Gall. 2022. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision . Springe...

    2022

  13. [21]

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. 2024. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401 (2024)

    2024 arXiv

  14. [22]

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. 2023. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv prepri...

    2023 arXiv

  15. [23]

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. 2023. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905 (2023)

    2023 arXiv

  16. [24]

    Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang

  17. [25]

    Yuhang Han, Xuyang Liu, Pengxiang Ding, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. 2024. Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for Training-Free Acceleration. arXiv preprint arXiv:2411.17686 (2024)

    2024

  18. [26]

    Yizeng Han, Zeyu Liu, Zhihang Yuan, Yifan Pu, Chaofei Wang, Shiji Song, and Gao Huang. 2024. Latency-aware Unified Dynamic Networks for Efficient Image Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  19. [27]

    Shuting He, Henghui Ding, Xudong Jiang, and Bihan Wen. 2024. Segpoint: Segment any point cloud via large language model. In European Conference on Computer Vision. Springer, 349–367

    2024

  20. [28]

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems 36 (2023), 20482– 20494

    2023

  21. [29]

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al . 2024. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Informa...

    2024

  22. [30]

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. 2023. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168 (2023)

    2023 arXiv

  23. [31]

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2023. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871 (2023)

    2023 arXiv

  24. [32]

    Kai Huang, Hao Zou, Ye Xi, BoChen Wang, Zhen Xie, and Liang Yu. 2024. IVTP: Instruction-Guided Visual Token Pruning for Large Vision-Language Models. In European Conference on Computer Vision . Springer, 214–230

    2024

  25. [33]

    Wencan Huang, Daizong Liu, and Wei Hu. 2023. Dense object grounding in 3d scenes. In Proceedings of the 31st ACM International Conference on Multimedia . 5017–5026

    2023

  26. [34]

    Wencan Huang, Daizong Liu, and Wei Hu. 2024. Advancing 3d object grounding beyond a single 3d scene. InProceedings of the 32nd ACM International Conference on Multimedia. 7995–8004

    2024

  27. [35]

    Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, and Yiyi Zhou. 2025. What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi- modal Large Language Models from the Perspective of Graph. arXiv preprint arXiv:2501.02268 (2025)

    2025 arXiv

  28. [36]

    Chen Ju, Haicheng Wang, Haozhe Cheng, Xu Chen, Zhonghua Zhai, Weilin Huang, Jinsong Lan, Shuai Xiao, and Bo Zheng. 2024. Turbo: Informativity-driven acceleration plug-in for vision-language large models. In European Conference on Computer Vision. Springer, 436–455

    2024

  29. [37]

    Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. 2024. Robin3d: Improving 3d large language model via robust instruction tuning.arXiv preprint arXiv:2410.00255 (2024)

    2024 arXiv

  30. [38]

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia

  31. [39]

    Hongliang Li, Jiaxin Zhang, Wenhui Liao, Dezhi Peng, Kai Ding, and Lianwen Jin. 2025. Beyond Token Compression: A Training-Free Reduction Framework for Efficient Visual Processing in MLLMs. arXiv preprint arXiv:2501.19036 (2025)

    2025 arXiv

  32. [40]

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. 2024. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392 (2024)

    2024 arXiv

  33. [41]

    Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision . Springer, 323–340

    2024

  34. [42]

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. 2022. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800 (2022)

    2022 arXiv

  35. [43]

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. 2024. Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. arXiv preprint arXiv:2405.05803 (2024)

    2024 arXiv

  36. [44]

    Daizong Liu and Wei Hu. 2025. Seeing is Not Believing: Adversarial Natural Object Optimization for Hard-Label 3D Scene Attacks. In Proceedings of the Computer Vision and Pattern Recognition Conference . 11886–11897

    2025

  37. [45]

    Daizong Liu, Yang Liu, Wencan Huang, and Wei Hu. 2024. A survey on text- guided 3D visual grounding: elements, recent advances, and future directions. arXiv preprint arXiv:2406.05785 (2024). MM ’25, October 27–31, 2025, Dublin, Ireland Wencan Huang, Daizong Liu, & Wei Hu

    2024 arXiv

  38. [46]

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. 2024. A survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.07403 (2024)

    2024 arXiv

  39. [47]

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Xiang Fang, Keke Tang, Yao Wan, and Lichao Sun. 2024. Pandora’s Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  40. [48]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

    2024

  41. [49]

    Ilya Loshchilov, Frank Hutter, et al. 2017. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101 5 (2017), 5

    2017 arXiv

  42. [50]

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. 2022. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474 (2022)

    2022 arXiv

  43. [51]

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. 2022. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12309–12318

    2022

  44. [52]

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. 2024. DeepStack: Deeply Stacking Visual Tokens is Surpris- ingly Simple and Effective for LMMs. arXiv preprint arXiv:2406.04334 (2024)

    2024 arXiv

  45. [53]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    2023 arXiv

  46. [54]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  47. [55]

    Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. 2024. Shapellm: Universal 3d object understanding for embodied interaction. In European Conference on Computer Vision . Springer, 214–238

    2024

  48. [56]

    Zhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. 2024. Gpt4point: A unified framework for point-language understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26417–26427

    2024

  49. [57]

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao

  50. [58]

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh

  51. [59]

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. 2023. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 8216–8223

    2023

  52. [60]

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava- prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 (2024)

    2024

  53. [61]

    Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. 2023. Crossget: Cross-guided ensemble of tokens for accelerating vision- language transformers. arXiv preprint arXiv:2305.17455 (2023)

    2023 arXiv

  54. [62]

    Advances in neural information processing systems 34 (2021), 13937–13949

    Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems 34 (2021), 13937–13949

    2021

  55. [63]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  56. [64]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  57. [65]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition . 4566–4575

    2015

  58. [66]

    Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. 2024. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. In Proceedings of the 32nd ACM International Conference on Multimedia . 6617–6626

    2024

  59. [67]

    Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding

  60. [68]

    Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. 2024. Zero-TPrune: Zero- shot token pruning through leveraging of the attention graph in pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16070–16079

    2024

  61. [69]

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, Tao Jin, and Zhou Zhao

  62. [70]

    Vicuna. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://vicuna.lmsys.org/

    2023

  63. [71]

    Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. 2024. Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-aware generation. In Proceedings of the...

    2024

  64. [72]

    arXiv preprint arXiv:2412.05819 (2024)

    [CLS] Token Tells Everything Needed for Training-free Efficient MLLMs. arXiv preprint arXiv:2412.05819 (2024)

    2024 arXiv

  65. [73]

    Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. 2024. DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models. arXiv preprint arXiv:2405.20985 (2024)

    2024 arXiv

  66. [74]

    Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. 2024. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. arXiv preprint arXiv:2409.10197 (2024)

    2024 arXiv

  67. [75]

    In Findings of the Association for Computational Linguistics: NAACL

    Data-Efficiently Learn Large Language Model for Universal 3D Scene Perception. In Findings of the Association for Computational Linguistics: NAACL

  68. [76]

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. 2023. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769 (2023)

    2023 arXiv

  69. [77]

    Tatiana Zemskova and Dmitry Yudin. 2024. 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding. arXiv preprint arXiv:2412.18450 (2024)

    2024 arXiv

  70. [78]

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2024. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision . Springer, 131–147

    2024

  71. [79]

    Yiming Zhang, ZeMing Gong, and Angel X Chang. 2023. Multi3drefer: Grounding text description to multiple 3d objects. InProceedings of the IEEE/CVF International Conference on Computer Vision . 15225–15236

    2023

  72. [80]

    Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. 2024. Acceler- ating Multimodel Large Language Models by Searching Optimal Vision Token Reduction. arXiv preprint arXiv:2412.00556 (2024)

    2024 arXiv

  73. [81]

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang

  74. [82]

    arXiv preprint arXiv:2406.12275 (2024)

    VoCo-LLaMA: Towards Vision Compression with Large Language Models. arXiv preprint arXiv:2406.12275 (2024)

    2024 arXiv

  75. [83]

    Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. 2025. Inst3d- lmm: Instance-aware 3d scene understanding with multi-modal instruction tun- ing. In Proceedings of the Computer Vision and Pattern Recognition Conference . 14147–14157

    2025

  76. [84]

    Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. 2024. Aim: Adaptive inference of multi-modal llms via token merging and pruning. arXiv preprint arXiv:2412.03248 (2024)

    2024 arXiv

  77. [85]

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al

  78. [86]

    arXiv preprint arXiv:2410.04417 (2024)

    Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417 (2024)

    2024 arXiv

  79. [87]

    Jiedong Zhuang, Lu Lu, Ming Dai, Rui Hu, Jian Chen, Qiang Liu, and Haoji Hu

  80. [89]

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, and Yang You. 2024. A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for accelerating Large VLMs. arXiv preprint arXiv:2412.03324 (2024)

    2024 arXiv

  81. [90]

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. 2024. Dynamic diffusion transformer. arXiv preprint arXiv:2410.03456 (2024)

    2024 arXiv

  82. [91]

    Wangbo Zhao, Jiasheng Tang, Yizeng Han, Yibing Song, Kai Wang, Gao Huang, Fan Wang, and Yang You. 2024. Dynamic tuning towards parameter and inference efficiency for vit adaptation. arXiv preprint arXiv:2403.11808 (2024)

    2024 arXiv

  83. [93]

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. 2023. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773 (2023)

    2023 arXiv

  84. [94]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  85. [96]

    arXiv preprint arXiv:2412.20105 (2024)

    ST 3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming. arXiv preprint arXiv:2412.20105 (2024)

    2024 arXiv

  86. [2021]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 11 (2021), 7436–7456

    Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 11 (2021), 7436–7456

    2021

  87. [2024]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9579–9589

  88. [2025]

    arXiv preprint arXiv:2501.01428 (2025)

    GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models. arXiv preprint arXiv:2501.01428 (2025)

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.