Multimodal 3D Reasoning Segmentation with Complex Scenes

Lewei Lu; Ling Shao; Shijian Lu; Xueying Jiang

REVIEW 3 major objections 5 minor 1 cited by

Multimodal 3D Reasoning Segmentation with Complex Scenes

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces a multi-object 3D reasoning segmentation task, builds the ReasonSeg3D benchmark with 20,113 question-answer pairs that carry 3D spatial relations, and proposes MORE3D, a network that extracts object-specific…

desk verdict A genuinely new multi-object 3D reasoning segmentation benchmark with a solid but incremental model; the reported gains are plausible, but the LLM-generated ground truth needs more scrutiny before the numbers can be fully trusted. read the letter →

arxiv 2411.13927 v4 pith:V463QNQS submitted 2024-11-21 cs.CV

Xueying Jiang , Lewei Lu , Ling Shao , Shijian Lu This is my paper · ORCID

classification cs.CV

keywords 3Dreasoningsegmentationmulti-objectpointcloudunderstandingspatialrelationsmultimodallargelanguagemodelReasonSeg3DMORE3DScanNet

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most 3D scene understanding models either segment objects without reasoning or answer text questions without segmentation, and the few 3D reasoning-segmentation models handle only single objects or single categories. This paper tries to establish that a machine can take a free-form question about an indoor 3D scene, reason out which several objects the question implies, and output both a text explanation with spatial relations and segmentation masks for each object. To make that claim testable, it builds ReasonSeg3D, a benchmark of 1,513 ScanNet scenes with 20,113 question-answer pairs whose answers name multiple objects and their 3D spatial relations, and MORE3D, which attaches a separate segmentation token to each object mentioned in the answer. On its own validation split, MORE3D reaches 30.19 cumulative IoU and 32.01 generalized IoU, above the previous 3D reasoning-segmentation method at 25.12 and 25.90. If the benchmark is valid, this gives the field a way to measure and improve multi-object 3D reasoning rather than single-object referring.

What carries the argument

The central mechanism is object-specific embedding extraction from <SEG> tokens. The LLM outputs a textual answer in which each object name is followed by a <SEG> token; a multi-seg index list records the positions of these tokens, and the LLM hidden states at those positions become per-object embeddings. A 3D decoder computes dot products between each object embedding and the per-point features to produce masks, with separate branches for mask prediction and classification. The textual answer loss supervises the explanations, and the mask loss combines binary cross-entropy and Dice loss.

What would settle it

Randomly sample ReasonSeg3D validation questions, have independent human annotators rewrite answers and select objects directly from the point cloud, then compare with the benchmark labels; if a meaningful fraction disagree on the objects or spatial relations, the reported 30.19 and 32.01 scores are measured against faulty labels. Alternatively, train MORE3D on a human-created subset of question-answer pairs and test on the same validation set; a large drop would show that the GPT-4o generation, not the model's reasoning, is doing the work.

Watch

Extended reading notes

Core claim

The central claim is that multi-object 3D reasoning segmentation is a learnable task and that the proposed MORE3D solves it better than existing methods. The paper argues that injecting 3D spatial relations into the textual answer gives the model the signal it needs to identify multiple objects of different categories in a cluttered point cloud, and that extracting one LLM embedding per <SEG> token, then decoding each embedding against shared per-point features, is the mechanism that turns one question into several masks. The created ReasonSeg3D is claimed to be large-scale and high-quality because the question-answer pairs are generated by GPT-4o with access to scene images and ground-truth segmentation, then human-verified, with about 3% corrected.

Load-bearing premise

The load-bearing premise is that ReasonSeg3D's question-answer pairs are trustworthy ground truth: they are generated by GPT-4o and then human-verified, with only about 3% corrected, and the paper reports no independent check for systematic errors in the spatial-relation language or in which objects the questions select.

Editorial extensions

If this is right

Future 3D reasoning-segmentation methods can be trained and compared on a common multi-object benchmark, since ReasonSeg3D provides 20,113 question-answer-mask triplets over 1,513 scenes.
Injecting 3D spatial relations into textual answers improves both explanation quality and mask accuracy; the ablation shows that either loss alone raises cIoU and gIoU, and both together give the largest gain.
Single-object 3D reasoning methods such as Reason3D transfer poorly to multi-object questions, indicating that multi-object reasoning requires multiple <SEG> tokens with object-specific embeddings.
Separating mask prediction and classification into independent decoder branches yields better cIoU and gIoU than a unified head.
Because the benchmark's masks come from ScanNetv2 instance annotations, ReasonSeg3D can be reused with existing 3D segmentation infrastructure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A second independent human annotation pass on a sample of ReasonSeg3D would test whether GPT-4o's spatial-relation descriptions contain systematic errors; the paper reports no such study.
Because the multi-seg index list is taken from ground-truth answers during training, MORE3D is not trained to decide how many objects a question implies; a variant that predicts the object count would probe whether the reasoning truly generalizes.
The reported comparisons train all baselines on the same ReasonSeg3D split, so the gap over open-vocabulary methods may partly reflect their inability to emit multi-object <SEG> answers rather than a pure reasoning deficit.
Per-question-type score breakdowns would show whether the performance gain comes from hard spatial-relation questions or from easy single-object cases that dominate the aggregated cIoU and gIoU.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate benchmark-plus-method paper. The benchmark is genuinely new: ReasonSeg3D is the first 3D reasoning segmentation dataset with multiple objects, multiple categories, and spatial-relation explanations. The model is a serviceable 3D adaptation of the SEG-token idea, and the ablations show both answer and mask losses matter. The reported numbers support the claim on their own benchmark.

The soft spots are real but not fatal. The ground truth for question-answer pairs is GPT-4o generated with only ~3% human correction and no verification protocol described. That means systematic bias in object selection or spatial-relation wording could infect both training and evaluation. The paper should report inter-annotator agreement or at least an error analysis on a sample. It also doesn't evaluate the quality of the generated text at all, so "strong reasoning abilities" is not directly measured. Missing error bars and repeated runs means the 30.19 vs 25.12 cIoU gap is a point estimate; that's a minor issue but worth noting. SegPoint is cited but missing from Table 2, which is a gap for a directly relevant baseline. Data and code are promised but not released, so the numbers can't be independently checked.

The stress-test's concern about GPT-4o bias is plausible but not demonstrated. The paper discloses the generation process, and the benchmark does use external ScanNet masks as targets, so there's no circularity. The main risk is that the benchmark may be too easy or biased toward salient objects, which would inflate the method's performance relative to true reasoning ability. That is a standard concern for LLM-generated benchmarks and can be addressed with better analysis rather than a rejection.

Who is this for: people working on 3D reasoning segmentation, embodied AI, and instruction-following in 3D. A serious referee should look at it, especially the dataset construction details. I'd recommend engagement with requests for the verification protocol, error bars, and the SegPoint baseline, but not a desk rejection.

Referee Report

3 major / 5 minor

Summary. The paper proposes a new task, multi-object 3D reasoning segmentation, in which a model takes a 3D point cloud and a user question as input and produces both a textual answer with 3D spatial relations and segmentation masks for multiple objects. The authors introduce ReasonSeg3D, a benchmark built from ScanNetv2 with GPT-4o-generated question-answer pairs (with about 3% human-corrected), and MORE3D, a method that combines a 3D encoder, a LLaMA-7B multimodal LLM with LoRA, object-specific <SEG> token embeddings, and a 3D decoder. On the ReasonSeg3D validation set, MORE3D reports 30.19 cIoU and 32.01 gIoU, outperforming the listed 3D segmentation, general scene understanding, and reasoning segmentation baselines. The paper also includes ablations on prediction-branch design and loss functions.

Significance. If the benchmark is shown to be trustworthy, this is a worthwhile contribution: it is the first multi-object 3D reasoning segmentation dataset with spatial-relation explanations, and MORE3D provides a clean, reproducible framework that integrates textual explanation with mask prediction. The design choices are clearly motivated, and the ablations show that both the textual answer loss and the mask loss contribute meaningfully. The main significance rests on the validity of ReasonSeg3D as an evaluation platform, because the same GPT-4o-derived data is used for both training and evaluation. The paper promises release of data and code, which would enable follow-up work; however, the current manuscript does not yet provide enough evidence about benchmark quality to fully support the claim of a high-quality, unbiased evaluation platform.

major comments (3)

[Dataset Generation Pipeline] The validity of ReasonSeg3D is load-bearing for the paper's central claims, but the manuscript provides no details on the human verification protocol, no inter-annotator agreement, no independent error analysis, and no study of systematic GPT-4o bias in object selection or spatial-relation phrasing. The text states that only about 3% of question-answer pairs required manual correction, but it does not report how many annotators were used, what instructions they received, what error taxonomy was applied, or how disagreements were resolved. Because the same benchmark is used both to train MORE3D (via the answer loss and the ground-truth-derived index list) and to evaluate every compared method, any systematic bias in GPT-4o's object choices or spatial vocabulary could inflate the reported gains. The authors should add an independent human evaluation on a held-out subset, report agreement statistics, and analyze the diversity of object categories and spatial-relation terms to demonstrate that the benchmark is not biased toward a narrow set of salient objects or phrases.
[Table 2] SegPoint (He et al. 2024) is cited in Related Work as a recent 3D reasoning segmentation method and is directly relevant to the comparison, but it is absent from Table 2. The claim that MORE3D 'achieves superior performance across all evaluation metrics' is incomplete without this baseline. The authors should either add SegPoint to Table 2 (training and evaluating it under the same protocol) or provide a concrete justification for its exclusion. If SegPoint cannot be run because of code or data constraints, that limitation should be stated explicitly.
[Evaluation Metrics and Experiments] The experimental section reports single-run cIoU and gIoU values with no error bars, repeated runs, or significance tests. The reported margin over Reason3D is about 5 cIoU points, and without variance estimates it is unclear whether this margin is reliable. Additionally, the definitions of cIoU and gIoU are not specified for the multi-object setting: it is not stated whether IoU is computed per object (averaged over objects then samples) or per sample (merging masks), nor how the cumulative union/intersection in cIoU is accumulated over multiple masks. The authors should report mean and standard deviation over at least three training runs and provide the exact aggregation formulas for both metrics.

minor comments (5)

[Table 1] The table header contains a typo: 'V enue' should be 'Venue'.
[Related Work and throughout] There are inconsistent spacing issues in method names, such as 'LLaV A' and 'V isionLLM'; these should be normalized to 'LLaVA' and 'VisionLLM'.
[Dataset Generation Pipeline] The sentence 'Samples that maximize diversity in both target objects and spatial-relation descriptions are then selected, thereby avoiding bias' is a claim without supporting evidence; the paper should describe the selection procedure concretely and show distribution statistics.
[Qualitative Results] Only two qualitative examples are shown in Figure 4, and there is no discussion of failure cases or limitations of the segmentation output; adding a few failure examples would strengthen the evaluation.
[Training Objectives and Ablations] The paper states that all three loss terms are assigned equal weights, but the classification loss Lcls is not included in the ablation study (Table 4); an ablation including Lcls would clarify its contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark targets come from external ScanNetv2 masks and GPT-4o-generated text, not from MORE3D's own outputs, and the experiments are standard train/validation evaluations.

full rationale

Walking the claimed derivation chain, the key quantities are not defined in terms of each other. ReasonSeg3D's segmentation masks come from ScanNetv2 ground-truth annotations, and the question-answer pairs are generated by GPT-4o with human verification, not by MORE3D. MORE3D is trained with standard cross-entropy, mask, and classification losses (Equations 3-6) against those external annotations and evaluated with cIoU and gIoU on a held-out validation split of the same benchmark. There is no equation or construction where a fitted parameter is renamed as a prediction, no self-definitional target, and no load-bearing self-citation chain. The one self-citation (Jiang et al. 2025 in Related Work) is not used to justify the central method or benchmark validity. The only mild concern is that training and evaluation share a self-created benchmark, which is normal for dataset papers and does not make the reported results equivalent to the inputs by construction. Thus the analysis finds no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on the benchmark's ground truth (ScanNet masks plus GPT-4o text) and on the choice of evaluation metrics; no new physical entities or conservation laws are introduced. Two model design choices (equal loss weights, separated prediction branches) are selected by hand or by validation-set comparison and affect the reported results.

free parameters (2)

Loss weights for Lans, Lmask, Lcls = equal (1:1:1)
Chosen by hand and not varied; Table 4 shows the mask and answer losses are necessary, but no sensitivity analysis is provided, so the equal weighting is an untested design choice affecting the reported performance.
Prediction-branch design (separated vs unified) = Separated mask and classification branches
The separated design is selected after comparing against a unified design on the validation set (Table 3), which is model selection on the evaluation data rather than a parameter-free choice.

assumptions (4)

domain assumption ScanNetv2 point cloud annotations (instance masks and semantic labels) are accurate enough to serve as ground truth for object masks and locations.
All ground-truth masks and categories are inherited from ScanNetv2 without additional verification; noise or missing instances in ScanNet would directly degrade ReasonSeg3D and the evaluation. Invoked throughout Dataset Construction and Experiment.
domain assumption GPT-4o-generated question-answer pairs, after human verification with about 3 percent corrections, correctly represent the objects and spatial relations needed for reasoning segmentation.
The benchmark's textual supervision and the reasoning component rest entirely on the quality of these pairs. The paper provides no inter-annotator agreement or error taxonomy, so the correctness of the untouched pairs is assumed. Location: Dataset Generation Pipeline.
domain assumption cIoU and gIoU from 2D reasoning segmentation literature are valid metrics for evaluating multi-object 3D reasoning segmentation.
Adopted from LISA-style benchmarks without adapting for text correctness; these metrics ignore the quality of the generated explanations and treat all objects equally. Location: Experimental Settings, Evaluation Metrics.
standard math Standard deep learning components (cross-entropy, BCE, Dice, LoRA, Adam) behave as expected for this task.
No formal guarantees are given, but these are standard tools and are not the target of the paper's contribution. Location: Training Objectives, Implementation Details.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multimodal 3D Reasoning Segmentation with Complex Scenes." pith.science (2026). https://pith.science/paper/V463QNQS

@misc{pith2026241113927,
  author       = {Pith},
  title        = {Pith review of: Multimodal 3D Reasoning Segmentation with Complex Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V463QNQS}},
  note         = {Machine review of arXiv:2411.13927}
}

read the original abstract

The recent development in multimodal learning has greatly advanced the research in 3D scene understanding in various real-world tasks such as embodied AI. However, most existing studies are facing two common challenges: 1) they are short of reasoning ability for interaction and interpretation of human intentions and 2) they focus on scenarios with single-category objects and over-simplified textual descriptions and neglect multi-object scenarios with complicated spatial relations among objects. We address the above challenges by proposing a 3D reasoning segmentation task for reasoning segmentation with multiple objects in scenes. The task allows producing 3D segmentation masks and detailed textual explanations as enriched by 3D spatial relations among objects. To this end, we create ReasonSeg3D, a large-scale and high-quality benchmark that integrates 3D segmentation masks and 3D spatial relations with generated question-answer pairs. In addition, we design MORE3D, a novel 3D reasoning network that works with queries of multiple objects and is tailored for 3D scene understanding. MORE3D learns detailed explanations on 3D relations and employs them to capture spatial information of objects and reason textual outputs. Extensive experiments show that MORE3D excels in reasoning and segmenting complex multi-object 3D scenes. In addition, the created ReasonSeg3D offers a valuable platform for future exploration of 3D reasoning segmentation. The data and code will be released.

Figures

Figures reproduced from arXiv: 2411.13927 by the authors.

**Figure 2.** Overview of our proposed MORE3D: Given an input point cloud, the 3D Encoder first extracts per-point features [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Extraction of object-specific point cloud embeddings. Each predicted textual answer contains multiple [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Segmentation visualization over the ReasonSeg3D validation set. Each case presents a user input question, the cor [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes
cs.CV 2025-07 conditional novelty 6.0 of 10

A large-scale 3D spatial reasoning segmentation benchmark with human-written queries that avoid object names shows current 3D vision-language models underperform.

Reference graph

Works this paper leans on

61 extracted references · 35 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716--23736

2022
[4]

Azuma, D.; Miyanishi, T.; Kurita, S.; and Kawanabe, M. 2022. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19129--19139

work page 2022
[5]

Boudjoghra, M. E. A.; Dai, A.; Lahoud, J.; Cholakkal, H.; Anwer, R. M.; Khan, S.; and Khan, F. S. 2024. Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation. arXiv preprint arXiv:2406.02548

arXiv 2024
[6]

Chen, S.; Chen, X.; Zhang, C.; Li, M.; Yu, G.; Fei, H.; Zhu, H.; Fan, J.; and Chen, T. 2024 a . LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26428--26438

work page 2024
[7]

Chen, S.; Zhu, H.; Chen, X.; Lei, Y.; Yu, G.; and Chen, T. 2023 a . End-to-end 3d dense captioning with vote2cap-detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11124--11133

work page 2023
[8]

Chen, T.; Yu, C.; Li, J.; Zhang, J.; Zhu, L.; Ji, D.; Zhang, Y.; Zang, Y.; Li, Z.; and Sun, L. 2024 b . Reasoning3D--Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models. arXiv preprint arXiv:2405.19326

arXiv 2024

Show all 61 references

[9]

Chen, Y.; Yang, S.; Huang, H.; Wang, T.; Xu, R.; Lyu, R.; Lin, D.; and Pang, J. 2024 c . Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370

2024 arXiv
[10]

Chen, Z.; Hu, R.; Chen, X.; Nie ner, M.; and Chang, A. X. 2023 b . Unit3d: A unified transformer for 3d dense captioning and visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 18109--18119

2023
[11]

X.; Savva, M.; Halber, M.; Funkhouser, T.; and Nie ner, M

Dai, A.; Chang, A. X.; Savva, M.; Halber, M.; Funkhouser, T.; and Nie ner, M. 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5828--5839

2017
[12]

Ding, R.; Yang, J.; Xue, C.; Zhang, W.; Bai, S.; and Qi, X. 2023. Pla: Language-driven open-vocabulary 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7010--7019

2023
[13]

Guo, Z.; Tang, Y.; Zhang, R.; Wang, D.; Wang, Z.; Zhao, B.; and Li, X. 2023 a . Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 15372--15383

2023
[14]

Guo, Z.; Zhang, R.; Zhu, X.; Tang, Y.; Ma, X.; Han, J.; Chen, K.; Gao, P.; Li, X.; Li, H.; et al. 2023 b . Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615

2023 arXiv
[15]

He, S.; and Ding, H. 2024. RefMask3D: Language-guided transformer for 3D referring segmentation. In Proceedings of the ACM International Conference on Multimedia, 8316--8325

2024
[16]

He, S.; Ding, H.; Jiang, X.; and Wen, B. 2024. SegPoint: Segment Any Point Cloud via Large Language Model. Proceedings of the IEEE/CVF European Conference on Computer Vision

2024
[17]

Hong, Y.; Zhen, H.; Chen, P.; Zheng, S.; Du, Y.; Chen, Z.; and Gan, C. 2023. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 20482--20494

2023
[18]

J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[19]

Huang, K.-C.; Li, X.; Qi, L.; Yan, S.; and Yang, M.-H. 2025. Reason3d: Searching and reasoning 3d segmentation via large language model. In International Conference on 3D Vision 2025

2025
[20]

Huang, P.-H.; Lee, H.-H.; Chen, H.-T.; and Liu, T.-L. 2021. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 1610--1618

2021
[21]

Huang, S.; Chen, Y.; Jia, J.; and Wang, L. 2022. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15524--15533

2022
[22]

Huang, Z.; Wu, X.; Chen, X.; Zhao, H.; Zhu, L.; and Lasenby, J. 2023. Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation. arXiv preprint arXiv:2309.00616

2023 arXiv
[23]

Jiang, X.; Li, W.; Zhang, X.; Shao, L.; and Lu, S. 2025. Exploring 3D Activity Reasoning and Planning: From Implicit Human Intentions to Route-Aware Planning. arXiv preprint arXiv:2503.12974

2025 arXiv
[24]

Kang, W.; Qu, M.; Kini, J.; Wei, Y.; Shah, M.; and Yan, Y. 2024. Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention. arXiv preprint arXiv:2405.18295

2024 arXiv
[25]

Kareem, A.; Lahoud, J.; and Cholakkal, H. 2024. PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model. Proceedings of the IEEE/CVF European Conference on Computer Vision

2024
[26]

Kingma, D. P. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations

2015
[27]

Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9579--9589

2024
[28]

Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 19730--19742. PMLR

2023
[29]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 a . Visual instruction tuning. Advances in Neural Information Processing Systems, 36

2023
[30]

Liu, K.; Zhan, F.; Zhang, J.; Xu, M.; Yu, Y.; El Saddik, A.; Theobalt, C.; Xing, E.; and Lu, S. 2023 b . Weakly supervised 3d open-vocabulary segmentation. Advances in Neural Information Processing Systems, 36: 53433--53456

2023
[31]

Ma, X.; Yong, S.; Zheng, Z.; Li, Q.; Liang, Y.; Zhu, S.-C.; and Huang, S. 2023. Sqa3d: Situated question answering in 3d scenes. International Conference on Learning Representations

2023
[32]

Milletari, F.; Navab, N.; and Ahmadi, S.-A. 2016. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision, 565--571. Ieee

2016
[33]

D.; Ngo, T

Nguyen, P. D.; Ngo, T. D.; Gan, C.; Kalogerakis, E.; Tran, A.; Pham, C.; and Nguyen, K. 2024. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

2024
[34]

Parelli, M.; Delitzas, A.; Hars, N.; Vlassis, G.; Anagnostidis, S.; Bachmann, G.; and Hofmann, T. 2023. Clip-guided vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5607--5612

2023
[35]

Peng, S.; Genova, K.; Jiang, C.; Tagliasacchi, A.; Pollefeys, M.; Funkhouser, T.; et al. 2023. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 815--824

2023
[36]

Pi, R.; Gao, J.; Diao, S.; Pan, R.; Dong, H.; Zhang, J.; Yao, L.; Han, J.; Xu, H.; Kong, L.; et al. 2023. Detgpt: Detect what you need via reasoning. Conference on Empirical Methods in Natural Language Processing

2023
[37]

Qian, Z.; Ma, Y.; Ji, J.; and Sun, X. 2024. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 4551--4559

2024
[38]

Ren, Z.; Huang, Z.; Wei, Y.; Zhao, Y.; Fu, D.; Feng, J.; and Jin, X. 2024. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26374--26383

2024
[39]

Rozenberszki, D.; Litany, O.; and Dai, A. 2022. Language-grounded indoor 3d semantic segmentation in the wild. In Proceedings of the IEEE/CVF European Conference on Computer Vision, 125--141. Springer

2022
[40]

W.; Pollefeys, M.; Tombari, F.; and Engelmann, F

Takmaz, A.; Fedele, E.; Sumner, R. W.; Pollefeys, M.; Tombari, F.; and Engelmann, F. 2023. Openmask3d: Open-vocabulary 3d instance segmentation. Advances in Neural Information Processing Systems

2023
[41]

Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[42]

Wang, J.; and Ke, L. 2024. LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1765--1774

2024
[43]

Wang, W.; Chen, Z.; Chen, X.; Wu, J.; Zhu, X.; Zeng, G.; Luo, P.; Lu, T.; Zhou, J.; Qiao, Y.; et al. 2023. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36

2023
[44]

Wu, C.; Ji, J.; Wang, H.; Ma, Y.; Huang, Y.; Luo, G.; Fei, H.; Sun, X.; Ji, R.; et al. 2024 a . Rg-san: Rule-guided spatial awareness network for end-to-end 3d referring expression segmentation. Advances in Neural Information Processing Systems, 37: 110972--110999

2024
[45]

Wu, C.; Liu, Y.; Ji, J.; Ma, Y.; Wang, H.; Luo, G.; Ding, H.; Sun, X.; and Ji, R. 2024 b . 3d-gres: Generalized 3d referring expression segmentation. Proceedings of the ACM International Conference on Multimedia

2024
[46]

Wu, C.; Ma, Y.; Chen, Q.; Wang, H.; Luo, G.; Ji, J.; and Sun, X. 2024 c . 3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 5940--5948

2024
[47]

Wu, J.; Zhong, M.; Xing, S.; Lai, Z.; Liu, Z.; Wang, W.; Chen, Z.; Zhu, X.; Lu, L.; Lu, T.; et al. 2024 d . VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks. arXiv preprint arXiv:2406.08394

2024 arXiv
[48]

Yan, C.; Wang, H.; Yan, S.; Jiang, X.; Hu, Y.; Kang, G.; Xie, W.; and Gavves, E. 2024 a . Visa: Reasoning video object segmentation via large language models. Proceedings of the IEEE/CVF European Conference on Computer Vision

2024
[49]

Yan, M.; Zhang, J.; Zhu, Y.; and Wang, H. 2024 b . MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

2024
[50]

F.; and Chai, J

Yang, J.; Chen, X.; Qian, S.; Madaan, N.; Iyengar, M.; Fouhey, D. F.; and Chai, J. 2024 a . Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In International Conference on Robotics and Automation, 7694--7701. IEEE

2024
[51]

Yang, J.; Ding, R.; Deng, W.; Wang, Z.; and Qi, X. 2024 b . Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19823--19832

2024
[52]

Yang, Y.; Jiang, P.-T.; Wang, J.; Zhang, H.; Zhao, K.; Chen, J.; and Li, B. 2024 c . Empowering Segmentation Ability to Multi-modal Large Language Models. arXiv preprint arXiv:2403.14141

2024 arXiv
[53]

Ye, Q.; Xu, H.; Xu, G.; Ye, J.; Yan, M.; Zhou, Y.; Wang, J.; Hu, A.; Shi, P.; Shi, Y.; et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178

2023 arXiv
[54]

Yuan, Z.; Yan, X.; Liao, Y.; Zhang, R.; Wang, S.; Li, Z.; and Cui, S. 2021. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Comp...

2021
[55]

Zhang, H.; Li, X.; and Bing, L. 2023. Video-llama: An instruction-tuned audio-visual language model for video understanding. Conference on Empirical Methods in Natural Language Processing

2023
[56]

Zhang, S.; Sun, P.; Chen, S.; Xiao, M.; Shao, W.; Zhang, W.; Liu, Y.; Chen, K.; and Luo, P. 2023. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601

2023 arXiv
[57]

Zhang, Y.; Gong, Z.; and Chang, A. X. 2023. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 15225--15236

2023
[58]

Zhang, Z.; Zhu, Z.; Li, P.; Liu, T.; Ma, X.; Chen, Y.; Jia, B.; Huang, S.; and Li, Q. 2024. Task-oriented sequential grounding and navigation in 3d scenes. arXiv preprint arXiv:2408.04034

2024 arXiv
[59]

Zhu, C.; Wang, T.; Zhang, W.; Chen, K.; and Liu, X. 2024 a . Scanreason: Empowering 3d visual grounding with reasoning capabilities. In Proceedings of the IEEE/CVF European Conference on Computer Vision, 151--168. Springer

2024
[60]

Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2024 b . Minigpt-4: Enhancing vision-language understanding with advanced large language models. International Conference on Learning Representations

2024
[61]

Zhu, Z.; Ma, X.; Chen, Y.; Deng, Z.; Huang, S.; and Li, Q. 2023. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2911--2921

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716--23736

2022

[4] [4]

Azuma, D.; Miyanishi, T.; Kurita, S.; and Kawanabe, M. 2022. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19129--19139

work page 2022

[5] [5]

Boudjoghra, M. E. A.; Dai, A.; Lahoud, J.; Cholakkal, H.; Anwer, R. M.; Khan, S.; and Khan, F. S. 2024. Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation. arXiv preprint arXiv:2406.02548

arXiv 2024

[6] [6]

Chen, S.; Chen, X.; Zhang, C.; Li, M.; Yu, G.; Fei, H.; Zhu, H.; Fan, J.; and Chen, T. 2024 a . LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26428--26438

work page 2024

[7] [7]

Chen, S.; Zhu, H.; Chen, X.; Lei, Y.; Yu, G.; and Chen, T. 2023 a . End-to-end 3d dense captioning with vote2cap-detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11124--11133

work page 2023

[8] [8]

Chen, T.; Yu, C.; Li, J.; Zhang, J.; Zhu, L.; Ji, D.; Zhang, Y.; Zang, Y.; Li, Z.; and Sun, L. 2024 b . Reasoning3D--Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models. arXiv preprint arXiv:2405.19326

arXiv 2024