Pith. sign in

REVIEW 5 major objections 5 minor 76 references

Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prior-SG frames region extraction as a MAP alignment against an LLM-generated prior graph, and beats wall-based and clustering baselines in open-plan scenes.

desk verdict A genuine advance in task-driven scene graph segmentation, but the uncalibrated LLM prior and manual per-dataset configuration leave the headline gains conditional on evidence the paper doesn't yet provide. read the letter →

arxiv 2608.06170 v1 pith:6J2BD6ME submitted 2026-08-06 cs.RO cs.CV

Giorgio Tonetti , Laurent Kneip , Abel Gawel , Marco Hutter This is my paper · ORCID
classification cs.ROcs.CV
keywords hierarchicalscenegraphsregionsegmentationMaximumAPosterioriinferenceLargeLanguageModelsopen-vocabularyperceptiongraphcutsMarkovrandomfieldtopologicalpriors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prior-SG claims that the region layer of a hierarchical scene graph should be computed not from walls or local clustering but from a probabilistic alignment against an LLM's expectation of the environment. As the robot moves, it aggregates RGB-D data into an Instance Graph of places and objects, while a Large Language Model synthesizes a Prior Graph stating which regions likely exist, which objects they contain, and how they connect. A maximum a posteriori (MAP) estimate over a Markov random field then labels every place with a region, fusing multi-scale visual textures, local geometry, discrete objects, and the topological prior. The authors report state-of-the-art semantic region segmentation on simulated residential datasets and large real-world open-plan sites, and show that changing the task prompt restructures the same physical map into different functional partitions. If right, this removes the need for hard structural assumptions and gives robots task-conditioned spatial understanding in arbitrary environments.

What carries the argument

The load-bearing object is the Prior Graph G_P, a probabilistic expectation of the environment generated by an LLM: it enumerates the task-relevant region vocabulary, declares expected objects per region with soft containment probabilities P(o|r), a region existence distribution P(r|E0), a pairwise adjacency matrix P(r_i<->r_j), and parametric geometry templates such as expected angles and variances. Its counterpart is the Instance Graph G_I, built incrementally: a topological place graph provides the spatial backbone, open-set detectors lift objects into 3D, and each place receives a visual descriptor via a CLIP feature pyramid with a scale-trust weighted integration kernel. The inference engine is a Markov random field whose unary potentials are a confidence-weighted log-linear pool of four experts (visual with a null sink for open-set clutter, geometric via a von Mises distribution on local inclination, object via proximity-discounted max-pooling, and the environment prior), and whose pairwise potential is a Potts-like term weighted by the LLM's transition log-likelihood. Because the LLM topology is non-metric (a Kitchen–Corridor transition and a Corridor–Bathroom transition may both be cheap while a direct Kitchen–Bathroom transition is expensive), the energy is minimized with alpha-beta swap rather than alpha-expansion.

What would settle it

Run Prior-SG on a space whose functional layout deliberately contradicts the LLM's common sense (for example, a single undivided studio flat where sleeping, cooking, and dining overlap), and compare its semantic segmentation against the same pipeline with a uniform, task-agnostic prior; if the prior-guided version does not outperform the prior-free baseline, the claim that the LLM topological priors drive the accuracy gains is falsified.

Watch

Extended reading notes

Core claim

Region extraction is formulated as a maximum a posteriori problem: find the labeling of place nodes that maximizes P(L | Z, E0, T), the posterior over region assignments given the sensor stream, the environment, and the task. The paper makes this tractable by decomposing it into a generative Prior Graph G_P synthesized by an LLM, an observable Instance Graph G_I built incrementally from the sensor stream, and a factorized posterior that is minimized by graph cuts. The Prior Graph supplies containment probabilities P(o|r), region existence P(r|E0), pairwise adjacency P(r_i<->r_j), and soft geometry templates; the Instance Graph supplies visual descriptors from a CLIP feature pyramid, geometric descriptors from the place-graph topology, and object detections lifted into 3D. The central empirical claim is that aligning these two graphs resolves local perceptual ambiguities that defeat bottom-up clustering and pure geometric heuristics, and reliably segments distant, partially observed regions in wall-less environments.

Load-bearing premise

The load-bearing premise is that the LLM's Prior Graph—its containment, adjacency, existence, and geometry probabilities—is a faithful logical expectation of the environment; if those priors are wrong for the actual layout or task, the MAP estimate is dragged toward incorrect regions and the reported gains over bottom-up methods could vanish.

Editorial extensions

If this is right

  • Scene graph construction no longer depends on wall detection or room-geometry heuristics; the same soft-prior machinery handles residential apartments, open-plan transit hubs, and outdoor campus runs.
  • Regions can be labeled before they are fully observed: on the train-station trajectory the system segments a staircase roughly 18 meters ahead and predicts the coming concourse before it is fully visible.
  • Task conditioning is zero-shot: with identical geometry and observations, changing only the LLM prompt reconfigures the map from real-estate rooms to hospitality service zones to utility maintenance zones.
  • The expert pool is modular: adding a geometric expert tightened boundaries at staircases without retraining, indicating that new sensing modalities can be plugged into the same alignment.
  • Enforcing the non-metric LLM topology through alpha-beta swap preserves the exact spatial rules rather than forcing them into a metric approximation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the Prior Graph comes from a nondeterministic LLM and the paper averages metrics over ten generations, deployment reliability would depend on prompt reproducibility; distilling the prior generator from accumulated Instance Graphs could remove that variance.
  • The visual expert's null sink absorbs out-of-distribution texture, so places that fit no hypothesized region carry near-zero confidence; that signal could be repurposed as a novelty map to drive exploration or request a revised task vocabulary.
  • The same MAP-alignment objective could be inverted into a calibration loop: observed object-region and region-region frequencies from the Instance Graph could revise the LLM's probability matrices, turning the static prior into an adaptive model of a specific building's quirks.
  • The ontological flexibility implies that one metric Instance Graph can serve multiple mission profiles simultaneously, with a separate Prior Graph per task sharing the same underlying geometry.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Prior-SG, a framework for hierarchical 3D scene graph construction that casts region segmentation as a MAP inference problem. An LLM generates a Prior Graph from the environment and task, specifying region/object vocabularies, containment and adjacency probabilities, and geometric templates. A separate Instance Graph is built online from RGB-D data using Hydra places, a multi-scale CLIP feature pyramid, and an open-vocabulary detector constrained to the prior object set. Region labels are assigned by graph cuts over a Markov Random Field that fuses visual, geometric, object, and environment experts with topological regularization. The method is evaluated on HM3D, TartanGround, CODa, and an internal train-station dataset, reporting large semantic F1 and mIoU gains over Clio and HOV-SG, plus ablations and qualitative demonstrations of task-conditioned partitioning.

Significance. The core idea of aligning a physically grounded graph with an LLM-generated topological prior is timely, and the formal MRF/MAP formulation is a clean way to combine heterogeneous cues. The multi-scale feature pyramid with explicit spatial and scale weighting is a practical improvement over global feature projection, and the paper includes thorough ablations and sensitivity analyses. The reported gains on CODa and the train station, if supported by a more rigorous evaluation, would be an advance for open-plan scene graphs. The main weaknesses are evaluation choices: the ground-truth vocabulary is given to the method, expert sets are activated per dataset, the internal train-station dataset lacks an annotation protocol, and the zero-shot flexibility claim rests on qualitative examples.

major comments (5)
  1. [VII-B.4, Table II, Table I] The quantitative headline results are not produced by a single fixed system. Table I reports HM3D Navigation F1 68.0, which in the ablation Table II corresponds to the V+E+O configuration (68.0); adding the geometric expert (all four experts, as stated in Sec. VII-B.4) gives 67.7. For the Train Station, Table I matches V+E+G, while CODa and TartanGround use V+E. Because the expert set is chosen per dataset, the claim that 'the system does not require scene-specific tuning' (Sec. VII-B.4) is weakened, and the reported margins may reflect model selection rather than a methodological advantage. Please either evaluate a single expert configuration everywhere or justify the per-dataset selection with a validation protocol.
  2. [IV-C and Table II] The environment prior P(r|E0) is a major driver of the reported gains; on CODa it raises F1 from 39.5 to 46.8 and mIoU from 34.4 to 43.8, and on the Train Station F1 from 58.4 to 65.1. These probabilities enter the unary and pairwise potentials directly in Eqs. (19) and (30), yet the paper offers no evidence that the LLM's containment, adjacency, or existence probabilities match the statistics of the evaluated environments, and no perturbation test of prior accuracy. Since the central claim is that the MAP alignment is reliable, the authors should add either a calibration study (comparing LLM probabilities to empirical distributions) or a sensitivity analysis in which the prior probabilities are corrupted or perturbed.
  3. [VII-B.2 and Table I] Fixing the target region set R_P to the ground-truth classes for all quantitative benchmarks is a strong oracle prior. It is not clear that the baselines receive the same advantage: Clio is task-driven and open-set, and HOV-SG selects labels from its vocabulary. The abstract's claim of 'state-of-the-art semantic region segmentation' is therefore conditional on providing the method with the evaluation vocabulary. Please clarify whether Clio and HOV-SG also received the ground-truth class list, or conduct an additional comparison with the LLM-generated vocabulary (without GT classes).
  4. [VII-A.4] The internal Train Station dataset is a key part of the empirical validation (large F1/mIoU gains over Clio), but the ground-truth annotation protocol is not described: who labeled the regions, what label set was used, whether the places were manually annotated, and how the LiDAR-RGB fusion was validated. Without this information or a release of the dataset, the reported train-station results cannot be independently verified. Please provide a detailed annotation protocol or use a public dataset.
  5. [VII-H and Fig. 4] The zero-shot ontological flexibility claim, highlighted in the abstract, is only demonstrated qualitatively with three task prompts on a single scene, with no ground-truth or quantitative evaluation. The authors themselves call it 'initial evidence' in the Conclusion, but the abstract's phrasing is stronger. Consider reporting task-conditioned metrics (e.g., label-consistent overlap between runs, or a comparison against a fixed-vocabulary baseline) to substantiate this claim.
minor comments (5)
  1. [V-B and VII-B.3] The theoretical derivation states r_Q is an intrinsic property of each place node, but Sec. VII-B.3 sets a homogeneous radius per scene (0.75m or 1.5m). Please reconcile the notation or clarify that the per-node value is currently simplified.
  2. [VII-B.2] Please report the exact LLM prompt and the parsing procedure for the probabilities; the paper averages over 10 runs but does not report the variance of the prior probabilities themselves.
  3. [Table II] The row labels are ambiguous; add explicit expert names to each row or a legend (V=visual, E=environment, G=geometric, O=object).
  4. [Eq. (14)] The denominator notation is typeset incorrectly; also specify the norm used (L2).
  5. [VII-G] The qualitative timestamps (t=0.0s, 10.4s, etc.) are useful, but please state the total trajectory length and the robot's speed to contextualize the 'long-range' claim of 18m.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM priors are external to the benchmark and no reported prediction reduces to a fitted value or self-citation chain.

full rationale

The paper's central derivation is a Bayesian MAP alignment: the LLM-produced Prior Graph supplies the label vocabulary, appearance templates, containment and adjacency probabilities, and the environment prior, while the Instance Graph supplies visual, geometric, and object observations. The reported region labels are the mode of a posterior that combines these inputs (Eqs. 17-19, 25, 30); no equation defines the output in terms of itself, and no parameter is fitted to the benchmark labels. The quantitative evaluation fixes R_P to the ground-truth classes (Sec. VII-B.2) and averages over 10 LLM generation runs; this is an evaluation constraint that isolates segmentation quality from vocabulary generation, not a fitted input renamed as a prediction. The object expert's P(r|o) is built from the LLM containment prior P(r|c), but the visual similarity P(c|o) comes from CLIP embeddings of the observed objects, so the evidence is not identical to the prior. The only overlapping-author citation is the TartanGround dataset reference [71], which is an external benchmark and not load-bearing for the method's derivation. The paper's own limitation statement (Sec. VIII) concedes that the prior is static and uncalibrated; that is a correctness risk about prior accuracy, not circularity. The skeptic concern about uncalibrated LLM priors is a legitimate external-validity threat, but the derivation chain itself is self-contained: the prior graph is generated from common sense rather than fitted to the evaluation data, and no self-citation chain or definitional equivalence forces the reported result.

Assumptions & free parameters 8 free parameters · 7 assumptions · 0 invented entities

The system depends on several hand-set parameters and domain assumptions. The largest freight is carried by the LLM prior quality and the manually selected per-dataset expert configuration, which are not learned or validated independently.

free parameters (8)
  • Spatial query radius r_Q = 0.75 m (HM3D, CODa, Train Station); 1.5 m (TartanGround)
    Defines expected physical size of semantic context per place; validated by ablation (Figure 7) and set per scene in Section VII-B.3.
  • Unary expert reliability weights lambda = lambda_vis=4, lambda_obj=1, lambda_geo=1, lambda_env=0.1
    Global hyperparameters for the log-linear pool; chosen by hand and fixed across experiments (Section VII-B.4).
  • Pairwise topological weight beta = 2
    Global regularizer strength for Potts-style transitions in Eq. 30.
  • Per-dataset expert activation = All experts on HM3D; visual+geometric+environment on Train Station; visual+environment on TartanGround and CODa
    Which experts run is chosen per dataset with justifications in Section VII-B.4; this is a data-dependent configuration choice.
  • Feature pyramid minimum crop resolution = 224x224 pixels (Medium)
    Selected via ablation (Figure 5); Fine and Coarse variants score lower.
  • Crop area threshold A_min = 2.25 m^2
    Candidate crops whose projected 3D area is below this are discarded (Section V-B).
  • Scale trust bandwidth gamma (Eq. 11) = not reported
    Bandwidth for the scale band-pass filter in the feature pyramid; numeric value not given in the text.
  • Softmax temperature tau and null score s_empty (Eq. 21) = not reported
    Temperature and null-class score for the visual expert's softmax; values not reported in the paper.
assumptions (7)
  • domain assumption LLM-synthesized probabilities (P(o|r), P(r|E0), P(r_i<->r_j|E0), geometry templates) are accurate enough to serve as ground-truth structural priors.
    Section IV-C generates these via prompting GPT-5.4 without verification against the observed environment or calibration.
  • domain assumption Hydra place graph faithfully represents the free-space topology and supports meaningful covariance-based geometry.
    Section V-A adopts Hydra's place extraction; geometric descriptors are derived from this graph in Eq. 16.
  • domain assumption Visual expert scores follow a temperature-scaled softmax with a calibrated null score; cosine similarities in CLIP space behave like log-likelihoods.
    Section VI-A.1 introduces this model for the visual expert.
  • domain assumption Expert independence in the Product of Experts is adequate for the MAP objective.
    Section VI-A factorizes the joint observation likelihood as a log-linear pool with independent experts.
  • standard math alpha-beta-swap finds a sufficient approximate minimum of the non-metric MRF energy.
    Section VI-B invokes Boykov et al. [68] for graph cut optimization.
  • domain assumption Ground-truth region annotations in all datasets, including the internal Train Station set, are correct and consistent.
    Section VII-A.4 does not describe the annotation process for the internal dataset.
  • domain assumption The feature pyramid with Gaussian spatial weights and scale band-pass approximates the optimal image crop for each place query.
    Section V-C, Eqs. 10-13, proposes this approximation without a formal optimality bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments." pith.science (2026). https://pith.science/paper/6J2BD6ME

@misc{pith2026260806170,
  author       = {Pith},
  title        = {Pith review of: Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6J2BD6ME}},
  note         = {Machine review of arXiv:2608.06170}
}
read the original abstract

Hierarchical 3D scene graphs are a promising representation for high-level spatial reasoning in autonomous mobile platforms. However, existing extraction frameworks typically rely on purely local visual clustering or strict geometric heuristics, such as wall-separated rooms, which fail in open-plan or arbitrarily-structured environments. We propose Prior-SG, a task- and prior-driven framework that casts scene graph generation fundamentally as a probabilistic alignment problem. As the robot explores, it continuously aggregates an incoming RGB-D sensor stream into a physically grounded Instance Graph utilizing a multi-scale, open-vocabulary feature fusion strategy. The system then infers the high-level functional semantics of this map through a Maximum A Posteriori (MAP) estimate, guided by a Prior Graph-a logical expectation of the environment's structure and task-relevant vocabulary synthesized dynamically by a Large Language Model. By optimizing a Markov Random Field that fuses heterogeneous experts (visual, geometric, and discrete objects) with these topological priors, the system resolves local perceptual ambiguities. We validate this approach across diverse simulated residential datasets and large, open-plan real-world environments. Prior-SG achieves state-of-the-art semantic region segmentation accuracy compared to recent baselines, robustly delineates distant functional boundaries in the absence of physical walls, and uniquely provides zero-shot ontological flexibility, enabling the robot to entirely restructure its spatial partitioning based on a given high-level task.

Figures

Figures reproduced from arXiv: 2608.06170 by the authors.

Figure 1
Figure 1. System architecture of the Prior-SG framework. As detailed in Section IV, the Prior Graph GP (left) is synthesized by a Large Language Model conditioned on a high-level task and environment. This involves top-down state-space truncation (A) to define the expected hierarchical ontology, and bottom-up attribute generation (B/C) to assign intrinsic visual templates, geometric priors, and structural edges. Concurrently,… view at source ↗
Figure 2
Figure 2. Qualitative comparison of region segmentation on the HM3D-00824 scene. Numbered trajectory markers (1–6) in the (a) ground truth (GT) correspond [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Qualitative online segmentation results along a continuous trajectory in the Train Station 0 scene. The rows represent sequential timesteps ( [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative demonstration of task-conditioned scene graph generation [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 7
Figure 7. Figure 7: System sensitivity to the spatial query radius ( [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 6
Figure 6. Figure 6: Semantic segmentation F1 score versus per-frame inference time [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 57 canonical work pages

  1. [1]

    3d scene graph: A structure for unified semantics, 3d space, and camera,

    I. Armeni, Z.-Y . He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese, “3d scene graph: A structure for unified semantics, 3d space, and camera,” inPro- ceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5664–5673

    work page 2019

  2. [2]

    Hydra: A real-time spatial perception system for 3D scene graph construction and optimization,

    N. Hughes, Y . Chang, and L. Carlone, “Hydra: A real-time spatial perception system for 3D scene graph construction and optimization,”Robotics: Science and Systems (RSS), 2022

    work page 2022

  3. [3]

    Foundations of spatial per- ception for robotics: Hierarchical representations and real-time systems,

    N. Hughes, Y . Chang, S. Hu, R. Talak, R. Abdulhai, J. Strader, and L. Carlone, “Foundations of spatial per- ception for robotics: Hierarchical representations and real-time systems,”The International Journal of Robotics Research, vol. 43, no. 10, pp. 1457–1505, 2024

    work page 2024

  4. [4]

    Ashita: Automatic scene-grounded hier- archical task analysis,

    Y . Chang, L. Fermoselle, D. Ta, B. Bucher, L. Carlone, and J. Wang, “Ashita: Automatic scene-grounded hier- archical task analysis,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 29 458–29 468

    work page 2025

  5. [5]

    Hierarchical Open-V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hierarchical Open-V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation,” in Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024

    work page 2024

  6. [6]

    Describe anything anywhere at any moment,

    N. Gorlo, L. Schmid, and L. Carlone, “Describe anything anywhere at any moment,”arXiv preprint arXiv:2512.00565, 2025

    arXiv 2025

  7. [7]

    Task and motion planning in hierarchical 3d scene graphs,

    A. Ray, C. Bradley, L. Carlone, and N. Roy, “Task and motion planning in hierarchical 3d scene graphs,” CoRR, vol. abs/2403.08094, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.08094

    arXiv doi:10.48550/arxiv.2403.08094 2024

  8. [8]

    Fungraph: Functionality aware 3d scene graphs for language-prompted scene interaction,

    D. Rotondi, F. Scaparro, H. Blum, and K. O. Arras, “Fungraph: Functionality aware 3d scene graphs for language-prompted scene interaction,”IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), 2025

    work page 2025

Show all 76 references
  1. [9]

    Open-V ocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces,

    C. Zhang, A. Delitzas, F. Wang, R. Zhang, X. Ji, M. Pollefeys, and F. Engelmann, “Open-V ocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  2. [10]

    Clio: Real-time task-driven open-set 3d scene graphs,

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone, “Clio: Real-time task-driven open-set 3d scene graphs,” IEEE Robotics and Automation Letters, 2024

    2024

  3. [11]

    Collaborative dynamic 3d scene graphs for automated driving,

    E. Greve, M. B ¨uchner, N. V ¨odisch, W. Burgard, and A. Valada, “Collaborative dynamic 3d scene graphs for automated driving,” in2024 IEEE International Confer- ence on Robotics and Automation (ICRA). IEEE, 2024, pp. 11 118–11 124

    2024

  4. [12]

    Opengraph: Open-vocabulary hierarchical 3d graph representation in large-scale outdoor environ- ments,

    Y . Deng, J. Wang, J. Zhao, X. Tian, G. Chen, Y . Yang, and Y . Yue, “Opengraph: Open-vocabulary hierarchical 3d graph representation in large-scale outdoor environ- ments,”IEEE Robotics and Automation Letters, 2024

    2024

  5. [13]

    Cognitive maps, cognitive collages, and spa- tial mental models,

    B. Tversky, “Cognitive maps, cognitive collages, and spa- tial mental models,” inEuropean conference on spatial information theory. Springer, 1993, pp. 14–24

    1993

  6. [14]

    J. M. Mandler,Stories, scripts, and scenes: Aspects of schema theory. Psychology Press, 2014

    2014

  7. [15]

    A survey on 3d scene graphs: Definition, generation and applica- tion,

    J. Bae, D. Shin, K. Ko, J. Lee, and U.-H. Kim, “A survey on 3d scene graphs: Definition, generation and applica- tion,” inRobot Intelligence Technology and Applications

  8. [16]

    Springer International Publishing, 2023, pp. 136–147

    2023

  9. [17]

    3d scene graphs: Open challenges and future directions,

    D. Rotondi, F. Argenziano, S. Koch, N. Hughes, M. Buechner, J. Wald, L. R. Schmid, D. Nardi, A. Valada, L. Paull, F. Tombari, L. Carlone, and K. O. Arras, “3d scene graphs: Open challenges and future directions,” 2026. [Online]. Available: https://arxiv.org/abs/2606.19383

    2026 arXiv

  10. [18]

    SG-nav: Online 3d scene graph prompting for LLM-based zero- shot object navigation,

    H. Yin, X. Xu, Z. Wu, J. Zhou, and J. Lu, “SG-nav: Online 3d scene graph prompting for LLM-based zero- shot object navigation,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  11. [19]

    3- d scene graph: A sparse and semantic representation of physical environments for intelligent agents,

    U.-H. Kim, J.-M. Park, T.-J. Song, and J.-H. Kim, “3- d scene graph: A sparse and semantic representation of physical environments for intelligent agents,”IEEE transactions on cybernetics, vol. 50, no. 12, pp. 4921– 4933, 2019

    2019

  12. [20]

    Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf, “Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,” inConference on Robot Learning. PMLR, 2023, pp. 23–72

    2023

  13. [21]

    3d scene graphs in robotics: A unified representation bridging geometry, semantics, and action,

    I. Catalano, C. Cueto Zumaya, J. Placed, J. Civera, W. Moreira Bessa, and J. Pe˜na Queralta, “3d scene graphs in robotics: A unified representation bridging geometry, semantics, and action,” 2025

    2025

  14. [22]

    When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models,

    X. Ma, B. Smart, Y . Bhalgat, S. Chen, X. Li, J. Ding, J. Gu, D. Z. Chen, S. Peng, J.-W. Bian, P. H. Torr, M. Pollefeys, M. Nießner, I. D. Reid, A. X. Chang, I. Laina, and V . A. Prisacariu, “When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-mo...

    2025

  15. [23]

    Image retrieval using scene graphs,

    J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3668–3678

    2015

  16. [24]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations,

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shamma et al., “Visual genome: Connecting language and vision using crowdsourced dense image annotations,”Interna- tional journal of computer vision, vol. 123, no. 1, pp. 3...

    2017

  17. [25]

    Kimera: From slam to spatial perception with 3d dynamic scene graphs,

    A. Rosinol, A. Violette, M. Abate, N. Hughes, Y . Chang, J. Shi, A. Gupta, and L. Carlone, “Kimera: From slam to spatial perception with 3d dynamic scene graphs,”The International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1510–1546, 2021

    2021

  18. [26]

    Sparse 3d topological graphs for micro-aerial vehicle planning,

    H. Oleynikova, Z. Taylor, R. Siegwart, and J. Nieto, “Sparse 3d topological graphs for micro-aerial vehicle planning,” in2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018, PREPRINT UNDER REVIEW 18 pp. 1–9

    2018

  19. [27]

    Situational graphs for robot navigation in structured indoor environments,

    H. Bavle, J. L. S ´anchez-L´opez, M. Shaheer, J. Civera, and H. V oos, “Situational graphs for robot navigation in structured indoor environments,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 9107–9114, 2022

    2022

  20. [28]

    S-graphs+: Real-time localization and mapping leveraging hierarchical representations,

    ——, “S-graphs+: Real-time localization and mapping leveraging hierarchical representations,”IEEE Robotics and Automation Letters, vol. 8, no. 8, pp. 4927–4934, 2023

    2023

  21. [29]

    S-graphs 2.0: A hierarchical-semantic optimiza- tion and loop closure for SLAM,

    ——, “S-graphs 2.0: A hierarchical-semantic optimiza- tion and loop closure for SLAM,”IEEE Robotics and Automation Letters, vol. 10, no. 12, 2025

    2025

  22. [30]

    Metric-semantic factor graph generation based on graph neural networks,

    J. A. Mill ´an-Romera, H. Bavle, M. Shaheer, H. V oos, and J. L. S ´anchez-L´opez, “Metric-semantic factor graph generation based on graph neural networks,” inIEEE International Conference on Robotics and Automation (ICRA), 2025

    2025

  23. [31]

    Area graph: Gen- eration of topological maps using the voronoi diagram,

    J. Hou, Y . Yuan, and S. Schwertfeger, “Area graph: Gen- eration of topological maps using the voronoi diagram,” inIEEE International Conference on Advanced Robotics, 2019

    2019

  24. [32]

    Towards terrain-aware task-driven 3D scene graph generation in outdoor environments,

    C. R. Samuelson, T. W. McLain, and J. G. Mangel- son, “Towards terrain-aware task-driven 3D scene graph generation in outdoor environments,” inIEEE ICRA Workshop on Field Robotics, 2025

    2025

  25. [33]

    Terra: Hierarchical terrain-aware 3D scene graph for task-agnostic outdoor mapping,

    C. R. Samuelson, A. Austin, S. Knoop, B. Romrell, G. R. Slade, T. W. McLain, and J. G. Mangelson, “Terra: Hierarchical terrain-aware 3D scene graph for task-agnostic outdoor mapping,” inIEEE International Conference on Robotics and Automation (ICRA), 2026

    2026

  26. [34]

    Hierarchical 3D scene graphs construction outdoors,

    J. Nyffeler, F. Tombari, and D. Bar ´ath, “Hierarchical 3D scene graphs construction outdoors,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  27. [35]

    Indoor and outdoor 3d scene graph gen- eration via language-enabled spatial ontologies,

    J. Strader, N. Hughes, W. Chen, A. Speranzon, and L. Carlone, “Indoor and outdoor 3d scene graph gen- eration via language-enabled spatial ontologies,”IEEE Robotics and Automation Letters, vol. 9, no. 6, pp. 4886– 4893, 2024

    2024

  28. [36]

    Learning 3d semantic scene graphs from 3d indoor reconstruc- tions,

    J. Wald, H. Dhamo, N. Navab, and F. Tombari, “Learning 3d semantic scene graphs from 3d indoor reconstruc- tions,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  29. [37]

    Scenegraphfusion: Incremental 3d scene graph pre- diction from rgb-d sequences,

    S.-C. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari, “Scenegraphfusion: Incremental 3d scene graph pre- diction from rgb-d sequences,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 7515–7525

    2021

  30. [38]

    3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans,

    A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone, “3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans,” inRobotics: Science and Systems, 2020

    2020

  31. [39]

    Khronos: A unified approach for spatio-temporal metric-semantic slam in dynamic environments,

    L. Schmid, M. Abate, Y . Chang, and L. Carlone, “Khronos: A unified approach for spatio-temporal metric-semantic slam in dynamic environments,” inProc. of Robotics: Science and Systems, 2024

    2024

  32. [40]

    3D VSG: Long-term seman- tic scene change prediction through 3D variable scene graphs,

    S. Looper, J. Rodriguez-Puigvert, R. Siegwart, C. Ca- dena, and L. Schmid, “3D VSG: Long-term seman- tic scene change prediction through 3D variable scene graphs,” inIEEE International Conference on Robotics and Automation (ICRA), 2023

    2023

  33. [41]

    Lost & found: Tracking changes from ego- centric observations in 3D dynamic scene graphs,

    T. Behrens, R. Zurbr ¨ugg, M. Pollefeys, Z. Bauer, and H. Blum, “Lost & found: Tracking changes from ego- centric observations in 3D dynamic scene graphs,”IEEE Robotics and Automation Letters, vol. 10, no. 4, pp. 3739–3746, 2025

    2025

  34. [42]

    Fross: Faster-than-real-time online 3d semantic scene graph generation from rgb-d images,

    H.-Y . Hou, C.-Y . Lee, M. Sonogashira, and Y . Kawanishi, “Fross: Faster-than-real-time online 3d semantic scene graph generation from rgb-d images,” inInternational Conference on Computer Vision (ICCV), 2025

    2025

  35. [43]

    SceneFun3D: Fine- grained functionality and affordance understanding in 3D scenes,

    A. Delitzas, A. Takmaz, F. Tombari, R. Sumner, M. Pollefeys, and F. Engelmann, “SceneFun3D: Fine- grained functionality and affordance understanding in 3D scenes,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  36. [44]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Y . Loet al., “Segment anything,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    2023

  37. [45]

    Fast segment anything,

    X. Zhao, W. Ding, Y . An, Y . Du, T. Yu, M. Li, M. Tang, and J. Wang, “Fast segment anything,” 2023

    2023

  38. [46]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Suet al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” inEuropean conference on computer vision. Springer, 2024, pp. 38–55

    2024

  39. [47]

    Sam 3: Segment anything with concepts,

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. R ¨adle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y . Zhou, L. Momeni, R. Hazra, S. Ding, S. ...

    2025 arXiv

  40. [48]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  41. [49]

    Mobileclip: Fast image-text models through multi-modal reinforced training,

    P. K. A. Vasu, H. Pouransari, F. Faghri, R. Vemula- palli, and O. Tuzel, “Mobileclip: Fast image-text models through multi-modal reinforced training,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 963–15 974

    2024

  42. [50]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inPro- ceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986

    2023

  43. [51]

    Siglip 2: Multilingual vision- language encoders with improved semantic understand- PREPRINT UNDER REVIEW 19 ing, localization, and dense features,

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafaet al., “Siglip 2: Multilingual vision- language encoders with improved semantic understand- PREPRINT UNDER REVIEW 19 ing, localization, and dense featu...

    2025 arXiv

  44. [52]

    Lerf: Language embedded radiance fields,

    J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, “Lerf: Language embedded radiance fields,” inInternational Conference on Computer Vision (ICCV), 2023

    2023

  45. [53]

    Visual language maps for robot navigation,

    C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” inProceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023

    2023

  46. [54]

    Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser, “Openscene: 3d scene understanding with open vocabularies,

    S. Peng, K. Genova, C. “. Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser, “Openscene: 3d scene understanding with open vocabularies,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 815–824

    2023

  47. [55]

    Language-driven semantic segmentation,

    B. Li, K. Q. Weinberger, S. Belongie, V . Koltun, and R. Ranftl, “Language-driven semantic segmentation,” in International Conference on Learning Representations,

  48. [56]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappaet al., “Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IE...

    2024

  49. [57]

    Findany- thing: Open-vocabulary and object-centric mapping for robot exploration in any environment,

    S. B. Laina, S. Boche, S. Papatheodorou, S. Schaefer, J. Jung, H. Oleynikova, and S. Leutenegger, “Findany- thing: Open-vocabulary and object-centric mapping for robot exploration in any environment,”arXiv preprint arXiv:2504.08603, 2025

    2025

  50. [58]

    The bare necessities: De- signing simple, effective open-vocabulary scene graphs,

    C. Kassab, M. Mattamala, S. Morin, M. B ¨uchner, A. Val- ada, L. Paull, and M. Fallon, “The bare necessities: De- signing simple, effective open-vocabulary scene graphs,” arXiv preprint arXiv:2412.01539, 2024

    2024 arXiv

  51. [59]

    Open-vocabulary indoor ob- ject grounding with 3d hierarchical scene graph,

    S. Linok and G. Naumov, “Open-vocabulary indoor ob- ject grounding with 3d hierarchical scene graph,”Optical Memory and Neural Networks, vol. 34, no. 3, pp. 323– 333, 2025

    2025

  52. [60]

    KeySG: Hierarchical keyframe-based 3D scene graphs,

    A. Werby, D. Rotondi, F. Scaparro, and K. Arras, “KeySG: Hierarchical keyframe-based 3D scene graphs,” inInternational Conference on Robotics and Automation (ICRA), 2026

    2026

  53. [61]

    Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships,

    S. Koch, N. Vaskevicius, M. Colosi, P. Hermosilla, and T. Ropinski, “Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June...

    2024

  54. [62]

    Agglomerative information bottleneck,

    N. Slonim and N. Tishby, “Agglomerative information bottleneck,”Advances in neural information processing systems, vol. 12, 1999

    1999

  55. [63]

    The information bottleneck method,

    N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” inProc. 37th Annual Allerton Conf. on Communication, Control and Computing, 1999, pp. 368–377, arXiv:physics/0004057

    1999 arXiv

  56. [64]

    Spatialvlm: Endowing vision- language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision- language models with spatial reasoning capabilities,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 14 455–14 465

    2024

  57. [65]

    Extract- ing zero-shot common sense from large language mod- els for robot 3d scene understanding,

    W. Chen, S. Hu, R. Talak, and L. Carlone, “Extract- ing zero-shot common sense from large language mod- els for robot 3d scene understanding,”arXiv preprint arXiv:2206.04585, 2022

    2022 arXiv

  58. [66]

    Extracting probabilistic knowledge from large language models for bayesian network parameterization,

    A. Nafar, K. B. Venable, Z. Cui, and P. Kordjamshidi, “Extracting probabilistic knowledge from large language models for bayesian network parameterization,”arXiv preprint arXiv:2505.15918, 2025

    2025 arXiv

  59. [67]

    LLM meets scene graph: Can large language models understand and generate scene graphs? a benchmark and empirical study,

    D. Yang, M. Kim, S. Kim, B.-w. Kwak, M. Park, J. Hong, W. Woo, and J. Yeo, “LLM meets scene graph: Can large language models understand and generate scene graphs? a benchmark and empirical study,” inProceedings of the 63rd Annual Meeting of the Association for Computational Li...

    2025

  60. [68]

    Fast approximate energy minimization via graph cuts,

    Y . Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,”IEEE Transactions on pattern analysis and machine intelligence, vol. 23, no. 11, pp. 1222–1239, 2002

    2002

  61. [69]

    To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty,

    Y . Abbasi Yadkori, I. Kuzborskij, A. Gy ¨orgy, and C. Szepesvari, “To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty,” Advances in Neural Information Processing Systems, vol. 37, pp. 58 077–58 117, 2024

    2024

  62. [70]

    Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI,

    S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y . Zhao, and D. Batra, “Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI,” inThirty-fifth Co...

  63. [71]

    Habitat 2.0: Training home assistants to rearrange their habitat,

    A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y . Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V . V ondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V . Koltun, J. Malik, M. Savva, and D. Batra, “Habitat 2.0: Training home assista...

    2021

  64. [72]

    Toward robust robot 3-d perception in urban environments: The ut campus object dataset,

    A. Zhang, C. Eranki, C. Zhang, J.-H. Park, R. Hong, P. Kalyani, L. Kalyanaraman, A. Gamare, A. Bagad, M. Estevaet al., “Toward robust robot 3-d perception in urban environments: The ut campus object dataset,” IEEE Transactions on Robotics, vol. 40, pp. 3322–3340, PREPRINT UNDE...

    2024

  65. [73]

    LAION-5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev, “LAION-5b: An open large-scale dataset for training next generation ...

    2022

  66. [74]

    Tartanground: A large-scale dataset for ground robot perception and navigation,

    M. Patel, F. Yang, Y . Qiu, C. Cadena, S. Scherer, M. Hutter, and W. Wang, “Tartanground: A large-scale dataset for ground robot perception and navigation,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025, pp. 20 524– 20 531

    2025

  67. [2021]

    Available: https://arxiv.org/abs/2109

    [Online]. Available: https://arxiv.org/abs/2109. 08238

  68. [2022]

    Available: https://openreview.net/forum? id=RriDjddCLN

    [Online]. Available: https://openreview.net/forum? id=RriDjddCLN

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.