Pith. sign in

REVIEW 2 major objections 4 minor 57 references

FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning

T0 review · 2 major / 4 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Joint factor-graph inference over candidate functional edges yields better-calibrated confidence scores for open-vocabulary 3D functional scene graphs.

desk verdict Solid systems paper: dual factor-graph joint inference actually improves calibration on ambiguous functional edges, and FunThor is a useful denser benchmark. read the letter →

arxiv 2604.03696 v1 submitted 2026-04-04 cs.CV

Zhengyu Fu , René Zurbrügg , Kaixian Qu , Marc Pollefeys , Marco Hutter , Hermann Blum , Zuria Bauer This is my paper · ORCID
classification cs.CV
keywords functional3Dscenegraphsfactor-graphreasoningopen-vocabularyunderstandingbeliefpropagationconfidencecalibrationpart-centricreconstructionRGB-Dmapping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Functional relationships in 3D scenes (which switch controls which light, which knob which burner) are often ambiguous from static RGB-D views alone. Existing methods score each candidate pair independently and therefore cannot exploit the mutual constraints that humans use to resolve those ambiguities. FunFact first reconstructs an object- and part-centric 3D map, lets foundation models propose plausible functional relations, then converts every candidate edge into a binary variable inside a dual factor graph. Soft cardinality factors (encoding one-to-one or one-to-many structure) and geometric proximity factors are placed on those variables; belief propagation produces the joint posterior and therefore well-calibrated marginal confidences. A new synthetic benchmark, FunThor, supplies dense part-level geometry and rule-based functional labels so that both recall and calibration can be measured rigorously. Across three datasets the method recovers more nodes and relations while sharply lowering expected calibration error on the hardest ambiguous cases.

What carries the argument

The dual factor graph: each candidate functional edge becomes a binary variable; cardinality factors (soft one-to-one / one-to-many penalties) and unary proximity factors (exponential decay with median-normalized length) jointly constrain the variables so that marginal inference yields calibrated posteriors.

What would settle it

On a set of real multi-switch multi-light scenes whose true wiring is known, measure whether the factor-graph marginals assign higher probability to the true switch–light pairings than an identical pipeline that scores each pair independently; if the independent scores are better calibrated, the joint-inference claim fails.

Watch

Extended reading notes

Core claim

Encoding every candidate functional edge as a binary variable inside a dual factor graph whose factors combine LLM-derived cardinality priors with Euclidean proximity priors, then performing belief propagation, produces jointly consistent and substantially better-calibrated per-edge confidences than independent pairwise scoring.

Load-bearing premise

The soft cardinality and proximity factors, together with the LLM’s one-to-one and proximity flags, are assumed to be faithful enough models of real functional exclusivity; if they are systematically wrong for a scene class, belief propagation will confidently reinforce the error.

Editorial extensions

If this is right

  • Downstream planners can threshold or sample functional edges according to true posterior probabilities rather than uncalibrated scores.
  • Ambiguous relations (switch–light, knob–burner) become explicitly ranked by remaining uncertainty, enabling active verification by a robot.
  • New evidence (an observed action) can be injected as an additional factor and the entire graph re-inferred, updating all related confidences at once.
  • Open-vocabulary functional scene graphs become usable as compact, uncertainty-aware world models for language-conditioned manipulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the factor graph is modular, the same machinery could absorb continuous geometric affordance scores or learned interaction embeddings without redesigning the inference engine.
  • The dramatic ECE drop on ambiguous classes suggests that calibration, not just recall, should become a standard reporting metric for any future functional scene-graph method.
  • The hierarchical object–part detection step is doing as much work as the factor graph; removing it collapses interactive-element recall to baseline levels, implying that part-aware mapping is a prerequisite for reliable functional reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. FunFact constructs open-vocabulary probabilistic functional 3D scene graphs from posed RGB-D images. It first builds an object- and part-centric 3D map via VLM proposals, GroundingDINO, SAM, and multi-view fusion, then uses an LLM to propose candidate functional relations (object–part and object–object). These candidates become binary variables in a dual factor graph regularized by soft cardinality factors (encoding one-to-one/many patterns) and unary geometric proximity factors; belief propagation yields joint marginals and calibrated per-edge confidences. The authors introduce FunThor, a synthetic AI2-THOR-based benchmark with part-level geometry and dense rule-based functional annotations. Experiments on SceneFun3D, FunGraph3D and FunThor report higher node/relation recall than OpenFunGraph and related baselines, plus substantially lower ECE (especially on ambiguous switch/knob cases), with ablations isolating the factor-graph and hierarchical-mapping contributions.

Significance. If the empirical claims hold, the work supplies a practical and principled advance for functional scene understanding: joint factor-graph inference over interdependent edges produces better-calibrated confidences than independent pairwise scoring, which is directly useful for downstream robotic planning and AR guidance under visual ambiguity. FunThor fills a genuine evaluation gap by providing exhaustive, rule-generated annotations that permit precision and ECE measurement. Strengths include multi-dataset validation, clear ablations (Table 3), alternative-VLM checks in the supplement, and an explicit dual-graph formulation that cleanly separates semantic proposals from structural priors. The contribution is incremental relative to OpenFunGraph yet technically solid and well-supported.

major comments (2)
  1. §4.4 and Table 3: ECE comparison with OpenFunGraph is partially confounded because the baseline supplies confidences only for the subset of edges involving “outlet/switch/remote” (all other edges are hard-coded to 1.0). Restricting both methods to the same edge classes, or reporting a class-conditional ECE, would make the calibration claim fully load-bearing; the current “All” column may overstate the gap even though the Ambiguous column remains convincing.
  2. §3.2 (cardinality factor definition and Eq. 1): The soft cardinality penalty base b and the median-length proximity scale λ are free hyper-parameters whose sensitivity is never reported. A short sweep on FunThor (or a statement that results are stable for b ∈ [0.1,0.5]) is needed to confirm that the reported ECE/F1 gains are not brittle to these ordinary but un-ablated choices.
minor comments (4)
  1. Table 2 SceneFun3D columns: the large drop relative to OpenFunGraph is attributed to generic GT labels (“handle/knob”) versus open-vocabulary predictions; this is acknowledged but should be quantified (e.g., fraction of mismatches) so readers can judge how much of the gap is protocol artifact versus model error.
  2. Fig. 3 and the dual-graph construction: the inversion (scene-graph edges become factor-graph variables) is clear once explained, yet a one-sentence formal definition of the dual would help readers less familiar with factor graphs.
  3. Supplement §D: wall-clock times are dominated by per-frame GPT calls; a note on whether the same scene analysis can be amortized across frames (or replaced by a lighter open model) would strengthen the practicality discussion in §5.
  4. Typos / notation: “foucet” appears in a qualitative caption; λ_{k,j} is introduced without an explicit range; the cardinality factor uses b^{d_n-1} which is fine but could be written with a clearer indicator for the zero-connection case.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical systems pipeline whose factor-graph marginals and calibration gains are measured against independent benchmarks, not forced by construction from fitted inputs.

full rationale

FunFact is a modular CV systems paper. Scene reconstruction (VLM proposals + GroundingDINO/SAM + multi-view fusion) produces candidate nodes; an LLM then proposes relation templates that become binary variables in a dual factor graph regularized by soft cardinality factors ϕ_card (hyper-parameter b) and unary proximity factors ϕ_prox (λ = median edge length). Belief propagation yields per-edge marginals that are thresholded and evaluated for recall and ECE on SceneFun3D, FunGraph3D and the newly introduced FunThor. None of the reported quantities (node/relation Recall@K, triplet F1, ECE on ambiguous switch/knob cases) is algebraically identical to any fitted parameter or to the LLM/geometric priors by definition. Free parameters are ordinary hyper-parameters; ablations (Table 3) isolate the contribution of the factor graph versus hierarchical mapping. There is no self-definitional loop, no “prediction” that re-uses a fit of the same statistic, no load-bearing uniqueness theorem imported from overlapping authors, and no renaming of a known empirical pattern. The derivation chain is therefore self-contained against external benchmarks; residual limitations (open-vocabulary matching friction, foundation-model latency) are acknowledged but do not constitute circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 2 invented entities

The method rests on standard probabilistic graphical-model machinery plus a collection of domain modeling choices (cardinality patterns, proximity bias, LLM proposal quality) and ordinary detection hyper-parameters. No new physical entities are postulated; FunThor is a synthetic data resource, not an ontological invention.

free parameters (5)
  • cardinality penalty base b
    Hyper-parameter in ϕ_card that controls how strongly multi- or zero-connections are down-weighted; chosen by hand, not derived.
  • proximity scale λ_k,j
    Set to the median edge length inside each local candidate set; a data-dependent but still free scaling choice.
  • part-filter overlap thresholds (30 % / 70 %)
    Hard geometric cut-offs that discard detections; affect which parts ever enter the factor graph.
  • 3D association IoU 0.03 and DINOv2 cosine 0.5
    Merging thresholds that determine the final object/part inventory before relation proposal.
  • confidence threshold 0.5 for precision/recall
    Operating point used for all reported F1 numbers.
assumptions (3)
  • domain assumption LLM-proposed relation templates correctly capture the typical cardinality (one-to-one vs. many) and proximity requirements of real functional interactions.
    Sec. 3.2; if the LLM systematically mis-labels a relation class, the factor graph encodes the wrong structural prior.
  • ad hoc to paper Soft cardinality factors of the stated form plus unary proximity factors are adequate to resolve the dominant ambiguities (knob-burner, switch-light).
    The particular functional form of ϕ_card and ϕ_prox is chosen by the authors; no derivation from first principles is given.
  • standard math Belief propagation on the resulting (generally loopy) factor graph yields sufficiently accurate marginals for calibration purposes.
    Standard approximate inference assumption; pgmpy implementation is used without further analysis of approximation error.
invented entities (2)
  • dual functional factor graph
    purpose: Invert scene-graph edges into binary variables so that cardinality and proximity can be expressed as factors and joint inference performed.
    Modeling device introduced in Sec. 3.2; not claimed to exist outside the algorithm.
  • FunThor dataset
    purpose: Provide dense part-level geometry and exhaustive rule-based functional annotations for calibration and precision evaluation.
    Synthetic resource constructed by the authors; useful but not an independent physical discovery.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning." pith.science (2026). https://pith.science/paper/2604.03696

@misc{pith2026260403696,
  author       = {Pith},
  title        = {Pith review of: FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.03696}},
  note         = {Machine review of arXiv:2604.03696}
}
read the original abstract

Recent work in 3D scene understanding is moving beyond purely spatial analysis toward functional scene understanding. However, existing methods often consider functional relationships between object pairs in isolation, failing to capture the scene-wide interdependence that humans use to resolve ambiguity. We introduce FunFact, a framework for constructing probabilistic open-vocabulary functional 3D scene graphs from posed RGB-D images. FunFact first builds an object- and part-centric 3D map and uses foundation models to propose semantically plausible functional relations. These candidates are converted into factor graph variables and constrained by both LLM-derived common-sense priors and geometric priors. This formulation enables joint probabilistic inference over all functional edges and their marginals, yielding substantially better calibrated confidence scores. To benchmark this setting, we introduce FunThor, a synthetic dataset based on AI2-THOR with part-level geometry and rule-based functional annotations. Experiments on SceneFun3D, FunGraph3D, and FunThor show that FunFact improves node and relation discovery recall and significantly reduces calibration error for ambiguous relations, highlighting the benefits of holistic probabilistic modeling for functional scene understanding. See our project page at https://funfact-scenegraph.github.io/

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 3 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Taskography: Evaluating robot task planning over large 3d scene graphs

    Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, and Florian Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. InProceed- ings of the 5th Conference on Robot Learning, pages 46–58. PMLR, 2022. 2

    2022

  3. [3]

    pgmpy: A python toolkit for bayesian networks.Journal of Machine Learning Re- search, 25(265):1–8, 2024

    Ankur Ankan and Johannes Textor. pgmpy: A python toolkit for bayesian networks.Journal of Machine Learning Re- search, 25(265):1–8, 2024. 5

    2024

  4. [4]

    3d scene graph: A structure for unified semantics, 3d space, and cam- era

    Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and cam- era. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 5664–5673, 2019. 2

    2019

  5. [5]

    Embodiedrag: Dynamic 3d scene graph retrieval for efficient and scalable robot task planning, 2024

    Meghan Booker, Grayson Byrd, Bethany Kemp, Aurora Schmidt, and Corban Rivera. Embodiedrag: Dynamic 3d scene graph retrieval for efficient and scalable robot task planning, 2024. 2

    2024

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InPro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 2

    2021

  7. [7]

    Vision transformers need registers, 2023

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023. 2

    2023

  8. [8]

    SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes

    Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Francis Engelmann. SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1, 2, 5

    2024

Show all 57 references
  1. [9]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InProceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human la...

    2019

  2. [10]

    Spotlight: Robotic scene un- derstanding through interaction and affordance detection

    Tim Engelbracht, Ren ´e Zurbr¨ugg, Marc Pollefeys, Hermann Blum, and Zuria Bauer. Spotlight: Robotic scene un- derstanding through interaction and affordance detection. In2025 IEEE-RAS 24th International Conference on Hu- manoid Robots (Humanoids), pages 1–8. IEEE, 2025. 2

    2025

  3. [11]

    Collaborative dynamic 3d scene graphs for automated driving

    Elias Greve, Martin B ¨uchner, Niclas V ¨odisch, Wolfram Burgard, and Abhinav Valada. Collaborative dynamic 3d scene graphs for automated driving. In2024 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 11118–11124, 2024. 2

    2024

  4. [12]

    Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning. In2024 IEEE International Confer- ...

    2024

  5. [13]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. InInternational conference on machine learning, pages 1321–1330. PMLR,

  6. [14]

    Language-grounded dy- namic scene graphs for interactive object search with mobile manipulation.IEEE Robotics and Automation Letters, 9(10): 8298–8305, 2024

    Daniel Honerkamp, Martin B ¨uchner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada. Language-grounded dy- namic scene graphs for interactive object search with mobile manipulation.IEEE Robotics and Automation Letters, 9(10): 8298–8305, 2024. 2

    2024

  7. [15]

    Hi-dyna graph: Hierarchical dynamic scene graph for robotic auton- omy in human-centric environments, 2025

    Jiawei Hou, Xiangyang Xue, and Taiping Zeng. Hi-dyna graph: Hierarchical dynamic scene graph for robotic auton- omy in human-centric environments, 2025. 2

    2025

  8. [16]

    Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents. InInterna- tional conference on machine learning, pages 9118–9147. PMLR, 2022. 1

    2022

  9. [17]

    Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakr- ishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu

    brian ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalash- nikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent V...

    2023

  10. [18]

    Conceptfusion: Open-set multimodal 3d mapping.arXiv preprint arXiv:2302.07241, 2023

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping.arXiv preprint arXiv:2302.07241, 2023. 2

    2023 arXiv

  11. [19]

    Chang, and Manolis Savva

    Mukul Khanna*, Yongsen Mao*, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat Synthetic Scenes Dataset (HSSD-200): An Analy- sis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Na...

    2023

  12. [20]

    3-d scene graph: A sparse and semantic rep- resentation of physical environments for intelligent agents

    Ue-Hwan Kim, Jin-Man Park, Taek-Jin Song, and Jong- Hwan Kim. 3-d scene graph: A sparse and semantic rep- resentation of physical environments for intelligent agents. IEEE transactions on cybernetics, 50(12):4921–4933, 2019. 2

    2019

  13. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 2

    2023

  14. [22]

    Open3dsg: Open- vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships

    Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pe- dro Hermosilla, and Timo Ropinski. Open3dsg: Open- vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  15. [23]

    Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474,

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474,

    arXiv

  16. [24]

    A theory of shape by space carving.International journal of computer vision, 38(3):199–218, 2000

    Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving.International journal of computer vision, 38(3):199–218, 2000. 2

    2000

  17. [25]

    Satori: Towards proactive ar assistant with belief-desire-intention user modeling

    Chenyi Li, Guande Wu, Gromit Yeuk-Yin Chan, Dishita Gdi Turakhia, Sonia Castelo Quispe, Dong Li, Leslie Welch, Claudio Silva, and Jing Qian. Satori: Towards proactive ar assistant with belief-desire-intention user modeling. InPro- ceedings of the 2025 CHI Conference on Human F...

    2025

  18. [26]

    Beyond bare queries: Open-vocabulary ob- ject grounding with 3d scene graph

    Sergey Linok, Tatiana Zemskova, Svetlana Ladanova, Ro- man Titkov, Dmitry Yudin, Maxim Monastyrny, and Alek- sei Valenkov. Beyond bare queries: Open-vocabulary ob- ject grounding with 3d scene graph. In2025 IEEE Inter- national Conference on Robotics and Automation (ICRA), pag...

    2025

  19. [27]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuro- pean conference on computer vision, pages 38–55. Springer,

  20. [28]

    Clio: Real-time task- driven open-set 3d scene graphs.IEEE Robotics and Au- tomation Letters, 9(10):8921–8928, 2024

    Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, and Luca Carlone. Clio: Real-time task- driven open-set 3d scene graphs.IEEE Robotics and Au- tomation Letters, 9(10):8921–8928, 2024. 2

    2024

  21. [29]

    Predicting good probabilities with supervised learning

    Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. InProceedings of the 22nd international conference on Machine learning, pages 625–632, 2005. 4

    2005

  22. [30]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 815–824, 2023. 2

    2023

  23. [31]

    Virtualhome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018. 1

    2018

  24. [32]

    Ifr-explore: Learning inter-object functional relationships in 3d indoor scenes

    LI Qi and Kaichun Mo. Ifr-explore: Learning inter-object functional relationships in 3d indoor scenes. InInter- national Conference on Learning Representations (ICLR), 2022, 2022. 2

    2022

  25. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  26. [34]

    Sayplan: Ground- ing large language models using 3d scene graphs for scalable robot task planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable robot task planning. InProceedings of The 7th Conference on Robot Learning, pages 23–72. PMLR, 2023. 1

    2023

  27. [35]

    Task and motion planning in hierarchical 3d scene graphs, 2024

    Aaron Ray, Christopher Bradley, Luca Carlone, and Nicholas Roy. Task and motion planning in hierarchical 3d scene graphs, 2024. 2

    2024

  28. [36]

    Kimera: From slam to spatial perception with 3d dynamic scene graphs.The International Journal of Robotics Research, 40(12-14):1510–1546, 2021

    Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupta, and Luca Carlone. Kimera: From slam to spatial perception with 3d dynamic scene graphs.The International Journal of Robotics Research, 40(12-14):1510–1546, 2021. 2

    2021

  29. [37]

    Dennis Rotondi, Fabio Scaparro, Hermann Blum, and Kai O. Arras. Fungraph: Functionality aware 3d scene graphs for language-prompted scene interaction. In2025 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 4083–4090, 2025. 1, 2

    2025

  30. [38]

    Open- mask3d: Open-vocabulary 3d instance segmentation.Ad- vances in Neural Information Processing Systems, 36: 68367–68390, 2023

    Ayca Takmaz, Elisabetta Fedele, Robert Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Open- mask3d: Open-vocabulary 3d instance segmentation.Ad- vances in Neural Information Processing Systems, 36: 68367–68390, 2023. 2

    2023

  31. [39]

    Curiousbot: Interactive mo- bile exploration via actionable 3d relational object graph,

    Yixuan Wang, Leonor Fermoselle, Tarik Kelestemur, Ji- uguang Wang, and Yunzhu Li. Curiousbot: Interactive mo- bile exploration via actionable 3d relational object graph,

  32. [40]

    Hierarchical open- vocabulary 3d scene graphs for language-grounded robot navigation

    Abdelrhman Werby, Chenguang Huang, Martin B ¨uchner, Abhinav Valada, and Wolfram Burgard. Hierarchical open- vocabulary 3d scene graphs for language-grounded robot navigation. InFirst Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024. 2

    2024

  33. [41]

    Scenegraphfusion: Incremen- tal 3d scene graph prediction from rgb-d sequences

    Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremen- tal 3d scene graph prediction from rgb-d sequences. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7515–7525, 2021. 2

    2021

  34. [42]

    Incremental 3d semantic scene graph predic- tion from rgb sequences

    Shun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Fed- erico Tombari. Incremental 3d semantic scene graph predic- tion from rgb sequences. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5064–5074, 2023. 2

    2023

  35. [43]

    Dynamic open- vocabulary 3d scene graphs for long-term language-guided mobile manipulation.IEEE Robotics and Automation Let- ters, 10(5):4252–4259, 2025

    Zhijie Yan, Shufei Li, Zuoxu Wang, Lixiu Wu, Han Wang, Jun Zhu, Lijiang Chen, and Jihong Liu. Dynamic open- vocabulary 3d scene graphs for long-term language-guided mobile manipulation.IEEE Robotics and Automation Let- ters, 10(5):4252–4259, 2025. 2

    2025

  36. [44]

    What can i do around here? deep functional scene understanding for cognitive robots

    Chengxi Ye, Yezhou Yang, Ren Mao, Cornelia Ferm ¨uller, and Yiannis Aloimonos. What can i do around here? deep functional scene understanding for cognitive robots. In2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4604–4611. IEEE, 2017. 1

    2017

  37. [45]

    Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation

    Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation. InAdvances in Neural Infor- mation Processing Systems, pages 5285–5307. Curran Asso- ciates, Inc., 2024. 2

    2024

  38. [46]

    Open-vocabulary functional 3d scene graphs for real-world indoor spaces

    Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, and Francis Engelmann. Open-vocabulary functional 3d scene graphs for real-world indoor spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages ...

    2025

  39. [47]

    Tag map: A text-based map for spatial rea- soning and navigation with large language models

    Mike Zhang, Kaixian Qu, Vaishakh Patil, Cesar Cadena, and Marco Hutter. Tag map: A text-based map for spatial rea- soning and navigation with large language models. InPro- ceedings of The 8th Conference on Robot Learning, pages 2120–2146. PMLR, 2025. 2

    2025

  40. [48]

    Rec- ognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, Yandong Guo, and Lei Zhang. Rec- ognize anything: A strong image tagging model. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern...

  41. [49]

    Part Filtering 1 B .FunThorGeneration Process 1 C

    2 FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning Supplementary Material A . Part Filtering 1 B .FunThorGeneration Process 1 C . Instance Association and Merging Details 2 D . Inference Time Analysis 2 E . Pseudo-code for Functional Scene ...

  42. [50]

    less than 30% of the part’s bounding box (i.e., back- ground objects are incorrectly detected as functional parts of the parent object), or

  43. [51]

    In ad- dition, we generate posed RGB-D images for each scene to support perceptual functional scene understanding

    more than 70% of the object’s bounding box (i.e., the model misdetects the object itself as one of its functional parts) B.FunThorGeneration Process To buildFunThor, we follow a principled pipeline that pro- duces part-aware geometries and dense functional-relation annotations...

  44. [52]

    relation

    Exact matching - We annotate edges between node pairs with relation type “relation” if and only if the first node’s label exactly matches the triplet’s “first label” and the second node’s label exactly matches the triplet’s “sec- ond label”. Representative examples of rules em...

  45. [53]

    second label

    Proximity-based matching - For every node whose label matches the first label of the triplet, we identify the clos- est node matching the triplet’s “second label”. If such a node exists and the Euclidean distance between the node pair’s centers is less than one meter, we annot...

  46. [54]

    Representative rules in this cate- gory include: lever pushes down to activate toaster; han- dle pulls to open door

    Part-object and part-part matching - For objects that ex- hibit toggleable or openable properties in AI2-THOR and possess explicit functional part annotations such as power switches or handles (see Appendix B), we anno- tate the corresponding node pairs with their respective f...

  47. [55]

    outlet,” “switch,

    Manual matching - For semantically ambiguous rela- tions, we manually record affected node pairs and store these annotations in a JSON file for subsequent loading by the annotation pipeline. Currently, only stove knob- burner associations require manual matching. These rules a...

  48. [56]

    If a handle appears rounded, resembles a knob, and is attached to a compartment to aid in opening or closing it (such as a cabinet or drawer), label this part as ‘knob handle‘

  49. [57]

    knob” and “handle

    For all other handles that are functional components of an object (such as a door handle or drawer handle, except when covered in point 1), simply use ‘handle‘ as the part name, without specifying the handle type. ## Input Format - A single RGB image of an indoor scene. ## Out...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.