The reviewed record of science sign in
Pith

arxiv: 2309.16650 · v1 · pith:FUUMMBBH · submitted 2023-09-28 · cs.RO · cs.CV

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

Qiao Gu , Alihusein Kuwajerwala , Sacha Morin , Krishna Murthy Jatavallabhula , Bipasha Sen , Aditya Agarwal , Corban Rivera , William Paul
show 8 more authors
Kirsty Ellis Rama Chellappa Chuang Gan Celso Miguel de Melo Joshua B. Tenenbaum Antonio Torralba Florian Shkurti Liam Paull
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:FUUMMBBHrecord.jsonopen to challenge →

classification cs.RO cs.CV
keywords planningconceptgraphsmodelsrepresentationsemanticapproachesdownstreamhttps
0
0 comments X
read the original abstract

For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Scaling Diverse Language Generation for 3D Visual Grounding

    cs.CL 2026-06 unverdicted novelty 7.0

    ViGiL3D++ generates diverse 3D visual grounding queries via scene-graph constraint sampling plus LLM language generation, yielding higher diversity than prior datasets and improved benchmark performance.

  2. Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphs

    cs.RO 2026-05 unverdicted novelty 6.0

    Dynamic scene graphs serve as explicit memory to improve imitation learning policies for spatial-temporal reasoning under partial observability in mobile and tabletop manipulation.

  3. A Survey on Vision-Language-Action Models for Embodied AI

    cs.RO 2024-05 unverdicted novelty 6.0

    This is the first survey on vision-language-action models, providing a taxonomy across three lines, plus summaries of datasets, simulators, benchmarks, challenges, and future directions in embodied AI.

  4. SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching

    cs.CV 2026-05 unverdicted novelty 5.0

    SceneGraphGrounder builds a persistent 3D scene graph from VLM-inferred relations in 2D views and solves grounding via constrained graph alignment, achieving competitive zero-shot results on ScanRefer with only RGB-D input.

  5. FUS3DMaps: Scalable and Accurate Open-Vocabulary Semantic Mapping by 3D Fusion of Voxel- and Instance-Level Layers

    cs.RO 2026-05 unverdicted novelty 5.0

    FUS3DMaps fuses voxel- and instance-level open-vocabulary layers inside a shared 3D voxel map to improve both layers and enable scalable accurate semantic mapping.

  6. A Modular Vision-Language-Action Robotics Framework for Indoor Environments

    cs.RO 2026-06 unverdicted novelty 3.0

    Describes a modular VLA framework with semantic voxel mapping via OwlViT and VLM-based command classification and grounding for the CMU VLA Challenge.