Pith. sign in

REVIEW 5 major objections 5 minor 74 references

3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 3-D scene graph turns a room scan into a sparse, queryable, semantic map for a robot.

desk verdict Useful incremental step toward 3D scene graphs from RGB-D, with public code and honest reporting of a thin evaluation; the four-property claim outruns the evidence. read the letter →

arxiv 1908.04929 v1 pith:2UQV4BJY submitted 2019-08-14 cs.CV cs.RO

Ue-Hwan Kim , Jin-Man Park , Taek-Jin Song , Jong-Hwan Kim This is my paper · ORCID
classification cs.CVcs.RO
keywords 3-DscenegraphenvironmentmodelunderstandingintelligentagentsvisualquestionansweringtaskplanningRGB-DSLAMspuriousdetectionrejection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

An intelligent agent that knows what objects are in a room, where they are, and how they relate can answer questions and plan actions without needing a dense 3-D reconstruction. This paper argues that the right container for that knowledge is a 3-D scene graph: a directed graph whose nodes are object instances carrying a semantic label, a 3-D position, color, and other attributes, and whose edges are typed relations such as spatial, action, and prepositional relations. The authors propose a construction framework that ingests an RGB-D video stream, discards blurry frames, groups overlapping frames into keyframe groups, detects objects and relations, rejects spurious detections, and merges local graphs into a single global graph. They claim this representation is accurate, applicable, usable, and scalable, and they demonstrate the first two properties by using the graph in a visual-question-answering system and in a robot task-planning scenario. A sympathetic reader would come away with a concrete route from raw sensor data to a semantic world model that downstream AI modules can search and reason over.

What carries the argument

The carrying object is the 3-D scene graph itself, a sparse and semantic representation of an environment as a directed graph: nodes are object instances with semantic labels, 3-D positions, colors, and other attributes; edges are typed pairwise relations (action, spatial, description, preposition, comparison). The load-bearing machinery that makes the representation usable is the construction framework, especially three pieces: adaptive blurry image rejection removes unstable frames; keyframe group extraction avoids re-processing redundant frames and bounds the work per group; and spurious detection rejection prunes false object and relation detections using 3-D positions, word-vector semantics, and a precomputed relation distribution. A fourth piece, same-node detection, computes a weighted similarity over labels, colors, and Gaussian positions to decide when a newly recognized object is the same as an existing graph node, so the global graph stays compact as the camera moves. Together these modules convert an RGB-D video stream into the graph that downstream applications can search and plan over.

What would settle it

Apply the framework to a second set of RGB-D sequences spanning several room types (bedroom, office, classroom) and count how many of the graph's nodes and edges match a human-annotated ground truth. If the full pipeline's precision or recall is no better than the baselines on these sequences, or if its runtime grows faster than linearly in the number of input frames as coverage expands, the accuracy and scalability parts of the claim would fail.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the 3-D scene graph is an effective environment model, and that its construction framework can generate such graphs from an RGB-D image sequence. Concretely, the graph is defined as $G = (V, E)$, where each vertex carries an identifier, a semantic label (with a set of scored candidates), physical attributes including a Gaussian-distributed 3-D position, a color histogram, and a thumbnail, and each directed edge carries one of five relation types: action, spatial, description, preposition, or comparison. The framework's contribution is a sequence of modules — adaptive blurry image rejection, keyframe group extraction, spurious detection rejection, same-node detection, and incremental graph merge and update — that together turn raw frames into a single global graph. The experiments reported in the paper show that simply extending a 2-D scene graph generator to 3-D produces crowded, spurious graphs, while the full pipeline produces compact graphs, and the paper concludes that the four properties of an effective environment model (accuracy, applicability, usability, and scalability) are thereby established.

Load-bearing premise

The four-property claim assumes that one pre-selected living-room sequence is representative enough to demonstrate accuracy and applicability, and that scalability follows from the graph format itself even though it is not measured.

Editorial extensions

If this is right

  • A robot can build a queryable semantic map of a new room from a single pass with an RGB-D camera, without dense reconstruction or human annotation.
  • The same graph representation feeds both perceptual tasks (counting objects, answering attribute and relation questions) and action tasks (generating a planning-language problem description, then a robot plan).
  • Because the graph is sparse and updated incrementally, adding new observations extends coverage without re-processing the whole scene, which is what makes scalability a plausible property.
  • The modular pipeline means each recognition component can be upgraded independently; better object detectors or relation extractors improve the graph without changing the representation.
  • A naive lift of 2-D graphs to 3-D is not enough; the spurious-detection and same-node stages are what keep the graph accurate, so the framework as a whole is the contribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's scalability claim is argued from the graph format rather than measured; a natural stress test would run the pipeline on multi-room or long-duration sequences and record node/edge counts and merge time as the number of keyframe groups grows.
  • The same-node detection machinery is a general object-track fusion device; it could be reused to merge graphs built by different robots exploring the same space, turning isolated 3-D scene graphs into a shared map.
  • The relation dictionary prior is fixed before deployment; learning scene-specific relation statistics per room type could raise relation recall without changing the representation, especially in environments different from the test sequence.
  • Dynamic objects are deliberately out of scope, but the graph structure itself suggests a direct extension: let nodes carry a temporal state and update edges in real time as humans and objects move, which would make the model suitable for human-robot interaction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper defines a 3-D scene graph as a sparse semantic environment model whose nodes represent objects, edges represent pairwise relations, and vertices carry physical and visual attributes. It then proposes a modular construction framework that takes RGB-D image sequences, rejects blurry frames with ABIR, extracts keyframe groups with KGE, recognizes objects and relations, removes spurious detections with SDR, and merges local graphs into a global 3-D scene graph. The paper evaluates accuracy on one ScanNet sequence using human judgments against several baselines, and demonstrates applicability through VQA and task-planning examples in a simulated kitchen. It concludes that the four claimed properties of an effective environment model—accuracy, applicability, usability, and scalability—are established by the experiments.

Significance. The proposed representation is timely and the modular pipeline is clearly described; making the source code public is a concrete strength, and the VQA and task-planning demonstrations illustrate a sensible path from environment graphs to reasoning and planning. If the central claims were adequately supported, the work would be a useful contribution to semantic mapping for intelligent agents. However, the current evidence is too narrow for the strength of the conclusion: the accuracy result rests on a single hand-picked ScanNet sequence and subjective human ratings, while usability and scalability are asserted rather than measured. The paper is therefore not yet at the level of validation its four-property claim requires.

major comments (5)
  1. [VII-A1, Table II] The accuracy evaluation is based on a single ScanNet sequence (sequence 0, a living room), which was explicitly selected after filtering out sequences with too few objects, narrow coverage, or blurry images. No other environments or sequences are used in the quantitative comparison. This cannot support the general claim in Section IX that the experiments established accuracy; multi-sequence evaluation across room types, with per-sequence results, is required.
  2. [VII-A3] The evaluation metric relies on six human participants, majority voting, and averaged ratings, with no reported inter-rater agreement, per-participant variance, or significance testing. Consequently the differences among the methods in Table II are not established as reliable; report variance and a suitable statistical test, or describe the results as illustrative rather than as verification.
  3. [VII-A5, Table II] The full model 3D-full receives a lower human overall-accuracy rating than 3D-efficient, and the paper explains this by asserting that human judges prefer excessive detail to missing entities. This is an unexplained reversal and it directly weakens the claim that SDR improves graph quality. The authors should test the preference assumption (for example, by asking judges about their preferences) or analyze the specific errors introduced by SDR.
  4. [VII-A, IX] Usability and scalability are never measured. Section VII-A says the graph structure "already guarantees" both properties, and Section IX states that the experimental results established all four properties, but no experiment quantifies ease of use, memory use, or how runtime and graph size scale with environment size. Either add such measurements or explicitly restrict the conclusion to accuracy and applicability.
  5. [VII-B3] The applicability demonstration is a single qualitative simulation scenario with no quantitative measures of task success, planning validity, or VQA answer correctness. This supports a feasibility claim, but not the "broad applicability" stated in Section VII-B3 and the conclusion; please provide metrics for the demonstration or moderate the claim accordingly.
minor comments (5)
  1. [VII-A4] Many free parameters (ABIR alpha, g, b; SDR threshold; histogram bins; same-node weights and threshold) are fixed without sensitivity analysis. A brief sensitivity study or discussion of their influence would increase confidence in the results.
  2. [VII-A3] The majority-voting rule should specify whether agreement is computed per entity and how ties among six raters are handled.
  3. [Table II, Fig. 4] Cells for 2D-basic and 3D-basic are incomplete because the graphs were overcrowded; mark these entries explicitly as not available and make the omitted graphs available as supplementary material.
  4. [V-C1, Eqs. (10)-(15)] The definitions of C_o, C_c, f_si, and the score function are dense and hard to follow; a notation table or a rewritten definition would improve readability.
  5. [IV-A, V-C1] There are typographical errors such as "Al o w e rα", "t o", and "stotal,i st h e"; the manuscript needs a careful proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the four-property claim rests on external data, human judgments, and application demonstrations, with evidential gaps but no self-referential reduction.

full rationale

The paper's derivation chain is not circular. The 3-D scene graph is defined independently in Section V-A as a directed graph G=(V,E) with object nodes and relation edges, and construction is a modular pipeline: ABIR, KGE, Faster R-CNN detection, F-Net relation extraction, SDR with Visual Genome priors, and graph merge/update. Accuracy is not a fitted output: it is measured against external ScanNet imagery by six human judges counting spurious and missing entities and rating overall accuracy (Section VII-A3), while the relation prior is built from Visual Genome in advance, not from the evaluation. Applicability is demonstrated by deploying the graph in VQA and PDDL-based task planning (Sections VI and VII-B), where the graph is queried or converted, not tuned to the demonstration outcome. The main weaknesses are evidential rather than circular: usability and scalability are asserted from the graph structure ('the widely used and well-known graph structure already guarantees both usability and scalability,' Section VII-A), and accuracy is evaluated on a single hand-filtered ScanNet sequence, which weakens the breadth of the conclusion but does not make any result equivalent to its input. Self-citations to prior Mybot modules ([30]-[32]) are used only in the simulation demonstration and are not load-bearing for the central environment-model claim. No equation or fitted parameter is renamed as a prediction.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The framework depends on many hand-tuned parameters (ABIR constants, KGE thresholds, same-node weights, SDR threshold, histogram bins), a single selected dataset sequence, and several modeling assumptions about the transfer of 2-D priors to 3-D scenes. None of these are derived or externally verified; they enter at specific modules and in the evaluation protocol.

free parameters (9)
  • ABIR smoothing factor alpha = 0.9
    Section VII-A4: 'For ABIR, we set alpha, g, and b as 0.9, 30, and 25, respectively. We tuned the values before applying the algorithm.'
  • ABIR gain g = 30
    Section VII-A4, hand-tuned before the experiment.
  • ABIR offset b = 25
    Section VII-A4, hand-tuned before the experiment.
  • KGE overlap thresholds tanchor and tkeyframe = not reported (only tanchor > tkeyframe stated)
    Section IV-B defines these thresholds; they control which frames become keyframes and anchors. Exact values are not disclosed, which hinders exact replication.
  • KGE sampling count = 1000 points
    Section VII-A4: 'we sampled 1000 points after projection from source to target frame to reduce the amount of computation.'
  • SDR relation probability threshold = 0.5
    Section VII-A4: 'Relations with the probabilities lower than the threshold were rejected.'
  • SDR rejected object classes = 68 predefined classes among 400
    Section VII-A4: 'SDR rejects 68 predefined object classes among 400 classes the recognition module can classify.'
  • Color histogram bins = 8 bins per axis (512 total)
    Section VII-A4: 'we divided each axis of the color space into eight bins, resulting in histogram size of 512.'
  • Same-node detection weights and threshold = w_label=0.375, w_color=0.25, w_position=0.375; threshold=0.7
    Section VII-A4: 'we set w_label, w_color, and w_position as 0.375, 0.25, and 0.375, respectively, and used 0.7 as the threshold.'
assumptions (6)
  • domain assumption The four properties (accuracy, applicability, usability, scalability) define a versatile environment model; no proof is given that they are necessary or sufficient.
    Section I lists these properties as claims, but they are never formally defined or shown to be the right criteria for environment models.
  • ad hoc to paper A graph structure 'readily guarantees usability as well as scalability'.
    Section I and the Conclusion assert this; it is not measured or derived.
  • domain assumption Visual Genome relation statistics transfer from 2-D images to 3-D indoor scenes.
    Section IV-C uses a relation dictionary built from Visual Genome as a prior for rejecting spurious relations in 3-D scene graphs.
  • domain assumption Human majority voting (two-thirds agreement) is a valid ground truth for graph accuracy.
    Section VII-A3 defines the evaluation metric using six participants and majority voting; the reliability of this ground truth is not validated.
  • domain assumption The single selected ScanNet sequence is representative of general indoor environments.
    Section VII-A1 selects one sequence after filtering; the paper generalizes from this sequence to the four-property claim.
  • domain assumption Each dimension of an object's 3-D position is independent and identically distributed Gaussian.
    Section V-B: 'we assume independent identically distributed for each dimension, x, y, and z' when modeling position uncertainty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents." pith.science (2026). https://pith.science/paper/2UQV4BJY

@misc{pith2026190804929,
  author       = {Pith},
  title        = {Pith review of: 3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UQV4BJY}},
  note         = {Machine review of arXiv:1908.04929}
}
read the original abstract

Intelligent agents gather information and perceive semantics within the environments before taking on given tasks. The agents store the collected information in the form of environment models that compactly represent the surrounding environments. The agents, however, can only conduct limited tasks without an efficient and effective environment model. Thus, such an environment model takes a crucial role for the autonomy systems of intelligent agents. We claim the following characteristics for a versatile environment model: accuracy, applicability, usability, and scalability. Although a number of researchers have attempted to develop such models that represent environments precisely to a certain degree, they lack broad applicability, intuitive usability, and satisfactory scalability. To tackle these limitations, we propose 3-D scene graph as an environment model and the 3-D scene graph construction framework. The concise and widely used graph structure readily guarantees usability as well as scalability for 3-D scene graph. We demonstrate the accuracy and applicability of the 3-D scene graph by exhibiting the deployment of the 3-D scene graph in practical applications. Moreover, we verify the performance of the proposed 3-D scene graph and the framework by conducting a series of comprehensive experiments under various conditions.

Figures

Figures reproduced from arXiv: 1908.04929 by the authors.

Figure 1
Figure 1. Example of 2-D scene graph. The scene graph abstracts the given [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Process of KGE. The input image frames are classified into three [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. First keyframe group of the first experiment sequence and the resulting 2-D and 3-D scene graphs. Red, green, and blue denote node, edge, and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Applicability demonstration. Human scanned through the kitchen environment and gathered relevant information for constructing a 3-D scene [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 73 canonical work pages

  1. [1]

    Accuracy: The model should delineate environments precisely and provide intelligent agents with correct information

    work page

  2. [2]

    Applicability: Intelligent agents should be able to utilize the model in performing various types of tasks rather than just one specific task

    work page

  3. [3]

    Usability: The model should not require a complicated procedure for usage, but provide intuitive user interface for application

    work page

  4. [4]

    The claimed properties of environment models enhance the autonomy of intelligent agents

    Scalability: The model should be able to depict both large- and small-scale environments and increment the coverage step by step. The claimed properties of environment models enhance the autonomy of intelligent agents. Intelligent agents could collect correct information regarding the environments in the process of constructing the models (accuracy), util...

    work page 2019

  5. [5]

    The two applications are under active research in computer vision [5], NLP [6], and robotics societies [7]

    visual question and answering (VQA) and 2) task planning. The two applications are under active research in computer vision [5], NLP [6], and robotics societies [7]. The proposed 3-D scene graph construction framework extracts relevant semantics within environments, such as object categories and relations between objects as well as phys- ical attributes, ...

    work page

  6. [6]

    We define the concept of the 3-D scene graph which represents the environments in an accurate, applicable, usable, and scalable way

    work page

  7. [7]

    We design the 3-D scene graph construction framework which generates 3-D scene graphs for environments upon receiving a sequence of observations

    work page

  8. [8]

    We provide two application examples of the 3-D scene graph: a) VQA and b) task planning

    work page

Show all 74 references
  1. [9]

    We conduct a series of thorough experiments and analyze the experiments both quantitatively and qual- itatively to verify the performance of the 3-D scene graph

  2. [10]

    The remainder of this paper is organized as follows

    We make the source code of the algorithms presented in this paper public 1 to contribute to the research society and the development of the field. The remainder of this paper is organized as follows. Section II introduces the related works with associated results and issues. Se...

  3. [11]

    The keyframe determines the first anchor frame and the coverage of the keyframe group

    Keyframe: A keyframe works as a reference. The keyframe determines the first anchor frame and the coverage of the keyframe group

  4. [12]

    Only the latest anchor frame is active and the active anchor frame determines the next anchor frame

    Anchor Frame: An anchor frame is either active or inac- tive. Only the latest anchor frame is active and the active anchor frame determines the next anchor frame

  5. [13]

    Garbage Frame: Frames other than the keyframes and the anchor frames are classified as garbage frames and discarded due to redundancy. Fig. 3 and Algorithm 1 depict the process of the KGE mod- ule. First of all, the module sets the first nonblurry incoming frame as the first keyf...

  6. [14]

    Identification Number (ID): A unique number assigned to each object in the environment

  7. [15]

    Semantic Label: Category of the object classified by the object recognition module

  8. [16]

    Physical Attributes: Physical characteristics, such as size (height and width), major colors, or position relative to the first keyframe

  9. [17]

    On the other hand, the following lists the types of relations an edge could stand for

    Visual Feature: A thumbnail, color histogram, or extracted visual features with descriptors. On the other hand, the following lists the types of relations an edge could stand for

  10. [18]

    This article has been accepted for inclusion in a future issue of this journal

    Actions: Behaviors shown by one object toward another one (e.g., feeding). This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 6 IEEE TRANSACTIONS ON CYBERNETICS

  11. [19]

    Spatial Relation: Spacious relations, such as distance and relative position (e.g., in front of)

  12. [20]

    Description: States of one object related by another object (e.g., wear)

  13. [21]

    Preposition: Semantic relations that are expressed by prepositions (e.g., with)

  14. [22]

    As one object is subjective and the other is objective given a pair of objects, the edges in 3-D scene graphs are directed (the subjective objects point at the objective objects)

    Comparison: Relative attributes of one object compared to another one (e.g., smaller). As one object is subjective and the other is objective given a pair of objects, the edges in 3-D scene graphs are directed (the subjective objects point at the objective objects). Therefore,...

  15. [23]

    In the process, we assume independent identically distributed for each dimension, x, y, and z, and keep the number of points used for the evaluation

    for the Gaussian distribution of the 3-D position ∼ N(μ, σ2). In the process, we assume independent identically distributed for each dimension, x, y, and z, and keep the number of points used for the evaluation. Next, we calculate the color histogram for the object, hH,S,V , a...

  16. [24]

    We utilize the following features for the same node detection: object label, 3-D position, and color histogram

    Same Node Detection: Without same node detection, 3-D scene graph would explode with same nodes added numer- ous times and multiple observations of the same objects cannot be integrated effectively. We utilize the following features for the same node detection: object label, 3...

  17. [25]

    Upon receiving image frames, local 3-D scene graphs are generated and merged to the global 3-D scene graph

    Merge and Update: The 3-D scene graph constructed by the first keyframe forms the initial global 3-D scene graph for the environment. Upon receiving image frames, local 3-D scene graphs are generated and merged to the global 3-D scene graph. The merge process compares the nodes...

  18. [26]

    Object Counting: Either simple (e.g., how many cups are in the environment?) or hierarchical (e.g., how many pieces of cutlery are there?)

  19. [27]

    Counting With Attributes: Number of objects with spe- cific attributes, such as size, visual feature, and location (e.g., how many red chairs are in the environment?)

  20. [28]

    Counting With Relations: Number of objects distinc- tively related to a specific object (e.g., how many objects are on the shelf?)

  21. [29]

    The above-listed questions can be easily converted to a query- form (machine readable form) even when asked in a free-form and open-ended manner by employing a few NLP algorithms

    Multimodal VQA: An answer by providing a thumb- nail of an object (e.g., show me the biggest bowl in the environment). The above-listed questions can be easily converted to a query- form (machine readable form) even when asked in a free-form and open-ended manner by employing ...

  22. [30]

    The problem description contains the information regarding the categories and states of objects and the goal definition

    domain description. The problem description contains the information regarding the categories and states of objects and the goal definition. The domain description describes the prim- itive actions robots can take. The descriptions are written in planning domain definition langu...

  23. [31]

    ScanNet consists of 1513 sequences, which were collected using one type of RGBD-sensor

    Dataset: We selected a few sequences from the ScanNet dataset [28]. ScanNet consists of 1513 sequences, which were collected using one type of RGBD-sensor. The resolutions of the image frames are 1269×968 (color) and 640×480 (depth) with 30-Hz frame rate and all the image and ...

  24. [32]

    We used the extended versions of base scene graph generation algorithm (F-Net) [17] as baselines (see Table I)

    Algorithms: We compared the quality of the 3-D scene graph with a few baseline methods. We used the extended versions of base scene graph generation algorithm (F-Net) [17] as baselines (see Table I). The first baseline computes 2-D scene graphs for every image frame, then conca...

  25. [33]

    We recruited six experiment participants (age: 22–35, male/female: 5/1) and gathered six responses for each graph

    Evaluation Metrics: We used a human judgment met- ric to evaluate the accuracy of each method. We recruited six experiment participants (age: 22–35, male/female: 5/1) and gathered six responses for each graph. For each result- ing graph, we showed the participants, their corre...

  26. [34]

    We used Intel Core i9-7980XE CPU@2.60 GHz and Titan XP for the experiment

    Implementation Details: For the implementation of the proposed 3-D scene graph construction framework, we used Python and Pytorch for seamless integration, since the build- ing blocks of the framework had been developed in the same environment. We used Intel Core i9-7980XE CPU...

  27. [35]

    4 shows the resulting 2-D and 3-D scene graphs for the experiment sequence with the first keyframe group and Table II reports the quantitative result of the comparative study

    Results and Analysis: Fig. 4 shows the resulting 2-D and 3-D scene graphs for the experiment sequence with the first keyframe group and Table II reports the quantitative result of the comparative study. The resulting scene graph from the 2D-basic baseline was too crowded by spu...

  28. [36]

    The kitchen simulation environment models the actual kitchen environment, so that the 3-D scene graph from the actual kitchen can be directly used in the simulation

    Environment: We constructed a kitchen simulation envi- ronment to verify the applicability of the proposed 3-D scene graph. The kitchen simulation environment models the actual kitchen environment, so that the 3-D scene graph from the actual kitchen can be directly used in the...

  29. [37]

    In the first phase, human scans through the actual kitchen and a 3-D scene graph is constructed for being used in the simulation

    Scenario: The demonstration scenario consists of four phases. In the first phase, human scans through the actual kitchen and a 3-D scene graph is constructed for being used in the simulation. The generated 3-D scene graph is trans- ferred to Mybot in the simulated environment a...

  30. [38]

    5 shows the demonstration process

    Results and Analysis: Fig. 5 shows the demonstration process. In the first step, human scanned through the envi- ronment and gathered relevant information to construct a 3-D scene graph. During the process, objects in the environment were detected and the categories, positions,...

  31. [39]

    R EFERENCES

    usability; and 4) scalability. R EFERENCES

  32. [40]

    Moveit! [ROS topics],

    S. Chitta, I. Sucan, and S. Cousins, “Moveit! [ROS topics],” IEEE Robot. Autom. Mag., vol. 19, no. 1, pp. 18–19, Mar. 2012

    2012

  33. [41]

    Semantic 3D object maps for everyday manipulation in human living environments,

    R. B. Rusu, “Semantic 3D object maps for everyday manipulation in human living environments,” KI-Künstliche Intelligenz, vol. 24, no. 4, pp. 345–348, 2010

    2010

  34. [42]

    Elasticfusion: Real-time dense SLAM and light source estimation,

    T. Whelan, R. F. Salas-Moreno, B. Glocker, A. J. Davison, and S. Leutenegger, “Elasticfusion: Real-time dense SLAM and light source estimation,” Int. J. Robot. Res., vol. 35, no. 14, pp. 1697–1716, 2016

    2016

  35. [43]

    Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge,

    O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 39, no. 4, pp. 652–663, Apr. 2017

    2015

  36. [44]

    VQA: Visual question answering,

    A. Agrawal, J. Lu, S. Antol, M. Mitchell, C. L. Zitnick, D. Parikh, and D. Batra, “VQA: Visual question answering,” Int. J. Comput. Vis., vol. 123, no. 1, pp. 4–31, 2017

    2017

  37. [45]

    Ask me anything: Dynamic memory networks for natural language processing,

    A. Kumar et al., “Ask me anything: Dynamic memory networks for natural language processing,” in Proc. Int. Conf. Mach. Learn. , 2016, pp. 1378–1387

    2016

  38. [46]

    Robot task planning and explanation in open and uncertain worlds,

    M. Hanheide et al., “Robot task planning and explanation in open and uncertain worlds,” Artif. Intell., vol. 247, pp. 119–150, Jun. 2017

    2017

  39. [47]

    Bundlefusion: Real-time globally consistent 3D reconstruction using on-the-fly surface reintegration,

    A. Dai, M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt, “Bundlefusion: Real-time globally consistent 3D reconstruction using on-the-fly surface reintegration,” ACM Trans. Graph. , vol. 36, no. 4, 2017, Art. no. 76a

    2017

  40. [48]

    3D is here: Point cloud library (PCL),

    R. B. Rusu and S. Cousins, “3D is here: Point cloud library (PCL),” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), Shanghai, China, 2011, pp. 1–4

    2011

  41. [49]

    Grid-based map- ping and tracking in dynamic environments using a uniform evidential environment representation,

    G. Tanzmeister, J. Thomas, D. Wollherr, and M. Buss, “Grid-based map- ping and tracking in dynamic environments using a uniform evidential environment representation,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), May 2014, pp. 6090–6095

    2014

  42. [50]

    OctoMap: An efficient probabilistic 3D mapping framework based on octrees,

    A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, “OctoMap: An efficient probabilistic 3D mapping framework based on octrees,”Auton. Robots, vol. 34, no. 3, pp. 189–206, Apr. 2013. [Online]. Available: https://doi.org/10.1007/s10514-012-9321-0

    2013 doi

  43. [51]

    SemanticFusion: Dense 3D semantic mapping with convolutional neu- ral networks,

    J. McCormac, A. Handa, A. Davison, and S. Leutenegger, “SemanticFusion: Dense 3D semantic mapping with convolutional neu- ral networks,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2017, pp. 4628–4635

    2017

  44. [52]

    PointNet++: Deep hierarchical feature learning on point sets in a metric space,

    C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 5099–5108

    2017

  45. [53]

    Video captioning with transferred semantic attributes,

    Y . Pan, T. Yao, H. Li, and T. Mei, “Video captioning with transferred semantic attributes,” in Proc. CVPR, vol. 2, 2017, p. 3

    2017

  46. [54]

    Scene graph generation by iterative message passing,

    D. Xu, Y . Zhu, C. B. Choy, and L. Fei-Fei, “Scene graph generation by iterative message passing,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 2, 2017, pp. 3097–3106

    2017

  47. [55]

    Scene graph generation from objects, phrases and region captions,

    Y . Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang, “Scene graph generation from objects, phrases and region captions,” in Proc. ICCV, 2017, pp. 1270–1279

    2017

  48. [56]

    Factorizable net: An efficient subgraph-based framework for scene graph generation,

    Y . Li, W. Ouyang, B. Zhou, J. Shi, C. Zhang, and X. Wang, “Factorizable net: An efficient subgraph-based framework for scene graph generation,” in Proc. Eur. Conf. Comput. Vis. (ECCV) , 2018, pp. 335–351

    2018

  49. [57]

    Image retrieval using scene graphs,

    J. Johnson et al., “Image retrieval using scene graphs,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , Boston, MA, USA, 2015, pp. 3668–3678

    2015

  50. [58]

    Learning spatial knowledge for text to 3D scene generation,

    A. Chang, M. Savva, and C. D. Manning, “Learning spatial knowledge for text to 3D scene generation,” in Proc. Conf. Empirical Methods Nat. Lang. Process. (EMNLP) , 2014, pp. 2028–2038

    2014

  51. [59]

    Graph-structured representa- tions for visual question answering,

    D. Teney, L. Liu, and A. van den Hengel, “Graph-structured representa- tions for visual question answering,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1–9

    2017

  52. [60]

    Faster R-CNN: Towards real- time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real- time object detection with region proposal networks,” in Proc. Adv. Neural Inf. Process. Syst. , 2015, pp. 91–99

    2015

  53. [61]

    Distributed representations of words and phrases and their composi- tionality,

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their composi- tionality,” inProc. Adv. Neural Inf. Process. Syst., 2013, pp. 3111–3119

    2013

  54. [62]

    Efficient non-maximum suppression,

    A. Neubeck and L. Van Gool, “Efficient non-maximum suppression,” in Proc. IEEE 18th Int. Conf. Pattern Recognit. (ICPR) , vol. 3, 2006, pp. 850–855

    2006

  55. [63]

    Visual Genome: Connecting language and vision using crowdsourced dense image annotations,

    R. Krishna et al., “Visual Genome: Connecting language and vision using crowdsourced dense image annotations,” Int. J. Comput. Vis., vol. 123, no. 1, pp. 32–73, 2017

    2017

  56. [64]

    Histogram-based color image retrieval,

    S. Jeong, “Histogram-based color image retrieval,” Project Rep. Psych221/EE362, 2001

    2001

  57. [65]

    Ff: The fast-forward planning system,

    J. Hoffmann, “Ff: The fast-forward planning system,” AI Mag., vol. 22, no. 3, p. 57, 2001

    2001

  58. [66]

    PDDL2.1: An extension to PDDL for expressing temporal planning domains,

    M. Fox and D. Long, “PDDL2.1: An extension to PDDL for expressing temporal planning domains,” J. Artif. Intell. Res., vol. 20, pp. 61–124, 2003

    2003

  59. [67]

    Scannet: Richly-annotated 3D reconstructions of indoor scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3D reconstructions of indoor scenes,” in Proc. CVPR, vol. 2, 2017, p. 10

    2017

  60. [68]

    Cheap and fast— But is it good? Evaluating non-expert annotations for natural language tasks,

    R. Snow, B. O’Connor, D. Jurafsky, and A. Y . Ng, “Cheap and fast— But is it good? Evaluating non-expert annotations for natural language tasks,” in Proc. Conf. Empirical Methods Nat. Lang. Process. , 2008, pp. 254–263

    2008

  61. [69]

    Realization of task intelligence for service robots in an unstructured environment,

    D.-H. Kim, G.-M. Park, Y .-H. Yoo, S.-J. Ryu, I.-B. Jeong, and J.-H. Kim, “Realization of task intelligence for service robots in an unstructured environment,” Annu. Rev. Control, vol. 44, pp. 9–18, 2017

    2017

  62. [70]

    Fuzzy integral-based gaze control of a robotic head for human robot interaction,

    B.-S. Yoo and J.-H. Kim, “Fuzzy integral-based gaze control of a robotic head for human robot interaction,” IEEE Trans. Cybern., vol. 45, no. 9, pp. 1769–1783, Sep. 2015

    2015

  63. [71]

    Hierarchical emotional episodic memory for social human robot collaboration,

    W.-H. Lee and J.-H. Kim, “Hierarchical emotional episodic memory for social human robot collaboration,” Auton. Robots , vol. 42, no. 5, pp. 1087–1102, 2018

    2018

  64. [72]

    Webots: Symbiosis between virtual and real mobile robots,

    O. Michel, “Webots: Symbiosis between virtual and real mobile robots,” in Proc. Int. Conf. Virtual Worlds , 1998, pp. 254–263

    1998

  65. [73]

    ROS: An open-source robot operating system,

    M. Quigley et al., “ROS: An open-source robot operating system,” in Proc. ICRA Workshop Open Source Softw., vol. 3. Kobe, Japan, 2009, p. 5

    2009

  66. [74]

    Hyperface: A deep multi- task learning framework for face detection, landmark localization, pose estimation, and gender recognition,

    R. Ranjan, V . M. Patel, and R. Chellappa, “Hyperface: A deep multi- task learning framework for face detection, landmark localization, pose estimation, and gender recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 1, pp. 121–135, Jan. 2019. Ue-Hwan Kim receiv...

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.