Time is on my sight: scene graph filtering for dynamic environment perception in an LLM-driven robot

Dimitri Ognibene; Giuseppe Boccignone; Luca Brini; Simone Colombani

REVIEW 3 major objections 4 minor 17 references

Time is on my sight: scene graph filtering for dynamic environment perception in an LLM-driven robot

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A particle filter updating a semantic scene graph keeps a robot's object positions accurate in dynamic scenes, cutting the mean absolute position error from (0.07, 0.18, 0.02) m to (0.02, 0.02, 0.01) m.

desk verdict A plausible scene-graph-plus-particle-filter integration that, as written, cannot track moved objects; the central dynamic claim needs resampling and real moving-object evidence. read the letter →

arxiv 2411.15027 v1 pith:ETAJFABL submitted 2024-11-22 cs.RO cs.AIcs.HC

Simone Colombani , Luca Brini , Dimitri Ognibene , Giuseppe Boccignone This is my paper · ORCID

classification cs.ROcs.AIcs.HC

keywords scenegraphsparticlefiltersemanticmappinglargelanguagemodelsrobottaskplanninghuman-robotinteractionRGB-Dperceptiondynamicenvironments

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a robot can keep its map of a dynamic room truthful by pairing a lightweight scene-graph generator with a particle filter, and that a large language model (LLM) planner should read that updated graph rather than a static map. This matters because natural-language commands like "pick the blue bottle on the table" produce correct robot actions only when the object positions the planner sees match the current scene. The paper's evidence is a 30-measurement comparison in which the particle filter drops the mean absolute position error from $(0.07, 0.18, 0.02)$ m to $(0.02, 0.02, 0.01)$ m. The architecture also closes a loop between perception and planning: when a task fails, the controller and explainer feed the failure back so the planner can replan against the updated semantic map.

What carries the argument

The mechanism that carries the argument is the particle filter used to track each object's 3D position inside the semantic map. A particle filter is a Monte Carlo state estimator: each object's position is represented by many weighted point hypotheses, here initialized as a Gaussian around the first observed position and predicted by applying the camera-motion transformation plus noise. The new observation comes from the scene-graph generator's object mask; the filter associates the new mask with a tracked object using the Intersection over Union (IoU) between the transformed previous mask and the current mask. Weights are updated as $w_i = 1/(1+d_i)$ where $d_i$ is the Euclidean distance between a particle's predicted position and the observed position, then normalized, and the object's estimated position is the weighted mean over particles. This turns noisy per-frame detections into a stable map coordinate, which is precisely what the planner reads when it translates a natural-language command into skills.

What would settle it

Place two visually similar objects in the same room, move one across the other's path, and record whether the IoU-based association swaps their identities; if labels swap, the scene graph will misrepresent the scene and the LLM planner will act on the wrong object. Repeat the 30-measurement protocol and compare the resulting position error with the reported $(0.02, 0.02, 0.01)$ m.

Watch

Extended reading notes

Core claim

The paper's central claim is that a robot perception module can keep a semantic scene graph current in a dynamic environment by fusing each new RGB-D frame into the graph through a particle filter, and that this continuously updated graph is what makes LLM-based planning reliable in human-inhabited spaces. The author would summarize the contribution as showing that real-time sensor data, a lightweight scene-graph generator, and a particle filter can be combined so that object positions in the semantic map track reality well enough for a planner to act on them. The supporting measurement is Table 1: over 30 readings, the mean absolute position error falls from $(0.07, 0.18, 0.02)$ m without the filter to $(0.02, 0.02, 0.01)$ m with it, and the error standard deviation falls from $(0.35, 0.24, 0.03)$ m to $(0.17, 0.12, 0.02)$ m. The paper also claims that when an action fails, the controller and explainer can return that information to the task planner, which replans against the updated graph.

Load-bearing premise

The load-bearing premise, supported only by one object over 30 measurements, is that overlap-based mask association plus inverse-distance particle weighting keeps object locations correct in general dynamic scenes without a motion model, without resampling, and without the paper reporting its IoU threshold, particle count, or initial covariance.

Editorial extensions

If this is right

A continuously updated semantic graph keeps the LLM planner's object references current, so a command like 'pick the blue bottle on the table' still names the right object after the scene has been rearranged.
Because the graph contains room nodes and object nodes, the same representation supports distinguishing same-named objects by room and applying graph-search algorithms for navigation between rooms.
The closed perception-planning loop lets the robot notice human-induced changes and replan after a failed action rather than continuing with a stale world model.
The reported inference time of about 400 ms per 480p frame for the scene-graph generator makes continuous updating feasible on modest hardware, not only on dedicated research setups.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: replacing the inverse-distance weight with a motion-aware likelihood could extend the same filter to handle brief occlusion and crossing objects, conditions the paper's 30-measurement test does not cover.
The graph could carry temporal attributes such as last-updated time or a velocity estimate per object, letting the planner prefer recently observed positions; the paper only anticipates new spatial relationships, not temporal ones.
A sensitivity analysis varying the particle count, the initial covariance, and the IoU threshold would show whether the reported error reduction is intrinsic to the method or depends on hand-set parameters that the paper does not report.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper is a sensible system integration with a load-bearing claim that, as written, does not hold up. What's new is the specific loop: PSGTR scene graph -> particle filter -> semantic map, feeding an LLM planner. I don't see that exact combination in SayPlan, DELTA, or SayNav, so the integration itself is a legitimate incremental contribution. The system description is clear, the hardware context is concrete, and the authors are honest that the planner details are deferred to a separate article.

The soft spot is not minor. Section 3.1 and Algorithm 1 specify a particle filter with no resampling, and the prediction step adds s0_i drawn from the initial-position distribution rather than zero-mean process noise. That means the particle cloud can only be dragged by camera motion; it has no mechanism to relocate toward an object that a human actually moved. So the abstract's central claim — that the system adapts to human-induced changes — is mechanistically implausible as specified. Table 1 is consistent with this: the 'real position' row is identical across all 30 measurements, so the reported error reduction is likely averaging of a static object, not dynamic tracking. No confidence intervals, no baseline, and none of the key parameters (N, initial covariance, IoU threshold, distance threshold) are given. This is a workshop-length paper, so I wouldn't demand a full benchmark, but the main claim needs at least one displaced-object trial and a proper motion model with resampling.

What the paper does well: the semantic map structure with room nodes and object nodes is practical, the choice of PSGTR is justified by inference speed, and the related work correctly situates the gap. No code or data is released, which is disappointing but normal for a workshop contribution. The citation pattern looks fair.

Bottom line: a reader interested in HRI system architectures can get a useful sketch of one way to combine scene graphs, particle filtering, and LLM planning. A reader looking for evidence that the architecture tracks dynamic scenes won't find it. I would send it to a serious referee only if the venue expects heavy revision; as is, I would not accept it. If I were the editor, I'd ask for a resampling-aware filter, moving-object experiments, and parameter disclosure before any further consideration.

Referee Report

3 major / 4 minor

Summary. The paper presents a robot control architecture in which a Large Language Model (LLM) planner receives an up-to-date semantic scene graph built by a Perception Module from RGB-D images. The Perception Module uses the PSGTR panoptic scene-graph model to extract objects, masks, and relationships, then applies a particle filter to refine the 3D position estimates of detected objects. The Planner Module is described at a high level as translating natural-language commands into skills and monitoring execution. The only quantitative evaluation is Table 1, which reports that over 30 measurements the particle filter reduces the mean absolute position error from (0.07, 0.18, 0.02) m to (0.02, 0.02, 0.01) m for a single object whose true position is constant.

Significance. The problem addressed is timely: enabling an LLM-based planner to act on a continuously updated semantic map is important for human-robot interaction in changing environments. The paper's strengths are its clear description of the scene-graph representation, the choice of a relatively lightweight scene-graph generator (PSGTR) with a motivated runtime trade-off, and the integration with a real humanoid robot platform. The particle-filter refinement is a simple, computationally cheap idea that could be useful if properly validated. However, the quantitative evidence is extremely limited, the filter as specified cannot track moving objects, and no experimental details are given for reproducibility. The claims in the abstract about adapting to human-induced changes and improving task efficiency are therefore not supported by the presented data.

major comments (3)

[Section 3.1, Algorithm 1, prediction and weight update equations] The particle filter as specified cannot track object motion. In the prediction step s_t^i = T_{t-1,t} s_{t-1}^i + s0^i, the additive term s0^i is a fixed sample from the initial distribution, not a time-varying process noise, and Algorithm 1 contains no resampling step. If an object is displaced by human action, all particles remain near the previous location; the weight update w_t^i = 1/(1 + d_t^i) only reweights the existing cloud, so the weighted mean cannot converge to the new observation. This directly contradicts the abstract's claim that the architecture adapts to human-induced changes. To support the dynamic tracking claim, the filter would need an object-motion model (e.g., a random-walk or constant-velocity model), a resampling step, and an experiment with a moving object.
[Table 1 and experimental setup] The quantitative evidence is a single table reporting 30 measurements of one object whose 'Real position' is identical in every row (0.67, 0.10, 0.95). This shows improvement on a static object, not on a dynamic scene. The table does not report the number of particles N, the initial covariance Sigma0, the IoU threshold lambda_IoU, the distance filter threshold, or the relationship probability threshold, so the result is not reproducible. Since the prediction step does not include any object-motion noise, the filter effectively averages repeated observations of a static object, which can trivially reduce mean absolute error. The authors should provide a dynamic test (e.g., an object displaced during the measurement sequence) and report the full experimental protocol, including confidence intervals or statistical tests.
[Section 3, Planner Module and overall evaluation completeness] The paper's title and abstract claim an 'LLM-driven robot' with dynamic environment perception, but the Planner Module is described in a single paragraph and the authors state that 'details on the planner will be provided in a separate article'. No task-planning or execution experiment is reported, so the system-level claims about improved adaptability, task efficiency, and human-robot collaboration are unsupported by any evaluation. If the intended contribution is limited to the Perception Module, the framing of the paper should be narrowed accordingly; otherwise, at least one integrated planning experiment is needed to substantiate the central claims.

minor comments (4)

[Table 1] There is a typo in the error standard deviation row: '0.12. 0.02' should read '0.12, 0.02'.
[Section 3.1, step 3] The abbreviation PSGTR is inconsistently written as 'PSTGR' in the sentence 'The Perception Module uses the result of PSTGR and builds the semantic map'.
[Author footer] The author email in the footer contains a garbled fragment 'envel⌢pe-⌢pensimone...'; this should be corrected to a standard email address.
[Algorithm 1] Line 3 applies the transformation to masks but is placed inside a loop 'for each object k'; clarify whether the loop is over masks or objects, and specify how the transformed mask is associated with an object.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the particle-filter evaluation uses an independent ground-truth position, and no load-bearing claim reduces to its inputs by construction.

full rationale

The paper reports a perception architecture and evaluates its particle-filter position estimate against a fixed ground-truth 'Real position' of (0.67, 0.10, 0.95) m. The particle filter's update rule uses the current observation s_new as an input and produces a weighted mean, while the reported error is computed against that independent ground truth, not against any value fitted to produce the table. No parameter in the particle filter is calibrated on the 30 measurements to minimize the reported error, so the improvement in Table 1 is an empirical evaluation rather than a fitted input renamed as a prediction. The author self-citations appear only in contextual remarks about intention prediction, action recognition, and action-word grounding; none of them supplies a load-bearing premise or a uniqueness theorem that forces the architecture's choices. The absence of an object-motion model, the lack of resampling, and the single static-object test are validity limitations that belong in a correctness assessment, not evidence of circularity. Since no equation or argument reduces the claimed result to its own inputs, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The architecture reuses existing models and filters without postulating new physical entities. Its main reliance is on the untested reliability of PSGTR, the accuracy of pose estimates, and the validity of the hand-designed particle filter update, all of which are domain assumptions rather than derived results.

free parameters (5)

Number of particles N
Algorithm 1 iterates over N particles per object but N is never specified.
Initial covariance matrix Sigma_0
Particle initialization uses Sigma_0 = diag(sigma_x^2, sigma_y^2, sigma_z^2) but no values are given.
IoU threshold lambda_IoU
Algorithm 1 uses an IoU threshold to associate masks, but the value is not reported.
Distance filter threshold
Objects beyond a distance are pruned to avoid tracking issues, but the threshold is not specified.
Relationship probability threshold
Edges are created only when the relationship probability exceeds a threshold, which is not given.

assumptions (4)

domain assumption PSGTR provides sufficiently accurate panoptic segmentation and relationship inference for the semantic map
The Perception Module relies on PSGTR model outputs as the source of objects and relationships (Section 3.1).
domain assumption Camera pose and robot odometry are known accurately enough to transform object positions into map coordinates
Graph construction assumes accurate robot poses relative to the geometric map (Section 3.1, steps 2 and 4a).
domain assumption IoU between transformed previous masks and current masks reliably identifies the same object across frames
Object association is performed via IoU thresholding without validation in occluded or cluttered scenes (Algorithm 1).
domain assumption The world is static except for camera motion during prediction; object motion is not modeled
The particle prediction step applies only the camera transformation T_{t-1,t} and does not model object dynamics, despite the claim of handling human-induced changes (Section 3.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Time is on my sight: scene graph filtering for dynamic environment perception in an LLM-driven robot." pith.science (2026). https://pith.science/paper/ETAJFABL

@misc{pith2026241115027,
  author       = {Pith},
  title        = {Pith review of: Time is on my sight: scene graph filtering for dynamic environment perception in an LLM-driven robot},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ETAJFABL}},
  note         = {Machine review of arXiv:2411.15027}
}

read the original abstract

Robots are increasingly being used in dynamic environments like workplaces, hospitals, and homes. As a result, interactions with robots must be simple and intuitive, with robots perception adapting efficiently to human-induced changes. This paper presents a robot control architecture that addresses key challenges in human-robot interaction, with a particular focus on the dynamic creation and continuous update of the robot state representation. The architecture uses Large Language Models to integrate diverse information sources, including natural language commands, robotic skills representation, real-time dynamic semantic mapping of the perceived scene. This enables flexible and adaptive robotic behavior in complex, dynamic environments. Traditional robotic systems often rely on static, pre-programmed instructions and settings, limiting their adaptability to dynamic environments and real-time collaboration. In contrast, this architecture uses LLMs to interpret complex, high-level instructions and generate actionable plans that enhance human-robot collaboration. At its core, the system Perception Module generates and continuously updates a semantic scene graph using RGB-D sensor data, providing a detailed and structured representation of the environment. A particle filter is employed to ensure accurate object localization in dynamic, real-world settings. The Planner Module leverages this up-to-date semantic map to break down high-level tasks into sub-tasks and link them to robotic skills such as navigation, object manipulation (e.g., PICK and PLACE), and movement (e.g., GOTO). By combining real-time perception, state tracking, and LLM-driven communication and task planning, the architecture enhances adaptability, task efficiency, and human-robot collaboration in dynamic environments.

Figures

Figures reproduced from arXiv: 2411.15027 by the authors.

**Figure 1.** The figure on the left showcases an example of a semantic map in an office environment, while the image on the right shows RoBee, the humanoid robot developed by Oversonic Robotics. 2. Reading robot poses: To generate the scene and semantic map, it is necessary to know the robot’s position relative to the geometric map, the camera’s position relative to the map, and the camera’s mounting position on the robot. 3. In… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

[1]

Di Napoli, G

C. Di Napoli, G. Ercolano, S. Rossi, Personalized home-care support for the elderly: a field experience with a social robot at home, User Modeling and User-Adapted Interaction 33 (2023) 405–440

work page 2023
[2]

Lucignano, F

L. Lucignano, F. Cutugno, S. Rossi, A. Finzi, A dialogue system for multimodal human-robot inter- action, in: Proceedings of the 15th ACM on International conference on multimodal interaction, 2013, pp. 197–204

work page 2013
[3]

Ognibene, L

D. Ognibene, L. Mirante, L. Marchegiani, Proactive intention recognition for joint human-robot search and rescue missions through monte-carlo planning in pomdp environments, in: Social Robotics: 11th International Conference, ICSR 2019, Madrid, Spain, November 26–29, 2019, Pro- ceedings 11, Springer, 2019, pp. 332–343

work page 2019
[4]

Galindo, J.-A

C. Galindo, J.-A. Fernández-Madrigal, J. González, A. Saffiotti, Robot task planning using semantic maps, Robotics and autonomous systems 56 (2008) 955–966

work page 2008
[5]

G. Zhu, L. Zhang, Y. Jiang, Y. Dang, H. Hou, P. Shen, M. Feng, X. Zhao, Q. Miao, S. A. A. Shah, et al., Scene graph generation: A comprehensive survey, arXiv e-prints (2022) arXiv–2201

2022
[6]

Armeni, Z.-Y

I. Armeni, Z.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, S. Savarese, 3d scene graph: A structure for unified semantics, 3d space, and camera, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5664–5673

2019
[7]

Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al., Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning, in: 2024 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 5021–5028

work page 2024
[8]

Chang, K

H. Chang, K. Boyalakuntla, S. Lu, S. Cai, E. Jing, S. Keskar, S. Geng, A. Abbas, L. Zhou, K. Bekris, et al., Context-aware entity grounding with open-vocabulary 3d scene graphs, arXiv preprint arXiv:2309.15940 (2023)

arXiv 2023

Show all 17 references

[9]

K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, N. Suenderhauf, Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning, in: 7th Annual Conference on Robot Learning, 2023

2023
[10]

Y. Liu, L. Palmieri, S. Koch, I. Georgievski, M. Aiello, Delta: Decomposed efficient long-term robot task planning using large language models, arXiv e-prints (2024) arXiv–2404

2024
[11]

Rajvanshi, K

A. Rajvanshi, K. Sikka, X. Lin, B. Lee, H.-P. Chiu, A. Velasquez, Saynav: Grounding large lan- guage models for dynamic planning to navigation in new environments, in: Proceedings of the International Conference on Automated Planning and Scheduling, volume 34, 2024, pp. 464–474

2024
[12]

J. Yang, Y. Z. Ang, Z. Guo, K. Zhou, W. Zhang, Z. Liu, Panoptic scene graph generation, in: European Conference on Computer Vision, Springer, 2022, pp. 178–196

2022
[13]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, At- tention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY, USA, 201...

2017
[14]

Ognibene, E

D. Ognibene, E. Chinellato, M. Sarabia, Y. Demiris, Contextual action recognition and target local- ization with an active allocation of attention on a humanoid robot, Bioinspiration & biomimetics 8 (2013) 035002

2013
[15]

Rossi, M

S. Rossi, M. Staffa, L. Bove, R. Capasso, G. Ercolano, User’s personality and activity influence on hri comfortable distances, in: Social Robotics: 9th International Conference, ICSR 2017, Tsukuba, Japan, November 22-24, 2017, Proceedings 9, Springer, 2017, pp. 167–177

2017
[16]

Marocco, A

D. Marocco, A. Cangelosi, K. Fischer, T. Belpaeme, Grounding action words in the sensorimotor interaction with the world: experiments with a simulated icub humanoid robot, Frontiers in neurorobotics 4 (2010) 1308

2010
[17]

Online Resources More information about RoBee and Oversonic Robotics are available: • RoBee, • Oversonic Robotics

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Di Napoli, G

C. Di Napoli, G. Ercolano, S. Rossi, Personalized home-care support for the elderly: a field experience with a social robot at home, User Modeling and User-Adapted Interaction 33 (2023) 405–440

work page 2023

[2] [2]

Lucignano, F

L. Lucignano, F. Cutugno, S. Rossi, A. Finzi, A dialogue system for multimodal human-robot inter- action, in: Proceedings of the 15th ACM on International conference on multimodal interaction, 2013, pp. 197–204

work page 2013

[3] [3]

Ognibene, L

D. Ognibene, L. Mirante, L. Marchegiani, Proactive intention recognition for joint human-robot search and rescue missions through monte-carlo planning in pomdp environments, in: Social Robotics: 11th International Conference, ICSR 2019, Madrid, Spain, November 26–29, 2019, Pro- ceedings 11, Springer, 2019, pp. 332–343

work page 2019

[4] [4]

Galindo, J.-A

C. Galindo, J.-A. Fernández-Madrigal, J. González, A. Saffiotti, Robot task planning using semantic maps, Robotics and autonomous systems 56 (2008) 955–966

work page 2008

[5] [5]

G. Zhu, L. Zhang, Y. Jiang, Y. Dang, H. Hou, P. Shen, M. Feng, X. Zhao, Q. Miao, S. A. A. Shah, et al., Scene graph generation: A comprehensive survey, arXiv e-prints (2022) arXiv–2201

2022

[6] [6]

Armeni, Z.-Y

I. Armeni, Z.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, S. Savarese, 3d scene graph: A structure for unified semantics, 3d space, and camera, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5664–5673

2019

[7] [7]

Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al., Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning, in: 2024 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 5021–5028

work page 2024

[8] [8]

Chang, K

H. Chang, K. Boyalakuntla, S. Lu, S. Cai, E. Jing, S. Keskar, S. Geng, A. Abbas, L. Zhou, K. Bekris, et al., Context-aware entity grounding with open-vocabulary 3d scene graphs, arXiv preprint arXiv:2309.15940 (2023)

arXiv 2023