REVIEW 3 major objections 5 minor 34 references
HOIverse: A Synthetic Scene Graph Dataset With Human Object Interactions
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A synthetic dataset called HOIverse claims to be the first scene graph dataset with accurate, procedurally generated ground truth for human-object interactions in indoor scenes.
desk verdict HOIverse is a genuinely new dense synthetic HOI-scene-graph dataset, but the claim of error-free ground truth is not yet backed by validation, and the paper's load-bearing weakness is exactly the one the stress-test flags. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the procedural relation-annotation pipeline. Spatial relations among objects and between humans and objects are computed from 3D geometry using volume-based overlap checks, while the human-specific relations looking at, pointing at, and body facing are extracted by casting rays from a person's head, index-finger, and spine joints and recording the ray with the smallest angular deviation from the viewing or pointing direction as a parameter. A deterministic rule set, rather than human annotators, decides every (subject, relation, object) triplet, which is exactly what underwrites the dataset's claim to complete and consistent annotations.
What would settle it
Sample a random set of HOIverse frames labeled with contact or gaze relations and check ground truth directly: does the human mesh intersect or touch the chair for sitting on, and does the head ray's hit object for looking at match a human gaze judgment? If a substantial fraction of sampled labeled relations fail such checks, the dataset's claim of accurate ground truth would be contradicted.
Extended reading notes
Core claim
The paper's claim is that complete and consistent scene graph ground truth for human-object interaction can be produced synthetically, avoiding the sparse, salient-only, and sometimes inconsistent annotations of manually labeled datasets. HOIverse populates each generated indoor scene with two to six humans in fourteen predefined interaction poses and then computes relations for every human-object and object-object pair procedurally. Relations are parametric: a predicate such as next to or looking at carries a measured distance or angle, so its meaning is fixed by geometry rather than by an annotator's judgment. The authors further claim this makes HOIverse the first synthetic scene graph dataset that describes human-object interactions with accurate ground truths for indoor 3D scenes, and they show existing scene graph generation models can be trained on it, with the best mean average precision achieved by a two-stage model that decouples detection from predicate classification.
Load-bearing premise
The dataset's usefulness rests on the assumption that geometric heuristics, such as ray casts from joints and volume-based relation checks, produce interactions that match what a human would call sitting on, looking at, touching, and so on, yet the pipeline never verifies that a placed pose actually contacts or engages its intended object.
Editorial extensions
If this is right
- Scene graph generation models trained on exhaustively annotated data cannot be judged by standard Recall@k, so the paper reports no-graph-constraint mean recall over 1000 predicates and mean average precision, with the best mAP achieved by a two-stage model that separates detection from predicate classification.
- Because relations carry parameters, an application can read an exact distance or angle from the annotation instead of retraining a classifier for each new definition of a spatial predicate.
- The egocentric camera attached to each human's head yields first-person views aligned with the person's pose, enabling evaluation of scene graph models from a first-person perspective.
- The procedural pipeline can produce effectively unlimited additional annotated frames by generating new scenes and sampling new human placements, without additional human labeling effort.
Reading between the lines
- Editorial extension: because the annotation rules are procedural, swapping a human pose or an object in a generated scene would produce paired counterfactual ground truths; models that rely on label correlations rather than genuine interaction evidence could be exposed by testing whether their predictions flip accordingly.
- Editorial extension: the ray-based definition of looking at does not model gaze as humans actually direct it, so a behavioral check against human judgments of gaze would clarify what the relation means and whether downstream robot planning should trust it.
- Editorial extension: the dataset currently centers on single-human-to-object relations; adding relations between two people, such as handing an object or pointing for another person, would be a natural next step for human-robot collaboration scenarios.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. HOIverse is a synthetic dataset for scene graph generation with human-object interactions, built by extending the CoPa-SG pipeline to Infinigen indoor scenes. Humans are placed deterministically using a set of predefined SMPL-X poses, and relations between all human-object and object-object pairs are computed procedurally as parametric relations, including three new relations (looking at, pointing at, body facing). The dataset provides RGB, depth, segmentation, human keypoints, first-person viewpoints, and negative ground truth, with roughly 16.2k images, 40k interaction annotations, and 22M scene graph relations. The paper also reports predicate-classification results for MotifNet, VCTree, and DSFormer on this dataset.
Significance. If the annotation pipeline is trustworthy, HOIverse fills a real gap: it combines scene graph generation with dense human-object interaction annotations in indoor scenes, offers procedural ground truth that avoids manual annotation inconsistency, and includes modalities (keypoints, egocentric views, negative relations) that are useful for human-robot collaboration and scene understanding. The deterministic pipeline, public dataset availability, and explicit parametric relation design are strengths. However, the central value proposition, accurate and unambiguous ground truth, is not yet demonstrated, and the evaluation protocol is too under-specified to support the benchmark conclusions. The significance is therefore conditional on validation of the annotation semantics and on making the relation definitions reproducible.
major comments (3)
- [III-B and III-D] The core claim that HOIverse provides 'accurate' and 'error-free' ground truth is not supported by any validation of the procedural annotation. The human populator places predefined poses based on free floor space and object presence, and the relation annotator assigns interactions such as 'sitting on' and 'holding' from the pose label rather than from contact/support or grasp checks. A 'sitting' human placed a few centimeters in front of a chair, or a 'looking at' ray that terminates on a wall within the 90-degree cone, would be silently annotated as correct. I recommend adding a geometric contact verification (e.g., body-mesh/object intersection or support-plane test), a human evaluation of a random sample, and an error analysis of the populator (e.g., percentage of placements with unintended penetrations or missed contacts).
- [III-A, III-D, and IV-B] The annotation definitions are not self-contained. The paper states that it follows CoPa-SG [8] for parametric relations, but [8] is under review and the supplementary PDF is not available to the reader. Load-bearing details, such as the exact ray-casting procedure, the parameter ranges for each relation, and the positive/negative thresholds (angle <= 10 degrees / >= 20 degrees, distance <= 1 m / >= 1.2 m in Section IV-B), are either inherited from an inaccessible source or introduced only in the evaluation section. Please provide the complete relation definitions, thresholds, and code in a public repository so the annotations can be reproduced and assessed.
- [IV-B and Table II] The evaluation protocol is under-specified and the benchmark numbers are not interpretable without additional detail. 'ng-mR' is introduced as 'No Graph Constraint Mean Recall@k' and is said to be computed over 1000 predicate relations, but no formula, no k value, and no averaging procedure are given; the mAP computation is likewise not described. The table reports no variance or number of runs, so the difference between MotifNet (0.504) and VCTree (0.511) mAP cannot be assessed. Please specify the metrics exactly and report mean +/- standard deviation over at least three seeds.
minor comments (5)
- [IV-B] The text mentions that models achieve '0.020 and 0.032 mR@50 and mR@100 respectively,' but these values are not shown in Table II; either include them or remove the sentence.
- [Figures] Figures 3, 4, 5, and 6 are not referenced in the body text; please add explicit references.
- [III-D] There is a typo in the looking-at description: 'everything above is not regarded aslooking at' should be 'as looking at'.
- [IV-A] Please define precisely what counts as a 'relation' and an 'interaction annotation' when reporting 40k interaction annotations and 22M scene graph annotations, since these numbers depend on whether camera-dependent duplicates and negative relations are included.
- [General] The dataset website is given, but the paper does not specify a license, versioning scheme, or download procedure; please add these details.
Circularity Check
No circular derivation: HOIverse is a procedurally generated dataset whose relation labels are defined by explicit geometric rules, not by a fitted input or a self-citation chain.
full rationale
The paper's contribution is a synthetic dataset with procedurally computed scene graph annotations, not a first-principles prediction. The relation labels are the output of stated geometric procedures: ray casts from head, finger, and spine joints, voxel-based relation computation, and parametric distance/angle thresholds (Section III-D). The benchmark trains external models on the dataset and reports their performance; it does not use those models as evidence for the correctness of the annotations. The self-citations to CoPa-SG [8] and DSFormer [11] are from the same group, and CoPa-SG is explicitly marked 'under review' in the references, which weakens independent verification of the prior parametric relation definitions. However, this is not circular in the sense required here: the new human-object relations (looking at, pointing at, body facing) are specified in the paper itself, and the dataset's accuracy claim is a construction claim about a synthetic generator, not a conclusion that depends on CoPa-SG's conclusions. Concerns that the human populator does not verify physical contact or gaze validity are correctness and validity risks for the ground truth, not circularity, since the annotations are defined as the outputs of the stated procedures. Under the hard rule requiring a quoted equation or fitted-parameter reduction, no circular step can be exhibited.
Assumptions & free parameters
free parameters (2)
- Angle threshold for positive looking/pointing/body-facing relations =
10 degrees (positive), 20 degrees (negative)
- Distance threshold for next-to relations =
1.0 m positive, 1.2 m negative
assumptions (4)
- domain assumption Infinigen procedural indoor scenes are valid stand-ins for real-world indoor environments.
- domain assumption SMPL-X human meshes with predefined poses and SMPLitex textures adequately represent humans in indoor scenes.
- domain assumption Ray casting from head, finger, and spine joints captures human attention, pointing, and body orientation.
- ad hoc to paper Parametric relation definitions from CoPa-SG are valid and reproduce across scenes.
Cite this review
Pith. "Pith review of HOIverse: A Synthetic Scene Graph Dataset With Human Object Interactions." pith.science (2026). https://pith.science/paper/OZJKCEW3
@misc{pith2026250619639,
author = {Pith},
title = {Pith review of: HOIverse: A Synthetic Scene Graph Dataset With Human Object Interactions},
year = {2026},
howpublished = {\url{https://pith.science/paper/OZJKCEW3}},
note = {Machine review of arXiv:2506.19639}
}
read the original abstract
When humans and robotic agents coexist in an environment, scene understanding becomes crucial for the agents to carry out various downstream tasks like navigation and planning. Hence, an agent must be capable of localizing and identifying actions performed by the human. Current research lacks reliable datasets for performing scene understanding within indoor environments where humans are also a part of the scene. Scene Graphs enable us to generate a structured representation of a scene or an image to perform visual scene understanding. To tackle this, we present HOIverse a synthetic dataset at the intersection of scene graph and human-object interaction, consisting of accurate and dense relationship ground truths between humans and surrounding objects along with corresponding RGB images, segmentation masks, depth images and human keypoints. We compute parametric relations between various pairs of objects and human-object pairs, resulting in an accurate and unambiguous relation definitions. In addition, we benchmark our dataset on state-of-the-art scene graph generation models to predict parametric relations and human-object interactions. Through this dataset, we aim to accelerate research in the field of scene understanding involving people.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[8]
Copa-sg: Dense scene graphs with parametric and proto-relations,
J. Lorenz, M. Phatak, R. Sch ¨on, K. Ludwig, N. H ¨ormann, A. Friedrich, and R. Lienhart, “Copa-sg: Dense scene graphs with parametric and proto-relations,” 2025, under review, see supplementary for PDF
work page 2025
-
[1]
Scene graph contrastive learning for embodied navigation,
K. P. Singh, J. Salvador, L. Weihs, and A. Kembhavi, “Scene graph contrastive learning for embodied navigation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 10 884–10 894
2023
-
[2]
Scene graph reasoning for visual question answering,
M. Hildebrandt, H. Li, R. Koner, V . Tresp, and S. G ¨unnemann, “Scene graph reasoning for visual question answering,” arXiv preprint arXiv:2007.01072, 2020
arXiv 2007
-
[3]
Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning,
Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa et al., “Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 5021–5028
2024
-
[4]
Visual relationship detection with language priors,
C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei, “Visual relationship detection with language priors,” 2016. [Online]. Available: https: //arxiv.org/abs/1608.00187
arXiv 2016
-
[5]
Visual semantic role labeling,
S. Gupta and J. Malik, “Visual semantic role labeling,” arXiv preprint arXiv:1505.04474, 2015
arXiv 2015
-
[6]
Visual genome: Connecting language and vision using crowdsourced dense image annotations,
R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei, “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” Int J Comput Vis , vol. 123, no. 1, pp. 32–73,
-
[7]
Panoptic scene graph generation,
J. Yang, Y . Z. Ang, Z. Guo, K. Zhou, W. Zhang, and Z. Liu, “Panoptic scene graph generation,” in ECCV, 2022
work page 2022
Show all 34 references
-
[9]
Neural motifs: Scene graph parsing with global context,
R. Zellers, M. Yatskar, S. Thomson, and Y . Choi, “Neural motifs: Scene graph parsing with global context,” 2018. [Online]. Available: http://arxiv.org/abs/1711.06640
2018 arXiv
-
[10]
Learning to compose dynamic tree structures for visual contexts,
K. Tang, H. Zhang, B. Wu, W. Luo, and W. Liu, “Learning to compose dynamic tree structures for visual contexts,” in Conference on Computer Vision and Pattern Recognition , 2019
2019
-
[11]
A fair ranking and new model for panoptic scene graph generation,
J. Lorenz, A. Pest, D. Kienzle, K. Ludwig, and R. Lienhart, “A fair ranking and new model for panoptic scene graph generation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.09216
2024 arXiv
-
[12]
The pascal visual object classes (voc) challenge,
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisser- man, “The pascal visual object classes (voc) challenge,” International journal of computer vision , vol. 88, pp. 303–338, 2010
2010
-
[13]
R-cnns for pose estimation and action detection. arxiv 2014,
G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik, “R-cnns for pose estimation and action detection. arxiv 2014,” arXiv preprint arXiv:1406.5212
2014 arXiv
-
[14]
2d human pose estimation: New benchmark and state of the art analysis,
M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, “2d human pose estimation: New benchmark and state of the art analysis,” in Proceedings of the IEEE Conference on computer Vision and Pattern Recognition , 2014, pp. 3686–3693
2014
-
[15]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13 . Springer,...
2014
-
[16]
Tuhoi: Trento universal human object interaction dataset,
D.-T. Le, J. Uijlings, and R. Bernardi, “Tuhoi: Trento universal human object interaction dataset,” in Proceedings of the Third Workshop on Vision and Language , 2014, pp. 17–24
2014
-
[17]
ImageNet Large Scale Visual Recognition Challenge,
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV) , vol. 115, no. 3, pp. 211–252, 2015
2015
-
[18]
Hico: A benchmark for recognizing human-object interactions in images,
Y .-W. Chao, Z. Wang, Y . He, J. Wang, and J. Deng, “Hico: A benchmark for recognizing human-object interactions in images,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1017– 1025
2015
-
[19]
The all-seeing project: Towards panoptic visual recognition and understanding of the open world,
W. Wang, M. Shi, Q. Li, W. Wang, Z. Huang, L. Xing, Z. Chen, H. Li, X. Zhu, Z. Cao et al. , “The all-seeing project: Towards panoptic visual recognition and understanding of the open world,” arXiv preprint arXiv:2308.01907, 2023
2023 arXiv
-
[20]
Haystack: A panoptic scene graph dataset to evaluate rare predicate classes,
J. Lorenz, F. Barthel, D. Kienzle, and R. Lienhart, “Haystack: A panoptic scene graph dataset to evaluate rare predicate classes,” in 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023
2023
-
[21]
Learning 3d semantic scene graphs from 3d indoor reconstructions,
J. Wald, H. Dhamo, N. Navab, and F. Tombari, “Learning 3d semantic scene graphs from 3d indoor reconstructions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3961–3970
2020
-
[22]
Vla- 3d: A dataset for 3d semantic scene understanding and navigation,
H. Zhang, N. Zantout, P. Kachana, Z. Wu, J. Zhang, and W. Wang, “Vla- 3d: A dataset for 3d semantic scene understanding and navigation,”arXiv preprint arXiv:2411.03540, 2024
2024 arXiv
-
[23]
Rio: 3d object instance re-localization in changing indoor environments,
J. Wald, A. Avetisyan, N. Navab, F. Tombari, and M. Nießner, “Rio: 3d object instance re-localization in changing indoor environments,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 7658–7667
2019
-
[24]
Infini- gen indoors: Photorealistic indoor scenes using procedural generation,
A. Raistrick, L. Mei, K. Kayan, D. Yan, Y . Zuo, B. Han, H. Wen, M. Parakh, S. Alexandropoulos, L. Lipson, Z. Ma, and J. Deng, “Infini- gen indoors: Photorealistic indoor scenes using procedural generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...
2024
-
[25]
Procthor: Large-scale embodied ai using procedural generation,
M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, and R. Mottaghi, “Procthor: Large-scale embodied ai using procedural generation,” 2022. [Online]. Available: https://arxiv.org/abs/2206.06994
2022 arXiv
-
[26]
Scenescript: Reconstructing scenes with an autoregressive structured language model,
A. Avetisyan, C. Xie, H. Howard-Jenkins, T.-Y . Yang, S. Aroudj, S. Patra, F. Zhang, D. Frost, L. Holland, C. Orme, J. Engel, E. Miller, R. Newcombe, and V . Balntas, “Scenescript: Reconstructing scenes with an autoregressive structured language model,” 2024. [Online]. Availab...
2024 arXiv
-
[27]
Scannet: Richly-annotated 3d reconstructions of indoor scenes,
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5828–5839
2017
-
[28]
Popu- lating 3D scenes by learning human-scene interaction,
M. Hassan, P. Ghosh, J. Tesch, D. Tzionas, and M. J. Black, “Popu- lating 3D scenes by learning human-scene interaction,” in Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , Jun. 2021
2021
-
[29]
Scene-aware human pose generation using transformer,
J. Yao, J. Chen, L. Niu, and B. Sheng, “Scene-aware human pose generation using transformer,” in Proceedings of the 31st ACM International Conference on Multimedia , ser. MM ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 2847–2855. [Online]. Available: ...
2023
-
[30]
AGORA: Avatars in geography optimized for regression analysis,
P. Patel, C.-H. P. Huang, J. Tesch, D. T. Hoffmann, S. Tripathi, and M. J. Black, “AGORA: Avatars in geography optimized for regression analysis,” in Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , Jun. 2021
2021
-
[31]
Expressive body capture: 3D hands, face, and body from a single image,
G. Pavlakos, V . Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black, “Expressive body capture: 3D hands, face, and body from a single image,” in Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 10 975–10 985
2019
-
[32]
SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image,
D. Casas and M. Comino-Trinidad, “SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image,” in British Machine Vision Conference (BMVC) , 2023
2023
-
[33]
Open3D: A modern library for 3D data processing,
Q.-Y . Zhou, J. Park, and V . Koltun, “Open3D: A modern library for 3D data processing,” arXiv:1801.09847, 2018
2018 arXiv
-
[2017]
Available: https://doi.org/10.1007/s11263-016-0981-7
[Online]. Available: https://doi.org/10.1007/s11263-016-0981-7
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.