Pith. sign in

REVIEW 3 major objections 5 minor 65 references

LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Depth-derived spatial instruction data raises open-vocabulary scene graph recall by 8.6% and mean recall by 28.4% over ASMv2.

desk verdict A useful new depth-informed SGG dataset, but the paper's signature spatial-reasoning claim leans on a small, partly circular validation set with unexplained ablation zeros; the PSG gain over ASMv2 is modest but external. read the letter →

arxiv 2412.06322 v1 pith:BUH2ZE5I submitted 2024-12-09 cs.CV

Mingjie Xu , Mengyang Wu , Yuzhi Zhao , Jason Chun Lok Li , Weifeng Ou This is my paper · ORCID
classification cs.CV
keywords scenegraphgenerationopen-vocabularyspatialrelationsmultimodallargelanguagemodelinstructiontuningdepthestimationlayeringPSGdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that scene graph generation, extracting (subject, predicate, object) triplets from an image, can be made open-vocabulary and spatially aware at the same time by training a multimodal large language model on synthetic data that encodes 3D layout. The authors build the SpaceSGG instruction dataset: monocular depth estimates are turned into object depth ranges, objects are assigned to near-to-far layers, and an LLM rewrites this layout with scene graph triplets into three data formats, spatial descriptions, single-turn QA, and multi-turn conversations. Training LLaVA-SpaceSGG with this mixture, they report recall of 15.43 versus 14.2 and mean recall of 13.23 versus 10.3 against the ASMv2 baseline on the PSG validation set, plus 52.48% accuracy on a spatial-relation validation set they constructed. If correct, the paper shows that explicit depth-derived spatial supervision is what an open-vocabulary SGG model needs to produce relations such as 'in front of' without regression on the open-set task.

What carries the argument

The load-bearing object is the depth-layered scene graph description produced by the SpaceSGG pipeline. A monocular depth estimator (Depth Anything) gives per-pixel depth; camera calibration converts it into a point cloud; Algorithm 1 computes each object's z-range and coverage relations; Algorithm 2 sorts objects into layers by distance and builds sub-layers for covered objects; and an LLM rewrites this structured layout into natural-language layer-by-layer descriptions. Those descriptions, paired with spatially focused QA and chain-of-thought multi-turn conversations, become instruction-tuning targets in the second stage. The mechanism works because spatial predicates are expressed not as abstract labels but as textual layer orderings, letting the MLLM reuse its language priors about 'in front of', 'behind', and 'attached to' while grounding them in geometry.

What would settle it

Re-train the identical model with the same SpaceSGG data but evaluate on an independent human-annotated spatial relation benchmark built from images the pipeline never processed; if accuracy falls to the 25% random-choice level, the depth-derived training data did not teach transferable spatial understanding.

Watch

Extended reading notes

Core claim

The central claim is that the reason existing open-vocabulary SGG models miss spatial relations is not a lack of model capacity but a lack of spatial signal in their training text: their instruction data is mostly 2D captions and common predicates. LLaVA-SpaceSGG demonstrates the fix by injecting depth-derived layering into the instruction stream. The depth map from a monocular estimator is calibrated into a point cloud, per-object depth ranges determine which objects cover or lie behind which, and objects are sorted into layers from near to far; this layered scene layout is then narrated by an LLM as SpaceSGG-Desc, SpaceSGG-QA, and SpaceSGG-Conv. On the external PSG benchmark the resulting model improves recall by 8.6% and mean recall by 28.4% over ASMv2, and on the authors' spatial benchmark it reaches 52.48%, outperforming LLaVA-1.5-13B at 45.13% and ASMv2-13B at 50.52%.

Load-bearing premise

The load-bearing premise is that the automatically generated and manually corrected spatial labels used for both training and evaluation faithfully describe true 3D layout.

Editorial extensions

If this is right

  • Open-vocabulary SGG can be improved by changing the training text, not the label space: adding 40K spatial instruction samples raises PSG recall from 14.2 to 15.43.
  • Monocular depth is sufficient to teach a model front-back and occlusion relations for SGG, avoiding the need for 3D sensors at inference time.
  • The same two-stage training paradigm, stage-1 image-level alignment followed by stage-2 SGG instruction tuning, is the recipe that transfers MLLM priors to structured scene graphs.
  • Because the SpaceSGG dataset and code are released, other models can be re-tuned on this data and compared under identical PSG evaluation.
  • Spatial-only data formats each contribute: the paper's ablations show removing any one of Desc, QA, or Conv lowers either PSG recall or spatial accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the spatial benchmark shares its construction pipeline with the training data, the 52.48% figure should be read as an upper bound on genuine spatial transfer; an independent human-annotated test set would give the sharper number.
  • The same depth-layering-to-text pipeline could be used to synthesize spatial supervision for other structured outputs, such as referring expressions, layout-based VQA, or robot instruction data, with the same caveat that model and benchmark would share biases.
  • The reported gain from depth-derived layers likely depends on the choice of monocular depth estimator, and scenes with ambiguous depth such as mirrors, transparent objects, or textureless regions are where the approach would most visibly fail.
  • If the method generalizes, a practical outcome is that open-vocabulary scene graphs with reliable 'in front of' and 'behind' relations become cheap to produce from ordinary photographs, which would directly help image retrieval and captioning systems that consume scene graphs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LLaVA-SpaceSGG, a multimodal LLM for open-vocabulary scene graph generation with enhanced spatial relations. It introduces the SpaceSGG instruction-tuning dataset, built from public datasets plus a synthetic data pipeline that combines 2D object detection, monocular depth estimation (DepthAnything), camera calibration, point-cloud layering, and LLM-generated descriptions, QA, and conversations. The model is trained with a two-stage paradigm and evaluated on the PSG validation set and on a self-constructed spatial relation validation set. The main reported results are Recall 15.43 vs. ASMv2's 14.2 and mRecall 13.23 vs. 10.3 on PSG, and 52.48% accuracy on the spatial validation set.

Significance. If the PSG results are reproducible, the paper makes a useful contribution: it demonstrates that adding depth-derived spatial instruction data can improve open-vocabulary triplet recall and mean recall for an MLLM-based SGG method, and it releases code, data, and models. The PSG evaluation is externally benchmarked and not circular. The main weakness is that the distinctive spatial-relation claim rests entirely on a self-constructed validation set generated by the same pipeline as the training data, with no error bars, no stated image disjointness, and a very small performance margin. The paper's central PSG claim is credible, but the spatial-relation contribution needs stronger independent evidence before the broader claims can be accepted.

major comments (3)
  1. [§4.1, §3.1.4] The spatial relation validation set is built by randomly selecting 30 images from COCO-Val-2017 and generating QA with the same data generation pipeline (DepthAnything, camera calibration, LLM prompting) used for SpaceSGG training data. The manuscript does not state that the 30 validation images are disjoint from the SpaceSGG training images, nor does it quantify noise in the manual correction step. With only 271 questions, the reported 52.48% vs. 50.52% difference is approximately five questions, and no error bars or significance test are provided. Because this benchmark is the sole support for the spatial-relation contribution, the spatial claim is not yet independently established. Please evaluate on an external or independently human-annotated spatial benchmark, report confidence intervals or repeated runs, and state the image disjointness explicitly.
  2. [Table 3, ab-train-1] The ablation without the SpaceSGG dataset reports 1.47% accuracy on a four-choice spatial benchmark, which is far below the 25% random-choice baseline. This strongly suggests that the free-form answer evaluation or output parsing is not robust for that model configuration, or that the benchmark is not measuring spatial ability in a controlled way. Since the same evaluation protocol produces the headline 52.48% result, this discrepancy must be explained and the evaluation protocol must be fixed (for example, with constrained decoding or manual grading).
  3. [Table 6, ab-data-9 and ab-data-10] Rows ab-data-9 and ab-data-10 report Recall=0, mRecall=0, and Accuracy=0. These values are implausible as genuine model performance and are not discussed in the text. If these runs failed to converge or the evaluation failed, they cannot support the placebo-ablation conclusion; if the numbers are real, the cause must be explained with training details. As written, these rows undermine the supplementary ablation evidence.
minor comments (5)
  1. [Figure 4] The prompt text contains a typo: 'Qeustion' should be 'Question'. Please proofread the figure text.
  2. [Algorithm 2] The algorithm title says 'Devide objects into layers'; 'Devide' should be 'Divide'. The pseudo-code also uses the phrase 'covered by' without a precise definition in the algorithm; please clarify how coverage is determined from the depth ranges and bounding boxes.
  3. [Throughout] The model name is written inconsistently as 'LLaVA-SpaceSGG' and 'LLaV A-SpaceSGG', and the baseline is sometimes 'ASMv2' and sometimes 'AS-V2'. Please standardize the notation.
  4. [§4.3.3, Table 5] The text states that results 'vary minimally' across generative models, but Table 5 shows GPT-4o giving 53.725% spatial accuracy vs. 52.48% for Llama 3, and mRecall ranging from 9.53 to 13.23. Please either provide a statistical or practical significance argument or soften the claim.
  5. [Figures 10 and 11] Some QA examples appear malformed, such as 'Is the snow in larger than the skis?' and answers that repeat the same entity names without disambiguation. If these are representative samples from the released dataset, they suggest the synthetic data may contain template errors; please clean the examples or note known artifacts.

Circularity Check

1 steps flagged · score 4.0 of 10

PSG gains are externally benchmarked and non-circular, but the distinctive spatial-relation accuracy claim is evaluated on a validation set generated by the same pipeline that produced the SpaceSGG training data, making that claim partially circular.

  1. fitted input called prediction [Section 4.1 (Spatial Relation Validation Set), cross-referencing Section 3.1.4]
    "We randomly select 30 images from COCO-Val-2017 and generate two types of questions (QA and multi-turn conversations) using the data generation pipeline in Section 3.1.4. These are manually annotated as single-choice QA with factual corrections, resulting in 271 questions."

    Section 3.1.4 is the same procedure that creates the SpaceSGG-QA and SpaceSGG-Conv training samples: it starts from scene graph descriptions, triplets, and object depth distributions, and uses LLM queries to produce front-back, up-down, occlusion, and multi-object sorting questions. The 271 validation questions are therefore not an independent spatial ground truth; they are outputs of the same depth-estimation, camera-calibration, layering, and LLM-prompting pipeline on which the model was trained. The paper says the questions are 'manually annotated as single-choice QA with factual corrections,' but this operates on pipeline-generated content and does not establish that the labels are independent of the pipeline's biases.

full rationale

The headline PSG comparison is not circular: LLaVA-SpaceSGG is evaluated on the external PSG validation set against ASMv2 and TextPSG using standard triplet recall and mean recall, so the reported 8.6% recall and 28.4% mRecall gains rest on independent annotations. No load-bearing self-citation chain is present; the cited prior works are external. However, the paper's novel spatial-relation contribution is supported by a self-constructed validation set that is generated with the exact data-generation pipeline used to create the SpaceSGG training data. Because the benchmark answers derive from DepthAnything-based depth extraction, camera calibration, layer assignment, and Llama-3 question generation, and because only 'factual corrections' are applied to those generated questions, the spatial accuracy claim is partly circular and does not yet independently establish genuine spatial reasoning. Additional warning signs, such as the below-random 1.47% accuracy for one ablation row and the absence of significance testing on a 271-question benchmark, reinforce that the spatial benchmark is not yet trustworthy; those are correctness concerns rather than circularity per se. Overall, the central open-vocabulary SGG claim remains externally grounded, so the paper is only partially circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces a dataset and a training recipe, not a new physical entity. The only 'postulated' things are the spatial layering rules (Algorithm 2) and the assumption that synthetic LLM-generated QA reflects true spatial layout. These are domain assumptions rather than free parameters; the trained network weights are fit to data, but no hand-fitted scalar constants are used in the method.

assumptions (4)
  • domain assumption PSG ground-truth scene graphs and the triplet matching protocol (IoU>0.5) are accepted as correct evaluation basis.
    Used in Section 4.1 metrics and Table 1; the central recall claim depends on these external labels.
  • domain assumption Depth-Anything monocular depth estimates are accurate enough to determine relative object depths and layer ordering.
    Section 3.1.2 uses DepthAnything to produce depth maps and z-ranges; if depth is noisy, the derived spatial relations and QA answers inherit errors.
  • domain assumption Camera calibration and 3D point cloud construction from a single RGB image preserve the true spatial layout of objects.
    Section 3.1.2 applies camera calibration to convert depth maps to point clouds; errors in intrinsics/scale propagate to the layering algorithms.
  • domain assumption LLM-generated descriptions, QA pairs, and conversations contain correct spatial facts about the image.
    Section 3.1.3 and 3.1.4 use Llama 3 70B to generate training and validation text; if the LLM hallucinates spatial relations, the model is trained on false targets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations." pith.science (2026). https://pith.science/paper/BUH2ZE5I

@misc{pith2026241206322,
  author       = {Pith},
  title        = {Pith review of: LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUH2ZE5I}},
  note         = {Machine review of arXiv:2412.06322}
}
read the original abstract

Scene Graph Generation (SGG) converts visual scenes into structured graph representations, providing deeper scene understanding for complex vision tasks. However, existing SGG models often overlook essential spatial relationships and struggle with generalization in open-vocabulary contexts. To address these limitations, we propose LLaVA-SpaceSGG, a multimodal large language model (MLLM) designed for open-vocabulary SGG with enhanced spatial relation modeling. To train it, we collect the SGG instruction-tuning dataset, named SpaceSGG. This dataset is constructed by combining publicly available datasets and synthesizing data using open-source models within our data construction pipeline. It combines object locations, object relations, and depth information, resulting in three data formats: spatial SGG description, question-answering, and conversation. To enhance the transfer of MLLMs' inherent capabilities to the SGG task, we introduce a two-stage training paradigm. Experiments show that LLaVA-SpaceSGG outperforms other open-vocabulary SGG methods, boosting recall by 8.6% and mean recall by 28.4% compared to the baseline. Our codebase, dataset, and trained models are publicly accessible on GitHub at the following URL: https://github.com/Endlinc/LLaVA-SpaceSGG.

Figures

Figures reproduced from arXiv: 2412.06322 by the authors.

Figure 1
Figure 1. The illustration of different tasks: (a) Object Detection, (b) Scene Graph Generation (SGG), and (c) Scene Graph Generation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SpaceSGG dataset construction pipeline. We utilize both SGG description and spatial relationships, where we generate 3 types [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. 3D Information Extraction: We retrieve the spatial lay [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Our proposed training paradigm and used training [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: An example of SpaceSGG-Desc, SpaceSGG-QA, and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Qualitative result of open-vocabulary SGG, particularly from traditionally state-of-arts models. Note that the cyan-coloured [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: We conduct two types of ablation studies: training [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: We conduct placebo ablation studies by testing the same [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Data Examples of SpaceSGG-Desc in SpaceSGG. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Data Examples of SpaceSGG-QA in SpaceSGG. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Data Examples of SpaceSGG-Conv in SpaceSGG. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Additional examples of LLaVA-SpaceSGG Open-Vocabulary SGG prediction compared with others on PSG validation set. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 42 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. 4, 5, 8

    work page 2024

  3. [3]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 1

    2015

  4. [4]

    Adabins: Depth estimation using adaptive bins

    Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4009–4018, 2021. 3

    work page 2021

  5. [5]

    Transformerfusion: Monocular rgb scene reconstruction using transformers

    Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. Transformerfusion: Monocular rgb scene reconstruction using transformers. Advances in Neural In- formation Processing Systems, 34:1403–1414, 2021. 3

    work page 2021

  6. [6]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 3

    arXiv 2005

  7. [7]

    Sift flow: Dense correspondence across different scenes

    LIU Ce. Sift flow: Dense correspondence across different scenes. ECCV 2008, 2008. 3

    work page 2008

  8. [8]

    Spatialvlm: Endowing vision-language mod- els with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language mod- els with spatial reasoning capabilities. arXiv preprint arXiv:2401.12168, 2024. 3

    arXiv 2024

Show all 65 references
  1. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 1

    2023 arXiv

  2. [10]

    Knowledge-embedded routing network for scene graph gen- eration

    Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embedded routing network for scene graph gen- eration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6163– 6171, 2019. 3

    2019

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 3

    2024 arXiv

  4. [12]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  5. [13]

    Spatial- rgpt: Grounded spatial reasoning in vision language model

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584, 2024. 4

    2024 arXiv

  6. [14]

    Deep- videomvs: Multi-view stereo on video with recurrent spatio- temporal fusion

    Arda Duzceker, Silvano Galliani, Christoph V ogel, Pablo Speciale, Mihai Dusmanu, and Marc Pollefeys. Deep- videomvs: Multi-view stereo on video with recurrent spatio- temporal fusion. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR...

    2021

  7. [15]

    Depth map prediction from a single image using a multi-scale deep net- work

    David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep net- work. Advances in neural information processing systems , 27, 2014. 3

    2014

  8. [16]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  9. [17]

    Recov- ering surface layout from an image

    Derek Hoiem, Alexei A Efros, and Martial Hebert. Recov- ering surface layout from an image. International Journal of Computer Vision, 75:151–172, 2007. 3

    2007

  10. [18]

    Language is not all you need: Aligning perception with language mod- els

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language mod- els. Advances in Neural Information Processing Systems , 36, 2024. 3

    2024

  11. [19]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022. 3

    2022

  12. [20]

    Image retrieval using scene graphs

    Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. InProceedings of the IEEE con- ference on computer vision and pattern recognition , pages 3668–3678, 2015. 1

    2015

  13. [21]

    Poisson surface reconstruction

    Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing , volume 7, 2006. 3

    2006

  14. [22]

    Two algorithms for constructing a delaunay triangulation

    Der-Tsai Lee and Bruce J Schachter. Two algorithms for constructing a delaunay triangulation. International Journal of Computer & Information Sciences, 9(3):219–242, 1980. 3

    1980

  15. [23]

    Depth and surface normal estimation from monocular images using regression on deep features and hi- erarchical crfs

    Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den Hengel, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hi- erarchical crfs. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1...

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 3

    2023

  17. [25]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 3

    2021

  18. [26]

    Factorizable net: an efficient subgraph-based framework for scene graph generation

    Yikang Li, Wanli Ouyang, Bolei Zhou, Jianping Shi, Chao Zhang, and Xiaogang Wang. Factorizable net: an efficient subgraph-based framework for scene graph generation. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 335–351, 2018. 1

    2018

  19. [27]

    Scene graph generation from objects, phrases and region captions

    Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xi- aogang Wang. Scene graph generation from objects, phrases and region captions. In Proceedings of the IEEE inter- national conference on computer vision , pages 1261–1270,

  20. [28]

    Stablellava: Enhanced visual instruction tun- ing with synthesized image-dialogue data

    Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yun- chao Wei. Stablellava: Enhanced visual instruction tun- ing with synthesized image-dialogue data. arXiv preprint arXiv:2308.10253, 2023. 1

    2023 arXiv

  21. [29]

    Binsformer: Revisiting adaptive bins for monocular depth estimation

    Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. IEEE Transactions on Image Processing , 2024. 3

    2024

  22. [30]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  23. [31]

    Gps-net: Graph property sensing network for scene graph generation

    Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3746– 3753, 2020. 5, 6

    2020

  24. [32]

    Visual spa- tial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spa- tial reasoning. Transactions of the Association for Computa- tional Linguistics, 11:635–651, 2023. 1

    2023

  25. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 6

    2024

  26. [34]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3, 5

    2024

  27. [35]

    Visual relationship detection with language priors

    Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationship detection with language priors. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Pro- ceedings, Part I 14, pages 852–869. Springer, 2016. 1, 2

    2016

  28. [36]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. 5

    2019

  29. [37]

    Atlas: End- to-end 3d scene reconstruction from posed images

    Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End- to-end 3d scene reconstruction from posed images. In Com- puter Vision–ECCV 2020: 16th European Conference, Glas- gow, UK, August 23–28, 2020, Proceedings, Part V...

    2020

  30. [38]

    Gpt-4o system card, August 2024

    OpenAI. Gpt-4o system card, August 2024. 8

    2024

  31. [39]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 5

    2023 arXiv

  32. [40]

    Spatial-temporal knowledge-embedded transformer for video scene graph generation

    Tao Pu, Tianshui Chen, Hefeng Wu, Yongyi Lu, and Liang Lin. Spatial-temporal knowledge-embedded transformer for video scene graph generation. IEEE Transactions on Image Processing, 2023. 1

    2023

  33. [41]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. 8

    2024

  34. [42]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  35. [43]

    Pixelwise view selection for unstructured multi-view stereo

    Johannes L Sch ¨onberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings, Part III 14 , pa...

    2016

  36. [44]

    Structured query- based image retrieval using scene graphs

    Brigit Schroeder and Subarna Tripathi. Structured query- based image retrieval using scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 178–179, 2020. 1

    2020

  37. [45]

    Nddepth: Normal-distance as- sisted monocular depth estimation

    Shuwei Shao, Zhongcai Pei, Weihai Chen, Xingming Wu, and Zhengguo Li. Nddepth: Normal-distance as- sisted monocular depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7931–7940, 2023. 3

    2023

  38. [46]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Pape...

    2018

  39. [47]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition , pages 8317–8326, 2019. 5

    2019

  40. [48]

    Neuralrecon: Real-time coherent 3d re- construction from monocular video

    Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d re- construction from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15598–15607, 2021. 3

    2021

  41. [49]

    Learning to compose dynamic tree structures for visual contexts

    Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6619–6628, 2019. 5, 6

    2019

  42. [50]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  43. [51]

    The all-seeing project v2: Towards general relation comprehension of the open world

    Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-seeing project v2: Towards general relation comprehension of the open world. arXiv preprint arXiv:2402.19474, 2024. 1, 2, 3, 4, 5, 6

    2024 arXiv

  44. [52]

    The all-seeing project: Towards panop- tic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhen- hang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panop- tic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023. 5, 6

    2023 arXiv

  45. [53]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 2, 3

    2022

  46. [54]

    Scene graph generation by iterative message passing

    Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. InPro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 5410–5419, 2017. 3, 5, 6

    2017

  47. [55]

    Panoptic scene graph gen- eration

    Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph gen- eration. In European Conference on Computer Vision, pages 178–196. Springer, 2022. 2, 3, 5, 6

    2022

  48. [56]

    Graph r-cnn for scene graph generation

    Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceed- ings of the European conference on computer vision (ECCV), pages 670–685, 2018. 3

    2018

  49. [57]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 2, 4

    2024

  50. [58]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 1, 3

    2022 arXiv

  51. [59]

    Neural motifs: Scene graph parsing with global con- text

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global con- text. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5831–5840, 2018. 5, 6

    2018

  52. [60]

    Conceptual and syntactical cross-modal alignment with cross-level consistency for image-text match- ing

    Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Shuaiqi Jing, and Jingkuan Song. Conceptual and syntactical cross-modal alignment with cross-level consistency for image-text match- ing. In Proceedings of the 29th ACM International Confer- ence on Multimedia, pages 2205–2213, 2021. 1

    2021

  53. [61]

    Textpsg: Panoptic scene graph generation from textual descriptions

    Chengyang Zhao, Yikang Shen, Zhenfang Chen, Mingyu Ding, and Chuang Gan. Textpsg: Panoptic scene graph generation from textual descriptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2839–2850, 2023. 5, 6

    2023

  54. [62]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3 The supplementary material contains:

    2023 arXiv

  55. [63]

    more ablation studies testing effectiveness of the pro- posed dataset

  56. [64]

    more examples about the SpaceSGG dataset includ- ing 3 components (SpaceSGG-Desc, SpaceSGG-QA and SpaceSGG-Conv)

  57. [65]

    house” as an example): object labels(“house

    more visual examples about our proposed LLaV A- SpaceSGG prediction compare with other models (TextPSG, ASMv2). A. More Ablation Studies To further validate the effectiveness of the proposed dataset, we replaced each element with equivalent compo- nents from the LLaV A-Instruc...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.