Pith. sign in

REVIEW 4 major objections 5 minor 72 references

From Data to Modeling: Fully Open-vocabulary Scene Graph Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single transformer framework now makes scene graph generation fully open-vocabulary, and it reports state-of-the-art Recall@K on VG150.

desk verdict The core architecture is the authors' ECCV 2024 paper, and the new empirical comparison of three pre-training pipelines plus GQA results is solid; the 'first fully open-vocabulary SGG' claim is unsupported as written because cited CVPR 2024 work [50] is never compared or scoped. read the letter →

arxiv 2505.20106 v1 pith:PY7FNUI5 submitted 2025-05-26 cs.CV

Zuyao Chen , Jinlin Wu , Zhen Lei , Chang Wen Chen This is my paper · ORCID
classification cs.CV
keywords scenegraphgenerationopenvocabularyrelation-awarepre-trainingvisual-conceptalignmentknowledgedistillationvision-languagemodelsDETRweaklysupervisedlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that scene graph generation—predicting objects and their pairwise relationships in an image—can be made fully open-vocabulary: the model should recognize objects and relationships that never appeared in its training categories. To demonstrate this, the authors introduce OvSGTR, a DETR-like transformer with a frozen image backbone and text encoder whose object and relation heads are replaced by similarity scoring against text embeddings, so the vocabulary is defined by the prompt rather than a fixed classifier. They further claim that large-scale relation-aware pre-training on synthesized scene graphs—generated by a scene parser, an LLM, or a multimodal LLM—is what makes the relation side transferable. On the VG150 benchmark the paper reports state-of-the-art R@K in closed-set, object-open, relation-open, and fully open settings, plus strong fine-tuned transfer to GQA200. A sympathetic reader would care because it suggests the same model can serve arbitrary visual relation vocabularies without new manual annotation.

What carries the argument

The load-bearing object is the visual–concept alignment module plus the learned relation query. Instead of a fixed classifier, node category scores are the similarity between transformer output hidden features and text embeddings of candidate noun phrases, and edge scores are the alignment between a relation feature $e_{s_i \to o_j} = f_\theta([v_{s_i}, v_{o_j}, r])$ (a two-layer MLP over the concatenated subject node, object node, and one learned relation-query embedding) and text embeddings of candidate predicates, trained with a binary cross-entropy loss. During fine-tuning, a distillation term $L_{\text{distill}} = \frac{1}{|\mathcal{N}|}\sum_{e\in\mathcal{N}}\|e_s - e_t\|_1$ pushes the student's negative edge features toward the frozen teacher's, which is the mechanism that prevents catastrophic forgetting of novel relationships and lets relation-aware pre-training survive fine-tuning.

What would settle it

Human-annotate a random sample of, say, 1,000 relation triplets from the 644k MegaSG pre-training set: if most predicates are ungrounded or semantically wrong, the relation-aware pre-training signal is not what the paper says it is. Separately, rerun the full pipeline with a stricter deduplication threshold (feature similarity below 0.7 against VG test images) and with the distillation loss removed; if novel-relation R@K barely moves in either ablation, the claimed transfer and retention mechanisms are not load-bearing.

Watch

Extended reading notes

Core claim

The central discovery the paper advances is that a single end-to-end transformer, trained with visual–concept alignment on both nodes and edges, can predict object categories and relation predicates outside the training set, and that this capability degrades sharply for novel relations unless the model is pre-trained on relation-aware supervision and then protected from catastrophic forgetting during fine-tuning. On the fully open OvD+R setting, where both novel objects and novel relations are withheld during training, the paper reports SGDet R@50 of 17.84, against 5.87 for the prior VS3 baseline, and novel-relation R@50 of 17.15 where VS3 scores zero. The authors attribute the gains to three complementary ingredients: large-scale synthesized scene graphs from the multimodal LLM-based MegaSG pipeline (644k images), a lightweight relation head that concatenates subject, object, and a learned relation-query feature, and a knowledge distillation loss that keeps the student's edge features close to the pre-trained teacher's on negative samples so novel relations are not forgotten.

Load-bearing premise

The results stand or fall on whether the machine-generated scene graphs used for pre-training—from the scene parser, GPT-4, and Gemini—are accurate enough to teach transferable visual relationships, and on whether the filter that discards pre-training images whose learned features are too close (similarity below 0.9) to any VG test image truly removes test-set overlap.

Editorial extensions

If this is right

  • A single OvSGTR model can be evaluated in all four SGG settings—closed-set, object-open, relation-open, and fully open—without changing architecture, only the training split and prompt vocabulary.
  • Scaling relation-aware pre-training from COCO Caption (104k images) to MegaSG (644k images) improves novel-relation SGDet R@50 from 16.39 to 22.86, so synthetic scene-graph data can substitute for expensive manual relation annotation.
  • The knowledge distillation mechanism moves novel-relation R@50 in the relation-open setting from 0.10 to 13.45 (Swin-T, SGDet), identifying catastrophic forgetting as a central obstacle to open-vocabulary relations.
  • Fine-tuning on GQA200 pushes SGDet R@50 to 33.9 and R@100 to 39.2 (Swin-B), above prior closed-set baselines, indicating that the pre-trained relation representation transfers to a different label distribution.
  • Fully open OvD+R remains much harder than closed-set SGG—SGDet R@50 is 17.84 versus 37.6—so the paper frames it as the benchmark that should drive future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the vocabulary is decided by text prompts, the same trained checkpoint could in principle be queried with any noun or predicate phrase, including hierarchical or multilingual labels; the paper does not test this directly.
  • The reported zero-shot gains hinge on the quality of machine-generated scene graphs; a natural next experiment is to measure how R@K degrades as synthesized labels are corrupted or downsampled.
  • The CLIP-similarity < 0.9 filter is a blunt instrument: near-duplicate images can still exist below that threshold, so a stricter deduplication against VG test images would test whether the 644k MegaSG pre-training is truly non-leaking.
  • OvD+R results suggest a new practical target: instead of separate object detectors and relation classifiers, a single open-vocabulary scene-graph transformer could serve downstream tasks like captioning and visual question answering, which the paper does not evaluate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces OvSGTR, a DETR-style transformer for fully open-vocabulary scene graph generation (SGG). It unifies four settings — closed-set, object-open (OvD), relation-open (OvR), and fully open (OvD+R) — by aligning node and edge features with text embeddings through a frozen image backbone and a frozen text encoder. The paper studies three weakly supervised pretraining pipelines (scene-parser, LLM, and multimodal LLM) and proposes a knowledge-distillation-based visual-concept retention mechanism to combat catastrophic forgetting of novel relations. Extensive experiments on VG150 report state-of-the-art numbers across all four settings, and additional experiments on GQA200 report strong transfer results after fine-tuning.

Significance. If the results hold, the main contributions are a single architecture spanning closed- and open-vocabulary node/edge prediction, evidence that large-scale synthesized scene graphs from GPT-4 and Gemini provide transferable relation supervision, and a simple distillation scheme that mitigates forgetting of novel predicates. The paper is strong in breadth: Tables I-VIII cover many baselines and protocols, the ablation of relation queries (Fig. 4) and the lambda study (Table VII) are informative, and the code is publicly available. However, the novelty claim and several protocol choices need to be tightened before the state-of-the-art conclusion is fully supported.

major comments (4)
  1. [Section II and Introduction/Conclusion] Section II lists [50] (Li et al., CVPR 2024) as an open-vocabulary SGG method, yet the Introduction and Conclusion claim that OvSGTR is 'the first framework towards fully open-vocabulary SGG.' No experiment in the paper compares with [50], and the text never states whether [50] already handles novel relation categories. If [50] supports open-set relations, the novelty and SOTA claims are not established; if it does not, the paper must say so explicitly. As written, the central claim is unsupported, and Tables III-VI omit a directly relevant baseline.
  2. [Section IV-A and Table I footnote] The leakage filter for MegaSG pretraining data is described only as requiring that each sampled image have CLIP similarity below 0.9 with the VG test set. No analysis is given for the choice of threshold, the number of images removed, or the residual near-duplicate rate. Because the zero-shot and novel-category results in Tables I, IV, V, and VI depend on pretraining data not containing VG test images, the threshold must be validated, e.g., with a similarity histogram and manual inspection of the closest pairs. The same check should be reported for GQA200 in Table VIII, since GQA shares its image corpus with Visual Genome.
  3. [Section IV-A (PredCls protocol)] The paper states that PredCls is implemented 'by selecting image regions that best match the ground-truth objects in post-processing before performing relation recognition.' This is a nonstandard protocol: it uses ground-truth boxes to choose among detector outputs, which can inflate relation-recognition performance relative to prior work that receives the ground-truth boxes directly. Please report the standard PredCls evaluation or demonstrate that the post-processing selection is equivalent, and re-examine Tables I, III, IV, V, and VIII under that protocol.
  4. [Section III-A and Table I] The largest gains are attributed to pretraining on GPT4SGG and MegaSG, but the synthesized scene-graph annotations are not released, and no quality validation (e.g., human agreement, per-predicate accuracy, or a small manually annotated subset) is provided. Since the claimed SOTA and open-vocabulary generalization rest on these machine-generated labels, the paper should either release the data or provide sufficient statistics and a quality audit so that the results are reproducible and the risk of noisy supervision is quantified.
minor comments (5)
  1. [Section IV-A] Similar to VQ150 should read VG150.
  2. [Section II] Large Language Models (LLMs) have been emerged should be have emerged.
  3. [Equation (3)] The notation v_j for a predicted node's visual representation conflicts with the use of v_i for ground-truth nodes; use a distinct symbol such as tilde-v-sub-j.
  4. [Section IV-B(6)] The text says the relation head is only a single MLP, while Section III-B describes a two-layer MLP; make this consistent.
  5. [Figure 4 caption] The x-axis labels 0, 1, 3, 5 are not defined; state that these are the number of relation queries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SOTA claims rest on external VG150/GQA200 benchmarks and on a controlled three-way pre-training comparison; the authors' self-citations to GPT4SGG, MegaSG, and their prior ECCV paper are empirical inputs rather than derivational shortcuts.

full rationale

No circular step is present. The core empirical claims are evaluated against external benchmarks (VG150, GQA200) under standard held-out splits, and the reported state-of-the-art numbers come from training a transformer on fixed datasets, not from an equation whose output coincides with an input. The three relation-aware pre-training pipelines (scene parser, GPT4SGG, MegaSG) are compared in controlled experiments in Tables I and II; although GPT4SGG [27] and MegaSG [43] are self-authored and the framework extends the authors' ECCV paper [55], the paper does not define its prediction targets in terms of those prior outputs, and the comparisons are independently re-run rather than imported by citation. The 'novel' relation categories in the OvR-SGG and OvD+R-SGG settings are present in the pre-training corpus, but the protocol only removes them from VG150 fine-tuning, which is a standard open-vocabulary transfer setup and not a circular derivation. One non-circular caveat belongs under correctness/novelty risk: the Related Work section lists Li et al. [50] (CVPR 2024, 'From pixels to graphs') as an open-vocabulary SGG method but neither compares to it nor scopes the 'first fully open-vocabulary SGG' claim against it; this is a missing-comparison issue, not a circularity issue, and it does not change the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The free parameters are all standard hyperparameters or filtering thresholds; none is fitted to the VG test labels, but the CLIP threshold and lambda directly affect the headline numbers and lack sensitivity analyses. The axioms are the background assumptions any open-vocabulary method of this family requires: text-encoder coverage, quality of synthetic labels, and leakage filtering.

free parameters (5)
  • distillation weight lambda = 0.1
    Selected via validation sweep in Table VII; controls trade-off between fine-tuning supervision and retention of novel relations.
  • prompt word budget M = 80
    Number of positive/negative text words concatenated in the prompt; stated in Section III-B with no sensitivity analysis.
  • number of relation queries = 1
    Chosen by ablation in Fig. 4; using more queries does not improve performance.
  • CLIP similarity leakage threshold = 0.9
    Used to filter MegaSG images against VG test set; no analysis of threshold sensitivity.
  • max detected objects per image = 100
    Top 100 objects kept for pairwise relation recognition; no sensitivity analysis.
assumptions (5)
  • domain assumption Frozen BERT text encoder contains the vocabulary for all base and novel object and relation categories, so open-vocabulary recognition is possible via text matching.
    Invoked throughout Section III-C; novel categories that are not in the text encoder's wordpiece vocabulary cannot be predicted, so 'open-vocabulary' is bounded by BERT's fixed vocabulary.
  • domain assumption Grounding DINO initialization provides a strong prior for object detection and its pretraining data overlap with VG test has been adequately removed.
    Used as initialization in Section IV-A; the authors remove images used by Grounding DINO from the test set but do not establish that removal is complete.
  • ad hoc to paper Synthesized scene graphs from GPT-4, Gemini 1.5 Flash, and a language parser can serve as reliable training supervision for relationships.
    Section III-A treats these pseudo-labels as ground truth for relation-aware pre-training without human verification or quality analysis.
  • ad hoc to paper CLIP similarity below 0.9 between pre-training and VG test images is sufficient to prevent information leakage.
    Stated in the Table I footnote; near-duplicate images can score below 0.9 and no leakage check is reported.
  • domain assumption Dot-product similarity between visual features and text embeddings is a valid classifier for open-vocabulary categories.
    Visual-concept alignment in Eqs. (3)-(4) assumes the learned visual and semantic spaces are aligned enough that similarity ranks categories correctly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Data to Modeling: Fully Open-vocabulary Scene Graph Generation." pith.science (2026). https://pith.science/paper/PY7FNUI5

@misc{pith2026250520106,
  author       = {Pith},
  title        = {Pith review of: From Data to Modeling: Fully Open-vocabulary Scene Graph Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PY7FNUI5}},
  note         = {Machine review of arXiv:2505.20106}
}
read the original abstract

We present OvSGTR, a novel transformer-based framework for fully open-vocabulary scene graph generation that overcomes the limitations of traditional closed-set models. Conventional methods restrict both object and relationship recognition to a fixed vocabulary, hindering their applicability to real-world scenarios where novel concepts frequently emerge. In contrast, our approach jointly predicts objects (nodes) and their inter-relationships (edges) beyond predefined categories. OvSGTR leverages a DETR-like architecture featuring a frozen image backbone and text encoder to extract high-quality visual and semantic features, which are then fused via a transformer decoder for end-to-end scene graph prediction. To enrich the model's understanding of complex visual relations, we propose a relation-aware pre-training strategy that synthesizes scene graph annotations in a weakly supervised manner. Specifically, we investigate three pipelines--scene parser-based, LLM-based, and multimodal LLM-based--to generate transferable supervision signals with minimal manual annotation. Furthermore, we address the common issue of catastrophic forgetting in open-vocabulary settings by incorporating a visual-concept retention mechanism coupled with a knowledge distillation strategy, ensuring that the model retains rich semantic cues during fine-tuning. Extensive experiments on the VG150 benchmark demonstrate that OvSGTR achieves state-of-the-art performance across multiple settings, including closed-set, open-vocabulary object detection-based, relation-based, and fully open-vocabulary scenarios. Our results highlight the promise of large-scale relation-aware pre-training and transformer architectures for advancing scene graph generation towards more generalized and reliable visual understanding.

Figures

Figures reproduced from arXiv: 2505.20106 by the authors.

Figure 1
Figure 1. Illustration of SGG Scenarios (best view in color). Dashed nodes or edges in (a) - (d) refer to unseen category instances [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of different pipelines for relation-aware pre-training. (a) Early weakly-supervised methods [25], [26] utilize [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of our proposed OvSGTR. The proposed OvSGTR is equipped with a frozen image backbone to extract visual features, a frozen text encoder to extract text features, and a transformer for decoding scene graphs. Visual features for nodes are the output hidden features of the transformer; Visual features for edges are obtained via a light-weight relation head (i.e., with only two-layer MLP). Visual-concept alignme… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation study of relation queries on VG150 validation [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of our model (w. Swin-T) on VG150 test set (best view in color). For clarity, we only show triplets [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: t-SNE [73] visualizations of object and relation features. (a)–(b) show object features extracted by OvSGTR-B without [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 58 canonical work pages

  1. [50]

    From pixels to graphs: Open-vocabulary scene graph generation with vision-language models,

    R. Li, S. Zhang, D. Lin, K. Chen, and X. He, “From pixels to graphs: Open-vocabulary scene graph generation with vision-language models,” inIEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 28 076–28 086

    work page 2024

  2. [55]

    Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention,

    Z. Chen, J. Wu, Z. Lei, Z. Zhang, and C. W. Chen, “Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention,” inEur . Conf. Comput. Vis., 2024, pp. 108–124

    work page 2024

  3. [1]

    Scene graph generation by iterative message passing,

    D. Xu, Y . Zhu, C. B. Choy, and L. Fei-Fei, “Scene graph generation by iterative message passing,” inIEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 3097–3106

    work page 2017

  4. [2]

    Neural motifs: Scene graph parsing with global context,

    R. Zellers, M. Yatskar, S. Thomson, and Y . Choi, “Neural motifs: Scene graph parsing with global context,” inIEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 5831–5840

    work page 2018

  5. [3]

    Learning to compose dynamic tree structures for visual contexts,

    K. Tang, H. Zhang, B. Wu, W. Luo, and W. Liu, “Learning to compose dynamic tree structures for visual contexts,” inIEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 6619–6628

    work page 2019

  6. [4]

    Unbiased scene graph generation from biased training,

    K. Tang, Y . Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased scene graph generation from biased training,” inIEEE Conf. Comput. Vis. Pattern Recog., 2020, pp. 3713–3722

    work page 2020

  7. [5]

    Recovering the unbiased scene graphs from the biased ones,

    M. Chiou, H. Ding, H. Yan, C. Wang, R. Zimmermann, and J. Feng, “Recovering the unbiased scene graphs from the biased ones,” inACM Int. Conf. Multimedia, 2021, pp. 1581–1590

    work page 2021

  8. [6]

    Bipartite graph network with adaptive message passing for unbiased scene graph generation,

    R. Li, S. Zhang, B. Wan, and X. He, “Bipartite graph network with adaptive message passing for unbiased scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 11 109–11 119

    work page 2021

Show all 72 references
  1. [8]

    Graphical contrastive losses for scene graph parsing,

    J. Zhang, K. J. Shih, A. Elgammal, A. Tao, and B. Catanzaro, “Graphical contrastive losses for scene graph parsing,” inIEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 11 535–11 543

    2019

  2. [9]

    Towards open-vocabulary scene graph generation with prompt-based finetuning,

    T. He, L. Gao, J. Song, and Y . Li, “Towards open-vocabulary scene graph generation with prompt-based finetuning,” inEur . Conf. Comput. Vis., 2022, pp. 56–73

    2022

  3. [10]

    Learning to generate language-supervised and open-vocabulary scene graph using pre-trained visual-semantic space,

    Y . Zhang, Y . Pan, T. Yao, R. Huang, T. Mei, and C. W. Chen, “Learning to generate language-supervised and open-vocabulary scene graph using pre-trained visual-semantic space,” inIEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 2915–2924

    2023

  4. [11]

    Auto-encoding scene graphs for image captioning,

    X. Yang, K. Tang, H. Zhang, and J. Cai, “Auto-encoding scene graphs for image captioning,” inIEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 10 685–10 694

    2019

  5. [12]

    Say as you wish: Fine-grained control of image caption generation with abstract scene graphs,

    S. Chen, Q. Jin, P. Wang, and Q. Wu, “Say as you wish: Fine-grained control of image caption generation with abstract scene graphs,” inIEEE Conf. Comput. Vis. Pattern Recog., 2020, pp. 9959–9968

    2020

  6. [13]

    Unpaired image captioning via scene graph alignments,

    J. Gu, S. R. Joty, J. Cai, H. Zhao, X. Yang, and G. Wang, “Unpaired image captioning via scene graph alignments,” inInt. Conf. Comput. Vis., 2019, pp. 10 322–10 331

    2019

  7. [14]

    On the role of scene graphs in image captioning,

    D. Wang, D. Beck, and T. Cohn, “On the role of scene graphs in image captioning,” inLANTERN@EMNLP-IJCNLP, 2019, pp. 29–34

    2019

  8. [15]

    In defense of scene graphs for image captioning,

    K. Nguyen, S. Tripathi, B. Du, T. Guha, and T. Q. Nguyen, “In defense of scene graphs for image captioning,” inInt. Conf. Comput. Vis., 2021, pp. 1387–1396

    2021

  9. [16]

    Graph-structured representa- tions for visual question answering,

    D. Teney, L. Liu, and A. van den Hengel, “Graph-structured representa- tions for visual question answering,” inIEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 3233–3241

    2017

  10. [17]

    Lightweight visual question answering using scene graphs,

    S. V . Nuthalapati, R. Chandradevan, E. Giunchiglia, B. Li, M. Kayser, T. Lukasiewicz, and C. Yang, “Lightweight visual question answering using scene graphs,” inCIKM, 2021, pp. 3353–3357

    2021

  11. [18]

    Robotvqa - A scene-graph- and deep-learning-based visual question answering system for robot manipulation,

    F. K. Kenfack, F. A. Siddiky, F. Balint-Benczedi, and M. Beetz, “Robotvqa - A scene-graph- and deep-learning-based visual question answering system for robot manipulation,” inIROS, 2020, pp. 9667– 9674

    2020

  12. [19]

    Visual question answering over scene graph,

    S. Lee, J. Kim, Y . Oh, and J. H. Jeon, “Visual question answering over scene graph,” inProc. Int. Conf. Graph Comput., 2019, pp. 45–50

    2019

  13. [20]

    Image generation from scene graphs,

    J. Johnson, A. Gupta, and L. Fei-Fei, “Image generation from scene graphs,” inIEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 1219– 1228

    2018

  14. [21]

    Diffusion-based scene graph to image gener- ation with masked contrastive pre-training,

    L. Yang, Z. Huang, Y . Song, S. Hong, G. Li, W. Zhang, B. Cui, B. Ghanem, and M. Yang, “Diffusion-based scene graph to image gener- ation with masked contrastive pre-training,”CoRR, vol. abs/2211.11138, 2022

    2022 arXiv

  15. [22]

    Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation,”CoRR, vol. abs/2403.17846, 2024

    2024 arXiv

  16. [23]

    Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation,

    H. Yin, X. Xu, Z. Wu, J. Zhou, and J. Lu, “Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation,”CoRR, vol. abs/2410.08189, 2024

    2024 arXiv

  17. [24]

    Scenegraphloc: Cross-modal coarse visual localization on 3d scene graphs,

    Y . Miao, F. Engelmann, O. Vysotska, F. Tombari, M. Pollefeys, and D. B. Bar ´ath, “Scenegraphloc: Cross-modal coarse visual localization on 3d scene graphs,” inEur . Conf. Comput. Vis., A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, Eds., vol. 1506...

    2024

  18. [25]

    Learning to generate scene graph from natural language supervision,

    Y . Zhong, J. Shi, J. Yang, C. Xu, and Y . Li, “Learning to generate scene graph from natural language supervision,” inInt. Conf. Comput. Vis., 2021, pp. 1823–1834

    2021

  19. [26]

    Integrating object-aware and interaction-aware knowledge for weakly supervised scene graph generation,

    X. Li, L. Chen, W. Ma, Y . Yang, and J. Xiao, “Integrating object-aware and interaction-aware knowledge for weakly supervised scene graph generation,” inACM Int. Conf. Multimedia, 2022, pp. 4204–4213

    2022

  20. [27]

    GPT4SGG: Synthe- sizing scene graphs from holistic and region-specific narratives,

    Z. Chen, J. Wu, Z. Lei, Z. Zhang, and C. Chen, “GPT4SGG: Synthe- sizing scene graphs from holistic and region-specific narratives,”arXiv preprint arXiv:2312.04314, 2023

    2023 arXiv

  21. [28]

    Open-vocabulary object detection using captions,

    A. Zareian, K. D. Rosa, D. H. Hu, and S. Chang, “Open-vocabulary object detection using captions,” inIEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 14 393–14 402

    2021

  22. [29]

    Aligning bag of regions for open-vocabulary object detection,

    S. Wu, W. Zhang, S. Jin, W. Liu, and C. C. Loy, “Aligning bag of regions for open-vocabulary object detection,” inIEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 15 254–15 264

    2023

  23. [30]

    Grounded language-image pre-training,

    L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y . Zhong, L. Wang, L. Yuan, L. Zhang, J. Hwang, K. Chang, and J. Gao, “Grounded language-image pre-training,” inIEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 10 955–10 965

    2022

  24. [31]

    Regionclip: Region-based language- image pretraining,

    Y . Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y . Li, and J. Gao, “Regionclip: Region-based language- image pretraining,” inIEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 16 772–16 782

    2022

  25. [32]

    Learning to prompt for open-vocabulary object detection with vision-language model,

    Y . Du, F. Wei, Z. Zhang, M. Shi, Y . Gao, and G. Li, “Learning to prompt for open-vocabulary object detection with vision-language model,” in IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 14 064–14 073

    2022

  26. [33]

    Scene graph parser,

    J. Mao, “Scene graph parser,” https://github.com/vacancy/ SceneGraphParser, 2022

    2022

  27. [34]

    Scene graph generation from objects, phrases and region captions,

    Y . Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang, “Scene graph generation from objects, phrases and region captions,” inInt. Conf. Comput. Vis., 2017, pp. 1270–1279

    2017

  28. [35]

    Knowledge-embedded routing network for scene graph generation,

    T. Chen, W. Yu, R. Chen, and L. Lin, “Knowledge-embedded routing network for scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 6163–6171

    2019

  29. [36]

    Sgtr: End-to-end scene graph generation with transformer,

    R. Li, S. Zhang, and X. He, “Sgtr: End-to-end scene graph generation with transformer,” inIEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 19 464–19 474

    2022

  30. [37]

    Iterative scene graph generation,

    S. Khandelwal and L. Sigal, “Iterative scene graph generation,” inAdv. Neural Inform. Process. Syst., S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022

    2022

  31. [38]

    Reltr: Relation transformer for scene graph generation,

    Y . Cong, M. Y . Yang, and B. Rosenhahn, “Reltr: Relation transformer for scene graph generation,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 9, pp. 11 169–11 183, 2023

    2023

  32. [39]

    Unbiased scene graph generation via two-stage causal modeling,

    S. Sun, S. Zhi, Q. Liao, J. Heikkil ¨a, and L. Liu, “Unbiased scene graph generation via two-stage causal modeling,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 10, pp. 12 562–12 580, 2023

    2023

  33. [40]

    Fast contextual scene graph generation with unbiased context augmentation,

    T. Jin, F. Guo, Q. Meng, S. Zhu, X. Xi, W. Wang, Z. Mu, and W. Song, “Fast contextual scene graph generation with unbiased context augmentation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 6302–6311

    2023

  34. [41]

    Semantic diversity-aware prototype-based learning for unbiased scene graph generation,

    J. Jeon, K. Kim, K. Yoon, and C. Park, “Semantic diversity-aware prototype-based learning for unbiased scene graph generation,” inEur . Conf. Comput. Vis., A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, Eds., vol. 15126, 2024, pp. 379–395

    2024

  35. [42]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,”Adv. Neural Inform. Process. Syst., vol. 28, 2015

    2015

  36. [43]

    What makes a scene? scene graph-based evaluation and feedback for controllable generation,

    Z. Chen, J. Wu, Z. Lei, and C. W. Chen, “What makes a scene? scene graph-based evaluation and feedback for controllable generation,”arXiv preprint arXiv:2411.15435, 2024. JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 14

    2024 arXiv

  37. [44]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” inICML, 2021, pp. 8748–8763

    2021

  38. [45]

    Grounding DINO: marrying DINO with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang, “Grounding DINO: marrying DINO with grounded pre-training for open-set object detection,”CoRR, vol. abs/2303.05499, 2023

    2023 arXiv

  39. [46]

    Microsoft COCO captions: Data collection and evaluation server,

    X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,”CoRR, vol. abs/1504.00325, 2015

    2015 arXiv

  40. [47]

    Open-vocabulary object detection via vision and language knowledge distillation,

    X. Gu, T. Lin, W. Kuo, and Y . Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” inInt. Conf. Learn. Represent., 2022

    2022

  41. [48]

    Scaling open-vocabulary image segmentation with image-level labels,

    G. Ghiasi, X. Gu, Y . Cui, and T. Lin, “Scaling open-vocabulary image segmentation with image-level labels,” inEur . Conf. Comput. Vis., 2022, pp. 540–557

    2022

  42. [49]

    Actionclip: A new paradigm for video action recognition,

    M. Wang, J. Xing, and Y . Liu, “Actionclip: A new paradigm for video action recognition,”CoRR, vol. abs/2109.08472, 2021

    2021 arXiv

  43. [51]

    Towards open vocabulary learning: A survey,

    J. Wu, X. Li, S. Xu, H. Yuan, H. Ding, Y . Yang, X. Li, J. Zhang, Y . Tong, X. Jianget al., “Towards open vocabulary learning: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 2024

    2024

  44. [52]

    A survey on open-vocabulary detection and segmentation: Past, present, and future,

    C. Zhu and L. Chen, “A survey on open-vocabulary detection and segmentation: Past, present, and future,”CoRR, vol. abs/2307.09220, 2023

    2023 arXiv

  45. [53]

    LLM4SGG: Large language models for weakly supervised scene graph generation,

    K. Kim, K. Yoon, J. Jeon, Y . In, J. Moon, D. Kim, and C. Park, “LLM4SGG: Large language models for weakly supervised scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 28 306–28 316

    2024

  46. [54]

    GPT-4v(ision) System Card,

    OpenAI, “GPT-4v(ision) System Card,” https://openai.com/research/ gpt-4v-system-card, 2023

    2023

  47. [56]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inInt. Conf. Comput. Vis., 2021, pp. 9992–10 002

    2021

  48. [57]

    BERT: pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” inNAACL- HLT, 2019, pp. 4171–4186

    2019

  49. [58]

    Deformable DETR: deformable transformers for end-to-end object detection,

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable DETR: deformable transformers for end-to-end object detection,” inInt. Conf. Learn. Represent., 2021

    2021

  50. [59]

    Generalized intersection over union: A metric and a loss for bounding box regression,

    H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. D. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” inIEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 658–666

    2019

  51. [60]

    Focal loss for dense object detection,

    T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” inInt. Conf. Comput. Vis., 2017, pp. 2999–3007

    2017

  52. [61]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  53. [62]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieseret al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  54. [63]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering,

    D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real-world visual reasoning and compositional question answering,” inIEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 6700–6709

    2019

  55. [64]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations,

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shammaet al., “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” IJCV, vol. 123, pp. 32–73, 2017

    2017

  56. [65]

    Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation,

    X. Dong, T. Gan, X. Song, J. Wu, Y . Cheng, and L. Nie, “Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 19 427–19 436

    2022

  57. [66]

    Vision relation transformer for unbiased scene graph generation,

    G. Sudhakaran, D. S. Dhami, K. Kersting, and S. Roth, “Vision relation transformer for unbiased scene graph generation,” inInt. Conf. Comput. Vis., 2023, pp. 21 882–21 893

    2023

  58. [67]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inInt. Conf. Learn. Represent., 2019

    2019

  59. [68]

    Linguistic structures as weak supervision for visual scene graph generation,

    K. Ye and A. Kovashka, “Linguistic structures as weak supervision for visual scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 8289–8299

    2021

  60. [69]

    UNITER: universal image-text representation learning,

    Y . Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “UNITER: universal image-text representation learning,” inEur . Conf. Comput. Vis., 2020, pp. 104–120

    2020

  61. [70]

    Hl-net: Heterophily learning network for scene graph generation,

    X. Lin, C. Ding, Y . Zhan, Z. Li, and D. Tao, “Hl-net: Heterophily learning network for scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 19 454–19 463

    2022

  62. [71]

    Fully convolutional scene graph generation,

    H. Liu, N. Yan, M. S. Mortazavi, and B. Bhanu, “Fully convolutional scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 11 546–11 556

    2021

  63. [72]

    Leveraging predicate and triplet learning for scene graph generation,

    J. Li, Y . Wang, X. Guo, R. Yang, and W. Li, “Leveraging predicate and triplet learning for scene graph generation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 28 369–28 379

    2024

  64. [73]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.”Journal of machine learning research, vol. 9, no. 11, 2008

    2008

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.