Pith. sign in

REVIEW 4 major objections 5 minor 136 references

Synthetic Visual Genome

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ROBIN-3B, trained on fewer than 3 million synthetic scene-graph instances, outperforms same-size models trained on over 300 million instances and surpasses models up to 13B on relationship understanding and referring expression…

desk verdict The data pipeline is a real contribution, but Table 2's headline comparison is contaminated by training/benchmark overlap. read the letter →

arxiv 2506.07643 v1 pith:GWAXIDEL submitted 2025-06-09 cs.CV

Jae Sung Park , Zixian Ma , Linjie Li , Chenhao Zheng , Cheng-Yu Hsieh , Ximing Lu , Khyathi Chandu , Quan Kong
show 4 more authors
Norimasa Kobori Ali Farhadi Yejin Choi Ranjay Krishna
This is my paper
classification cs.CV
keywords scenegraphgenerationvisualrelationshipunderstandingsyntheticdatainstructiontuningmultimodallanguagemodelself-distillationreferringexpressioncomprehensionfiltering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that dense, machine-generated scene graphs—not ever-larger datasets—are the missing training signal for visual relationship reasoning. It builds SVG, a synthetic dataset of 146K images with 5.6M relationships over 2.6M objects, by having a teacher multimodal model fill in missing relations in existing human-annotated scene graphs and then filtering the results. Training a 3B mask-aware model, ROBIN, on this data yields better relationship understanding than same-size models trained on 300M+ instances, and the best referring-expression score (88.9 average) among models up to 13B. A second stage, SG-EDIT, has ROBIN generate scene graphs and a frontier model refine them, and each stage adds consistent benchmark gains. If the claim holds, synthetic relational annotation becomes a cheap, scalable substitute for massive human-curated relational data.

What carries the argument

The load-bearing object is the two-stage synthetic scene-graph data engine, with filtering at each stage. Stage 1 uses human-annotated seed regions (COCO/LVIS detections, VG/GQA scene graphs, region captions, depth maps) and prompts a teacher model to complete missing relationships in five categories; spatial relations survive seven geometric rules, and non-spatial relations survive only if both a LLaVA-based and a Qwen-VL-based binary VQA check answer 'Yes'. Stage 2 is a self-distillation loop in which ROBIN generates candidate dense scene graphs from segmentation masks, GPT-4o edits them by removing unlikely relations and adding relevant ones, and the edited graphs become the next training set. The model's architecture represents each region by both pixel-level masks and normalized text coordinates, which lets one forward pass handle up to 99 regions per image for dense relation generation.

What would settle it

Have human annotators label a larger random sample (say 5,000) of relations that the two-model VQA filter keeps versus discards, and compare per-type precision; the dense-scene-graph claim weakens if kept relations are not clearly more accurate than discarded ones, or if gains on relationship benchmarks vanish when the model is trained on an equal-size random subset of unfiltered relations.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a 3B multimodal model can reach or exceed the relationship reasoning of much larger models when it is instruction-tuned on dense, synthetic scene graphs rather than on hundreds of millions of web-scale image-text pairs. Starting from 33K seed images whose regions are grounded by human labels and segmentation masks, the pipeline asks GPT-4V to list at least five relationship categories per subject, producing SVG-RELATIONS with roughly five relations per region; then ROBIN is trained and used to generate scene graphs for ADE20K, PSG, and VG images, which GPT-4o edits before a second training pass. The final ROBIN-3B reports 61.6 GQA accuracy, 77.6 MMBench, 90.1 SugarCrepe replace-relation, 86.2 What's Up controlled, 88.8 average RefCOCO/+/g, and 21.0 R@20 on PSG scene graph generation, surpassing same-size baselines such as Phi-3-Vision and BLIP-3 and larger models such as ASM-V2-13B. The paper's own ablations attribute the gains to adding scene-graph data to the instruction mixture and to GPT-4o-edited self-distillation.

Load-bearing premise

The pipeline assumes the two-model 'Yes/Yes' VQA filter actually removes bad relationships, but on the paper's own 600-example human evaluation that filter's balanced accuracy of 0.6165 barely beats the 0.5484 of simply always answering 'Yes', so if this filter mostly keeps easy relations and throws out hard but valid ones, the reported gains could come from an easier training distribution rather than dense scene graphs.

Editorial extensions

If this is right

  • If ROBIN-3B's results are taken at face value, dense synthetic scene graphs can replace hundreds of millions of web-scale training instances for relationship reasoning.
  • Scene-graph instruction tuning transfers to grounding and region recognition: the same model improves RefCOCO/+/g, ADE20K, LVIS, and PACO scores, so relational data helps beyond relation-specific benchmarks.
  • The SG-EDIT loop shows a model's own generations, when edited by a stronger teacher, are a reusable data source; training on edited graphs beats training on raw generations on every relationship benchmark.
  • A mask-aware 3B model can produce open-ended panoptic scene graphs competitive with closed-set models fine-tuned on the target dataset, which suggests general-purpose MLMs can absorb scene graph generation as a standard task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would not yet generalize the data-efficiency claim to images far outside the COCO-style seed distribution; the paper itself lists in-the-wild evaluation as future work, and all seed images come from datasets with human annotations.
  • The VQA filter's balanced accuracy on the paper's own 600-example human audit (0.6165 vs. 0.5484 for majority voting) is thin support for the claim that filtering is what makes the data high quality; a larger human audit of kept versus discarded relations would separate true filtering gains from an easier training distribution.
  • An implication the authors leave implicit is that the same five-relation taxonomy could be carried into video or 3D scene graphs; if the efficiency result is real, adding scene-graph objectives to other mid-size MLMs is a natural next test.
  • The referring-expression gains may be inflated by RefCOCO-family grounding data in the training mixture; an evaluation on referring expressions from a held-out source would show how much the scene-graph data alone contributes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SVG (Synthetic Visual Genome), a pipeline for creating dense scene graph annotations at scale: Stage 1 uses GPT-4V to complete missing relationships for selected objects in seed images with human annotations, followed by rule-based and model-based filtering; Stage 2 describes SG-EDIT, a self-distillation framework in which ROBIN generates scene graphs that GPT-4o edits, and these edited graphs are used for further training. The authors train ROBIN-3B, a Qwen2.5-3B-based multimodal model, on about 1.73M Stage 1 instances and 1.23M Stage 2 instances, then report results on relationship understanding benchmarks, referring expression comprehension, region recognition, and panoptic scene graph generation. The central claim is that training on refined dense scene graphs yields strong relationship understanding with far fewer training instances than larger-scale baselines.

Significance. If the central claim holds, the paper makes a valuable contribution: dense synthetic scene graphs could be an efficient training signal for multimodal relationship reasoning, and the proposed filtering and self-distillation pipeline is a concrete recipe for generating such data at scale. The paper also includes useful ablations (e.g., the role of scene graph data in Stage 1, shown in Table 6, and the effect of GPT-4 editing in Figure 4) and the authors state that data, code, and checkpoints will be released. However, the headline evaluation is materially weakened by training/benchmark overlap for VSR, GQA, and PSG, and by filter selection on the same SugarCREPE/CREPE benchmarks that are then reported as results. The remaining clean evidence (e.g., What's Up?, MMBench, and the VSR-excluded ablation) still suggests some benefit, but the paper's strongest claims of cross-model superiority on relationship understanding are not yet established.

major comments (4)
  1. [Table 2; Tables 7 and 8] Table 2 reports VSR ZS-test and GQA Dev-EN as headline results, but Tables 7 and 8 show that VSR, GQA, and GQA-CoT are included in the Stage 1 and Stage 2 visual instruction VQA training mixtures. The reported scores (76.4 on VSR, 61.6 on GQA) therefore cannot support the claim that ROBIN-3B outperforms models trained on over 300M instances on relationship understanding. The paper itself acknowledges the overlap in Sec. 4.5, where excluding VSR from training lowers Stage 1 VSR to 69.7 (Table 6, last row), which is below BLIP-3's 72.6. Please re-report Table 2 using only genuinely held-out benchmarks, or clearly label the in-distribution columns and remove them from the headline comparison.
  2. [Appendix E.2; Table 2] The model-based filtering method is selected by evaluating on SugarCREPE and CREPE (Table 17), yet CRPE and SugarCrepe are then reported as benchmark results in Table 2. This is a selection-on-test-set loop: the reported gains on these benchmarks are not independent evidence for the filtering method. The human evaluation in Table 18 shows only a modest advantage for the chosen LLaVA+Qwen filter (balanced accuracy 0.6165 vs. 0.5484 for majority voting), which is a weak validation for a filter that determines the quality of the entire SVG dataset. Please validate the filter on a held-out set that is not subsequently reported as a benchmark, and either remove or explicitly relabel the CRPE/SugarCrepe columns as in-domain diagnostics.
  3. [Table 5; Tables 7 and 8] Table 5 reports panoptic scene graph generation on PSG, but PSG appears as scene graph detection data in both Stage 1 (Table 7) and Stage 2 (Table 8), and Stage 2 uses PSG as a source for SVG-SG generation. The comparison with closed-set models that are fine-tuned on PSG is therefore not a fair open-ended comparison: ROBIN has also been trained on PSG scene graph annotations. Please either evaluate on a held-out scene graph benchmark, or explicitly state the training overlap and avoid claiming superiority over PSG-fine-tuned models in an open-ended setting.
  4. [Table 4; Sec. 2] Table 4 reports ADE20K open-vocabulary segmentation results, but Sec. 2 states that SVG-SG includes 25K ADE20K images used in Stage 2 training. The ADE20K scores are therefore obtained on images whose scene graph annotations were part of the training data, so the comparison against Osprey-7B on ADE20K is not an out-of-distribution evaluation. Please either re-evaluate on a held-out split or clearly identify the ADE20K result as in-domain.
minor comments (5)
  1. [Sec. 3.2; Table 8] The text in Sec. 3.2 says Stage 2 training uses 1.23M samples, while the caption of Table 8 says 1.19M instances; please reconcile these numbers.
  2. [Sec. 4.1] The text refers to the 'CPRE dataset' but the benchmark is CRPE; this typo appears in the first paragraph of Sec. 4.1.
  3. [Table 2] The header of the CRPE column cites reference [81] (ASM-V2), but the benchmark described in Sec. 4.1 and Appendix E.2 is CREPE from reference [56]; please correct the citation.
  4. [Appendix E.1] The phrase 'identify at lest K = 5 subjects' should read 'at least K = 5 subjects'.
  5. [Appendix E.2] The model name is written as 'LLaVa-v1.6-vicuna-13b', while elsewhere it appears as 'LLaVA'; please use a consistent notation.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline relationship-benchmark comparisons are partly in-distribution: VSR and GQA appear in ROBIN's training mixtures, and the filtering pipeline is tuned on SugarCrepe/CRPE before those same benchmarks are reported as gains.

  1. fitted input called prediction [Section 3.2 (Training stages), Tables 7-8 (Stage 1/2 dataset mixtures), Table 2 (relationship benchmarks), Section 4.5 (ablation note)]
    "Visual Instruction VQA 430K VQAv2 [21], GQA[28], GQA-CoT [12, 28], VSR [50], OKVQA [58] (Table 7); Table 2 reports VSR[50] ZS-test and GQA Dev-EN; Section 4.5: "We excluded VSR from the Visual Instruction (Vis-Ins) data to avoid train-test overlap and ensure fair zero-shot evaluation.""

    ROBIN's Stage 1 and Stage 2 mixtures include VSR and GQA (and GQA-CoT) as Visual Instruction VQA training tasks. Table 2 then reports VSR ZS-test and GQA Dev-EN as evidence that ROBIN outperforms models trained on over 300M instances on relationship understanding. Because these exact benchmarks were training objectives, the reported scores are in-distribution fittings, not zero-shot predictions; the ZS-test label is contradicted by the paper's own Tables 7 and 8. The paper concedes the overlap in Section 4.5 by removing VSR for its ablation, where Stage-1 VSR drops from 73.7 (Table 2) to 69.7 (Table 6), below BLIP-3's 72.6. Thus the headline cross-model superiority on VSR/GQA is at least partly forced by training-set membership rather than by dense scene graphs.

  2. fitted input called prediction [Appendix E.2 (Data Filtering Details), Tables 17-18; Table 2 columns CRPE Relation and SugarCrepe Relation; Section 4.1]
    "We compare these models' performance on (1) existing image-text evaluation datasets that require relation understanding such as SugarCREPE [27] and CREPE [56] ... Through these experiments, we find it most optimal to use both LLaVa-v1.6-vicuna-13b and Qwen-VL-Chat and filter out relationships where either model answers 'No'."

    The filtering pipeline is selected by ranking candidate model-based filters on SugarCREPE and CREPE (Table 17), including the exact choice of the two teacher models and their AND-combination. The same two benchmarks appear as headline evidence in Table 2 (SugarCrepe Relation, CRPE Relation) and in the Stage1-vs-Stage2 improvement argument (Section 4.1: gains of 68.2 vs 65.9 on CRPE, 90.1 vs 89.1 on SugarCrepe). Reporting performance on the objective used to tune the filter double-counts the selection signal; those columns are not independent confirmation. The 600-example human evaluation (Table 18) is separate but weak (0.6165 balanced accuracy), so it does not fully de-circularize the filter choice.

full rationale

The central derivation is not entirely circular: the paper trains a VSR-excluded ablation (Table 6), reports uncontaminated benchmarks (MMBench, SeedBench, What's Up, RefCOCO/RefCOCO+/RefCOCOg, region classification on LVIS/PACO), and provides code/data release and a real human-evaluation set for the filter. These give independent content that keeps the score below 8. However, the headline claim of superiority on relationship understanding rests in part on VSR and GQA, which are literally in ROBIN's Stage 1 and Stage 2 training tables, and on SugarCrepe/CRPE, which were used to select the filtering rule. The paper's own ablation concedes the VSR overlap, and the clean VSR score (69.7) no longer beats BLIP-3 (72.6). This is a partial circularity: some reported 'predictions' reduce to training-set membership or to the filter-tuning objective. I do not count the weak balanced accuracy of the human filter validation as circularity; that is a data-quality risk. I also do not treat the self-citations to SugarCrepe and CREPE as load-bearing by themselves; they are public benchmarks, but their use as both tuning and evaluation targets is what creates the circular step.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical entities or mediators are introduced. SVG (dataset) and SG-EDIT (pipeline) are artifacts of the method, not postulated entities with independent falsifiable handles, so the invented-entities list is empty.

free parameters (6)
  • IoU threshold for region retention = 0.5
    Regions are kept only if IoU with SAM/Semantic-SAM masks exceeds 0.5 (Sec 2, Appendix E.1); this choice controls which objects appear in SVG and affects downstream training.
  • Minimum number of subjects per scene K = 5
    GPT-4V is asked to identify at least 5 subjects per image (Appendix E.1); this determines scene graph density.
  • Maximum relationships per object = 20
    Scene graph data uses a maximum of 20 relations per object (Appendix C, Table 7 note); this caps density and could truncate dense annotations.
  • Spatial rule definitions and overlap interpretation = above, below, left, right, overlap variants
    Seven hand-written rules define 22 spatial relations using bounding box centers (Listing 1); the 95.1% agreement with VG is on VG examples, not a guarantee for GPT-4V outputs.
  • NMS IoU threshold for proposals = 0.6
    SAM and Semantic-SAM union is non-maximum suppressed at IoU 0.6 (Appendix E.3); this changes object proposals.
  • VQA filter agreement rule = both LLaVA and Qwen answer Yes
    Relations are kept only if both models say yes (Appendix E.2); validated on 600 examples with balanced accuracy 0.6165, barely above majority.
assumptions (5)
  • domain assumption Scene graphs are a valid training target for improving general visual relationship reasoning
    The paper assumes dense relationship annotations transfer to VQA, REC, and region recognition; this is the core premise of the method (Sec 1, 3.2).
  • domain assumption GPT-4V and GPT-4o can produce and refine accurate relationships when seeded with human annotations
    Stage 1 relies on GPT-4V completing relations and Stage 2 on GPT-4o editing; quality is only indirectly validated via downstream benchmarks and 600 human examples (Sec 2, Appendix E).
  • ad hoc to paper Filtering with LLaVA and Qwen removes more bad than good relations
    Balanced accuracy 0.6165 on 600 examples is the only direct evidence; this is a weak basis for a data-scale assumption (Appendix E.2, Table 18).
  • domain assumption SentenceBERT cosine similarity is an adequate label matcher for evaluating open-ended outputs
    REC, region classification, and PSG evaluation all map free-form text to labels via SentenceBERT max similarity (Appendix D), which can inflate scores if paraphrases are accepted.
  • ad hoc to paper VSR and GQA benchmarks measure relationship understanding independently of training
    Both datasets appear in Stage 1 and Stage 2 training (Tables 7, 8), yet are reported as evaluation benchmarks in Table 2; this assumption is explicitly weakened by the ablation in Sec 4.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synthetic Visual Genome." pith.science (2026). https://pith.science/paper/GWAXIDEL

@misc{pith2026250607643,
  author       = {Pith},
  title        = {Pith review of: Synthetic Visual Genome},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWAXIDEL}},
  note         = {Machine review of arXiv:2506.07643}
}
read the original abstract

Reasoning over visual relationships-spatial, functional, interactional, social, etc.-is considered to be a fundamental component of human cognition. Yet, despite the major advances in visual comprehension in multimodal language models (MLMs), precise reasoning over relationships and their generations remains a challenge. We introduce ROBIN: an MLM instruction-tuned with densely annotated relationships capable of constructing high-quality dense scene graphs at scale. To train ROBIN, we curate SVG, a synthetic scene graph dataset by completing the missing relations of selected objects in existing scene graphs using a teacher MLM and a carefully designed filtering process to ensure high-quality. To generate more accurate and rich scene graphs at scale for any image, we introduce SG-EDIT: a self-distillation framework where GPT-4o further refines ROBIN's predicted scene graphs by removing unlikely relations and/or suggesting relevant ones. In total, our dataset contains 146K images and 5.6M relationships for 2.6M objects. Results show that our ROBIN-3B model, despite being trained on less than 3 million instances, outperforms similar-size models trained on over 300 million instances on relationship understanding benchmarks, and even surpasses larger models up to 13B parameters. Notably, it achieves state-of-the-art performance in referring expression comprehension with a score of 88.9, surpassing the previous best of 87.4. Our results suggest that training on the refined scene graph data is crucial to maintaining high performance across diverse visual reasoning task.

Figures

Figures reproduced from arXiv: 2506.07643 by the authors.

Figure 1
Figure 1. Example of our Synthetic Visual Genome (SVG) dataset – the first automatically generated large-scale scene graph dataset with diverse open-set categories, fine-grained regions, and densely annotated relationships. SVG averages four times more relations per object than Visual Genome [40]. make sense of new scenes by stitching together individual objects and their pairwise relationships [7, 16, 29, 31]. For instance, … view at source ↗
Figure 2
Figure 2. Overview of our Synthetic Visual Genome data engine. Stage 1: GPT-4 completes missing relationships for selected objects in existing image annotations, spanning five categories: spatial, interactional, emotional, functional, and social. Stage 2: Self-distillation pipeline where ROBIN generates dense scene graphs from images with segmentations, and GPT-4 refines these graphs by enriching content and correcting errors… view at source ↗
Figure 3
Figure 3. Architecture of ROBIN, consisting of a visual encoder that embeds the global image, a mask encoder that embeds segmentation masks individually, and a text encoder that encodes the textual input. 3.2. Training stages We next describe the three training stages designed to pro￾gressively distill scene-graph reasoning into our model. A detailed breakdown of these datasets, including data format￾ting and distribution spe… view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: VQA performance of blind LLMs using different scene graphs as context: (1) original Visual Genome (VG) scene graphs, (2) GPT-edited VG scene graphs, (3) VG scene graphs appended with Robin-3B relations (Stage 1), and (4) the GPT-edited scene graphs from Stage 2. run an…
Figure 4
Figure 4. Figure 4: Effectiveness of SG-EDIT distillation. Self-distillation and GPT4 editing Based on our SG-EDIT framework, we investigate the empirical gains of using GPT￾4 edited scene graphs compared to training on the model’s own generated scene graphs without refinement [PITH_FULL…
Figure 6
Figure 6. Figure 6: Dense scene graph generated by ROBIN on the Panoptic Scene Graph dataset [87] [14] side mirror of silver convertible ca  part of [1] [18] right taillight of blue ca  part of [5] [7] black car in the backgroun<  in front of [2] [19] red lighW  in front of [2] [PITH…
Figure 7
Figure 7. Figure 7: Dense scene graph generated by ROBIN on the ADE20K dataset [105] 8 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: presents a qualitative comparison between the Stage 1 and Stage 2 trained versions of ROBIN-3B. Notably, the Stage 2 model demonstrates a more accurate and precise understanding of the segmentation mask, correctly labeling [0] and [1] as the left and right walls, respe…
Figure 9
Figure 9. Figure 9: Example of scene graph quality generated with GPT-4o from scratch with provided regions. Errors are highlighted and underlined in red. A.3. GPT-4 Edited Scene Graph In [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Example of SVG-SG collection with SG-EDIT. GPT-4 can provide more precise object descriptions with attributes (bolded) and remove incorrect relations (highlighted in red) while adding relevant ones (underlined). B. Training Details B.1. Implementation details We train…
Figure 11
Figure 11. Figure 11: Data generation pipeline for SVG-RELATIONS, consisting of four steps: (1) gathering annotations for all the regions; (2) keeping only the annotations of the regions detected by SAM-SEEM; (3) selecting and annotating regions with GPT4-V and (4) filtering out incorrect …
Figure 12
Figure 12. Figure 12: Distribution of spatial relations before and after filtering has wearing holding held by using supporting carrying accompanied by looking at riding facing resting on displaying supported by touching placed on equipped with observed by playing with ridden by observing …
Figure 13
Figure 13. Figure 13: Distribution of interactional relations before and after filtering supports worn by used by holds used for eaten with transporting displays used with served with houses covers walked on by reflects eaten by served on illuminates uses provides shade for carries enterta…
Figure 14
Figure 14. Figure 14: Distribution of functional relations before and after filtering 7 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Distribution of emotional relations before and after filtering interacting with participating with sitting with being watched by accompanying posing with standing with engaging with teammate of performing for working with competing against coexisting with in the prese…
Figure 16
Figure 16. Figure 16: Distribution of social relations before and after filtering E.3. Object Proposal Generations • SAM [39]: We explore the subpart, part, and whole object modes available in the SAM model as region candidates for scene graph generation. We found the part and whole mode t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

136 extracted references · 43 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024. 4, 5, 6

    arXiv 2024

  2. [2]

    Tallyqa: Answering complex counting ques- tions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting ques- tions. In AAAI Conference on Artificial Intelligence,

  3. [3]

    Qwen-vl: A frontier large vision- language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, 9 Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023. 6, 9

    arXiv 2023

  4. [4]

    Qwen2.5- vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shi- jie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report...

    2025

  5. [5]

    Recognition-by-components: a theory of human image understanding

    Irving Biederman. Recognition-by-components: a theory of human image understanding. Psychological review, 94(2):115, 1987. 1

    1987

  6. [6]

    Scene perception: Detecting and judging objects undergoing relational violations

    Irving Biederman, Robert J Mezzanotte, and Jan C Rabinowitz. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive psychology, 14(2):143–177, 1982. 1

    1982

  7. [7]

    From machine learning to machine reasoning

    L´eon Bottou. From machine learning to machine reasoning. Machine learning, 94(2):133–149, 2014. 1

    2014

  8. [8]

    Internlm2 technical report.arXiv preprint arXiv:2403.17297, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report.arXiv preprint arXiv:2403.17297, 2024. 4, 5, 6, 3

    arXiv 2024

Show all 136 references
  1. [9]

    Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14...

    2024

  2. [10]

    Scene graph generation with role-playing large language models

    Guikun Chen, Jin Li, and Wenguan Wang. Scene graph generation with role-playing large language models. In NeurIPS, 2024. 9

    2024

  3. [11]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mo- hamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:...

    arXiv

  4. [12]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 3, 6, 9

    2023 arXiv

  5. [13]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 9

    2023 arXiv

  6. [14]

    De- tect what you can: Detecting and representing ob- jects using holistic models and body parts

    Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Loddon Yuille. De- tect what you can: Detecting and representing ob- jects using holistic models and body parts. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 1979–1986...

    2014

  7. [15]

    How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 2

    2024 arXiv

  8. [16]

    Some contro- versial questions in phonological theory

    Noam Chomsky and Morris Halle. Some contro- versial questions in phonological theory. Journal of linguistics, 1(2):97–138, 1965. 1

    1965

  9. [17]

    In- structblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. In- structblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024. 1, 9

    2024

  10. [18]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023. 3, 9, 6

    2023 arXiv

  11. [19]

    Blink: Multimodal large language models can see but not perceive.arXiv preprint arXiv:2404.12390, 2024

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei- Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive.arXiv preprint arXiv:2404.12390, 2024. 2

    2024 arXiv

  12. [20]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Syste...

    2024

  13. [21]

    Making the v in vqa matter: Elevating the role of image understand- ing in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understand- ing in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, ...

    2017

  14. [22]

    Agqa: A benchmark for compo- sitional spatio-temporal reasoning

    Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compo- sitional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 1

    2021

  15. [23]

    Lvis: A dataset for large vocabulary instance segmenta- tion

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmenta- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356– 5364, 2019. 3, 5, 6

    2019

  16. [24]

    Dsgg: Dense rela- tion transformer for an end-to-end scene graph gener- 10 ation

    Zeeshan Hayder and Xuming He. Dsgg: Dense rela- tion transformer for an end-to-end scene graph gener- 10 ation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28317–28326, 2024. 6

    2024

  17. [25]

    Partim- agenet: A large, high-quality dataset of parts, 2022

    Ju He, Shuo Yang, Shaokang Yang, Adam Ko- rtylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. Partim- agenet: A large, high-quality dataset of parts, 2022. 4

    2022

  18. [26]

    The curious case of neural text degenera- tion, 2020

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degenera- tion, 2020. 3

    2020

  19. [27]

    Sugarcrepe: Fixing hackable benchmarks for vision-language composi- tionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language composi- tionality. Advances in Neural Information Processing Systems, 36, 2024. 4, 5, 6

    2024

  20. [28]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019. 3, 4, 5, 9

    2019

  21. [29]

    Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757–795, 2020

    Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757–795, 2020. 1

    2020

  22. [30]

    Openclip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. If you use this software, please cite it as below. 4

    2021

  23. [31]

    Composi- tionality

    Theo MV Janssen and Barbara H Partee. Composi- tionality. In Handbook of logic and language, pages 417–473. Elsevier, 1997. 1

    1997

  24. [32]

    Action genome: Actions as compositions of spatio-temporal scene graphs

    Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Car- los Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10236–10247, 2020. 1

    2020

  25. [33]

    Dvqa: Understanding data visualiza- tions via question answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christo- pher Kanan. Dvqa: Understanding data visualiza- tions via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2018. 3

    2018

  26. [34]

    What’s “up” with vision-language models? inves- tigating their struggle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? inves- tigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 4

    2023

  27. [35]

    What’s” up” with vision-language models? inves- tigating their struggle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s” up” with vision-language models? inves- tigating their struggle with spatial reasoning. arXiv preprint arXiv:2310.19785, 2023. 5

    2023 arXiv

  28. [36]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min- joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251...

    2016

  29. [37]

    Are you smarter than a sixth grader? textbook ques- tion answering for multimodal machine comprehen- sion

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook ques- tion answering for multimodal machine comprehen- sion. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVP...

    2017

  30. [38]

    Llm4sgg: Large language models for weakly supervised scene graph generation

    Kibum Kim, Kanghoon Yoon, Jaeyeong Jeon, Yeon- jun In, Jinyoung Moon, Donghyun Kim, and Chany- oung Park. Llm4sgg: Large language models for weakly supervised scene graph generation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28306–28316,

    2024

  31. [39]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 3, 7, 5, 8

    2023

  32. [40]

    Vi- sual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yan- nis Kalantidis, Li-Jia Li, David A Shamma, et al. Vi- sual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of com...

    2017

  33. [41]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visu...

    1956

  34. [42]

    Seed-bench: Benchmark- ing multimodal llms with generative comprehension,

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmark- ing multimodal llms with generative comprehension,

  35. [43]

    Mimic-it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 9 11

    2023 arXiv

  36. [44]

    Semantic-sam: Segment and rec- ognize anything at any granularity

    Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shi- long Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and rec- ognize anything at any granularity. arXiv preprint arXiv:2307.04767, 2023. 3, 5, 8

    2023 arXiv

  37. [45]

    Sgtr: End-to-end scene graph generation with transformer,

    Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer,

  38. [46]

    From pixels to graphs: Open-vocabulary scene graph generation with vision- language models

    Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open-vocabulary scene graph generation with vision- language models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28076–28086, 2024. 9

    2024

  39. [47]

    Vila: On pre- training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 4, 5

    2024

  40. [48]

    Microsoft coco: Common ob- jects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common ob- jects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedi...

    2014

  41. [49]

    Gps-net: Graph property sensing net- work for scene graph generation

    Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing net- work for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3746–3753, 2020. 5, 6

    2020

  42. [50]

    Visual spatial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. 4, 5, 7, 9, 3

    2023

  43. [51]

    Improved baselines with visual instruction tun- ing

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tun- ing. arXiv preprint arXiv:2310.03744, 2023. 9, 3

    2023 arXiv

  44. [52]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 9, 6

    2024

  45. [53]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 4, 5

    2023 arXiv

  46. [54]

    Visual relationship detection with lan- guage priors

    Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with lan- guage priors. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 852–869. Springer, 2016. 1, 9

    2016

  47. [55]

    Groma: Localized visual tokeniza- tion for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokeniza- tion for grounding multimodal large language models. In European Conference on Computer Vision, pages 417–435. Springer, 2025. 3, 6

    2025

  48. [56]

    Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910–10921, 2023

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910–10921, 2023. 4, 6

    2023

  49. [57]

    Generation and comprehension of unambigu- ous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Mur- phy. Generation and comprehension of unambigu- ous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 5, 6

    2016

  50. [58]

    Ok-vqa: A visual ques- tion answering benchmark requiring external knowl- edge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual ques- tion answering benchmark requiring external knowl- edge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195– 3204, 2019. 3

    2019

  51. [59]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    arXiv

  52. [60]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawa- har. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209,

  53. [61]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fau- connier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 5

    2024 arXiv

  54. [62]

    et al. OpenAI. Gpt-4 technical report, 2024. 9

    2024

  55. [63]

    Localized symbolic knowledge distillation for visual commonsense models

    Jae Sung Park, Jack Hessel, Khyathi Raghavi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Youngjae Yu, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, and Yejin Choi. Localized symbolic knowledge distillation for visual commonsense models. ArXiv, abs/2312.04837,

    arXiv

  56. [64]

    Kosmos-2: Grounding multimodal large 12 language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large 12 language models to the world. arXiv preprint arXiv:2306.14824, 2023. 6

    2023 arXiv

  57. [65]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models, 2016. 3

    2016

  58. [66]

    Filtering, distillation, and hard negatives for vision-language pre-training

    Filip Radenovic, Abhimanyu Dubey, Abhishek Ka- dian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. In Proceedings of the IEEE/CVF conference on compute...

    2023

  59. [67]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning trans- ferable visual models from natural language supervi- sion. In ICML, 2021. 9

    2021

  60. [68]

    Paco: Parts and attributes of common ob- jects

    Vignesh Ramanathan, Anmol Kalia, Vladan Petro- vic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common ob- jects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2023

  61. [69]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming- Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03356, 2023. 9

    2023 arXiv

  62. [70]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3...

    2020

  63. [71]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. 5

    1908 arXiv

  64. [72]

    Scienceqa: A novel resource for question answering on scholarly articles

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23 (3):289–301, 2022. 3

    2022

  65. [73]

    Laion-400m: Open dataset of clip- filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beau- mont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Ko- matsuzaki. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 9, 6

    2021 arXiv

  66. [74]

    A- okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022. 7, 3

    2022

  67. [75]

    Textcaps: a dataset for image captioning with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020. 3

    2020

  68. [76]

    Learning to compose dynamic tree structures for visual contexts

    Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6619–6628, 2019. 5, 6

    2019

  69. [77]

    Unbiased scene graph generation from biased training

    Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3716–3725, 2020. 9

    2020

  70. [78]

    Is a picture worth a thousand words? delving into spatial rea- soning for vision language models

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial rea- soning for vision language models. arXiv preprint arXiv:2406.14852, 2024. 1

    2024 arXiv

  71. [79]

    Cogvlm: Visual ex- pert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual ex- pert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 9, 6

    2023 arXiv

  72. [80]

    Finetuned multimodal language models are high-quality image- text data filters

    Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Ku- mar, Yu Tian, Xifeng Yan, and Heng Wang. Finetuned multimodal language models are high-quality image- text data filters. arXiv preprint arXiv:2403.02677,

    arXiv

  73. [81]

    The all-seeing project v2: Towards gen- eral relation comprehension of the open world, 2024

    Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, Yu Qiao, and Jifeng Dai. The all-seeing project v2: Towards gen- eral relation comprehension of the open world, 2024. 2, 3, 5, 6, 9

    2024

  74. [82]

    Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoor- molabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv ...

    2022 arXiv

  75. [83]

    Visionllm v2: An end- to-end generalist multimodal large language model for hundreds of vision-language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, 13 Lewei Lu, Tong Lu, et al. Visionllm v2: An end- to-end generalist multimodal large language model for hundreds of vision-language tasks. arXiv preprint arXiv:2406.08394, 2024. 6

    2024 arXiv

  76. [84]

    Scene graph generation by iterative message passing

    Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410– 5419, 2017. 5, 6, 9

    2017

  77. [85]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 4, 5

    2024

  78. [86]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzhe...

    2024 arXiv

  79. [87]

    Panoptic scene graph generation

    Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In European Conference on Computer Vision, pages 178–196. Springer, 2022. 3, 4, 5, 6, 7, 8, 9

    2022

  80. [88]

    Set-of-mark prompt- ing unleashes extraordinary visual grounding in gpt- 4v

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompt- ing unleashes extraordinary visual grounding in gpt- 4v. arXiv preprint arXiv:2310.11441, 2023. 1

    2023 arXiv

  81. [89]

    Depth any- thing: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 3

    2024

  82. [90]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 4, 5

    2024 arXiv

  83. [91]

    mplug-owl2: Revolutionizing multi- modal large language model with modality collabora- tion

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi- modal large language model with modality collabora- tion. arXiv preprint arXiv:2311.04257, 2023. 9

    2023 arXiv

  84. [92]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 6, 9

    2023 arXiv

  85. [93]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexan- der C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 20...

    2016

  86. [94]

    Modeling context in re- ferring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in re- ferring expressions. pages 69–85. Springer, 2016. 4

    2016

  87. [95]

    Rlip: Relational language-image pre-training for human-object interaction detection, 2022

    Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Relational language-image pre-training for human-object interaction detection, 2022. 9

    2022

  88. [96]

    Os- prey: Pixel understanding with visual instruction tun- ing, 2024

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Os- prey: Pixel understanding with visual instruction tun- ing, 2024. 3, 4, 5, 6, 9

    2024

  89. [97]

    Neural motifs: Scene graph pars- ing with global context

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph pars- ing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5831–5840, 2018. 5, 6, 9

    2018

  90. [98]

    From recognition to cognition: Vi- sual commonsense reasoning

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Vi- sual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019. 4, 3

    2019

  91. [99]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.02949, 2023. 4, 6, 9, 3

    2023 arXiv

  92. [100]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning. arXiv preprint arXiv:2409.20566, 2024. 4, 5, 6

    2024 arXiv

  93. [101]

    Gpt4roi: Instruction tuning large lan- guage model on region-of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large lan- guage model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. 6, 3

    2023 arXiv

  94. [102]

    Llavar: Enhanced visual instruction tuning for text-rich im- 14 age understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich im- 14 age understanding. arXiv preprint arXiv:2306.17107,

    arXiv

  95. [103]

    Textpsg: Panop- tic scene graph generation from textual descrip- tions

    Chengyang Zhao, Yikang Shen, Zhenfang Chen, Mingyu Ding, and Chuang Gan. Textpsg: Panop- tic scene graph generation from textual descrip- tions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2839–2850,

  96. [104]

    Bubogpt: Enabling visual grounding in multi-modal llms

    Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581, 2023. 9

    2023 arXiv

  97. [105]

    Semantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019. 3, 5, 6, 7, 8

    2019

  98. [106]

    Hilo: Exploiting high low frequency relations for unbiased panoptic scene graph generation

    Zijian Zhou, Miaojing Shi, and Holger Caesar. Hilo: Exploiting high low frequency relations for unbiased panoptic scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21637–21648, 2023. 6, 9

    2023

  99. [107]

    round mirror,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 9 15 Synthetic Visual Genome Supplementary Material A. Qualitative Analysis In t...

    2023 arXiv

  100. [108]

    curtain → behind [2] right of [13]

  101. [109]

    chair → sat on by [14]

  102. [110]

    person is sitting → filler filler filler has [3,7,8,9,10], in front of [0,11,12], wearing [1] filler

  103. [111]

    head → part of [2], in front of [13] filling this empty line

  104. [113]

    wall spotlight is on → illuminating [13]

  105. [114]

    drawing → being drawn on [14]

  106. [115]

    person → touching [13] Generated Scene GraphEdited Scene Graph

  107. [116]

    blue backdrop → behind [1,2]

  108. [117]

    black chair → supporting [2]

  109. [118]

    person in maroon sweater sitting → has [3,7,8,9,10], in front of [0,11,12], fillerfill posing for [13], sitting on [1]

  110. [119]

    head of seated person → part of [2], fil looking at [13]

  111. [120]

    left hand → part of [2]

  112. [121]

    spotlight on wall → filererererer illuminating [2, 13]

  113. [122]

    sketch of person → drawn by [14]

  114. [123]

    The object is {class name}

    artist’s hand drawing → holding [13] … GPT4 Editing1.More precise object labels 2.Remove incorrect relations 3.Add relevant relations 03 2 1 97 108 13 14 1112 … Figure 10. Example of SVG-SG collection with SG-E DIT. GPT-4 can provide more precise object descriptions with attri...

  115. [124]

    The predicted subject and object bounding boxes have an Intersection over Union (IoU) greater than 0.5 with the ground truth bounding boxes

  116. [125]

    The predicted subject class, predicate, and object class match those of the ground truth. E. SVG Data Generation Details In this section, we provide more details of the data generation pipeline. E.1. SVG-R ELATIONS Figure 11 shows an overview of data curation for SVG-R ELATION...

  117. [126]

    4 people sitting in a large inflatable boat

    Keep Annotations of Regions Detected with SAM variants COCO + LVIS Objects VG Scene Graphs COCO + VG Captions Depth Maps A group of people riding in a black raft a bunch of white waves Paddle is in the water. 4 people sitting in a large inflatable boat

  118. [127]

    [315,077 ,500,428] man: paddling [20]

  119. [128]

    [266,294,320,377] baseball cap

  120. [129]

    [457 ,561,516,706] human, white leg

  121. [130]

    Does this relation correctly describe the image? Answer with Yes or No

    Select Regions and Annotate Relations with GPT4-V Filter with Rules and/or Models Subject 0: Woman wearing baseball cap sitting in front of [3] blonde-haired woma left of [16] man in a brown cowboy hat Subject 11... sharing e x perience with [3] blonde- haired woma riding [20]...

  122. [131]

    Examples include ’above’, ’below’, ’over’, ’across from’, ’behind’, ’in front of’, ’inside’, ’outside’ etc

    **Spatial Relationships**: These refer to the physical location or position of the person in relation to other objects or individuals. Examples include ’above’, ’below’, ’over’, ’across from’, ’behind’, ’in front of’, ’inside’, ’outside’ etc

  123. [132]

    Examples include ’holding’, ’touching’, ’looking at’, ’talking to’, ’playing with’, ’using’, etc

    **Interactional Relationships**: These involve some form of action or interaction between the person and other objects or individuals. Examples include ’holding’, ’touching’, ’looking at’, ’talking to’, ’playing with’, ’using’, etc

  124. [133]

    Examples include ’worn by’, ’used by’, ’owned by’, ’part of’, etc

    **Functional Relationships**: These refer to the purpose or function of an object in relation to the person. Examples include ’worn by’, ’used by’, ’owned by’, ’part of’, etc

  125. [134]

    Examples include ’friend of’, ’sibling of’, ’parent of’, ’colleague of’, ’boss of’, etc

    **Social Relationships** (for human): These refer to the social connection or interaction between the person and other individuals. Examples include ’friend of’, ’sibling of’, ’parent of’, ’colleague of’, ’boss of’, etc

  126. [135]

    Examples include ’loves’, ’likes’, ’dislikes’, ’hates’, etc

    **Emotional Relationships** (for human): These refer to the emotional connection or feelings between the person and other individuals. Examples include ’loves’, ’likes’, ’dislikes’, ’hates’, etc. **Description Requirement**: - The description should cover the interesting featu...

  127. [136]

    Go through all the objects in the scene graph and determine if you can confidently say the object mentioned by noisy description is really present and visible in the image. - If you think you can confidently verify the presence of an object and the description can be improved,...

  128. [137]

    bbox"][0] + obj[

    Then, make edits to the relationships by adding the prominent relationships current scene graph is missing, or/and removing erroneous relationships between objects in the scene. For example, object 1 has relationship: ’on’: [2,3,4,5] and you think it should be ’on’: [2,3], the...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.