Pith. sign in

REVIEW 4 major objections 5 minor 80 references

The paper sets out to show that explicit, grounded modeling of actions, objects, and relations—not just global image-text alignment—improves vision-language scene understanding, and backs the claim with a 615,805-image dataset and a tri-lev

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:17 UTC pith:FCMCLP6V

load-bearing objection A plausible 615K-image scene dataset and relation-aware CLIP variant, but every evaluation is measured against the same auto-generated labels used to build the dataset, so the 'grounded' claim is not yet supported. the 4 major comments →

arxiv 2511.20274 v2 pith:FCMCLP6V submitted 2025-11-25 cs.CV

SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking

Advik Sinha , Saurabh Atreya , Aashutosh A V , Sk Aziz Ali , Abhijit Das This is my paper
classification cs.CV
keywords vision-language pretrainingscene understandingrelation tripletsgroundingobject detectionknowledge distillationzero-shot retrievalscene graph
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that dual-stream vision-language models, which align whole images with short captions, miss the compositional structure of real scenes: multiple objects, actions, and inter-object relations cannot be captured by one global embedding. To fix this, it introduces SCLARO, a 615,805-image dataset spanning indoor, outdoor, and driving scenes, where every image carries a global action caption, object bounding boxes, and relation triplets of the form (object1, relation, object2), plus relation-focused regions that localize each interaction. It then proposes ScenarioCLIP, a model with separate visual and text encoders for the global scene, objects, and relations, trained with contrastive alignment at each level and an EMA-based knowledge-distillation objective that transfers global context into local representations. On zero-shot retrieval, linear-probe classification, object detection, predicate classification, and scene-graph classification, the model outperforms a hierarchical shared-encoder baseline, with the largest gains at the object and relation levels (e.g., +14.19 points Top-1 on object zero-shot retrieval). If correct, the work supplies both a large-scale training resource and a concrete recipe for making relational structure explicit in foundation-model pretraining.

Core claim

The core claim is that relational scene understanding can be learned by vision-language pretraining if the model is given grounded relational inputs and an architecture that keeps global, object, and relation semantics separate. ScenarioCLIP encodes, per image, the full scene, cropped objects, and relation-focused regions through three visual encoders, and the corresponding action caption, object names, and relation triplets through three text encoders; contrastive losses align each level across modalities, and exponential-moving-average teachers distill global visual semantics into object and relation embeddings (and fine-grained text semantics into the global text embedding). On the Action

What carries the argument

The load-bearing mechanism is a tri-level disentangled encoder set plus an EMA knowledge-distillation loop. Six encoders—global, object, and relation for vision and text, all initialized from a pretrained vision-language backbone—map the full image, object crops, and relation-focused regions, and their text counterparts, into a shared space. Contrastive alignment is applied separately at the global, object, and relation levels, with synthetic hard negatives formed by swapping objects or replacing predicates with opposites. Simultaneously, an exponential-moving-average teacher transfers knowledge from the global visual embedding to object and relation visual embeddings via KL divergence, whil

Load-bearing premise

The entire benchmark rests on auto-generated annotations being true: the action captions, object boxes, and relation triplets come from a vision-language model plus detection and segmentation tools, and the paper reports no human verification or independent agreement check, so every measured improvement is computed against labels that could contain hallucinated objects, wrong relations, or mislocalized boxes.

What would settle it

Take a random sample of a few hundred dataset images, have annotators list the objects actually present and the relations that hold between the detected boxes, and compute agreement with the auto-generated triplets. A large fraction of unsupported or absent-object triplets would show that the grounded claim fails; a companion check would be to train ScenarioCLIP only on human-verified samples and see whether the object and relation retrieval gains survive on unseen object-predicate combinations.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Explicit object- and relation-level encoders, rather than a single shared encoder, yield large object zero-shot retrieval gains (+14.19 Top-1) without hurting action-level performance, so relational structure can be added to existing vision-language pipelines.
  • Frozen ScenarioCLIP representations separate action, object, and relation classes better under linear probing (+3.83 to +4.56 Top-1), implying cleaner class boundaries at every level.
  • The relation encoder transfers to object detection, where using the global encoder for proposals and the object encoder for RoI features nudges AP from 9.7 to 9.9 (AP50 18.6 to 18.9), and to predicate/scene-graph classification (R@1 gains of 4.51 and 5.70).
  • The dataset itself, with 225,609 relation classes and focused regions for each triplet, provides a large-scale benchmark for future work on grounded scene understanding, and the synthetic hard negatives give a ready-made contrastive training signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the auto-generated annotations are accurate enough, an obvious testable extension is to ablate the hard-negative triplets versus the disentangled encoders separately; knowing which ingredient drives the +14.19 object-retrieval gain would help practitioners decide what to adopt.
  • Because the paper does not report human verification of the vision-language-generated labels, a before-using-the-dataset sanity check is to re-annotate a random subset with independent annotators and measure agreement; that number would bound how much of the measured gain is real semantics versus label artifacts.
  • The EMA distillation direction—global-to-local on vision, local-to-global on text—is a recipe that could transfer to other modalities (video, LiDAR, robotics state), where a single caption also under-specifies the scene; scenario-specific grounding would be the test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces a large-scale visual scene understanding resource (615,805 images) with three levels of annotation: global action captions, object bounding boxes, and relation triplets, together with relation-focused regions. The annotations are produced entirely automatically by a pipeline comprising a VLM (Ovis-Gemma 9B), GroundingDINO, and SAM, with only vocabulary-level curation. The paper also proposes ScenarioCLIP, a CLIP-style model with disentangled global, object, and relation encoders trained with contrastive alignment and EMA-based knowledge distillation. Experiments on the proposed dataset report gains over PyramidCLIP on zero-shot retrieval, linear probing, object detection, predicate classification, and scene graph classification, and the abstract claims out-of-domain generalization. The central claims are (a) the dataset provides 'grounded' scenario-level annotations and (b) ScenarioCLIP improves relational scene understanding relative to prior CLIP-style models.

Significance. If the dataset annotations were trustworthy, the resource would be potentially valuable: it is large, spans diverse domains, and combines action, object, and relation supervision in one corpus, which is uncommon. The ScenarioCLIP architecture is also a reasonable extension of CLIP-style pretraining to hierarchical scene understanding, and the disentangled-encoder design with EMA distillation is clearly described. The paper ships code and a data pipeline, which aids reproducibility. However, the significance is conditional on the correctness of the annotations and on the validity of the evaluation. As it stands, the evaluation is self-referential: all labels used for training and testing are generated by the same automatic pipeline, with no human verification or external benchmark. This undermines the dataset's 'grounded' claim and makes the reported performance gains hard to interpret as evidence of real scene understanding rather than of agreement with a particular VLM's biases.

major comments (4)
  1. [§3.1, §3.2, §7, Tables 1–6] The evaluation is circular. All ground-truth action, object, and relation labels are produced automatically by Ovis-Gemma, GroundingDINO, and SAM, with no human verification or external agreement check. Every number in Tables 1, 2, 3, 5, and 6 is measured against these auto-generated labels. If the VLM hallucinates objects or relations, or if GroundingDINO/SAM mislocalize them, the reported gains may simply reflect how well ScenarioCLIP fits the generator's output. The manuscript should include (i) a human evaluation of a random sample of the generated labels, (ii) an agreement study against human annotations for a subset, and/or (iii) evaluation on existing human-annotated datasets (e.g., COCO, Visual Genome, OpenImages) for object and relation tasks. Without such an anchor, the 'grounded' dataset claim is unsupported.
  2. [Abstract and §5] The abstract explicitly promises 'out-of-domain generalisation' and the introduction highlights it, but the main text and supplementary contain no out-of-domain experiment. All downstream evaluations in §5 are on the Action-Genome test set, which is generated by the same pipeline. Table 1B uses released CLIP/SLIP checkpoints, but these are not evaluated on external data. To support the out-of-domain claim, the authors must add experiments that transfer ScenarioCLIP to independent datasets or tasks (e.g., zero-shot or fine-tuned evaluation on COCO/Visual Genome, or on an existing action recognition benchmark). This is a load-bearing missing piece.
  3. [§7, Eq. (4)–(5)] The relation-focused region is a purely spatial construction: an RBF-weighted blend of two SAM masks, with non-focused pixels blurred. There is no evidence that these regions correspond to the semantics of the relation triplet. The relation localization evaluation in §8.1.5 and Table 6 uses the same auto-generated relation masks as supervision, so it cannot validate grounding. The paper should provide a human or external assessment of whether the focused regions meaningfully localize the interaction, or demonstrate performance on a dataset with human-annotated relation regions.
  4. [§3.2 and §8.1.4] The dataset's relation class count is 225,609, which is extremely large relative to 615,805 images, and the paper claims the vocabulary was 'cleaned' and 'consolidated.' Yet the predicate classification and scene graph evaluation in Table 5 restrict to a subset of ~700 predicates, suggesting that the full 225k relation set is not being meaningfully evaluated. This inconsistency raises doubts about the quality and usability of the relation taxonomy. The authors should clarify how many relations are actually usable after filtering and whether the huge class count reflects duplicate or near-duplicate relations that survived the curation.
minor comments (5)
  1. [Title/Abstract] The arXiv metadata titles the paper 'SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking,' but the manuscript's internal title is 'ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis.' The dataset is referred to as 'Action-Genome' throughout the main text. This inconsistency should be resolved in revision.
  2. [§5.2, Table 1] The 'zero-shot' naming is confusing: Table 1A is described as in-domain pretraining on Action-Genome, and the comparison is against PyramidCLIP that is also fine-tuned on the test distribution. This is not the conventional zero-shot setting used for CLIP-style models. Clarify what 'zero-shot' means here (no fine-tuning on the specific task?) and avoid conflating it with the standard cross-dataset zero-shot evaluation.
  3. [§5.5 and Table 7] The ablation of λ_KD is only reported for zero-shot retrieval; the effect on detection and linear probe is not shown for the other λ_KD schedules. The claim that 'KD refines the structure' would be stronger with ablations across tasks.
  4. [References] Several references are incomplete or incorrectly formatted (e.g., [56] 'Viualizing data using t-sne'; [40] Ovis appears in the authors list as 'Lu, Shiyin, et al.' with an incomplete author list). Please check all entries.
  5. [Figure 3/4] The figures are dense and the annotations are small. In particular, Figure 4's embedding-dimension labels (e.g., 'nO×3×512×512' for object crops) are confusing and should be clarified in the caption.

Circularity Check

2 steps flagged

Benchmark-level circularity: all reported scores are measured against auto-generated VLM/GroundingDINO/SAM labels from the same pipeline that produced the training targets, with no independent anchor; relation localization is self-referential to the generated masks.

specific steps
  1. fitted input called prediction [Section 3.1 (Stages 1-3) and Section 5.1; Tables 1-3, 5-6]
    "Stage-1 uses an open-weight VLM (Ovis-Gemma 9B [11, 40]) to generate a global action caption, object list, and relation triplets for each image, based on a custom prompt. Stage-2 applies GroundingDINO [35], prompted with the object list, to predict bounding boxes (xmin, ymin, xmax, ymax) for all mentioned objects. Stage-3 uses SAM [25] to obtain object masks and then constructs relation-focused regions by spatially weighting and blending the masks with the original image. ... All experiments are carried out on the test set of the proposed Action-Genome Dataset."

    The test labels that define every reported metric are produced by the same VLM/GroundingDINO/SAM pipeline used to create the training targets. ScenarioCLIP is fitted to these annotations, and its outputs are then presented as predictions of grounded scenario understanding. There is no independent human or external annotation in the loop, so the benchmark measures the model's agreement with the generator's annotations, not with scene content. If Ovis hallucinates an object or relation, the model is rewarded for retrieving that hallucinated label. The 'grounded' claim therefore reduces to the generator's output by construction.

  2. self definitional [Supplementary Section 8.1.5 (Relation Localization)]
    "To evaluate how well the relation encoder captures the spatial extent of an interaction, we train a lightweight decoder to predict a dense relation mask. For each annotated (object1, relation, object2) triple in Action-Genome, we construct a focused input image by zeroing out all pixels outside the union of the object1 and object2 bounding boxes, and use the corresponding relation mask provided in the dataset as the supervision signal."

    The 'relation mask provided in the dataset' is itself the Stage-3 output: SAM masks blended by the RBF/Gaussian weighting (Equations 4-5). The decoder is trained and evaluated against this same constructed target, so the localization metric measures how well the model reproduces the generation formula, not whether the relation is independently grounded in the image. The task's ground truth is defined by the same procedure that creates the relation-focused regions, making the experiment self-referential.

full rationale

The derivation of ScenarioCLIP's objective (Eqs. 1-3) is self-contained and not circular, and the relative comparison between ScenarioCLIP and PyramidCLIP on identical auto-generated labels is internally valid; this is not a score of 8-10. However, the paper's central resource claim — 'grounded annotations at all three levels' — and every reported absolute benchmark number are evaluated on the test split of the Action-Genome dataset, whose labels were generated by the same three-stage VLM/GroundingDINO/SAM pipeline that created the training targets. No human audit or external agreement check is reported; the single sentence about reusing Kinetics/OpenPVSG labels is not accompanied by any split-level quality analysis. The relation-localization experiment is explicitly self-referential: the supervision signal is the dataset's RBF/SAM-generated mask. The abstract also promises out-of-domain generalization, but no OOD experiments appear in the main text or supplementary. Consequently, the reported gains are evidence of consistency with the generator's annotations, not of independent grounded scenario understanding.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on unverified auto-generated labels and ad hoc focused-region construction. The free parameters and axioms listed here are the load-bearing, independently unverified choices that the dataset and model performance depend on.

free parameters (3)
  • λ_KD (distillation weight) = 1 (fixed; ablated to 10 and annealed schedules in Table 7)
    Hand-selected loss weight balancing KD and contrastive terms; the ablation in Table 7 shows small performance differences, so the final choice is tuned on the evaluation set.
  • RBF/Gaussian blur scale σ in focused-region construction = not specified in the paper
    Equations 4-5 define RBF and Gaussian kernels, but σ controls how much of the image is highlighted and is chosen ad hoc; exact values are deferred to the supplementary and not given in the provided text.
  • EMA decay / warm-up steps = 0.9995 / 2000
    Hyperparameters chosen for training stability; they affect the distillation targets but are not fitted to external benchmarks.
axioms (4)
  • ad hoc to paper VLM outputs from Ovis-Gemma 9B (action, object, and relation texts) are treated as ground-truth annotations without human verification.
    Section 3 Stage 1; all benchmark labels and supervision derive from this assumption.
  • domain assumption GroundingDINO bounding boxes and SAM masks localize the VLM-mentioned objects and relations accurately.
    Sections 3.1-3.2; object detection and relation-focused region supervision depend on these pretrained models being correct on the source images.
  • ad hoc to paper Relation-focused regions built by RBF-weighted blending of SAM masks are a valid supervisory signal for inter-object relations.
    Equations 4-5 and Section 3; no independent evidence that these masks correspond to human-perceived interactions.
  • domain assumption In-domain 'zero-shot' retrieval on the same label vocabulary used during pretraining measures generalization.
    Section 5.2 and Tables 1-2; test classes appear in the pretraining set, so the zero-shot setting is not conventional open-vocabulary transfer.
invented entities (1)
  • Relation-focused region (RBF-blended mask) no independent evidence
    purpose: A new input representation that highlights the spatial context of a (object1, relation, object2) triplet for the relation visual encoder.
    Constructed from SAM masks and RBF weighting; the paper provides no external validation that these regions isolate the true interacting objects.

pith-pipeline@v1.3.0-alltime-deepseek · 18612 in / 12433 out tokens · 129316 ms · 2026-08-03T20:17:58.067753+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking." pith.science (2026). https://pith.science/paper/FCMCLP6V

@misc{pith2026251120274,
  author       = {Pith},
  title        = {Pith review of: SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCMCLP6V}},
  note         = {Machine review of arXiv:2511.20274}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In the paradigm of computer vision-based precise real-world scene understanding, joint reasoning in terms of contextual understanding about the objects present in a scene, their inter-object relations, and the action being performed is an essential prerequisite. However, prior works have not addressed all three jointly, and no large-scale dataset provides grounded annotations at all three levels across diverse visual scenarios. Hence, this work introduces the SCLARO (Scene-Contextual Localisation of Actions, Relations & Objects) dataset, consisting of 615,805 images spanning indoor, outdoor, and driving scenarios, annotated with global action captions, object bounding boxes, and relation triplets that supply structured scene context beyond a free-text caption. To benchmark the dataset, we propose ScenarioCLIP, a tri-level reference model that jointly encodes global scene context, objects, and inter-object relations using disentangled encoders and EMA-based knowledge distillation. We benchmark across a comprehensive suite of tasks on the SCLARO Dataset, namely zero-shot retrieval, linear probe, object detection, predicate classification, scene-graph classification, and out-of-domain generalisation. ScenarioCLIP's disentangled encoders improve over the previous works, such as PyramidCLIP's shared encoder, most notably at the object and relation levels and on out-of-domain generalisation. Code for the data generation pipeline and ScenarioCLIP is available at https://github.com/scenario-clip/SCLARO-ScenarioCLIP

Figures

Figures reproduced from arXiv: 2511.20274 by Aashutosh A V, Abhijit Das, Advik Sinha, Saurabh Atreya, Sk Aziz Ali.

Figure 1
Figure 1. Figure 1: ScenarioCLIP can not only detect actions and ob￾jects but also localize the relations between objects in both single￾relation (a) and multi-relation (b) scenes. VLM (OpenVLM/Ovis-Gemma [11, 40]) for action, object, and relation text, GroundingDINO [35] for object ground￾ing, and SAM [25] plus RBF-based masking to construct relation-focused regions. In addition, we synthesise hard negative relation triplets… view at source ↗
Figure 2
Figure 2. Figure 2: Focused regions and scenario-centric grounding. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Data generation pipeline for the Action-Genome Dataset [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of ScenarioCLIP. A global, object, and relation encoder extracts visual features from the full image, object crops, and focused regions, while corresponding text encoders embed the action caption, object names, and relation triplets. Contrastive losses align image-action, object-object, and relation-relation pairs, and EMA teachers provide knowledge-distillation targets. 5 [PITH_FULL_IMAGE:figure… view at source ↗
Figure 5
Figure 5. Figure 5: t-SNE visualization of object-level semantic embeddings of 10,000 objects from the Action Genome Dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Object processing pipeline, illustrating the refinement from raw VLM-generated object strings to final grouped object classes by removing adjectives and color descriptors, eliminating redundant plural forms, and clustering semantically similar objects using BERT embeddings and HDBSCAN. Predicate Classification Scene Graph Classification Model R@1 R@5 R@10 R@1 R@5 R@10 PyramidCLIP [14] 30.33 63.94 78.37 25.… view at source ↗
Figure 7
Figure 7. Figure 7: Action processing and consolidation, where diverse raw action phrases are normalized and merged into a smaller set of canonical action labels. β = (0.9, 0.999), ε = 10−8 and a weight decay of 0.2 for all trainable parameters in the vision-language backbone and contrastive loss modules. We employ a learning-rate schedule with a linear warm-up over the first 10% of the total optimization steps, where the lea… view at source ↗
Figure 8
Figure 8. Figure 8: Relation processing pipeline. Raw VLM-generated relation strings are converted into relation triplets by mapping objects to groups, removing stopwords, and collapsing duplicate triplets into a final relation set. Frozen Encoder Trainable Encoder Model Dice Coefficient (↑) IoU (↑) MAE (↓) Dice Coefficient (↑) IoU (↑) MAE (↓) PyramidCLIP [14] 0.6378 0.5269 0.1535 0.5971 0.4858 0.1625 ScenarioCLIP w/o KD 0.63… view at source ↗
Figure 9
Figure 9. Figure 9: Grad-CAM [53] visualizations of the object encoder for PyramidCLIP [14], ScenarioCLIP w/o KD, and ScenarioCLIP. For each image, we feed the whole frame into the object encoder and compute a patch-level relevance map, which is overlaid on the original image. ScenarioCLIP produces more focused and semantically meaningful activations around the objects and interactions of interest, while PyramidCLIP [14] atte… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 2 linked inside Pith

  1. [1]

    Adapting contrastive language-image pretrained (clip) models for out-of-distribution detection, 2023

    Nikolas Adaloglou, Felix Michels, Tim Kaiser, and Markus Kollmann. Adapting contrastive language-image pretrained (clip) models for out-of-distribution detection, 2023. 3

    2023

  2. [2]

    Behley, M

    J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. SemanticKITTI: A Dataset for Se- mantic Scene Understanding of LiDAR Sequences. InProc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019. 4

    2019

  3. [3]

    ” O’Reilly Media, Inc.”, 2009

    Steven Bird, Ewan Klein, and Edward Loper.Natural lan- guage processing with Python: analyzing text with the natu- ral language toolkit. ” O’Reilly Media, Inc.”, 2009. 1

    2009

  4. [4]

    Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density-based clustering based on hierarchical den- sity estimates. InAdvances in Knowledge Discovery and Data Mining, pages 160–172, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. 1

    2013

  5. [5]

    A simple framework for contrastive learning of visual representations, 2020

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations, 2020. 2

    2020

  6. [6]

    Clip guided image-perceptive prompt learning for image enhancement,

    Weiwen Chen, Qiuhong Ke, and Zinuo Li. Clip guided image-perceptive prompt learning for image enhancement,

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2

    2009

  8. [8]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019. 1

    2019

  9. [9]

    De- coupling zero-shot semantic segmentation, 2022

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation, 2022. 3

    2022

  10. [10]

    Learning to prompt for open-vocabulary ob- ject detection with vision-language model, 2022

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary ob- ject detection with vision-language model, 2022. 3

    2022

  11. [11]

    Gemma 2: Improving open language models at a practical size, 2024

    Gemma Team et al. Gemma 2: Improving open language models at a practical size, 2024. 2, 4, 1

    2024

  12. [12]

    Clip-adapter: Better vision-language models with feature adapters, 2021

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters, 2021. 3

    2021

  13. [13]

    Pyramidclip: Hierarchical fea- ture alignment for vision-language model pretraining.Ad- vances in neural information processing systems, 35:35959– 35970, 2022

    Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Ron- grong Ji, and Chunhua Shen. Pyramidclip: Hierarchical fea- ture alignment for vision-language model pretraining.Ad- vances in neural information processing systems, 35:35959– 35970, 2022. 1

    2022

  14. [14]

    Pyramidclip: Hierarchical fea- ture alignment for vision-language model pretraining, 2022

    Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Ron- grong Ji, and Chunhua Shen. Pyramidclip: Hierarchical fea- ture alignment for vision-language model pretraining, 2022. 1, 3, 7, 8, 2, 4, 5

    2022

  15. [15]

    Geiger, P

    A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Au- tonomous Driving? The KITTI Vision Benchmark Suite. InProc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3354–3361, 2012. 4

    2012

  16. [16]

    Fast r-cnn, 2015

    Ross Girshick. Fast r-cnn, 2015. 2

    2015

  17. [17]

    Open-vocabulary object detection via vision and language knowledge distillation, 2022

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation, 2022. 3

    2022

  18. [18]

    A survey on self-supervised learning: Algorithms, applications, and future trends, 2024

    Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends, 2024. 2

    2024

  19. [19]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. 1

    2015

  20. [20]

    Momentum contrast for unsupervised visual rep- resentation learning, 2020

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning, 2020. 2

    2020

  21. [21]

    Structure-clip: To- wards scene graph knowledge to enhance multi-modal struc- tured representations, 2023

    Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, and Wen Zhang. Structure-clip: To- wards scene graph knowledge to enhance multi-modal struc- tured representations, 2023. 2

    2023

  22. [22]

    Le, Yunhsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. 3

    2021

  23. [23]

    The kinetics human action video dataset, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017. 4

    2017

  24. [24]

    Human pose descriptions and subject-focused attention for improved zero-shot transfer in human-centric classification tasks, 2024

    Muhammad Saif Ullah Khan, Muhammad Ferjad Naeem, Federico Tombari, Luc Van Gool, Didier Stricker, and Muhammad Zeshan Afzal. Human pose descriptions and subject-focused attention for improved zero-shot transfer in human-centric classification tasks, 2024. 3

    2024

  25. [25]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything, 2023. 2, 3, 4, 5, 1

    2023

  26. [26]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32–33, 2009. 3

    2009

  27. [27]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural net- works. InProceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1, page 1097–1105, Red Hook, NY , USA, 2012. Curran Associates Inc. 1

    2012

  28. [28]

    Veclip: Improving clip training via visual-enriched captions,

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Improving clip training via visual-enriched captions,

  29. [29]

    Dongxu Li, Junnan Li, and Steven C. H. Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to- image generation and editing, 2023. 3

    2023

  30. [30]

    Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training, 2019

    Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training, 2019. 3

    2019

  31. [31]

    Visualbert: A simple and performant baseline for vision and language, 2019

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language, 2019. 3 9

    2019

  32. [32]

    Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm, 2022

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm, 2022. 1

    2022

  33. [33]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 3

    2023

  34. [34]

    One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion, 2023

    Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Ji- ayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion, 2023. 3

    2023

  35. [35]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023. 2, 3, 4, 5, 1

    Pith/arXiv arXiv 2023

  36. [36]

    C-CLIP: Multimodal continual learning for vision-language model

    Wenzhuo Liu, Fei Zhu, Longhui Wei, and Qi Tian. C-CLIP: Multimodal continual learning for vision-language model. In The Thirteenth International Conference on Learning Repre- sentations, 2025. 3

    2025

  37. [37]

    Wonder3d: Single image to 3d using cross-domain diffusion,

    Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, and Wenping Wang. Wonder3d: Single image to 3d using cross-domain diffusion,

  38. [38]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 7, 2, 3

    2019

  39. [39]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks, 2019. 3

    2019

  40. [40]

    Ovis: Structural em- bedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural em- bedding alignment for multimodal large language model. arXiv:2405.20797, 2024. 2, 4, 1

    Pith/arXiv arXiv 2024

  41. [41]

    Mm1: Methods, analysis & insights from multimodal llm pre- training, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu H`e, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zi...

    2024

  42. [42]

    Rareact: A video dataset of unusual interactions, 2020

    Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Rareact: A video dataset of unusual interactions, 2020. 3

    2020

  43. [43]

    S-CLIP: Semi-supervised vision-language learning us- ing few specialist captions

    Sangwoo Mo, Minkyu Kim, Kyungmin Lee, and Jinwoo Shin. S-CLIP: Semi-supervised vision-language learning us- ing few specialist captions. InThirty-seventh Conference on Neural Information Processing Systems, 2023. 6

    2023

  44. [44]

    Multi- moments in time: Learning and interpreting models for multi-action video understanding, 2021

    Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry A McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogerio Feris, and Aude Oliva. Multi- moments in time: Learning and interpreting models for multi-action video understanding, 2021. 4

    2021

  45. [45]

    Slip: Self-supervision meets language-image pre- training, 2021

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training, 2021. 7

    2021

  46. [46]

    Understand- ing fine-tuning CLIP for open-vocabulary semantic segmen- tation in hyperbolic space

    Zelin Peng, Zhengqin Xu, Changsong Wen, Yu Huang, Menglin Yang, Feilong Tang, and Wei Shen. Understand- ing fine-tuning CLIP for open-vocabulary semantic segmen- tation in hyperbolic space. In2nd Beyond Euclidean Work- shop: Hyperbolic and Hyperspherical Learning for Com- puter Vision, 2025. 3

    2025

  47. [47]

    Filtering, distil- lation, and hard negatives for vision-language pre-training,

    Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distil- lation, and hard negatives for vision-language pre-training,

  48. [48]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 1, 2, 3, 4, 7

    2021

  49. [49]

    Faster r-cnn: Towards real-time object detection with region proposal networks, 2016

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016. 3

    2016

  50. [50]

    Interpreting and analyzing clip’s zero-shot image classification via mutual knowledge, 2024

    Fawaz Sammani and Nikos Deligiannis. Interpreting and analyzing clip’s zero-shot image classification via mutual knowledge, 2024. 3

    2024

  51. [51]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,

  52. [52]

    Laion-5b: An open large-scale dataset for training next generation image-text models, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text model...

    2022

  53. [53]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Ba- tra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Ba- tra. Grad-cam: Visual explanations from deep networks via gradient-based localization.International Journal of Com- puter Vision, 128(2):336–359, 2019. 4, 5

    2019

  54. [54]

    Ex- plore the potential of clip for training-free open vocabulary semantic segmentation, 2024

    Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Ex- plore the potential of clip for training-free open vocabulary semantic segmentation, 2024. 3

    2024

  55. [55]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. 3

    2024

  56. [56]

    Viualizing data using t-sne.Journal of Machine Learning Research, 9:2579–2605, 2008

    Laurens van der Maaten, Geoffrey Hinton, and Yoesoep Rachmad. Viualizing data using t-sne.Journal of Machine Learning Research, 9:2579–2605, 2008. 8

    2008

  57. [57]

    Clipn for zero-shot ood detection: Teaching clip to say no,

    Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no,

  58. [58]

    Actionclip: A new paradigm for video action recognition, 2021

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition, 2021. 3 10

    2021

  59. [59]

    See- ing in flowing: Adapting clip for action recognition with mo- tion prompts learning, 2023

    Qiang Wang, Junlong Du, Ke Yan, and Shouhong Ding. See- ing in flowing: Adapting clip for action recognition with mo- tion prompts learning, 2023. 3

    2023

  60. [60]

    Videocomposer: Compositional video synthesis with motion controllability, 2023

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability, 2023. 3

    2023

  61. [61]

    Enhancing visual-language modality alignment in large vision language models via self-improvement, 2024

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Par- minder Bhatia, Furong Huang, and Cao Xiao. Enhancing visual-language modality alignment in large vision language models via self-improvement, 2024. 2

    2024

  62. [62]

    Clip-gen: Language-free training of a text-to-image genera- tor with clip, 2022

    Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. Clip-gen: Language-free training of a text-to-image genera- tor with clip, 2022. 3

    2022

  63. [63]

    Hq-clip: Leveraging large vision-language models to create high-quality image- text datasets and clip models, 2025

    Zhixiang Wei, Guangting Wang, Xiaoxiao Ma, Ke Mei, Hua- ian Chen, Yi Jin, and Fengyun Rao. Hq-clip: Leveraging large vision-language models to create high-quality image- text datasets and clip models, 2025. 3

    2025

  64. [64]

    Groupvit: Semantic segmentation emerges from text supervision, 2022

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision, 2022. 3

    2022

  65. [65]

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee. K. Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model, 2024. 3

    2024

  66. [66]

    Post-pre-training for modality alignment in vision-language foundation models, 2025

    Shin’ya Yamaguchi, Dewei Feng, Sekitoshi Kanai, Kazuki Adachi, and Daiki Chijiwa. Post-pre-training for modality alignment in vision-language foundation models, 2025. 3

    2025

  67. [67]

    Instance-aware repeat factor sampling for long-tailed object detection, 2023

    Burhaneddin Yaman, Tanvir Mahmud, and Chun-Hao Liu. Instance-aware repeat factor sampling for long-tailed object detection, 2023. 3

    2023

  68. [68]

    Panoptic video scene graph generation, 2023

    Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, and Ziwei Liu. Panoptic video scene graph generation, 2023. 4

    2023

  69. [69]

    Filip: Fine-grained interactive language-image pre-training, 2021

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training, 2021. 1, 3

    2021

  70. [70]

    Coca: Contrastive captioners are image-text foundation models, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022

    2022

  71. [71]

    Vision-language models for vision tasks: A survey, 2024

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey, 2024. 3

    2024

  72. [72]

    Tip-adapter: Training-free clip-adapter for better vision- language modeling, 2021

    Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling, 2021. 3

    2021

  73. [73]

    Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

  74. [74]

    Conditional prompt learning for vision-language models, 2022

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Conditional prompt learning for vision-language models, 2022. 3

    2022

  75. [75]

    Aligning modalities in vision large lan- guage models via preference fine-tuning, 2024

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large lan- guage models via preference fine-tuning, 2024. 2

    2024

  76. [76]

    Openpsg: Open-set panoptic scene graph generation via large multimodal models, 2024

    Zijian Zhou, Zheng Zhu, Holger Caesar, and Miaojing Shi. Openpsg: Open-set panoptic scene graph generation via large multimodal models, 2024. 4

    2024

  77. [77]

    Clip-vis: Adapting clip for open-vocabulary video instance segmentation, 2024

    Wenqi Zhu, Jiale Cao, Jin Xie, Shuangming Yang, and Yan- wei Pang. Clip-vis: Adapting clip for open-vocabulary video instance segmentation, 2024. 3 11 ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis Supplementary Material

    2024

  78. [78]

    taking a photo

    Models used for Stage-wise Dataset Genera- tion Stage-1 of our generation pipeline involves using an OpenVLM(Ovis-Gemma 9B [11, 40]). This model gen- erates annotations based on a carefully crafted custom prompt. We experimented with different phrasing struc- tures, adjusting syntax. These experiments were crucial in impacting the quality of our final ann...

  79. [79]

    garbage truck

    model to get semantically meaningful embeddings for the objects. On these embeddings, we use HDBSCAN (Hi- erarchical Density-Based Spatial Clustering of Applications with Noise) [4] for clustering. HDBSCAN assigns clusters based on density of the data points. Amutual reachability distance, defined as: dmreach-k(a, b) = max{corek(a),core k(b), d(a, b)} is ...

  80. [80]

    Anneal1→0

    Experiments 8.1. Implementation Details 8.1.1. Pre-training Stage All ScenarioCLIP variants are pretrained on the Action- Genome dataset for12epochs using the AdamW opti- mizer [38]. We use a base learning rate of2×10 −5, Figure 7. Action processing and consolidation, where diverse raw action phrases are normalized and merged into a smaller set of canonic...

    arXiv