pith. sign in

arxiv: 2207.00221 · v2 · pith:3HHEZLGLnew · submitted 2022-07-01 · 💻 cs.CV · cs.CL· cs.LG

VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations

Tiancheng Zhao , Tianqi Zhang , Mingwei Zhu , Haozhan Shen , Kyusong Lee , Xiaopeng Lu , Jianwei Yin This is my paper
classification 💻 cs.CV cs.CLcs.LG
keywords modelsdownstreammethodproposedvl-checklistattributesframeworkfurther
0
0 comments X
read the original abstract

Vision-Language Pretraining (VLP) models have recently successfully facilitated many cross-modal downstream tasks. Most existing works evaluated their systems by comparing the fine-tuned downstream task performance. However, only average downstream task accuracy provides little information about the pros and cons of each VLP method, let alone provides insights on how the community can improve the systems in the future. Inspired by the CheckList for testing natural language processing, we exploit VL-CheckList, a novel framework to understand the capabilities of VLP models. The proposed method divides the image-texting ability of a VLP model into three categories: objects, attributes, and relations, and uses a novel taxonomy to further break down these three aspects. We conduct comprehensive studies to analyze seven recently popular VLP models via the proposed framework. Results confirm the effectiveness of the proposed method by revealing fine-grained differences among the compared models that were not visible from downstream task-only evaluation. Further results show promising research direction in building better VLP models. Our data and code are available at: https://github.com/om-ai-lab/VL-CheckList.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    DO-Bench is a controlled benchmark that attributes VLM object hallucination errors to textual prior pressure, perceptual limits, or their interaction via two diagnostic dimensions and metrics.

  2. HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

    cs.CV 2026-07 unverdicted novelty 6.0

    HyFL-CLIP distills Euclidean CLIP alignment into hyperbolic space using cross-manifold similarity and Einstein midpoint aggregation to capture hierarchical part-whole relations, achieving up to 19.5% gains in long-tex...

  3. All in One: A Unified Synthetic Data Pipeline for Multimodal Video Understanding

    cs.CV 2026-04 unverdicted novelty 6.0

    A unified synthetic data generation pipeline produces unlimited annotated multimodal video data across multiple tasks, enabling models trained mostly on synthetic data to generalize effectively to real-world video und...

  4. Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

    cs.CV 2026-06 unverdicted novelty 5.0

    MACCO applies cross-modal masked reconstruction of compositional concepts with inter- and intra-modal auxiliary objectives to improve visio-linguistic compositionality in VLMs.

  5. Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding

    cs.LG 2026-04 unverdicted novelty 5.0

    Using lexical concreteness to guide contrastive negative mining and a new margin-based Cement loss, the Slipform framework reaches state-of-the-art on compositional benchmarks for vision-language models.