Pith

open record

sign in

arxiv: 2503.09837 · v2 · pith:EGOL54Y7 · submitted 2025-03-12 · cs.CV · cs.AI· cs.CL

On the Limitations of Vision-Language Models in Understanding Image Transforms

Ahmad Mustafa Anis , Hasnain Ali , Saquib Sarfraz This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:EGOL54Y7record.jsonopen to challenge →

classification cs.CV cs.AIcs.CL
keywords imagemodelsunderstandingdownstreamimage-leveltaskstransformationsvideo
0
0 comments X
read the original abstract

Vision Language Models (VLMs) have demonstrated significant potential in various downstream tasks, including Image/Video Generation, Visual Question Answering, Multimodal Chatbots, and Video Understanding. However, these models often struggle with basic image transformations. This paper investigates the image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by Google. Our findings reveal that these models lack comprehension of multiple image-level augmentations. To facilitate this study, we created an augmented version of the Flickr8k dataset, pairing each image with a detailed description of the applied transformation. We further explore how this deficiency impacts downstream tasks, particularly in image editing, and evaluate the performance of state-of-the-art Image2Image models on simple transformations.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks

    cs.CV 2026-06 unverdicted novelty 7.0

    The paper constructs the VIBE benchmark and evaluates six visual in-context learning models on 14 datasets, 12 tasks, and 106 combinations under a unified one-shot protocol, revealing limitations and failure modes.

  2. Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspective

    cs.CV 2026-04 unverdicted novelty 6.0

    TaCo contrastively embeds semantic, generative, and transformation tasks from medical imaging into a joint space to reveal which tasks cluster, blend, or remain distinct.

  3. Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks

    cs.CV 2026-04 unverdicted novelty 4.0

    Robust CLIP models amplify vulnerabilities to natural adversarial scenarios while standard CLIP shows large performance drops on natural language-induced adversarial examples in zero-shot classification, segmentation,...