REVIEW 7 cited by
How Much Can CLIP Benefit Vision-and-Language Tasks?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.
Forward citations
Cited by 7 Pith papers
-
Visual Question Answering on Multiple Remote Sensing Image Modalities
A new remote sensing VQA dataset with RGB, multispectral, and SAR modalities, plus a VisualBERT baseline, reaches 65.56% average accuracy and shows that multi-modal context helps.
-
LSDM: LLM-Enhanced Spatio-temporal Diffusion Model for Service-Level Mobile Traffic Prediction
LSDM predicts next-hour mobile traffic per app category by feeding a diffusion model with satellite imagery, POI counts, and LLM-generated text descriptions, outperforming eight baselines on a single real-world dataset.
-
(Almost) Free Modality Stitching of Foundation Models
A hypernetwork that generates connector weights for all image-text model pairs can rank pairs like grid search at about 10x lower training cost, but the best connector lags grid search by a few points.
-
Multimodal Plant Root Phenotyping with Integration of 3D Skeleton Extraction and Language Analysis
An unsupervised 3D root skeleton extractor plus evidence-first GPT-4o fine-tuning is claimed to improve root phenotyping VQA accuracy on a private 12-species dataset.
-
ReVision : A Post-Hoc, Vision-Based Technique for Replacing Unacceptable Concepts in Image Generation Pipeline
ReVision uses a vision-language model's bounding box to gate attention-based image editing, suppressing unsafe concepts while better preserving benign background in multi-concept scenes.
-
A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications
A survey that classifies visual semantic communication into preservation, expansion, and refinement categories and reviews their machine-learning components and applications.
-
Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward
A survey paper reviews multimodal generative AI and autoregressive LLMs for text-driven human motion generation, with comparative tables of models, datasets, and metrics.
Discussion (0). Continue with ORCID to comment.