REVIEW 4 cited by
What does CLIP know about a red circle? Visual prompt engineering for VLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large-scale Vision-Language Models, such as CLIP, learn powerful image-text representations that have found numerous applications, from zero-shot classification to text-to-image generation. Despite that, their capabilities for solving novel discriminative tasks via prompting fall behind those of large language models, such as GPT-3. Here we explore the idea of visual prompt engineering for solving computer vision tasks beyond classification by editing in image space instead of text. In particular, we discover an emergent ability of CLIP, where, by simply drawing a red circle around an object, we can direct the model's attention to that region, while also maintaining global information. We show the power of this simple approach by achieving state-of-the-art in zero-shot referring expressions comprehension and strong performance in keypoint localization tasks. Finally, we draw attention to some potential ethical concerns of large language-vision models.
Forward citations
Cited by 4 Pith papers
-
LPOI: Listwise Preference Optimization for Vision Language Models
LPOI reduces VLM hallucination by training the model to prefer the original image over progressively masked versions of the same image, using a listwise ranking loss built from pairwise preference data.
-
Decouple before Align: Visual Disentanglement Enhances Prompt Tuning
Decoupling images into foreground and background before aligning them with text improves CLIP prompt tuning on few-shot and generalization benchmarks.
-
ConText: Driving In-context Learning for Text Removal and Segmentation
ConText is the first visual in-context learning model for text removal and segmentation, chaining the two tasks and using self-prompting to reach new state-of-the-art scores.
-
Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?
A new benchmark and method (Spot-IT) aim to improve multimodal LLMs' ability to locate fine details in documents, with reported significant gains.
Discussion (0). Continue with ORCID to comment.