REVIEW 30 cited by
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.
Forward citations
Cited by 30 Pith papers
-
WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation
WaveZip, a training-free wavelet method for video token condensation, reports about 10x token reduction while retaining roughly 98-99% accuracy on multi-benchmark LVLM evaluation.
-
Testing chatbots on the creation of encoders for audio conditioned image generation
All chatbot-designed audio encoders failed to align with CLIP text embeddings and produced incoherent images, while showing a surprising architectural similarity across chatbots.
-
Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation
Screen2AX generates hierarchical macOS accessibility metadata from a screenshot and reports improved GPT-4 UI task success compared with native accessibility and OmniParser V2.
-
PoemTale Diffusion: Minimising Information Loss in Poem to Image Generation with Multi-Stage Prompt Refinement
PoemTale Diffusion generates a coherent set of images from a poem by combining emotion-based segmentation, multi-stage LLM prompt refinement, and consistent self-attention, outperforming direct poem-to-image approache...
-
Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities
Vision-language models consistently recognize unsafe content better from text than from images, and a simplified reinforcement learning fine-tune narrows that gap.
-
Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models
Video-LLMs can be trained, via SFT or DPO on a new synthetic dataset UVQA, to refuse questions that cannot be answered from the video content, with modest cost to answerable QA performance.
-
RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction
RICO refines image captions by reconstructing them into images with a text-to-image model and asking GPT-4o to fix discrepancies against the original, iteratively, with a DPO-distilled fast variant.
-
Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Method
OmniVQA is a first open-source dataset and benchmark for 360-degree visual question answering, and 360-R1 uses GRPO with three LLM-based rewards to improve an existing multimodal model on it.
-
The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model
CLIP embeddings are modeled as a mixture of von Mises-Fisher distributions on the unit sphere, improving out-of-distribution detection and semantic decomposition over single-Gaussian baselines.
-
Multilingual Training and Evaluation Resources for Vision-Language Models
Releases regenerated multilingual training data and translated benchmarks for VLMs in five languages and demonstrates consistent benefits from multilingual training over English-only baselines.
-
Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
CoCoA forces an MLLM to reconstruct masked text through a single EOS token, improving multimodal embedding quality on MMEB-V1 and matching MoCa at 3B with far less pretraining data.
-
Invisible Injections: Exploiting Vision-Language Models Through Steganographic Prompt Embedding
Steganographic prompt injection is reported to covertly manipulate vision-language models with up to 31.8% success, but the evidence is not reproducible.
-
Affect-aware Cross-Domain Recommendation for Art Therapy via Music Preference Elicitation
A 200-person study of music-driven cross-domain recommendation for art therapy shows music-based and visual-based engines perform equally, contradicting the paper's 'outperforming' claim.
-
CLIP-RL: Surgical Scene Segmentation Using Contrastive Language-Vision Pretraining & Reinforcement Learning
A CLIP-based encoder with RL residual refinement and curriculum learning reaches 81% mIoU on EndoVis 2018 and 74.12% on EndoVis 2017 surgical segmentation.
-
AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluation
A finetuned vision-language model jointly predicts nine aspect scores and written comments for AI-generated videos, with a new benchmark and claims of state-of-the-art alignment with human judgment.
-
Graph-MLLM: Harnessing Multimodal Large Language Models for Multimodal Graph Learning
A unified comparison across six multimodal graph datasets shows that fine-tuned multimodal LLMs used as direct predictors achieve the highest node classification accuracy, even without graph structure input.
-
EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models
EfficientVLA combines LLM layer pruning, task-aware visual token selection, and diffusion-head feature caching to cut CogACT's inference cost to 28.9% of baseline FLOPs with a 0.6% SIMPLER success drop.
-
CF-VLM:CounterFactual Vision-Language Fine-tuning
CF-VLM fine-tunes VLMs on counterfactual image-text pairs with three objectives, reporting gains on compositional reasoning benchmarks and modest hallucination reductions.
-
SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
SOLVE couples a vision-language model and an end-to-end planner via a shared encoder and a trajectory chain-of-thought, reporting small but state-of-the-art open-loop planning gains on nuScenes.
-
Diff-ID: Identity Consistent Facial Image Generation and Morphing via Diffusion Models
Diff-ID trades a bit of ArcFace identity score for much lower FID, yielding the best FS/FID trade-off among tested face generators, plus qualitative morphing without per-identity fine-tuning.
-
Effectively obtaining acoustic, visual and textual data from videos
A video-processing pipeline created a 2.24 million-sample audio-image-text dataset, with text captions generated by BLIP from video frames.
-
CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering
A specialist classifier feeding a pruned VLM with knowledge-graph grounding reports 82.1% diagnostic accuracy on a 39-image dermatology test set, about 18 percentage points above a fine-tuned VLM baseline.
-
LEARN: A Story-Driven Layout-to-Image Generation Framework for STEM Instruction
A layout-conditioned diffusion model, fine-tuned on book covers with contrastive and semantic losses, generates story-like STEM illustrations.
-
Visual Language Models as Zero-Shot Deepfake Detectors
Zero-shot VLMs scored by normalized yes/no token probabilities beat most trained deepfake detectors on a new SimSwap dataset, and a lightly fine-tuned InstructBLIP is near-perfect on DFDC-P.
-
Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learning
A video-language model that combines adaptive frame sampling, explicit timestamps, and a reinforcement-learning reward for refusing irrelevant queries, beating prior methods on QVHighlights by about 3.5%.
-
From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challenge
A challenge report showing that MLLM-generated scene graphs and ConceptNet knowledge graphs each give small accuracy gains over a video-only baseline, and that per-category selection reaches 44.21% on the HD-EPIC VQA ...
-
Seamless and Efficient Interactions within a Mixed-Dimensional Information Space
A thesis that three design strategies, multimodal AI, context-aware placement, and combined 2D/3D views, make mixed-dimensional information spaces seamless and efficient, demonstrated with three systems.
-
Investigating the Invertibility of Multimodal Latent Spaces: Limitations of Optimization-Based Methods
Optimization can force BLIP, Flux, Whisper, and Chatterbox to hit textual targets, but the inverted inputs are perceptually incoherent and the recovered text embeddings are semantically meaningless.
-
E.A.R.T.H.: Structuring Creative Evolution through Model Error in Generative AI
A five-stage pipeline that induces, scores, rewrites, and validates model errors reports large creativity gains that largely arise from selection on the measurement metric itself.
-
Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model
SegVLM reports 53.87 IoU on PhraseCut referring segmentation by adding SE blocks, deformable convolutions, residual shortcuts, and a fused BCE-Focal-Dice loss to CRIS.
Discussion (0). Sign in to comment.