REVIEW 21 cited by
An Introduction to Vision-Language Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.
Forward citations
Cited by 21 Pith papers
-
GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks
Graph-as-Policy (GaP) multi-agent harnesses generate and self-refine robot skill graphs that outperform VLA and TAMP baselines on eight variational automation benchmarks.
-
Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability
Telescope Perplexity, the average negative log probability a reference LM assigns to each token immediately after seeing it, yields strong zero-shot LLM-text detection by probing an early-training aversion to repetition.
-
In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models
VLMs can run Picbreeder but produce less refined, more mode-collapsed archives than humans; modest selection noise, short context, and many prompted personalities improve diversity metrics at quality cost.
-
Beyond Invisibility: Learning Robust Visible Watermarks for Stronger Copyright Protection
HARVIM learns watermark placement to maximize reconstruction error under an inpainting-based removal model, showing modest gains over random watermarks.
-
RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
Expert-curated modular Robot-VQA benchmark of 474 questions across 39 robot tasks shows SOTA VLMs have large gaps that correlate with physical robot execution.
-
Multimodal Benchmark for Safety Assessment in Industrial Inspection Scenarios
A real-world multimodal benchmark with 5,013 annotated inspection instances and a safety-assessment evaluation of 15 vision-language models.
-
Controlling Multimodal LLMs via Reward-guided Decoding
MRGD guides MLLM decoding with a learned hallucination reward and a detector-based recall reward, allowing users to trade off object precision, recall, and test-time compute while reducing object hallucinations on CHA...
-
Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models
Vision-language models can handle some 2D shape puzzles but nearly all fail at multi-step 3D spatial deformation reasoning.
-
IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments
IntPhys 2 evaluates models on permanence, immutability, continuity, and solidity in synthetic videos, and finds most models near chance while humans near perfect.
-
Rethinking Machine Unlearning in Image Generation Models
A new taxonomy and multi-aspect evaluation framework for image generation unlearning, with a curated dataset, shows that ten existing unlearning methods perform poorly on preservation and robustness.
-
Circuit Stability Characterizes Language Model Generalization
Circuit stability, measured as rank correlation between soft circuits across subtasks, is proposed as a predictor of language model generalization.
-
Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering
A training-free pipeline constructs a spatiotemporal knowledge graph from egocentric video, enabling low-latency, explainable embodied question answering.
-
Participatory AI: A Scandinavian Approach to Human-Centered AI
Participatory AI applies five Scandinavian Participatory Design principles to four AI design challenges, illustrated through five diverse case studies.
-
PB-IAD: Utilizing multimodal foundation models for semantic industrial anomaly detection in dynamic manufacturing environments
With carefully layered prompts and one or three reference samples, GPT-4.1 detects anomalies in cable images and crimp-force features at F1 levels that PatchCore and Isolation Forest reach only after training on dozen...
-
Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models
VLMs score far below adult humans on a new 13,188-question benchmark of 36 atomic 2D geometry perception skills.
-
Towards channel foundation models (CFMs): Motivations, methodologies and opportunities
A survey and position paper proposing channel foundation models, with experiments on two pretrained CSI models showing gains over a vanilla ViT baseline.
-
Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions
Zero-shot vision language models achieve moderate accuracy on five-class academic facial expression recognition, with happy easiest and distracted almost never detected.
-
DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
DynTok dynamically merges similar adjacent visual tokens into groups, reducing video token counts to 44.4% with comparable or better video understanding accuracy.
-
Learning Sparsity for Effective and Efficient Music Performance Question Answering
Sparsify reports state-of-the-art accuracy on Music AVQA benchmarks by borrowing three existing sparsification techniques, cutting training time by 28% and retaining 70-80% of accuracy on a 25% data subset.
-
Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks
Qwen2-VL and Qwen3-VL beat 2022 ImageArg baselines in zero-shot Logos and Ethos F1, but Qwen3 falls below baseline on Pathos despite the abstract's claim.
-
Can Vision Transformers with ResNet's Global Features Fairly Authenticate Demographic Faces?
An empirical comparison of three ViT backbones with ResNet for few-shot demographic face authentication reports Swin Transformer as best, but the fairness conclusion is not supported by the experimental design.
Discussion (0). Continue with ORCID to comment.