REVIEW 6 cited by
Do Vision-Language Models Really Understand Visual Language?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Visual language is a system of communication that conveys information through symbols, shapes, and spatial arrangements. Diagrams are a typical example of a visual language depicting complex concepts and their relationships in the form of an image. The symbolic nature of diagrams presents significant challenges for building models capable of understanding them. Recent studies suggest that Large Vision-Language Models (LVLMs) can even tackle complex reasoning tasks involving diagrams. In this paper, we investigate this phenomenon by developing a comprehensive test suite to evaluate the diagram comprehension capability of LVLMs. Our test suite uses a variety of questions focused on concept entities and their relationships over a set of synthetic as well as real diagrams across domains to evaluate the recognition and reasoning abilities of models. Our evaluation of LVLMs shows that while they can accurately identify and reason about entities, their ability to understand relationships is notably limited. Further testing reveals that the decent performance on diagram understanding largely stems from leveraging their background knowledge as shortcuts to identify and reason about the relational information. Thus, we conclude that LVLMs have a limited capability for genuine diagram understanding, and their impressive performance in diagram reasoning is an illusion emanating from other confounding factors, such as the background knowledge in the models.
Forward citations
Cited by 6 Pith papers
-
ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams
VLMs recover common ERD elements at F1>0.74 but drop to 0.07–0.28 on N-ary relationships, multivalued attributes, and weak entities; reasoning models gain 15–25% yet stay prior- and complexity-sensitive.
-
Prior Bias in Vision Language Models on UML Diagram Interpretation
Reversing only the UML relation arrow while keeping class names and layout fixed cuts open-source VLM relation accuracy by about 33%, revealing prior-over-vision bias.
-
Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models
Vision-language models can handle some 2D shape puzzles but nearly all fail at multi-step 3D spatial deformation reasoning.
-
ViStruct: Simulating Expert-Like Reasoning Through Task Decomposition and Visual Attention Cues
ViStruct automatically breaks visualization questions into ordered subtasks tied to highlighted chart regions, imitating expert analysis strategies for chart reading.
-
Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
Prompt specificity measurably affects counting accuracy and attention allocation in Qwen2.5-VL and Kimi-VL, and can partially overcome learned visual priors.
-
WIP: Large Language Model-Enhanced Smart Tutor for Undergraduate Circuit Analysis
A GPT-4o-based smart tutor with problem-specific documents provided homework feedback in a circuit analysis course, and 90.9% of 66 student feedback responses were positive.
Discussion (0). Continue with ORCID to comment.