REVIEW 34 cited by
PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
read the original abstract
Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited. To close this gap, we introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks. PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions. Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world -- likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors. To tackle the shortfall, we introduce PhysAgent, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results demonstrate that enhancing VLMs' physical world understanding capabilities can help embodied agents such as MOKA. We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding.
Forward citations
Cited by 34 Pith papers
-
Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
A law-grounded benchmark, Apple-PI, grades video models stage-by-stage on physics reasoning and finds they top out at 0.473, well short of reliable simulation.
-
RetroHolmes: When Semantic Plausibility Fails Retrospective Physical Process Reasoning
Current VLMs fail retrospective physical reachability and causal reconstruction on RetroHolmes; a simple analysis-by-synthesis loop with video simulation reduces bias and belief-conflict sensitivity.
-
Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks
Frontier agentic AI models consistently fail at computational imaging tasks requiring physics-aware inversion, producing visually plausible but physically incorrect outputs.
-
ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?
ChronoPhyBench is a new benchmark and dataset for chronological physical dynamics reasoning that combines video-conditioned next-state prediction with VQA to reduce language bias in MLLM evaluation.
-
Physics in 2-Steps: Locking Motion Priors Before Visual Refinement Erases Them
PhaseLock extracts motion priors from 2-step inference and enforces them via Latent Delta Guidance to raise physical consistency scores by 6.2 points on average in image-to-video diffusion models.
-
Benchmarking Single-Factor Physical Video-to-Audio Generation
FlatSounds benchmark shows state-of-the-art V2A models rely more on text captions than visual input for physical and semantic accuracy, with captions improving correctness but degrading temporal alignment.
-
World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models
WMW audits VLMs by requiring typed physical state-transition traces and using a verifier to detect inconsistencies missed by answer-only evaluation, with TraceBank as a released resource of synthetic scenarios.
-
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
ESI-Bench shows active exploration outperforms passive observation in multimodal LLMs on spatial tasks but reveals failures from poor action choices and overconfident belief commitment unlike humans.
-
Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs
Temporal information in Video-LLMs is encoded well by video-centric encoders but disrupted by standard projectors; time-preserved MLPs plus AoT supervision yield 98.1% accuracy on arrow-of-time and gains on other temp...
-
Grounding Video Reasoning in Physical Signals
A new benchmark converts video clips into shared grounded event records and tests models across physics, semantic, and control prompts under original, shuffled, ablated, and masked conditions, finding selective robust...
-
OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving
OptiVerse is a new benchmark spanning neglected optimization domains that shows LLMs suffer sharp accuracy drops on hard problems due to modeling and logic errors, with a Dual-View Auditor Agent proposed to improve pe...
-
SCP: Spatial Causal Prediction in Video
SCP defines a new benchmark task for predicting spatial causal outcomes beyond direct observation and shows that 23 leading models lag far behind humans on it.
-
V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions
V-REX shows that VLMs' multi-step visual exploration can be measured separately as planning (choosing sub-questions) and following (answering them), and that planning is the key bottleneck.
-
Abstract 3D Perception for Spatial Intelligence in Vision-Language Models
SandboxVLM enhances VLMs' spatial intelligence by encoding 3D geometry with abstract bounding boxes in a four-stage zero-shot pipeline, yielding an 8.3% improvement on SAT Real benchmark.
-
RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation
RigPI combines VLM semantic priors with two-stage gradient optimization in differentiable simulation to identify inertial and frictional parameters of rigid bodies from robot-object interactions.
-
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Introduces CausalPhys benchmark with causal graphs and CRFT fine-tuning to improve VLMs' causal physical reasoning accuracy and interpretability.
-
$\Delta$ynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos
A vision-language framework generates text-based rigid-body scene configurations from videos using motion reasoning and optical flow, reporting 0.30 IoU on CLEVRER (7x over baselines) and transfer to 235 real videos.
-
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
ESI-Bench is a new benchmark for embodied spatial intelligence with 10 task categories on OmniGibson that requires agents to actively explore via perception, locomotion, and manipulation, revealing that MLLMs suffer f...
-
GeoWorld-VLM: Geometry from World Models for Vision-Language Models
GeoWorld-VLM aligns VLM image features with intermediate representations from camera-conditioned world models via fine-tuning only the encoder and projector, yielding ~4% gains on What'sUp and VSR spatial benchmarks a...
-
Quantitative Video World Model Evaluation for Geometric-Consistency
PDI-Bench computes 3D projective residuals from segmented and tracked points to quantify geometric inconsistency in AI-generated videos.
-
From Priors to Perception: Grounding Video-LLMs in Physical Reality
Video-LLMs fail physical reasoning due to semantic prior dominance rather than perception deficits; a new programmatic adversarial curriculum and visual-anchored reasoning chain enable substantial gains via standard L...
-
Dual-Cluster Memory Agent: Resolving Multi-Paradigm Ambiguity in Optimization Problem Solving
DCM-Agent improves LLM performance on multi-paradigm optimization problems by 11-21% via dual-cluster memory construction and dynamic inference guidance.
-
Multimodal Language Models Cannot Spot Spatial Inconsistencies
Multimodal LLMs significantly underperform humans at spotting objects that break 3D consistency in multi-view image pairs.
-
CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates
VLMs perform near chance at detecting and correcting an erroneous step in visual sequential planning, and incrementally updating scene graphs step-by-step (SGI) yields small, partially inconsistent gains.
-
SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios
SWITCH introduces a 193-video benchmark of tangible control-interface interactions and shows that frontier LMMMs struggle with fine-grained grounding and outcome verification.
-
OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping
OmniView-Space framework with MPSM, tool-guided reasoning, and distillation achieves SOTA on spatial reasoning benchmarks for MLLMs while reducing external geometry dependencies.
-
RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation
RigPI combines VLM initialization with two-stage gradient-based optimization in differentiable simulation to estimate dynamic parameters of rigid bodies from real robot interactions.
-
Physically Viable World Models: A Case for Query-Conditioned Embodied AI
Embodied AI requires query-conditioned world models that select the simplest physical abstraction sufficient to answer intervention queries.
-
GeoWorld-VLM: Geometry from World Models for Vision-Language Models
GeoWorld-VLM distills geometric structure from camera-conditioned world models into VLMs by aligning visual features, improving spatial reasoning by about 4% on What'sUp and VSR benchmarks across two architectures whi...
-
PhysBrain 1.0 Technical Report
PhysBrain 1.0 extracts scene elements, spatial dynamics, actions and depth relations from human egocentric video to create QA supervision for VLMs, then transfers the resulting physical priors to VLA policies via capa...
-
Dual-Cluster Memory Agent: Resolving Multi-Paradigm Ambiguity in Optimization Problem Solving
DCM-Agent improves LLM optimization solving by 11–21% on seven benchmarks via dual-cluster memory of Approaches, Checklists, and Pitfalls plus adaptive path switching.
-
Agentic Physical AI toward a Domain-Specific Foundation Model for Nuclear Reactor Control
A compact language model trained on scaled synthetic nuclear reactor control data exhibits variance collapse and emergent concentration on a single actuation strategy driven by physical execution success.
-
MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models
MASS adds spatiotemporal motion signals and 3D grounding to VLMs and releases MASS-Bench, yielding physics-reasoning performance within 2% of Gemini-2.5-Flash after reinforcement fine-tuning.
-
Evidence of a Cognitive Shift in AI Education: How Students Are Rethinking Human Intelligence?
Longitudinal poll data from 471 students in AI courses shows a shift toward preferring human intelligence, reaching 65% in technical courses and 90% in design courses by 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.