REVIEW 16 cited by
PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Current benchmarks for evaluating the reasoning capabilities of Large Language Models (LLMs) face significant limitations: task oversimplification, data contamination, and flawed evaluation items. These deficiencies necessitate more rigorous assessment methods. To address these limitations, we introduce PHYBench, a benchmark of 500 original physics problems ranging from high school to Physics Olympiad difficulty. PHYBench addresses data contamination through original content and employs a systematic curation pipeline to eliminate flawed items. Evaluations show that PHYBench activates more tokens and provides stronger differentiation between reasoning models compared to other baselines like AIME 2024, OlympiadBench and GPQA. Even the best-performing model, Gemini 2.5 Pro, achieves only 36.9% accuracy compared to human experts' 61.9%. To further enhance evaluation precision, we introduce the Expression Edit Distance (EED) Score for mathematical expression assessment, which improves sample efficiency by 204% over binary scoring. Moreover, PHYBench effectively elicits multi-step and multi-condition reasoning, providing a platform for examining models' reasoning robustness, preferences, and deficiencies. The benchmark results and dataset are publicly available at https://www.phybench.cn/.
Forward citations
Cited by 16 Pith papers
-
Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging
A 57-task, expert-verified benchmark shows frontier LLM coding agents systematically fail on physical conventions, inverse-solver choice, and end-to-end imaging pipelines.
-
Agentic Exploration of Physics Models
A general-purpose LLM agent can discover physics models, including ODEs and spin Hamiltonians, by autonomously choosing experiments and fitting hypotheses to numeric data.
-
On Path to Multimodal Historical Reasoning: HistBench and HistAgent
HistAgent, a history-specialized agent, scores 27.54% pass@1 and 36.47% pass@2 on the new 414-question HistBench benchmark, surpassing generalist agents tested on the same data.
-
Vision Language Models Cannot Reason About Physical Transformation
Current VLMs cannot maintain transformation-invariant representations of number, length, volume or size and instead rely on textual invariance priors that reverse on matched non-conserving controls.
-
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...
-
CLVisc Agent for autonomous relativistic hydrodynamics studies
An LLM agent autonomously created a CLVisc skill and ran two hydrodynamic studies, finding that the high-temperature branch of η/s dominates flow suppression and that PGCM-uniform 16O decouples ellipticity from size.
-
ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems
A new physics benchmark with static and dynamically varied numeric problems shows top LLMs solve at most 43 percent of the static set and drop sharply when problem constants change.
-
OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation
OneIG-Bench introduces a 2,440-prompt, six-dimension benchmark with automated metrics for text-to-image models, covering alignment, text, reasoning, style, and diversity in English and Chinese.
-
Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment
Post-training alignment on synthetic graph data yields statistically significant gains on some real-world implicit-graph tasks, but gains are inconsistent and multi-step reasoning remains fragile.
-
PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models
A new benchmark of 380 principle-based physics problems shows that state-of-the-art LLMs struggle to apply symmetry, conservation, and dimensional-analysis shortcuts, achieving under 50 percent average accuracy with h...
-
PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
PhyX is a new 3,000-question visual physics benchmark; the best AI model tested scores 45.8 percent, well below the 75.6 to 78.9 percent of a small human student sample.
-
PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions
PhysicsArena introduces a 5,103-instance multimodal physics benchmark that separately scores variable identification, process formulation, and solution derivation in MLLMs.
-
lmgame-Bench: How Good are LLMs at Playing Games?
lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.
-
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...
-
SciDA: Scientific Dynamic Assessor of LLMs
SciDA is a dynamically initialized, multi-discipline olympiad benchmark that shows LLMs perform substantially worse when problem variables are randomized, which the authors attribute to memorization of fixed numerical...
-
KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation
KORGym introduces a 51-game, text and visual, multi-turn benchmark with a normalized scoring scheme, and uses it to compare 19 LLMs and 8 VLMs on six reasoning dimensions.
Discussion (0). Continue with ORCID to comment.