Pith. sign in

REVIEW 4 major objections 6 minor 10 cited by

PhyX: Does Your Model Have the "Wits" for Physical Reasoning?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces PhyX, a 3,000-question multimodal benchmark built to test whether AI models can reason about physics from realistic images, and reports that every model it evaluated falls far short of human physics students, with…

desk verdict A useful benchmark idea with a clean text-de-redundancy design, but the headline human-model gap rests on an answer-extraction pipeline that can silently discard correct responses. read the letter →

arxiv 2505.15929 v2 pith:DSCWIYEF submitted 2025-05-21 cs.AI

Hui Shen , Taiqiang Wu , Qi Han , Yunta Hsieh , Jizhou Wang , Yuyue Zhang , Yuxin Cheng , Zijian Hao
show 11 more authors
Yuansheng Ni Xin Wang Zhongwei Wan Kai Zhang Wendong Xu Jing Xiong Ping Luo Wenhu Chen Chaofan Tao Zhuoqing Mao Ngai Wong
This is my paper · ORCID
classification cs.AI
keywords physicalreasoningmultimodalbenchmarklargelanguagemodelsvisualquestionansweringphysicsproblemsolvingevaluationchain-of-thoughtLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing benchmarks measure knowledge recall and symbolic math but miss physical reasoning: the ability to read a realistic visual scenario, extract implicit physical conditions, apply the right laws, and compute a grounded answer. To make that measurable, it introduces PhyX, 3,000 expert-validated multimodal questions spanning six physics domains and six reasoning types, with each question stripped of the text-image redundancy that lets models rely on only one modality. On a 1,000-question test subset, all 16 evaluated models scored far below three groups of physics students; the strongest multimodal model, GPT-o4-mini, reached 45.8%, while the weakest student group scored 75.6%. The authors take this roughly 30-point gap as evidence that current models lean on memorized formulas, textual hints, and surface visual patterns instead of genuine physical understanding. If the benchmark is valid, it provides a diagnostic that separates reasoning about the physical world from other forms of machine intelligence.

What carries the argument

The load-bearing mechanism is the benchmark's de-redundancy design. Every PhyX question exists in three versions — Full-Text, Text-DeRedundancy (which strips whatever the image already conveys), and Text-Minimal (question only) — so that a correct answer requires integrating both modalities instead of matching surface cues from either alone. The claim that models fail at genuine physical reasoning is carried by the accuracy drop along this text-reduction gradient, by the six-type reasoning taxonomy used to categorize questions, and by a three-step evaluation pipeline (chain-of-thought prediction, rule-based answer extraction, then a DeepSeek-V3 judge reported to agree with human grading on more than 99% of a 200-example pilot) that is meant to make the measured accuracies trustworthy.

What would settle it

Recruit a substantially larger panel of independent physics graduate students, have each answer more than 18 questions drawn evenly from the full 3,000-question set, and compare. If the worst-expert accuracy falls near or below 50% on the same judged rubrics, the claimed 30-point human-model gap collapses; alternatively, if the testmini's domain and reasoning-type distribution differs measurably from the full set, the reported model scores are not a valid estimate for the benchmark as a whole.

Watch

Extended reading notes

Core claim

The central claim is that PhyX is the first large-scale benchmark able to measure physics-grounded reasoning in visual scenarios, and that state-of-the-art multimodal models fail it by a wide margin. On the 1,000-question testmini subset, the best multimodal model (GPT-o4-mini) scores 45.8%, below the best text-only LLM (DeepSeek-R1 at 51.2%) and far below the worst of three human-expert groups at 75.6%; the gap persists across all six domains and reaches roughly 46 points in Modern Physics (human 86.7% versus model 40.6%). An error analysis of 96 GPT-4o failures attributes 39.6% to visual reasoning errors, 38.5% to missing knowledge, 13.5% to text reasoning errors, and 8.3% to calculation slips, and accuracy falls monotonically as the textual description is removed, showing that models lean on redundant text rather than the image. The paper concludes that physical reasoning is a distinct capability that is harder than the mathematical reasoning measured by benchmarks like MathVista and MATH-V, and that current models have not acquired it.

Load-bearing premise

The headline human-versus-model gap rests on the performance of just 15 physics students, each answering 18 questions in groups of five, and on the assumption that the 1,000-question testmini subset fairly represents the full 3,000-question benchmark; if the students were not genuinely expert, or the sample questions were unrepresentative, the gap could shrink or disappear even while model accuracies remain low.

Editorial extensions

If this is right

  • Open-ended questions separate models far more sharply than multiple-choice questions, so multiple-choice evaluations of physics reasoning systematically overstate model competence.
  • Because the worst human-expert group outperformed every model in every domain, current multimodal systems should not be treated as reliable for university-level physics tutoring, analysis, or scientific assistance.
  • The error profile — visual reasoning 39.6%, missing knowledge 38.5%, text reasoning 13.5%, calculation 8.3% — locates the bottleneck in visual grounding and physics knowledge rather than arithmetic.
  • Text-only LLMs fed image captions rival the best multimodal models, showing that much of the apparent visual competence of MLLMs can be replicated by a captioning pipeline.
  • Comparing GPT-4o at 32.5% on PhyX against its reported 63.8% on MathVista and MATH-V indicates that physical reasoning is a harder test of foundation models than mathematical reasoning in visual contexts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The de-redundancy protocol is a general-purpose modality-lock test: it could be applied to any multimodal benchmark (medicine, navigation, engineering) to detect when a model is pattern-matching one modality rather than integrating evidence from both.
  • The authors note that the images are schematic textbook-style scenes rather than photographs; a photo-realistic variant might change the human-model gap in either direction — humans could improve with richer detail while models could degrade on noisy natural images — so the 30-point gap should not be assumed to transfer to real-world perception.
  • The six-type taxonomy could be used predictively: a model's per-type accuracy profile may indicate which reasoning weaknesses stem from missing physics knowledge versus weak visual encoding, suggesting targeted training data rather than general scaling.
  • A testable extension of the error analysis: train a model to correct its own visual-reading errors on PhyX examples and measure whether gains transfer to the other five reasoning types; the 39.6% visual error share predicts that fixing perception would lift every category.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces PhyX, a multimodal physics benchmark of 3,000 unique questions (with a 1,000-question testmini subset) spanning six physics domains and six reasoning types, each question provided in multiple-choice and open-ended forms. The authors evaluate 16 LLMs and MLLMs under three textual input conditions and report that the worst group of human physics students achieves 75.6% accuracy on open-ended text-de-redundancy questions while the best MLLM, GPT-o4-mini, achieves 45.8%. They interpret this gap as evidence that current models lack robust physical reasoning, and they provide domain-level breakdowns, an error taxonomy, case studies, and an evaluation protocol compatible with VLMEvalKit.

Significance. If the headline comparison is valid, PhyX would be a useful diagnostic benchmark for a genuinely under-tested capability: integrating visual perception, physical laws, and symbolic computation. The paper's strengths are its broad domain coverage, the paired MC/OE design, the three textual-variation conditions, and the explicit error taxonomy with case studies. However, the benchmark's utility depends on two load-bearing assumptions that are not adequately supported: that open-ended model answers are scored fairly, and that the small human sample provides a reliable expert baseline. The paper also contains concrete data-quality errors in its own illustrative examples, which undermines confidence in the expert-validation claim. The central direction is defensible, but the quantitative headline needs substantial additional support.

major comments (4)
  1. [§3.2, Appendix D.2, Figure 17] The open-ended answer extraction regex only accepts answers introduced by 'final answer:' or 'correct answer:' and has no fallback to the full raw prediction. A correct answer written as a bare numeric value, as 'Therefore, the heat transfer is -0.0147 kJ.', or in any other natural format is discarded and almost certainly scored as wrong. This is asymmetric with the human baseline, which was presumably hand-graded, and it can systematically depress all model OE scores. The magnitude of the risk is visible in Table 2: GPT-o4-mini drops from 86.9% on the MC version to 45.8% on the OE version. The reported '>99%' judge accuracy from a 200-example preliminary study validates the LLM judge on inputs already produced by the extraction step; it does not establish that extraction succeeds on a large fraction of correct answers. The authors should report the extraction failure rate per model, provide a fallback that evaluates the full prediction when no marker phrase is found, and re-run the OE evaluation before the headline gap is taken as established.
  2. [§3.1, Table 3] The human expert baseline rests on only 15 students, each answering 18 questions, for a total of 270 human responses. The three reported groups are partitions of five students, so the per-domain percentages in Table 3 are based on roughly 13–17 answers per cell; for example, the worst-human Electromagnetism value of 60.0% is consistent with 9 correct out of 15. The 95% confidence interval for 60% on 15 trials is roughly 32% to 84%, which overlaps substantially with the best model score of 45.8%. As reported, the data do not statistically support the claim that all human groups outperform all models. The authors should either collect a much larger human sample stratified across the testmini domains or report exact counts and confidence intervals and soften the gap claim accordingly.
  3. [Figure 2] The paper's showcase examples contain incorrect answers. The Optics example asks for the magnification of a 1.00 cm object placed 10.0 cm from a concave mirror of radius 30.0 cm; the correct magnification is 3.0, but the displayed answer is '5.6 kW', which appears to be copied from the Electromagnetism row and is dimensionally impossible for a magnification. The Modern Physics example reports the observed dimensions as '-0.65m', but a dimension cannot be negative and the contracted width should be +0.65 m while the height remains 1.00 m. These errors in the paper's own curated examples directly contradict the claim that scenarios undergo rigorous expert validation and cast doubt on the annotation quality of the full benchmark. The authors should correct the figure and audit the full dataset for this class of answer errors.
  4. [§2.1, Table 1; §2.2] The dataset size is stated inconsistently. Table 1 reports 'Total new questions 6,000' with 'Unique number of questions 3,000', while §2.2 says the process 'results in a diverse collection of 3,300 questions', and the abstract and §1 say 3K questions. Since each physical problem is converted into both an MC and an OE version, the relationship between 'questions', 'problems', and 'instances' needs a precise definition, and the 3,300 figure needs reconciliation. The manuscript also states the data and code 'will be made publicly available' but neither is linked or inspectable in the current version; for a benchmark paper, release of the dataset and the exact evaluation scripts is a condition for verifying the central claims.
minor comments (6)
  1. [§3.1] The testmini subset is described as a 'proportional random sampling strategy', but no comparison between the testmini and full 3,000-question distributions over domains and reasoning types is provided; since all quantitative results are on testmini, the authors should report domain counts and reasoning-type counts for both sets.
  2. [§3.2] The preliminary study of 200 examples for the LLM judge is described in one sentence; the authors should report how the 200 examples were sampled, what the ground-truth judge labels were, and the agreement metric used to justify the '99%' claim.
  3. [§3.5] The error taxonomy (Visual Reasoning Errors, Text Reasoning Errors, Lack of Knowledge, Calculation Error) is said to be produced 'based on GPT-4o', but no human validation of the error labels is reported; a small human-annotated agreement study would make the taxonomy more credible.
  4. [Figure 18 caption] The caption for Figure 18 reads 'Rule-based answer extraction strategy for OE questions', but the content appears to be the prompt for the LLM judge; the figure and caption should be matched.
  5. [§3.3, Table 2] The Open-Ended columns report 'Random Choice - 25' for MC-style random guessing; for open-ended numeric answers random chance is not 25%, so this row should be clarified or omitted for the OE columns.
  6. [Throughout] There are naming inconsistencies between the text and code, such as 'PhyX' vs 'MetaPhyX', and model names such as 'LLaV A-OneVision-7B' with an odd space; these should be normalized.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark study with no fitted inputs or self-citation chain in its central claim.

full rationale

PhyX is a measurement study: it curates 3,000 questions, runs fixed prompts through 16 models, extracts answers with rule-based regexes, and scores them via an LLM judge. The headline gap (human experts 75.6% vs GPT-o4-mini 45.8%) is a reported measurement, not a quantity derived from a fitted parameter or from the benchmark's own definitions. No equation in the paper defines a benchmark metric in terms of the model outputs being predicted; the reasoning-type taxonomies are descriptive labels, and the human baseline is an external, independently sampled performance. GPT-4o is used to generate captions while also being evaluated, and DeepSeek-V3 serves as the LLM judge while also being evaluated, but these choices affect measurement validity rather than circularity: the reported accuracies are not forced by construction, and the paper does not invoke a self-citation or a uniqueness theorem to rule out alternatives. Any concern about regex extraction discarding correct answers or about judge bias is a correctness/evaluation-fairness concern, not a circularity concern under the stated criteria. Therefore no specific circular step can be exhibited with quote evidence from the paper.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims rest on data quality and baseline assumptions, not on fitted parameters or invented entities. The benchmark contains no numerical parameters fitted to data; the human baseline and LLM judge are measurement assumptions rather than free parameters.

assumptions (5)
  • domain assumption The 3,000 PhyX questions are scientifically accurate and correctly labeled by physics Ph.D. students and cross-checking annotators.
    Section 2.2 describes expert annotation but provides no inter-annotator agreement statistics; several visible sample answers in Figure 2 appear wrong, casting doubt on the validation.
  • domain assumption The 1,000-question testmini subset is representative of the full 3,000-question benchmark.
    Section 3.1 says testmini is built by proportional random sampling across domains, but no confidence intervals or repeat-sampling analysis are provided.
  • domain assumption The human baseline from 15 physics students, each answering 18 questions, estimates expert-level performance.
    Section 3.1 reports only 15 students and 18 questions each, split into groups of five. This is too small to support the headline 29% human-model gap.
  • domain assumption DeepSeek-V3 as an LLM judge evaluates open-ended answers with 99% accuracy.
    Section 3.2 claims a preliminary study of 200 examples shows 99% judge accuracy, but the study details, sample, and error categories are not shown.
  • domain assumption Questions were selected so answers are not easily available online, mitigating data contamination.
    Section 2.2 and Appendix F.7 instruct annotators to prefer questions with answers in obscure locations, but no contamination test is run on the released models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PhyX: Does Your Model Have the "Wits" for Physical Reasoning?." pith.science (2026). https://pith.science/paper/DSCWIYEF

@misc{pith2026250515929,
  author       = {Pith},
  title        = {Pith review of: PhyX: Does Your Model Have the "Wits" for Physical Reasoning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DSCWIYEF}},
  note         = {Machine review of arXiv:2505.15929}
}
read the original abstract

Existing benchmarks fail to capture a crucial aspect of intelligence: physical reasoning, the integrated ability to combine domain knowledge, symbolic reasoning, and understanding of real-world constraints. To address this gap, we introduce PhyX: the first large-scale benchmark designed to assess models capacity for physics-grounded reasoning in visual scenarios. PhyX includes 3K meticulously curated multimodal questions spanning 6 reasoning types across 25 sub-domains and 6 core physics domains: thermodynamics, electromagnetism, mechanics, modern physics, optics, and wave\&acoustics. In our comprehensive evaluation, even state-of-the-art models struggle significantly with physical reasoning. GPT-4o, Claude3.7-Sonnet, and GPT-o4-mini achieve only 32.5%, 42.2%, and 45.8% accuracy respectively-performance gaps exceeding 29% compared to human experts. Our analysis exposes critical limitations in current models: over-reliance on memorized disciplinary knowledge, excessive dependence on mathematical formulations, and surface-level visual pattern matching rather than genuine physical understanding. We provide in-depth analysis through fine-grained statistics, detailed case studies, and multiple evaluation paradigms to thoroughly examine physical reasoning capabilities. To ensure reproducibility, we implement a compatible evaluation protocol based on widely-used toolkits such as VLMEvalKit, enabling one-click evaluation. More details are available on our project page: https://phyx-bench.github.io/.

Figures

Figures reproduced from arXiv: 2505.15929 by the authors.

Figure 1
Figure 1. Accuracies of three leading MLLMs, two leading LLM and human performance on our [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Sampled PHYX examples from each domain. Physical problem-solving fundamentally differs from pure mathematical reasoning or science knowl￾edge question answering by requiring models to: (1) decode implicit conditions in the questions (e.g., interpreting "smooth surface" in the question as the coefficient of friction equals to zero), (2) maintain physical consistency across the reasoning chains since the laws of physi… view at source ↗
Figure 3
Figure 3. Comparison with existing physics benchmarks. Realistic refers to the extent to which the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (48 more)
Figure 4
Figure 4. Figure 4: Existing benchmarks that contain physics-related questions suffer from information redun [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: An real example of reasoning trajectory based on GPT-4o and the comparison of required [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: The error distribution over 90 annotated errors based on GPT-4o with a typical visual [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The distribution of the number of words per question in P [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Examples of the visual context for the Mechanics domain. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Examples of the visual context for the Electromagnetism domain. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Examples of the visual context for the Thermodynamics domain. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Examples of the visual context for the Wave/Acoustics domain. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Examples of the visual context for the Optics domain. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Examples of the visual context for the Modern Physics domain. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: CoT prompting for generating answer. C.4 Physical Reasoning Definition Six physical reasoning types are defined in [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Rule-based answer extraction strategy for MC questions. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Rule-based answer extraction strategy for OE questions. [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Rule-based answer extraction strategy for OE questions. [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Prompt template for caption generation. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Prompt for reasoning type labeling (1). Prompt for Reasoning Type Labeling(2) **Instructions:** 1. **Read and Analyze:** Carefully understand the problem and the likely steps/concepts needed for its solution. 2. **Identify Potential Types:** Determine which of the 6 r…
Figure 21
Figure 21. Figure 21: Prompt for reasoning type labeling (2). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 1
Figure 1. Figure 1: A sample correct case of Mechanics. Back to List of Figures | Back to Table Index 28 [PITH_FULL_IMAGE:figures/full_fig_p028_1.png]
Figure 2
Figure 2. Figure 2: A sample correct case of Mechanics. Back to List of Figures | Back to Table Index 29 [PITH_FULL_IMAGE:figures/full_fig_p029_2.png]
Figure 3
Figure 3. Figure 3: A sample error case of Mechanics. Error category: Visual Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p030_3.png]
Figure 4
Figure 4. Figure 4: A sample error case of Mechanics. Error category: Text Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p031_4.png]
Figure 5
Figure 5. Figure 5: A sample error case of Mechanics. Error category: Lack of Knowledge [PITH_FULL_IMAGE:figures/full_fig_p032_5.png]
Figure 6
Figure 6. Figure 6: A sample correct case of Electromagnetism. [PITH_FULL_IMAGE:figures/full_fig_p033_6.png]
Figure 7
Figure 7. Figure 7: A sample correct case of Electromagnetism. [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]
Figure 8
Figure 8. Figure 8: A sample error case of Electromagnetism. Error category: Visual Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]
Figure 9
Figure 9. Figure 9: A sample error case of Electromagnetism. Error category: Text Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p036_9.png]
Figure 10
Figure 10. Figure 10: A sample error case of Electromagnetism. Error category: Lack of Knowledge [PITH_FULL_IMAGE:figures/full_fig_p037_10.png]
Figure 11
Figure 11. Figure 11: A sample correct case of Thermodynamics. [PITH_FULL_IMAGE:figures/full_fig_p038_11.png]
Figure 12
Figure 12. Figure 12: A sample correct case of Thermodynamics. [PITH_FULL_IMAGE:figures/full_fig_p039_12.png]
Figure 13
Figure 13. Figure 13: A sample error case of Thermodynamics. Error category: Visual Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p040_13.png]
Figure 14
Figure 14. Figure 14: A sample error case of Thermodynamics. Error category: Text Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p041_14.png]
Figure 15
Figure 15. Figure 15: A sample error case of Thermodynamics. Error category: Lack of Knowledge [PITH_FULL_IMAGE:figures/full_fig_p042_15.png]
Figure 16
Figure 16. Figure 16: A sample correct case of Wave/Acoustics. [PITH_FULL_IMAGE:figures/full_fig_p043_16.png]
Figure 17
Figure 17. Figure 17: A sample correct case of Wave/Acoustics. [PITH_FULL_IMAGE:figures/full_fig_p044_17.png]
Figure 18
Figure 18. Figure 18: A sample error case of Wave/Acoustics. Error category: Visual Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p045_18.png]
Figure 19
Figure 19. Figure 19: A sample error case of Wave/Acoustics. Error category: Text Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p046_19.png]
Figure 20
Figure 20. Figure 20: A sample error case of Wave/Acoustics. Error category: Lack of Knowledge [PITH_FULL_IMAGE:figures/full_fig_p047_20.png]
Figure 21
Figure 21. Figure 21: A sample correct case of Optics. Back to List of Figures | Back to Table Index 48 [PITH_FULL_IMAGE:figures/full_fig_p048_21.png]
Figure 22
Figure 22. Figure 22: A sample correct case of Optics. Back to List of Figures | Back to Table Index 49 [PITH_FULL_IMAGE:figures/full_fig_p049_22.png]
Figure 23
Figure 23. Figure 23: A sample error case of Optics. Error category: Visual Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p050_23.png]
Figure 24
Figure 24. Figure 24: A sample error case of Optics. Error category: Text Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p051_24.png]
Figure 25
Figure 25. Figure 25: A sample error case of Optics. Error category: Lack of Knowledge [PITH_FULL_IMAGE:figures/full_fig_p052_25.png]
Figure 26
Figure 26. Figure 26: A sample correct case of Modern Physics. [PITH_FULL_IMAGE:figures/full_fig_p053_26.png]
Figure 27
Figure 27. Figure 27: A sample correct case of Modern Physics. [PITH_FULL_IMAGE:figures/full_fig_p054_27.png]
Figure 28
Figure 28. Figure 28: A sample error case of Modern Physics. Error category: Visual Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p055_28.png]
Figure 29
Figure 29. Figure 29: A sample error case of Modern Physics. Error category: Text Reasoning Error [PITH_FULL_IMAGE:figures/full_fig_p056_29.png]
Figure 30
Figure 30. Figure 30: A sample error case of Modern Physics. Error category: Lack of Knowledge [PITH_FULL_IMAGE:figures/full_fig_p057_30.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

    cs.AI 2026-04 unverdicted novelty 8.0 of 10

    FeynmanBench is the first benchmark for evaluating multimodal LLMs on diagrammatic reasoning with Feynman diagrams, revealing systematic failures in enforcing physical constraints and global topology.

  2. Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

    cs.CV 2026-07 conditional novelty 7.0 of 10

    A law-grounded benchmark, Apple-PI, grades video models stage-by-stage on physics reasoning and finds they top out at 0.473, well short of reliable simulation.

  3. PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Frontier MLLMs remain far from mastering atomic visual perception: none reach 60% on a failure-derived, perception-only benchmark of ten capabilities.

  4. Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.

  5. Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    Multimodal reasoning models hallucinate at high-entropy cognitive bifurcation points due to loss of visual semantic anchoring, and the V-STAR training paradigm with HVAR rewards and FRM reflection mitigates this by re...

  6. DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A unified visual benchmark of six physics environments shows current VLMs, including frontier models, mostly fail at interactive physical reasoning and cannot convert descriptive physics knowledge into reliable control.

  7. Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTS

    cs.CL 2025-07 conditional novelty 6.0 of 10

    R2-LLMs retrieves hierarchical problem templates and step-level examples during Monte Carlo tree search, improving math reasoning accuracy of small LLMs at inference time without additional training.

  8. Skywork-R1V3 Technical Report

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A 38B open-source VLM reaches 76.0% on MMMU using RL post-training and connector-only tuning, with a critical-token entropy metric for checkpoint selection.

  9. Though Language Models Err While They Strive: Conformal Prediction for Self-Correcting Scientific Generation

    cs.CL 2026-07 reject novelty 4.0 of 10

    A graph-based conformal wrapper that filters and regenerates LLM reasoning steps claims formal coverage guarantees on scientific validity, but its evaluation is circular and its gains are confounded with sampling effo...

  10. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

45 extracted references · 16 canonical work pages · cited by 10 Pith papers

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/

    2024

  3. [3]

    Gemini 2.5: Our most intelligent ai model, 2025

    Gemini Team. Gemini 2.5: Our most intelligent ai model, 2025. URLhttps://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking

    work page 2025

  4. [4]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. InAmerican Invita- tional Mathematics Examination - AIME 2024, February 2024. URL https://maa.org/ math-competitions/american-invitational-mathematics-examination-aime

    work page 2024

  5. [5]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  6. [6]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    arXiv 2021

  7. [7]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024

  8. [8]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URLhttps://arxiv.org/abs/2410.21276

    arXiv 2024

Show all 45 references
  1. [9]

    Claude 3.7 sonnet and claude code

    claude. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/ claude-3-7-sonnet, 2025

    2025

  2. [10]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  3. [11]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    2024

  4. [12]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186. Springer, 2024

    2024

  5. [13]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InThe Twelfth International Conference on Learning ...

  6. [14]

    Phybench: Holistic evaluation of physical perception and reasoning in large language models.arXiv preprint arXiv:2504.16074, 2025

    Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, et al. Phybench: Holistic evaluation of physical perception and reasoning in large language models.arXiv preprint arXiv:2504.16074, 2025

    2025 arXiv

  7. [15]

    Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models.arXiv preprint arXiv:2502.00334, 2025

    Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models.arXiv preprint arXiv:2502.00334, 2025. 10

    2025 arXiv

  8. [16]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/, 2025

    2025

  9. [17]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  10. [18]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  11. [19]

    Introducing claude 3.5 sonnet

    claude. Introducing claude 3.5 sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet, 2024

    2024

  12. [20]

    Claude 3.5 haiku.https://www.anthropic.com/claude/haiku, 2024

    claude. Claude 3.5 haiku.https://www.anthropic.com/claude/haiku, 2024

    2024

  13. [21]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  14. [22]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai.arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  15. [23]

    Openai o3-mini: Pushing the frontier of cost-effective reasoning

    OpenAI. Openai o3-mini: Pushing the frontier of cost-effective reasoning. https://openai. com/index/openai-o3-mini/, 2025

    2025

  16. [24]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412. 19437

    2025

  17. [25]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  18. [26]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  19. [27]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  20. [28]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations

  21. [29]

    Scieval: A multi-level large language model evaluation benchmark for scientific research

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053–19061, 2024

    2024

  22. [30]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021. 11

    2021 arXiv

  23. [31]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  24. [32]

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. InInternational Conference on Machine Lear...

    2024

  25. [33]

    Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai.Advances in Neural Information Processing Systems, 37:19209–19253, 2024

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai.Advances in Neural Information Processing Systems, 37:1920...

    2024

  26. [34]

    Physreason: A comprehensive benchmark towards physics-based reasoning.arXiv preprint arXiv:2502.12054, 2025

    Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning.arXiv preprint arXiv:2502.12054, 2025

    2025 arXiv

  27. [35]

    the fair use of a copyrighted work, including such use by ...... scholarship, or research, is not an infringement of copyright

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark.arXiv preprint arXiv:2501.05444, 2025. 12 Table of Contents in Appendix A Ethics Statement 14 B ...

    2025 arXiv

  28. [36]

    Translating the scenario into a physics framework

    **Physical Model Grounding Reasoning:** * Explanation: Connecting problem details to physical concepts, laws, and idealized models (e.g., point mass, frictionless surface, ideal gas). Translating the scenario into a physics framework

  29. [37]

    **Spatial Relation Reasoning:** * Explanation: Understanding and manipulating geometric aspects (positions, angles, vectors, diagrams, coordinate systems)

  30. [38]

    **Multi-Formula Reasoning:** * Explanation: Requiring the combination or sequential use of multiple distinct physics formulas or principles to find the solution

  31. [39]

    starts from rest

    **Implicit Condition Reasoning:** * Explanation: Recognizing and using conditions not explicitly stated but implied by context or keywords (e.g., "starts from rest", "smooth surface", "maximum height")

  32. [40]

    Do *not* select this for basic algebra or substitutions

    **Numerical Reasoning:** (Revised Definition) * **Explanation:** Requiring advanced mathematical methods beyond basic algebra/trigonometry (e.g., calculus - integration/ differentiation, solving differential equations, Fourier analysis) as a core part of manipulating physical ...

  33. [41]

    Figure 20: Prompt for reasoning type labeling (1)

    **Predictive Reasoning:** * Explanation: Using physical laws and initial conditions to forecast a future state or behavior (e.g., final velocity, trajectory, final temperature). Figure 20: Prompt for reasoning type labeling (1). Prompt for Reasoning Type Labeling(2) **Instructions:**

  34. [42]

    **Read and Analyze:** Carefully understand the problem and the likely steps/concepts needed for its solution

  35. [43]

    **Identify Potential Types:** Determine which of the 6 reasoning types are involved in the solution process

  36. [44]

    most critical

    **Prioritize and Select:** From the potentially involved types, select **at most two** that are the **most critical, dominant, or uniquely challenging** aspects of solving this *particular* problem. * Think about what makes the problem non-trivial. Is it complex geometry? Comb...

  37. [45]

    Reasoning Type A

    **Output Format:** **CRITICAL:** Your entire response must consist *only* of a single Python-style list containing strings of the exact names for the selected zero, one, or two most critical reasoning types. Do **NOT** include any introductory text, explanations, labels, apolo...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.