Pith. sign in

REVIEW 3 major objections 5 minor 16 references

ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ABench-Physics shows that state-of-the-art LLMs solve under half of graduate-level physics problems and lose 22.5 points on average when only the numbers change.

desk verdict Static benchmark is useful; the dynamic 22.5% drop mostly reflects the all-variants scoring rule, not memorization. read the letter →

arxiv 2507.04766 v1 pith:FIM5QBGW submitted 2025-07-07 cs.LG cs.AIcs.CL

Yiming Zhang , Yingfan Ma , Yanmei Gu , Zhengkai Yang , Yihong Zhuang , Feng Wang , Zenan Huang , Yuanyuan Wang
show 4 more authors
Chao Huang Bowen Song Cheng Lin Junbo Zhao
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords physicalreasoningevaluationdynamicbenchmarknumericalperturbationmemorizationvsgeneralizationgraduate-levelphysicslargelanguagemodelsreinforcementlearningrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ABench-Physics is a benchmark built to test whether LLMs genuinely reason about physics or just match familiar solution patterns. It pairs a fixed set of 400 graduate- and Olympiad-level numerical problems with a dynamic subset of 100 problems whose numerical constants are automatically perturbed, and it credits a model on the dynamic subset only if it solves every variant of a problem. The paper's central claim is that current state-of-the-art models fail this test: the strongest model solves only about 43% of the static problems, and moving from unperturbed to perturbed variants lowers every model's accuracy, by 22.5 percentage points on average. The authors read this drop as evidence that models lean on superficial pattern matching rather than stable algebraic and physical modeling, and that reinforcement-trained models are somewhat more robust to such perturbations than instruction-tuned ones. If this is right, static benchmarks, especially multiple-choice or single-number ones, overstate LLM physics ability.

What carries the argument

The load-bearing mechanism is the parameter variation engine: it takes a Phy_B problem's ground-truth LaTeX solution and automatically regenerates fresh variants by changing the embedded numerical constants, while leaving the physical setup and solution structure intact. Coupled with the all-or-nothing scoring rule, credit only if every variant is solved, this turns a single problem into a probe of whether a model can redo the same physical derivation under shifted numbers. The 1% relative-error tolerance and explicit unit and significant-figure specifications make the numerical answers machine-gradable, and the static Phy_A set supplies a high-difficulty ceiling against which the dynamic drop is measured.

What would settle it

A decisive control would compare two ways of generating Phy_B variants: one where constants are rescaled by powers of ten, making arithmetic harder but leaving the physics the same, and one where constants are changed so the problem crosses into a different physical regime. If model accuracy drops sharply in the first condition too, the 22.5-point decline is at least partly an arithmetic effect, and the memorization conclusion would need revision; if it drops only in the second, the benchmark cleanly isolates physical modeling.

Watch

Extended reading notes

Core claim

The paper introduces ABench-Physics, a two-part benchmark for LLM physical reasoning. Phy_A contains 400 fixed, high-difficulty problems drawn from graduate and Olympiad sources; Phy_B contains 100 parameterised problems together with an automatic engine that regenerates variants by perturbing the numerical constants inside the ground-truth LaTeX solutions. Every answer is a number, scored with a 1% relative-error tolerance, and a Phy_B problem counts as solved only if the model answers all of its variants correctly. Evaluated across a range of frontier systems, the paper reports that the strongest model reaches 43.0% on Phy_A, that no model approaches human-level performance on the static set, and that every system loses accuracy on the dynamic variant task, with an average drop of 22.5 percentage points and the largest drop near 28 points. Because the semantic structure of each problem is unchanged, the paper attributes the dynamic drop to reliance on memorized solution patterns rather than robust physical modeling, and notes that RL-trained reasoning models show smaller relative losses and sometimes overtake SFT models that had scored higher on the static set.

Load-bearing premise

The central conclusion assumes that changing only the numbers inside a problem's LaTeX solution leaves the physical model and reasoning required to solve it unchanged, so that a lower score on the variants isolates memorization rather than arithmetic difficulty, rounding, or formatting failures.

Editorial extensions

If this is right

  • If the benchmark's results hold, current LLMs are far from reliable graduate-level or Olympiad physics: even the best tested system solves only about 43% of the static problems.
  • A static-only evaluation score materially overstates physical reasoning ability, since every tested model scores lower on the numerically perturbed variants of the same problems.
  • The average 22.5-point dynamic drop gives a quantitative target for robustness work: models need to retain accuracy when problem constants change without altering the physical model.
  • RL-trained reasoning models appear to generalize across numerical perturbations better than SFT instruction-tuned models, suggesting a training-direction hypothesis the benchmark can track over time.
  • The dynamic variation engine and all-variants metric provide a reusable template for contamination-resistant physics evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sharper test of the memorization explanation would separate arithmetic sensitivity from physics sensitivity: e.g., perturb constants by powers of ten to change computation without changing the physics, versus small changes that cross a physical regime boundary, and compare the two accuracy drops.
  • Because the preprint releases only three variants per Phy_B problem, the all-or-nothing score is coarse at the per-problem level; evaluating with more variants per template would tighten the estimate of how much of the drop is memorization versus arithmetic.
  • The same parameter-perturbation design could transfer to other quantitative disciplines, such as chemistry, circuits, or engineering mechanics, where numerical constants carry much of a problem's identity and memorization is easy to mask.
  • If future models close the dynamic gap, training on procedurally varied physics problems or reinforcement learning against variant-solved rewards would be the natural mechanism, though the paper does not test training interventions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ABench-Physics is a benchmark for LLM physical reasoning consisting of 500 numerical-answer problems: Phy_A contains 400 static problems screened to be hard for current models, and Phy_B contains 100 parameterized problems with an automatic variation engine that perturbs numerical constants in the LaTeX solutions. The evaluation protocol uses a 1% relative-error tolerance, and for Phy_B a model is credited only if it solves all released variants. Across 15 models, the best Phy_A accuracy is 43.0%; the paper reports an average 22.5% drop from Phy_B reference-variant accuracy to all-variants dynamic accuracy and interprets this as evidence of reliance on memorization and of RL models generalizing better than SFT models. The paper's central quantitative claims are compromised by a scoring mismatch between the static and dynamic Phy_B metrics and by missing uncertainty quantification.

Significance. The resource itself addresses a real gap: static multiple-choice benchmarks are vulnerable to contamination, and dynamic numerical variants are a sensible way to probe robustness. The dataset appears carefully constructed, with manual annotation, independent verification, paraphrasing, and unique numerical answers, and the evaluation covers a broad set of models. If the experimental claims were supported by per-variant accuracy, the benchmark could be a useful community diagnostic. As written, the headline '22.5% drop' and the RL/SFT reversal are not established because they are confounded by the all-or-nothing scoring rule.

major comments (3)
  1. [§3.3, §4.1, Table 2] The headline dynamic decline is computed under two different scoring rules. In Table 2, 'Stactic Acc' (sic) for Phy_B credits a model for solving one reference variant, while 'Dynamic Acc' credits it only if all three released variants are solved. For any per-variant success probability p, the expected all-variants accuracy is p^3 under independent errors, so a model totally insensitive to perturbation would still show an expected decline of p - p^3 (about 0.33 at p = 0.75). The observed declines, 0.19–0.28, are actually below that benchmark, consistent with positive error correlation or with non-reference variants being easier; in no case does Δ isolate sensitivity to numerical perturbation. The paper never reports average per-variant accuracy on Phy_B, so the central memorization conclusion is not currently supported. Please report per-variant accuracies for each variant and compare static and dynamic performance under a matched scoring rule, with confidence intervals.
  2. [§4.1] The 'RL Outperforms SFT on Dynamic Questions' subsection is not supported by the reported numbers. Dynamic Acc is a nonlinear function of per-variant accuracy (p^3 under independence), so small differences in per-variant performance can produce the observed ranking reversals, and the correlation structure across variants is unknown. Moreover, Table 2 has no error bars or significance tests on a 100-question subset, and the systems are not explicitly labeled by training paradigm. This claim should either be removed or supported with per-variant accuracies for each group and a statistical comparison.
  3. [§3.1] Phy_A was explicitly constructed by 'experimental screening that identified cases where models tend to produce long responses with low accuracy.' Thus the 43.0% top accuracy on Phy_A is an estimate for an adversarially selected set, not an unbiased estimate of performance on graduate- or Olympiad-level physics. The abstract and §4.1 present the 43% as evidence of a general 'substantial gap'; this framing should be softened or supplemented with human performance or with an unscreened sample of similar problems. The selection rule is a legitimate benchmark-design choice, but it must be reflected in the strength of the claims.
minor comments (5)
  1. [Table 2] The column header 'Stactic' is a typo, and the text should clarify that the 'Static Acc' column refers to Phy_B reference variants, not to Phy_A, to avoid the ambiguity in the running text.
  2. [§4.1] The opening sentence says 'four widely used large language models' but Table 2 reports fifteen systems; update the sentence accordingly.
  3. [Footnote 2] The fact that only three variants per Phy_B problem are released in the preprint is important for interpreting the all-variants score and should be stated in the main text (§3.1 or §3.3) rather than in a footnote.
  4. [References] The reference 'OpenAI. Learning to reason with llms, Spetember 2024b' contains a typo ('Spetember') and the OpenAI o3 system-card reference is incomplete; also the title line 'AB ENCH -PHYSICS' has a formatting artifact.
  5. [Figure 1] Figure 1 has no error bars or confidence intervals and largely repeats Table 2; consider adding uncertainty information or removing the figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical benchmark measurements; the Dynamic-Acc scoring confound is a validity issue, not a definitional reduction.

full rationale

ABench-Physics is an empirical evaluation rather than a derivation: it reports observed pass rates on human-verified problems under a stated evaluation protocol, with no fitted parameters, no first-principles predictions, and no theorem whose conclusion is assumed in its premises. The static Phy_A results are externally anchored to independently annotated ground-truth answers, and the benchmark is a measurement instrument rather than a chain of reasoning. The chief concern is the comparison between Phy_B 'Static Acc' and 'Dynamic Acc': Section 3.3 defines Dynamic Acc as requiring the model to answer all regenerated variants of a template, so a drop from a one-variant score to an all-variants score partly reflects the stricter conjunction rule rather than perturbation sensitivity. This is a genuine construct-validity and scoring-scale confound, but it is not circular in the sense used here: the paper does not claim a derivation, and the observed declines are actually smaller than the mechanical p - p^3 expectation under an independence null model, so the drop is not forced by construction in the direction claimed. Similarly, the Phy_A screening for cases where models produce long, low-accuracy responses is selection bias affecting external validity, not circularity. There are no load-bearing self-citations; the cited prior benchmarks and model reports are used as context and comparison, not as evidence for the paper's own conclusions. A benchmark's results can be confounded or over-interpreted without being circular, and no quote in the manuscript exhibits an equation or quantity that reduces to its own input by construction. I therefore find no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, particles, or dimensions are postulated. The benchmark is a measurement artifact rather than a theoretical entity; all listed free parameters and axioms are methodological choices or untested domain assumptions that the central claims depend on.

free parameters (5)
  • Relative-error tolerance = 1%
    Answers are accepted when |y_hat - y*| <= 0.01|y*|. This threshold is chosen by hand and directly determines reported accuracy.
  • Number of variants per Phy_B problem = 3 in the preprint release
    The dynamic engine is said to produce many variants, but the preprint releases three variations per question. The all-variants-correct metric depends on this number.
  • All-variants-correct scoring rule = all variants must be correct
    A model is credited only if it solves every generated variant. This binary rule amplifies the measured drop and can conflate a single arithmetic slip with failed generalization.
  • Phy_A selection screen = cases where models produce long responses with low accuracy
    Phy_A questions were selected based on screening for low model accuracy, so the 43 percent ceiling is partly built in by the selection criterion.
  • Subset sizes = 400 static and 100 dynamic
    Subset sizes were chosen by the authors; the dynamic subset is small, which limits the precision of the 22.5 percent drop estimate.
assumptions (5)
  • domain assumption Every problem has a unique and unambiguous numerical answer.
    Section 3.2 asserts this property, but no inter-annotator agreement or calibration data are reported.
  • domain assumption Perturbing numerical constants preserves the physical model and the reasoning steps required.
    Section 3.1 and 3.3 claim variants have identical physical models but different conditions; if changing constants changes the concepts or only increases arithmetic burden, the dynamic-accuracy conclusion changes.
  • domain assumption Human annotators' solutions and ground truths are correct.
    Each question was verified by one or two human annotators, but no error rate or disagreement metric is reported.
  • domain assumption DeepSeek-R1 rewrites and human paraphrases remove overlap with pretraining corpora.
    Section 3.2 asserts paraphrasing to minimize overlap, but no contamination measurement such as n-gram or embedding leakage testing is presented.
  • domain assumption A single 1 percent tolerance numerical comparison is a valid measure of physics reasoning.
    The evaluation protocol reduces the construct to numeric accuracy; formatting and unit specification are enforced, but reasoning quality is not assessed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems." pith.science (2026). https://pith.science/paper/FIM5QBGW

@misc{pith2026250704766,
  author       = {Pith},
  title        = {Pith review of: ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FIM5QBGW}},
  note         = {Machine review of arXiv:2507.04766}
}
read the original abstract

Large Language Models (LLMs) have shown impressive performance in domains such as mathematics and programming, yet their capabilities in physics remain underexplored and poorly understood. Physics poses unique challenges that demand not only precise computation but also deep conceptual understanding and physical modeling skills. Existing benchmarks often fall short due to limited difficulty, multiple-choice formats, and static evaluation settings that fail to capture physical modeling ability. In this paper, we introduce ABench-Physics, a novel benchmark designed to rigorously evaluate LLMs' physical reasoning and generalization capabilities. ABench-Physics consists of two components: Phy_A, a static set of 400 graduate- or Olympiad-level problems; and Phy_B, a dynamic subset of 100 problems equipped with an automatic variation engine to test model robustness across changing conditions. All questions require precise numerical answers, with strict formatting and tolerance constraints. Our evaluation of several state-of-the-art LLMs reveals substantial performance gaps, highlighting persistent limitations in physical reasoning, especially in generalization to dynamic variants. ABench-Physics provides a challenging and diagnostic framework for advancing scientific reasoning in LLMs.

Figures

Figures reproduced from arXiv: 2507.04766 by the authors.

Figure 1
Figure 1. Model accuracy on the two benchmark subsets. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages

  1. [3]

    Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang

    URL https://arxiv.org/abs/2502.17521. Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501.16411,

    arXiv

  2. [4]

    DeepSeek Team

    URL https://arxiv.org/abs/2502.15815. DeepSeek Team. Introducing deepseek-v3. Technical report, December

    arXiv

  3. [6]

    doi: https://doi.org/ 10.1016/j.neucom.2025.130135

    ISSN 0925-2312. doi: https://doi.org/ 10.1016/j.neucom.2025.130135. URL https://www.sciencedirect.com/science/article/pii/ S0925231225008070. 5 Google DeepMind. Gemini’s new thinking capabilities and updates, March

    work page arXiv 2025

  4. [7]

    FOLIO: Natural language rea- soning with first-order logic

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szab´o, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech M...

    work page 2024

  5. [8]

    doi: 10.18653/v1/2024.emnlp-main.1229

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.1229. URL https://aclanthology.org/2024.emnlp-main.1229/. Stephen Hawking. A brief history of time: from big bang to black holes. Random House,

    doi:10.18653/v1/2024.emnlp-main.1229 2024

  6. [10]

    doi: 10.18653/v1/2024.inlg-main.45

    Association for Computational Linguistics. doi: 10.18653/v1/2024.inlg-main.45. URL https: //aclanthology.org/2024.inlg-main.45/. Hynek Kydl´ıˇcek. Math-verify: Math verification library,

    doi:10.18653/v1/2024.inlg-main.45 2024

  7. [12]

    URL https://arxiv.org/abs/2406.15720. OpenAI. Gpt-4o system card, 2024a. URL https://openai.com/index/gpt-4o-system-card/ . OpenAI. Learning to reason with llms, Spetember 2024b. OpenAI. Introducing gpt-4.1, March 2025a. OpenAI. Introducing openai o3 and o4-mini, April 2025b. Max Planck. Scientific Autobiography: And Other Papers. Citadel Press,

    arXiv

  8. [14]

    Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang

    URL https://arxiv.org/ abs/2410.23123. Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models, February 2025a. Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, an...

    arXiv

Show all 16 references
  1. [15]

    Shenghe Zheng, Qianjia Cheng, Junchi Yao, Mengsong Wu, Haonan He, Ning Ding, Yu Cheng, Shuyue Hu, Lei Bai, Dongzhan Zhou, Ganqu Cui, and Peng Ye

    URL https://arxiv.org/abs/2504.10885. Shenghe Zheng, Qianjia Cheng, Junchi Yao, Mengsong Wu, Haonan He, Ning Ding, Yu Cheng, Shuyue Hu, Lei Bai, Dongzhan Zhou, Ganqu Cui, and Peng Ye. Scaling physical reasoning with the physics dataset,

    arXiv

  2. [16]

    URL https://arxiv.org/abs/2506.00022. 7

  3. [1949]

    Phybench: Holistic evaluation of physical perception and reasoning in large language models, 2025a

    6 Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin,...

    arXiv

  4. [2009]

    Mea- suring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. arXiv preprint arXiv:2009.03300,

    2009 arXiv

  5. [2019]

    Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray

    URL https://arxiv.org/abs/1908.05656. Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. Recent advances in large langauge model benchmarks against data contamination: From static to dynamic evaluation,

    1908 arXiv

  6. [2020]

    URL https://arxiv.org/abs/2007. 08124. Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. Scaling laws for fact memo- rization of large language models,

    2007

  7. [2024]

    Jingzhe Ding, Yan Cen, and Xinyuan Wei

    URL https://arxiv.org/abs/2406.14644. Jingzhe Ding, Yan Cen, and Xinyuan Wei. Using large language model to solve and explain physics word problems approaching human level, September

    arXiv

  8. [2025]

    Have llms advanced enough? a challenging problem solving benchmark for large language models

    Daman Arora, Himanshu Singh, and Mausam. Have llms advanced enough? a challenging problem solving benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7527–7543. Association for Computational Linguistics,

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.