{"id":"243b2d96-f7b4-40cd-9dac-ac050f0d53e1","arxiv_id":"2505.15929","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PhyX is a new 3,000-question visual physics benchmark; the best AI model tested scores 45.8 percent, well below the 75.6 to 78.9 percent of a small human student sample.","lead":"PhyX is a newly built benchmark of 3,000 multimodal physics questions that asks AI models to read realistic images and reason through mechanics, electromagnetism, thermodynamics, waves, optics, and modern physics. The paper reports that even the best tested model, GPT-o4-mini, answers only 45.8% correctly, while a small group of human physics students scored about 75% or higher.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-ended scoring can discard correct model answers before the LLM judge sees them: the regex in Figure 17 only accepts 'final answer:' or 'correct answer:', with no fallback, so the reported 45.8%-75.6% human-model gap may be partly an evaluation artifact.","rationale":"I read the paper's central claim as the 29-point human-model gap. The reader's weakest assumption was the small human sample; I agree that is a problem but regard the open-ended scoring pipeline as more directly load-bearing. The paper's own appendix exposes a deterministic filter that deletes any answer not phrased with 'final answer:' or 'correct answer:'. Since the LLM judge is described as comparing the extracted answer to the ground truth, an empty extraction becomes a wrong answer. This is not a speculation about model quality; it is a property of the released evaluation code (the regex in Figure 17). The 200-example judge validation does not cover extraction failures. The MC/OE gap in Table 2 makes the potential impact concrete. The human baseline issue compounds this: even the 'worst' group is only 5 students and 90 answers, so the exact size of the gap is uncertain from both sides. I also note visible data quality issues, such as the Optics example in Figure 2 showing '5.6 kW' (the same answer as the Electromagnetism example) and Table 1 stating 6,000 new questions while the text and Table 2 use 3,000; these are not my primary concern but reinforce the need for data release. I do not think this warrants rejection; the benchmark may still be useful and the gap may survive a fairer evaluation. But the authors should be required to release model outputs and evaluation logs, add a fallback extraction path, and report extraction success rates. That is consistent with the reader's CONDITIONAL verdict, so I leave the verdict unchanged. My concern is not that the authors are dishonest, only that an easily fixable scoring rule may be doing part of the work in the headline number.","tokens_in":19676,"tokens_out":10737,"duration_ms":96586,"concrete_test":"Take a random 200-question subset of the Text-DeRedundancy open-ended predictions for GPT-o4-mini and DeepSeek-R1. Re-score each with a fallback: when the Figure 17 regex returns no match, feed the entire raw prediction to the DeepSeek-V3 judge using the Appendix D.3 prompt, and also have one physics graduate student score 50 of those fallback cases. Compare the resulting accuracies to Table 2. If either model gains more than 10 absolute points, the reported human-model gap is substantially an artifact of answer extraction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that state-of-the-art MLLMs fail physical reasoning, quantified by a 75.6% (worst human group) vs. 45.8% (GPT-o4-mini) gap on Text-DeRedundancy open-ended questions (Tables 2-3). To establish this, model answers must be scored fairly; this is where the paper is least secure. In Step 2 (Appendix D.2, Figure 17), open-ended answers are extracted with the regex `\\b(?:final\\s+answer|correct\\s+answer)\\b[^:：]*[:：]\\s*(.*?)(?=\\n\\n\\n|\\Z)`. If the model gives the correct value in any other format, such as 'Therefore, the heat transfer is -0.0147 kJ.' or as a bare numeric answer, the extracted answer is empty and Step 3 compares an empty string to the ground truth, almost certainly marking the response wrong. No fallback to the full raw prediction is described. The reported '>99%' judge accuracy comes from only 200 examples and does not establish that successful extraction is common. This is asymmetric with the human baseline, which was almost certainly hand-graded. The magnitude is visible in Table 2: GPT-o4-mini scores 86.9% on the MC version but only 45.8% on the OE version; a large share of that 41-point drop could be extraction failure rather than lack of physical reasoning. If so, the headline '29% gap' shrinks and the central claim loses its quantitative support. The small human sample (15 students x 18 questions) is a real secondary worry, but the scoring pipeline is the load-bearing issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PhyX, a multimodal physics benchmark of 3,000 unique questions (with a 1,000-question testmini subset) spanning six physics domains and six reasoning types, each question provided in multiple-choice and open-ended forms. The authors evaluate 16 LLMs and MLLMs under three textual input conditions and report that the worst group of human physics students achieves 75.6% accuracy on open-ended text-de-redundancy questions while the best MLLM, GPT-o4-mini, achieves 45.8%. They interpret this gap as evidence that current models lack robust physical reasoning, and they provide domain-level breakdowns, an error taxonomy, case studies, and an evaluation protocol compatible with VLMEvalKit.","tokens_in":20000,"tokens_out":5079,"duration_ms":48437,"significance":"If the headline comparison is valid, PhyX would be a useful diagnostic benchmark for a genuinely under-tested capability: integrating visual perception, physical laws, and symbolic computation. The paper's strengths are its broad domain coverage, the paired MC/OE design, the three textual-variation conditions, and the explicit error taxonomy with case studies. However, the benchmark's utility depends on two load-bearing assumptions that are not adequately supported: that open-ended model answers are scored fairly, and that the small human sample provides a reliable expert baseline. The paper also contains concrete data-quality errors in its own illustrative examples, which undermines confidence in the expert-validation claim. The central direction is defensible, but the quantitative headline needs substantial additional support.","major_comments":[{"comment":"The open-ended answer extraction regex only accepts answers introduced by 'final answer:' or 'correct answer:' and has no fallback to the full raw prediction. A correct answer written as a bare numeric value, as 'Therefore, the heat transfer is -0.0147 kJ.', or in any other natural format is discarded and almost certainly scored as wrong. This is asymmetric with the human baseline, which was presumably hand-graded, and it can systematically depress all model OE scores. The magnitude of the risk is visible in Table 2: GPT-o4-mini drops from 86.9% on the MC version to 45.8% on the OE version. The reported '>99%' judge accuracy from a 200-example preliminary study validates the LLM judge on inputs already produced by the extraction step; it does not establish that extraction succeeds on a large fraction of correct answers. The authors should report the extraction failure rate per model, provide a fallback that evaluates the full prediction when no marker phrase is found, and re-run the OE evaluation before the headline gap is taken as established.","section":"§3.2, Appendix D.2, Figure 17"},{"comment":"The human expert baseline rests on only 15 students, each answering 18 questions, for a total of 270 human responses. The three reported groups are partitions of five students, so the per-domain percentages in Table 3 are based on roughly 13–17 answers per cell; for example, the worst-human Electromagnetism value of 60.0% is consistent with 9 correct out of 15. The 95% confidence interval for 60% on 15 trials is roughly 32% to 84%, which overlaps substantially with the best model score of 45.8%. As reported, the data do not statistically support the claim that all human groups outperform all models. The authors should either collect a much larger human sample stratified across the testmini domains or report exact counts and confidence intervals and soften the gap claim accordingly.","section":"§3.1, Table 3"},{"comment":"The paper's showcase examples contain incorrect answers. The Optics example asks for the magnification of a 1.00 cm object placed 10.0 cm from a concave mirror of radius 30.0 cm; the correct magnification is 3.0, but the displayed answer is '5.6 kW', which appears to be copied from the Electromagnetism row and is dimensionally impossible for a magnification. The Modern Physics example reports the observed dimensions as '-0.65m', but a dimension cannot be negative and the contracted width should be +0.65 m while the height remains 1.00 m. These errors in the paper's own curated examples directly contradict the claim that scenarios undergo rigorous expert validation and cast doubt on the annotation quality of the full benchmark. The authors should correct the figure and audit the full dataset for this class of answer errors.","section":"Figure 2"},{"comment":"The dataset size is stated inconsistently. Table 1 reports 'Total new questions 6,000' with 'Unique number of questions 3,000', while §2.2 says the process 'results in a diverse collection of 3,300 questions', and the abstract and §1 say 3K questions. Since each physical problem is converted into both an MC and an OE version, the relationship between 'questions', 'problems', and 'instances' needs a precise definition, and the 3,300 figure needs reconciliation. The manuscript also states the data and code 'will be made publicly available' but neither is linked or inspectable in the current version; for a benchmark paper, release of the dataset and the exact evaluation scripts is a condition for verifying the central claims.","section":"§2.1, Table 1; §2.2"}],"minor_comments":[{"comment":"The testmini subset is described as a 'proportional random sampling strategy', but no comparison between the testmini and full 3,000-question distributions over domains and reasoning types is provided; since all quantitative results are on testmini, the authors should report domain counts and reasoning-type counts for both sets.","section":"§3.1"},{"comment":"The preliminary study of 200 examples for the LLM judge is described in one sentence; the authors should report how the 200 examples were sampled, what the ground-truth judge labels were, and the agreement metric used to justify the '99%' claim.","section":"§3.2"},{"comment":"The error taxonomy (Visual Reasoning Errors, Text Reasoning Errors, Lack of Knowledge, Calculation Error) is said to be produced 'based on GPT-4o', but no human validation of the error labels is reported; a small human-annotated agreement study would make the taxonomy more credible.","section":"§3.5"},{"comment":"The caption for Figure 18 reads 'Rule-based answer extraction strategy for OE questions', but the content appears to be the prompt for the LLM judge; the figure and caption should be matched.","section":"Figure 18 caption"},{"comment":"The Open-Ended columns report 'Random Choice - 25' for MC-style random guessing; for open-ended numeric answers random chance is not 25%, so this row should be clarified or omitted for the OE columns.","section":"§3.3, Table 2"},{"comment":"There are naming inconsistencies between the text and code, such as 'PhyX' vs 'MetaPhyX', and model names such as 'LLaV A-OneVision-7B' with an odd space; these should be normalized.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper targets a real gap and the benchmark concept is valuable, but the current version is not ready for publication as is. The open-ended scoring pipeline and the tiny human baseline are load-bearing for the headline 29% gap, and the Figure 2 answer errors are a red flag for dataset quality. If the authors release the data and code, fix the extraction fallback, re-run the OE evaluation, and substantially strengthen the human study, the contribution could become publishable. The 'first large-scale' claim is also worth checking against PhysReason and other concurrent benchmarks during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is worth having, but the headline 29% human-model gap is not yet trustworthy. The open-ended scoring pipeline in Appendix D.2 extracts only text following a 'final answer:' or 'correct answer:' marker; any correct answer phrased differently — including a bare numeric value — is silently dropped before the LLM judge sees it. The claimed >99% judge accuracy comes from 200 examples and does not cover extraction success. GPT-o4-mini's 41-point MC-to-OE drop (86.9% to 45.8%) is consistent with a substantial share of correct answers being discarded. Since the human baseline was surely hand-graded, the asymmetry is real, and the specific gap number may be mostly an evaluation artifact. The direction survives — MC scores are far from ceiling too — but the quantitative claim as stated does not.\n\nWhat is genuinely good: the three-version design (Full-Text, Text-DeRedundancy, Text-Minimal) is a clean way to measure text-over-reliance, and the 16-model sweep with a sensible error taxonomy makes this a useful diagnostic. At 3,000 physics questions across six domains it is larger than the physics subsets of PhysReason and EMMA, so 'large-scale' is roughly fair, though 'first' overstates it.\n\nSecondary worries, in order. The human baseline is 15 students × 18 questions each, split into groups of five; the domain-level human numbers in Table 3 come from roughly a dozen responses per domain per group. Figure 2 has visible errors — the Optics example's answer is the satellite power from the Electromagnetism row ('5.6 kW'), and the Modern Physics answer '-0.65 m' is not a plausible contracted dimension. Data and code are promised but not yet released, so none of this can be checked independently.\n\nWho this is for: people building or benchmarking multimodal physics reasoning. The de-redundancy protocol is worth borrowing now; the dataset numbers are not worth citing until the pipeline is fixed and the data is out. I would send it to review — it deserves referee time — but with a clear expectation of major revision.","headline":"A useful benchmark idea with a clean text-de-redundancy design, but the headline human-model gap rests on an answer-extraction pipeline that can silently discard correct responses.","tokens_in":20607,"tokens_out":5489,"would_cite":false,"duration_ms":43411,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces PhyX, a 3,000-question multimodal benchmark built to test whether AI models can reason about physics from realistic images, and reports that every model it evaluated falls far short of human physics students, with…","keywords":["physical reasoning","multimodal benchmark","multimodal large language models","visual question answering","physics problem solving","benchmark evaluation","chain-of-thought reasoning","LLM-as-judge"],"falsifier":"Recruit a substantially larger panel of independent physics graduate students, have each answer more than 18 questions drawn evenly from the full 3,000-question set, and compare. If the worst-expert accuracy falls near or below 50% on the same judged rubrics, the claimed 30-point human-model gap collapses; alternatively, if the testmini's domain and reasoning-type distribution differs measurably from the full set, the reported model scores are not a valid estimate for the benchmark as a whole.","tokens_in":19471,"feed_emoji":"🧲","tokens_out":10090,"duration_ms":78033,"temperature":0.7,"pith_summary":"The paper argues that existing benchmarks measure knowledge recall and symbolic math but miss physical reasoning: the ability to read a realistic visual scenario, extract implicit physical conditions, apply the right laws, and compute a grounded answer. To make that measurable, it introduces PhyX, 3,000 expert-validated multimodal questions spanning six physics domains and six reasoning types, with each question stripped of the text-image redundancy that lets models rely on only one modality. On a 1,000-question test subset, all 16 evaluated models scored far below three groups of physics students; the strongest multimodal model, GPT-o4-mini, reached 45.8%, while the weakest student group scored 75.6%. The authors take this roughly 30-point gap as evidence that current models lean on memorized formulas, textual hints, and surface visual patterns instead of genuine physical understanding. If the benchmark is valid, it provides a diagnostic that separates reasoning about the physical world from other forms of machine intelligence.","feed_headline":"Top AI vision models score 45.8% on new physics test","feed_subtitle":"A 3,000-question visual benchmark finds even the best models trail physics students by roughly 30 points.","key_machinery":"The load-bearing mechanism is the benchmark's de-redundancy design. Every PhyX question exists in three versions — Full-Text, Text-DeRedundancy (which strips whatever the image already conveys), and Text-Minimal (question only) — so that a correct answer requires integrating both modalities instead of matching surface cues from either alone. The claim that models fail at genuine physical reasoning is carried by the accuracy drop along this text-reduction gradient, by the six-type reasoning taxonomy used to categorize questions, and by a three-step evaluation pipeline (chain-of-thought prediction, rule-based answer extraction, then a DeepSeek-V3 judge reported to agree with human grading on more than 99% of a 200-example pilot) that is meant to make the measured accuracies trustworthy.","core_discovery":"The central claim is that PhyX is the first large-scale benchmark able to measure physics-grounded reasoning in visual scenarios, and that state-of-the-art multimodal models fail it by a wide margin. On the 1,000-question testmini subset, the best multimodal model (GPT-o4-mini) scores 45.8%, below the best text-only LLM (DeepSeek-R1 at 51.2%) and far below the worst of three human-expert groups at 75.6%; the gap persists across all six domains and reaches roughly 46 points in Modern Physics (human 86.7% versus model 40.6%). An error analysis of 96 GPT-4o failures attributes 39.6% to visual reasoning errors, 38.5% to missing knowledge, 13.5% to text reasoning errors, and 8.3% to calculation slips, and accuracy falls monotonically as the textual description is removed, showing that models lean on redundant text rather than the image. The paper concludes that physical reasoning is a distinct capability that is harder than the mathematical reasoning measured by benchmarks like MathVista and MATH-V, and that current models have not acquired it.","pith_inferences":["The de-redundancy protocol is a general-purpose modality-lock test: it could be applied to any multimodal benchmark (medicine, navigation, engineering) to detect when a model is pattern-matching one modality rather than integrating evidence from both.","The authors note that the images are schematic textbook-style scenes rather than photographs; a photo-realistic variant might change the human-model gap in either direction — humans could improve with richer detail while models could degrade on noisy natural images — so the 30-point gap should not be assumed to transfer to real-world perception.","The six-type taxonomy could be used predictively: a model's per-type accuracy profile may indicate which reasoning weaknesses stem from missing physics knowledge versus weak visual encoding, suggesting targeted training data rather than general scaling.","A testable extension of the error analysis: train a model to correct its own visual-reading errors on PhyX examples and measure whether gains transfer to the other five reasoning types; the 39.6% visual error share predicts that fixing perception would lift every category."],"forward_implications":["Open-ended questions separate models far more sharply than multiple-choice questions, so multiple-choice evaluations of physics reasoning systematically overstate model competence.","Because the worst human-expert group outperformed every model in every domain, current multimodal systems should not be treated as reliable for university-level physics tutoring, analysis, or scientific assistance.","The error profile — visual reasoning 39.6%, missing knowledge 38.5%, text reasoning 13.5%, calculation 8.3% — locates the bottleneck in visual grounding and physics knowledge rather than arithmetic.","Text-only LLMs fed image captions rival the best multimodal models, showing that much of the apparent visual competence of MLLMs can be replicated by a captioning pipeline.","Comparing GPT-4o at 32.5% on PhyX against its reported 63.8% on MathVista and MATH-V indicates that physical reasoning is a harder test of foundation models than mathematical reasoning in visual contexts."],"supporting_citations":[{"why":"Supplies the contrast of olympiad-level physics problems whose simplified diagrams PhyX claims to surpass with realistic visuals.","marker":"[7]"},{"why":"Is the model used for image captioning and for the 96-case error analysis, and is also a leading evaluated baseline.","marker":"[8]"},{"why":"Is the broad multi-discipline benchmark whose information redundancy PhyX's de-redundancy design is explicitly built against.","marker":"[10]"},{"why":"Provides the MATH-V comparison used to argue that physical reasoning is harder than multimodal mathematical reasoning (GPT-4o at 63.8%).","marker":"[11]"},{"why":"Provides the MathVista comparison supporting the same harder-than-math claim.","marker":"[13]"},{"why":"Is the existing text-centric physics benchmark that PhyX claims to extend with modality de-redundancy and a larger question count.","marker":"[14]"},{"why":"Is the LLM judge whose more than 99% agreement on a 200-question pilot anchors the reported human and model accuracy numbers.","marker":"[24]"},{"why":"Is the existing multimodal physics benchmark with a smaller physics subset that PhyX claims to supersede in scale and reasoning-type coverage.","marker":"[34]"}],"fun_headline_variants":["New physics benchmark: best AI vision model scores 45.8%","AI vision models trail humans by 30 points on new physics test","PhyX: multimodal models score below 50% on visual physics reasoning","Physics benchmark reveals AI leans on text, not images, to answer","Even top AI models struggle with visual physics reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline human-versus-model gap rests on the performance of just 15 physics students, each answering 18 questions in groups of five, and on the assumption that the 1,000-question testmini subset fairly represents the full 3,000-question benchmark; if the students were not genuinely expert, or the sample questions were unrepresentative, the gap could shrink or disappear even while model accuracies remain low.","fun_headline_variants_meta":{"raw":{"variants":["New physics benchmark: best AI vision model scores 45.8%","AI vision models trail humans by 30 points on new physics test","PhyX: multimodal models score below 50% on visual physics reasoning","Physics benchmark reveals AI leans on text, not images, to answer","Even top AI models struggle with visual physics reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1650,"prompt_tokens":1050,"completion_tokens":600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":666,"tokens_out":600,"duration_ms":5476,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:09:47.480127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recruit a substantially larger panel of independent physics graduate students, have each answer more than 18 questions drawn evenly from the full 3,000-question set, and compare. If the worst-expert accuracy falls near or below 50% on the same judged rubrics, the claimed 30-point human-model gap collapses; alternatively, if the testmini's domain and reasoning-type distribution differs measurably from the full set, the reported model scores are not a valid estimate for the benchmark as a whole.","supporting_citations":[],"review_version":1}