{"id":"f2f3b731-ad7e-42be-b324-19daf8708a41","arxiv_id":"2508.18124","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"CMPhysBench is a new condensed matter physics benchmark with a fine-grained expression-tree score, and the best tested model scores only 36 out of 100 with 28% accuracy.","lead":"This paper introduces CMPhysBench, a test of over 520 graduate-level calculation problems in condensed matter physics for large language models. It also proposes a score called SEED that gives partial credit based on how close a model's symbolic answer is to the correct expression tree.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SEED metric's validity is unverified; the claimed capability gap rests entirely on it.","rationale":"The reader identified the SEED metric and question quality as the weakest assumptions. I focus specifically on SEED because it is novel and the entire quantitative claim is expressed through it. The abstract provides no validation of SEED's behavior, so the paper's main result is unverifiable from the abstract alone. My proposed concrete tests would directly address this. The verdict remains UNVERDICTED, consistent with the reader's assessment, but I do not move it because the concern is a call for evidence rather than a demonstrated error.","tokens_in":683,"tokens_out":2531,"duration_ms":29651,"concrete_test":"Take 50 ground-truth expressions from the benchmark. For each, generate several algebraically equivalent forms using standard identities (commutativity, expansion, factoring, etc.). Compute SEED between each ground-truth and its equivalent rewrite. If the mean SEED is not near the maximum (e.g., >0.9), the metric penalizes valid answers and the reported model scores understate capability. Additionally, have two condensed-matter experts independently judge a sample of model outputs for correctness, and compute the rank correlation between expert scores and SEED scores; a weak correlation would invalidate SEED as a measure of mathematical correctness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that even the best LLM reaches only 36 SEED and 28% accuracy—depends on the SEED score faithfully measuring mathematical correctness. SEED is introduced as a tree-based expression edit distance with fine-grained partial credit, but the abstract gives no evidence that it correctly handles algebraic equivalence. If SEED assigns high similarity to structurally similar but incorrect expressions, or low similarity to valid rewrites (e.g., (x+1)^2 vs. x^2+2x+1), then the reported numbers do not reflect a true capability gap. Tree edit distance is sensitive to expression form, not just semantics; without a validation study against human expert grading or known-equivalent expression pairs, the metric is uninterpretable. This is load-bearing because the paper's main result reduces to two scores.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces CMPhysBench, a benchmark of more than 520 graduate-level condensed matter physics calculation problems aimed at evaluating large language models (LLMs). The authors propose the Scalable Expression Edit Distance (SEED) score, a tree-based metric intended to provide fine-grained partial credit by comparing predicted expressions to ground-truth answers. They report that the best tested model, Grok-4, achieves only a 36 average SEED score and 28% accuracy, concluding that current LLMs have a significant capability gap in condensed matter physics. The code and dataset are promised to be publicly available. The review is based on the abstract only, as the full text was not available.","tokens_in":901,"tokens_out":1517,"duration_ms":21291,"significance":"If the benchmark and the SEED metric are validated, CMPhysBench could be a useful resource for measuring LLM problem-solving ability in a specialized STEM domain, and the public release would support reproducibility. The reported performance gap is potentially important for guiding future model development. However, the significance depends entirely on whether the ground-truth questions are correctly calibrated to be graduate-level and whether SEED faithfully captures mathematical correctness; neither is established in the abstract.","major_comments":[{"comment":"The central claim that SEED provides a 'more accurate assessment of similarity' is unsupported. SEED is described as a tree-based edit distance, which is inherently syntactic and may not recognize algebraically equivalent expressions (e.g., (x+1)^2 vs. x^2+2x+1). No validation is reported showing that SEED correlates with human expert grading, handles equivalent rewrites, or distinguishes plausible but incorrect derivations. Because the headline result (36 SEED score, 28% accuracy) is defined by this metric, this is load-bearing and must be addressed by a validation study or equivalence-invariance analysis.","section":"Abstract (SEED metric)"},{"comment":"The statement that the 520+ questions are 'meticulously curated' and 'graduate-level' is not backed by any protocol. There is no mention of expert agreement, pilot testing, or criteria for excluding ambiguous or incorrectly solved problems. Since the benchmark's ground truth is the standard against which LLM outputs are scored, a documented curation and verification process is essential to interpret the reported numbers.","section":"Abstract (question curation)"},{"comment":"The reported '36 average SEED score and 28% accuracy' are given without statistical detail. No error bars, number of runs, variance across questions, or significance tests are reported. Also, the relationship between the binary accuracy and the continuous SEED score is unclear: does 'accuracy' correspond to exact match, a SEED threshold, or another criterion? Without this information, it is impossible to gauge the reliability of the capability-gap claim.","section":"Abstract (results and uncertainty)"}],"minor_comments":[{"comment":"The phrase 'as a novel Benchmark' is awkwardly capitalized; 'benchmark' need not be capitalized. Also, 'we focus exclusively on calculation problems, requiring LLMs to independently generate comprehensive solutions' could be clarified by stating whether model-generated solutions are parsed automatically or human-evaluated.","section":"Abstract (typo/style)"}],"recommendation":"major_revision","confidential_remarks":"This is an abstract-only review, so my assessment is necessarily limited. The core issue is that the paper's main empirical claims rest on the SEED metric and dataset quality, neither of which is substantiated in the abstract. If the full manuscript contains thorough validation of SEED (e.g., correlation with human grading, equivalence handling, ablation on incorrect/partial solutions) and a documented curation protocol, a major revision could resolve these concerns. The current abstract alone is not sufficient to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a benchmark paper, not a physics breakthrough. It introduces CMPhysBench, 520+ graduate-level calculation problems in condensed matter physics, and a new scoring metric, SEED, based on expression-tree edit distance. The headline result is that the best model (Grok-4) gets only 36 average SEED and 28% accuracy. That claimed capability gap is the paper's selling point.\n\nWhat is actually new: a domain-specific benchmark for a technical science field, focused exclusively on calculation problems rather than factual QA, and a non-binary scoring method that gives partial credit. The authors have made code and dataset public, which is real evidence and should be credited. If the benchmark is well-constructed, it is genuinely useful for model evaluation and for tracking progress in AI-assisted condensed matter research.\n\nNow the soft spots. The abstract gives no details on question validation, human-expert agreement, contamination checks, or statistical uncertainty. The SEED metric is described only as tree-based edit distance; the stress-test worry is real: tree edit distance is sensitive to expression form, not semantics. If SEED rewards structurally similar but wrong algebra, or penalizes valid rewrites like (x+1)^2 vs. x^2+2x+1, the reported numbers might not measure mathematical correctness. That is a load-bearing concern because the entire conclusion rests on two scores. However, it is not a fatal flaw in the abstract alone. The public repository is the place to check whether SEED has been validated against known-equivalent expressions or human grading. If the full paper includes such validation, the results stand; if not, the benchmark is still useful but the capability gap needs qualification.\n\nI also note that the reader's soundness score of 4/10 is a bit harsh given that code and data are available for independent checking. The abstract is thin, but the paper is not unverifiable.\n\nWho this is for: anyone building or evaluating LLMs for scientific problem-solving, and researchers in condensed matter who care about AI tools. It deserves a serious referee. The review should focus on the SEED validation, contamination controls, and whether the questions truly require graduate-level reasoning. I would take it.\n\nRecommendation: send to peer review with a request for a SEED validation study and more transparency on curation. If that holds, this becomes a citeable benchmark.","headline":"A useful new benchmark for LLMs in condensed matter physics, but the central scoring metric is under-described in the abstract; worth sending to review given public code/data.","tokens_in":1425,"tokens_out":1259,"would_cite":true,"duration_ms":16512,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CMPhysBench shows top LLMs solve only 28% of graduate condensed-matter calculation problems.","keywords":["condensed matter physics","large language models","benchmark","SEED score","expression edit distance","graduate-level problems","symbolic evaluation","reasoning evaluation"],"falsifier":"Have a panel of physics graduate students solve a random sample of CMPhysBench problems, grade their answers with SEED, and compare their scores to Grok-4's; if human SEED scores are comparable to the model's, the benchmark is not measuring graduate-level capability. Also inspect cases where SEED gives high partial credit but the answer is algebraically wrong, or where correct equivalent expressions are scored low.","tokens_in":669,"feed_emoji":"🧲","tokens_out":2669,"duration_ms":30394,"temperature":0.7,"pith_summary":"CMPhysBench asks whether large language models can actually do graduate-level condensed matter physics, not just talk about it. The paper builds a benchmark of over 520 calculation problems spanning magnetism, superconductivity, and strongly correlated systems, and requires models to produce full solutions. To grade those solutions, it introduces SEED, a tree-based expression edit distance that gives partial credit for near-miss algebra. On this test the best model, Grok-4, scores 36 on the SEED scale and only 28 percent exact accuracy. The paper's claim is that current LLMs show a large capability gap in this practical, mathematically heavy domain.","feed_headline":"Top LLMs solve 28% of graduate condensed-matter problems","feed_subtitle":"New benchmark CMPhysBench tests 520+ calculation problems; best model Grok-4 reaches only 28% exact accuracy.","key_machinery":"The SEED (Scalable Expression Edit Distance) score is the central object. It represents mathematical expressions as trees and measures the minimum edit operations needed to transform a model's predicted expression into the ground-truth expression. This gives non-binary partial credit and is intended to be a finer-grained and more accurate similarity metric than exact-match accuracy.","core_discovery":"The paper's central claim is that state-of-the-art LLMs cannot yet solve most graduate-level condensed matter calculation problems, and that the shortfall is measurable precisely because SEED grades partial progress rather than binary right/wrong. By restricting the benchmark to calculation problems where a ground-truth expression exists, and scoring predictions through tree edit distance, the authors argue they capture how close a model's reasoning comes to correct algebra. Their reported numbers—36 average SEED, 28 percent exact accuracy for the best model—are the evidence for the capability gap.","pith_inferences":["A natural extension: use SEED on other symbolic-math domains (quantum chemistry, fluid dynamics) to see whether the 28% ceiling is generic or specific to condensed matter.","The 36 SEED score may overstate similarity because tree edit distance rewards structurally similar but algebraically wrong expressions; testing against human expert grading on a sample would tell.","Models trained on CMPhysBench's public data could inflate scores via memorization; a held-out dynamic question generation would test genuine transfer.","If the gap is real, it implies that current LLM scaling on web text will not automatically produce graduate-level physics calculation ability; targeted training on derivation traces may be needed."],"forward_implications":["If true, LLM performance on graduate physics calculation is far behind typical text tasks.","CMPhysBench can serve as a stable evaluation target for future physics-capable LLMs.","SEED-style scoring could be used beyond this benchmark for any symbolic math evaluation.","The public code and data allow direct comparison and reproduction of the reported gap."],"supporting_citations":[],"fun_headline_variants":["LLMs flunk graduate condensed-matter calculations","Grok-4 scores only 28% on new physics benchmark","CMPhysBench: AI still weak in condensed matter","Top AI models fail most condensed-matter problems","New benchmark shows LLMs' physics gap: 28% accuracy"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"CMPhysBench's validity rests on the curated problems being genuine graduate-level condensed matter calculations and on SEED faithfully scoring mathematical correctness—if ground-truth expressions are wrong, leaked, or if SEED penalizes valid alternative rewrites, the reported capability gap would be mismeasured.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flunk graduate condensed-matter calculations","Grok-4 scores only 28% on new physics benchmark","CMPhysBench: AI still weak in condensed matter","Top AI models fail most condensed-matter problems","New benchmark shows LLMs' physics gap: 28% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1137,"prompt_tokens":696,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":440,"tokens_out":441,"duration_ms":5206,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:32:35.723699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of physics graduate students solve a random sample of CMPhysBench problems, grade their answers with SEED, and compare their scores to Grok-4's; if human SEED scores are comparable to the model's, the benchmark is not measuring graduate-level capability. Also inspect cases where SEED gives high partial credit but the answer is algebraically wrong, or where correct equivalent expressions are scored low.","supporting_citations":[],"review_version":1}