{"id":"f1385472-60d7-45b8-831a-fadb94b3a790","arxiv_id":"2501.04277","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On the MaScQA benchmark, Claude-3.5-Sonnet and GPT-4o achieve about 84 percent accuracy, while the best open-source models (Llama3-70b, Phi3-14b) reach about 56 and 43 percent.","lead":"This paper tests 15 large language models on a materials science question-answering benchmark and reports that the best closed-source models answer about 84 percent correctly, while the best open-source model reaches about 56 percent. The authors intend these numbers as a baseline for future fine-tuning of open-source models for materials science.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Judge bias on unvalidated weak models is the load-bearing risk: GPT-4o was manually checked on 4 of 15 models, and known errors run one-way for weak models, so reported open-source accuracies may be understated.","rationale":"The reader's weakest assumption correctly identifies the central reliability issue. The paper's headline ranking is backed by a large margin (Claude-3.5-Sonnet and GPT-4o near 84% versus Llama3-70b near 56%), so that ordering is unlikely to flip. However, the quantitative claims about open-source models and the implied fine-tuning improvement depend on accuracies for weak models that were never manually validated. The paper itself shows that misclassification by GPT-4o was one-directional for Llama2-7b in the first prompt (48 correct answers converted from incorrect to correct), and the final prompt still leaves 15 errors out of 644 for that model. Since outputs from weak models are disproportionately ambiguous, the same bias can plausibly persist for unvalidated models and compress or exaggerate the open-source gap. A deterministic letter-parsing check for the multiple-choice categories plus targeted human rescoring of NUM answers would settle whether the reported weak-model accuracies are accurate. This does not warrant rejection, but it justifies a conditional acceptance pending that validation.","tokens_in":839,"tokens_out":720,"duration_ms":40704,"concrete_test":"Download the raw response files from the GitHub repository. For the 420 MCQ/MATCH/MCQN questions, extract the letter from the final answer list with a deterministic parser and compare to ground truth, bypassing GPT-4o entirely. For the 224 NUM questions, have two independent human annotators score a stratified sample (at least 150 outputs from Llama2-70b, Phi3-14b, Llama3-70b, and GPT-4o), then compute corrected accuracies. If any weak-model accuracy shifts by more than 2-3 percentage points or the open-source ordering changes, the automated-judge bias concern lands; if numbers match, the current tables stand.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2.2 validates the GPT-4o judge on only four models (Llama2-7b, Llama3-8b, Mistral-7b, GPT-4) under the final prompt; the other eleven models are scored without any manual check. The reported errors are not symmetric: with the first prompt, 48/644 Llama2-7b correct answers were marked incorrect, raising its score from 85 to 133 (+56%); the second prompt still leaves 15/644 errors for Llama2-7b. Because weak models produce ambiguous answers (Fig. 3), an automated judge can systematically under-count correct answers for exactly the models whose accuracies (Llama2-70b 28.1%, Phi3-3.8b 36.5%, Phi3-14b 42.8%) matter for the paper's fine-tuning-gap claim. The top ranking (84% vs 56%) is likely robust, but the absolute open-source numbers and the size of the improvement that fine-tuning could bring are not established without judge validation on those models.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates 15 LLMs (closed- and open-source) on the MaScQA benchmark of 644 questions, using GPT-4o as an automated judge with a three-run protocol at temperature 0, and compares results to a random baseline and to the original Zaki et al. study. It reports Claude-3.5-Sonnet and GPT-4o as top performers at roughly 84% accuracy, Llama3-70b at 56.0%, Phi3-14b at 42.8%, and weaker open-source models far below. The authors frame the results as a baseline for fine-tuning an open-source materials-science LLM.","tokens_in":18127,"tokens_out":5294,"duration_ms":51027,"significance":"If the measurements are accurate, the paper provides a useful and reproducible baseline for LLM performance on a domain-specific benchmark, with open code and data, three repeated runs per question, and a transparent comparison to prior work. The central qualitative ranking (closed-source models outperform open-source ones) is plausible and broadly consistent with other LLM evaluations. However, the quantitative claims about open-source models, and hence the perceived fine-tuning gap, rest on validation of the automated judge on only four of fifteen models, so the absolute numbers should be treated with caution until the judge is validated on the remaining models.","major_comments":[{"comment":"The GPT-4o judge is manually validated on only four of fifteen models, and the reported misclassifications are systematically one-directional for weaker models: for Llama2-7b, 48 of 644 correct answers were labeled incorrect under the first prompt, and 15 of 644 remain misclassified under the final prompt. The remaining eleven models, including all Phi3 models, Llama2-70b, Llama3-70b, Mistral-7b, and several Claude/GPT variants, are scored without any manual check. Because the paper's claims about open-source performance (Llama3-70b at 56.0%, Phi3-14b at 42.8%, Llama2-70b at 28.1%) and the size of the fine-tuning gap depend on these absolute numbers, the authors should manually score a random sample of responses for each unvalidated model and report corrected accuracies or a bias-adjusted confidence interval.","section":"§2.2, Tables 2–3"},{"comment":"The evaluation prompt instructs GPT-4o to judge an answer as correct only if the reasoning is sound, yet §2.2 states that for MATCH, MCQ, and MCQN questions the response is assessed solely by the selected letter. This contradiction means that answers with the correct option but flawed reasoning—precisely the ambiguous cases shown in Figure 3—may be marked incorrect, systematically lowering scores for weak models that produce verbose but incorrect explanations. The prompt should be aligned with the stated letter-based policy, or the policy should be revised and the judge revalidated under the actual scoring rule.","section":"§2.2.2, Figure 4(b)"},{"comment":"The prose misreports the numerical results in Table 5: it says Claude-3-Opus and GPT-4-turbo 'closely follow with ∼64−63%' when the table lists 77.9% and 76.4%, and it describes Llama3-70b as achieving results 'with ∼32.6%' when its overall accuracy is 56.0% (32.6% is its NUM-category score). These discrepancies make the results difficult to interpret and should be corrected.","section":"§3, text versus Table 5"}],"minor_comments":[{"comment":"The statement that manually removing six questions 'does not significantly bias the evaluation outcomes' is an assertion without supporting analysis; please provide the list of removed questions and a brief justification, or quantify the sensitivity.","section":"§1.2"},{"comment":"For the four models tested on both machines (GPT-4, GPT-3.5-turbo, Llama2-7b, Llama3-8b), Table 5 reports a single accuracy, but the paper does not state how the two machine-specific results are combined; please clarify whether the reported value is the average of the two runs or a pooled result.","section":"Tables 4–5"},{"comment":"The caption says the figure includes 'the 15 unique LLMs tested' but the plotted quantities are per-category total correct answers; please clarify whether these are per-category totals averaged over all models, and label the axes accordingly.","section":"Figure 5"},{"comment":"Setting the random baseline for NUM questions to exactly 0% with zero standard deviation is a reasonable simplification, but in practice the accepted numerical range may be large enough that random guessing has a tiny nonzero probability; a brief note acknowledging this would be more precise.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a benchmark evaluation that is within the scope of physics.comp-ph only insofar as it concerns computational methods for materials science; the main technical weakness is the incomplete validation of the automated judge on the models whose accuracies drive the paper's conclusions. The authors' candid discussion of evaluator bias is a positive sign, but it does not replace a quantitative correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate and useful benchmark extension, and the headline ranking is probably right, but the paper overstates the precision of the open-source numbers. The GPT-4o judge was manually checked on only four models, and the known errors run one-way for weak models, so the bottom of the table carries more uncertainty than the error bars show.\n\nWhat's actually new: Zaki et al. tested three models on MaScQA; this paper sweeps fifteen, adds a three-run repeat, a random baseline, per-category breakdowns, and a hardware comparison. They even ship the raw responses and code on GitHub. That's real empirical work, and the paper is honest about the judge problem: they report misclassification counts for the four models they checked, show examples of ambiguous Llama2-7b answers, and try a second, more detailed prompt that cuts errors from 48 to 15 out of 644. Credit where due.\n\nThe soft spots are not fatal but they matter. The judge was validated on four models; the other eleven are scored with no manual check. For Llama2-7b the final prompt still misclassifies 15 of 644 answers, and the errors are asymmetric—correct answers marked incorrect much more often than the reverse. Weak models produce ambiguous outputs, and an automated judge can systematically undercount them. So the absolute accuracies for Llama2-70b (28%), Phi3-3.8b (36%), and Phi3-14b (43%) are probably lower bounds, and the size of the \"fine-tuning gap\" they advertise is not pinned down. The ranking of closed versus open is robust; the specific numbers less so.\n\nThe hardware split is a smaller but real nuisance: some local models run on a MacBook M1, some on an 8×A100 box, and they report a 16% difference for Llama3-8b between the two. That makes cross-model comparisons a bit dirty, though it likely doesn't change the top-tier ordering. They also removed six benchmark questions ad hoc; the details are in the SI, but the paper should at least list them in the main text. And they never discuss training-data contamination: MaScQA is built from GATE, a public exam, so these models may have seen the exact questions. That's a known risk and should be addressed.\n\nMinor internal issues: the abstract says GPT-4 where the body says GPT-4o, and one results paragraph says the top models exceeded ~70% just before the tables report 84%. These look like typos, but they'll confuse readers.\n\nBottom line: this is a good empirical baseline for anyone working on materials-science LLMs, and it deserves a serious referee. The right fix is not rejection; it's asking for judge validation on more model families, a transparent list of the removed questions, and a contamination discussion. I'd cite it as a comparison point and bring it to a reading group.\n\nRecommendation: engage with it, send it to peer review, and push for the revisions above.","headline":"A useful and honest benchmark sweep of 15 LLMs on MaScQA; the headline ranking is credible, but the absolute numbers for weak open-source models are shakier than the tables suggest because the automated judge was validated on only 4 of the 15 models.","tokens_in":18658,"tokens_out":2384,"would_cite":true,"duration_ms":24282,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Closed-source LLMs answer materials-science exam questions with about 84 percent accuracy, while the best open-source model reaches only 56 percent.","keywords":["LLM evaluation","materials science","MaScQA benchmark","question answering","numerical reasoning","open-source language models","LLM-as-a-judge","metallurgical engineering"],"falsifier":"Re-score every model's answers by hand, or with a judge whose error rate is measured per model, and compare the accuracy table; in particular, manually grading the 224 open numerical answers of the weakest models would reveal whether their 2–8 percent NUM scores are real or an artifact of the judge marking correct but messily formatted answers as incorrect.","tokens_in":17720,"feed_emoji":"🧪","tokens_out":6933,"duration_ms":57485,"temperature":0.7,"pith_summary":"The paper sets out to measure how well existing large language models know materials science before any domain-specific training. Using the MaScQA benchmark, a set of 644 exam questions from the Graduate Aptitude Test in Engineering covering four question formats and fourteen sub-fields, it evaluates fifteen models and reports that the closed-source Claude-3.5-Sonnet and GPT-4o answer about 84 percent correctly. The best open-source model, Llama3-70b, reaches about 56 percent, and the smaller Phi3-14b about 43 percent. These numbers matter because the authors need a baseline against which to measure a planned fine-tuned open-source model and retrieval-augmented generation system for materials science.","feed_headline":"Top LLMs hit 84 percent on materials-science exam","feed_subtitle":"Best open-source model trails at 56 percent, setting a baseline for fine-tuning open models.","key_machinery":"The central machinery is the MaScQA benchmark itself plus the scoring pipeline built around it. MaScQA contributes 644 questions divided into four types — MATCH (match the following), MCQ (multiple choice), MCQN (multiple choice numerical), and NUM (open numerical) — which together test pattern matching, conceptual knowledge, and arithmetic. To score roughly 37,000 model answers without manual grading, the authors use an LLM-as-a-judge setup in which GPT-4o classifies each answer against the gold answer, using a revised prompt that instructs the judge to check both the selected option and any accompanying reasoning. They also fix the generation temperature to 0, submit each question three times, and compare every model against a random baseline derived from the binomial distribution with success probability $p = 0.25$ for the multiple-choice formats and zero for NUM. The benchmark defines what counts as success, the judge converts responses into accuracy numbers, and the random baseline separates real knowledge from guessing.","core_discovery":"On the authors' measurements, Claude-3.5-Sonnet is the top model at 83.9±0.2 percent overall accuracy, with GPT-4o essentially tied at 83.8±1.3 percent; Claude-3-Opus (77.9), GPT-4-turbo (76.4), and GPT-4o-mini (70.5) follow, while GPT-4 sits at 60.8 percent. Among open-source models, Llama3-70b leads at 56.0 percent, Phi3-14b reaches 42.8 percent, and Phi3-3.8b reaches 36.5 percent; Llama2-7b, Llama2-70b, Llama3-8b, and Mistral-7b all fall below 32 percent, with Llama2-7b and Mistral-7b scoring below the random guessing baseline of 16.3 percent on parts of the test. The open-ended numerical questions (NUM) are the hardest category: even the best model, Claude-3.5-Sonnet, gets 74.6 percent there, while the weaker open models score in the low single digits. The paper interprets the results as a raw-capability baseline and as evidence that fine-tuning and prompt engineering have a large potential upside for open-source models.","pith_inferences":["If the judge's errors are systematic against poorly formatted answers, the open-source scores are likely lower bounds rather than point estimates, and the gap between closed and open models may be smaller than reported.","The Phi3-3.8b result — 36.5 percent at 3.8 billion parameters, above several larger models — suggests parameter count is not the main driver, so fine-tuning a small open model may yield outsized returns at low hardware cost.","Because GATE questions are public exam materials, the closed-source models may have encountered them during pretraining, so the 84 percent figure could overstate genuine materials-science reasoning; a fresh question set would test transfer.","The NUM category could serve as a cheap diagnostic for tokenization and arithmetic quality when comparing future model versions."],"forward_implications":["Claude-3.5-Sonnet and GPT-4o set the current ceiling at roughly 84 percent, with GPT-4o strongest on multiple-choice questions and Claude-3.5-Sonnet strongest on open numerical ones.","The best open-source model, Llama3-70b, trails the closed-source leaders by about 28 points, quantifying the headroom that fine-tuning would need to close.","Open-ended numerical questions are the universal weak spot: every model scores lowest there, and the weakest models fall to 2–9 percent.","Several open-source models, including Llama2-7b and Mistral-7b, perform at or below random guessing in MATCH and MCQN categories, indicating systematic reasoning flaws rather than mere knowledge gaps.","Local hardware matters for local models: Llama3-8b improved by about 16 percent when moved from a MacBook M1 to a GPU server, while API-based models were unaffected."],"supporting_citations":[{"why":"Supplies the MaScQA benchmark, its question taxonomy, and the original three-model results that this study extends.","marker":"[1]"},{"why":"Provides the LLM-as-a-judge methodology that the authors adapt, using GPT-4o to score model answers.","marker":"[27]"},{"why":"Cited as evidence that multi-step numerical tasks are hard for LLMs, used to interpret the NUM and MCQN results.","marker":"[20]"},{"why":"Cited as a chemistry benchmark with numerical difficulty patterns that the authors compare against their MCQN and NUM findings.","marker":"[13]"},{"why":"Cited for showing that MATCH-type and MCQ-type weaknesses are not unique to materials science.","marker":"[12]"}],"fun_headline_variants":["Materials exam: closed LLMs 84%, open best 56%","Claude-3.5 and GPT-4o top materials exam at 84%","Open-source LLMs lag big on materials science exam","Best open model hits 56% vs 84% for closed on materials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy table rests on the assumption that GPT-4o, as automated judge, classifies answers correctly for all fifteen models, but it was manually validated on only four models, and even with the final prompt it still misclassified 15 of Llama2-7b's 644 answers — errors that could bias the reported scores, especially for weak open-source models, and shift the ranking.","fun_headline_variants_meta":{"raw":{"variants":["Materials exam: closed LLMs 84%, open best 56%","Claude-3.5 and GPT-4o top materials exam at 84%","Open-source LLMs lag big on materials science exam","Best open model hits 56% vs 84% for closed on materials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3589,"prompt_tokens":1065,"completion_tokens":2524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":2444}},"tokens_in":681,"tokens_out":2524,"duration_ms":16371,"temperature":1.0,"reasoning_tokens":2444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:35:58.831291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score every model's answers by hand, or with a judge whose error rate is measured per model, and compare the accuracy table; in particular, manually grading the 224 open numerical answers of the weakest models would reveal whether their 2–8 percent NUM scores are real or an artifact of the judge marking correct but messily formatted answers as incorrect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MaScQA benchmark, its question taxonomy, and the original three-model results that this study extends."}],"review_version":1}