{"id":"2132710c-deb2-42b7-b4d9-31c8e539f315","arxiv_id":"2505.03049","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A hackathon report showing 34 LLM-based prototypes across materials science and chemistry, with a few preliminary quantitative evaluations of property prediction, educational QA, and multimodal reasoning.","lead":"This paper describes 34 prototype projects from a 2024 hackathon that applied large language models to materials science and chemistry tasks. It organizes them into seven categories and highlights a few examples, such as LLM-based phonon prediction, MOF design agents, and a multimodal chemistry QA benchmark.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'constellation of capabilities' claim treats self-selected, largely unvalidated hackathon prototypes as demonstrated capacity; reproducibility of the 34 projects is the load-bearing unknown.","rationale":"The reader's CONDITIONAL verdict and weakest_assumption—that the report's usefulness claim depends on hackathon projects being functional and representative—match the most load-bearing concern I can identify. The paper is transparent about many limitations, and it is not internally inconsistent: it labels projects as prototypes and proof-of-concept. The problem is that the Abstract and Conclusion move from those caveated descriptions to a strong general capability claim, while the strongest quantitative anchor (phonon MAE) has no error bars and the hypothesis counts come from unvalidated LLM judges. This is an empirical question, not an ideological one, and it is checkable from the provided GitHub and Zenodo links: do the repositories run, and do the headline numbers survive reseeding? I do not see grounds to reject the paper; it is a useful survey of a hackathon. But the breadth claim should be conditioned on an independent reproducibility audit, which is exactly what the CONDITIONAL verdict requests. Since my concern does not move the reader's verdict, I mark the verdict as UNCHANGED.","tokens_in":28108,"tokens_out":6809,"duration_ms":77594,"concrete_test":"Audit a random sample of eight projects from Table 1 by cloning their repositories and running each documented quickstart to reproduce one claimed output per project, scored pass/fail against the paper's text. Separately, re-run the Section 1.1 phonon fine-tuning with the published Zenodo data and five random seeds, computing a paired bootstrap confidence interval for the 44-to-38 cm^-1 MAE improvement. If fewer than six of eight audits pass, or if the bootstrap interval includes zero, the 'constellation of capabilities' conclusion should be downgraded to 'reported prototypes.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the 34 projects collectively form a demonstrated 'constellation of capabilities' spanning the research lifecycle—rests on treating hackathon submissions as evidence of functioning tools. The paper provides no selection protocol, no independent evaluation, and no reproducibility check. Several highlighted results are internally flagged as preliminary: Section 1.1 reports the phonon MAE improvement (44 vs 38 cm^-1) from a single split with no error bars and states the model 'has not yet been exhaustively analyzed, validated, or optimized yet'; Section 6.1 derives feasibility/utility/novelty counts (243/175/12) from LLM-as-a-judge agents without human validation; and Section 5.2 concedes the NOMAD Query Reporter handled homogenized data but struggled on heterogeneous ELN data. These caveats appear in the body text, yet the Abstract and Conclusion generalize to broad utility and 'immense promise.' If a substantial fraction of the 34 repositories do not run as documented, or if the headline numbers do not reproduce under reseeding, the conclusion overstates what the hackathon actually demonstrated. This is a structural evidence-quality concern, not a question of author intent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on 34 projects from the second LLM Hackathon for Applications in Materials Science and Chemistry. It categorizes the submissions into seven research areas, highlights an exemplar project in each area, provides a table of all projects with links to their code repositories, and argues that the projects collectively form a 'constellation of capabilities' spanning the research lifecycle. The supporting evidence is primarily qualitative description plus selected quantitative results, including a phonon peak prediction MAE comparison, multiple-choice accuracy scores for LLMs on materials science questions, multimodal accuracy comparisons for ChemQA, and counts of generated hypotheses deemed feasible, useful, and novel.","tokens_in":28298,"tokens_out":3527,"duration_ms":37045,"significance":"If taken as a community report on early LLM applications, the paper is a useful catalog: the repository table alone is a practical resource, and the breadth of application areas gives a realistic snapshot of what hackathon teams attempt with current models. The paper also deserves credit for explicitly flagging several limitations, such as the preliminary nature of the phonon model, the difficulty of heterogeneous ELN data in the NOMAD Query Reporter, and the reliance on automated judges for hypothesis evaluation. However, the central claim that the projects form a demonstrated 'constellation of capabilities' goes beyond what the evidence supports: most results are self-reported prototypes without independent evaluation, error bars, or reproducibility checks. The paper is therefore valuable as a collection of use cases and community-building reflections, but it does not yet constitute a rigorous demonstration of LLM utility across the research lifecycle.","major_comments":[{"comment":"The phonon peak prediction result (MAE 44 vs. 38 cm^-1) is the paper's headline quantitative evidence for property prediction, but it is based on a single train/test/validation split with no error bars, no repeated seeds, and no significance test. The text itself states that the model 'has not yet been exhaustively analyzed, validated, or optimized yet.' Because this result is used to support the claim that bonding information improves LLM predictions, the paper should either provide repeated-split or bootstrap statistics, or explicitly label this as a preliminary illustration rather than a demonstrated improvement.","section":"Section 1.1"},{"comment":"The hypothesis counts (243 feasible, 175 useful, 12 novel) are produced by LLM-as-a-judge evaluation agents without any human validation or inter-annotator agreement, yet the text describes the evaluation as 'rigorous.' This is load-bearing for the hypothesis-generation category. The paper should either temper the claim by describing these as model-generated estimates, or provide a human-validated sample to support the counts.","section":"Section 6.1"},{"comment":"The NOMAD Query Reporter section concedes that the prototype 'was able to manage homogenized hits well' but struggled with heterogeneous ELN data. Despite this, the Abstract and Conclusion generalize to broad utility across the research lifecycle. Please narrow the generalization to the demonstrated scope, or add evidence that the tool handles heterogeneous data successfully.","section":"Section 5.2 and Abstract/Conclusion"},{"comment":"The paper does not state a selection protocol for the 34 projects or the choice of exemplar projects, and the evaluations are self-reported by teams, several of whom are also authors. For a claim of 'demonstrated' capabilities, please specify how submissions were included or excluded and what independent verification, if any, was performed beyond team self-reports.","section":"Overview of Submissions and Table 1"}],"minor_comments":[{"comment":"The table header reads 'Overview of the tools developed by the various tools, and links to source code repositories' — 'tools' should be 'teams'.","section":"Table 1 heading"},{"comment":"The ChemQA evaluation reports accuracy differences across modalities but does not state the number of questions per task or provide error bars; please include these details for interpretability.","section":"Section 7.3"},{"comment":"The comparison of the team's surrogate MOFormer MAE (0.467) with the published MOFormer MAE (0.387) appears to be on different model versions or test sets; please clarify the comparison.","section":"Section 2.1"},{"comment":"LangSim is demonstrated on a single inverse-alloy-design example without reporting any success rate across multiple runs; a brief statement of reliability would strengthen the prototype description.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is written largely by hackathon organizers and participating team members, which creates an inherent conflict of interest in evaluating the projects. The paper is more of a community report than a scientific study, and the journal should consider whether that framing is appropriate for its scope. The repository table and the candid limitation statements are genuine strengths; the overgeneralization in the Abstract and Conclusion is the main fixable weakness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is not really a research paper with new results; it is an event report summarizing 34 hackathon projects, and its value is as a structured, well-linked catalog of what LLM teams tried in materials and chemistry over a weekend. Second, the paper is better than the promotional bits suggest: the authors repeatedly flag their own limitations in the body text, and the descriptive claims are mostly consistent with the linked repositories.\n\nWhat works: the seven-category organization is sensible, the table of projects and code links is genuinely useful, and the highlighted mini-evaluations are honest about their preliminary status. The phonon prediction fine-tuning (MAE 44 vs 38 cm^-1 with bonding descriptors) is a real signal, even if it is one split with no error bars. The MaSTeA benchmark on MaScQA and the ChemQA multimodal comparison are usable early data points. The paper also explicitly concedes that the NOMAD query reporter struggled on heterogeneous ELN data and that the hypothesis counts (243/175/12) came from LLM-as-judge without human validation. That level of candor earns credit.\n\nThe soft spots are real but proportionate. The central claim that the projects form a demonstrated 'constellation of capabilities' outruns the evidence: the projects were self-selected, many are prototypes, and no independent evaluation or reproducibility check is provided. The highlighted numbers are thin, and the abstract/conclusion generalize more strongly than the body supports. The overlap with companion report [31] is substantial, though this version adds the mini-evaluations and a cleaner synthesis. The promotional language ('immense promise,' 'Copernican revolution') is unnecessary and should be toned down.\n\nThe stress-test note is essentially right: the load-bearing unknown is reproducibility of the 34 repositories, and the 'constellation' framing treats prototypes as demonstrated capacity. But that is a structural feature of hackathon reports, not a sign of author dishonesty. The paper's own caveats should be surfaced in the abstract and conclusion, not just buried in project descriptions.\n\nWho is this for? Practitioners who want a quick map of LLM applications, hackathon organizers, and newcomers looking for project ideas and baseline examples. It deserves a serious referee, but as a review/event report, not as a venue for the quantitative claims. My recommendation: send it to peer review with a request for revision—move the caveats forward, soften the conclusion, and either add reproducibility statements for the featured repositories or explicitly label them as unvalidated prototypes.","headline":"A well-organized, transparent hackathon report that is genuinely useful as a map of the landscape, but the 'constellation of capabilities' framing outruns the thin, self-selected evidence.","tokens_in":29050,"tokens_out":1097,"would_cite":false,"duration_ms":14165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that LLMs are broadly useful across the materials science and chemistry research lifecycle, and uses 34 hackathon projects to support that claim.","keywords":["large language models","materials science","chemistry","scientific workflows","property prediction","hypothesis generation","retrieval-augmented generation","hackathon prototypes"],"falsifier":"A rerun of the phonon fine-tuning on the same data with independent code and held-out splits, compared against the same benchmark range: if the mean absolute error does not fall from 44 toward 38 cm$^{-1}$ or land inside the benchmark band, the central example fails. Independently, human expert review of a random sample of the 12 highly novel hypotheses would test the multi-agent pipeline; if no genuine novelties survive, the claim that the pipeline contributes to discovery is not supported.","tokens_in":27888,"feed_emoji":"🧪","tokens_out":10171,"duration_ms":93996,"temperature":0.7,"pith_summary":"This paper argues that large language models have become broadly useful across the entire materials science and chemistry research cycle, not just for isolated text tasks. It bases that claim on a curated review of 34 projects built during a 24-hour global hackathon, grouped into seven areas: property prediction, material design, automation and interfaces, education and communication, data management, hypothesis generation, and knowledge extraction. The concrete demonstrations include a fine-tuned model that predicts phonon peak positions with a mean absolute error of 38 cm$^{-1}$ when given quantum-chemical bonding descriptions, within the 29 to 68 cm$^{-1}$ range of established benchmarks, and a multimodal question-answering study showing that models perform better when text and images are combined. The paper presents these prototypes as a \"constellation of capabilities\" that, taken together, could let LLMs assist at every stage of research. If the pattern holds, scientists get a low-resource route to domain-specific tools without waiting for specialized models.","feed_headline":"LLMs now cover every stage of materials research","feed_subtitle":"A 34-project hackathon review maps language-model tools onto property prediction, design, automation, and discovery.","key_machinery":"The load-bearing object is a taxonomy: a seven-cell map of LLM capabilities arranged around the research lifecycle, from property prediction and material design through automation, education, data management, hypothesis generation, and knowledge extraction. What carries the argument is the corpus of 34 prototypes slotted into that map, each combining a general-purpose LLM with an orchestration technique such as fine-tuning on domain text, retrieval-augmented generation, tool-calling agents, or multi-agent prompting. The mechanism is aggregate demonstration: because each cell contains at least one working prototype built in a short timeframe, the taxonomy is presented as evidence of real capability rather than future aspiration.","core_discovery":"The central claim, stated on the authors' terms, is that 34 working prototypes demonstrate LLMs as versatile predictive models, platforms for rapid prototyping of domain-specific tools, and more, across the research lifecycle. The strongest quantitative evidence is the phonon-peak prediction experiment, where adding orbital-based bonding descriptions to crystal-structure text lowered the fine-tuned model's mean absolute error from 44 to 38 cm$^{-1}$, inside the 29 to 68 cm$^{-1}$ band of established benchmark models. A second strand of evidence shows that current models answer chemistry questions better when given both molecular images and text than when given images alone. The paper does not claim any single tool is production-ready; its claim is that the breadth itself is the finding, because every stage of research has at least one demonstrated LLM application.","pith_inferences":["Inference: The report's strongest evidence is narrower than its broad framing; the first testable extension is independent human scoring of a sample of outputs from each showcased prototype.","Inference: The pattern suggests the near-term payoff of LLMs in these fields is rapid prototyping and workflow integration, not autonomous discovery, so autonomous claims should be read as feasibility demonstrations.","Inference: The phonon improvement from adding bonding descriptors hints that text descriptions can encode physical information that pure structure strings miss; a cheap experiment would test whether the same descriptors improve other thermal and elastic property predictions.","Inference: Because several tools depend on proprietary model APIs, rebuilding the same constellation with open-weight models and comparing error rates would directly test how much of the reported capability is tied to any specific model."],"forward_implications":["Fine-tuning a general-purpose LLM on domain text descriptions is a low-resource route to property predictors: the phonon experiment reached benchmark-competitive accuracy with standard fine-tuning on just over a thousand structures.","Natural-language agents can lower the barrier to specialized instruments and simulation codes, letting non-experts run tasks such as finding an alloy composition with a target bulk modulus.","Retrieval-augmented tools can turn large unstructured data sources into queryable summaries and structured knowledge, which should make repository data easier to audit and reuse.","Multi-agent hypothesis pipelines can generate hundreds of candidate research questions, but their feasibility, utility, and novelty scores still need human adjudication before they count as discovery.","Because model improvements propagate across all seven categories, each new model release should raise the floor of every prototype, not just one task."],"supporting_citations":[{"why":"Makes the baseline case that LLMs can be leveraged for predictive chemistry, which the property-prediction section builds on.","marker":"[1]"},{"why":"Describes the first hackathon and its 14 examples, the template this event extends to 34 projects.","marker":"[30]"},{"why":"Provides the full per-project descriptions from which the review draws its seven categories and highlights.","marker":"[31]"},{"why":"Supplies the quantum-chemical bonding database used in the phonon-peak fine-tuning experiment.","marker":"[34]"},{"why":"Defines the benchmark leaderboard range for phonon peak prediction against which the fine-tuned model's error is judged.","marker":"[37]"},{"why":"Provides the undergraduate materials-science question set used in the teaching-assistant evaluation.","marker":"[80]"},{"why":"Defines retrieval-augmented generation, the technique several of the reported tools rely on to ground their answers.","marker":"[88]"}],"fun_headline_variants":["34 LLM projects tackle materials science from all angles","LLMs power prediction, design, and automation in materials","Materials research gets 34 LLM tools across the lifecycle","Hackathon's 34 LLM apps cover materials discovery end-to-end","LLMs now aid every stage of materials and chemistry research"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 34 hackathon projects work as described and that their results are representative rather than cherry-picked; the paper itself notes that the phonon model is not yet exhaustively validated and that the hypothesis counts come from model self-assessment.","fun_headline_variants_meta":{"raw":{"variants":["34 LLM projects tackle materials science from all angles","LLMs power prediction, design, and automation in materials","Materials research gets 34 LLM tools across the lifecycle","Hackathon's 34 LLM apps cover materials discovery end-to-end","LLMs now aid every stage of materials and chemistry research"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1510,"prompt_tokens":978,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":594,"tokens_out":532,"duration_ms":5791,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:00:31.607701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A rerun of the phonon fine-tuning on the same data with independent code and held-out splits, compared against the same benchmark range: if the mean absolute error does not fall from 44 toward 38 cm$^{-1}$ or land inside the benchmark band, the central example fails. Independently, human expert review of a random sample of the 12 highly novel hypotheses would test the multi-agent pipeline; if no genuine novelties survive, the claim that the pipeline contributes to discovery is not supported.","supporting_citations":[{"cited_title":"Mascqa: investigating materials science knowledge of large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the undergraduate materials-science question set used in the teaching-assistant evaluation."}],"review_version":1}