{"id":"c1ea5ef1-2637-43b4-9dc0-7f6553642cb7","arxiv_id":"2504.16116","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"DMind is a 3,543-item, nine-domain benchmark for LLMs in Web3; evaluation of 31 models shows strong fundamentals but weak security, token economics, and meme-related reasoning.","lead":"This paper builds a 3,543-question benchmark to test how well large language models understand Web3 topics, from smart contracts and DeFi to memecoins and security. It evaluates 31 models and finds that even top models struggle with security auditing and token economics, so the benchmark is meant as a diagnostic tool for safer AI deployment in blockchain finance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (5) with K_scale=100/9 cannot yield the reported 0–100 percentage scores; all quantitative results are unsupported as presented.","rationale":"The paper makes a valuable contribution by introducing a Web3-specific benchmark and an extensive evaluation of 31 models. The open-source release, the cross-judge reliability analysis (Appendix A), and the Pareto cost analysis are positive elements. However, the manuscript as written contains a fundamental quantitative error in its scoring formula. Eq. (5) with K_scale=100/9 bounds any total score at approximately 11.1, yet the paper reports scores around 70–80 and classifies tiers at ≥75. No amount of normalization of the individual components removes this contradiction because the weighted combination of two [0,1] values is itself in [0,1]; multiplying by 100/9 cannot reach 100. This means the reported numbers cannot have been produced by the described method. A reader cannot verify the rankings, the subdomain comparisons, or the cost-effectiveness frontier without the code—and the code is not included in the manuscript. This is more load-bearing than the contamination-resistance concern: even if the LoRA fine-tuning experiment were fully convincing, the quantitative profile of model capabilities would still be unsupported. Conversely, a correct K_scale would not rescue the paper as written because the textual and appendix inconsistencies (subjective item counts 389 vs 341 vs 48) remain unresolved. Therefore the reader's REJECT verdict is appropriate; the paper could become acceptable after correcting the formula, reconciling item counts, and releasing versioned artifacts. We partially agree with the reader's weakest_assumption: the contamination-resistance argument is indeed fragile, but the scoring formula is the single most load-bearing flaw.","tokens_in":18509,"tokens_out":7739,"duration_ms":70449,"concrete_test":"Inspect the released evaluation pipeline (test_objective.py, test_subjective.py) and recompute one reported headline score, e.g., GPT-5 Medium, from raw item responses using the code's actual scoring constants. If the code uses K_scale=100 and reproduces the table value, Eq. (5) is a typo and the concern is resolved. If the code literally implements K_scale=100/9, the computed maximum is ≤11.1, confirming the reported results are impossible. Also verify the subjective item count (389 vs 341) in the released data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines the total score via Eq. (5): Stotal = (ωobj·S̃obj + ωsubj·S̃subj)·K_scale, with S̃obj = Sobj/Sobj,max and S̃subj = Ssubj/Ssubj,max. Since ωobj+ωsubj=1 and each normalized score lies in [0,1], Stotal is bounded by K_scale = 100/9 ≈ 11.1. Yet every reported result—Figure 3, Figure 4, Table 3, and the tier thresholds (≥75, 70–75, <70)—is on a 0–100 percentage scale. The paper provides no alternate normalization that would reconcile this. As written, the formula cannot produce the headline numbers. A correct K_scale=100 would fix the scale, but the manuscript does not say this, and ancillary inconsistencies (Appendix A says |Q_subj|=341 while §3.1 says 389 subjective items; Appendix B applies IAA to only 48 of these) further erode confidence that the released pipeline matches the text. This is the most load-bearing concern because it directly invalidates the benchmark's reported scores, rankings, and thresholds that constitute the paper's central quantitative contribution. The contamination-resistance issue is secondary: even if that experiment were flawless, a wrong scoring formula still makes the measured values meaningless.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DMind Benchmark, a Web3-domain evaluation suite of 3,543 items spanning nine subfields, combining 3,154 objective multiple-choice items with 389 subjective tasks such as smart-contract auditing, on-chain numerical reasoning, and strategy analysis. The authors evaluate 31 proprietary and open-weight LLMs with a fixed protocol, report an overall ranking and per-subdomain heatmap, perform a cost-effectiveness/Pareto analysis, and include a fine-tuning-based contamination-resistance study. The central claims are that DMind provides a valid, contamination-resistant measure of Web3 competence and that current LLMs show a consistent capability profile: strong on fundamentals and infrastructure, weaker on token economics, meme concepts, and security. The dataset and evaluation pipeline are released to support reproducibility.","tokens_in":18774,"tokens_out":3577,"duration_ms":36091,"significance":"If the measurement issues are resolved, DMind could be a useful community resource: it targets an under-benchmarked high-stakes domain, ships an open dataset and pipeline, evaluates a broad model set, and provides cost-effectiveness guidance that practitioners can act on. The cross-judge agreement study and the inter-annotator agreement analysis are also valuable methodological efforts. However, the paper's central quantitative contribution is currently undermined by a direct inconsistency in the scoring formula, and the contamination-resistance evidence is weaker than the text claims. The strengths of the resource make the manuscript worth repairing, but the repair must address load-bearing technical points rather than only presentation.","major_comments":[{"comment":"Equation (5) defines Stotal = (ω_obj·S̃_obj + ω_subj·S̃_subj)·K_scale with K_scale = 100/9. Since ω_obj + ω_subj = 1 and both normalized scores lie in [0,1], the maximum possible Stotal is 100/9 ≈ 11.11. Yet Figure 3, Figure 4, Table 3, and the tier thresholds in Section 4.2 (≥75, 70–75, <70) are all presented on a 0–100 percentage scale. No alternative normalization is provided anywhere in the manuscript. This means the reported scores, rankings, and thresholds cannot be derived from the stated formula. The constant should almost certainly be K_scale = 100, but as written the manuscript is internally inconsistent. Because every quantitative claim in the paper depends on this formula, this is a load-bearing error that must be corrected in both the text and the released pipeline.","section":"Section 3.2, Eq. (5)"},{"comment":"The number of subjective items is inconsistent: Section 3.1 states the benchmark comprises 389 subjective items, Appendix A states |Q_subj| = 341, and Appendix B describes a rubric for 'each of the 48 subjective questions.' These numbers affect S_subj,max and therefore the weight ω_subj in Eq. (5) and the final composite scores. If Appendices A and B analyze only subsets, that must be stated explicitly; if the counts are errors, they must be corrected. As written, the reader cannot determine the actual composition of the benchmark or verify the claimed totals.","section":"Section 3.1, Appendix A, Appendix B"},{"comment":"The contamination-resistance experiment is internally inconsistent and overinterpreted. Section C.1 says all models undergo LoRA fine-tuning for three epochs, but Table 3 reports scores at Epoch 0 through Epoch 4 and describes 'five training iterations.' This discrepancy must be resolved: were there three epochs with five evaluation checkpoints, or five epochs? More substantively, the flat learning curves after fine-tuning on the complete DMind dataset are taken as 'strong evidence' of genuine conceptual understanding, but the experiment does not include a held-out generalization control, does not test whether the models can recall item-specific answers under different phrasing, and the paper's own Limitations section concedes that contamination cannot be definitively ruled out for proprietary models. The claim in C.3 should be tempered to a suggestive negative result rather than a definitive proof of contamination resistance.","section":"Appendix C, Table 3"}],"minor_comments":[{"comment":"The text says 'Model performance is quantified by accuracy scores (in percentages)' but does not explicitly state that the percentages are the Stotal from Eq. (5); once the scaling issue is fixed, this connection should be stated.","section":"Section 4.1"},{"comment":"The related-work section cites 'the very recent DMind Benchmark (Huang et al., 2025)' where the reference is the paper itself. This self-citation should be removed or replaced with an external comparison, since it does not provide independent context.","section":"Section 2.1"},{"comment":"The description says the panel consists of 'five mutually-unaware experts' and that none were involved in the original creation, but immediately adds that two of the paper's authors also served as raters. This needs clarification: were the author-raters among the five, and how does that square with the 'mutually-unaware' and 'impartial' description?","section":"Appendix B.1"},{"comment":"There is a typo: 'to the best of out knowledge' should be 'to the best of our knowledge.'","section":"Section 1"},{"comment":"The author list in the main text includes 'Haobo Wang' but a reference entry for the authors' own paper lists 'Hobert Wang'; this should be checked for consistency.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The scoring-formula inconsistency is the kind of issue that could be a simple typo in K_scale, but it is central enough that the paper cannot be accepted without verification of the released pipeline's actual behavior. I would ask the authors to run a small sanity check (e.g., compute Eq. (5) for one model by hand and confirm it matches the reported score) and to make the subjective-item counts consistent across the main text and appendices. The contamination-resistance claim also needs either a held-out control or a substantially more cautious wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: DMind is a real asset for the Web3 evaluation niche, but the paper's own scoring formula makes the headline numbers impossible as written. Fix the formula, reconcile the item counts, and this becomes a conditional accept rather than a reject.\n\nWhat's new and done well: this is the first benchmark I know of that covers nine Web3 subdomains with 3,543 items, mixes objective and subjective tasks, and evaluates 31 models with a sensible Pareto cost analysis. The data curation pipeline—license-aware crawls, provenance tracking, deduplication—is more careful than most benchmark papers. The qualitative finding that models are strong on fundamentals/infrastructure and weak on token economics and security is plausible and useful for deployment. Open-sourcing the dataset and pipeline is the right call, and the Hugging Face traction suggests the community wants it.\n\nSoft spots, in proportion. The load-bearing one is Equation (5). With normalized scores in [0,1] and weights summing to 1, Stotal = (omega_obj * Stilde_obj + omega_subj * Stilde_subj) * K_scale with K_scale = 100/9 has a maximum of about 11.1, not 100. The paper then reports percentages, tier thresholds at 75 and 70, and rankings on a 0–100 scale, with no alternate normalization stated. That is not a nitpick: it means every numeric result in Section 4 is unsupported unless the reader assumes a typo and K_scale should be 100. The manuscript needs to state which.\n\nNext, the item counts do not line up: the main text says 389 subjective items, Appendix A says |Q_subj| = 341, and the IAA study in Appendix B scores only 48. These might be different subsets, but the paper does not explain. The IAA panel also includes two authors as raters, which weakens the impartiality claim even though the reported agreement numbers are high.\n\nThe contamination-resistance argument is secondary. LoRA fine-tuning three open models on the full dataset and seeing flat curves is suggestive, but it is not proof that proprietary models cannot memorize. The Limitations section honestly concedes that contamination cannot be ruled out; the conclusion in C.3 overstates what the experiment shows.\n\nWho this is for: anyone building or using Web3-specific LLM evaluation, and researchers studying benchmark contamination. It deserves a serious referee—the resource is valuable and the problems are fixable—but the version I read should not be accepted until the scoring scale is corrected and the numbers are re-run.","headline":"Useful Web3 benchmark with a real community footprint, but the scoring equation cannot produce the reported percentages, so the headline numbers need a re-run before this is trustworthy.","tokens_in":19375,"tokens_out":3266,"would_cite":false,"duration_ms":30950,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The DMind Benchmark tests 31 LLMs across nine Web3 domains and finds a consistent failure profile: strong fundamentals, weak token economics and security.","keywords":["DMind Benchmark","Web3 evaluation","LLM benchmarking","smart contract security","token economics","DeFi","data contamination","cost-effectiveness analysis"],"falsifier":"Train any capable open model on the exact 3,543 DMind items until training loss is near zero, then test it on a held-out set of newly written items covering the same nine domains; if performance on seen items jumps by more than the observed +0.82 to +0.91 points, or if gains do not transfer to the held-out set, the paper's claim that the benchmark resists memorization and demands genuine reasoning would be called into question.","tokens_in":18275,"feed_emoji":"⛓️","tokens_out":10453,"duration_ms":88197,"temperature":0.7,"pith_summary":"The paper sets out to establish DMind Benchmark as a shared yardstick for how well large language models understand the Web3 stack—cryptographic fundamentals, infrastructure, smart contracts, DeFi, DAOs, NFTs, token economics, meme culture, and security—and to show that current models are not equally reliable across those areas. It evaluates 31 proprietary and open-weight models on 3,543 curated items, combining factual multiple-choice questions with open-ended tasks that mimic real operations such as auditing a vulnerable smart contract and calculating liquidation prices. The results consistently show strong recall on fundamentals and infrastructure but serious weaknesses in token economics, meme concepts, and security auditing. If the benchmark is valid, it matters because LLMs are already used for smart contract auditing and DeFi analytics, where reasoning failures translate directly into financial loss; the benchmark would give model developers a concrete list of which gaps to close and give deployers a cost-effectiveness map for choosing models.","feed_headline":"Web3 benchmark: 31 LLMs strongest on basics, weakest on security","feed_subtitle":"A 3,543-item suite across nine Web3 domains maps where models can be trusted and where they cannot.","key_machinery":"The central object is the DMind Benchmark itself: a curated set of 3,543 evaluation items (3,154 objective, 389 subjective) spanning nine Web3 subdomains. Objective items use multiple-choice scoring with partial credit for multi-select questions; subjective items are scored by a fixed LLM judge against rubricized criteria, with keyword fallback. The suite's argument is carried by three validation mechanisms: normalized total scores that combine objective and subjective sections, cross-judge and inter-annotator agreement statistics that show rankings are not judge artifacts, and a contamination probe in which fine-tuning models on the complete dataset produces gains below one point. This last mechanism is what supports the paper's claim that the benchmark measures understanding rather than memorization. A cost-effectiveness analysis using vendor prices adds a Pareto frontier for deployment choices.","core_discovery":"The paper claims that DMind Benchmark is a valid, contamination-resistant measurement of Web3 competence in LLMs, and that evaluating 31 leading models with it reveals a stable capability profile: strong recall of blockchain fundamentals and infrastructure, moderate performance on smart contracts, DeFi, DAOs, and NFTs, and the largest deficits in token economics, meme concepts, and security auditing. The paper further claims that this profile is not an artifact of the scoring setup, because ten different LLM judges agree closely with the primary judge, and because fine-tuning three open models on the entire benchmark for several epochs produces near-flat learning curves. On the paper's own terms, this means that a model scoring well on DMind has to reason about Web3 material rather than pattern-match memorized answers, and that the benchmark can serve as a diagnostic tool for targeted data curation and a shared standard for tracking progress.","pith_inferences":["Editorial inference: the same nine-domain design could be re-run with time-stamped item splits, which would separate knowledge that has gone stale from genuine reasoning deficits; the meme-concept and token-economics gaps could narrow as newer training data arrives.","Editorial inference: the subjective scoring protocol could be extended to agentic settings where models can call tools or interact with testnets; the paper's static setup may underestimate what tool-augmented models can do, and the same rubric could test that.","Editorial inference: the contamination experiment's logic could be extended to proprietary models by testing them on paraphrased versions of benchmark items; that would either strengthen or qualify the paper's claim that high scores require genuine reasoning.","Editorial inference: the weakest column, security auditing, suggests that production deployment of LLM auditors should be gated by demonstrated performance on this kind of task rather than by general reasoning ability."],"forward_implications":["A reproducible Web3 evaluation standard now exists, so future models can be compared against the same fixed protocol and tracked over time.","The consistent weak columns—token economics, meme concepts, and security auditing—identify specific targets for data curation and safety-oriented post-training.","The Pareto cost frontier means deployment choices can be guided by accuracy per dollar, with cheaper open models occupying a sensible mid-range operating point.","The high cross-judge agreement implies that rankings on the subjective tasks are not a quirk of any single scoring model.","The near-flat fine-tuning curves imply that open release of the dataset does not automatically let models game the benchmark by memorization, supporting continued public benchmarking."],"supporting_citations":[{"why":"General-purpose MMLU benchmark used as the contrast case DMind is designed to go beyond.","marker":"(Hendrycks et al., 2021)"},{"why":"BIG-Bench family of broad tasks that motivates the need for domain-specific evaluation.","marker":"(Kazemi et al., 2025)"},{"why":"HELM's holistic evaluation framework supplies the multi-dimension evaluation idea DMind adapts.","marker":"(Liang et al., 2023)"},{"why":"FinBen, a domain-specific financial benchmark, is the precedent for a bespoke evaluation suite.","marker":"(Chen et al., 2024)"},{"why":"Defines smart contracts, the foundational object probed in the smart-contract and security subdomains.","marker":"(Szabo, 1997)"},{"why":"Defines the blockchain ledger and consensus that anchor the fundamentals subdomain.","marker":"(Nakamoto, 2008)"},{"why":"Defines Ethereum and programmable decentralized applications covered by the benchmark's infrastructure and smart-contract items.","marker":"(Buterin, 2014)"},{"why":"Introduces the Web3 concept and decentralized application model that the benchmark's nine subdomains operationalize.","marker":"(Wood, 2014)"}],"fun_headline_variants":["31 LLMs weakest at Web3 security auditing, benchmark shows","DMind benchmark: LLMs ace basics, fail security audits","Web3 LLM benchmark: 31 models, 9 domains, gap at security","LLM Web3 skills: strong on infrastructure, weak on audits","DMind: where LLMs stumble in Web3—security auditing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that fine-tuning three open-weight models on the entire benchmark is a valid memorization test, so that the near-flat learning curves prove high scores require real reasoning; the paper itself concedes that contamination cannot be definitively ruled out for proprietary models.","fun_headline_variants_meta":{"raw":{"variants":["31 LLMs weakest at Web3 security auditing, benchmark shows","DMind benchmark: LLMs ace basics, fail security audits","Web3 LLM benchmark: 31 models, 9 domains, gap at security","LLM Web3 skills: strong on infrastructure, weak on audits","DMind: where LLMs stumble in Web3—security auditing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1724,"prompt_tokens":1013,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":618}},"tokens_in":629,"tokens_out":711,"duration_ms":6534,"temperature":1.0,"reasoning_tokens":618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:00:10.922208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train any capable open model on the exact 3,543 DMind items until training loss is near zero, then test it on a held-out set of newly written items covering the same nine domains; if performance on seen items jumps by more than the observed +0.82 to +0.91 points, or if gains do not transfer to the held-out set, the paper's claim that the benchmark resists memorization and demands genuine reasoning would be called into question.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FinBen, a domain-specific financial benchmark, is the precedent for a bespoke evaluation suite."}],"review_version":1}