Pith. sign in

REVIEW 3 major objections 5 minor 73 references

DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The DMind Benchmark tests 31 LLMs across nine Web3 domains and finds a consistent failure profile: strong fundamentals, weak token economics and security.

desk verdict Useful Web3 benchmark with a real community footprint, but the scoring equation cannot produce the reported percentages, so the headline numbers need a re-run before this is trustworthy. read the letter →

arxiv 2504.16116 v4 pith:3RKZ2ZAB submitted 2025-04-18 cs.CR cs.AI

Enhao Huang , Pengyu Sun , Shuxun Wang , Zixin Lin , Alex Chen , Kaichun Hu , Joey Ouyang , Frank Li
show 8 more authors
Zhiyu Zhang Haobo Wang Yiming Li Zhan Qin James Yi Gang Zhao Ziang Ling Lowes Yang
This is my paper · ORCID
classification cs.CRcs.AI
keywords DMindBenchmarkWeb3evaluationLLMbenchmarkingsmartcontractsecuritytokeneconomicsDeFidatacontaminationcost-effectivenessanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish DMind Benchmark as a shared yardstick for how well large language models understand the Web3 stack—cryptographic fundamentals, infrastructure, smart contracts, DeFi, DAOs, NFTs, token economics, meme culture, and security—and to show that current models are not equally reliable across those areas. It evaluates 31 proprietary and open-weight models on 3,543 curated items, combining factual multiple-choice questions with open-ended tasks that mimic real operations such as auditing a vulnerable smart contract and calculating liquidation prices. The results consistently show strong recall on fundamentals and infrastructure but serious weaknesses in token economics, meme concepts, and security auditing. If the benchmark is valid, it matters because LLMs are already used for smart contract auditing and DeFi analytics, where reasoning failures translate directly into financial loss; the benchmark would give model developers a concrete list of which gaps to close and give deployers a cost-effectiveness map for choosing models.

What carries the argument

The central object is the DMind Benchmark itself: a curated set of 3,543 evaluation items (3,154 objective, 389 subjective) spanning nine Web3 subdomains. Objective items use multiple-choice scoring with partial credit for multi-select questions; subjective items are scored by a fixed LLM judge against rubricized criteria, with keyword fallback. The suite's argument is carried by three validation mechanisms: normalized total scores that combine objective and subjective sections, cross-judge and inter-annotator agreement statistics that show rankings are not judge artifacts, and a contamination probe in which fine-tuning models on the complete dataset produces gains below one point. This last mechanism is what supports the paper's claim that the benchmark measures understanding rather than memorization. A cost-effectiveness analysis using vendor prices adds a Pareto frontier for deployment choices.

What would settle it

Train any capable open model on the exact 3,543 DMind items until training loss is near zero, then test it on a held-out set of newly written items covering the same nine domains; if performance on seen items jumps by more than the observed +0.82 to +0.91 points, or if gains do not transfer to the held-out set, the paper's claim that the benchmark resists memorization and demands genuine reasoning would be called into question.

Watch

Extended reading notes

Core claim

The paper claims that DMind Benchmark is a valid, contamination-resistant measurement of Web3 competence in LLMs, and that evaluating 31 leading models with it reveals a stable capability profile: strong recall of blockchain fundamentals and infrastructure, moderate performance on smart contracts, DeFi, DAOs, and NFTs, and the largest deficits in token economics, meme concepts, and security auditing. The paper further claims that this profile is not an artifact of the scoring setup, because ten different LLM judges agree closely with the primary judge, and because fine-tuning three open models on the entire benchmark for several epochs produces near-flat learning curves. On the paper's own terms, this means that a model scoring well on DMind has to reason about Web3 material rather than pattern-match memorized answers, and that the benchmark can serve as a diagnostic tool for targeted data curation and a shared standard for tracking progress.

Load-bearing premise

The load-bearing premise is that fine-tuning three open-weight models on the entire benchmark is a valid memorization test, so that the near-flat learning curves prove high scores require real reasoning; the paper itself concedes that contamination cannot be definitively ruled out for proprietary models.

Editorial extensions

If this is right

  • A reproducible Web3 evaluation standard now exists, so future models can be compared against the same fixed protocol and tracked over time.
  • The consistent weak columns—token economics, meme concepts, and security auditing—identify specific targets for data curation and safety-oriented post-training.
  • The Pareto cost frontier means deployment choices can be guided by accuracy per dollar, with cheaper open models occupying a sensible mid-range operating point.
  • The high cross-judge agreement implies that rankings on the subjective tasks are not a quirk of any single scoring model.
  • The near-flat fine-tuning curves imply that open release of the dataset does not automatically let models game the benchmark by memorization, supporting continued public benchmarking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same nine-domain design could be re-run with time-stamped item splits, which would separate knowledge that has gone stale from genuine reasoning deficits; the meme-concept and token-economics gaps could narrow as newer training data arrives.
  • Editorial inference: the subjective scoring protocol could be extended to agentic settings where models can call tools or interact with testnets; the paper's static setup may underestimate what tool-augmented models can do, and the same rubric could test that.
  • Editorial inference: the contamination experiment's logic could be extended to proprietary models by testing them on paraphrased versions of benchmark items; that would either strengthen or qualify the paper's claim that high scores require genuine reasoning.
  • Editorial inference: the weakest column, security auditing, suggests that production deployment of LLM auditors should be gated by demonstrated performance on this kind of task rather than by general reasoning ability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces DMind Benchmark, a Web3-domain evaluation suite of 3,543 items spanning nine subfields, combining 3,154 objective multiple-choice items with 389 subjective tasks such as smart-contract auditing, on-chain numerical reasoning, and strategy analysis. The authors evaluate 31 proprietary and open-weight LLMs with a fixed protocol, report an overall ranking and per-subdomain heatmap, perform a cost-effectiveness/Pareto analysis, and include a fine-tuning-based contamination-resistance study. The central claims are that DMind provides a valid, contamination-resistant measure of Web3 competence and that current LLMs show a consistent capability profile: strong on fundamentals and infrastructure, weaker on token economics, meme concepts, and security. The dataset and evaluation pipeline are released to support reproducibility.

Significance. If the measurement issues are resolved, DMind could be a useful community resource: it targets an under-benchmarked high-stakes domain, ships an open dataset and pipeline, evaluates a broad model set, and provides cost-effectiveness guidance that practitioners can act on. The cross-judge agreement study and the inter-annotator agreement analysis are also valuable methodological efforts. However, the paper's central quantitative contribution is currently undermined by a direct inconsistency in the scoring formula, and the contamination-resistance evidence is weaker than the text claims. The strengths of the resource make the manuscript worth repairing, but the repair must address load-bearing technical points rather than only presentation.

major comments (3)
  1. [Section 3.2, Eq. (5)] Equation (5) defines Stotal = (ω_obj·S̃_obj + ω_subj·S̃_subj)·K_scale with K_scale = 100/9. Since ω_obj + ω_subj = 1 and both normalized scores lie in [0,1], the maximum possible Stotal is 100/9 ≈ 11.11. Yet Figure 3, Figure 4, Table 3, and the tier thresholds in Section 4.2 (≥75, 70–75, <70) are all presented on a 0–100 percentage scale. No alternative normalization is provided anywhere in the manuscript. This means the reported scores, rankings, and thresholds cannot be derived from the stated formula. The constant should almost certainly be K_scale = 100, but as written the manuscript is internally inconsistent. Because every quantitative claim in the paper depends on this formula, this is a load-bearing error that must be corrected in both the text and the released pipeline.
  2. [Section 3.1, Appendix A, Appendix B] The number of subjective items is inconsistent: Section 3.1 states the benchmark comprises 389 subjective items, Appendix A states |Q_subj| = 341, and Appendix B describes a rubric for 'each of the 48 subjective questions.' These numbers affect S_subj,max and therefore the weight ω_subj in Eq. (5) and the final composite scores. If Appendices A and B analyze only subsets, that must be stated explicitly; if the counts are errors, they must be corrected. As written, the reader cannot determine the actual composition of the benchmark or verify the claimed totals.
  3. [Appendix C, Table 3] The contamination-resistance experiment is internally inconsistent and overinterpreted. Section C.1 says all models undergo LoRA fine-tuning for three epochs, but Table 3 reports scores at Epoch 0 through Epoch 4 and describes 'five training iterations.' This discrepancy must be resolved: were there three epochs with five evaluation checkpoints, or five epochs? More substantively, the flat learning curves after fine-tuning on the complete DMind dataset are taken as 'strong evidence' of genuine conceptual understanding, but the experiment does not include a held-out generalization control, does not test whether the models can recall item-specific answers under different phrasing, and the paper's own Limitations section concedes that contamination cannot be definitively ruled out for proprietary models. The claim in C.3 should be tempered to a suggestive negative result rather than a definitive proof of contamination resistance.
minor comments (5)
  1. [Section 4.1] The text says 'Model performance is quantified by accuracy scores (in percentages)' but does not explicitly state that the percentages are the Stotal from Eq. (5); once the scaling issue is fixed, this connection should be stated.
  2. [Section 2.1] The related-work section cites 'the very recent DMind Benchmark (Huang et al., 2025)' where the reference is the paper itself. This self-citation should be removed or replaced with an external comparison, since it does not provide independent context.
  3. [Appendix B.1] The description says the panel consists of 'five mutually-unaware experts' and that none were involved in the original creation, but immediately adds that two of the paper's authors also served as raters. This needs clarification: were the author-raters among the five, and how does that square with the 'mutually-unaware' and 'impartial' description?
  4. [Section 1] There is a typo: 'to the best of out knowledge' should be 'to the best of our knowledge.'
  5. [References] The author list in the main text includes 'Haobo Wang' but a reference entry for the authors' own paper lists 'Hobert Wang'; this should be checked for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DMind is an external measurement instrument and its self-references and validation experiments are not load-bearing reductions.

full rationale

DMind Benchmark is an externally constructed evaluation suite: 3,543 expert-curated items are drawn from a crawled, provenance-tracked 6.1 GB corpus and are scored against fixed rubrics before being used to evaluate 31 models. No reported result is defined in terms of the quantity it is supposed to measure, and no parameter is fitted to the evaluated models and then renamed as a prediction. The only self-citations, such as Section 2.1 describing DMind as "among the first attempts" to fill the Web3 evaluation gap, are introductory and carry no load-bearing argument. The contamination-resistance experiment in Appendix C is an empirical robustness check, not a derivation whose output equals its input: fine-tuning three models on the full dataset and measuring score movement is an external falsification test, even if its design is limited. The paper's Limitations section explicitly concedes that "Contamination cannot be definitively ruled out for proprietary models," which weakens the strength of that empirical claim but does not make it circular. The K_scale=100/9 inconsistency in Eq. (5) is a scoring-scale arithmetic problem that, if unreconciled, would invalidate the reported percentages; however, this is a correctness and reproducibility defect, not a circular reduction, because Eq. (5) is not derived from the reported scores. Overall, the benchmark construction and evaluation chain are self-contained empirical measurements rather than derivationally circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities such as particles, forces, or dimensions; the benchmark is a dataset and evaluation protocol, not a postulated entity under the schema. The main assumptions are about expert ground truth, LLM-as-judge reliability, representativeness of the crawl, and the validity of the fine-tuning contamination probe.

free parameters (3)
  • Objective scoring point values (V_SC_corr=2, V_MC_perf=3, V_MC_part=1) = 2, 3, 1
    Hand-chosen in Equations (2) and (3). These values determine each objective item's contribution to the total score and are not derived from data or an external standard.
  • Total score scaling constant K_scale = 100/9
    Hand-set in Equation (5). With the normalized weighted sum bounded by 1, K_scale caps the total at about 11.11, which contradicts the 0-100 percentage interpretation in Section 4.2.
  • IAA and judge-panel sample sizes = 48 subjective items for IAA; 100 sampled per trial in Appendix A
    The paper validates subjective scoring on only 48 of the stated 389 subjective questions, and Appendix A reports 341. These sampling choices affect how strongly the reliability claims generalize.
assumptions (5)
  • domain assumption Expert-authored items and rubrics are correct ground truth for Web3 competence.
    The benchmark's validity rests on the five expert panelists' judgments in Section 3.1 and the rubrics in Appendix B. No external audit of item correctness is provided.
  • domain assumption LLM judges, including Claude-3.7-sonnet and the ten-model panel, can score subjective answers reliably.
    Subjective scores are assigned by AI judges under Equation (4). Appendix A reports high agreement among judges but does not validate the judges against independent expert ground truth.
  • ad hoc to paper LoRA fine-tuning on the complete DMind dataset is a sufficient test of whether models can memorize the benchmark.
    Appendix C interprets flat learning curves as evidence of contamination resistance. This assumption is introduced specifically for this paper and is not independently established.
  • domain assumption The 39 whitelisted communities and 6.1GB crawl are representative of practitioner-facing Web3 discourse.
    This is used in Section 3.1 to justify coverage across nine subdomains. No quantitative representativeness check is reported.
  • domain assumption Option-order randomization and paraphrasing materially reduce training-data contamination.
    Stated in Section 3.1 as a mitigation strategy. The paper provides no measurement of how much contamination risk remains, and the Limitations concede contamination cannot be ruled out for proprietary models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain." pith.science (2026). https://pith.science/paper/3RKZ2ZAB

@misc{pith2026250416116,
  author       = {Pith},
  title        = {Pith review of: DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RKZ2ZAB}},
  note         = {Machine review of arXiv:2504.16116}
}
read the original abstract

The Web3 ecosystem, underpinned by cryptographic primitives and decentralized consensus, represents a high-stakes environment where software vulnerabilities and incentive misalignments translate directly into financial loss. As Large Language Models (LLMs) are increasingly integrated into this domain for tasks ranging from smart contract auditing to decentralized finance analytics, ensuring their reliability is paramount. However, general-purpose benchmarks fail to capture the specialized reasoning required for these adversarial and protocol-driven settings. To bridge this gap, we introduce DMind Benchmark, a comprehensive evaluation suite designed to rigorously assess LLM proficiency across the Web3 stack. DMind Benchmark encompasses nine distinct subdomains (spanning infrastructure, smart contracts, token economics, etc.) and combines objective knowledge retrieval with complex open-ended reasoning tasks that emulate real-world operational challenges. We conduct an extensive evaluation of 31 leading proprietary and open-weights models, employing a contamination-aware pipeline and verifying the statistical robustness of our scoring protocol through rigorous cross-judge consistency checks. Our analysis reveals a critical dichotomy: while models demonstrate competence in foundational infrastructure concepts, they exhibit significant vulnerabilities in high-reasoning tasks such as security auditing. Furthermore, we provide a Pareto analysis to guide cost-effective deployment and demonstrate through adversarial experiments that high performance on DMind Benchmark necessitates genuine reasoning rather than superficial memorization. Since its open-source release in April 2025, DMind Benchmark achieved the #1 trending position on Hugging Face for nearly a week and accumulated over 13k downloads by June 2026, establishing itself as a standard for advancing secure and trustworthy AI in Web3.

Figures

Figures reproduced from arXiv: 2504.16116 by the authors.

Figure 1
Figure 1. An overview of the interconnected domains within the Web3 ecosystem, highlighting the nine key [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The DMind Benchmark framework, illustrating its nine evaluated Web3 domains, diverse objective and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overall performance of all evaluated LLMs on the DMind Benchmark, sorted by mean score. Colors [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Unified heatmap of model accuracy across nine Web3 subdimensions: Fundamentals (Fund.), Infrastructure [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Overall mean score versus total cost in USD [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 39 canonical work pages

  1. [1]

    Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, and et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  3. [3]

    Bartoletti and L

    M. Bartoletti and L. Pompianu. 2017. An empirical analysis of smart contracts: platforms, applications, and design patterns. In Financial Cryptography and Data Security: FC 2017 International Workshops, Lecture Notes in Computer Science, pages 494--509. Springer International Publishing

    work page 2017

  4. [4]

    Belchior, A

    R. Belchior, A. Vasconcelos, S. Guerreiro, and M. Correia. 2021. A survey on blockchain interoperability: Past, present, and future trends. ACM Computing Surveys, 54(8):1--41

    work page 2021

  5. [5]

    Momtaz, and et al

    Cristiano Bellavitis, Christian Fisch, Paul P. Momtaz, and et al. 2023. The rise of decentralized autonomous organizations (daos): a first empirical glimpse. Venture Capital

    work page 2023

  6. [6]

    V. Buterin. 2014. Ethereum: A next-generation smart contract and decentralized application platform. Ethereum White Paper

    work page 2014

  7. [7]

    Vitalik Buterin and 1 others. 2013. Ethereum white paper. GitHub repository, 1(22-23):5--7

    work page 2013

  8. [8]

    Christian Catalini, Alonso de Gortari, and Nihar Shah. 2022. Some simple economics of stablecoins. Annual Review of Financial Economics, 14(1):117--135

    work page 2022

Show all 73 references
  1. [9]

    Siddhartha Chatterjee and Bina Ramamurthy. 2025. Efficacy of various large language models in generating smart contracts. In Advances in Information and Communication, pages 482--500, Cham. Springer Nature Switzerland

    2025

  2. [10]

    Qianqian Chen, Wen Han, Zhihao Chen, and et al. 2024. Finben: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems, 37:95716--95743

    2024

  3. [11]

    Yan Chen and Cristiano Bellavitis. 2020. Blockchain disruption and decentralized finance: The rise of decentralized business models. Journal of Business Venturing Insights, 13:e00151

    2020

  4. [12]

    K. R. Dearstyne, A. D. Rodriguez, and J. Cleland-Huang. 2024. Supporting software maintenance with dynamically generated document hierarchies. In ICSME, pages 426--437

    2024

  5. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, and et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Yuchuan Fu, Xiaohan Yuan, and Dongxia Wang. 2025. Ras-eval: A comprehensive benchmark for security evaluation of llm agents in real-world environments. arXiv preprint arXiv:2506.15253. Includes 3,802 attack tasks, supports real-world tool execution

    2025 arXiv

  7. [15]

    Dagher, Tim Andersen, and Jun Zhuang

    Caleb Geren, Amanda Board, Gaby G. Dagher, Tim Andersen, and Jun Zhuang. 2025. https://doi.org/10.1145/3715073.3715075 Blockchain for large language model security and safety: A holistic survey . SIGKDD Explor. Newsl., 26(2):1–20

    2025

  8. [16]

    Team GLM, Aohan Zeng, and Bin et al. Xu. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    2024 arXiv

  9. [17]

    Neel Guha, Julian Nyarko, Daniel Ho, and et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems, volume 36, pages 44123--44279. Curran Associates, Inc

    2023

  10. [18]

    Dong Guo and Faming et al. Wu. 2025. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062

    2025 arXiv

  11. [19]

    Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, and et al. 2024. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models. arXiv preprint arXiv:2308.09975

    2024 arXiv

  12. [20]

    Samer Hassan and Primavera De Filippi. 2021. Decentralized autonomous organization. Internet Policy Review, 10(2)

    2021

  13. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  14. [22]

    Enhao Huang, Pengyu Sun, Zixin Lin, Alex Chen, Joey Ouyang, Hobert Wang, Dong Dong, Gang Zhao, James Yi, Frank Li, Ziang Ling, and Lowes Yang. 2025. Dmind benchmark: Toward a holistic assessment of llm capabilities across the web3 domain. arXiv preprint arXiv:2504.16116. Open ...

    2025 arXiv

  15. [23]

    Zheng Hui, Yijiang River Dong, Ehsan Shareghi, and Nigel Collier. 2025. Trident: Benchmarking llm safety in finance, medicine, and law. arXiv preprint arXiv:2507.21134. Domain-specific safety benchmark across multiple regulated fields

    2025 arXiv

  16. [24]

    Md Rafiqul Islam, Muhammad Mahbubur Rahman, Md Mahmud, Mohammed Ataur Rahman, Muslim Har Sani Mohamad, and Abd Halim Embong. 2021. A review on blockchain security issues and challenges. In 2021 IEEE 12th control and system graduate research colloquium (ICSGRC), pages 227--232. IEEE

    2021

  17. [25]

    Kensuke Ito. 2024. Cryptoeconomics and tokenomics as economics: A survey with opinions. arXiv preprint arXiv:2407.15715

    2024 arXiv

  18. [26]

    Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, and et al. 2025. Big-bench extra hard. arXiv preprint arXiv:2502.19187

    2025 arXiv

  19. [27]

    Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. Medexqa: Medical question answering benchmark with multiple explanations. arXiv preprint arXiv:2406.06331

    2024 arXiv

  20. [28]

    David Krause. 2024. Beyond the hype: A meme coin reality check for retail investors. Available at SSRN 4891841

    2024

  21. [29]

    Ennan Lai and Wenjun Luo. 2020. Static analysis of integer overflow of smart contracts in ethereum. In ICCSP

    2020

  22. [30]

    Aonian Li and Bangwei et al. Gong. 2025. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313

    2025 arXiv

  23. [31]

    Y. Li, B. Luo, and Q. et al. Wang. 2024. A reflective llm-based agent to guide zero-shot cryptocurrency trading. arXiv preprint arXiv:2407.09546

    2024 arXiv

  24. [32]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, and et al. 2023. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2023 arXiv

  25. [33]

    Chao Liu, Han Liu, Zhao Cao, Zhong Chen, Bangdao Chen, and Bill Roscoe. 2018. Reguard: finding reentrancy bugs in smart contracts. In ICSE

    2018

  26. [34]

    Zefang Liu1, Jialei Shi1, and John F. Buford1. 2024. Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity. In AICS. Curran Associates, Inc

    2024

  27. [35]

    Hou-Wan Long, Hongyang Li, and Wei Cai. 2024. Coinclip: A multimodal framework for evaluating the viability of memecoins in the web3 ecosystem. arXiv preprint arXiv:2412.07591

    2024 arXiv

  28. [36]

    Looijenga

    M.S. Looijenga . 2024. http://essay.utwente.nl/104811/ Rechtbert : Training a dutch legal bert model to enhance legaltech

    2024

  29. [37]

    H. Luo, J. Luo, and A. V. Vasilakos. 2024. Bc4llm: A perspective of trusted artificial intelligence when blockchain meets large language models. Neurocomputing, 599:128089

    2024

  30. [38]

    Mistral AI . 2025. https://mistral.ai/news/mistral-medium-3 Mistral medium 3

    2025

  31. [39]

    Mothukuri, R

    V. Mothukuri, R. M. Parizi, and J. L. et al. Massa. 2024. An ai multi-model approach to defi project trust scoring and security. In Blockchain, pages 19--28

    2024

  32. [40]

    Matthieu Nadini, Laura Alessandretti, Flavio Di Giacinto, Mauro Martino, Luca Maria Aiello, and Andrea Baronchelli. 2021. Mapping the nft revolution: market trends, trade networks, and visual features. Scientific reports, 11(1):20902

    2021

  33. [41]

    Satoshi Nakamoto. 2008. Bitcoin: A peer-to-peer electronic cash system

    2008

  34. [42]

    D. Nam, A. Macvean, and V. et al. Hellendoorn. 2024. Using an llm to help with code understanding. In ICSE, pages 1--13

    2024

  35. [43]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474

    2022 arXiv

  36. [44]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, and Sam Altman et al. 2024. Gpt-4 technical report

    2024

  37. [45]

    Peterson K. Ozili. 2022. Decentralized finance research and developments around the world. Journal of Banking and Financial Technology

    2022

  38. [46]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning. PMLR

    2022

  39. [47]

    Ruhul Saha, Gaurav Kumar, Mauro Conti, and Sujata Pal. 2021. Dhacs: Smart contract-based decentralized hybrid access control for industrial internet-of-things. IEEE Transactions on Industrial Informatics, 18(5):3452--3461

    2021

  40. [48]

    Karan Singhal, Shekoofeh Azizi, Tu, and et al. 2023. https://doi.org/10.1038/s41586-023-06291-2 Large language models encode clinical knowledge . Nature, 620(7972):172--180

    2023 doi

  41. [49]

    Sara Mahdavi, Jason Wei, and et al

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, and et al. 2022. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138

    2022 arXiv

  42. [50]

    M. Suri, P. Mathur, and F. et al. Dernoncourt. 2024. Docedit-v2: Document structure editing via multimodal llm grounding. In EMNLP, pages 15485--15505

    2024

  43. [51]

    N. Szabo. 1997. Formalizing and securing relationships on public networks. First Monday, 2(9)

    1997

  44. [52]

    G. Team, R. Anil, S. Borgeaud, and et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  45. [53]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, and et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  46. [54]

    Kimi Team and Yifan et al. Bai. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534

    2025 arXiv

  47. [56]

    Hugo et al. Touvron. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  48. [57]

    Toyoda, X

    K. Toyoda, X. Wang, and M. et al. Li. 2024. Blockchain data analysis in the era of large-language models. arXiv preprint arXiv:2412.09640

    2024 arXiv

  49. [58]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. https://arxiv.org/abs/1905.00537 Superglue: A stickier benchmark for general-purpose language understanding systems . Preprint, arXiv:1905.00537

    2020 arXiv

  50. [59]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 a . https://arxiv.org/abs/1804.07461 Glue: A multi-task benchmark and analysis platform for natural language understanding . Preprint, arXiv:1804.07461

    2019 arXiv

  51. [60]

    Qin Wang, Rujia Li, Qi Wang, and Shiping Chen. 2021. Non-fungible token (nft): Overview, evaluation, opportunities and challenges. arXiv preprint arXiv:2105.07447

    2021 arXiv

  52. [61]

    Shuai Wang, Wenwen Ding, Juanjuan Li, Yong Yuan, Liwei Ouyang, and Fei-Yue Wang. 2019 b . Decentralized autonomous organizations: Concept, model, and applications. IEEE Transactions on Computational Social Systems, 6(5):870--878

    2019

  53. [62]

    Sam Werner, Daniel Perez, Lewis Gudgeon, Ariah Klages-Mundt, Dominik Harz, and William Knottenbelt. 2022. Sok: Decentralized finance (defi). In Proceedings of the 4th ACM Conference on Advances in Financial Technologies, pages 30--46

    2022

  54. [63]

    Gavin Wood. 2014. DÆps: What web 3.0 looks like. http://gavwood.com/dappsweb3.html

    2014

  55. [64]

    Gavin Wood and 1 others. 2014. Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper, 151(2014):1--32

    2014

  56. [65]

    C. Wu, J. Chen, and Z. et al. Wang. 2024. Semantic sleuth: Identifying ponzi contracts via large language models. In ASE, pages 582--593

    2024

  57. [66]

    Shijie Wu, Ozan Irsoy, Lu, and et al. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564

    2023 arXiv

  58. [67]

    xAI . 2025. https://grok.online/zh/about About grok

    2025

  59. [68]

    Dylan Yaga, Peter Mell, Nik Roby, and Karen Scarfone. 2019. Blockchain technology overview. arXiv preprint arXiv:1906.11078

    2019 arXiv

  60. [69]

    Yli-Huumo, D

    J. Yli-Huumo, D. Ko, S. Choi, S. Park, and K. Smolander. 2016. Where is current research on blockchain technology?—a systematic review. PloS one, 11(10):e0163477

    2016

  61. [70]

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, and et al. 2024. Safetybench: Evaluating the safety of large language models. arXiv preprint arXiv:2309.07045

    2024 arXiv

  62. [71]

    Zhong and Z

    L. Zhong and Z. Wang. 2024. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In AAAI

    2024

  63. [72]

    Q. Zhou, H. Huang, Z. Zheng, and J. Bian. 2020. Solutions to scalability of blockchain: A survey. IEEE Access, 8:16440--16455

    2020

  64. [73]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  65. [74]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.