Pith. sign in

REVIEW 5 major objections 4 minor 23 references

Multi-source Multi-level Multi-token Ethereum Dataset and Benchmark Platform

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces 3MEthTaskforce, a publicly released Ethereum dataset and benchmark platform that combines 303 million token-transaction records, 3,880 token profiles, global market indicators, and a decade of Reddit sentiment…

desk verdict Useful dataset resource, but the LLM sentiment features almost certainly leak future information, so the benchmark results need a causal redo. read the letter →

arxiv 2501.11906 v1 pith:R5F7VPCI submitted 2025-01-21 cs.CE

Haoyuan Li , Mengxiao Zhang , Maoyuan Li , Jianzheng Li , Junyi Yang , Shuangyan Deng , Zijian Zhang , Jiamou Liu This is my paper · ORCID
classification cs.CE
keywords Ethereumblockchaindatasetuserbehaviorpredictiontokenpricetemporalgraphneuralnetworkstime-seriesforecastingmarketsentimentdecentralizedfinance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces 3MEthTaskforce, a publicly released Ethereum dataset and benchmark platform that combines 303 million token-transaction records from 3,880 tokens, 3,880 token profiles, global market indicators, and ten years of Reddit sentiment. The authors aim to give researchers a single integrated resource for studying how user behavior, market sentiment, and token prices interact. To show the dataset is usable, they define two tasks—user behavior prediction and token price prediction—and benchmark six dynamic graph neural networks and nineteen time-series models on them. A sympathetic reading takes the paper's contribution to be the dataset and its baseline results, not a new learning method. If the resource is as useful as claimed, it would lower the barrier to multimodal blockchain analytics and decentralized-finance risk research.

What carries the argument

The central object is the integrated multi-level dataset itself: transaction-level records (sender, receiver, token, value, timestamp), token-level metadata (price, market capitalization, 24-hour volume), global-level market indices, and textual-level sentiment indices computed from Reddit posts. The benchmark's second key mechanism is the task formulation: user behavior prediction is cast as edge prediction on a temporal bipartite graph of wallets and tokens, while price prediction is cast as univariate and multivariate time-series forecasting with feature sets that progressively add global and sentiment signals. Sentiment indices are produced by prompting a large language model to score posts, with one version giving a single 0–10 score and another giving separate positive and negative scores.

What would settle it

Compare the distribution of transaction volumes, token ages, and wallet activity in the evaluation subsets against the same statistics computed on the full 303-million-transaction, 3,880-token dataset; a large divergence would mean the benchmark results cannot be assumed to hold for the whole platform.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that 3MEthTaskforce is the first multi-source, multi-level, multi-token Ethereum dataset and benchmark platform. It integrates over 300 million transaction records, 3,880 token profiles, global market indicators such as Bitcoin dominance and total market capitalization, and Reddit-derived sentiment scores spanning 2014–2024. The platform defines two core tasks: predicting whether a user will buy or sell a token by treating transactions as temporal edges in a bipartite graph, and predicting token prices from price series augmented with global indices and sentiment. The benchmark results show that LLM-derived positive/negative sentiment scores improve user behavior prediction over a single overall sentiment score, and that adding global and sentiment features improves multivariate price forecasting, with iTransformer ranking best among the nineteen time-series models and JODIE and DyGFormer leading the dynamic GNNs.

Load-bearing premise

The benchmark conclusions assume that the roughly 260,000-transaction, 29,164-wallet subset used for user behavior experiments and the 211 long-history tokens used for multivariate price prediction are representative of the full 303-million-transaction, 3,880-token dataset.

Editorial extensions

If this is right

  • Researchers in blockchain analytics get a single multimodal dataset that links transaction graphs, token fundamentals, market-level indicators, and sentiment, instead of stitching separate sources together.
  • The published benchmark numbers give a baseline map of current model performance: dynamic GNNs reach above 0.94 average precision on behavior prediction, and adding sentiment and global features consistently improves multivariate price forecasts.
  • The two defined tasks—user behavior prediction and token price prediction—provide standardized evaluation protocols for future work on Ethereum data.
  • The platform's design supports downstream risk analysis and market-fluctuation modeling, including applications such as detecting abnormal trading behavior or modeling panic events like the UST/LUNA collapse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own experiments run on a 260,000-transaction slice and a 211-token subset; if those slices are not representative of the full 303-million-transaction, 3,880-token corpus, the benchmark rankings may not carry over to the whole platform.
  • Because the release provides only LLM-generated sentiment scores rather than raw Reddit text, users cannot build their own sentiment models or inspect the original posts, which limits the asset's reusability for NLP research.
  • The sentiment signal comes from Reddit subreddits, so it likely captures a particular community's mood; extending the data with other social feeds could change the feature's predictive value.
  • The dataset covers a period with major market regime changes, so it could be used to test whether models that win on average still fail at turning points such as crashes, a question the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces 3MEthTaskforce, a publicly released Ethereum dataset and benchmark platform integrating over 300 million transaction records, 3,880 token profiles, global market indicators, and Reddit sentiment data from 2014 to 2024. It defines two tasks—user behavior prediction and token price prediction—and reports benchmark results for six dynamic graph neural networks and nineteen time-series forecasting models. The central claims are that the multi-source, multi-level, multi-token design enables integrated studies of user behavior, sentiment, and token performance, and that the provided benchmarks map current model performance on these tasks.

Significance. If the technical concerns are resolved, this is a potentially valuable community resource: it is one of the few datasets to combine transaction-level Ethereum data with token metadata, global market indices, and long-horizon Reddit sentiment, and it ships with FAIR-aligned documentation and a benchmark suite covering a broad set of GNN and time-series baselines. The scale (303M transactions, 3,880 tokens) and the explicit task definitions are strengths, and the authors are to be credited for releasing data and code. However, the benchmark conclusions are currently undermined by a likely lookahead bias in the LLM-generated sentiment features, by the unexamined representativeness of the evaluation subsets, and by missing metric definitions and error reporting.

major comments (5)
  1. [Prompt (Appendix) and feature-ablation results in Tables 2–4] The sentiment scores spos_t and sneg_t are generated by DeepSeek under Prompt 2, which instructs the model to consider 'the trends and news about cryptocurrencies at the time of the timestamp' and to judge whether the context 'seemed to have a positive or negative impact on cryptocurrencies at the time.' Because DeepSeek's pretraining data includes information posterior to posts from 2013–2024, these scores can encode future price movements. There is no temporal cutoff, masking, or causal constraint in the sentiment generation pipeline. This directly affects the central ablation claims: in Table 2, TCL NAP rises from 0.737 on trans text to 0.786 on trans text llm, and in Table 4 the multivariate results are used to argue that sentiment features improve forecasting. As written, the reported sentiment-feature gains may be inflated by leakage. Please regenerate the sentiment indices with a causal pipeline (e.g., prompting with only text and public knowledge available at the timestamp, or applying a strict knowledge cutoff), or explicitly re-frame the sentiment-ablation results as non-causal upper bounds and add a temporal holdout in which the LLM's training cutoff precedes the test period.
  2. [User Behavior Experiment Setup; Price Prediction Experiment Setup] The user behavior experiments use a subset of about 260,000 transactions from 29,164 wallets and 1,194 tokens, while the multivariate price experiments use 211 tokens with at least six years of history; the full dataset spans 303 million transactions, 35 million wallets, and 3,880 tokens. The paper provides no representativeness analysis comparing these evaluation subsets to the full population along dimensions such as token type, market capitalization, transaction frequency, or time period. Without such evidence, the benchmark conclusions about which models and features work best cannot be generalized to the platform the paper claims to provide. Please add a distribution comparison between the subsets and the full dataset, or explicitly scope the benchmark claims to the subsets.
  3. [Price Prediction Experiment Setup, Performance Metrics] The metrics called Normalized MSE, Normalized MAE, WAPE, and MSMAPE are never defined by equations. The LUNC values in Table 3 (e.g., 7.44e-11 for LinearRegression and 2.86e-8 for ARIMA) are impossible to interpret without knowing the normalization scheme, and the multivariate metrics in Table 4 cannot be checked or reproduced. Define all four metrics precisely, including denominators, aggregation over tokens and time steps, and the exact normalization applied to each token's price series.
  4. [User Behavior Experiment Setup; Price Prediction Experiment Setup; Tables 2–4] The text states that results are averaged over three runs with a confidence level of 0.05 and that errors fall within the confidence interval, but Tables 2–4 report only point estimates. Many comparisons in Table 2 are separated by less than 0.01 in TAP/NAP (e.g., DyGFormer 0.928 on trans versus 0.939 on trans global), so without standard deviations or confidence intervals the reader cannot judge whether the reported differences are meaningful. Please include error bars or intervals for all entries, or state explicitly which differences are statistically significant.
  5. [Feature Impact paragraph and Table 4] The text contains internal contradictions with the reported numbers. It says 'iTransformer's msMAPE on the All dataset (17.03) was notably better than on the Price dataset (16.23)', but lower msMAPE is better, so 16.23 is the better value and the statement is backwards. It also says TCN achieved the lowest msMAPE of 11.19 on the Price dataset, whereas Table 4 shows TCN's Price msMAPE is 30.7; the value 11.19 appears to be the Global wAPE. These inconsistencies directly affect the paper's feature-impact claims and must be corrected.
minor comments (4)
  1. [Appendix, Datasheets for Datasets, Composition] The datasheet asserts that 'the dataset is free from errors, noise, and redundancies.' Given the scale of 303 million transaction records and the multi-source cleaning pipeline, this claim needs support from explicit validation steps, or it should be softened to describe known limitations and any residual noise.
  2. [Data Collection] The paragraph on Ethereum Public ETL states that the free Ethereum API gathered '5,855 active wallet addresses and their transactions' and then reports '267,242 transaction records among 29,164 wallet addresses'; please clarify the relationship between these two numbers, since one would expect the API-gathered addresses to match the final wallet count.
  3. [User Behavior Experiment Setup] The five input vectors are sometimes called 'transaction input vectors' and sometimes 'transaction labels'; please use one consistent term throughout the section to avoid confusion between the prediction target and the input features.
  4. [Price Prediction Experiment Setup, Performance Metrics] The metric names appear as 'Normalized MSE', 'Normalized MAE', 'WAPE', and 'MSMAPE' in the text and as 'wape' and 'msmape' in Table 4; standardize the spelling and capitalization of the metric acronyms.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's benchmarks are empirical evaluations using independently collected data; the LLM sentiment leakage concern is a data-validity issue, not a derivation that reduces to its inputs.

full rationale

The paper's claimed derivation chain is a data-construction and benchmarking pipeline, not a formal derivation. The two benchmark tasks (User Behavior Prediction and Token Price Prediction) feed externally collected transaction, price, global-index, and Reddit-derived sentiment features into standard dynamic-GNN and time-series models; none of the task definitions or result tables equates the predicted output with any input by construction. The sentiment indices are generated by DeepSeek from Reddit text using the quoted prompts and are then used as covariates; they are not fitted parameters calibrated to the benchmark test labels, and no equation in the paper defines the sentiment score as the target price or transaction. The skeptical concern that DeepSeek's pretraining may include information posterior to a Reddit post's timestamp is a real data-leakage and external-validity threat to the 'sentiment helps' benchmark conclusion, but it is not circularity: leakage would make the feature spuriously informative, not make the prediction identical to the input by definition. The paper also contains no load-bearing self-citation chain or imported uniqueness theorem; the GNN and time-series baselines are standard external methods, and the dataset is independently released. Accordingly, no circular step can be exhibited under the required 'quote and reduce' standard, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central contribution is a curated dataset, so the main dependencies are data quality assumptions and the validity of the sentiment annotation. There are no new physical or model entities; the only hand-chosen free parameter is the sentiment decay factor.

free parameters (1)
  • sentiment decay factor k = 0.5
    Hand-chosen in Algorithm 1 to spread each post's sentiment, comment count, and scores across the following six days with exponential decay; it directly shapes all sentiment features used in both benchmark tasks.
assumptions (4)
  • domain assumption Each wallet address represents a unique user.
    Explicitly stated in the Tasks section. In practice users control many addresses and exchanges aggregate many users, so the user behavior task actually models wallet behavior.
  • ad hoc to paper The 260,000-transaction subset and the 211-token subset are representative of the full 303-million-transaction, 3,880-token dataset.
    The benchmarks use only these subsets with no representativeness analysis, yet the conclusions are presented as describing the platform as a whole.
  • domain assumption LLM-generated sentiment scores are valid, unbiased measures of market sentiment.
    Sentiment is generated by DeepSeek from prompts referencing contemporaneous trends and news; no human validation or temporal leakage check is provided.
  • domain assumption Ethereum Public ETL, DefiLlama, and PRAW data are accurate and complete for the covered tokens and time range.
    The paper relies on these third-party tools and asserts the resulting dataset is free of errors without external audit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-source Multi-level Multi-token Ethereum Dataset and Benchmark Platform." pith.science (2026). https://pith.science/paper/R5F7VPCI

@misc{pith2026250111906,
  author       = {Pith},
  title        = {Pith review of: Multi-source Multi-level Multi-token Ethereum Dataset and Benchmark Platform},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5F7VPCI}},
  note         = {Machine review of arXiv:2501.11906}
}
read the original abstract

This paper introduces 3MEthTaskforce (https://3meth.github.io), a multi-source, multi-level, and multi-token Ethereum dataset addressing the limitations of single-source datasets. Integrating over 300 million transaction records, 3,880 token profiles, global market indicators, and Reddit sentiment data from 2014-2024, it enables comprehensive studies on user behavior, market sentiment, and token performance. 3MEthTaskforce defines benchmarks for user behavior prediction and token price prediction tasks, using 6 dynamic graph networks and 19 time-series models to evaluate performance. Its multimodal design supports risk analysis and market fluctuation modeling, providing a valuable resource for advancing blockchain analytics and decentralized finance research.

Figures

Figures reproduced from arXiv: 2501.11906 by the authors.

Figure 1
Figure 1. 3MEthTaskforce Data Pipeline. The third column illustrates datasets: Transaction Records (blue), Token Information (green), Global Market Indices (red), and Textual Indices (yellow). User Behavior Prediction and Token Price Prediction. • Offering a comprehensive set of benchmarks for the pro￾posed tasks, validated through systematic experimental results. Related Work Cryptocurrency-related data repositories. Most ex… view at source ↗
Figure 2
Figure 2. Time-series analysis of cryptocurrency market activity and online discourse from data in this dataset. (a) displays the aggregate trading volume, indicating periods of high market activity. (b) shows the market capitalization breakdown by cryptocurrency category, reveal￾ing the relative dominance of BTC over time. (c) illustrates the temporal dynamics of submissions across various cryptocurrency subreddits. (d) pres… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 13 canonical work pages

  1. [1]

    Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J

    arXiv preprint arXiv:2310.03589. Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J. W.; Wallach, H.; Iii, H. D.; and Crawford, K. 2021. Datasheets for datasets. Communications of the ACM, 64(12): 86–92. Hamayel, M. J.; and Owda, A. Y . 2021. A novel cryp- tocurrency price prediction model using GRU, LSTM and bi-LSTM machine learning algorithms. Ai, 2(...

    arXiv 2021

  2. [2]

    Additionally, if your study involves hypotheses testing... (a) Did you clearly state the assumptions underlying all theoretical results? NA (b) Have you provided justifications for all theoretical re- sults? NA (c) Did you discuss competing hypotheses or theories that might challenge or complement your theoretical re- sults? NA (d) Have you considered alt...

  3. [3]

    (a) Did you state the full set of assumptions of all theoret- ical results? NA (b) Did you include complete proofs of all theoretical re- sults? NA

    Additionally, if you are including theoretical proofs... (a) Did you state the full set of assumptions of all theoret- ical results? NA (b) Did you include complete proofs of all theoretical re- sults? NA

  4. [4]

    Additionally, if you ran machine learning experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (ei- ther in the supplemental material or as a URL)? Yes (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? Yes (c) Did you report error bars (...

    work page

  5. [5]

    Advances in Neural Information Processing Systems , 35: 32928–32941

    Towards better evaluation for dynamic link predic- tion. Advances in Neural Information Processing Systems , 35: 32928–32941. Qi, Y .; Wu, J.; Xu, H.; and Guizani, M. 2023. Blockchain Data Mining With Graph Learning: A Survey. IEEE Trans- actions on Pattern Analysis and Machine Intelligence. Qiu, X.; Hu, J.; Zhou, L.; Wu, X.; Du, J.; Zhang, B.; Guo, C.; Z...

    arXiv 2023

  6. [6]

    Additionally, if you used crowdsourcing or conducted research with human subjects, without compromising anonymity... (a) Did you include the full text of instructions given to participants and screenshots? NA (b) Did you describe any potential participant risks, with mentions of Institutional Review Board (IRB) ap- provals? NA (c) Did you include the esti...

    work page

  7. [7]

    arXiv preprint arXiv:2210.02186

    Timesnet: Temporal 2d-variation modeling for gen- eral time series analysis. arXiv preprint arXiv:2210.02186. Xu, D.; Ruan, C.; Korpeoglu, E.; Kumar, S.; and Achan, K

    arXiv

  8. [8]

    arXiv preprint arXiv:2002.07962

    Inductive representation learning on temporal graphs. arXiv preprint arXiv:2002.07962. Xu, Z.; Zeng, A.; and Xu, Q. 2023. FITS: Modeling time se- ries with 10k parameters. arXiv preprint arXiv:2307.03756. Yenido˘gan, I.; C ¸ ayir, A.; Kozan, O.; Da˘g, T.; and Arslan, C ¸

    arXiv 2002

Show all 23 references
  1. [10]

    In Proceedings of the 2019 4th International Conference on Machine Learn- ing Technologies, 15–19

    Prediction of cryptocurrency price dynamics with multiple machine learning techniques. In Proceedings of the 2019 4th International Conference on Machine Learn- ing Technologies, 15–19. Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; and Zhang, W. 2021. Informer: ...

    2019

  2. [11]

    For most authors... (a) Would answering this research question advance sci- ence without violating social contracts, such as violat- ing privacy norms, perpetuating unfair profiling, exac- erbating the socio-economic divide, or implying disre- spect to societies or cultures? Y...

  3. [15]

    Additionally, if you are using existing assets (e.g., code, data, models) or curating/releasing new assets, without compromising anonymity... (a) If your work uses existing assets, did you cite the cre- ators? Yes (b) Did you mention the license of the assets? Yes (c) Did you ...

    2021

  4. [17]

    Motivation What is the purpose of creating the dataset? Is it de- signed for specific tasks? The dataset was created to sup- port in-depth analysis of the cryptocurrency market, partic- ularly for tasks such as user behavior prediction and token price prediction. By integratin...

  5. [18]

    How many instances are there in total? The dataset con- tains 303 million transaction records, covering 35 million users and 3,880 tokens

    Composition What do the dataset instances represent (e.g., doc- uments, images)? The dataset instances represent blockchain transaction records, token information, market indices, and textual sentiment data from Reddit. How many instances are there in total? The dataset con- t...

  6. [19]

    What mechanisms or procedures were used for data collection? The data was primarily obtained using blockchain extraction tools, transaction APIs, and sentiment analysis APIs

    Collection Process How was the data for each instance acquired (e.g., di- rect observation, subject reports)? The data was col- lected using various open-source tools and platforms, in- cluding the Ethereum Public ETL tool, DefiLlama, and Red- dit API (PRAW). What mechanisms o...

    2014

  7. [20]

    Was the “raw” data retained? Yes, the raw data was re- tained for subsequent verification and further analysis

    Preprocessing/Cleaning/Annotation Was any preprocessing, cleaning, or annotation per- formed on the data? Yes, the data was cleaned and an- notated, including the removal of irrelevant content and the extraction of sentiment indices. Was the “raw” data retained? Yes, the raw d...

  8. [21]

    Use Cases For which tasks has the dataset been used? The dataset has been used for tasks such as user behavior prediction, token price prediction, and user behavior marking. How might the dataset’s composition or collection pro- cess affect its future uses? The multi-source an...

  9. [22]

    How will the dataset be distributed (e.g., tarball on a website, API)? The dataset is provided in CSV format through the Figshare platform, with a permanent DOI link

    Distribution Will the dataset be distributed to third parties? Yes, the dataset has been made publicly available. How will the dataset be distributed (e.g., tarball on a website, API)? The dataset is provided in CSV format through the Figshare platform, with a permanent DOI li...

  10. [23]

    Will the dataset be updated? Yes, updates and exten- sions may be made in the future as needed

    Maintenance Who is responsible for maintaining the dataset? The dataset is maintained by the creation team. Will the dataset be updated? Yes, updates and exten- sions may be made in the future as needed. Is there a mechanism for others to contribute to or extend the dataset? C...

    2019

  11. [2018]

    In 2018 3rd international conference on computer science and engineering (UBMK), 621–624

    Bitcoin forecasting using ARIMA and PROPHET. In 2018 3rd international conference on computer science and engineering (UBMK), 621–624. IEEE. Yu, L.; Sun, L.; Du, B.; and Lv, W. 2023. Towards bet- ter dynamic graph learning: New architecture and unified li- brary. Advances in N...

    2018

  12. [2019]

    In 2019 International Seminar on Application for Technology of Information and Communication (iSemantic), 260–265

    Short term prediction on bitcoin price using ARIMA method. In 2019 International Seminar on Application for Technology of Information and Communication (iSemantic), 260–265. IEEE. Wu, H.; Hu, T.; Liu, Y .; Zhou, H.; Wang, J.; and Long, M

    2019

  13. [2020]

    IEEE Transactions on Emerging Topics in Computing, 10(1): 438–449

    A graph learning based approach for identity infer- ence in dapp platform blockchain. IEEE Transactions on Emerging Topics in Computing, 10(1): 438–449. Liu, Y .; Hu, T.; Zhang, H.; Wu, H.; Wang, S.; Ma, L.; and Long, M. 2023. itransformer: Inverted transformers are effective ...

    2023 arXiv

  14. [2022]

    arXiv preprint arXiv:2211.14730

    A time series is worth 64 words: Long-term forecast- ing with transformers. arXiv preprint arXiv:2211.14730. Ozturan, C.; Sen, A.; and Kilic, B. 2021. Transaction Graph Dataset for the Ethereum Blockchain. Zenodo, 4718440. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury,...

    2021 arXiv

  15. [2023]

    In E3S Web of Conferences, volume 391, 01112

    A Cryptocurrency Price Prediction Model using Deep Learning. In E3S Web of Conferences, volume 391, 01112. EDP Sciences. Alahmari, S. A. 2019. Using Machine Learning ARIMA to Predict the Price of Cryptocurrencies. ISeCure, 11(3). Bhatt, S.; Ghazanfar, M.; and Amirhosseini, M. ...

    2019 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.