{"id":"a468a9fd-eb3a-49f0-b841-eafd22b3f8a6","arxiv_id":"2501.11906","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"3MEthTaskforce is a new multi-source Ethereum dataset combining transactions, token prices, market indices, and Reddit sentiment, with benchmarks for user behavior and price prediction.","lead":"This paper releases 3MEthTaskforce, a large Ethereum dataset that combines over 300 million transactions, 3,880 token profiles, global market indicators, and Reddit sentiment from 2014 to 2024. It benchmarks six dynamic graph models and nineteen time-series models on user behavior and token price prediction tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM-generated sentiment indices likely encode post-timestamp information, inflating the value of sentiment features in both benchmark tasks.","rationale":"The paper's central contribution is a dataset and benchmark platform. The reader's concern about subset representativeness is valid but secondary: representativeness affects generalizability, whereas sentiment lookahead affects the validity of the experimental comparisons themselves. The leaked sentiment would also affect any downstream user of the dataset who relies on the provided sentiment indices. The paper's own datasheet answer that the data is 'free from errors, noise, and redundancies' is implausible, but the leakage is a concrete, testable methodological flaw. Therefore, the conditional verdict stands: the paper should be accepted only if the authors either demonstrate that sentiment scores are temporally clean (e.g., by using a model with training cutoff or a causal method) or re-release the sentiment data with appropriate safeguards and re-benchmark.","tokens_in":22736,"tokens_out":5111,"duration_ms":48877,"concrete_test":"Regenerate the sentiment scores from the same Reddit posts using a prompt that omits timestamps and any instruction to consider temporal context (e.g., 'score this text by its literal sentiment'), or use a sentiment lexicon like VADER on the raw text where available, then re-run the key experiments: for user behavior, compare TCL/TGN on trans text llm versus trans text; for price prediction, compare iTransformer on price all versus price on the 211-token multivariate setup. If the performance gap narrows significantly or reverses, the original sentiment features leak future information and the benchmark claims must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the sentiment indices used as features in both user behavior prediction (Section 'User Behavior Experiment Setup', vectors trans text and trans text llm) and token price prediction (Section 'Price Prediction Experiment Setup', vectors price text and price all) are produced by an LLM with instructions that invite lookahead bias. In Appendix 'Prompt', Prompt 2 asks the model to 'consider the trends and news about cryptocurrencies at the time of the timestamp' and to judge whether the context 'seemed to have a positive or negative impact on cryptocurrencies at the time.' The model, DeepSeek, has pre-training data that includes information posterior to the post's timestamp (e.g., a 2013 post can be scored with knowledge of Bitcoin's later price trajectory). There is no temporal cutoff, masking, or causal constraint in the sentiment generation pipeline. Consequently, the sentiment features may encode future price movements, inflating the measured benefit of sentiment in the benchmark tables (e.g., TCL NAP rising from 0.737 on trans text to 0.786 on trans text llm in Table 2; iTransformer's msMAPE improving on All vs Price in Table 4). This does not disprove the dataset's value, but it directly undermines the paper's benchmark conclusions about the contribution of sentiment features and the relative model rankings on those features.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 3MEthTaskforce, a publicly released Ethereum dataset and benchmark platform integrating over 300 million transaction records, 3,880 token profiles, global market indicators, and Reddit sentiment data from 2014 to 2024. It defines two tasks—user behavior prediction and token price prediction—and reports benchmark results for six dynamic graph neural networks and nineteen time-series forecasting models. The central claims are that the multi-source, multi-level, multi-token design enables integrated studies of user behavior, sentiment, and token performance, and that the provided benchmarks map current model performance on these tasks.","tokens_in":22971,"tokens_out":5438,"duration_ms":51918,"significance":"If the technical concerns are resolved, this is a potentially valuable community resource: it is one of the few datasets to combine transaction-level Ethereum data with token metadata, global market indices, and long-horizon Reddit sentiment, and it ships with FAIR-aligned documentation and a benchmark suite covering a broad set of GNN and time-series baselines. The scale (303M transactions, 3,880 tokens) and the explicit task definitions are strengths, and the authors are to be credited for releasing data and code. However, the benchmark conclusions are currently undermined by a likely lookahead bias in the LLM-generated sentiment features, by the unexamined representativeness of the evaluation subsets, and by missing metric definitions and error reporting.","major_comments":[{"comment":"The sentiment scores spos_t and sneg_t are generated by DeepSeek under Prompt 2, which instructs the model to consider 'the trends and news about cryptocurrencies at the time of the timestamp' and to judge whether the context 'seemed to have a positive or negative impact on cryptocurrencies at the time.' Because DeepSeek's pretraining data includes information posterior to posts from 2013–2024, these scores can encode future price movements. There is no temporal cutoff, masking, or causal constraint in the sentiment generation pipeline. This directly affects the central ablation claims: in Table 2, TCL NAP rises from 0.737 on trans text to 0.786 on trans text llm, and in Table 4 the multivariate results are used to argue that sentiment features improve forecasting. As written, the reported sentiment-feature gains may be inflated by leakage. Please regenerate the sentiment indices with a causal pipeline (e.g., prompting with only text and public knowledge available at the timestamp, or applying a strict knowledge cutoff), or explicitly re-frame the sentiment-ablation results as non-causal upper bounds and add a temporal holdout in which the LLM's training cutoff precedes the test period.","section":"Prompt (Appendix) and feature-ablation results in Tables 2–4"},{"comment":"The user behavior experiments use a subset of about 260,000 transactions from 29,164 wallets and 1,194 tokens, while the multivariate price experiments use 211 tokens with at least six years of history; the full dataset spans 303 million transactions, 35 million wallets, and 3,880 tokens. The paper provides no representativeness analysis comparing these evaluation subsets to the full population along dimensions such as token type, market capitalization, transaction frequency, or time period. Without such evidence, the benchmark conclusions about which models and features work best cannot be generalized to the platform the paper claims to provide. Please add a distribution comparison between the subsets and the full dataset, or explicitly scope the benchmark claims to the subsets.","section":"User Behavior Experiment Setup; Price Prediction Experiment Setup"},{"comment":"The metrics called Normalized MSE, Normalized MAE, WAPE, and MSMAPE are never defined by equations. The LUNC values in Table 3 (e.g., 7.44e-11 for LinearRegression and 2.86e-8 for ARIMA) are impossible to interpret without knowing the normalization scheme, and the multivariate metrics in Table 4 cannot be checked or reproduced. Define all four metrics precisely, including denominators, aggregation over tokens and time steps, and the exact normalization applied to each token's price series.","section":"Price Prediction Experiment Setup, Performance Metrics"},{"comment":"The text states that results are averaged over three runs with a confidence level of 0.05 and that errors fall within the confidence interval, but Tables 2–4 report only point estimates. Many comparisons in Table 2 are separated by less than 0.01 in TAP/NAP (e.g., DyGFormer 0.928 on trans versus 0.939 on trans global), so without standard deviations or confidence intervals the reader cannot judge whether the reported differences are meaningful. Please include error bars or intervals for all entries, or state explicitly which differences are statistically significant.","section":"User Behavior Experiment Setup; Price Prediction Experiment Setup; Tables 2–4"},{"comment":"The text contains internal contradictions with the reported numbers. It says 'iTransformer's msMAPE on the All dataset (17.03) was notably better than on the Price dataset (16.23)', but lower msMAPE is better, so 16.23 is the better value and the statement is backwards. It also says TCN achieved the lowest msMAPE of 11.19 on the Price dataset, whereas Table 4 shows TCN's Price msMAPE is 30.7; the value 11.19 appears to be the Global wAPE. These inconsistencies directly affect the paper's feature-impact claims and must be corrected.","section":"Feature Impact paragraph and Table 4"}],"minor_comments":[{"comment":"The datasheet asserts that 'the dataset is free from errors, noise, and redundancies.' Given the scale of 303 million transaction records and the multi-source cleaning pipeline, this claim needs support from explicit validation steps, or it should be softened to describe known limitations and any residual noise.","section":"Appendix, Datasheets for Datasets, Composition"},{"comment":"The paragraph on Ethereum Public ETL states that the free Ethereum API gathered '5,855 active wallet addresses and their transactions' and then reports '267,242 transaction records among 29,164 wallet addresses'; please clarify the relationship between these two numbers, since one would expect the API-gathered addresses to match the final wallet count.","section":"Data Collection"},{"comment":"The five input vectors are sometimes called 'transaction input vectors' and sometimes 'transaction labels'; please use one consistent term throughout the section to avoid confusion between the prediction target and the input features.","section":"User Behavior Experiment Setup"},{"comment":"The metric names appear as 'Normalized MSE', 'Normalized MAE', 'WAPE', and 'MSMAPE' in the text and as 'wape' and 'msmape' in Table 4; standardize the spelling and capitalization of the metric acronyms.","section":"Price Prediction Experiment Setup, Performance Metrics"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a plausible and useful contribution for a data/benchmark venue, and I would not reject it outright. The most serious issue is the likely lookahead bias in the LLM sentiment features; if the authors can recompute the sentiment indices with a causal constraint or clearly delimit the claims that depend on them, the paper could become publishable. The unresolved representativeness question and the metric definition gaps also need substantial work before the benchmark results can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: 3MEthTaskforce is a genuinely useful dataset resource, but the benchmark results that come with it are not trustworthy as reported because the LLM-generated sentiment features almost certainly leak future information.\n\nWhat is new: they've assembled 303M transaction records, 3,880 token profiles, global market indices, and Reddit sentiment (2014-2024) into one public platform with code, a DOI, and a data sheet. No existing dataset combines all of these. The collection pipeline is straightforward and the FAIR claims are credible. They also define two tasks and run a broad set of baselines: six dynamic GNNs and nineteen time-series models. That is real work and, once cleaned up, a valuable service to the community.\n\nThe soft spots are not tiny. The sentiment annotation prompt (Appendix: Prompt 2) explicitly tells the LLM to 'consider the trends and news about cryptocurrencies at the time of the timestamp' and to judge whether the context 'seemed to have a positive or negative impact on cryptocurrencies at the time.' DeepSeek's training data includes material posterior to those timestamps; there is no temporal cutoff, masking, or causal constraint in the pipeline. So the sentiment scores can encode hindsight about price movements. That explains the apparently large gains from sentiment features in Tables 2 and 4 (e.g., TCL's NAP jump from 0.737 to 0.786; iTransformer's msMAPE improvement). The benchmark conclusions about sentiment value are not reliable.\n\nTwo more concerns, in decreasing order. The evaluation uses a 260k-transaction subset and a 211-token subset with no representativeness check, so the model ranking may not generalize to the full 303M-transaction, 3,880-token platform. And the paper never defines 'Normalized MSE', 'WAPE', or 'MSMAPE', and error bars are claimed but not shown in the tables. The datasheet's assertion that the data is 'free from errors, noise, and redundancies' is also an overclaim.\n\nI would not desk-reject this. The resource is worth refereeing seriously; the fix is specific and feasible. The authors should regenerate sentiment using only information available at the post timestamp (or at least demonstrate that the scores do not vary with future data), add uncertainty estimates, define metrics, and scope the benchmark claims to the subsets actually used. The dataset will still be useful even if the sentiment benchmarks need to be redone.\n\nFor your reading group, this is a good case study in data-annotation pitfalls for temporal benchmarks. Take it to peer review, but expect heavy revision.","headline":"Useful dataset resource, but the LLM sentiment features almost certainly leak future information, so the benchmark results need a causal redo.","tokens_in":23501,"tokens_out":2624,"would_cite":true,"duration_ms":26230,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces 3MEthTaskforce, a publicly released Ethereum dataset and benchmark platform that combines 303 million token-transaction records, 3,880 token profiles, global market indicators, and a decade of Reddit sentiment…","keywords":["Ethereum","blockchain dataset","user behavior prediction","token price prediction","temporal graph neural networks","time-series forecasting","market sentiment","decentralized finance"],"falsifier":"Compare the distribution of transaction volumes, token ages, and wallet activity in the evaluation subsets against the same statistics computed on the full 303-million-transaction, 3,880-token dataset; a large divergence would mean the benchmark results cannot be assumed to hold for the whole platform.","tokens_in":22515,"feed_emoji":"📊","tokens_out":6031,"duration_ms":52234,"temperature":0.7,"pith_summary":"This paper introduces 3MEthTaskforce, a publicly released Ethereum dataset and benchmark platform that combines 303 million token-transaction records from 3,880 tokens, 3,880 token profiles, global market indicators, and ten years of Reddit sentiment. The authors aim to give researchers a single integrated resource for studying how user behavior, market sentiment, and token prices interact. To show the dataset is usable, they define two tasks—user behavior prediction and token price prediction—and benchmark six dynamic graph neural networks and nineteen time-series models on them. A sympathetic reading takes the paper's contribution to be the dataset and its baseline results, not a new learning method. If the resource is as useful as claimed, it would lower the barrier to multimodal blockchain analytics and decentralized-finance risk research.","feed_headline":"New Ethereum dataset links 303M transactions, prices, and sentiment","feed_subtitle":"Benchmarks for user-behavior and price-prediction tasks give researchers a shared base for multimodal blockchain analytics.","key_machinery":"The central object is the integrated multi-level dataset itself: transaction-level records (sender, receiver, token, value, timestamp), token-level metadata (price, market capitalization, 24-hour volume), global-level market indices, and textual-level sentiment indices computed from Reddit posts. The benchmark's second key mechanism is the task formulation: user behavior prediction is cast as edge prediction on a temporal bipartite graph of wallets and tokens, while price prediction is cast as univariate and multivariate time-series forecasting with feature sets that progressively add global and sentiment signals. Sentiment indices are produced by prompting a large language model to score posts, with one version giving a single 0–10 score and another giving separate positive and negative scores.","core_discovery":"On its own terms, the paper claims that 3MEthTaskforce is the first multi-source, multi-level, multi-token Ethereum dataset and benchmark platform. It integrates over 300 million transaction records, 3,880 token profiles, global market indicators such as Bitcoin dominance and total market capitalization, and Reddit-derived sentiment scores spanning 2014–2024. The platform defines two core tasks: predicting whether a user will buy or sell a token by treating transactions as temporal edges in a bipartite graph, and predicting token prices from price series augmented with global indices and sentiment. The benchmark results show that LLM-derived positive/negative sentiment scores improve user behavior prediction over a single overall sentiment score, and that adding global and sentiment features improves multivariate price forecasting, with iTransformer ranking best among the nineteen time-series models and JODIE and DyGFormer leading the dynamic GNNs.","pith_inferences":["The paper's own experiments run on a 260,000-transaction slice and a 211-token subset; if those slices are not representative of the full 303-million-transaction, 3,880-token corpus, the benchmark rankings may not carry over to the whole platform.","Because the release provides only LLM-generated sentiment scores rather than raw Reddit text, users cannot build their own sentiment models or inspect the original posts, which limits the asset's reusability for NLP research.","The sentiment signal comes from Reddit subreddits, so it likely captures a particular community's mood; extending the data with other social feeds could change the feature's predictive value.","The dataset covers a period with major market regime changes, so it could be used to test whether models that win on average still fail at turning points such as crashes, a question the paper does not address."],"forward_implications":["Researchers in blockchain analytics get a single multimodal dataset that links transaction graphs, token fundamentals, market-level indicators, and sentiment, instead of stitching separate sources together.","The published benchmark numbers give a baseline map of current model performance: dynamic GNNs reach above 0.94 average precision on behavior prediction, and adding sentiment and global features consistently improves multivariate price forecasts.","The two defined tasks—user behavior prediction and token price prediction—provide standardized evaluation protocols for future work on Ethereum data.","The platform's design supports downstream risk analysis and market-fluctuation modeling, including applications such as detecting abnormal trading behavior or modeling panic events like the UST/LUNA collapse."],"supporting_citations":[{"why":"Prior transaction-graph dataset that lacks market indices and sentiment, motivating the multi-source design.","marker":"(Ozturan, Sen, and Kilic 2021)"},{"why":"Graph benchmark for limited cryptocurrencies that omits sentiment and global indices, defining the comparison gap.","marker":"(Shamsi et al. 2022)"},{"why":"EX-Graph links Ethereum to social profiles but its text cannot serve as a market sentiment indicator.","marker":"(Wang et al. 2024a)"},{"why":"Open-source Ethereum ETL tool used to collect the 303-million-transaction records.","marker":"(Medvedev 2018)"},{"why":"JODIE, one of the six dynamic GNN baselines and a strong performer in behavior prediction.","marker":"(Kumar, Zhang, and Leskovec 2019)"},{"why":"TGAT baseline and source of the TAP/NAP evaluation metrics for link prediction.","marker":"(Xu et al. 2020)"},{"why":"TGN baseline, a scalable temporal graph network used in the user behavior benchmark.","marker":"(Rossi et al. 2020)"},{"why":"iTransformer, the best-ranked multivariate price forecasting model in the benchmark.","marker":"(Liu et al. 2023)"}],"fun_headline_variants":["Multi-source Ethereum dataset: 303M transactions, 3,880 tokens, sentiment","Ethereum benchmark dataset links 303M transactions to sentiment and prices","Ethereum dataset: 303M txns, 3,880 tokens, Reddit sentiment, benchmarks","Benchmarks for Ethereum behavior and price from 303M transactions and sentiment","First multi-source Ethereum dataset for behavior, price, and sentiment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark conclusions assume that the roughly 260,000-transaction, 29,164-wallet subset used for user behavior experiments and the 211 long-history tokens used for multivariate price prediction are representative of the full 303-million-transaction, 3,880-token dataset.","fun_headline_variants_meta":{"raw":{"variants":["Multi-source Ethereum dataset: 303M transactions, 3,880 tokens, sentiment","Ethereum benchmark dataset links 303M transactions to sentiment and prices","Ethereum dataset: 303M txns, 3,880 tokens, Reddit sentiment, benchmarks","Benchmarks for Ethereum behavior and price from 303M transactions and sentiment","First multi-source Ethereum dataset for behavior, price, and sentiment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002894,"raw_usage":{"total_tokens":10940,"prompt_tokens":834,"completion_tokens":10106,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":10001}},"tokens_in":450,"tokens_out":10106,"duration_ms":67522,"temperature":1.0,"reasoning_tokens":10001,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:44:00.614672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the distribution of transaction volumes, token ages, and wallet activity in the evaluation subsets against the same statistics computed on the full 303-million-transaction, 3,880-token dataset; a large divergence would mean the benchmark results cannot be assumed to hold for the whole platform.","supporting_citations":[],"review_version":1}