REVIEW 4 major objections 5 minor 32 references
PulseReddit: A Novel Reddit Dataset for Benchmarking MAS in High-Frequency Cryptocurrency Trading
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that LLM-based multi-agent traders using a new Reddit sentiment dataset beat SMA, MACD, and LSTM baselines, especially in bull markets, with up to 50% higher returns.
desk verdict Useful dataset, but the trading evaluation is not reliable: duplicated baselines, no transaction costs, and no error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is PulseReddit itself: a cleaned corpus of Reddit posts and first-level comments from r/Bitcoin, r/ethereum, r/dogecoin, r/solana, r/binance, and r/pepecoin, timestamp-aligned with Binance OHLCV data at four trading intervals. The second mechanism is the four-agent multi-agent system built on the CryptoTrade design, where a Market Analyst, News Analyst, Trading Agent, and Reflection Agent work together and the News Analyst injects PulseReddit sentiment while the Reflection Agent adjusts strategy after each episode. The Trading Agent outputs a continuous action in [-1,1] representing proportional buy or sell volume, which makes the benchmarked returns depend directly on the quality of the synthesized signals.
What would settle it
Recompute the returns in Tables 3 through 6 under realistic conditions: a taker fee per trade, one bid-ask tick of slippage, and a one-step order lag at each 5-minute decision point. If the multi-agent advantage over SMA and MACD vanishes or reverses at 5-minute granularity, the paper's high-frequency superiority claim fails.
Extended reading notes
Core claim
The central discovery, stated on the paper's own terms, is that a four-agent LLM trading system augmented with PulseReddit social data outperforms traditional technical and learning-based baselines over the tested market days, especially in bull regimes. The strongest single result is deepseek-chat returning 5.40% on BTC at 4-hour intervals in the bull window versus 3.41% for SMA; multi-agent models also take the best or second-best slot across most coin, frequency, and regime blocks in Tables 3 through 6. The authors interpret this as evidence that off-chain Reddit sentiment, synthesized with on-chain technical analysis, provides a tangible edge for short-term trading. They further claim that adding Reddit data improves both total return and Sharpe ratio in their ablation, and that GPT-4o-mini offers a favorable speed-performance trade-off for practical deployment.
Load-bearing premise
The reported returns assume every trade fills at the recorded price with no fees, slippage, or latency, and the simulation's few selected days stand in for high-frequency trading generally.
Editorial extensions
If this is right
- If the claim holds, Reddit sentiment becomes a usable off-chain feature for short-horizon crypto trading agents, not just a curiosity.
- Multi-agent systems with PulseReddit data would be the recommended benchmark configuration for bull-market trading simulations, while classic indicators remain competitive in sideways regimes.
- Practitioners would prefer the lighter GPT-4o-mini over GPT-4o or DeepSeek-Chat when latency or compute budget matters, accepting a small performance gap.
- The dataset provides a common evaluation ground for future multi-agent high-frequency trading research by aligning social and market data at matching time resolutions.
Reading between the lines
- Editorial extension: the paper's own tables show that most multi-agent returns are negative even in bull windows, so the claimed edge is relative loss reduction against baselines, not absolute profitability.
- Editorial extension: because the simulation ignores transaction costs, the 5-minute results are upper bounds; the more defensible practical claim may be at 4-hour horizons.
- Editorial extension: the same dataset construction could be applied to other social platforms, such as X posts or Telegram channels, to test whether Reddit-specific sentiment is the active ingredient or any retail-discussion feed would do.
- Editorial extension: a direct test of the dataset's value would be an event study around the November 2024 activity spikes, measuring whether sentiment spikes precede or follow price moves.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PulseReddit, a dataset of Reddit posts and comments from six cryptocurrency subreddits, synchronized with Binance price data at 5-minute to 4-hour intervals, and evaluates LLM-based multi-agent systems (MAS) built on the CryptoTrade framework against traditional and LSTM baselines in bull, bear, and sideways markets. The central claim is that MAS augmented with PulseReddit data achieve superior trading outcomes, particularly in bull markets, with up to 50% higher returns. The dataset is publicly available and the paper includes a preprocessing pipeline, an ablation study, and a qualitative case study.
Significance. If the empirical claims were reliable, the paper would be a useful contribution: PulseReddit could serve as a benchmark for social-signal-driven trading research, and the comparison of LLM backends for trading agents would inform practical model selection. The dataset itself appears to be a plausible new resource, and the authors make it openly available. However, the evaluation has several load-bearing flaws that undermine the central claim of superior MAS trading performance: transaction costs are omitted, one baseline (LSTM) contains internally inconsistent data, hyperparameters appear to be selected on the test periods, and all results are single runs. The dataset contribution may still have value, but the trading-performance evidence is not credible as presented.
major comments (4)
- [Section 3.1] The Return metric is defined solely as (w_end - w_start)/w_start, and the Market Setup text describes agents acting at 5-minute to 4-hour intervals with continuous actions in [-1,1]. No transaction cost, exchange fee, or slippage term is introduced anywhere. At 5-minute rebalancing, the number of trades is large, and even a 0.1% per-side fee plus a realistic spread would substantially erode the reported returns. For example, the headline result in Table 3, DeepSeek-chat at BTC 4h bull (5.40% versus SMA 3.41%), is a narrow raw margin that could vanish under execution costs. Because every comparison in Tables 3-6 uses this cost-free return, the claimed superiority of MAS in high-frequency trading is not established.
- [Tables 3-6] For all four coins, the LSTM row under 'Sideways Market' is identical to the LSTM row under 'Bull Market' in every time frequency (e.g., BTC: -9.09, -4.86, -1.86, -0.68; DOGE: -11.00, -6.84, -3.37, -2.08; ETH: -10.07, -4.20, -1.95, -0.93; SOL: -11.95, -4.66, -4.64, -2.41). A data-dependent baseline cannot produce identical results across two distinct market regimes, indicating a copy/paste or data-loading error. The LSTM baseline is therefore invalid, and any conclusion that compares MAS to LSTM in sideways markets is unsupported.
- [Appendix D] The parameter sensitivity analysis reports that the optimal price window and reflection window (both set to 5) are chosen to maximize daily return and Sharpe ratio, but no separate validation set or temporal holdout is described. If these windows are selected using the same market periods whose returns are reported in Tables 3-6, the MAS results are selected on the test data, overstating performance. The paper must either define a proper tuning protocol using a validation period or report results across all tested window settings.
- [Section 3.1 and Tables 3-6] Every reported result is a point estimate from a single run at temperature 0.7. There is no repetition with different random seeds, no reporting of variance, and no significance tests. Given the stochastic nature of LLM outputs and the large variance typical of short-horizon trading simulations, the abstract's 'conclusive' claims about superior performance and robustness are not statistically supported.
minor comments (5)
- [Section 2.2] The text between the first sentence of Section 2.2 and Figure 2 contains a long corrupted segment of '/uni00000015/uni00000013/...' that appears to be an encoding artifact. This should be removed or fixed.
- [Table 2] The text says that three sideways market days were randomly selected, but Table 2 lists only bull and bear dates; no sideways rows appear in the table.
- [Table 7] For gpt-4o-mini, the 15-minute execution time for BTC and DOGE is identical (44.50 minutes); this may be coincidental, but it should be checked to rule out a data copying error.
- [References] The reference for 'Agenttaxo' is cited as 'Wang et al., 2015', but the entry itself describes an ICLR 2025 Workshop paper; the year appears to be a typo.
- [Abstract and Section 3.4] The abstract states 'up to 50% higher returns' without a pointer to the specific table or cell, and Section 3.4 contains a typo ('shwon' instead of 'shown').
Circularity Check
No significant circularity: PulseReddit is an independently constructed dataset and the evaluation is self-contained.
full rationale
PulseReddit is a newly collected dataset (Section 2) built from Reddit API data and Binance klines; its construction does not depend on the paper's conclusions. The central claim is supported by direct backtests in Tables 3-6 and by an ablation (Table 8) that compares the same MAS with and without Reddit input. No equation in Section 3.1 defines Return in terms of the Reddit signal or vice versa, so the 'superior trading outcomes' conclusion is not equivalent to the dataset by construction. The reuse of CryptoTrade (Li et al., 2024a) is a self-citation, but it is not load-bearing in the circularity sense: the framework is executed in this paper rather than being justified solely by the citation, and no uniqueness claim is imported from the cited work. Appendix D's selection of price/reflection windows on the evaluation data is a statistical-reliability concern, not a circular reduction: those hyperparameters are not a renamed version of the reported return or Sharpe ratio, and the baseline comparisons do not rest on a claim that the settings were predicted without access to outcomes. Likewise, the absence of transaction costs and slippage undermines the practical HFT interpretation, but this is a validity limitation, not a case of the conclusion being built into the premises. No self-definitional, fitted-as-prediction, or uniqueness-imported circularity can be exhibited with a specific equation or reduction, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- SMA window =
chosen among [5,10,15,20,25,30] by validation performance
- MAS price window =
5
- MAS reflection window =
5
- Reddit comment length filter =
10 to 100 words
assumptions (5)
- domain assumption Reddit sentiment contains predictive information for short-term crypto price movements
- domain assumption The three selected days per market regime are representative of bull, bear, and sideways conditions
- domain assumption LLM agents' textual outputs can be converted into a continuous trading action in [-1,1] that reflects a real executable strategy
- domain assumption Zero transaction costs and unlimited liquidity
- domain assumption Reddit data density is sufficient at 5-minute granularity
Cite this review
Pith. "Pith review of PulseReddit: A Novel Reddit Dataset for Benchmarking MAS in High-Frequency Cryptocurrency Trading." pith.science (2026). https://pith.science/paper/PG3YZR6T
@misc{pith2026250603861,
author = {Pith},
title = {Pith review of: PulseReddit: A Novel Reddit Dataset for Benchmarking MAS in High-Frequency Cryptocurrency Trading},
year = {2026},
howpublished = {\url{https://pith.science/paper/PG3YZR6T}},
note = {Machine review of arXiv:2506.03861}
}
read the original abstract
High-Frequency Trading (HFT) is pivotal in cryptocurrency markets, demanding rapid decision-making. Social media platforms like Reddit offer valuable, yet underexplored, information for such high-frequency, short-term trading. This paper introduces \textbf{PulseReddit}, a novel dataset that is the first to align large-scale Reddit discussion data with high-frequency cryptocurrency market statistics for short-term trading analysis. We conduct an extensive empirical study using Large Language Model (LLM)-based Multi-Agent Systems (MAS) to investigate the impact of social sentiment from PulseReddit on trading performance. Our experiments conclude that MAS augmented with PulseReddit data achieve superior trading outcomes compared to traditional baselines, particularly in bull markets, and demonstrate robust adaptability across different market regimes. Furthermore, our research provides conclusive insights into the performance-efficiency trade-offs of different LLMs, detailing significant considerations for practical model selection in HFT applications. PulseReddit and our findings establish a foundation for advanced MAS research in HFT, demonstrating the tangible benefits of integrating social media.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
High-frequency trading: a practical guide to algorithmic strategies and trading systems
Aldridge, I. High-frequency trading: a practical guide to algorithmic strategies and trading systems. John Wiley & Sons, 2013
work page 2013
-
[3]
Phishing scams detection in ethereum transaction network
Chen, L., Peng, J., Liu, Y., Li, J., Xie, F., and Zheng, Z. Phishing scams detection in ethereum transaction network. ACM Transactions on Internet Technology (TOIT), 21 0 (1): 0 1--16, 2020
work page 2020
-
[4]
Judgelrm: Large reasoning models as a judge
Chen, N., Hu, Z., Zou, Q., Wu, J., Wang, Q., Hooi, B., and He, B. Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050, 2025
arXiv 2025
-
[5]
Cheng, J. and Chin, P. Sociodojo: Building lifelong analytical agents with real-world text and time series. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=s9z0HzWJJp
work page 2024
-
[6]
Ferdiansyah, F., Othman, S. H., Radzi, R. Z. R. M., Stiawan, D., Sazaki, Y., and Ependi, U. A lstm-method for bitcoin price prediction: A case study yahoo finance stock market. In 2019 international conference on electrical engineering and computer science (ICECOS), pp.\ 206--210. IEEE, 2019
work page 2019
-
[7]
Non-linear prediction of security returns with moving average rules
Gencay, R. Non-linear prediction of security returns with moving average rules. Journal of Forecasting, 15 0 (3): 0 165--174, 1996
work page 1996
-
[8]
Ethereum account classification based on graph convolutional network
Huang, T., Lin, D., and Wu, J. Ethereum account classification based on graph convolutional network. IEEE Transactions on Circuits and Systems II: Express Briefs, 69 0 (5): 0 2528--2532, 2022
work page 2022
Show all 32 references
-
[9]
Jones, C. M. What do we know about high-frequency trading? Columbia Business School Research Paper, 0 (13-11), 2013
2013
-
[10]
Crafting papers on machine learning
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann
2000
-
[11]
Ttagn: Temporal transaction aggregation graph network for ethereum phishing scams detection
Li, S., Gou, G., Liu, C., Hou, C., Li, Z., and Xiong, G. Ttagn: Temporal transaction aggregation graph network for ethereum phishing scams detection. In Proceedings of the ACM Web Conference 2022, pp.\ 661--669, 2022
2022
-
[12]
Cryptotrade: A reflective llm-based agent to guide zero-shot cryptocurrency trading
Li, Y., Luo, B., Wang, Q., Chen, N., Liu, X., and He, B. Cryptotrade: A reflective llm-based agent to guide zero-shot cryptocurrency trading. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 1094--1106, 2024 a
2024
-
[13]
A reflective llm-based agent to guide zero-shot cryptocurrency trading
Li, Y., Luo, B., Wang, Q., Chen, N., Liu, X., and He, B. A reflective llm-based agent to guide zero-shot cryptocurrency trading. arXiv preprint arXiv:2407.09546, 2024 b
2024 arXiv
-
[14]
Modeling and understanding ethereum transaction records via a complex network approach
Lin, D., Wu, J., Yuan, Q., and Zheng, Z. Modeling and understanding ethereum transaction records via a complex network approach. IEEE Transactions on Circuits and Systems II: Express Briefs, 67 0 (11): 0 2737--2741, 2020
2020
-
[15]
Lin, D., Wu, J., Xuan, Q., and Chi, K. T. Ethereum transaction tracking: Inferring evolution of transaction networks via link prediction. Physica A: Statistical Mechanics and its Applications, 600: 0 127504, 2022
2022
-
[16]
Fingpt: Democratizing internet-scale data for financial large language models
Liu, X.-Y., Wang, G., and Zha, D. Fingpt: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485, 2023
2023 arXiv
-
[17]
Multi-chain graphs of graphs: A new approach to analyzing blockchain datasets
Luo, B., Zhang, Z., Wang, Q., and He, B. Multi-chain graphs of graphs: A new approach to analyzing blockchain datasets. Advances in Neural Information Processing Systems, 37: 0 28490--28514, 2024
2024
-
[18]
Market efficiency and stability in the era of high-frequency trading: A comprehensive review
Nahar, J., Nishat, N., Shoaib, A., and Hossain, Q. Market efficiency and stability in the era of high-frequency trading: A comprehensive review. International Journal of Business and Economics, 1 0 (3): 0 1--13, 2024
2024
-
[19]
Debagreement: A comment-reply dataset for (dis) agreement detection in online debates
Pougu \'e -Biyong, J., Semenova, V., Matton, A., Han, R., Kim, A., Lambiotte, R., and Farmer, D. Debagreement: A comment-reply dataset for (dis) agreement detection in online debates. In Thirty-fifth conference on neural information processing systems datasets and benchmarks t...
2021
-
[20]
Shamsi, K., Victor, F., Kantarcioglu, M., Gel, Y., and Akcora, C. G. Chartalist: Labeled graph datasets for utxo and account-based blockchains. Advances in Neural Information Processing Systems, 35: 0 34926--34939, 2022
2022
-
[21]
and Kim, J
Wang, J. and Kim, J. Predicting stock price trend using macd optimized by historical volatility. Mathematical Problems in Engineering, 2018: 0 1--12, 2018
2018
-
[22]
Tsgn: Transaction subgraph networks assisting phishing detection in ethereum
Wang, J., Chen, P., Xu, X., Wu, J., Shen, M., Xuan, Q., and Yang, X. Tsgn: Transaction subgraph networks assisting phishing detection in ethereum. arXiv preprint arXiv:2208.12938, 2022
2022 arXiv
-
[23]
Agenttaxo: Dissecting and benchmarking token distribution of llm multi-agent systems
Wang, Q., Tang, Z., JIANG, Z., Chen, N., Wang, T., and He, B. Agenttaxo: Dissecting and benchmarking token distribution of llm multi-agent systems. In ICLR 2025 Workshop on Foundation Models in the Wild, 2015
2025
-
[24]
Ex-graph: A pioneering dataset bridging ethereum and x
Wang, Q., Zhang, Z., Liu, Z., Lu, S., Luo, B., and He, B. Ex-graph: A pioneering dataset bridging ethereum and x. arXiv preprint arXiv:2310.01015, 2023
2023 arXiv
-
[25]
Megaagent: A practical framework for autonomous cooperation in large-scale llm agent systems
Wang, Q., Wang, T., Li, Q., Liang, J., and He, B. Megaagent: A practical framework for autonomous cooperation in large-scale llm agent systems. arXiv preprint arXiv:2408.09955, 2024
2024 arXiv
-
[26]
Assessing judging bias in large reasoning models: An empirical study
Wang, Q., Lou, Z., Tang, Z., Chen, N., Zhao, X., Zhang, W., Song, D., and He, B. Assessing judging bias in large reasoning models: An empirical study. arXiv preprint arXiv:2504.09946, 2025 a
2025 arXiv
-
[27]
What limits llm-based human simulation: Llms or our design? arXiv preprint arXiv:2501.08579, 2025 b
Wang, Q., Wu, J., Tang, Z., Luo, B., Chen, N., Chen, W., and He, B. What limits llm-based human simulation: Llms or our design? arXiv preprint arXiv:2501.08579, 2025 b
2025 arXiv
-
[28]
Bloomberggpt: A large language model for finance
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023
2023 arXiv
-
[29]
Pixiu: A large language model, instruction data and evaluation benchmark for finance
Xie, Q., Han, W., Zhang, X., Lai, Y., Peng, M., Lopez-Lira, A., and Huang, J. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443, 2023
2023 arXiv
-
[30]
W., and Khashanah, K
Yu, Y., Li, H., Chen, Z., Jiang, Y., Li, Y., Zhang, D., Liu, R., Suchow, J. W., and Khashanah, K. Finmem: A performance-enhanced llm trading agent with layered memory and character design. In Proceedings of the AAAI Symposium Series, volume 3, pp.\ 595--597, 2024
2024
-
[31]
A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist
Zhang, W., Zhao, L., Xia, H., Sun, S., Sun, J., Qin, M., Li, X., Zhao, Y., Zhao, Y., Cai, X., et al. A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Da...
2024
-
[32]
Live graph lab: Towards open, dynamic and real transaction graphs with nft
Zhang, Z., Luo, B., Lu, S., and He, B. Live graph lab: Towards open, dynamic and real transaction graphs with nft. Advances in Neural Information Processing Systems, 36, 2024 b
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.