REVIEW 4 major objections 6 minor 12 references
FinRL-DeepSeek: LLM-Infused Risk-Sensitive Reinforcement Learning for Trading Agents
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that an LLM-based news risk score, folded into CVaR-PPO, lets a trading agent outperform the Nasdaq-100 after two million training steps.
desk verdict A simple LLM-risk-injection idea with public code, but the current evidence—unvalidated risk scores, hand-set multipliers, and two stochastic runs—cannot support the claimed CPPO-DeepSeek outperformance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a pair of LLM-generated multipliers wrapped around CVaR-PPO. Conditional Value-at-Risk (CVaR) is the average loss in the worst tail of outcomes, and CVaR-PPO adds a penalty proportional to how far trajectory returns fall below a threshold. A recommendation score $S_f$ from a 1–5 LLM prompt scales the agent's action, $a_t^{\mathrm{mod}} = S_f a_t$, with multipliers kept close to 1 (0.9–1.1) for stability. A separate risk score $R_i^f$ from a 1–5 risk prompt is mapped to per-stock multipliers, combined by portfolio weight into $R_f$, and used to rescale the full trajectory return before the CVaR penalty is computed. This lets the news-derived risk score change how the algorithm weighs tail losses; the paper treats the risk prompt as its main new component, distinct from the recommendation prompt taken from prior work.
What would settle it
Rerun the same two-million-step backtest with the LLM risk and recommendation scores replaced by random draws from the same 1–5 distribution, keeping every other setting fixed; if the random-score CPPO variant still matches or beats the Nasdaq-100 benchmark, the news signal is not what is carrying the result.
Extended reading notes
Core claim
The central claim is that news carries a risk signal that price-only reinforcement learning misses, and that this signal can be captured by asking an LLM for a 1–5 risk score. The paper introduces an aggregate news-risk multiplier $R_f = \sum_i w_i R_i^f$, built from per-stock LLM risk scores weighted by portfolio weights, and applies it to the trajectory return in the CPPO objective as $D_{R_f}(\pi_\theta) = R_f D(\pi_\theta)$. In two stochastic backtests run for two million training steps, PPO and CPPO-DeepSeek outperform the other listed methods, including the Nasdaq-100 benchmark; the author notes that plain PPO looks stronger in bull markets while CPPO-DeepSeek looks stronger in bear markets, with the transition placed at the end of 2021. The paper presents this as evidence that LLMs can serve as feature extractors for risk from news beyond standard sentiment analysis.
Load-bearing premise
The whole comparison rests on the assumption that one randomly chosen news article per stock per day, converted to a 1–5 score by an LLM and then to a multiplier between 0.9 and 1.1, retains real information about future risk or returns; the paper reports no check that LLM scores predict anything out-of-sample.
Editorial extensions
If this is right
- LLM risk scores can be added to an existing risk-sensitive RL pipeline with prompt calls and two scalar multipliers, requiring no extra model training.
- The infusion-strength findings indicate that the multiplier range matters: stronger LLM infusion helped CPPO but hurt PPO in the reported runs.
- Training duration conditions the result: no LLM variant beat the Nasdaq-100 benchmark at 400–500k steps, while the two-million-step run did, so short backtests can miss the effect.
- The reported bull/bear split suggests plain PPO and news-risk CPPO are complementary in different market regimes.
Reading between the lines
- A shuffled-score control is the decisive missing experiment; without it, the gap could come from added variance or optimizer interactions rather than from the meaning of the news.
- Because the paper subsamples one article per stock per day, an article-selection policy (for example, choosing the highest-impact headline) is a natural extension that could change the signal's strength.
- The bull/bear division points to a regime-switching agent that runs PPO in rising markets and CPPO-DeepSeek in falling markets; this is an extension the paper notes but does not build.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FinRL-DeepSeek, a trading agent that augments PPO and CVaR-PPO (CPPO) with LLM-generated stock recommendation scores and risk assessment scores obtained from financial news. The LLM scores are injected multiplicatively into actions and trajectory returns, and the method is evaluated in backtests on Nasdaq-100 constituents using news from the FNSPID dataset, comparing DeepSeek V3, Qwen 2.5, and Llama 3.3. The central claim is that after 2 million training steps, CPPO-DeepSeek outperforms other methods, including the Nasdaq-100 benchmark, thereby demonstrating the value of LLM-based risk assessment for risk-sensitive RL trading.
Significance. The direction is timely and the manuscript is transparent in releasing code and data. If the central claim were established, the contribution would be a simple, modular way to inject LLM news signals into risk-sensitive reinforcement learning, which could be useful to practitioners. However, the current evidence does not establish the claim: the paper reports contradictory results across training horizons, relies on one or two stochastic runs without error bars, uses hand-set multipliers without any validation of the LLM signal, and reports metrics (near-zero Information Ratios, negative CVaR for every model) that do not support the stated outperformance. The manuscript is best viewed as a preliminary exploratory report rather than a substantiated research contribution.
major comments (4)
- [§5.1.2, §5.2.2] Section 5.1.2 explicitly states, 'The use of LLM always worsens performance in this test,' for PPO and CPPO at 400–500k training steps, while Section 5.2.2 claims that after 2M steps PPO-DeepSeek and CPPO-DeepSeek outperform other methods, including the Nasdaq-100 benchmark. The paper offers no mechanism, no statistical comparison, and no discussion to reconcile this reversal. As written, the central claim is contradicted by the paper's own earlier experimental section, and the authors need to either explain the reversal with controlled experiments or substantially weaken the claim.
- [§5.2, Tables 1–3] The reported metrics do not support the 'outperform' claim. Information Ratios in Tables 1–3 range from -0.0252 to 0.0100, and every model has negative CVaR. In Table 1, PPO achieves an Information Ratio of 0.0100, which is higher than the headline CPPO-DeepSeek value of 0.0078. The text in Section 5.2.2 does not specify which metric supports the claim, and the reliance on one or two stochastic runs without seed information, confidence intervals, or significance tests means that the apparent advantage of CPPO-DeepSeek could be sampling noise. Multiple independent seeds, error bars, and a formal comparison against the benchmark's IR and CVaR are required.
- [§4.2, §4.3, §5.3] The LLM signal injection is built on hand-set multipliers (0.9, 0.95, 1, 1.05, 1.1) applied multiplicatively to actions and trajectory returns, with no evidence that the LLM risk scores or recommendation scores correlate with subsequent realized risk or returns. Without a control condition using shuffled or randomly generated scores, the reported difference between CPPO and CPPO-DeepSeek in Table 3 (IR 0.0078 vs -0.0148 at 10% infusion) is indistinguishable from arbitrary scaling plus sampling noise. The choice to report the 10% variant for CPPO and the 0.1% variant for PPO in Section 5.2 appears to have been made after observing test-set performance; this selection bias needs to be addressed by pre-registering the infusion strength or reporting all variants with a multiple-comparison correction.
- [§3] Section 3 states that one news article per stock per day is randomly selected to reduce LLM API costs. This introduces sampling noise and ignores the possibility that the selected article is uninformative, redundant, or unrepresentative of the day's news flow. The paper provides no sensitivity analysis over article selection, so the LLM score could reflect the arbitrary article draw rather than a genuine news signal. The authors should report results across multiple article-selection seeds or otherwise demonstrate that the choice of article does not drive the backtest differences.
minor comments (6)
- [§4.1.2] The claim 'To our knowledge, we are the first to apply CVaR-PPO to stock trading' is a strong novelty assertion with no literature search or comparative discussion; it should either be substantiated with a broader related-work review or softened.
- [§5.1] The sentence 'In these tests, we use a 10% infusion of LLM, which means that the original PPO and CPPO up to 10%' is incomplete and unclear; please rewrite to specify exactly how the 10% infusion is defined and applied.
- [§4.3, Eq. (1)] The relationship between the trajectory return D(πθ) in Eq. (1) and the adjusted return D_{R_f}(πθ) = R_f · D(πθ) in Section 4.3 should be made explicit; currently the notation appears without a clear definition of D(πθ) before Eq. (1).
- [Figures 1–6, Tables 1–3] Figure and table captions do not consistently identify which LLM (DeepSeek, Qwen, or Llama), which infusion strength, and which random seed produced each result; this information is essential for reproducibility.
- [§5.2.2] The phrase 'In these two runs (RL agents are stochastic)' is vague; the paper should state the total number of independent runs per configuration and report the seeds or a seed range.
- [§5.2, Tables 1–3] The backtest appears to ignore transaction costs, slippage, and portfolio turnover; given the near-zero Information Ratios, these costs could plausibly eliminate any reported advantage, so they should be modeled or explicitly discussed.
Circularity Check
No circularity: the LLM risk scores are external inputs and the reported outperformance is an empirical backtest comparison, not an equation-level consequence of the paper's definitions.
full rationale
The paper's derivation chain is not circular. The LLM recommendation and risk scores are generated from FNSPID news by pretrained LLMs through prompts, and the constants 0.9-1.1 in Sections 4.2 and 4.3 are hand-set multipliers defining perturbed actions and adjusted returns; they are not fitted from the backtest or defined in terms of the reported performance metric. The claimed CPPO-DeepSeek outperformance is an empirical observation from two stochastic RL runs, not a quantity forced by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known result under new coordinates. The paper's weaknesses—lack of validation that LLM risk scores predict future risk, random selection of one article per stock per day, and choosing the 10% infusion level after inspecting test results—are correctness and overfitting concerns, not circularity. Under the hard rules, a circularity claim requires exhibiting a specific reduction of a prediction to its input, which is absent here. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- LLM recommendation multiplier S_f =
1.1, 1.05, 0.95, 0.9 depending on score and action sign
- LLM risk multiplier R_i^f =
1.1, 1.05, 1.0, 0.95, 0.9 for risk scores 5, 4, 3, 2, 1
- LLM infusion strength =
10%, 1%, 0.1% variants; the favorable configuration is reported
assumptions (4)
- domain assumption LLM scores from the prompts are meaningful financial risk and recommendation signals
- domain assumption One randomly selected news article per stock per day preserves the information of the full FNSPID dataset
- ad hoc to paper Multiplicative perturbation of actions (S_f * a_t) and returns (R_f * D) is a sound way to inject LLM signals into PPO and CVaR-PPO
- domain assumption Backtest on Nasdaq-100 with FNSPID news is a valid out-of-sample evaluation
Cite this review
Pith. "Pith review of FinRL-DeepSeek: LLM-Infused Risk-Sensitive Reinforcement Learning for Trading Agents." pith.science (2026). https://pith.science/paper/Z7J2DJGH
@misc{pith2026250207393,
author = {Pith},
title = {Pith review of: FinRL-DeepSeek: LLM-Infused Risk-Sensitive Reinforcement Learning for Trading Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z7J2DJGH}},
note = {Machine review of arXiv:2502.07393}
}
read the original abstract
This paper presents a novel risk-sensitive trading agent combining reinforcement learning and large language models (LLMs). We extend the Conditional Value-at-Risk Proximal Policy Optimization (CPPO) algorithm, by adding risk assessment and trading recommendation signals generated by a LLM from financial news. Our approach is backtested on the Nasdaq-100 index benchmark, using financial news data from the FNSPID dataset and the DeepSeek V3, Qwen 2.5 and Llama 3.3 language models. The code, data, and trading agents are available at: https://github.com/benstaf/FinRL_DeepSeek
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[3]
Zihan Dong, Xinyu Fan, and Zhiyuan Peng
URL https://arxiv.org/abs/2411.00782. Zihan Dong, Xinyu Fan, and Zhiyuan Peng. Fnspid: A comprehensive financial news dataset in time series. arXiv preprint arXiv:2402.06698,
-
[4]
URL https: //arxiv.org/abs/2402.06698. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[5]
Shreyas Lele, Kavit Gangar, Harshal Daftary, and De- washish Dharkar
URL https://arxiv.org/abs/2407.21783. Shreyas Lele, Kavit Gangar, Harshal Daftary, and De- washish Dharkar. Stock market trading agent using on-policy reinforcement learning algorithms. https: //ssrn.com/abstract=3582014,
-
[8]
URL https://arxiv. org/abs/2304.07619. A. Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115,
-
[9]
URL https://arxiv. org/abs/2412.15115. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization al- gorithms. arXiv preprint arXiv:1707.06347,
-
[11]
Financial News-Driven LLM Reinforcement Learning for Portfolio Management
URL https://arxiv. org/abs/2411.11059. Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031,
-
[13]
doi: 10.24963/ijcai.2022/
-
[15]
URL https:// arxiv.org/abs/2407.06567. 5
Show all 12 references
-
[2017]
URL https://arxiv.org/abs/1707.06347. A. Unnikrishnan. Financial news-driven llm reinforce- ment learning for portfolio management. arXiv preprint arXiv:2411.11059,
-
[2022]
Alejandro Lopez-Lira and Yuehua Tang
URL https:// arxiv.org/abs/2011.09607. Alejandro Lopez-Lira and Yuehua Tang. Can chat- gpt forecast stock price movements? return pre- dictability and large language models. arXiv preprint arXiv:2304.07619,
2011 arXiv
-
[2023]
ChengYang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu
URL https: //arxiv.org/abs/2306.06031. ChengYang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. In Proceedings of the Thirty-First International Joint Con- ference on Artificial Int...
-
[2024]
4 Qianggang Ding, Haochen Shi, and Bang Liu
URL https:// arxiv.org/abs/2412.19437. 4 Qianggang Ding, Haochen Shi, and Bang Liu. Tradex- pert: Revolutionizing trading with mixture of expert llms. arXiv preprint arXiv:2411.00782 ,
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.