Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Enhancing Trading Performance Through Sentiment Analysis with Large Language Models: Evidence from the S&P 500

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that combining LLM-based news sentiment with technical indicators improves S&P 500 trading, with the best hybrid returning 5.77% versus -0.696% for buy-and-hold.

desk verdict The 5.77% backtest result likely reflects in-sample label leakage, since the sentiment classifiers are trained on return-derived labels from the same May–August 2024 window used for evaluation. read the letter →

arxiv 2507.09739 v1 pith:WQ5PCYQH submitted 2025-07-13 q-fin.CP q-fin.TR

Haojie Liu , Zihan Lin , Randall R. Rojas This is my paper
classification q-fin.CPq-fin.TR
keywords S&P500sentimentanalysislargelanguagemodeltime-seriesforecastingtradingstrategyGPT-2FinBERTtechnicalindicators
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that adding real-time news sentiment extracted by fine-tuned GPT-2 and FinBERT to traditional technical indicators and time-series forecasts improves S&P 500 trading performance. On a May 10 to August 7, 2024 backtest, the best hybrid—GPT-2 sentiment from Dow Jones news combined with volume-weighted MACD—returned 5.77%, while buy-and-hold lost 0.696%. If true, this supports a hybrid approach to market timing that adapts to volatile environments. The paper also reports that sentiment classifiers beat pure technical indicators at predicting daily direction, with FinBERT reaching 75.56% accuracy on Benzinga news and GPT-2 reaching 65.48% on WSJ.

What carries the argument

The load-bearing mechanism is the normalized combined signal $I_t = \sum_i I_{it} / |\sum_i I_{it}|$, which maps each day's sentiment and technical votes to a value in $[-1,1]$ that dictates buying, selling, or holding. The other essential piece is the labeling scheme that converts daily S&P 500 returns into three sentiment classes and uses those labels to fine-tune the language models, so news text becomes a daily sentiment score.

What would settle it

Re-run the trading simulation with a hard temporal split: fine-tune GPT-2 and FinBERT only on news-return pairs dated before May 10, 2024, then apply them to the test window and recompute the VW MACD hybrid return. If the 5.77% figure shrinks to near zero or negative, the claimed out-of-sample improvement is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that fusing LLM-derived news sentiment with a momentum indicator can produce trading signals that outperform passive holding. The authors construct daily sentiment labels from S&P 500 returns (above +1%, between -1% and +1%, below -1%), fine-tune GPT-2 and FinBERT on news paired to those labels, aggregate predictions to a daily mode, and combine them with technical signals through a normalized sum. In the tested window, the combination of GPT-2 sentiment on Dow Jones news with VW MACD yields 5.77%, compared with -0.696% for buy-and-hold. The paper interprets this as evidence that sentiment adds predictive value when layered on technical indicators, even though sentiment alone is not consistently profitable.

Load-bearing premise

The load-bearing premise is that the sentiment models' predictions are genuinely out-of-sample; the paper never states a train/test split, so the labels used to fine-tune GPT-2 and FinBERT may include the May–August 2024 period on which the 5.77% return is measured.

Editorial extensions

If this is right

  • The best hybrid strategy (GPT-2 on Dow Jones news plus VW MACD) returns 5.77% over the test period, beating buy-and-hold by about 6.5 percentage points.
  • Sentiment models in isolation do not consistently outperform technical or time-series baselines; the improvement comes from combining sentiment with technical indicators.
  • FinBERT's best classification accuracy (75.56% on Benzinga) and GPT-2's best (65.48% on WSJ) exceed all tested technical indicators, none of which exceeded 10% accuracy.
  • Volume-weighted MACD and Dual MACD consistently generate positive returns across news sources and sentiment models, suggesting momentum confirmation is a robust complement to sentiment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A strict temporal train/test split—excluding May through August 2024 from the fine-tuning labels—would be needed to confirm the 5.77% return is out-of-sample; the paper does not report one, so leakage is a live risk.
  • The same vote-aggregation mechanism with its 16:00–23:59 ET and weekend-to-Monday lag rules could be applied to intraday news to produce same-day signals, potentially increasing trade frequency.
  • Because the paper assumes zero transaction costs and perfect execution, live returns would likely be lower; a natural test is to re-run the backtest with realistic spreads and slippage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes to combine LLM-based news sentiment analysis (GPT-2 and FinBERT) with technical indicators and time-series forecasts to time the S&P 500 index. News articles are labeled by the sign of the same day's realized return, the LLMs are fine-tuned on these labels, and the resulting daily sentiment is then combined with MACD, SAR, VW MACD, Dual MACD, ARIMA, Prophet, and ETS to generate buy/sell signals in a simulated trading strategy. The best hybrid strategy (GPT-2 sentiment on Dow Jones news with VW MACD) is reported to earn 5.77% versus -0.696% for buy-and-hold over May 10 to August 7, 2024.

Significance. If the results were valid, the paper would provide evidence that LLM-based news sentiment adds value to technical trading rules for index timing. The paper has some strengths: it uses multiple news sources, two LLMs, and a transparent set of performance tables. However, the central empirical claim is not supported because the sentiment models are trained on labels constructed from the same return series used to evaluate the strategy, and no out-of-sample separation is established. As such, the headline return could be an artifact of in-sample fitting. The paper also contains methodological gaps in the trading simulation that prevent replication. These issues outweigh the potential contribution.

major comments (4)
  1. [§2.4, §3.4.1, §3.5, §4.4] The sentiment labels are created from realized daily returns over August 2019–August 2024 (§2.4), and GPT-2 and FinBERT are fine-tuned on these labels (§3.4.1). The trading test period May 10–August 7, 2024 (§3.5) lies inside this labeling window, and no train/test split is described. Consequently, the classification accuracies in Table 1 and the 5.77% return in Table 3 can be achieved by memorizing the test-period returns; the backtest is not out-of-sample. This is the load-bearing piece of evidence for the paper's central claim, and it must be redone with a strict temporal holdout.
  2. [§3.5] The trading simulation is not fully specified. The combined signal It is defined as ΣI_it / |ΣI_it|, but the paper never defines I_it or how sentiment and technical indicators are encoded, weighted, and aggregated. The capital update rules are contradictory: when It < 0 the strategy sells all shares (Ct+1 = Pt * St) but when It ≤ 0 retains shares (St+1 = St), and when It > 0 it buys with all cash (St+1 = Ct/Pt) but sets Ct+1 = Ct, which together imply holding both cash and shares. Without a precise, consistent trading rule the reported returns in Tables 3–5 cannot be reproduced or interpreted.
  3. [Tables 3–5] The headline 5.77% is the best result across a large grid of combinations: two LLMs, five news sources, seven technical indicators, and three time-series models (plus sentiment-only and technical-only baselines). No correction for multiple testing is reported, and the tables show many negative returns, so the maximum could be an in-sample selection artifact. The paper should either pre-register the strategy, use a separate validation period, or report the full distribution of returns with appropriate multiplicity-adjusted inference.
  4. [§4.1, Table 1] The sentiment classification accuracies are computed against the same return labels used to fine-tune the models, so they measure in-sample fit rather than predictive accuracy. Additionally, no baseline accuracy (e.g., majority-class) is given, making the 75.56% figure difficult to interpret. For the central claim, what matters is the out-of-sample agreement between news sentiment and subsequent returns, which is not reported.
minor comments (6)
  1. [Abstract] The abstract contains a grammatical issue: 'By merging sentiment data with momentum and trend-based metrics, including a benchmark buy-and-hold and sentiment-based approach, is evaluated' is an incomplete sentence and should be rewritten.
  2. [§2.2] The text states that the k = 1 specification 'delivers the best out-of-sample accuracy,' but no out-of-sample evaluation is described anywhere; this usage of 'out-of-sample' is inconsistent with the rest of the paper, which never establishes a holdout period.
  3. [Tables 3 and 5] The row labels in Tables 3 and 5 are confusing: Table 3 rows are labeled 'GPT Dow Jones,' etc., but the table includes FinBERT results; please clarify whether the label refers to the sentiment source or the model used to generate the sentiment, and align the captions accordingly.
  4. [§4.2] The accuracy metric for time-series models is not defined; it is presumably the proportion of correctly predicted return signs, but this should be stated explicitly. Without this definition, Table 2 is difficult to interpret, especially because technical indicator accuracies below 10% seem inconsistent with a three-class prediction problem.
  5. [Figure 1] The figure caption describes the actual returns as a 'dashed black line,' while the text in Section 4.4 refers to a 'gray line'; these descriptions should be reconciled.
  6. [§6 (Limitations)] The limitations section lists zero transaction costs, equal headline weighting, a short sample period, and focus on the S&P 500, but it does not mention the absence of a train/test split, which is the most serious limitation affecting the validity of the results.

Circularity Check

1 steps flagged · score 7.0 of 10

The headline 5.77% hybrid return is not established as out-of-sample: the sentiment labels are return categories, the LLMs are fine-tuned on those labels, and the May–August 2024 backtest lies inside the labeling window with no stated train/test split.

  1. fitted input called prediction [Section 2.4; Section 3.4.1; Section 3.5; Section 4.4 and Table 3]
    "After merging, the stock price data was converted into categorical sentiment labels (positive, neutral, negative) as described earlier. We then appended this stock sentiment to each news article for that day. ... They fine-tuned these models for financial sentiment analysis and integrated the resulting sentiment scores with stock price data. ... It creates trading signals from the overall market sentiment and daily stock returns and utilize these for simulating trades ... The strategy is designed to adjust the assets dynamically based on predicted sentiment signals for each trading day."

    Section 2.1 defines the sentiment labels as the sign/magnitude of the S&P 500 daily return, and Section 2.4 appends those return-derived labels to each day's news articles. Section 3.4.1 then fine-tunes GPT-2 and FinBERT on this labeled corpus, with no train/test split stated anywhere in the manuscript. Section 3.5 feeds the fine-tuned models' outputs back as 'predicted sentiment signals' and computes trading returns for May 10–August 7, 2024, a period strictly inside the August 2019–August 2024 labeling window. Consequently the models may have memorized the realized return categories of the evaluation days; the 'prediction' is a fitted value, not an out-of-sample forecast.

full rationale

Walking the derivation chain: daily S&P 500 returns are thresholded into positive/neutral/negative labels (Section 2.1); these labels are merged onto news articles (Section 2.4); GPT-2 and FinBERT are fine-tuned on the merged labeled corpus (Section 3.4.1); the models' daily outputs are used as 'predicted sentiment signals' in the trading simulation (Section 3.5), whose performance is reported for May 10–August 7, 2024 (Section 4.4, Table 3). Because the backtest window is contained in the labeling window and no train/test split is stated, the sentiment component of the hybrid signal reduces to a fitted mapping from news text to contemporaneous/future return categories. The 5.77% result therefore does not by itself demonstrate out-of-sample skill. This is a leakage/circularity issue in the paper's central claim. No load-bearing self-citation chain was found: the authors' own prior work (Huang et al. 2018) is only background motivation, and no uniqueness theorem or ansatz is imported from it. The paper's limitations section acknowledges zero transaction costs and the short May–August window, but not the missing holdout split. If a proper temporal split had been documented, the design could support the claim; as written, the derivation chain is incomplete and the main prediction is at best an in-sample fit.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central claim depends on the sentiment classifier being a genuine out-of-sample predictor, but the classifier is trained on labels derived from the same returns used in the backtest, with no described train/test split. Additional hand-chosen elements include return thresholds, dropout rates, lag selection, and Dual MACD parameters. No new entities are introduced.

free parameters (5)
  • Return category thresholds = >1%, -1% to 1%, <-1%
    Section 2.1 defines positive, neutral, and negative labels from daily return thresholds; these thresholds determine the target the sentiment model learns.
  • GPT-2 dropout rate = 0.5
    Section 3.4.1: dropout rate set to 0.5 for GPT-2 during fine-tuning.
  • FinBERT dropout rate = 0.8
    Section 3.4.1: dropout rate set to 0.8 for FinBERT during fine-tuning.
  • News-to-return lag k = 1
    Section 2.2: lag k=1 selected because it 'delivers the best out-of-sample accuracy' among k=0,1,2; this is model selection on the same data.
  • Dual MACD long-term parameters = 19, 39, 9
    Section 3.2.4: the longer-term MACD settings (19-39-9) are introduced as an example with no justification, affecting the Dual MACD signal.
assumptions (5)
  • domain assumption Daily news sentiment labels can be constructed from same-day categorized stock returns
    Section 2.4 appends return-based categorical labels to each news article, assuming news text is predictive of the same day's return category.
  • domain assumption Zero transaction costs and perfect execution
    Section 3.5 and Section 6 state this assumption explicitly and acknowledge it overstates returns.
  • ad hoc to paper No train/test split is needed for the fine-tuned sentiment models
    The paper fine-tunes GPT-2 and FinBERT on the full sample (or an unspecified sample) and then backtests in May-August 2024 without describing temporal separation, so leakage is not ruled out.
  • ad hoc to paper Sentiment, technical, and time-series signals can be summed without weighting or dependency checks
    Section 3.5 defines the combined signal as the normalized sum of indicator votes; the paper does not justify additivity or account for correlations among signals.
  • ad hoc to paper The mode of per-article sentiments represents daily market sentiment
    Sections 3.4.1 and 4.1 aggregate multiple daily articles using the mode, but provide no evidence that the mode is a better summary than the mean or a volume-weighted score.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Trading Performance Through Sentiment Analysis with Large Language Models: Evidence from the S&P 500." pith.science (2026). https://pith.science/paper/WQ5PCYQH

@misc{pith2026250709739,
  author       = {Pith},
  title        = {Pith review of: Enhancing Trading Performance Through Sentiment Analysis with Large Language Models: Evidence from the S&P 500},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQ5PCYQH}},
  note         = {Machine review of arXiv:2507.09739}
}
read the original abstract

This study integrates real-time sentiment analysis from financial news, GPT-2 and FinBERT, with technical indicators and time-series models like ARIMA and ETS to optimize S&P 500 trading strategies. By merging sentiment data with momentum and trend-based metrics, including a benchmark buy-and-hold and sentiment-based approach, is evaluated through assets values and returns. Results show that combining sentiment-driven insights with traditional models improves trading performance, offering a more dynamic approach to stock trading that adapts to market changes in volatile environments.

Figures

Figures reproduced from arXiv: 2507.09739 by the authors.

Figure 1
Figure 1. This plot compares the actual returns with predicted returns from ARIMA, ETS, [PITH_FULL_IMAGE:figures/full_fig_p026_1.png] view at source ↗
Figure 2
Figure 2. This plot compares the assets values of various trading models from May to early [PITH_FULL_IMAGE:figures/full_fig_p027_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 22 canonical work pages

  1. [1]

    G. Appel. Technical Analysis: Power Tools for Active Investors. FT Press, 2005

    work page 2005

  2. [2]

    D. Araci. Finbert: Financial sentiment analysis with pre-trained language models, 2019

    work page 2019

  3. [3]

    G. E. P. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung. Time Series Analysis: Forecasting and Control. Holden-Day, San Francisco, 1970

    work page 1970

  4. [4]

    A comparative study of MACD -based trading strategies: Evidence from the us stock market

    Pat Tong Chio. A comparative study of MACD -based trading strategies: Evidence from the us stock market. arXiv preprint 2206.12282, 2022. URL https://arxiv.org/abs/2206.12282

    arXiv 2022

  5. [5]

    Financial sentiment analysis: Techniques and applications

    Kelvin Du, Frank Xing, Rui Mao, and Erik Cambria. Financial sentiment analysis: Techniques and applications. ACM Computing Surveys, 56 0 (9), 2024. doi:10.1145/3649451. URL https://doi.org/10.1145/3649451

    work page doi:10.1145/3649451 2024

  6. [6]

    Engelberg and C

    J. Engelberg and C. A. Parsons. The causal impact of media in financial markets. The Journal of Finance, 66 0 (1): 0 67--97, 2011

    work page 2011

  7. [7]

    Using financial news sentiment for stock price direction prediction

    Bledar Fazlija and Pedro Harder. Using financial news sentiment for stock price direction prediction. Mathematics, 10 0 (13): 0 2156, 2022. doi:10.3390/math10132156. URL https://doi.org/10.3390/math10132156

    work page doi:10.3390/math10132156 2022

  8. [8]

    Twitter sentiment and stock market movements: The predictive power of social media

    Talita Greyling and Stephanié Rossouw. Twitter sentiment and stock market movements: The predictive power of social media. VoxEU Column, 2025. URL https://cepr.org/voxeu/columns/twitter-sentiment-and-stock-market-movements-predictive-power-social-media

    work page 2025

Show all 27 references
  1. [9]

    Informational role of social media: Evidence from twitter sentiment

    Chen Gu and Alexander Kurov. Informational role of social media: Evidence from twitter sentiment. Journal of Banking & Finance, 121: 0 105969, 2020. doi:10.1016/j.jbankfin.2020.105969. URL https://doi.org/10.1016/j.jbankfin.2020.105969

    2020

  2. [10]

    Revised short screening version of the profile of mood states (poms-16): Validity and reliability

    Andreas Hinz et al. Revised short screening version of the profile of mood states (poms-16): Validity and reliability. Frontiers in Psychology, 12: 0 631668, 2021. doi:10.3389/fpsyg.2021.631668. URL https://doi.org/10.3389/fpsyg.2021.631668

    2021

  3. [11]

    Huang, Randall R

    Melody Y. Huang, Randall R. Rojas, and Patrick D. Convery. News sentiment as leading indicators for recessions. arXiv preprint arXiv:1805.04160v2, 2018

    2018 arXiv

  4. [12]

    Hyndman and George Athanasopoulos

    Rob J. Hyndman and George Athanasopoulos. Forecasting: Principles and Practice. OTexts, 2 edition, 2018. URL https://otexts.com/fpp2/

    2018

  5. [13]

    P. J. Kaufman. Trading Systems and Methods. Wiley, 5 edition, 2013

    2013

  6. [14]

    Sentiment trading with large language models

    Kemal Kirtac and Guido Germano. Sentiment trading with large language models. Finance Research Letters, 62: 0 105227, 2024. doi:10.1016/j.frl.2024.105227. URL https://doi.org/10.1016/j.frl.2024.105227

    2024

  7. [15]

    Finvader: VADER sentiment classifier updated with financial lexicons

    Petr Korab. Finvader: VADER sentiment classifier updated with financial lexicons. GitHub repository, 2023. URL https://github.com/PetrKorab/FinVADER

    2023

  8. [16]

    Loughran and B

    T. Loughran and B. McDonald. When is a liability not a liability? textual analysis, dictionaries, and 10-ks. The Journal of Finance, 66 0 (1): 0 35--65, 2011

    2011

  9. [17]

    Design and evaluation of SentiEcon : A fine-grained economic/financial sentiment lexicon from a corpus of business news

    Antonio Moreno-Ortiz, Javier Fernandez-Cruz, and Chantal Pérez-Hernández. Design and evaluation of SentiEcon : A fine-grained economic/financial sentiment lexicon from a corpus of business news. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages ...

    2020

  10. [18]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1 0 (8), 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  11. [19]

    R. P. Schumaker and H. Chen. Textual analysis of stock market prediction using breaking financial news: The azfin text system. ACM Transactions on Information Systems (TOIS), 27 0 (2): 0 1--19, 2009

    2009

  12. [20]

    Shumway and David S

    Robert H. Shumway and David S. Stoffer. Time Series Analysis and Its Applications: With R Examples . Springer, 5 edition, 2025. doi:10.1007/978-3-031-70584-7. URL https://doi.org/10.1007/978-3-031-70584-7

    2025 doi

  13. [21]

    Forecasting at scale

    Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72 0 (1): 0 37--45, 2018

    2018

  14. [22]

    P. C. Tetlock. Giving content to investor sentiment: The role of media in the stock market. The Journal of Finance, 62 0 (3): 0 1139--1168, 2007

    2007

  15. [23]

    Bitcoin price change and trend prediction through twitter sentiment and data volume

    Jacques Vella Critien, Albert Gatt, and Joshua Ellul. Bitcoin price change and trend prediction through twitter sentiment and data volume. Financial Innovation, 8: 0 45, 2022. doi:10.1186/s40854-022-00352-7. URL https://doi.org/10.1186/s40854-022-00352-7

    2022 doi

  16. [24]

    Wang and J

    J. Wang and J. Kim. Predicting stock price trend using macd optimized by historical volatility. Journal of Applied Mathematics, 2018: 0 1--12, 2018

    2018

  17. [25]

    Efficient market hypothesis in contemporary applications: A systematic review on theoretical models, experimental validation, and practical application

    Chuer Xu. Efficient market hypothesis in contemporary applications: A systematic review on theoretical models, experimental validation, and practical application. Highlights in Business, Economics and Management, 21: 0 231--239, 2023. doi:10.54097/hbem.v21i.14337. URL https://...

    2023 doi

  18. [26]

    Y. Yang, M. C. S. Uy, A. Huang, and M. Liu. Finbert: A pretrained language model for financial communications, 2019

    2019

  19. [27]

    Finbert: A pretrained language model for financial communications

    Yi Yang, Mark Christopher Siy UY, and Allen Huang. Finbert: A pretrained language model for financial communications. arXiv preprint, 2020. URL https://arxiv.org/abs/2006.08097

    2020 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.