Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

FinMarBa: A Market-Informed Dataset for Financial Sentiment Classification

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FinMarBa labels financial headlines by market reaction, and its sentiment signal outperforms human-annotated Financial-Phrasebank in an S&P 500 backtest.

desk verdict A useful dataset pipeline undermined by a confounded comparison and a wrong t-statistic; the resource deserves a second chance, the central claim does not. read the letter →

arxiv 2507.22932 v1 pith:QI6ES6NV submitted 2025-07-24 cs.CL q-fin.GN

Baptiste Lefort , Eric Benhamou , Beatrice Guez , Jean-Jacques Ohana , Ethan Setrouk , Alban Etienne This is my paper
classification cs.CLq-fin.GN
keywords financialsentimentclassificationmarket-basedannotationBloombergMarketWrapslargelanguagemodelsdatasetconstructionSharperatiobacktestingGPT-4
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces FinMarBa, a dataset of 61,252 financial headlines labeled not by human readers but by the market's own reaction: a headline is positive if the associated ticker's next-day return exceeds its 60th percentile over the prior five years, negative if below the 30th percentile, and indecisive otherwise. The authors claim this market-informed labeling produces sentiment signals that are more predictive than human-annotated Financial-Phrasebank, evidenced by a Sharpe ratio of 0.30 versus -0.13 in an S&P 500 backtest from 2019 to 2024. The paper argues that human annotation carries biases and fails to capture actual market impact, so a market-based dataset is a better foundation for financial NLP and signal generation. The dataset, fine-tuned BERT models, and labeling code are released openly.

What carries the argument

The annotation pipeline starts with Bloomberg Market Wraps, which are condensed into headlines by GPT-4; GPT-4 also assigns each headline a list of affected tickers. For each ticker, the next trading day's percentage price change is compared to the 30th and 60th quantiles of the prior five-year rolling return distribution, producing +1, -1, or 0 labels. This quantile thresholding is the load-bearing device that turns market data into sentiment labels without human judgment.

What would settle it

Recompute FinMarBa labels using the return from the publication day's close to the next day's close instead of the day-after-publication return, then rerun the same backtest; if the Sharpe advantage over Financial-Phrasebank disappears or reverses, the reported predictive power is an artifact of the one-day labeling lag.

Watch

Extended reading notes

Core claim

The central claim is that FinMarBa's market-driven annotation yields a sentiment classification dataset whose labels reflect actual price reactions, and that models trained on it produce a trading signal with positive risk-adjusted returns, while the same signal built from Financial-Phrasebank-trained FinBERT is negative. The paper reports a Sharpe ratio of 0.30 for FinMarBaBERT versus -0.13 for FinBERT, with a t-statistic of 10 (p below 0.01), and robustness tests where swapping headlines into future windows increases the Sharpe gap relative to backward windows, supporting that the labels encode forward-looking information.

Load-bearing premise

The label is valid only if the next trading day's return is caused by the headline, but the daily Market Wrap is published after market close and summarizes that day's news, so the following day's move may be driven by different information.

Editorial extensions

If this is right

  • Models fine-tuned on FinMarBa produce a daily sentiment score with a Sharpe ratio of 0.30 over 4.5 years, implying the market-informed labels can support tradable signals.
  • The dataset's label distribution (42% positive, 31% negative, 26% indecisive) better matches the long-term positive drift of equity markets than Financial-Phrasebank's 59% indecisive share.
  • The method scales to large corpora without manual annotation, allowing fine-tuning and evaluation sets far larger than the roughly 5,000 sentences of Financial-Phrasebank.
  • The open release of the dataset, models, and code lets others reproduce the backtest and reuse the labeling pipeline on new headlines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal timing assumption fails, the reported Sharpe may partly reflect that the label window begins after the Market Wrap's publication, so part of the 'signal' could be lookahead; a timestamp-aware backtest would settle this.
  • The quantile thresholds 30/60 are fixed across all tickers and regimes; tuning them per ticker or volatility state could improve the labeling, a testable extension the paper does not run.
  • The comparison varies both dataset and model (FinBERT vs FinMarBaBERT); an ablation that trains both architectures on both datasets would isolate the labeling effect from model initialization.
  • Because headlines are generated by GPT-4 from Bloomberg summaries, the dataset inherits any selection bias of Bloomberg editors and any summarization errors; its coverage is US-equity-heavy and may not transfer to other regions or asset classes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces FinMarBa, a financial sentiment dataset whose labels are assigned automatically from the one-day-ahead return of tickers mentioned in Bloomberg Market Wrap headlines, using a quantile rule (Eq. 1). The authors compare a BERT model fine-tuned on FinMarBa (FinMarBaBERT) against FinBERT trained on Financial-Phrasebank, reporting a Sharpe ratio of 0.30 versus -0.13 on an S&P 500 backtest and claiming statistical significance via t-statistics. They also report label-distribution statistics and robustness experiments as supporting evidence. The manuscript's central claim is that market-informed annotation yields sentiment labels that are more predictive than human-annotated labels.

Significance. If the central claim were established, market-informed labeling would be a valuable, low-cost alternative to human annotation for financial sentiment datasets, with clear practical relevance for signal generation. The paper also contributes a large corpus, an automated annotation pipeline, and open-source resources, all of which are useful to the community. However, the empirical demonstration as written does not support the claim: the headline comparison is confounded, the statistical tests are miscalibrated, and the label-distribution argument is largely predetermined by the chosen thresholds. These issues are load-bearing for the paper's main conclusion, though the dataset itself may still be a useful resource.

major comments (5)
  1. [§5.2, Table 4] The t-statistics in Table 4 are computed as Sharpe ratio multiplied by sqrt(4.5*250). This is incorrect by a factor of sqrt(250) regardless of whether the Sharpe ratio is annualized or daily. If the reported Sharpe ratio is annualized, the correct t-statistic over 4.5 years is approximately SR*sqrt(4.5) = 0.30*2.12 = 0.64, not 10. If it is daily, the correct t-statistic is SR_daily*sqrt(1125), which equals the annualized SR*sqrt(4.5); either way, the reported values and the claimed p-values below 1 percent are unsupported.
  2. [§5.3, Table 5] The robustness analysis treats differences in Sharpe ratios as if they were directly comparable to a normal-theory critical value. The statement that a ratio is significant if it exceeds 1.65*sqrt(4.5*250) ≈ 0.05 conflates units of a Sharpe-ratio difference with a t-statistic and repeats the sqrt(250) error from §5.2. The standard error of a difference in Sharpe ratios depends on the return moments and the overlap of the samples, so none of the 'statistically significant at a 5 percent confidence level' claims in this section are supported.
  3. [§5.1–§5.2] The central comparison is confounded: FinMarBaBERT is trained on roughly 61,000 Bloomberg Market Wrap headlines, while FinBERT is trained on about 5,000 Finnish-Phrasebank sentences. The two training sets differ simultaneously in size, text genre, geographical focus, class priors, and label semantics, so the claim in §5.1 that 'the only difference lies in the dataset used for training' is inaccurate. The observed Sharpe gap (0.30 vs. -0.13) cannot be attributed to the labeling method rather than to these other dimensions. A controlled experiment that varies only the label source—for example, market labels versus human labels on the same corpus—is needed to support the paper's central claim.
  4. [§4, Table 2 and Eq. (1)] The label distribution in Table 2 (42.11% positive, 31.43% negative, 26.45% indecisive) is not independent evidence of a market positive bias. With thresholds Q0.3 and Q0.6 in Eq. (1), the expected proportions under iid returns are approximately 40% positive, 30% negative, and 30% neutral; the observed values are close to these mechanically induced rates. Thus, the claim that FinMarBa 'reflects the long-term positive bias of equity markets' is largely circular rather than a validation of the labeling method.
  5. [§3.2, step 2] The labeling assumes that the return on the day after a headline's publication is caused by that headline. Since Bloomberg Market Wraps are daily summaries published after the market close, the next day's move may be driven by entirely different news. The paper itself acknowledges in §3.2 that 'the hypothesis whereby headlines are entirely responsible for market movements cannot be completely dismissed.' A concrete test—for example, a placebo experiment with randomly shifted headline dates or with headlines matched to unrelated tickers—is needed before the labels can be taken as measuring the market's reaction to the specific headline.
minor comments (6)
  1. [Abstract (front matter)] The abstract at the very beginning of the document describes a hierarchical reinforcement-learning framework for portfolio optimization, which is unrelated to the FinMarBa dataset described in the rest of the paper; this appears to be a copy-paste error and must be corrected before resubmission.
  2. [Table 4] Reporting a p-value of exactly 0 is not meaningful; the paper should report the computed statistic and a proper p-value, and should correct the t-statistic formula as noted above.
  3. [Table 1] There are typos in the example rows, such as 'V AIAS' in the introduction and inconsistent formatting in Table 1; these should be cleaned up.
  4. [§5.2] The paper does not state whether the Sharpe ratios in Table 4 are annualized or daily, which is essential for interpreting the t-statistic and the backtest result; the frequency should be stated explicitly.
  5. [§4, Figure 5 vs. Figure 6] The word-frequency comparison would be more informative with a quantitative measure of regional bias rather than only top-word lists, since the lists alone do not establish the claimed coverage difference.
  6. [References] Several references are incomplete or contain formatting errors (e.g., 'Maloet al., 2013' in the introduction, and the missing page numbers in some entries); a careful reference pass is needed.

Circularity Check

1 steps flagged · score 4.0 of 10

Label-distribution validation is self-definitional; central backtest is independent but confounded.

  1. self definitional [Section 4, 'The Annotation Consistency', using Eq. (1) thresholds from Section 3.2]
    "given that all major equity markets have a slight long-term positive bias, we also expect positive headlines to be slightly more represented. Table 2 is consistent with the expected behavior for both on the data set, producing a fairly balanced classification. The main difference is that the FinMarBa dataset reflects this positive market bias, while the Financial-Phrasebank dataset does not. Thus, the distribution of labels in the FinMarBa dataset is more consistent with actual market behavior."

    Eq. (1) defines +1 as ΔP > Q0.6 and −1 as ΔP < Q0.3, so for any stationary return distribution the expected shares are about 40% positive, 30% negative, 30% indecisive. The observed 42/31/26 split in Table 2 is therefore a direct consequence of the chosen quantile boundaries, not an independent observation of market behavior. The paper presents this split as evidence that FinMarBa 'reflects this positive market bias' and reduces indecisive news, but that 'prediction' is baked into the labeling rule itself. It cannot validate the annotation method.

full rationale

The one clear circular step is the validation of the FinMarBa label distribution: the quantile thresholds in Eq. (1) mechanically produce roughly 40% positive, 30% negative, and 30% indecisive labels, so the observed distribution in Table 2 cannot serve as independent evidence of market-aligned sentiment. The central Sharpe-ratio comparison is not circular in the reduction-by-construction sense: FinMarBaBERT is trained on market-derived labels and then evaluated out-of-sample, so the result is an empirical outcome rather than a logical tautology. However, the comparison is confounded because the two models differ simultaneously in training-set size, text domain, regional focus, and label semantics, which undermines the attribution of the Sharpe gap to the labeling method alone. The self-citations to Lefort et al. (2024a) are not load-bearing here because the daily score formula is given in the paper and the macro-impact assumption is stated explicitly. Overall, the circularity is partial and confined to a supporting statistical claim, not to the entire derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions about the reliability of GPT-4 and the causal link between headlines and next-day returns. The quantile thresholds are free parameters that determine the label distribution, and the equal-impact assumption is ad hoc. No new physical or conceptual entities are introduced.

free parameters (3)
  • quantile thresholds Q0.3, Q0.6 = 0.3, 0.6
    Chosen by hand to define positive/negative/neutral labels in Eq. 1; the expected label distribution is 40/30/30, which drives the claimed market-positive-bias result.
  • return horizon = 1 day
    The label uses the closing price change from day t to t+1; the choice of one day is not derived.
  • rolling window length = 5 years
    Historical percentile window chosen to capture a substantial portion of return distributions; not justified quantitatively.
assumptions (5)
  • domain assumption GPT-4-generated headlines faithfully summarize Bloomberg Market Wraps and are market-relevant.
    Headline generation in Section 3.1 uses a GPT-4 prompt with no validation of output quality.
  • domain assumption GPT-4 correctly identifies all relevant tickers for each headline.
    Ticker identification in Section 3.2 relies on GPT-4; no accuracy evaluation is provided.
  • domain assumption The next-day return of the associated ticker is caused by the headline (causal interpretation).
    Section 3.2 step 2 labels based on next-day return; the paper acknowledges the causal hypothesis cannot be dismissed.
  • ad hoc to paper Equal impact of each headline on all associated tickers.
    Section 3.2 states the headline is considered to have equal impact on all tickers, which affects labeling when tickers move oppositely.
  • domain assumption Bloomberg Market Wraps provide a comprehensive, unbiased sample of market-moving news.
    Used in Section 3; the paper's limitations section acknowledges source biases may propagate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FinMarBa: A Market-Informed Dataset for Financial Sentiment Classification." pith.science (2026). https://pith.science/paper/QI6ES6NV

@misc{pith2026250722932,
  author       = {Pith},
  title        = {Pith review of: FinMarBa: A Market-Informed Dataset for Financial Sentiment Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QI6ES6NV}},
  note         = {Machine review of arXiv:2507.22932}
}
read the original abstract

This paper presents a novel hierarchical framework for portfolio optimization, integrating lightweight Large Language Models (LLMs) with Deep Reinforcement Learning (DRL) to combine sentiment signals from financial news with traditional market indicators. Our three-tier architecture employs base RL agents to process hybrid data, meta-agents to aggregate their decisions, and a super-agent to merge decisions based on market data and sentiment analysis. Evaluated on data from 2018 to 2024, after training on 2000-2017, the framework achieves a 26% annualized return and a Sharpe ratio of 1.2, outperforming equal-weighted and S&P 500 benchmarks. Key contributions include scalable cross-modal integration, a hierarchical RL structure for enhanced stability, and open-source reproducibility.

Figures

Figures reproduced from arXiv: 2507.22932 by the authors.

Figure 1
Figure 1. Distribution of the main tickers by region. The sum of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Automatic Classification of Financial Headlines. Blue [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Time ordering of the automatic labelling steps. The head [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Full Process of Dataset Annotation. Sentiment Financial-Phrasebank (%) FinMarBar (%) Positive 28.13 42.11 Negative 12.46 31.43 Indecisive 59.41 26.45 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Top 10 words in Financial-Phrasebank [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Top 10 words in FinMarBa Regional Coverage Comparison To construct a predictive macro signal for equity markets, headlines should predomi￾nantly cover the U.S., the largest and most influential market, while maintaining balanced coverage of other regions [PITH_FULL_IM…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Frontier Financial Judgement: Can agents tell what might move a stock?

    cs.CL 2026-07 conditional novelty 6.0 of 10

    No evaluated AI agent can fully match professional analysts' newness/importance/direction labels on the new 82-case Frontier Financial Judgement benchmark; GPT-5.5 tops out at 52.4%.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Finbert: Financial sentiment analysis with pre-trained language models, 2019

    Dogu Araci. Finbert: Financial sentiment analysis with pre-trained language models, 2019

    work page 2019

  4. [4]

    Sentiment analysis for stock price prediction

    Rubi Gupta and Min Chen. Sentiment analysis for stock price prediction. In 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) , pages 213--218, 2020

    work page 2020

  5. [5]

    Twitter sentiments dataset

    Sherif Hussein. Twitter sentiments dataset. Mendeley Data , 2021

    work page 2021

  6. [6]

    Evaluating sentiment analysis in the context of securities trading

    Siavash Kazemian, Shunan Zhao, and Gerald Penn. Evaluating sentiment analysis in the context of securities trading. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2094--2103, Berlin, Germany, August 2016. Association for Computational Linguistics

    work page 2016

  7. [7]

    Can ChatGPT compute trustworthy sentiment scores from Bloomberg Market Wraps? , 2024

    Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, David Saltiel, Beatrice Guez, and Damien Challet. Can ChatGPT compute trustworthy sentiment scores from Bloomberg Market Wraps? , 2024

    work page 2024

  8. [8]

    Stress index strategy enhanced with financial news sentiment analysis for the equity markets, 2024

    Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, David Saltiel, Beatrice Guez, and Thomas Jacquot. Stress index strategy enhanced with financial news sentiment analysis for the equity markets, 2024

    work page 2024

Show all 19 references
  1. [9]

    A deep learning approach with extensive sentiment analysis for quantitative investment

    Wang Li, Chaozhu Hu, and Youxi Luo. A deep learning approach with extensive sentiment analysis for quantitative investment. Electronics , 12(18), 2023

    2023

  2. [10]

    Finsslx: A sentiment analysis model for the financial domain using text simplification

    Macedo Maia, André Freitas, and Siegfried Handschuh. Finsslx: A sentiment analysis model for the financial domain using text simplification. In 2018 IEEE 12th International Conference on Semantic Computing (ICSC) , pages 318--319, 2018

    2018

  3. [11]

    Good debt or bad debt: Detecting semantic orientations in economic texts, 2013

    Pekka Malo, Ankur Sinha, Pyry Takala, Pekka Korhonen, and Jyrki Wallenius. Good debt or bad debt: Detecting semantic orientations in economic texts, 2013

    2013

  4. [12]

    GPTEval : A survey on assessments of ChatGPT and GPT-4 , 2023

    Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. GPTEval : A survey on assessments of ChatGPT and GPT-4 , 2023

    2023

  5. [13]

    Multilingual twitter sentiment classification: The role of human annotators

    Igor Mozeti c , Miha Gr c ar, and Jasmina Smailovi \'c . Multilingual twitter sentiment classification: The role of human annotators. PloS one , 11(5):e0155036, 2016

    2016

  6. [14]

    Twitter sentiment classification using machine learning techniques for stock markets

    Mohammed Qasem, Ruppa Thulasiram, and Parimala Thulasiram. Twitter sentiment classification using machine learning techniques for stock markets. In 2015 International Conference on Advances in Computing, Communications and Informatics (ICACCI) , pages 834--840, 2015

    2015

  7. [15]

    a mper and Frederik Simon B \

    Sergej Schultenk \"a mper and Frederik Simon B \"a umer. Structured knowledge extraction for digital twins: Leveraging LLMs to analyze tweets. In Frank Phillipson, Gerald Eichler, Christian Erfurth, and G \"u nter Fahrnberger, editors, Innovations for Community Services , page...

    2024

  8. [16]

    Generating insights about financial asks from Reddit posts and user interactions

    Sachin Thukral, Suyash Sangwan, Vipul Chauhan, Arnab Chatterjee, and Lipika Dey. Generating insights about financial asks from Reddit posts and user interactions. In Proceedings of the International Conference on Advances in Social Networks Analysis and Mining , ASONAM ’23. AC...

    2023

  9. [17]

    Bloomberggpt: A large language model for finance, 2023

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance, 2023

    2023

  10. [18]

    Investlm: A large language model for investment using financial domain instruction tuning, 2023

    Yi Yang, Yixuan Tang, and Kar Yan Tam. Investlm: A large language model for investment using financial domain instruction tuning, 2023

    2023

  11. [19]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.