{"id":"51135b29-cfaf-4269-828b-a8cbaa96f506","arxiv_id":"2505.16090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs outperform traditional sentiment tools on financial phrase classification, and business-line sentiment from Microsoft earnings calls shows patterns that overall tone misses.","lead":"This study tests how well ChatGPT, Gemini, Copilot, and traditional NLP tools judge sentiment in financial text, then applies the best tool to Microsoft earnings calls. It finds LLMs beat older tools on nuance and that sentiment split by business line reveals non-obvious links to stock moves.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark comparisons rest on 250-row point estimates with no confidence intervals or significance tests; the headline claim of clear LLM superiority is not yet statistically supported.","rationale":"The reader's weakest assumption was that the business-line segmentation is unvalidated, and that is indeed a real fragility in the case study. My stress-test identifies an even more foundational vulnerability: the benchmark claim that motivates the whole paper is not statistically supported by the reported evidence. The 250-row point estimates in Figures 1 and 2 have no confidence intervals, no repeated runs, no hypothesis tests, and no prompt transparency, so the reported ordering of models could easily be sampling noise. This matters because the Section 2 conclusion is the paper's headline result and the stated basis for recommending LLM-based sentiment tools. The Section 3 segmentation concern is also important but is secondary: even if the segmentation were perfect, the benchmark claim would still lack statistical grounding. I therefore partially agree with the reader. The conditional verdict remains appropriate: the direction of the claim is plausible and the authors are appropriately hedged, but the paper needs prompt artifacts, a larger or stratified evaluation, significance testing, and segmentation validation before the central claims can be accepted as reliable. No change to the reader's verdict is needed.","tokens_in":4280,"tokens_out":3432,"duration_ms":29844,"concrete_test":"Obtain the exact prompts and model settings used for Figure 1, then rerun the benchmark on a larger stratified sample (e.g., 500 sentences) with temperature 0 and three independent trials per model; compute 95% Wilson confidence intervals and a McNemar test for each pairwise comparison. If the Copilot-App advantage over ChatGPT-4o or FinBERT is not significant at p<0.05, Section 2's 'clearly outperformed' claim must be downgraded. Separately, have two independent annotators verify Copilot's business-line segmentation on two quarters of transcripts against Microsoft's reported segment definitions and report Cohen's kappa; a kappa below 0.8 would invalidate the segment-level sentiment and SHAP conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 2 ('LLMs clearly outperformed traditional tools in financial sentiment analysis, especially in detecting filler words and subtle cues') rests on Figure 1: accuracy point estimates on the 'First 250 Rows' of the Financial Phrase Bank. The paper reports no confidence intervals, no repeated trials, no significance tests, and no disclosure of the exact prompts or model settings. With n=250, the standard error of an 82.0% accuracy is about 2.4 percentage points; the reported lead over ChatGPT-4o (77.6%) is less than two standard errors, and the gap over FinBERT/NLTK is similarly within plausible sampling noise. If the 250-row subset is not stratified or representative, the comparison is even less reliable. The business-line case study in Section 3 is a separate fragile step: Copilot's segmentation of transcripts into business lines is unvalidated, and the SHAP beeswarm (Figure 4) is computed from sentiment values derived from that segmentation without evidence that segment assignment is accurate or that the SHAP model is stable. If segments are misassigned, the 'Search and News Advertising inversion' and 'Devices spike' findings are not reliable. The paper's own hedging in the conclusion does not repair missing measurement error and validation artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a benchmark of nine sentiment-analysis tools (Microsoft Copilot variants, ChatGPT-4o, Gemini, Azure, FinBERT, NLTK, TextBlob) on the first 250 rows of the Financial Phrase Bank, and a case study applying ChatGPT-4o to Microsoft earnings-call transcripts segmented by business line using Microsoft Copilot. The authors claim that LLMs, especially the Copilot App, clearly outperformed traditional tools in detecting nuanced financial sentiment, and that segment-level sentiment, visualized with a SHAP beeswarm plot, reveals business-line-specific relationships with stock price movements. The manuscript is written as a project report rather than a full research paper, and its central claims rest on point-estimate accuracy figures and an exploratory SHAP analysis without reported statistical support or validation of the transcript-segmentation step.","tokens_in":4629,"tokens_out":3035,"duration_ms":28261,"significance":"If the benchmark claims were statistically supported, the paper would offer practically useful guidance for practitioners selecting sentiment tools for financial text: it directly compares several widely used cloud LLM interfaces and Python libraries on a standard dataset, and it attempts to extend sentiment analysis from whole-transcript tone to business-line-level tone. The real-world case study, despite being exploratory, illustrates a plausible workflow that could be valuable if validated. The paper also gives credit for documenting concrete failure modes of LLM deployment (CSV handling, fallback to TextBlob, hallucinations) and for explicit hedging in the conclusion that stock movement prediction remains out of reach. However, the current evidence is insufficient to establish the headline claims, and the lack of statistical inference and segmentation validation limits the paper's contribution to an industry anecdote rather than a reliable research result.","major_comments":[{"comment":"The claim that LLMs 'significantly outperformed' traditional tools is not supported by the reported statistics. The benchmark reports only point accuracies on 250 sentences, with no confidence intervals, significance tests, or repeated trials. For n=250, the standard error of an 82.0% accuracy is about 2.4 percentage points, so the reported 4.4-point lead over ChatGPT-4o is within roughly two standard errors, and the differences against other tools are similarly plausible under sampling noise. The authors should provide per-model standard errors or exact binomial confidence intervals, a formal significance test that accounts for multiple comparisons, and a description of how the 250-row subset was selected.","section":"Section 2, Figure 1"},{"comment":"The business-line analysis depends on Microsoft Copilot's segmentation of earnings-call transcripts into segments (Devices, Dynamics, Gaming, etc.), but the manuscript reports no validation of this segmentation. If sentences are misassigned to business lines, the segment-level sentiment scores and the subsequent SHAP-based interpretation would be unreliable. The authors should validate the segmentation on a random sample against human annotation, or at least report inter-coder or model-consistency metrics, and describe the exact instructions given to Copilot for this task.","section":"Section 3, transcript segmentation"},{"comment":"The SHAP beeswarm plot is not reproducible and its interpretation is not statistically grounded. The manuscript does not specify the predictive model used to map sentiment values to stock price changes, the number of data points (segments by quarter), the feature set, or any train/test or stability analysis. With a small number of quarters and multiple segments, SHAP values are known to be unstable, and the claim that 'positive tone in Search and News Advertising is associated with stock decline' is a post-hoc reading of a few anecdotal quarters rather than a tested hypothesis. The authors should report the model formulation, sample size, and a permutation or bootstrap-based stability check, and explicitly label these findings as exploratory.","section":"Section 3, Figure 4 (SHAP analysis)"},{"comment":"The summary bullet 'LLMs clearly outperformed traditional tools in financial sentiment analysis' overstates the evidence. The benchmark is a single, unduplicated run on 250 items, and the paper's own Figure 1 shows a range of LLM accuracies that overlaps once realistic uncertainty is considered. Without significance tests or repeated trials, the correct claim is that the LLM tools achieved higher point estimates in this particular run. This language should be softened to match the statistical support.","section":"Section 4, bullet list"}],"minor_comments":[{"comment":"The exact prompts, model settings, temperature parameters, and versions of each LLM are not disclosed, which limits reproducibility; provide a supplementary appendix with the full prompt and configuration used for each model.","section":"Section 2"},{"comment":"The paper refers to a 'Condensed Sentiment Accuracy Comparison' but does not explain which models are included or what 'condensed' means; please clarify the subset of models and the metric depicted.","section":"Section 2, Figure 2"},{"comment":"The sentence 'FinBERT was particularly effective for finance-specific cases' is vague and potentially in tension with the earlier statement that LLMs outperformed traditional tools; please specify the quantitative basis for this claim.","section":"Section 2, p. 3"},{"comment":"The term 'filler words' is used to describe what are apparently hedged or nuanced expressions; consider using more precise terminology such as 'hedging phrases' or 'qualifying language' to avoid confusion.","section":"Section 2 and 4"},{"comment":"Reference [2] points to a Kaggle dataset rather than the original Financial Phrase Bank paper; please cite the original Aalto University publication and report the exact version of the dataset used.","section":"References"},{"comment":"The figures should include axis labels, units, and a caption describing the number of quarters, the source of the transcripts, and the exact stock-price return window; currently the reader must infer these details from the text.","section":"Section 3, Figure 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short industry-practicum report with a useful empirical setup but insufficient statistical rigor for a research venue. The core comparison on 250 rows is presented as decisive, yet a simple standard-error calculation shows the headline gap is weak; the business-line finding is also built on unvalidated segmentation and an undocumented SHAP model. These issues are fixable in a revision if the authors add confidence intervals/significance tests, disclose prompts and settings, validate the segmentation, and reframe the SHAP analysis as exploratory. The paper's contribution is more of a technical report than a scientific article; the editor may wish to consider whether the target journal's criteria for a full paper are met even after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this as an honest industry practicum report, not a research paper, and it is most useful when read that way. What is genuinely new here is narrow but real: the discovery that Copilot 365 silently routes sentiment analysis to TextBlob, the accuracy numbers for Copilot variants on a standardized financial dataset, and the business-line segmentation of Microsoft earnings calls. The TextBlob fallback is the kind of concrete, reproducible observation that practitioners will actually care about, and I give them credit for flagging it instead of burying it.\n\nThe paper does some things well. It is readable, the benchmark design is simple, and the final section properly hedges the stock-movement claims and says LLMs are not replacements for experts. The exploratory framing of the SHAP analysis is appropriate on the surface.\n\nNow the soft spots, in proportion. The central claim in Section 2 that \"LLMs clearly outperformed traditional tools\" is not statistically supported. The headline numbers come from the first 250 rows of one dataset, with no confidence intervals, no significance tests, and no disclosure of the exact prompts or model settings. The gap between the Copilot App at 82.0% and ChatGPT-4o at 77.6% is within plausible sampling noise, and even the lead over FinBERT is not obviously outside it. I am not saying the direction is wrong; I am saying the evidence is thinner than the word \"clearly\" allows. The case study has a separate fragile step: Copilot's segmentation of transcripts by business line is unvalidated, and the SHAP beeswarm is computed from sentiment values that inherit any segmentation error. The \"Search and News Advertising inversion\" and the \"Devices spike\" are interesting hypotheses, but they are post-hoc stories on a handful of quarters, not findings. The authors' own conclusion is appropriately tentative, which helps, but the abstract and the early sections push the claims further than the methods support.\n\nThe citation pattern is thin for a benchmark paper: two references, one being a Kaggle dataset, with no comparison to prior work on LLM financial sentiment or to earlier findings on business-line tone. That is a minor issue if the paper is positioned as a practical report, but it would matter in a peer-reviewed venue.\n\nWho is this for? Practitioners choosing between Copilot variants and open libraries for financial sentiment work, and people studying how LLM-based tools actually behave in production settings. It does not deserve a hard desk reject—the TextBlob fallback alone justifies a referee’s time—but it needs a serious revision to add statistical rigor, release the prompts and data subset, and validate the segmentation before the business-line conclusions can be trusted. I would not cite it in my own work as evidence for a claim beyond the TextBlob observation, and I would not bring it to a reading group for its methods, but I would engage with it as a useful field report.","headline":"A useful practitioner benchmark with one genuinely interesting finding (Copilot 365 silently falls back to TextBlob), but the stats are too thin to support \"clearly outperformed\" and the earnings-call segmentation is unvalidated.","tokens_in":5023,"tokens_out":1432,"would_cite":false,"duration_ms":13682,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims LLMs outperform traditional sentiment tools on financial text, and that segment-level earnings-call tone reveals stock-moving patterns that overall tone misses.","keywords":["financial sentiment analysis","large language models","earnings call transcripts","sentiment benchmarking","business-line segmentation","SHAP values","prompt engineering","stock price prediction"],"falsifier":"Run the nine benchmark systems on a fresh, expert-labeled corpus of earnings-call sentences: if standard lexicon tools match or beat the LLMs' accuracy, the performance gap collapses. Separately, redo the segment-level SHAP analysis using human-verified business segments: if the Search and News Advertising inversion and the Devices spike no longer align with stock moves, the business-line insight is an artifact of the segmentation.","tokens_in":4091,"feed_emoji":"📈","tokens_out":8732,"duration_ms":68775,"temperature":0.7,"pith_summary":"The paper asks whether large language models can interpret the hedged, forward-looking language of earnings calls well enough to be useful in finance. It answers by benchmarking nine tools on a standardized financial phrase dataset, finding that LLM-based systems, led by the Copilot App at 82.0% accuracy, outperform lexicon-based libraries, which lose signal when text is aggressively cleaned. It then applies ChatGPT-4o to quarterly earnings call transcripts segmented by business line and argues that sentiment patterns inside specific segments align with stock moves in ways overall transcript tone does not. If correct, practitioners should prefer LLM-based sentiment tools over traditional libraries and should segment earnings calls by business line to find market-relevant signals.","feed_headline":"LLMs beat lexicon tools on financial tone; segment view shows stock clues","feed_subtitle":"Copilot hit 82 percent accuracy and business-line sentiment lined up with post-call stock moves.","key_machinery":"The analysis is carried by three linked components: a standardized financial sentiment benchmark used as the accuracy yardstick, a segmentation step that splits each quarter's earnings call transcript by business line before sentiment scoring, and SHAP values that map each segment's sentiment to its contribution to a stock-price-change prediction. The benchmark establishes the model rankings, the segmentation is what exposes per-segment sentiment signals that aggregate tones obscure, and SHAP provides the quantitative link between segment tone and market response. The paper also relies on identical prompts across LLMs and on prompt engineering to keep the comparison fair.","core_discovery":"Across all benchmark comparisons, the paper's central finding is that LLM-based tools read financial tone more accurately than traditional lexicon-based libraries. On the Financial Phrase Bank, the Copilot App (both online and local) reached 82.0% accuracy, ChatGPT-4o reached 77.6%, prompt-engineered ChatGPT 75.6%, and Gemini 2.0 Flash 68.0%, while NLTK, TextBlob, and a Copilot 365 configuration that silently fell back to TextBlob scored lower. The paper attributes the gap to text cleaning: traditional models strip away filler and hedged words that carry tone, while LLMs use that context. A second finding comes from the earnings-call case study: after segmenting transcripts by business line and scoring each segment with ChatGPT-4o, sentiment in specific segments such as Search and News Advertising and Devices aligned with post-call stock moves, while overall transcript sentiment did not. The paper interprets the SHAP beeswarm plot as evidence that positive tone in some segments can coincide with falling prices, so tone should not be read as a simple stock predictor.","pith_inferences":["If segment-level sentiment is genuinely predictive, the same segmented design could be applied to earnings calls of other large firms, and firm-specific tell segments might emerge that general sentiment models would miss.","The observed inversion (positive tone, falling stock) invites a testable explanation: upbeat sentiment in a segment may signal that good news is already priced in, and conditioning segment sentiment on prior analyst expectations could confirm this.","Because the business-line segmentation was not independently validated, an audit that re-segments a few quarters by human labelers and re-runs the SHAP analysis would show whether the stock correlations hold up.","Combining LLM sentiment with finance-specific tools such as FinBERT could push past the 85% ceiling, since the paper's own data show different error patterns across model families."],"forward_implications":["Practitioners should prefer LLM-based sentiment tools over lexicon libraries for hedged or forward-looking financial text, but should expect accuracy below 85% and plan for verification.","Analysts should segment earnings call transcripts by business line before scoring sentiment, because aggregate tone can miss or even invert the market signal.","A positive tone in a specific segment is not reliably a positive stock signal; context and expectations can make upbeat language read as over-optimism, as with Search and News Advertising.","Deployment choices matter: Copilot 365's fallback to a simpler sentiment engine and CSV-handling issues can silently lower quality, so users need to know which engine is actually running.","Human domain expertise remains necessary for framing questions, choosing analyses, and checking outputs; LLMs support but do not replace the analyst."],"supporting_citations":[{"why":"Supplies the quarterly earnings call transcripts used for the business-line sentiment case study.","marker":"[1]"},{"why":"Provides the Financial Phrase Bank dataset with pre-labeled positive, neutral, and negative sentences used as the benchmark ground truth.","marker":"[2]"}],"fun_headline_variants":["LLMs beat lexicons on financial tone; segment sentiment aligns with stocks","Tone insight: LLMs top lexicons; business-line sentiment tracks stocks","LLM sentiment beats lexicon tools; segments mirror stock moves","AI reads financial nuance: LLMs beat lexicons; line-level tone moves stocks","Copilot leads LLM accuracy on fin tone; segment views hint at stock clues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The business-line findings depend on the automatic splitting of each earnings call transcript into business segments being correct, and the paper does not validate that splitting against human judgment.","fun_headline_variants_meta":{"raw":{"variants":["LLMs beat lexicons on financial tone; segment sentiment aligns with stocks","Tone insight: LLMs top lexicons; business-line sentiment tracks stocks","LLM sentiment beats lexicon tools; segments mirror stock moves","AI reads financial nuance: LLMs beat lexicons; line-level tone moves stocks","Copilot leads LLM accuracy on fin tone; segment views hint at stock clues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001027,"raw_usage":{"total_tokens":4384,"prompt_tokens":1059,"completion_tokens":3325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":3227}},"tokens_in":675,"tokens_out":3325,"duration_ms":20803,"temperature":1.0,"reasoning_tokens":3227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:05:56.875834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the nine benchmark systems on a fresh, expert-labeled corpus of earnings-call sentences: if standard lexicon tools match or beat the LLMs' accuracy, the performance gap collapses. Separately, redo the segment-level SHAP analysis using human-verified business segments: if the Search and News Advertising inversion and the Devices spike no longer align with stock moves, the business-line insight is an artifact of the segmentation.","supporting_citations":[{"cited_title":"Investor relations.https://www.microsoft.com/en-us/investor/default, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the quarterly earnings call transcripts used for the business-line sentiment case study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Financial Phrase Bank dataset with pre-labeled positive, neutral, and negative sentences used as the benchmark ground truth."}],"review_version":1}