{"id":"a0582a3a-5eb7-4f3e-9183-34a29b77b721","arxiv_id":"2502.10008","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ChatGPT's monthly good-news ratio from Wall Street Journal headlines predicts S&P 500 excess returns in and out of sample; DeepSeek, BERT, and word lists do not.","lead":"Using ChatGPT-3.5 and DeepSeek to classify 84,535 Wall Street Journal headlines, the authors find that only ChatGPT's ratio of good news predicts future S&P 500 returns, with out-of-sample gains. The result suggests large language models can extract economically meaningful information from news that investors only slowly incorporate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Out-of-sample R2_OS of 1.17% depends on an unstated standardization of the news ratio; if full-sample moments are used, the recursive forecasts leak future information.","rationale":"The reader's weakest_assumption identifies the recursive out-of-sample standardization as the key unresolved detail, and my independent reading reaches the same conclusion. The paper's central claim is that ChatGPT's good-news ratio forecasts aggregate returns, with the OOS R2_OS being the strongest evidence of real-time economic value. If the predictor is standardized with full-sample moments, the OOS forecast at each date uses information not yet available, and the reported R2_OS and certainty-equivalent gains become overstated. This is a precise, checkable assumption rather than a general objection, and it is not resolved by the paper's indirect look-ahead tests in Section 4.7 because those tests also do not describe the standardization timing. I do not see a more load-bearing concern: the in-sample regressions are standard, the Hodrick standard errors are appropriate for overlapping horizons, and the qualitative comparison across models is informative. The broader GPT training-cutoff concern is real but harder to test decisively; the standardization issue is the sharpest point of attack. Since the reader already made acceptance conditional on clarifying the recursive procedure and releasing code, my verdict is unchanged.","tokens_in":38681,"tokens_out":4069,"duration_ms":42925,"concrete_test":"Obtain or reconstruct the recursive out-of-sample procedure and recompute Table 6 under three standardization rules: (i) raw NRG with no standardization, (ii) expanding-window standardization using only data through month t, and (iii) full-sample standardization. For each rule, re-estimate Equation (4) recursively, compute R2_OS and the Clark-West MSFE-adjusted statistic, and also recompute the CER gains in Table 7. If variants (i) or (ii) yield R2_OS materially below 1.17% or statistically insignificant, the published OOS evidence is an artifact of full-sample standardization. The authors should also state explicitly which standardization was used in the published Table 6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing evidence is the out-of-sample R2_OS of 1.17% in Table 6, which supports the claim that ChatGPT's good-news ratio has real-time predictive content. Section 4.5 states that alpha_t and beta_t are re-estimated recursively using data available at time t, but it never states how the news ratio is standardized before entering Equation (4). Elsewhere in the paper, predictors are explicitly 'standardized to have a zero mean and unit variance' (e.g., Table 2 note and Section 5.3.2). If that standardization is done once with full-sample moments (1996-2022), then every recursive forecast in Equation (4) uses the full-sample mean and variance of NRG, which include future observations. That is a look-ahead leak in the forecast construction itself, independent of ChatGPT's training cutoff. Because R2_OS = 1.17% is modest, even a small shift in the standardized predictor could move the Clark-West statistic below significance, and the same leak would inflate the CER gains in Table 7. The Section 4.7 post-2021 weekly test does not resolve this issue because it also does not specify whether the weekly news ratios are standardized recursively.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether large language models can extract predictive information from Wall Street Journal front-page headlines for the aggregate U.S. stock market and the macroeconomy. Using ChatGPT-3.5 and DeepSeek-R1 to classify headlines from 1996 to 2022 into good, bad, and neutral news, the authors construct monthly good-news and bad-news ratios and test their in-sample and out-of-sample predictive power for S&P 500 excess returns. The central finding is that ChatGPT's good-news ratio is positively correlated with contemporaneous returns and predicts subsequent returns for up to six months, with a one-month in-sample R2 of 1.37% and an out-of-sample R2_OS of 1.17%. The bad-news ratio, DeepSeek, word-list methods, BERT, and RoBERTa do not predict future returns. The paper also connects the predictability to underreaction to good news during economic downturns, high economic policy uncertainty, and novel news, and shows that ChatGPT's news ratios forecast macroeconomic conditions.","tokens_in":38908,"tokens_out":4502,"duration_ms":48007,"significance":"If the central claim holds, the paper makes a valuable contribution by showing that an LLM-based news measure contains real-time information about the equity risk premium that traditional predictors and dictionary methods miss. The empirical design has clear strengths: the regression framework is standard, Hodrick (1992) standard errors are used for overlapping horizons, the out-of-sample evaluation uses Campbell and Thompson (2008) and Clark and West (2007), and the paper includes multiple robustness checks with alternative prompts, fine-tuned models, and ChatGPT-4. The economic-mechanism tests in Section 5 are thoughtful and falsifiable. The main quantitative results, however, are modest in magnitude and depend on the details of the out-of-sample protocol, especially the standardization of the predictor and the training-data cutoff of the language model. Because the paper does not report code or machine-checked proofs, its contribution rests on the credibility of the empirical procedure; that procedure is currently not fully specified in two load-bearing respects.","major_comments":[{"comment":"","section":"Section 4.5, Table 6"},{"comment":"","section":"Section 4.7"},{"comment":"","section":"Tables 2-5"}],"minor_comments":[{"comment":"","section":"Section 4.5, Equation (4)"},{"comment":"","section":"Section 3.5, Figure 1 discussion"},{"comment":"","section":"Section 4.5, Equations (5) and (6)"},{"comment":"","section":"Throughout"},{"comment":"","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of an economics/finance journal and addresses a timely question. The main concern is not novelty but the rigor of the out-of-sample evaluation: the recursive standardization issue and the training-data look-ahead issue are both load-bearing for the headline R2_OS of 1.17%. I recommend requesting a revision that specifies and fixes the standardization protocol and provides a direct look-ahead test. I did not identify any citation or ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's contribution is real: it applies ChatGPT-style news classification to aggregate stock market prediction, adds a DeepSeek comparison, and connects predictability to macro fundamentals. That combination is new in the equity premium literature. The authors also do better than most LLM-finance preprints at addressing the obvious GPT training-cutoff concern, with three indirect tests in Section 4.7. I came away thinking the in-sample predictability is probably genuine, and the mechanism stories (underreaction to good news, stronger in downturns and high uncertainty) are plausible.\n\nThe soft spot is exactly where the reader's report puts it. The out-of-sample procedure re-estimates alpha and beta recursively, but never says how the news ratio is standardized before entering the forecast equation. Since the paper standardizes predictors elsewhere (zero mean, unit variance), the missing detail is load-bearing. If the standardization uses full-sample moments, every forecast from 2006 onward uses future data, and with an R2_OS of only 1.17%, that could easily push the Clark-West statistic below significance. The Section 4.7 weekly test has the same ambiguity, so it doesn't resolve the issue. This is fixable with a clear sentence and a robustness table using recursive standardization, but without that, the main out-of-sample claim is not yet established.\n\nI'm less worried about the multiple-horizon and multiple-prompt searching than the reader is. The alternative prompts and fine-tuning results are consistent, and the horizon pattern (predictability up to six months) has an economic interpretation. The absence of code and data makes replication hard, but that's not a fatal flaw for a finance paper.\n\nOverall, the paper is honest, the writing is direct, and the empirical work is extensive. I'd send it to a serious referee. The editor should ask for a precise description of the recursive standardization, a version of the out-of-sample results using only expanding-window moments, and ideally a code release for the news ratios. If the OOS result survives that, it's a solid contribution.","headline":"Original and careful empirical work, but the headline out-of-sample result has an unresolved standardization detail that could be a look-ahead leak; the paper deserves refereeing, not desk rejection.","tokens_in":39469,"tokens_out":1700,"would_cite":true,"duration_ms":20374,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChatGPT's good-news ratio, the fraction of Wall Street Journal headlines it labels positive, predicts up to six months of future stock-market returns, with an out-of-sample R2 of 1.17%.","keywords":["large language models","ChatGPT","stock market predictability","news sentiment","good news ratio","out-of-sample forecasting","macroeconomic forecasting","DeepSeek"],"falsifier":"Recompute the out-of-sample $R^2_{OS}$ using only expanding-window standardization of the good-news ratio, or raw counts, and compare with the reported 1.17%; if it collapses, the reported gain comes from full-sample normalization rather than real-time forecasting. Then rerun the 1996-2005 in-sample regressions with a language model whose training data ends before 1996; if predictability disappears in the earlier window and survives only after 2006, look-ahead bias rather than underreaction is the cause.","tokens_in":38442,"feed_emoji":"📈","tokens_out":9525,"duration_ms":76306,"temperature":0.7,"pith_summary":"The paper asks whether a large language model can read Wall Street Journal headlines and forecast the aggregate U.S. stock market. It instructs ChatGPT-3.5 to label each headline as good, bad, or neutral for stocks and computes the monthly ratio of good news to all headlines. The central claim is that this good-news ratio is positively correlated with contemporaneous market returns and significantly predicts returns over the next one to six months, with a one-month-ahead in-sample $R^2$ of 1.37% and an out-of-sample $R^2_{OS}$ of 1.17% from 2006 to 2022. Bad news moves prices immediately but does not forecast future returns, and the predictability is stronger in downturns, high economic-policy uncertainty, and novel news. If right, the result implies that ChatGPT extracts macro-relevant information that investors are slow to incorporate, and that the same information has economic value in asset allocation.","feed_headline":"ChatGPT's good-news ratio predicts six months of stock returns","feed_subtitle":"Investors appear to underreact to positive Wall Street Journal news, giving the signal real trading value.","key_machinery":"The central object is the ChatGPT good-news ratio, NRG, computed each month as the share of Wall Street Journal front-page headlines and alerts that GPT-3.5 labels GOING UP under a fixed zero-shot prompt. The machinery is the predictive regression of future average market excess returns on NRG, with overlapping-horizon standard errors for in-sample inference and the standard out-of-sample $R^2_{OS}$ test against the historical average. The ratio carries the argument because, in the paper's account, the large model captures context-sensitive meaning in headlines—words like 'bounce' and 'buoy'—that word lists and smaller models miss.","core_discovery":"Using front-page headlines and alerts from the Wall Street Journal for January 1996 to December 2022, the paper asks GPT-3.5, with a single zero-shot prompt, to label each headline as GOING UP, GOING DOWN, or UNKNOWN for U.S. stock prices. The monthly good-news ratio NRG is the fraction of headlines labeled GOING UP. The paper reports that a one-standard-deviation increase in NRG is followed by 0.53% higher average excess market return over the next month, and the predictive regression $R^2$ rises from 1.37% at the one-month horizon to 8.52% at the twelve-month horizon. Recursive out-of-sample forecasts from January 2006 to December 2022 produce $R^2_{OS} = 1.17\\%$ against the historical-average benchmark, and a mean-variance investor with risk aversion 3 earns a certainty-equivalent gain of 4.92% per year (3.55% after 50 basis point transaction costs). The paper interprets the asymmetry—good news predicts, bad news does not—as investors' underreaction to good news, consistent with ambiguity aversion and limited attention, and supports it with interaction tests showing stronger predictability in downturns, high economic-policy uncertainty, and novel news.","pith_inferences":["If the underreaction story is right, the signal should decay as LLM-based sentiment tools become standard investment inputs; the paper's own cumulative forecast-error plot already appears to flatten after 2021, consistent with a publication effect.","The DeepSeek comparison raises a testable model-specific question: a replication with other large English-trained models should show whether the predictive edge is about English-language training intensity or about something specific to the GPT architecture.","The macro results imply a sharper, untested prediction: the good-news ratio should lead not just the equity premium but also real activity indicators such as payroll growth and industrial production at horizons beyond one month, and that lead should survive controls for lagged macro announcements."],"forward_implications":["If the central claim is right, the ChatGPT good-news ratio joins the short list of predictors that beat the historical average out of sample, and it does so over horizons investors care about: one to six months.","The absence of predictive power in the bad-news ratio implies that the market's fast reaction to bad news is not an artifact of the model; the asymmetry itself is the economic content.","Because predictability concentrates in downturns, high-policy-uncertainty periods, and novel news, a real-time trading rule would tilt toward good-news signals in exactly those states rather than applying the ratio uniformly.","The estimated economic value—4.92% annual certainty-equivalent gain at risk aversion 3, still 3.55% after 50 basis point transaction costs—means a mean-variance investor should be willing to pay a substantial fee for the forecast."],"supporting_citations":[{"why":"First study using ChatGPT to predict stock returns; this paper extends the setting from individual stocks to the aggregate market.","marker":"Lopez-Lira and Tang (2023)"},{"why":"Provides the 14 economic predictor variables and the historical-average benchmark that the good-news ratio must beat out of sample.","marker":"Welch and Goyal (2008)"},{"why":"Supplies the out-of-sample $R^2_{OS}$ statistic and the 0.5% monthly threshold for economic significance, plus the certainty-equivalent return framework.","marker":"Campbell and Thompson (2008)"},{"why":"The standard financial word-list method used as the baseline alternative textual analysis that fails to predict market returns.","marker":"Loughran and McDonald (2011)"},{"why":"Theoretical mechanism for why ambiguity-averse investors underreact to good news, which the paper uses to explain the predictive asymmetry.","marker":"Epstein and Schneider (2008)"},{"why":"Provides the economic policy uncertainty index used in the interaction test showing stronger predictability under high uncertainty.","marker":"Baker, Bloom, and Davis (2016)"},{"why":"Theoretical basis for stronger underreaction to good news in bad economic times, tested with the CFNAI interaction.","marker":"Veronesi (1999)"},{"why":"Provides the MSFE-adjusted test used to assess whether the out-of-sample $R^2_{OS}$ is statistically significant.","marker":"Clark and West (2007)"}],"fun_headline_variants":["ChatGPT's good-news ratio forecasts stock returns; DeepSeek lags","Good news from ChatGPT predicts market, bad news does not","ChatGPT news signal yields 4.9% annual extra return","Underreaction to good news makes ChatGPT a market predictor"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the out-of-sample forecasts use only information available at the forecast date, and that a model trained through September 2021 can classify 1996-2005 headlines without encoding later market outcomes into its labels.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT's good-news ratio forecasts stock returns; DeepSeek lags","Good news from ChatGPT predicts market, bad news does not","ChatGPT news signal yields 4.9% annual extra return","Underreaction to good news makes ChatGPT a market predictor"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3267,"prompt_tokens":910,"completion_tokens":2357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":2283}},"tokens_in":526,"tokens_out":2357,"duration_ms":15581,"temperature":1.0,"reasoning_tokens":2283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:42:24.433653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the out-of-sample $R^2_{OS}$ using only expanding-window standardization of the good-news ratio, or raw counts, and compare with the reported 1.17%; if it collapses, the reported gain comes from full-sample normalization rather than real-time forecasting. Then rerun the 1996-2005 in-sample regressions with a language model whose training data ends before 1996; if predictability disappears in the earlier window and survives only after 2006, look-ahead bias rather than underreaction is the cause.","supporting_citations":[],"review_version":1}