REVIEW 5 major objections 5 minor 19 references
Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment
T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A two-stage 'Classify-and-Rethink' prompt can make ChatGPT avoid the framing effect in gold news scoring and generate excess returns in a 2018-2023 backtest.
desk verdict The CAR prompt idea is reasonable but the backtest confounds rethink with window and trade-direction changes, so the excess-return claim isn't supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing design is the Classify-and-Rethink (CAR) prompt. In the Classify leg, ChatGPT assigns each gold news item to one of six economic categories and outputs an initial score between -1 and 1 in 0.1 steps; the categories force the model to reason through an economic channel instead of reacting to wording. In the Rethink leg, the model is instructed to evaluate whether its own score is reasonable from a long-term trend perspective and to revise it, which is intended to un-anchor the score from short-term framing. Trading converts the revised scores into positions: CAR buys when the 20-news average score exceeds a threshold and sells when it falls below, whereas the no-rethink baselines trade on the 5-news average in the opposite direction.
What would settle it
Run a crossed ablation that keeps the averaging window, trade direction, and threshold identical across all strategies and toggles only the rethink instruction; if CAR's excess return over the classify-only strategy disappears, the reported effect is not caused by rethinking.
Extended reading notes
Core claim
The central claim is that the rethink step is what allows ChatGPT to overcome the framing effect. On the paper's account, a one-step prompt makes the model mirror the short-term sentiment of the news—just as a biased investor would—while a classify-only prompt improves the economic reasoning but still overreacts to the latest item. The CAR prompt adds a final instruction to reflect on the initial score from a long-term trend perspective and correct it, producing a score that deliberately moves away from market sentiment. The authors report that this sequence outperforms both simpler prompts and buy-and-hold in return and Sharpe ratio, and that the resulting scores are roughly normally distributed rather than clustered at extremes, which they take as a sign of more rational scoring.
Load-bearing premise
The result depends on attributing the return gap to the rethink step rather than to the simultaneous change in averaging window (from 5 to 20 news items) and trading direction, and it assumes the buy/sell threshold was fixed in advance rather than chosen after inspecting the backtest.
Editorial extensions
If this is right
- ChatGPT can serve as a zero-shot news scorer inside a tradeable strategy without fine-tuning or training labels.
- The classification stage is essential: removing it and applying the rethink prompt directly caused the strategy's profit to fall to 42%, below buy-and-hold.
- The rethink stage makes the score distribution closer to normal, which the authors connect to more rational behavior and easier downstream processing.
- Each scored news item comes with a textual explanation, so the resulting investment signals are auditable by a human.
Reading between the lines
- Our inference: the reported comparison does not isolate the rethink step, because CAR also switches the averaging window from 5 to 20 news items and reverses the trade direction relative to the baselines; a crossed ablation varying one factor at a time would be needed to confirm the mechanism.
- Our inference: if the rethink prompt is what matters, a similar benefit may appear in other news-driven assets, such as equities or exchange rates, where the framing effect is thought to distort short-term reactions; this is testable with the same CAR prompt and backtest template.
- Our inference: because the rethink step pushes scores toward a near-normal distribution, part of the return gain might be reproducible by any score-averaging procedure that damps outliers, independent of the model's reasoning; comparing CAR against a mechanically smoothed sentiment score would separate those effects.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes Classify-and-Rethink (CAR), a multi-step zero-shot prompt strategy in which ChatGPT first classifies gold-related news into six categories, scores each article on [-1,1], and then reconsiders the score from a long-term perspective. The authors backtest a long/flat trading rule on Shanghai Gold Exchange Au9999 from January 2018 to June 2023 and report that CAR achieves an 80.35% return and a Sharpe ratio of 1.071, outperforming Buy-and-Hold (63.53%, 0.811), One-Step (63.44%, 0.902), and Classify (73.41%, 1.019). They conclude that CAR overcomes the framing effect and generates excess returns. The paper also includes an ablation study, a score-distribution analysis, and a qualitative case study.
Significance. If the causal claim were established, the work would be a useful demonstration that multi-step prompting can reduce framing effects in LLM-based financial news scoring and improve trading performance. The research question is interesting and timely, and the CAR pipeline is clearly described. The paper also contains an explicit discussion of limitations in Section 4 and a declared conflict of interest, both of which are commendable. However, the current evidence does not isolate the effect of the rethink step: the comparison in Table 1 changes the averaging window and the trading direction together with the prompt, the buy/sell threshold is never disclosed, transaction costs are ignored, and no statistical or out-of-sample validation is provided. These are not presentation issues; they bear directly on the central claim. As a result, the paper currently does not support its headline conclusion that the rethink mechanism itself is what overcomes behavioral biases and produces excess returns.
major comments (5)
- [§2.4 and Table 1] The comparison between Classify and Classify+Rethink (CAR) in Table 1 is confounded. Section 2.4 states that the baseline strategies use the average score of the latest five news items and buy when the average is below the threshold and sell when above, whereas CAR uses the average of the latest 20 news items and buys when the average is above the threshold and sells when below. Consequently, the CAR column differs from the Classify column in three ways simultaneously: the presence of the rethink prompt, the averaging window (5 vs 20), and the sign of the trading rule (buy-low/sell-high vs buy-high/sell-low). The observed return gap (73.41% vs 80.35%) cannot therefore be attributed to the rethink mechanism. Please provide a factorial ablation that varies the prompt, the window, and the direction independently, for example Classify with a 20-item window and the same direction rule as CAR, and Rethink with a 5-item window and the same direction rule as the baselines.
- [§2.4] The buy/sell threshold is never disclosed. The text refers to 'the threshold' and 'the set value' without giving its numerical value or how it was chosen. Without this value, the backtest is not reproducible, and it is impossible to rule out that the threshold was selected after inspecting the same 2018–2023 data, which would make the reported returns in-sample rather than predictive. Please report the threshold value, show sensitivity of the Table 1 results to the threshold, and, if the threshold was tuned on the backtest period, state this explicitly and provide an out-of-sample or cross-validation evaluation.
- [§3.2 and Fig. 2] The backtest ignores transaction costs. Figure 2's caption explicitly states 'without considering transaction costs,' and the paper claims 'excess returns' and 'higher investment returns.' With a long/flat strategy that switches on news-score crossings, transaction costs and bid-ask spreads can materially change net performance, especially given the ablation's mention of 30 sell operations over the period. Please include realistic transaction costs (and, if applicable, slippage) in the reported returns, or at least report turnover and a break-even cost analysis.
- [§3.2 and §4] No statistical significance, confidence intervals, or out-of-sample tests are reported. The results are a single historical path over one asset class and one time period, and Section 4 itself acknowledges that 'our experiments are based solely on historical data.' The 80.35% vs 73.41% gap could easily be within sampling variation, especially given that the strategy parameters may have been chosen after observing the same period. Please add bootstrap or Monte Carlo tests (for example, re-sampling the news order or randomizing entry/exit), a split-sample or out-of-sample evaluation, and a multiple-testing awareness for the prompt variants.
- [§3.4 and Table 2] The case study is a single anecdote and does not support the general claim that CAR 'overcomes the framing effect.' Moreover, the Rethink explanation in Table 2 is internally inconsistent: it states that rating the news as 0.5 'may be too optimistic' but then adjusts the score upward to 0.7. This undercuts the illustrative value of the example. A systematic evaluation is needed, such as blind rating of the explanations by independent judges or a quantitative comparison of score revisions against subsequent price movements.
minor comments (5)
- [Abstract] The abstract contains the phrase 'still remains paid limited attention'; this should read 'has received limited attention.'
- [Throughout] The manuscript uses 'frame effect' and 'framing effect' interchangeably; please standardize the terminology.
- [§3.3] The ablation study does not specify the prompts, window sizes, or thresholds used for the 'Rethink only' variant or the random-trading baseline, so the reported 42% profit and 20–40% random fluctuation range are not fully reproducible.
- [Data Availability] The data availability statement says 'Data requests can be made to Datayes via DFSupport@wind.com.cn,' which appears to assign a Wind email address to Datayes; please verify the correct contact information for each data provider.
- [Declarations] The conflict-of-interest statement says that Shuoling Liu, Gaoguo Jia, Yuhang Jiang, and Liyuan Chen are all affiliated with E Fund Management, but the affiliations list only the latter three with E Fund and the corresponding author with HKUST and CAS; please reconcile this discrepancy.
Circularity Check
No significant circularity: the study is an empirical backtest, not a derivation; the reported excess returns are not equivalent to the paper's inputs by construction.
full rationale
The paper contains no formal derivation chain whose output is equivalent to its inputs. The CAR method is an empirical prompt-and-backtest setup: ChatGPT scores gold news, scores are averaged and thresholded, and returns are measured on historical Au9999 data. The central claim that CAR 'effectively overcomes the framing effect and facilitates excess returns' is supported by a table of backtest returns, not by a mathematical identity. No fitted parameter is later renamed as a prediction in a way that forces the reported result; the threshold is described as a 'predetermined value' in Section 3.2, but its numerical value and selection procedure are not reported, which is a transparency and validity concern rather than circularity. The comparison between Classify and CAR simultaneously changes the averaging window (5 vs 20 news items) and the sign of the trading rule, so attribution of the performance gap to the rethink step is confounded; this is an experimental-design limitation, not a circular reduction. The paper contains no load-bearing self-citations: all listed references are to external prior work, and no 'uniqueness theorem' from the authors is invoked. The Discussion candidly notes that 'our experiments are based solely on historical data' and that previously profitable strategies may regress to the mean as assets become more fully priced, further confirming that the claims are empirical and time-bound rather than definitionally guaranteed. Honest non-finding: score 0.
Assumptions & free parameters
free parameters (4)
- Buy/sell threshold =
not reported
- News averaging window (baselines) =
5
- News averaging window (CAR) =
20
- Trading direction rule =
contrarian for baselines, momentum for CAR
assumptions (4)
- domain assumption LLM scoring of Chinese gold news in zero-shot mode yields economically meaningful sentiment signals.
- domain assumption Au9999 gold can be traded at daily close prices without transaction costs, liquidity constraints, or slippage.
- domain assumption The backtest period 2018-2023 is representative for evaluating the strategy's performance.
- ad hoc to paper A single case study demonstrates that the rethink step avoids the framing effect.
Cite this review
Pith. "Pith review of Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment." pith.science (2026). https://pith.science/paper/QG724ZMT
@misc{pith2026241113599,
author = {Pith},
title = {Pith review of: Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment},
year = {2026},
howpublished = {\url{https://pith.science/paper/QG724ZMT}},
note = {Machine review of arXiv:2411.13599}
}
read the original abstract
Large Language Models (LLMs) have achieved remarkable success recently, displaying exceptional capabilities in creating understandable and organized text. These LLMs have been utilized in diverse fields, such as clinical research, where domain-specific models like Med-Palm have achieved human-level performance. Recently, researchers have employed advanced prompt engineering to enhance the general reasoning ability of LLMs. Despite the remarkable success of zero-shot Chain-of-Thoughts (CoT) in solving general reasoning tasks, the potential of these methods still remains paid limited attention in the financial reasoning task.To address this issue, we explore multiple prompt strategies and incorporated semantic news information to improve LLMs' performance on financial reasoning tasks.To the best of our knowledge, we are the first to explore this important issue by applying ChatGPT to the gold investment.In this work, our aim is to investigate the financial reasoning capabilities of LLMs and their capacity to generate logical and persuasive investment opinions. We will use ChatGPT, one of the most powerful LLMs recently, and prompt engineering to achieve this goal. Our research will focus on understanding the ability of LLMs in sophisticated analysis and reasoning within the context of investment decision-making. Our study finds that ChatGPT with CoT prompt can provide more explainable predictions and overcome behavioral biases, which is crucial in finance-related tasks and can achieve higher investment returns.
Reference graph
Works this paper leans on
-
[1]
Preprint at https://arxiv.org/abs/2206.07682 (2022)
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. Preprint at https://arxiv.org/abs/2206.07682 (2022)
arXiv 2022
-
[2]
Preprint at https://arxiv.org/abs/2212.13138 (2022) 12
Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large Language Models Encode Clinical Knowledge. Preprint at https://arxiv.org/abs/2212.13138 (2022) 12
arXiv 2022
-
[3]
Preprint at https://arxiv.org/abs/2205.11916 (2022)
Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large Language Models are Zero-Shot Reasoners. Preprint at https://arxiv.org/abs/2205.11916 (2022)
arXiv 2022
-
[4]
Preprint at https://arxiv.org/abs/2201.11903
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E.H.-h., Xia, F., Le, Q., Zhou, D.: Chain of thought prompting elicits reasoning in large language models (2022). Preprint at https://arxiv.org/abs/2201.11903
arXiv 2022
-
[5]
Preprint at https: //arxiv.org/abs/2203.02155 (2022)
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. Preprint at https: //arxiv.org/abs/2203.02155 (2022)
arXiv 2022
-
[6]
Preprint at https://arxiv.org/ abs/1706.03741 (2017)
Christiano, P., Leike, J., Brown, T.B., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Preprint at https://arxiv.org/ abs/1706.03741 (2017)
arXiv 2017
-
[7]
Preprint at https://arxiv.org/abs/2109.01652 (2021)
Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., Le, Q.V.: Finetuned Language Models Are Zero-Shot Learners. Preprint at https://arxiv.org/abs/2109.01652 (2021)
arXiv 2021
-
[8]
Preprint at https://arxiv.org/abs/2303.17564 (2023)
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kam- badur, P., Rosenberg, D., Mann, G.: BloombergGPT: A Large Language Model for Finance. Preprint at https://arxiv.org/abs/2303.17564 (2023)
arXiv 2023
Show all 19 references
-
[9]
Preprint at https://arxiv.org/abs/1908.10063 (2019)
Araci, D.: Finbert: Financial sentiment analysis with pre-trained language models. Preprint at https://arxiv.org/abs/1908.10063 (2019)
2019 arXiv
-
[10]
Preprint at https: //arxiv.org/abs/2211.00083 (2022)
Shah, R.S., Chawla, K., Eidnani, D., Shah, A., Du, W., Chava, S., Raman, N., Smiley, C., Chen, J., Yang, D.: WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain. Preprint at https: //arxiv.org/abs/2211.00083 (2022)
2022 arXiv
-
[11]
Preprint at https://arxiv.org/abs/2301.03136 (2023)
Son, G., Lee, H., Kang, N., Hahm, M.: Removing Non-Stationary Knowledge From Pre-Trained Language Models for Entity-Level Sentiment Classification in Finance. Preprint at https://arxiv.org/abs/2301.03136 (2023)
2023 arXiv
-
[12]
Preprint at https://arxiv
Son, G., Jung, H.-N., Hahm, M.S., Na, K., Jin, S.: Beyond Classification: Finan- cial Reasoning in State-of-the-Art Language Models. Preprint at https://arxiv. org/abs/2305.01505 (2023)
2023 arXiv
-
[13]
Preprint at https://arxiv.org/abs/2304.05351 (2023)
Xie, Q., Han, W., Lai, Y., Peng, M., Huang, J.: The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges. Preprint at https://arxiv.org/abs/2304.05351 (2023)
2023 arXiv
-
[14]
13 Preprint at https://arxiv.org/abs/2211.12588
Chen, W., Ma, X., Wang, X., Cohen, W.W.: Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks (2022). 13 Preprint at https://arxiv.org/abs/2211.12588
2022 arXiv
-
[15]
Preprint at https://arxiv.org/abs/2209.14610
Lu, P., Qiu, L., Chang, K.-W., Wu, Y.N., Zhu, S.-C., Rajpurohit, T., Clark, P., Kalyan, A.: Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning (2022). Preprint at https://arxiv.org/abs/2209.14610
2022 arXiv
-
[16]
Preprint at https://arxiv.org/abs/ 2212.09561
Weng, Y., Zhu, M., He, S., Liu, K., Zhao, J.: Large language models are reasoners with self-verification abs/2212.09561 (2022). Preprint at https://arxiv.org/abs/ 2212.09561
2022 arXiv
-
[17]
Preprint at https://arxiv.org/abs/2210.03350
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N.A., Lewis, M.: Measuring and narrowing the compositionality gap in language models (2022). Preprint at https://arxiv.org/abs/2210.03350
2022 arXiv
-
[18]
Preprint at https://arxiv.org/abs/2210.03493 (2022)
Zhang, Z., Zhang, A., Li, M., Smola, A.J.: Automatic Chain of Thought Prompt- ing in Large Language Models. Preprint at https://arxiv.org/abs/2210.03493 (2022)
2022 arXiv
-
[19]
Preprint at https://arxiv.org/abs/2304.09797 (2023) 14
Zheng, C., Liu, Z., Xie, E., Li, Z., Li, Y.: Progressive-hint prompting improves reasoning in large language models. Preprint at https://arxiv.org/abs/2304.09797 (2023) 14
2023 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.