Pith. sign in

REVIEW 4 major objections 7 minor 17 references

Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Study

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reasoning-specialized LLMs can pass the multiple-choice portions of all three CFA exam levels, with retrieval-augmented generation raising the hardest level to 87.73 percent accuracy.

desk verdict Useful new CFA benchmark numbers for GPT-o1 and o3-mini, but the error-taxonomy conclusions rest on unvalidated model self-explanations. read the letter →

arxiv 2509.04468 v1 pith:VKBC3BSZ submitted 2025-08-29 cs.CL cs.AI

Xuan Yao , Qianteng Wang , Xinbo Liu , Ke-Wei Huang This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelsfinancialreasoningCFAmockexamsretrieval-augmentedgenerationzero-shotevaluationerroranalysisbenchmarkmodelselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a reasoning-specialized large language model, when aided by retrieval from the official CFA curriculum, can pass the multiple-choice portions of all three CFA exam levels, and that the dominant remaining failure mode is missing domain knowledge rather than weak reading or arithmetic. It reaches this conclusion by testing a general multimodal model, a reasoning-specialized model, and a lightweight model on 1,560 official mock-exam questions under zero-shot prompting and under a two-stage retrieval-augmented pipeline. The central reported numbers are zero-shot accuracies of 94.78%, 89.32%, and 79.09% for the reasoning-specialized model on Levels I, II, and III, with retrieval raising Level III to 87.73%. If correct, these results give practitioners a concrete model-selection and augmentation strategy, and redirect future effort toward knowledge-base quality and numerical verification rather than question rewording.

What carries the argument

The load-bearing mechanism is a two-stage domain-reasoning retrieval-augmented generation pipeline. First, the model transforms each multiple-choice question into a retrieval query consisting of a fifty-word summary and five to ten topic keywords; second, five curriculum segments retrieved by semantic similarity from a level- and topic-specific vector database are injected into a structured reasoning prompt that asks the model to identify relevant CFA concepts, analyze options, and answer with a brief explanation. This pipeline is what produces the reported Level III improvement from 79.09% to 87.73%, and it is also the setting in which the paper's error taxonomy is computed. A further result of the paper is that retrieval helps conceptual questions but leaves calculation accuracy essentially unchanged.

What would settle it

A stratified sample of the 1,560 questions re-scored by independent CFA experts, blind to the model explanations, would settle whether knowledge gaps truly account for most errors; if experts attribute most failures to reasoning instead, the error-taxonomy conclusion collapses. Likewise, replacing the curriculum corpus with an equivalent general finance corpus and seeing the Level III RAG gain disappear would show that retrieval content, not reasoning ability, drives the improvement.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is a performance hierarchy: the model designed to spend extra compute on reasoning beats the general multimodal flagship and the lightweight model at every CFA level in zero-shot settings, and its advantage widens when retrieval-augmented generation supplies curriculum context. The reported numbers are 94.78% / 89.32% / 79.09% zero-shot accuracy for the reasoning-specialized model across Levels I-III, with RAG lifting Level III by 8.64 percentage points to 87.73%. Error analysis on RAG answers attributes nearly two-thirds of residual failures to knowledge gaps, with calculation and inconsistency errors accounting for under 15% combined, and Flesch Reading Ease distributions for correct versus incorrect answers show no meaningful readability effect. The authors conclude that targeted retrieval from authoritative curriculum materials yields larger practical gains than scaling to larger models.

Load-bearing premise

The whole argument assumes that accuracy on CFA mock multiple-choice questions, scored with passing thresholds estimated from online forum advice, is a valid stand-in for real financial-analysis ability, and that the models' own explanations truthfully reveal why they got answers wrong.

Editorial extensions

If this is right

  • Under the study's estimated passing thresholds, the reasoning-specialized model and the lightweight model both clear the multiple-choice bar for all three CFA levels, with or without retrieval.
  • RAG's benefit grows with exam complexity: Level I gains are at most 0.89 percentage points, while Level III gains reach 8.64 points, so retrieval matters most where case contexts and domain-knowledge demands are heaviest.
  • Because knowledge errors dominate residual failures and readability has no measurable effect, future accuracy gains should come from richer curriculum corpora for retrieval and from deterministic numerical checks, not from simplifying question prose.
  • RAG raises conceptual accuracy by roughly 2-3 percentage points for every model but leaves calculation accuracy nearly unchanged, implying that context cannot fix arithmetic and that verification layers are needed for quantitative tasks.
  • The reported cost-performance split supports a tiered deployment strategy: use the reasoning-specialized model for high-stakes, complex analysis, use the lightweight model for routine high-volume tasks, and treat the general multimodal model as reliable only when augmented with retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The passing-level conclusion applies only to the multiple-choice components of the exams; the real CFA also contains constructed-response items at Level III, so full-exam readiness remains untested by this design.
  • The claim that knowledge gaps are the primary failure mode assumes the models' written explanations faithfully report internal reasoning; if those explanations are post-hoc rationalizations, the knowledge-error share may be overstated.
  • A natural extension is to test whether the same retrieval mechanism transfers to messier proprietary sources such as internal regulations and firm policies, where the corpus is not cleanly partitioned by topic and level.
  • The thresholds used to declare passing come from online forum estimates rather than the charter body's undisclosed minimum passing scores; re-running the same benchmark against validated thresholds would strengthen or weaken every pass/fail conclusion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper evaluates three OpenAI models (GPT-4o, GPT-o1, o3-mini) on 1,560 multiple-choice questions from official CFA Institute mock exams across Levels I–III, under zero-shot prompting and under a retrieval-augmented generation (RAG) pipeline that retrieves passages from the official CFA curriculum. It reports model accuracy by level and topic, estimates whether models meet passing thresholds, analyzes error types using a four-category taxonomy, examines the effect of passage readability, and separates calculation from conceptual questions. The central empirical claims are that GPT-o1 outperforms the other models at all levels, that RAG improves accuracy particularly on more complex Level III questions, and that knowledge gaps are the primary failure mode.

Significance. If the results are robust, the paper provides a useful, externally-scored benchmark for LLM financial reasoning: the evaluation is not circular, since accuracy is measured against official CFA answer keys, and the use of 1,560 real mock-exam questions across three levels and ten topics gives the study unusual breadth. The practical implications for model selection and RAG deployment are actionable and clearly presented, and the topic-level breakdown is a valuable resource for practitioners. However, the paper's diagnostic and comparative claims currently outrun its evidence. The error taxonomy rests on unvalidated model-generated explanations, no uncertainty quantification is provided for any accuracy difference, and the possibility of training-data contamination is not addressed. These gaps affect the headline conclusions about knowledge gaps, RAG gains, and the meaning of zero-shot 'intrinsic' knowledge. With targeted revisions, the benchmark itself could be a solid contribution.

major comments (4)
  1. [§5.1, Table 7, §6.4] The headline claim that knowledge gaps are the primary failure mode is not empirically supported as presented. Table 7 is built entirely from explanations generated by the same models that produced the wrong answers; there is no human annotation, inter-annotator reliability check, or gold standard. The 'inconsistency errors' category is defined as the model's thinking being entirely correct while it picks the wrong answer, but internal reasoning is never observed, only a post-hoc textual explanation. The problem is compounded by the fact that Table 7 is computed on RAG runs: when the correct curriculum passage was retrieved and the model still errs, labeling the failure a 'knowledge gap' conflates retrieval/application failure with missing parametric knowledge. In addition, the aggregate statement in §6.4 that '61.68% of mistakes' are knowledge errors is not reproducible from Table 7: summing the table gives 357 knowledge errors out of 765 total errors, i.e., 46.7%, and several rows contradict 'knowledge errors dominate across all models and levels' (e.g., GPT-o1 Level I: 40.43% knowledge vs. 48.94% reasoning; o3-mini Level I: 36.19% knowledge vs. 57.14% reasoning). The authors should either validate the taxonomy with human annotators and report agreement, or substantially weaken the knowledge-gap conclusion and re-label the categories to reflect what is actually measured.
  2. [§4.1, Tables 3–6] No confidence intervals, bootstrap resamples, or significance tests are reported anywhere in the results, yet strong claims are made about small differences. For example, Table 3 reports a GPT-4o Level I RAG improvement of +0.89 percentage points, which corresponds to 8 changed answers out of 900 questions; this is well within the range of sampling noise. Topic-level cells are even more fragile, with some containing as few as 4 questions (e.g., Level III Alternative Investments in Table 6), while the text draws conclusions about topic-specific strengths and weaknesses. The authors should report bootstrap or Bayesian intervals for at least the overall and level-level accuracies, and should exercise appropriate caution when interpreting differences of a few percentage points as substantive RAG gains or model advantages.
  3. [§3.1, §4.1] The possibility of training-data contamination is not addressed. The benchmark uses official CFA Institute mock exams from 2022–2025 and the official CFA curriculum as retrieval sources; these documents may well appear in the pretraining corpora of GPT-4o, GPT-o1, and o3-mini. This directly affects the interpretation of zero-shot accuracy as 'intrinsic knowledge' and could inflate the reported numbers, including GPT-o1's 94.78% Level I score. The authors should perform or report a contamination check (e.g., probing for memorized verbatim passages, or measuring performance on newly written questions of matched difficulty) or, at minimum, explicitly discuss this risk as a limitation rather than treating zero-shot accuracy as unproblematic intrinsic capability.
  4. [§3.1, §4.4] The passing-criteria claims are based on an 'estimated passing criteria' taken from online forum discussions cited in Callanan et al. (2023), not on the official CFA minimum passing score (MPS), which the paper itself notes is not disclosed and varies by exam version. Statements such as 'GPT-o1 and o3-mini can pass CFA Level I, Level II, and the MCQ part of Level III' overstate what the data support. The authors should either present accuracy against a plausible range of MPS thresholds or clearly frame these as informal heuristics rather than actual pass/fail determinations.
minor comments (7)
  1. [Abstract and title] The abstract contains a subject-verb agreement error ('reasoning and accuracy oriented models consistently outperforms'), and the title has a typo ('CF A' should be 'CFA').
  2. [§4.3, Table 5] The text states that GPT-4o zero-shot accuracy is 52.08% for Level II Equity Investments and 54.17% for Level II Derivatives, but Table 5 reports 51.56% and 50.00%, respectively; the numbers should be reconciled.
  3. [§4.4] The sentence 'The GPT-4o model alone can pass Level II, but struggles with Level II and III' is self-contradictory; the first 'Level II' is presumably a typo for 'Level I'.
  4. [§5.1] The sentence 'We focus on the RAG pipeline to probe the intrinsic capabilities of GPT models' is misleading, because the RAG pipeline supplies external curriculum context; 'intrinsic' here should be replaced with something like 'context-augmented' or the sentence should be rephrased.
  5. [References and Appendix A] The MinerU reference is duplicated as Wang et al. 2024a and 2024b, and Appendix Figures A1–A3 all carry the caption 'Level I' although they appear intended to show Levels I, II, and III.
  6. [§5.3] The binary classification of questions into calculation vs. conceptual is itself performed by prompting a GPT model, with no human validation or agreement reported; a short validation note would strengthen this section.
  7. [Overall] No data or code availability statement is included; providing the dataset (or a sample), prompts, and evaluation scripts would materially aid reproducibility, especially for the RAG pipeline.

Circularity Check

0 steps flagged · score 2.0 of 10

Benchmark accuracies are measured against external CFA answer keys and are not circular; the main caveat is an unvalidated model-generated error taxonomy, which is an evidentiary weakness rather than a circular reduction.

full rationale

The headline accuracies (Table 3) are measured against official CFA mock-exam answer keys, an external gold standard; no parameter is fitted to the target outcomes, so the central benchmark is not circular. The RAG pipeline's improvement numbers are differences between two externally scored runs, not a construction that encodes the result. The estimated passing thresholds are imported from Callanan et al. (2023) — a source sharing author Xinbo Liu with this paper — but those thresholds are prior, externally stated rules, and varying them would not change the accuracy measurements; this is a minor self-citation, not load-bearing circularity. The error taxonomy in Section 5.1 is a construct-validity weakness: labels are assigned from models' own explanations without human annotation or agreement checks, and the 'inconsistency' category assumes access to unobserved internal thinking. However, this is an unvalidated measurement of failure type, not a reduction of the conclusion to its input by definition; the accuracy scores stand independently of it. The Section 6.4 claim that knowledge errors are 61.68% of mistakes is not reproducible from Table 7's rows (which sum to about 46.7% across RAG errors), and the readability conclusion in Section 5.2 is asserted without a significance test; both are correctness and evidence concerns, not circularity. Overall, no claimed prediction or derivation is equivalent by construction to its inputs, so the paper has no significant circularity beyond a minor self-citation and self-reported error labels.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The study is empirical, so the ledger focuses on design assumptions: the proxy validity of CFA questions, the external passing criteria, and the reliability of self-reported error explanations. The top-k retrieval count is the main hand-chosen design parameter.

free parameters (1)
  • Number of retrieved curriculum segments (top-k) = 5
    The RAG pipeline retrieves exactly five segments per question (Section 3.3.3). Results are not ablated across k, so the reported RAG improvements depend on this hand-chosen value.
assumptions (3)
  • domain assumption CFA mock exam performance is a valid proxy for real-world financial analysis capability.
    Section 1.2 justifies the study by treating CFA examinations as an authentic benchmark for professional financial practice, but this proxy relationship is asserted rather than validated against real work outcomes.
  • domain assumption Estimated passing criteria from online forum discussions are adequate for pass/fail conclusions.
    Section 3.1 adopts the estimated criteria proposed by Callanan et al. (2023), which are based on forum discussions, and uses them to state which models can pass each level.
  • domain assumption Model-generated explanations accurately reflect the true cause of each error.
    Section 5.1 categorizes errors from the LLMs' written explanations without human annotation or inter-annotator agreement, so the error taxonomy rests on self-reports.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Study." pith.science (2026). https://pith.science/paper/VKBC3BSZ

@misc{pith2026250904468,
  author       = {Pith},
  title        = {Pith review of: Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKBC3BSZ}},
  note         = {Machine review of arXiv:2509.04468}
}
read the original abstract

The rapid advancement of large language models presents significant opportunities for financial applications, yet systematic evaluation in specialized financial contexts remains limited. This study presents the first comprehensive evaluation of state-of-the-art LLMs using 1,560 multiple-choice questions from official mock exams across Levels I-III of CFA, most rigorous professional certifications globally that mirror real-world financial analysis complexity. We compare models distinguished by core design priorities: multi-modal and computationally powerful, reasoning-specialized and highly accurate, and lightweight efficiency-optimized. We assess models under zero-shot prompting and through a novel Retrieval-Augmented Generation pipeline that integrates official CFA curriculum content. The RAG system achieves precise domain-specific knowledge retrieval through hierarchical knowledge organization and structured query generation, significantly enhancing reasoning accuracy in professional financial certification evaluation. Results reveal that reasoning-oriented models consistently outperform others in zero-shot settings, while the RAG pipeline provides substantial improvements particularly for complex scenarios. Comprehensive error analysis identifies knowledge gaps as the primary failure mode, with minimal impact from text readability. These findings provide actionable insights for LLM deployment in finance, offering practitioners evidence-based guidance for model selection and cost-performance optimization.

Figures

Figures reproduced from arXiv: 2509.04468 by the authors.

Figure 1
Figure 1. Domain Reasoning RAG Pipeline 3.3. RAG Technical Implementation and Architecture Our Retrieval-Augmented Generation (RAG) system employs a two-stage pipeline de￾signed specifically for financial knowledge integration and domain-specific reasoning enhancement. The implementation addresses the unique challenges of professional fi￾nancial certification evaluation through carefully engineered prompt templates and vector… view at source ↗
Figure 2
Figure 2. Flesch Reading Ease of CFA questions Since the mock questions contain long case description, multiple tables, and num￾bers, it is helpful to assess how difficult the questions are to understand, and how LLMs are affected by the readability. To quantify textual difficulty, we compute Flesch Reading Ease (FRE) for every question (Rudolf Flesch (2016)). FRE converts sentence length and syllable density into a score tha… view at source ↗
Figure 3
Figure 3. Flesch Reading Ease by model tion formulas, and ethical guidelines, and in numeric-verification techniques to catch arithmetic errors. 5.3. Question Type Topic Level 1 Level 2 Level 3 Calculation Conceptual Calculation Conceptual Calculation Conceptual Alternative Investments 7 58 15 17 0 4 Corporate Issuers 6 63 16 12 0 0 Derivatives 11 59 15 13 14 6 Economics 6 60 11 17 3 19 Equity Investments 30 85 40 24 0 16 Eth… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [1]

    Callanan, A

    E. Callanan, A. Mbakwe, A. Papadimitriou, Y. Pei, M. Sibue, X. Zhu, Z. Ma, X. Liu, and S. Shah. Can gpt models be financial analysts? an evaluation of chatgpt and gpt-4 on mock cfa exams, 2023. URL https://arxiv.org/abs/2310.08678

    arXiv 2023

  2. [2]

    Cfa program

    CFA. Cfa program. https://www.cfainstitute.org/programs/cfa-program

    work page

  3. [3]

    Cfa program structure

    CFA Institute . Cfa program structure. https://www.cfainstitute.org/en/programs/cfa/exam/structure, 2024

    work page 2024

  4. [4]

    Fairhurst and D

    D. Fairhurst and D. Greene. How much does chatgpt know about finance? Financial Analysts Journal, 81 0 (1): 0 12--32, 2025. doi:10.1080/0015198X.2024.2411941. URL https://doi.org/10.1080/0015198X.2024.2411941

    work page arXiv 2025

  5. [5]

    D. K. C. Lee, C. Guan, Y. Yu, and Q. Ding. A comprehensive review of generative ai in finance. FinTech, 3 0 (3): 0 460--478, 2024. doi:10.3390/fintech3030025

    doi:10.3390/fintech3030025 2024

  6. [6]

    X. Li, S. Chan, X. Zhu, Y. Pei, Z. Ma, X. Liu, and S. Shah. Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? a study on several typical tasks. arXiv preprint arXiv:2305.05862, 2023. URL https://arxiv.org/abs/2305.05862

    arXiv 2023

  7. [7]

    Mahfouz, E

    M. Mahfouz, E. Callanan, M. Sibue, A. Papadimitriou, Z. Ma, X. Liu, and X. Zhu. The state of the art of large language models on chartered financial analyst exams. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1068--1082. Association for Computational Linguistics, Nov. 2024. doi:10.18653/v...

    work page doi:10.18653/v1/2024.emnlp-industry.80 2024

  8. [8]

    Hello gpt-4o

    OpenAI . Hello gpt-4o. https://openai.com/index/hello-gpt-4o/?utm_source=chatgpt.com, 2024 a

    work page 2024

Show all 17 references
  1. [9]

    Introducing openai o1

    OpenAI . Introducing openai o1. https://openai.com/o1/?utm_source=chatgpt.com, 2024 b

    2024

  2. [10]

    Openai o3-mini

    OpenAI . Openai o3-mini. https://openai.com/index/openai-o3-mini/?utm_source=chatgpt.com, 2025

    2025

  3. [11]

    How to write plain english

    Rudolf Flesch . How to write plain english. https://web.archive.org/web/20160712094308/http://www.mang.canterbury.ac.nz/writing_guide/writing/flesch.shtml, 2016

    2016

  4. [12]

    Srivastava, M

    P. Srivastava, M. Malik, V. Gupta, T. Ganu, and D. Roth. Evaluating llms' mathematical reasoning in financial document question answering. arXiv preprint arXiv:2402.11194, 2024. URL https://arxiv.org/abs/2402.11194

    2024

  5. [15]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  6. [16]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  7. [17]

    The State of the Art of Large Language Models on Chartered Financial Analyst Exams

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024 arXiv

  8. [18]

    , " * write output.state after.block = add.period

    ENTRY address archive author booktitle chapter collaboration edition editor eid howpublished institution journal key lastchecked month note number numpages organization pages publisher school series title type url urldate volume year label extra.label sort.label INTEGERS outpu...

  9. [19]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.