{"id":"a6a83cb6-ad88-4275-8ba3-793ec37ce0cd","arxiv_id":"2509.04468","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On 1,560 CFA mock questions, GPT-o1 beats GPT-4o and o3-mini, and a curriculum-based retrieval pipeline adds the most accuracy on Level III questions.","lead":"This paper benchmarks three OpenAI models on 1,560 official CFA mock exam questions, with and without a retrieval pipeline built from CFA curriculum documents. It finds the reasoning-focused GPT-o1 scores highest, and that retrieval helps most on the hardest, Level III questions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The error taxonomy in §5.1 is built from the models' own unvalidated explanations, so the headline claim that knowledge gaps dominate failures is not empirically grounded; a human-annotation check would settle it.","rationale":"The reader's conditional verdict is appropriate, and this stress-test reaches the same conclusion for a narrower reason. The reader's weakest assumption combined external-validity concerns about CFA mock exams as a proxy for real-world ability with the unvalidated error taxonomy. I focus on the error taxonomy because it is the one internal-validity problem that directly undermines an advertised headline result, not merely the broader generalization of the benchmark. The performance comparisons in Table 3 are large and internally coherent: GPT-o1 is above o3-mini above GPT-4o across levels, and the Level III RAG gain for GPT-o1 is meaningful even under paired-data uncertainty. Those results do not depend on the error taxonomy. What does depend on it is the paper's third main finding, repeated in the abstract and in §6.4, that knowledge gaps are the primary failure mode and that targeted knowledge augmentation should therefore be the top deployment priority. Since Table 7 is generated from the same models' post-hoc explanations, with no human validation and with the additional confound that RAG supplies knowledge at inference time, the 'knowledge gap' conclusion is not trustworthy as stated. This is addressable through a relatively small annotation study, so it does not justify rejection; it strengthens the case for conditional acceptance with explicit validation requirements. The external-validity question is real but is a standard limitation of exam-based benchmarks and is less centrally load-bearing than the error taxonomy.","tokens_in":14910,"tokens_out":7513,"duration_ms":76058,"concrete_test":"Sample roughly 200 wrong RAG answers across models and levels. For each sampled question, record the five retrieved curriculum chunks. Have two CFA-competent annotators independently classify (a) whether the correct answer or the fact needed to answer correctly is present in the retrieved context, and (b) the error type using the Callanan et al. taxonomy, blind to the model's explanation. Compare their labels to Table 7 using Cohen's kappa and category proportions. If kappa is below 0.6, or more than one-third of 'knowledge' errors occur in cases where the correct fact is present in the retrieved chunks, the knowledge-dominance conclusion is unsupported and the error analysis in §5.1 would need to be redone with human-validated labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central diagnostic claim — knowledge gaps are the primary failure mode (abstract, §6.4) — rests on the error classification in Table 7, which is derived solely from the model's own explanation of a wrong answer. No human annotation, inter-annotator reliability, or comparison against a gold standard is reported. The category 'inconsistency errors' is defined as the model's thinking being entirely correct while it selects the wrong answer, but the model's internal reasoning is never observed; only its generated explanation is available, and that explanation can be a post-hoc rationalization. The problem is compounded by the fact that Table 7 is computed on RAG runs: the pipeline is designed to supply the relevant curriculum content, so if the correct fact is present in the retrieved chunks and the model still answers incorrectly, the failure is better characterized as retrieval or application failure, not as an internal 'knowledge gap'. Labeling such cases as knowledge errors conflates retrieval failure with missing parametric knowledge. Consequently, the '61.68% of mistakes are knowledge errors' claim in §6.4 is not an established empirical result about model knowledge; it is an unvalidated classification of model-generated text.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates three OpenAI models (GPT-4o, GPT-o1, o3-mini) on 1,560 multiple-choice questions from official CFA Institute mock exams across Levels I–III, under zero-shot prompting and under a retrieval-augmented generation (RAG) pipeline that retrieves passages from the official CFA curriculum. It reports model accuracy by level and topic, estimates whether models meet passing thresholds, analyzes error types using a four-category taxonomy, examines the effect of passage readability, and separates calculation from conceptual questions. The central empirical claims are that GPT-o1 outperforms the other models at all levels, that RAG improves accuracy particularly on more complex Level III questions, and that knowledge gaps are the primary failure mode.","tokens_in":15135,"tokens_out":6929,"duration_ms":64551,"significance":"If the results are robust, the paper provides a useful, externally-scored benchmark for LLM financial reasoning: the evaluation is not circular, since accuracy is measured against official CFA answer keys, and the use of 1,560 real mock-exam questions across three levels and ten topics gives the study unusual breadth. The practical implications for model selection and RAG deployment are actionable and clearly presented, and the topic-level breakdown is a valuable resource for practitioners. However, the paper's diagnostic and comparative claims currently outrun its evidence. The error taxonomy rests on unvalidated model-generated explanations, no uncertainty quantification is provided for any accuracy difference, and the possibility of training-data contamination is not addressed. These gaps affect the headline conclusions about knowledge gaps, RAG gains, and the meaning of zero-shot 'intrinsic' knowledge. With targeted revisions, the benchmark itself could be a solid contribution.","major_comments":[{"comment":"The headline claim that knowledge gaps are the primary failure mode is not empirically supported as presented. Table 7 is built entirely from explanations generated by the same models that produced the wrong answers; there is no human annotation, inter-annotator reliability check, or gold standard. The 'inconsistency errors' category is defined as the model's thinking being entirely correct while it picks the wrong answer, but internal reasoning is never observed, only a post-hoc textual explanation. The problem is compounded by the fact that Table 7 is computed on RAG runs: when the correct curriculum passage was retrieved and the model still errs, labeling the failure a 'knowledge gap' conflates retrieval/application failure with missing parametric knowledge. In addition, the aggregate statement in §6.4 that '61.68% of mistakes' are knowledge errors is not reproducible from Table 7: summing the table gives 357 knowledge errors out of 765 total errors, i.e., 46.7%, and several rows contradict 'knowledge errors dominate across all models and levels' (e.g., GPT-o1 Level I: 40.43% knowledge vs. 48.94% reasoning; o3-mini Level I: 36.19% knowledge vs. 57.14% reasoning). The authors should either validate the taxonomy with human annotators and report agreement, or substantially weaken the knowledge-gap conclusion and re-label the categories to reflect what is actually measured.","section":"§5.1, Table 7, §6.4"},{"comment":"No confidence intervals, bootstrap resamples, or significance tests are reported anywhere in the results, yet strong claims are made about small differences. For example, Table 3 reports a GPT-4o Level I RAG improvement of +0.89 percentage points, which corresponds to 8 changed answers out of 900 questions; this is well within the range of sampling noise. Topic-level cells are even more fragile, with some containing as few as 4 questions (e.g., Level III Alternative Investments in Table 6), while the text draws conclusions about topic-specific strengths and weaknesses. The authors should report bootstrap or Bayesian intervals for at least the overall and level-level accuracies, and should exercise appropriate caution when interpreting differences of a few percentage points as substantive RAG gains or model advantages.","section":"§4.1, Tables 3–6"},{"comment":"The possibility of training-data contamination is not addressed. The benchmark uses official CFA Institute mock exams from 2022–2025 and the official CFA curriculum as retrieval sources; these documents may well appear in the pretraining corpora of GPT-4o, GPT-o1, and o3-mini. This directly affects the interpretation of zero-shot accuracy as 'intrinsic knowledge' and could inflate the reported numbers, including GPT-o1's 94.78% Level I score. The authors should perform or report a contamination check (e.g., probing for memorized verbatim passages, or measuring performance on newly written questions of matched difficulty) or, at minimum, explicitly discuss this risk as a limitation rather than treating zero-shot accuracy as unproblematic intrinsic capability.","section":"§3.1, §4.1"},{"comment":"The passing-criteria claims are based on an 'estimated passing criteria' taken from online forum discussions cited in Callanan et al. (2023), not on the official CFA minimum passing score (MPS), which the paper itself notes is not disclosed and varies by exam version. Statements such as 'GPT-o1 and o3-mini can pass CFA Level I, Level II, and the MCQ part of Level III' overstate what the data support. The authors should either present accuracy against a plausible range of MPS thresholds or clearly frame these as informal heuristics rather than actual pass/fail determinations.","section":"§3.1, §4.4"}],"minor_comments":[{"comment":"The abstract contains a subject-verb agreement error ('reasoning and accuracy oriented models consistently outperforms'), and the title has a typo ('CF A' should be 'CFA').","section":"Abstract and title"},{"comment":"The text states that GPT-4o zero-shot accuracy is 52.08% for Level II Equity Investments and 54.17% for Level II Derivatives, but Table 5 reports 51.56% and 50.00%, respectively; the numbers should be reconciled.","section":"§4.3, Table 5"},{"comment":"The sentence 'The GPT-4o model alone can pass Level II, but struggles with Level II and III' is self-contradictory; the first 'Level II' is presumably a typo for 'Level I'.","section":"§4.4"},{"comment":"The sentence 'We focus on the RAG pipeline to probe the intrinsic capabilities of GPT models' is misleading, because the RAG pipeline supplies external curriculum context; 'intrinsic' here should be replaced with something like 'context-augmented' or the sentence should be rephrased.","section":"§5.1"},{"comment":"The MinerU reference is duplicated as Wang et al. 2024a and 2024b, and Appendix Figures A1–A3 all carry the caption 'Level I' although they appear intended to show Levels I, II, and III.","section":"References and Appendix A"},{"comment":"The binary classification of questions into calculation vs. conceptual is itself performed by prompting a GPT model, with no human validation or agreement reported; a short validation note would strengthen this section.","section":"§5.3"},{"comment":"No data or code availability statement is included; providing the dataset (or a sample), prompts, and evaluation scripts would materially aid reproducibility, especially for the RAG pipeline.","section":"Overall"}],"recommendation":"major_revision","confidential_remarks":"The paper overlaps with prior work by Callanan et al. (2023) and Mahfouz et al. (2024) on CFA exams and RAG; the authors position their study as broader and more recent, but should more explicitly state what is new beyond using newer models and a curriculum-based RAG pipeline. The benchmark results are likely useful to the community even if the diagnostic analysis needs substantial revision. I would not reject the paper, but the current error-taxonomy and uncertainty issues are too central for acceptance without changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a practical benchmark paper with fresh numbers worth having. The central ranking — GPT-o1 above o3-mini above GPT-4o across all three CFA levels — is large and internally consistent. The RAG gains at Level III, especially GPT-o1's +8.64 points, are plausible and align with the intuition that retrieval helps more when questions demand specific curriculum knowledge. If you need current accuracy figures for these models on CFA-style questions, this is the first place I'd point.\n\nWhat's genuinely new: the specific measurements for GPT-o1 and o3-mini on 1,560 official mock questions, plus the two-stage query generation (summary plus keywords) feeding hierarchical retrieval. That's a legitimate extension of Mahfouz et al., though the paper oversells it as a novel pipeline. The practical guidance on model selection and cost-performance trade-offs is sensible and not overreaching.\n\nThe soft spots are real but mostly fixable. The paper reports no confidence intervals, no repeated sampling, and no significance tests, so small differences like GPT-4o's +0.89% at Level I should be treated as noise. More seriously, the stress-test concern about the error taxonomy is correct. Section 5.1 classifies errors solely from the models' own explanations, with no human annotation or agreement check. That makes the headline claim that knowledge gaps dominate failures (61.68%) an unvalidated classification of model-generated text, not an empirical result. The problem is compounded because Table 7 uses RAG runs: when the correct content is in the retrieved chunks and the model still answers wrong, labeling that a \"knowledge error\" conflates retrieval or application failures with missing parametric knowledge. The \"inconsistency\" category is even sketchier, since it requires knowing the model's internal thinking, which the explanations don't reliably reveal.\n\nThe central accuracy benchmark holds up; the diagnostic section does not. The passing criteria borrowed from online forum discussions are also unvalidated, but those are not load-bearing for the main ranking.\n\nMy verdict: deserving of peer review, with a clear request to either add human-validated error annotations or substantially soften the knowledge-gap claims. The raw performance data is citable; the error taxonomy is not, in its current form.","headline":"Useful new CFA benchmark numbers for GPT-o1 and o3-mini, but the error-taxonomy conclusions rest on unvalidated model self-explanations.","tokens_in":15655,"tokens_out":1809,"would_cite":true,"duration_ms":17914,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reasoning-specialized LLMs can pass the multiple-choice portions of all three CFA exam levels, with retrieval-augmented generation raising the hardest level to 87.73 percent accuracy.","keywords":["large language models","financial reasoning","CFA mock exams","retrieval-augmented generation","zero-shot evaluation","error analysis","benchmark evaluation","model selection"],"falsifier":"A stratified sample of the 1,560 questions re-scored by independent CFA experts, blind to the model explanations, would settle whether knowledge gaps truly account for most errors; if experts attribute most failures to reasoning instead, the error-taxonomy conclusion collapses. Likewise, replacing the curriculum corpus with an equivalent general finance corpus and seeing the Level III RAG gain disappear would show that retrieval content, not reasoning ability, drives the improvement.","tokens_in":14726,"feed_emoji":"📊","tokens_out":6322,"duration_ms":56879,"temperature":0.7,"pith_summary":"This paper claims that a reasoning-specialized large language model, when aided by retrieval from the official CFA curriculum, can pass the multiple-choice portions of all three CFA exam levels, and that the dominant remaining failure mode is missing domain knowledge rather than weak reading or arithmetic. It reaches this conclusion by testing a general multimodal model, a reasoning-specialized model, and a lightweight model on 1,560 official mock-exam questions under zero-shot prompting and under a two-stage retrieval-augmented pipeline. The central reported numbers are zero-shot accuracies of 94.78%, 89.32%, and 79.09% for the reasoning-specialized model on Levels I, II, and III, with retrieval raising Level III to 87.73%. If correct, these results give practitioners a concrete model-selection and augmentation strategy, and redirect future effort toward knowledge-base quality and numerical verification rather than question rewording.","feed_headline":"Reasoning-first LLM passes all three CFA mock levels","feed_subtitle":"A reasoning-specialized model scores 94.78 percent at Level I; retrieval adds 8.64 points at Level III.","key_machinery":"The load-bearing mechanism is a two-stage domain-reasoning retrieval-augmented generation pipeline. First, the model transforms each multiple-choice question into a retrieval query consisting of a fifty-word summary and five to ten topic keywords; second, five curriculum segments retrieved by semantic similarity from a level- and topic-specific vector database are injected into a structured reasoning prompt that asks the model to identify relevant CFA concepts, analyze options, and answer with a brief explanation. This pipeline is what produces the reported Level III improvement from 79.09% to 87.73%, and it is also the setting in which the paper's error taxonomy is computed. A further result of the paper is that retrieval helps conceptual questions but leaves calculation accuracy essentially unchanged.","core_discovery":"On its own terms, the paper's central discovery is a performance hierarchy: the model designed to spend extra compute on reasoning beats the general multimodal flagship and the lightweight model at every CFA level in zero-shot settings, and its advantage widens when retrieval-augmented generation supplies curriculum context. The reported numbers are 94.78% / 89.32% / 79.09% zero-shot accuracy for the reasoning-specialized model across Levels I-III, with RAG lifting Level III by 8.64 percentage points to 87.73%. Error analysis on RAG answers attributes nearly two-thirds of residual failures to knowledge gaps, with calculation and inconsistency errors accounting for under 15% combined, and Flesch Reading Ease distributions for correct versus incorrect answers show no meaningful readability effect. The authors conclude that targeted retrieval from authoritative curriculum materials yields larger practical gains than scaling to larger models.","pith_inferences":["The passing-level conclusion applies only to the multiple-choice components of the exams; the real CFA also contains constructed-response items at Level III, so full-exam readiness remains untested by this design.","The claim that knowledge gaps are the primary failure mode assumes the models' written explanations faithfully report internal reasoning; if those explanations are post-hoc rationalizations, the knowledge-error share may be overstated.","A natural extension is to test whether the same retrieval mechanism transfers to messier proprietary sources such as internal regulations and firm policies, where the corpus is not cleanly partitioned by topic and level.","The thresholds used to declare passing come from online forum estimates rather than the charter body's undisclosed minimum passing scores; re-running the same benchmark against validated thresholds would strengthen or weaken every pass/fail conclusion."],"forward_implications":["Under the study's estimated passing thresholds, the reasoning-specialized model and the lightweight model both clear the multiple-choice bar for all three CFA levels, with or without retrieval.","RAG's benefit grows with exam complexity: Level I gains are at most 0.89 percentage points, while Level III gains reach 8.64 points, so retrieval matters most where case contexts and domain-knowledge demands are heaviest.","Because knowledge errors dominate residual failures and readability has no measurable effect, future accuracy gains should come from richer curriculum corpora for retrieval and from deterministic numerical checks, not from simplifying question prose.","RAG raises conceptual accuracy by roughly 2-3 percentage points for every model but leaves calculation accuracy nearly unchanged, implying that context cannot fix arithmetic and that verification layers are needed for quantitative tasks.","The reported cost-performance split supports a tiered deployment strategy: use the reasoning-specialized model for high-stakes, complex analysis, use the lightweight model for routine high-volume tasks, and treat the general multimodal model as reliable only when augmented with retrieval."],"supporting_citations":[{"why":"Supplies the estimated passing criteria (70% overall for Level I, 60% for Levels II and III) and the four-way error taxonomy used throughout the paper.","marker":"Callanan et al. (2023)"},{"why":"Established the all-CFA-level evaluation with RAG and chain-of-thought prompting that this study extends by retrieving from raw curriculum documents.","marker":"Mahfouz et al. (2024)"},{"why":"Provided the foundational finding that LLMs struggle with domain-knowledge application in finance, motivating the benchmark design.","marker":"Li et al. (2023)"},{"why":"Identified the mathematical-reasoning bottleneck in financial document QA, which the paper targets with its calculation-versus-conceptual split.","marker":"Srivastava et al. (2024)"},{"why":"Provided the document-processing tool that converts CFA curriculum PDFs to markdown while preserving tables and formulas, a prerequisite for the RAG knowledge base.","marker":"Wang et al. (2024b)"},{"why":"Defines the three-level program structure and career-progression framing that the paper uses to argue CFA exams mirror real-world financial analysis.","marker":"CFA Institute (2024)"}],"fun_headline_variants":["CFA benchmark: reasoning models top the pack, RAG adds value","Reasoning-specialized LLM scores 94.78% on CFA Level I","Knowledge gaps, not readability, trip LLMs on CFA exam","RAG boosts CFA Level III accuracy by 8.64 points","First CFA-based LLM evaluation shows reasoning, not size, wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument assumes that accuracy on CFA mock multiple-choice questions, scored with passing thresholds estimated from online forum advice, is a valid stand-in for real financial-analysis ability, and that the models' own explanations truthfully reveal why they got answers wrong.","fun_headline_variants_meta":{"raw":{"variants":["CFA benchmark: reasoning models top the pack, RAG adds value","Reasoning-specialized LLM scores 94.78% on CFA Level I","Knowledge gaps, not readability, trip LLMs on CFA exam","RAG boosts CFA Level III accuracy by 8.64 points","First CFA-based LLM evaluation shows reasoning, not size, wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1797,"prompt_tokens":923,"completion_tokens":874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":779}},"tokens_in":539,"tokens_out":874,"duration_ms":8658,"temperature":1.0,"reasoning_tokens":779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:40:15.617483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A stratified sample of the 1,560 questions re-scored by independent CFA experts, blind to the model explanations, would settle whether knowledge gaps truly account for most errors; if experts attribute most failures to reasoning instead, the error-taxonomy conclusion collapses. Likewise, replacing the curriculum corpus with an equivalent general finance corpus and seeing the Level III RAG gain disappear would show that retrieval content, not reasoning ability, drives the improvement.","supporting_citations":[{"cited_title":"Cfa program structure","cited_arxiv_id":null,"evidence_quote":"Defines the three-level program structure and career-progression framing that the paper uses to argue CFA exams mirror real-world financial analysis."}],"review_version":1}