{"id":"4cbf29f5-a2d7-46d5-ad12-72756df2ed1a","arxiv_id":"2509.09727","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A role-aware multi-agent pipeline with retrieval and expert critique raises financial multiple-choice accuracy by 6.6-8.3 percentage points over zero-shot CoT across four LLMs.","lead":"This paper builds a three-agent system where one agent writes an answer, one fetches textbook passages, and a role-playing expert critic critiques the draft before a final answer is produced. On 3,532 financial education questions, the system beats plain zero-shot reasoning by roughly 7 to 8 accuracy points across four language models, and it lets a small general model match the zero-shot accuracy of a finance-tuned model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 6.6–8.3% gain is credited to 'critique-based refinement' but is actually the full M-3 pipeline (evidence + critique + hint) versus M-0; the isolated critique effect (M-2 − M-0) is only 3.8–6.1 pp, and even that is confounded with a second inference pass.","rationale":"The paper has real merits: 3,532 expert-designed questions, four models, consistently positive directions of improvement, and a cost analysis. The M-2 gain is positive for every model, so the framework is plausibly useful. However, the central quantitative claim is not supported by the experimental design as written. The abstract's 6.6–8.3% figure comes from M-3 − M-0, not from the critique-only ablation; the critique-only delta is smaller and still confounded by the second generation pass. The paper's own footnote says role prompting is applied only to the Expert Reviewer to maintain a clean baseline, but that does not remove the need for a no-critique second-pass control. This concern is more consequential to the headline claim than the reader's stated weakest assumption about distractor model family, though the reader's rationale independently flags the uncontrolled hint and the non-isolated critic. The fix is straightforward: report M-2 − M-0 as the critique effect, add a neutral second-pass control and a hint-free M-3 condition, and compare GPT-4o-mini M-3 to FinGPT M-3 before claiming equivalence. Until those controls are provided, the strongest quantitative claim is over-credited to the role-aware reviewer, and the paper should remain conditional rather than being accepted on its current evidence.","tokens_in":11573,"tokens_out":13311,"duration_ms":116175,"concrete_test":"Recompute the per-model deltas from Table 2 (M-2 − M-0 and M-3 − M-0) and run paired McNemar tests on the 3,532 questions. If the 6.6–8.3% range is attained only by M-3 − M-0 while M-2 − M-0 is 3.8–6.1 pp, the abstract's attribution of the headline gain to 'critique-based refinement' is not supported; the full-pipeline gain may be driven by retrieval, the hint h, or simple self-correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2's rightmost 'Gain' column is M-3 − M-0, not the contribution of the Expert Reviewer. The abstract and Section 5.1 present this as critique-based refinement, but M-3 additionally enables the Evidence Retriever, prepends an uncontrolled hint h (Section 3.2: 'We also prepend a brief hint h ... for each question'), and adds a third/fourth LLM call. The isolated critique mode M-2 − M-0 is +3.77, +6.09, +4.33, +4.96 for GPT-4o-mini, Gemini-2.0-Flash, Llama-3.1-70B-Instruct, and FinGPT, materially below the claimed 6.6–8.3%. Even M-2 does not isolate the reviewer's role or critique content: relative to M-0 it adds a second generation pass, so a plain 'reconsider' instruction could produce a similar gain. Section 5.1 compounds this by reporting critique's effect as M-2 − M-1, a comparison that removes evidence while adding critique. The 'alternative to fine-tuning' conclusion is also weaker than stated: FinGPT under the same M-3 pipeline reaches 72.84%, above GPT-4o-mini's 69.93%, so the comparison only holds against FinGPT's zero-shot baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a role-aware multi-agent framework for financial education question answering. The framework comprises three agents — a Base Generator, an Evidence Retriever, and an Expert Reviewer — and is evaluated in four configurations (M-0 through M-3) on 3,532 Study.com finance questions with four LLMs: GPT-4o-mini, Gemini-2.0-Flash, Llama-3.1-70B-Instruct, and FinGPT-mt_Llama3-8B_LoRA. The authors report that the full pipeline (M-3) improves answer accuracy by 6.6-8.3% over zero-shot Chain-of-Thought baselines, that the gains are monotonic across configurations, and that GPT-4o-mini under M-3 achieves accuracy comparable to the fine-tuned FinGPT baseline. The paper concludes that structured agent collaboration can serve as a cost-effective alternative to fine-tuning.","tokens_in":11893,"tokens_out":5361,"duration_ms":43109,"significance":"The paper addresses a practically relevant problem and provides a reasonably large evaluation across four models and 82 finance topics. The consistent monotonic ordering M-0 < M-1 < M-2 < M-3 across all four models is a useful empirical observation, and the detailed prompt listings and cost analysis support reproducibility. The claim that agent collaboration can reduce the need for fine-tuning is of practical interest. However, the headline attribution of the 6.6-8.3% gain to 'critique-based refinement' is not supported by the reported ablations, and the evaluation setup contains confounds involving a prepended hint, multiple extra LLM calls, and a benchmark constructed by the same model family that includes an evaluated model. These issues must be resolved before the quantitative claims can be accepted as stated.","major_comments":[{"comment":"The claimed 6.6-8.3% improvement is attributed to 'critique-based refinement', but the rightmost 'Gain' column is M-3 - M-0. M-3 includes evidence retrieval, the prepended hint, and two additional LLM calls beyond the baseline. The isolated critique effect, M-2 - M-0, is materially smaller: +3.77% (GPT-4o-mini), +6.09% (Gemini-2.0-Flash), +4.33% (Llama-3.1-70B-Instruct), and +4.96% (FinGPT). Please report the full decomposition of the gain and restrict the abstract's 'critique-based' phrasing to the M-2 - M-0 comparison or the marginal effect of critique when added to evidence.","section":"Abstract and Section 5.1, Table 2"},{"comment":"The text states that critique drives gains of +2.24%, +4.19%, +2.70%, and +3.34% for the four models. These numbers correspond to M-2 - M-1, not to the effect of the reviewer relative to the baseline. Comparing M-2 with M-1 conflates removing evidence with adding critique. The correct isolated critique effect is M-2 - M-0, which should be reported and discussed. Without this correction, the section misrepresents the reviewer's contribution.","section":"Section 5.1, Impact of Critique (M-2)"},{"comment":"A per-question hint h is prepended in the evidence modes (M-1 and M-3) but not in M-0 or M-2. Consequently, the evidence effect (M-1 - M-0) and the combined effect (M-3 - M-0) conflate retrieval with the hint. Since the hint is derived from the dataset and may itself supply useful concept-level information, it is an uncontrolled variable. Please either include the same hint in all four modes or run an additional ablation that isolates the hint contribution.","section":"Section 3.2 and M-1/M-3 configurations"},{"comment":"The benchmark is modified by using GPT-o4-mini-high to generate distractor options for free-response questions, and the role prompts for the Expert Reviewer are also generated with the same model. Because GPT-4o-mini is one of the evaluated models, there is a risk of model-family bias: the distractors may be systematically easier for this family to reject, which would inflate GPT-4o-mini's accuracy and affect the comparison with FinGPT. Please provide a human evaluation of distractor plausibility or a sensitivity analysis using distractors generated by a different model or source.","section":"Section 4.1 and 4.2"},{"comment":"The claim that the pipeline 'enables GPT-4o-mini to achieve performance comparable to the finance-tuned FinGPT' is based on comparing GPT-4o-mini under M-3 with FinGPT at its zero-shot baseline M-0. Under the same M-3 pipeline, FinGPT reaches 72.84%, which is higher than GPT-4o-mini's 69.93%. The conclusion that structured agent collaboration is an alternative to fine-tuning should be qualified to a comparison against FinGPT's zero-shot, non-augmented baseline, not against FinGPT when it also receives the benefits of the agent framework.","section":"Section 5.2 and Conclusion"}],"minor_comments":[{"comment":"The caption says 'System prompts for the Base Generator (left) and Expert Reviewer agents (right)', but the left panel is the Evidence Retriever Agent, not the Base Generator. Please correct the caption.","section":"Figure 4 caption"},{"comment":"The text says the configurations are 'labeled M0 through M4', but only M-0 through M-3 are defined. Please align the numbering.","section":"Section 3.3"},{"comment":"There are typos: 'GPT-o4-mini-high' appears twice (should likely be GPT-4o-mini-high), 'reviwer' appears in Section 5.1, and 'the agent then is then instructed' appears in Section 3.2. Please proofread.","section":"Throughout"},{"comment":"The hint h is mentioned but not defined precisely; please state whether it is taken from the Study.com dataset, generated, or manually written, and clarify the concatenation notation used in the context block.","section":"Section 3.2"},{"comment":"Category names differ between Table 1 ('Financial Statements & Analysis') and Figure 6 ('Fin. Stmts. & Analysis'). Please use consistent category names.","section":"Table 1 and Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The core empirical finding — that the multi-agent pipeline improves accuracy consistently across models — is plausible and potentially useful, but the manuscript's central attribution claims are overstated relative to the ablations actually run. The benchmark-construction bias from using GPT-4o-mini-high to generate distractors and prompts deserves particular scrutiny because it directly affects the GPT-4o-mini vs. FinGPT comparison. With a corrected analysis and qualified claims, the paper could become publishable; as it stands, the headline numbers do not support the stated conclusion about critique-based refinement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper ships a useful new dataset and an honest cost analysis, but the headline 6.6–8.3% gain is not actually the isolated effect of critique. The real isolated effect (M-2 − M-0) is 3.8–6.1 pp, and even that adds a second inference pass and a role-prompted reviewer, so it's not a clean measure of critique content. That said, the overall M-3 pipeline is a solid applied result.\n\nWhat's new: the 3,532-question finance education set across 82 topics is a real asset, and the monotonic improvement across four models is a useful datapoint. The cost analysis is straightforward and believable. The observation that evidence alone is a small contributor while the extra pass matters is interesting, though undercut by the hint confound in M-1.\n\nSoft spots: First, the M-1 evidence condition prepends a hint h that is not present in M-0, so the +1.5–1.9 pp for evidence is conflated with a concept hint. Second, the abstract and conclusion credit 'critique-based refinement' for the full M-3 gain, but M-3 also enables retrieval, the hint, and the second pass. The stress-test calculation holds: isolated critique (M-2 − M-0) is materially smaller. Also, Section 5.1 reports 'Impact of Critique' as M-2 − M-1, which removes evidence while adding critique—not a fair decomposition. Third, the FinGPT comparison only beats FinGPT's zero-shot baseline; FinGPT under the same M-3 pipeline scores higher than GPT-4o-mini M-3 (72.84 vs 69.93), so 'comparable performance' is misleading. Fourth, no error bars or significance tests; with 3,532 questions, a few percentage points might be real, but variance across topics could be large. The distractor generation by GPT-4o-mini-high is a minor concern, not a load-bearing flaw.\n\nWho is this for: researchers building applied educational QA systems who want a cheap prompting pipeline and a new benchmark. It deserves a serious referee; the edits should focus on re-labeling the gains, adding an isolated evidence ablation without the hint, and reporting significance. I'd accept a revised version after those changes.","headline":"Useful dataset and a plausible multi-agent pipeline, but the headline accuracy gain is mis-attributed to critique and the ablations need cleaning.","tokens_in":12409,"tokens_out":2283,"would_cite":true,"duration_ms":18897,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a role-aware multi-agent loop—draft, retrieve evidence, critique, revise—raises multiple-choice accuracy on 3,532 finance education questions by 6.6–8.3 percentage points over zero-shot Chain-of-Thought, and lets an…","keywords":["financial question answering","multi-agent LLM","role prompting","retrieval-augmented generation","critique-based refinement","finance education","Chain-of-Thought","FinGPT baseline"],"falsifier":"Regenerate the distractors for the same 3,532 questions with a different model family or with human-written distractors, then rerun the baseline and full-pipeline evaluations; if GPT-4o-mini's margin over FinGPT shrinks or the critique-driven gains disappear, the core comparison is an artifact of distractor-generation bias.","tokens_in":11319,"feed_emoji":"🎓","tokens_out":6485,"duration_ms":51680,"temperature":0.7,"pith_summary":"This paper claims that a three-agent prompting pipeline—a Base Generator that reasons step by step, an Evidence Retriever that pulls relevant passages from finance textbooks, and an Expert Reviewer that critiques the draft answer under a topic-specific role—raises multiple-choice question accuracy on 3,532 finance education questions by 6.6–8.3 percentage points over zero-shot Chain-of-Thought across four LLMs. The biggest gains come from the reviewer's critique, which catches arithmetic and procedural errors, while retrieval adds smaller but compounding factual grounding. A sympathetic reading of the results is that structured role-aware collaboration can give out-of-the-box models accuracy comparable to a finance fine-tuned model, without weight updates. The claim matters because it suggests a cheap, prompt-only route to domain question-answering improvement in finance education.","feed_headline":"Three-agent critic loop lifts finance exam accuracy 6.6-8.3%","feed_subtitle":"A reviewer that checks every calculation lets low-cost LLMs match finance-tuned models without fine-tuning.","key_machinery":"The load-bearing mechanism is a single-pass, three-agent loop: a Base Generator produces a zero-shot Chain-of-Thought answer; an Evidence Retriever returns the top three textbook chunks by cosine similarity plus a per-question hint; and an Expert Reviewer, primed with a role prompt such as \"capital-structure strategist,\" audits the draft—recomputing every numeric value and checking conceptual correctness—and returns a critique. The Base Generator then makes a fresh second pass that integrates evidence and critique. Each additional agent adds one LLM call, so cost scales linearly with accuracy rather than with model size, and the critique step is where most of the measured gain comes from.","core_discovery":"The paper's central discovery is that critique-based refinement is the main driver of improvement: adding an Expert Reviewer that recomputes numbers and verifies conceptual correctness lifts accuracy by 2.24–4.19 percentage points across the four models, roughly twice the effect of adding retrieved textbook evidence (1.53–1.90 points). Combining both in the full M-3 configuration produces the best accuracy for every model, with Gemini-2.0-Flash reaching 79.14% and GPT-4o-mini reaching 69.93%, enough to surpass the 65.21% zero-shot accuracy of the finance-tuned FinGPT-mt_Llama3-8B_LoRA baseline. The authors interpret this as evidence that structured, role-aware agent collaboration can substitute for expensive fine-tuning in financial question answering.","pith_inferences":["If the distractor-generation bias is controlled, the framework's relative gains likely persist, but the specific claim that GPT-4o-mini matches FinGPT may not: the distractors were made by GPT-4o-mini-high, a sibling of the evaluated model, and same-family distractors could be easier for it to reject.","Because the mechanism is prompt-plus-retrieval and the workload is recomputable arithmetic, the same three-agent structure should transfer to other certification-style domains—accounting, actuarial science, engineering fundamentals—where a textbook corpus and unambiguous numeric answers exist.","The paper leaves critique structure unexamined; coding the reviewer's feedback by error type (arithmetic slip, wrong formula, misread table) and measuring which types the second pass fixes could turn the observed gain into a targeted prompt-design recipe.","A natural stress test is what happens when retrieved evidence contradicts the critique: the paper does not explore whether the second-pass generator defers to evidence or to reviewer feedback when they disagree."],"forward_implications":["A critique agent that verifies arithmetic and conceptual steps is worth more than retrieved evidence alone: the critique-only configuration beats the evidence-only configuration for every model, and the full pipeline compounds the two error reductions.","Out-of-the-box models can reach or beat a finance-tuned 8B model on these questions: GPT-4o-mini's 69.93% full-pipeline accuracy exceeds FinGPT's 65.21% closed-book baseline, and Llama-3.1-70B-Instruct exceeds it by 3.2 points.","Accuracy improves at roughly linear cost: each agent added costs one extra LLM call, and the full GPT-4o-mini run cost about $65 end-to-end, making this an inexpensive alternative to fine-tuning.","Weak categories such as Budgeting & Personal Finance show the largest gains from the full pipeline, suggesting retrieval clarifies definitions and critique fixes arithmetic where everyday language meets multi-step calculation.","All four tested models, including the finance-tuned FinGPT itself, improve under the full pipeline, so the benefit is not specific to one model family."],"supporting_citations":[{"why":"This citation supplies the prompting method used to generate the three distractor options that convert free-response questions into multiple-choice form.","marker":"[6]"},{"why":"This citation provides the finance-tuned FinGPT LoRA model whose zero-shot accuracy the pipeline is compared against and claimed to match or exceed.","marker":"[26]"},{"why":"This citation establishes role-play prompting as a zero-shot reasoning enhancer, which the paper's role-aware agents build on.","marker":"[14]"},{"why":"This citation supports the use of LLM-generated role prompts as stable and relevant steering signals for domain-expert agents.","marker":"[13]"},{"why":"These six finance textbooks form the indexed corpus that the Evidence Retriever searches to supply contextual evidence.","marker":"[7, 8, 10, 19, 20, 22]"}],"fun_headline_variants":["Critic agent, not retrieval, drives finance QA gains","Cheap agent loop lets budget LLMs beat finance-tuned models","Agent critic boosts finance answers 6.6-8.3% without fine-tuning","Multi-agent critique lifts finance quiz scores by up to 8.3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured results assume that the multiple-choice distractors, generated by GPT-4o-mini-high, are no easier for the evaluated GPT-4o-mini to reject than for other models; if same-family distractors are systematically easier, the reported GPT-4o-mini accuracy and its comparison to FinGPT are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Critic agent, not retrieval, drives finance QA gains","Cheap agent loop lets budget LLMs beat finance-tuned models","Agent critic boosts finance answers 6.6-8.3% without fine-tuning","Multi-agent critique lifts finance quiz scores by up to 8.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000365,"raw_usage":{"total_tokens":1968,"prompt_tokens":955,"completion_tokens":1013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":934}},"tokens_in":571,"tokens_out":1013,"duration_ms":8447,"temperature":1.0,"reasoning_tokens":934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:08:16.938402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the distractors for the same 3,532 questions with a different model family or with human-written distractors, then rerun the baseline and full-pipeline evaluations; if GPT-4o-mini's margin over FinGPT shrinks or the critique-driven gains disappear, the core comparison is an artifact of distractor-generation bias.","supporting_citations":[{"cited_title":"Chiral active matter in external potentials","cited_arxiv_id":"2306.11133","evidence_quote":"This citation provides the finance-tuned FinGPT LoRA model whose zero-shot accuracy the pipeline is compared against and claimed to match or exceed."}],"review_version":1}