{"id":"ba75e4c9-d7e8-4595-a104-0912d52b28c3","arxiv_id":"2506.06282","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new multimodal financial reasoning benchmark and a retrieval-based error feedback prompting method that improves model accuracy, with the improvement partly confounded by information leakage.","lead":"This paper introduces FinMR, a multimodal benchmark with 3,200 finance question-answer pairs across 15 topics, and a prompting method called EFL that retrieves past mistakes to improve reasoning. It reports that multimodal input helps and that EFL boosts accuracy, but the EFL gain may partly come from giving models the answer through feedback.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EFL's reported gains are confounded: retrieved error feedback embeds the correct answer and gold explanation, so the improvement over zero-shot CoT may be few-shot answer leakage, not learning from errors.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: EFL's retrieved feedback includes the correct answer and expert explanation, so the improvement over CoT may be due to providing the solution in the prompt. This is not a minor implementation detail; it directly undermines the paper's second main conclusion ('EFL consistently outperformed CoT') and its claimed mechanism of learning from errors. The confound is structural because the error database is constructed using gold explanations, and the test prompt then contains that explanation. The paper provides no control for this leakage, and the missing data/code release prevents external verification of the retrieval contents. I do not see a different concern that is more load-bearing: the benchmark itself may be a useful contribution, but the central method claim is not established as presented. A conditional verdict is appropriate: accept only after the proposed ablations are run and the data/code are released. The reader's verdict of CONDITIONAL matches this assessment.","tokens_in":15391,"tokens_out":3041,"duration_ms":29598,"concrete_test":"Run an ablation on the 640-question test set using the same retrieval and prompt template, but with the correct answer and gold explanation redacted from the retrieved example (keep only the question, options, and the error-analysis/hint text). Compare accuracy against full EFL and against CoT. If accuracy drops to near CoT, the gain is answer/explanation leakage, not error feedback. As a second control, build an analogous 'positive example' database from correct responses (with correct reasoning but no error analysis), retrieve the most similar positive example, and prompt with it; if this matches EFL accuracy, the active ingredient is a solved example, not the error. Also report the distribution of retrieval similarity between test and development questions to rule out near-duplicate leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central EFL claim is that retrieving similar error feedback improves reasoning beyond CoT, with gains up to 12.44% (Qwen VL) and Gemini-1.5-Pro reaching 82.06%. However, the EFL prompt (Figure 4) supplies a 'Retrieved Example' built from the error database, and that database stores the correct answer, the gold explanation, and feedback derived from them (Section 4.1, Figure 3). Thus the test prompt contains a solved similar problem with its answer and reasoning. The zero-shot CoT baseline has no such demonstration, so the comparison conflates two variables: (1) the presence of a relevant in-context example, and (2) the specific value of error feedback. The improvement could come entirely from few-shot in-context learning of a solved example, not from 'learning from past mistakes.' This is a structural confound, not a tuning issue. Moreover, retrieval selects the most similar negative example from the development set, which is drawn from the same exam sources as the test set; if a test question is a near-duplicate of a development question (same template, different numbers), retrieval may effectively leak the solution. The paper does not report retrieval similarity distributions, does not exclude near-duplicates, and does not include control conditions (e.g., retrieving positive examples, or retrieving the example without its answer/explanation). Without such controls, the headline EFL result does not establish the mechanism claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinMR, a multimodal financial reasoning benchmark with 3,200 (also stated as 3,700 in Table 1) expert-level QA pairs across 15 topics, and proposes Error Feedback Learning (EFL), a retrieval-augmented prompting method that retrieves a similar negative example with model-generated feedback from a development-set error database. The authors evaluate several LLMs and MLLMs under CoT and EFL prompting, reporting that EFL consistently improves accuracy over CoT (up to 12.44% for Qwen-VL) and that Gemini-1.5-Pro with EFL reaches 82.06%. They also present an error-type analysis identifying image recognition as the dominant failure mode. The paper claims three contributions: the FinMR benchmark, the EFL method, and a systematic evaluation/error analysis.","tokens_in":15697,"tokens_out":4834,"duration_ms":42037,"significance":"If the benchmark is validated and the EFL gains are shown to arise from error feedback, the paper would be a useful contribution to multimodal financial reasoning: FinMR covers a broader topic set than FAMMA, includes manual explanations, and the EFL method is training-free and simple. Credit is due for the expert-verification pipeline, the manual annotation of explanations, and the breadth of models and topics evaluated. However, the central EFL claim is currently confounded: the retrieved example embeds the correct answer and gold explanation, so the comparison to zero-shot CoT cannot separate the effect of error feedback from the effect of receiving a solved in-context example. The benchmark also has an internal counting inconsistency and the data are not released, which weakens reproducibility. The findings are therefore promising but not yet established.","major_comments":[{"comment":"The EFL evaluation is structurally confounded. The feedback prompt in Figure 3 explicitly includes the Correct Answer and Explanation as inputs, and the EFL test prompt in Figure 4 supplies a Retrieved Example drawn from the resulting database. Thus the EFL condition gives the model a fully solved similar problem, while the zero-shot CoT baseline provides no such demonstration. The reported improvements (e.g., 12.44% for Qwen-VL) could therefore be entirely due to few-shot in-context learning of a solved example rather than to learning from errors. To support the paper's central claim, the authors must add control conditions, such as retrieving a positive example, retrieving an example without the answer/explanation, or ablating the feedback component while keeping the retrieved problem.","section":"Section 4.1–4.2, Figures 3–4"},{"comment":"The number of examples in FinMR is inconsistent: Table 1 reports 3,700, while the abstract, Section 1, and Table 3 all report 3,200. Since the benchmark size is a headline statistic and the paper relies on it in multiple places, this discrepancy must be resolved and the correct count stated consistently throughout.","section":"Table 1 vs. Abstract, Section 1, Table 3"},{"comment":"The retrieval index is built from the development set, which is drawn from the same exam sources as the test set. If a test question is a near-duplicate of a development question (e.g., the same template with different numbers), the top-1 retrieved example could effectively leak the solution. The paper does not report retrieval similarity distributions, does not exclude near-duplicates, and does not analyze how often retrieved examples are near-duplicates. This leakage risk directly affects the validity of the EFL results and needs to be addressed.","section":"Section 4.2, retrieval setup"},{"comment":"The claim that image recognition failures account for 72.84% of total errors is presented without describing the categorization protocol, the number of annotators, the coding instructions, or inter-annotator agreement. Given that the error analysis is one of the three stated contributions, the reliability of this statistic should be documented, and the paper should clarify whether the percentage is computed over all errors or a sampled subset.","section":"Section 5.3, error type analysis"}],"minor_comments":[{"comment":"The image statistics are internally inconsistent: 2,643 + 1,118 = 3,761, not 3,764, and the '30%' label for multiple-image QAs does not match either the QA proportion (557/3,200 ≈ 17.4%) or the image proportion (1,118/3,764 ≈ 29.7%). Please correct the totals and percentages.","section":"Table 3"},{"comment":"Several entries in the LLaMa-3.2-Vision rows appear to be typos (e.g., '43.00' in the PM column and '77.00' in the FRA column), and they are formatted inconsistently with the other entries. Please verify all numbers in the table.","section":"Table 4"},{"comment":"The FAMMA reference is listed as 'Famma' in the bibliography; please unify the capitalization and ensure all citations match the reference list.","section":"References"},{"comment":"The anonymous link in the abstract is not a functional data release; for reproducibility, the paper should state where and when the data and code will be made available, or explicitly describe any restrictions.","section":"Abstract and data availability"},{"comment":"The claim that 'multimodal inputs significantly enhance performance' is only directly supported by the within-model comparison for Gemini-1.5-Pro (text+caption vs. text+image); the other comparisons are between different model families. Please qualify the claim accordingly.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be a workshop submission with promising benchmark construction but insufficiently controlled evaluation. The EFL confound and the counting inconsistency are fixable with additional experiments and careful corrections, so I do not recommend rejection, but the authors must supply control conditions and address the retrieval leakage risk before the central claim can be accepted. The lack of released data is also a concern for reproducibility in a journal context."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, FinMR is a genuinely useful benchmark resource: 3,200 expert QA pairs across 15 financial topics, with images, manual explanations, and a dev/test split. That fills a real gap. Second, the paper's headline claim about EFL is not established: the 'retrieved example' in the EFL prompt includes the correct answer and gold explanation, so the gains over zero-shot CoT could just be few-shot leakage, not learning from errors.\n\nWhat the paper does well: the data collection is documented and domain-credible (CFA/FRM-aligned exam papers, PhD-level verification). The evaluation covers a broad set of closed and open models, and the error analysis, especially the image-recognition bottleneck, is useful. The benchmark could be a community asset if the data and code are actually released.\n\nThe soft spots. The abstract says 3,200 QA pairs but Table 1 says 3,700; minor but needs fixing. More important, the EFL comparison is structurally confounded. Figure 3's feedback generation prompt is fed the correct answer, the explanation, and the wrong reasoning; the stored 'feedback' therefore contains the solution. Figure 4 then places the retrieved example — which includes that solution — into the test prompt. CoT gets no such solved example. So the reported 12.44% gain and 82.06% accuracy may largely reflect in-context learning of a similar solved problem, not 'learning from mistakes.' The paper reports no control (e.g., retrieving a solved example without error feedback, or a positive example, or feedback stripped of the answer). Retrieval from the dev set, drawn from the same exam sources, also raises near-duplicate leakage. Also, the data link is anonymous and no code is shipped, so nothing can be independently checked.\n\nProportionately: the benchmark is the more solid contribution; the EFL method is the weak load-bearing claim. If a revision adds controls, error bars, and a stable data release, the paper would be worth accepting. As is, the central claim overreaches.\n\nThis is for a workshop-level venue, and I'd send it to a serious referee who can demand those controls. The benchmark alone deserves peer review.","headline":"FinMR is a plausible benchmark resource, but the EFL gains are confounded by answer-plus-explanation leakage in the retrieved prompt.","tokens_in":16193,"tokens_out":1961,"would_cite":false,"duration_ms":17760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Error feedback from past mistakes lifts financial reasoning accuracy in multimodal AI models by up to 12.44 percentage points.","keywords":["FinMR benchmark","financial reasoning","multimodal large language models","error feedback learning","chain-of-thought prompting","retrieval-augmented reasoning","image recognition errors","financial math reasoning"],"falsifier":"A control experiment that retrieves a similar solved example with the same correct answer and explanation but no error feedback, or that strips the correct answer and explanation out of the feedback, would settle it: if the control matches EFL, the claimed error-learning mechanism is not the cause.","tokens_in":1592,"feed_emoji":"📊","tokens_out":2229,"duration_ms":83077,"temperature":0.7,"pith_summary":"The paper introduces FinMR, a benchmark of 3,200 expert-level financial question-answer pairs that pair text with charts, tables, and other visuals across 15 finance topics. Its central claim is that multimodal models reason better on finance when they receive both the image and a retrieved example of a similar past mistake with corrective feedback, a training-free method the paper calls Error Feedback Learning (EFL). The paper reports that EFL improves accuracy over plain chain-of-thought prompting across every model tested, with the largest reported gain 12.44 percentage points for Qwen-VL and with Gemini-1.5-Pro reaching 82.06 percent. It also finds that direct image input beats image captions, that math-heavy financial questions are roughly 10 points harder than expertise questions, and that image recognition failure accounts for the majority of errors. If these results hold, retrieving past errors is a cheap way to push financial reasoning accuracy without fine-tuning.","feed_headline":"Error-feedback prompts lift multimodal finance scores by up to 12.44%","feed_subtitle":"The FinMR benchmark shows Gemini-1.5-Pro reaching 82.06 percent accuracy when given examples of past mistakes.","key_machinery":"The load-bearing machinery is the error database plus top-1 retrieval. For each development-set question a model got wrong, the paper has the model produce step-by-step reasoning, then prompts a financial expert with the wrong reasoning, the correct answer, and the gold explanation to generate feedback: corrected steps, a comparison of right versus wrong, and a reusable hint. At test time, EFL encodes the current question, retrieves the most semantically similar negative example from that database, and inserts the example and its feedback into the prompt before asking the model to reason step by step. This is what carries the claimed improvement: no weights change, only the prompt content changes.","core_discovery":"The paper's central claim is that financial reasoning in large models is best measured with a multimodal benchmark and best improved, without retraining, by showing the model a similar solved-with-mistakes example and its feedback. FinMR supplies that measurement: 3,200 questions, each with text, at least one image, a correct option, and a manually verified expert explanation, drawn from professional certification-style university exams across 15 topics, split 80/20 into development and test sets. The EFL method builds an error database from the development set by taking the models' wrong answers, attaching AI-generated feedback that uses the gold explanation, and then, at test time, retrieving the most semantically similar error case and feeding it into the prompt. On this benchmark, the paper reports that EFL beats chain-of-thought prompting for every model surveyed, that the best multimodal result is Gemini-1.5-Pro at 82.06% accuracy, and that most failures, 72.84%, are image recognition errors rather than financial knowledge errors.","pith_inferences":["A direct ablation of EFL, comparing feedback that includes the correct answer against feedback that only says where the reasoning went wrong, would test whether the mechanism is error-based learning or answer leakage; the paper does not report this comparison.","If the gain is mostly leakage, the practical takeaway changes from 'models learn from mistakes' to 'models benefit from retrieving solved examples,' which is a retrieval-augmented generation result rather than a self-reflection result.","The same error database could be used to build few-shot curricula, retrieving not one but a sequence of errors of increasing similarity, which the paper does not explore.","The benchmark's manually verified explanations could support fine-grained per-topic diagnosis, for example separating formula errors from visual extraction errors within quantitative topics."],"forward_implications":["Multimodal financial QA systems can be improved on a fixed model simply by maintaining a growing database of its own past failures and retrieving the nearest one at inference time.","Because EFL works across all tested models, prompt-level error feedback could be folded into production financial assistants without GPU retraining.","Direct image access matters: the same Gemini-1.5-Pro model scores 82.06% with images versus 61.37% with captions, so finance benchmarks that omit images underestimate multimodal models.","Financial math remains a separate bottleneck, roughly 10 points below expertise questions, so gains in visual input do not remove the need for stronger calculation and formula application.","Image recognition failure, not financial knowledge, dominates errors at 72.84%, directing future work toward chart and table reading."],"supporting_citations":[{"why":"FAMMA is the closest prior multimodal financial benchmark; FinMR extends its 8 topics to 15 and adds risk-management coverage.","marker":"Xue et al., 2024"},{"why":"FinQA provides the text-only numerical financial reasoning baseline that motivates adding charts and tables to financial QA.","marker":"Chen et al., 2022c"},{"why":"MathVista supplies the visual-math evaluation design and the external comparison point for FinMR's multimodal questions.","marker":"Lu et al., 2024"},{"why":"MMMU defines college-level multimodal reasoning scope, which FinMR aligns with for expertise questions.","marker":"Yue et al., 2024"},{"why":"Chain-of-thought prompting is the baseline reasoning method that EFL is measured against.","marker":"Wei et al., 2022b"},{"why":"This survey supplies prior evidence that error feedback helps multimodal mathematical reasoning, which EFL adapts into retrieval-based learning.","marker":"Yan et al., 2024b"},{"why":"This survey of MLLM reasoning motivates learning from feedback as a training-free route to better multimodal performance.","marker":"Wang et al., 2024b"},{"why":"TAT-QA demonstrates table-and-text financial QA, marking the visual-chart gap that FinMR is designed to close.","marker":"Zhu et al., 2021"}],"fun_headline_variants":["Multimodal benchmark FinMR: error feedback lifts AI without fine-tuning","Gemini-1.5-Pro hits 82.06% on FinMR with error-feedback prompting","72.84% of AI finance errors are image recognition, not knowledge gaps","FinMR benchmark: error feedback beats chain-of-thought on finance QA","No fine-tuning needed: error feedback lifts multimodal finance reasoning"],"cache_read_input_tokens":18304,"weakest_assumption_plain":"The load-bearing premise is that the accuracy gain attributed to EFL comes from learning from past errors, rather than from the retrieved prompt already containing the correct answer and expert explanation.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal benchmark FinMR: error feedback lifts AI without fine-tuning","Gemini-1.5-Pro hits 82.06% on FinMR with error-feedback prompting","72.84% of AI finance errors are image recognition, not knowledge gaps","FinMR benchmark: error feedback beats chain-of-thought on finance QA","No fine-tuning needed: error feedback lifts multimodal finance reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001017,"raw_usage":{"total_tokens":4279,"prompt_tokens":919,"completion_tokens":3360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":3259}},"tokens_in":535,"tokens_out":3360,"duration_ms":22726,"temperature":1.0,"reasoning_tokens":3259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:20:12.144729+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment that retrieves a similar solved example with the same correct answer and explanation but no error feedback, or that strips the correct answer and explanation out of the feedback, would settle it: if the control matches EFL, the claimed error-learning mechanism is not the cause.","supporting_citations":[],"review_version":1}