REVIEW 5 major objections 7 minor 18 references
FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain
T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A new finance benchmark demands answers backed by visual citations, and its experiments show today's multimodal models pass text inference but stumble on numerical reasoning and precise block-level attribution.
desk verdict Solid, useful benchmark for financial multimodal RAG with visual citation; the GPT-4o labeler/judge overlap is a real but contained bias, not a fatal one, and the resource deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the ground-truth citation format: every QA pair in FinRAGBench-V specifies which page(s) the answer comes from, and the evaluation further checks whether the answer is supported by specific blocks inside those pages. To make models produce such evidence, RGenCite prompts a multimodal LLM with the question and the top-k retrieved page images, asking for an answer plus citations, where each citation is a page index and block coordinates $[x_1, y_1, x_2, y_2]$ marking the supporting region. To score citations automatically, the paper defines citation recall (the union of cited pages/blocks must support the answer; otherwise 0) and citation precision (the fraction of citations that are each individually necessary), and it compares two evaluation strategies: showing the evaluator LLM the full page with a bounding box, or cropping the cited block out of the page. The image-cropping strategy agrees with IoU overlap and human ratings far better than box-bounding, and it is the default used in the main results. This citation machinery is what turns 'answer accuracy' into 'answer accuracy with traceable evidence'.
What would settle it
Independently write a new set of financial QA pairs on the same 1,394 documents without using GPT-4o to draft questions, have three experts verify them, and rerun the RGenCite evaluation; if the ordering of category accuracy and the page-versus-block citation gap change substantially or shrink, the headline results are artifacts of the GPT-4o-based construction rather than stable properties of current multimodal models.
Extended reading notes
Core claim
FinRAGBench-V is built as a multimodal RAG benchmark for finance: a retrieval corpus of 60,780 Chinese and 51,219 English real document pages (research reports, financial statements, prospectuses, academic papers, magazines, news) and a QA set of 855 Chinese and 539 English pairs across seven categories spanning text, chart, table, and multi-page reasoning. Each question requires the model to retrieve the relevant pages, compose an answer, and emit both page-level and block-level visual citations, i.e., coordinates marking the exact image regions that support each claim. The paper reports that on this benchmark the best multimodal retrievers (e.g., ColQwen2) far outperform text-only retrievers, that proprietary multimodal LLMs clearly outperform open-source ones, and that all tested models are noticeably weaker on chart/table numerical calculations and multi-page questions than on text inference. It further reports that page-level citation precision/recall is high for several proprietary models, while block-level citation precision/recall drops sharply, especially for open-source models. The authors conclude that FinRAGBench-V exposes the gap between current MLLMs and the traceability demanded by financial applications.
Load-bearing premise
The benchmark's ground truth answers were drafted by GPT-4o and then human-filtered, so if the questions carry GPT-4o's own phrasing or blind spots, the measured difficulty and error patterns may partly reflect that one model's style rather than general financial reasoning ability.
Editorial extensions
If this is right
- Multimodal retrieval is decisively better than text-only retrieval for financial documents: the best multimodal retriever in the paper reaches about 86–90% Recall@10 while the best text retriever reaches about 36–43%.
- Numerical answers extracted from charts and tables, and answers that require combining several pages, should be treated as high-risk outputs with current models; the benchmark consistently shows these are the weak spots.
- Page-level attribution is near-solved for the best proprietary models but block-level attribution is not, so systems advertising grounded answers need to be checked at the block level rather than trusted because the page is right.
- The benchmark's retrieval corpus, QA pairs, and RGenCite pipeline give future systems a single testbed on which answer accuracy and citation quality can be compared side by side.
Reading between the lines
- A testable extension: re-author a fresh set of financial questions on the same corpus without involving GPT-4o in the drafting, then rerun the identical evaluation; if the difficulty ranking (text over numerics, page over block citation) persists, the headline findings reflect general model limitations rather than the question-generation process.
- The coordinate-based citation scheme and its precision/recall evaluation transfer directly to other high-stakes domains such as medical records or legal filings, where answers must be traced to a specific image region.
- Because the paper shows image-cropping citation evaluation aligns much better with human judgment than box-bounding, user-facing RAG products could adopt a cropped-evidence display as both an evaluation and a presentation strategy.
- The block-level weakness suggests a concrete training recipe: jointly optimizing citation localization with answer generation, for instance by supervising the model to output supporting coordinates before the final answer, could be a higher-leverage improvement than retrieval-side changes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. FinRAGBench-V is a bilingual (Chinese/English) benchmark for retrieval-augmented generation over financial documents containing charts, tables, and text. The resource comprises a retrieval corpus of 60,780 Chinese and 51,219 English pages drawn from six financial document types, and a QA dataset of 1,394 pairs (855 Chinese, 539 English) that were synthesized by GPT-4o, then filtered and revised by human annotators, spanning seven question categories (text inference; chart extraction, calculation, time-sensitive; table calculation, comparison; multi-page). The paper also presents RGenCite, a RAG baseline that generates answers together with page-level and block-level visual citations (coordinate boxes), and an automatic citation evaluation method computing precision/recall at both levels via box-bounding or image-cropping, validated against human ratings. Experiments across five multimodal and four text retrievers and thirteen MLLMs yield three claims: multimodal retrievers strongly outperform OCR-based text retrieval; MLLMs handle text inference well but struggle with numerical reasoning on charts and tables and with multi-page inference; and page-level citation is markedly easier than block-level citation. The paper concludes that FinRAGBench-V exposes current MLLMs' unreliability for financial multimodal reasoning with traceable citations.
Significance. The benchmark fills a genuine gap: prior financial RAG benchmarks are text-only (FinQA, OmniEval) or lack retrieval support (MME-Finance), and none couples multimodal RAG with visual citation in finance. If the findings hold, the resource is valuable: it quantifies a large multimodal-versus-text retriever gap, identifies numerical reasoning on charts/tables and multi-page inference as the current bottleneck, and shows that block-level visual attribution is unsolved even for strong proprietary models. The paper's practice is strong: the dataset and code are released, the construction pipeline is documented with prompts and annotation guidelines (Appendices A and F), retrieval uses standard IR metrics, and the citation metric is checked against both IoU and human ratings. The caveat is that the headline measurements rest on an LLM-judged pipeline: GPT-4o-authored (human-verified) ground truth, an unvalidated GPT-4o answer judge whose prompt does not include the source image, and a citation judge validated on a single generator with moderate correlation. The difficulty numbers and rankings should therefore be read as provisional until an independent-judge study is reported.
major comments (5)
- [Section 6.2, Table 9, Figure 5] The central empirical claim that MLLMs 'handle text inference well but struggle with numerical reasoning on charts, tables, and multi-page inferences' (Section 7.2, Figure 5) rests entirely on the Acc metric, which is produced by a GPT-4o judge whose prompt (Table 9) contains only the question, the ground-truth answer, and the model answer, with no source image. This contradicts Section 6.2's statement that the judge checks whether responses are 'consistent with the visual context'; in fact the judge can only measure textual agreement with a GPT-4o-written reference. Unlike the citation evaluator (Table 6), this judge is never validated against human judgments. Because the QA pairs themselves were authored by GPT-4o (Section 4.2.1), a model that answers correctly but in a different style, with different rounding, or with different answer granularity may be marked 'false', distorting both the model rankings in Table 5 and the category-level difficulty profile in Figure 5. The low self-score of GPT-4o (33.26 Chinese, 43.41 English) mitigates any concern of judge leniency toward the author model, but the false-negative risk remains. The paper's own Appendix H admits possible 'annotation errors, ambiguities, or biases' in the dataset, but the judge-validity gap is not listed in the Limitations. Requested fix: include the source image(s) in the judge prompt, and report a human-agreement study on a stratified sample of outputs (Cohen's kappa and accuracy against human labels).
- [Section 7.3, Table 6, Table 5] The automatic citation evaluator is validated only on citations produced by a single generator (Qwen2.5-VL-72B, Section 7.3), and the adopted configuration (GPT-4o with image-cropping) agrees with human ratings only moderately (Pearson 68.01, Spearman 64.03, Table 6). Since the headline citation observation in Table 5 is the large page-versus-block gap, a systematic evaluator difficulty with cropped blocks could inflate that gap without reflecting genuine model weakness. In addition, the adopted evaluator, GPT-4o, is itself one of the evaluated generators in Table 5, so potential self-preference is not controlled for. Validation should be repeated on outputs of at least two or three other generators, with agreement reported separately for page-level and block-level judgments and for each evaluation approach. Finally, under Equation (3) and its accompanying note, precision is set to 0 whenever recall is 0, so P_Prec and B_Prec in Table 5 partly encode citation-set recall failures; this dependency should be stated when interpreting the page-versus-block comparison.
- [Sections 7.1-7.2] Generation is always evaluated on the top-10 retrieved pages, with no analysis conditioned on retrieval success. ColQwen2's Recall@10 is 85-90% (Table 4), so for roughly 10-15% of questions the supporting page is absent from the generator's input, and multi-page questions additionally require multiple pages to fall inside the top-10 simultaneously. The observed weakness on multi-page inference (and, to a lesser degree, on numerical questions) in Figure 5 and Table 5 may therefore reflect retrieval misses rather than MLLM reasoning failures, and retrieval errors may affect some categories more than others. Reporting Acc and citation quality on the subset of questions whose ground-truth pages are all retrieved (or in an oracle-retrieval condition) would separate the two failure sources and is necessary to support the category-level claims.
- [Section 4.1, Table 2] Table 2 is inconsistent with the corpus statistics stated in Section 4.1. The text reports 1,104 Chinese and 1,105 English documents (2,209 total) and 60,780 Chinese plus 51,219 English pages; the rows of Table 2, however, sum to 2,209 documents and 60,780 pages, and the table gives no indication of whether rows are Chinese-only, English-only, or combined. None of the three readings reconciles all the stated numbers. Please correct the table or the text, give per-language counts by data source, and recompute the average pages per document.
- [Sections 4.2.1-4.2.2, 6.2] The QA authorship and the evaluation pipeline overlap in a way that a human filter does not fully neutralize. The QA pairs are synthesized by GPT-4o (Section 4.2.1), retained or revised by humans (Section 4.2.2), and later judged by GPT-4o (Section 6.2). The human filter verifies clarity, category, answer correctness, and page sources (Table 27), so the retained ground-truth answers are factually checked; but the question wording, the answer granularity, and the selection of what counts as the answer all originate from GPT-4o. If the surviving 1,394 questions carry GPT-4o's question-answering style, the headline findings on numerical reasoning and block-level citation may partly measure GPT-4o-derived biases rather than general financial multimodal ability. I do not consider this disqualifying circularity; the construction is transparent and the resource is valuable. As a concrete test: independently re-author the reference answers for a stratified sample of 100-200 questions (human-written from the documents, without seeing the GPT-4o drafts) and check whether the model ranking in Table 5 and the category ordering in Figure 5 are preserved.
minor comments (7)
- [Figures 3 and 5] In the version under review, the category labels of Figures 3 and 5 are rendered as corrupted glyph sequences (strings such as '/uni00000037/uni00000048/...'), making both figures unreadable; they must be regenerated so that the seven question categories and the per-category Acc values can be verified.
- [Table 3] Table 3 is hard to parse: it is unclear whether 12.23 is the average question length and 10.17 the average answer length, and whether lengths are whitespace tokens (which differ substantially between Chinese and English). Report lengths separately for the two languages and state the tokenizer used.
- [Appendix G] Appendix G states that the dataset was annotated by three experienced annotators, but no inter-annotator agreement is reported for either the QA verification or the rating-based citation evaluation; reporting Cohen's kappa or Krippendorff's alpha would substantiate the quality claims.
- [Table 6] Reporting Pearson, Spearman, and Kendall coefficients as percentages (e.g., 68.01) is unconventional and likely to confuse; the values appear to be 0.68, 0.64, and 0.57 and should be labeled as correlations.
- [Section 5.2, Appendix D] The connection between the model's output coordinates and the 'blocks' used in block-level citation evaluation is not specified: it is unclear how the box-bounding method draws boxes from coordinates and whether overlapping boxes are merged before evaluation; a short algorithmic description of both evaluation approaches would improve reproducibility.
- [Section 3] The notation is mostly consistent, but in Section 3 the symbol 'i' is reused both as the running index of citations (C={c1, c2, ..., ci}) and as an arbitrary citation's index (ci = (pi, Bi)), which makes Equations (1)-(3) harder to read; a notational cleanup would help.
- [Limitations and Appendix H] The Limitations section mentions only the absence of a trained model; given Appendix H's own admission of possible 'annotation errors, ambiguities, or biases' in the QA dataset, the Limitations should also state the dependence of Acc on LLM-based evaluation and the single-generator validation of the citation evaluator.
Circularity Check
No significant circularity: benchmark construction and evaluation are empirically independent despite GPT-4o overlap.
full rationale
FinRAGBench-V is a benchmark-construction paper, not a derivation that reduces predicted quantities to fitted inputs. The QA ground truth is generated with GPT-4o assistance (Section 4.2.1, Table 7) and then human-filtered against explicit correctness and category criteria (Section 4.2.2, Table 27), which breaks any definitional tie between raw model output and the benchmark labels. The headline findings, such as MLLMs struggling with numerical reasoning (Figure 5) and block-level citation being harder than page-level citation (Table 5), are empirical observations obtained by running thirteen external MLLMs through the RGenCite pipeline and scoring their outputs with ROUGE, a GPT-4o Acc judge (Section 6.2, Table 9), and a citation evaluator that was separately calibrated against both IoU-based and rating-based human judgment (Section 7.3, Table 6). No equation in the paper maps a reported metric back into a construction choice by construction. The GPT-4o-as-synthesizer-and-judge overlap is a real validity concern, and the paper itself acknowledges possible bias in Appendix H, but it is not circular in the sense required here: the human filtering of QA pairs and the human/IoU validation of the citation evaluator provide independent checks, and no load-bearing self-citation chain is present. The only adapted external method, VISA (Ma et al. 2024b), is cited as prior work rather than as a substitute for the benchmark's own evidence. Therefore no specific circular step can be exhibited, and the appropriate score is 0.
Assumptions & free parameters
free parameters (1)
- Top-k retrieved pages for generation =
10
assumptions (4)
- domain assumption GPT-4o-generated QA pairs that pass human filtering are accurate and representative of real financial RAG queries.
- domain assumption Retrieval at page granularity with top-k pages and image input is the right task formulation for financial multimodal RAG.
- domain assumption GPT-4o with image-cropping is a reliable proxy for human citation quality.
- standard math Standard retrieval and generation metrics capture meaningful differences.
Cite this review
Pith. "Pith review of FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain." pith.science (2026). https://pith.science/paper/GI5UNGXI
@misc{pith2026250517471,
author = {Pith},
title = {Pith review of: FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain},
year = {2026},
howpublished = {\url{https://pith.science/paper/GI5UNGXI}},
note = {Machine review of arXiv:2505.17471}
}
read the original abstract
Retrieval-Augmented Generation (RAG) plays a vital role in the financial domain, powering applications such as real-time market analysis, trend forecasting, and interest rate computation. However, most existing RAG research in finance focuses predominantly on textual data, overlooking the rich visual content in financial documents, resulting in the loss of key analytical insights. To bridge this gap, we present FinRAGBench-V, a comprehensive visual RAG benchmark tailored for finance which effectively integrates multimodal data and provides visual citation to ensure traceability. It includes a bilingual retrieval corpus with 60,780 Chinese and 51,219 English pages, along with a high-quality, human-annotated question-answering (QA) dataset spanning heterogeneous data types and seven question categories. Moreover, we introduce RGenCite, an RAG baseline that seamlessly integrates visual citation with generation. Furthermore, we propose an automatic citation evaluation method to systematically assess the visual citation capabilities of Multimodal Large Language Models (MLLMs). Extensive experiments on RGenCite underscore the challenging nature of FinRAGBench-V, providing valuable insights for the development of multimodal RAG systems in finance.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Text-Text Inference: Extraction or reasoning based on textual information. Chart-based QA:
-
[2]
Ragbench: Explainable benchmark for retrieval-augmented generation systems.CoRR, abs/2407.11005. Ziliang Gan, Yu Lu, Dong Zhang, Haohan Li, Che Liu, Jian Liu, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, Rongjunchen Zhang, and Yong Dai
-
[3]
Mme-finance: A multimodal finance bench- mark for expert-level understanding and reasoning. CoRR, abs/2411.03314. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023a. Enabling large language models to gener- ate text with citations. InProceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2023, Singapore, Dec...
arXiv 2023
-
[4]
Correctness of Multi-Page Sources:For multi-page queries, if the answer is derived from multiple pages, all referenced page sources must be accurately identified. Decision Rule:Retain the data if all criteria are met, revise it if there are minor issues (e.g., unclear query, incorrect category), and discard it if there are major errors or cannot be fixed ...
-
[6]
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia
Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.CoRR, abs/2403.05530. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: an automated evalua- tion framework for retrieval-augmented generation systems. InProceedings of the 2024 Conference of the North American Chapter of the Association fo...
arXiv 2024
-
[9]
Chart-Information Extraction: Extract key metrics or features from the chart
-
[10]
Chart-Numerical Calculation: Includes calculations such as growth rates, interest rates, total costs, etc
-
[11]
Chart-Time-Sensitive: Includes trend descriptions, causal relationships, event sequences, frequencies, dura- tions, etc. Table-based QA:
Show all 18 references
-
[12]
Table-Numerical Calculation: Perform calculations such as growth rates, interest rates, total costs, etc., using table data
-
[13]
result":[ {
Table-Comparison and Sorting: Compare or rank entities based on specific criteria (e.g., return rates, risks). Here is the format of your output: { "result":[ { "query" : "", "category":"", "answer": "", "answer_type":"" }, { "answer": "", "query" : "", "category":"", "answer_...
2022
-
[14]
Executive Directors received higher payments
Which category received higher payments? Category:Table-Comparison and Sorting Answer: For the year 2023, Executive Directors received $929,609 in share- based payments, while Other key management received $824,662. Executive Directors received higher payments. Reference Image...
2023
-
[15]
Query Clarity:The query should be specific and unambiguous, targeting a particular topic in a document, and avoiding vague or overly general questions
-
[16]
It should not include hallucinations or inferred information beyond what is presented
Answer Correctness:The answer must be factually correct and directly supported by the visual content. It should not include hallucinations or inferred information beyond what is presented. If calculation is involved, the answer should be accurate
-
[17]
Mislabelled or ambiguous categories should lead to revision or rejection
Category Appropriateness:The question should match its assigned category (e.g., table- numerical calculations). Mislabelled or ambiguous categories should lead to revision or rejection
-
[354]
Based on the report from EastMoney, what are the opening and clos- ing prices of Zheshang Securities on October 10, 2024?
Association for Computational Linguistics. Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, and Ido Dagan. 2024. Attribute first, then gener- ate: Locally-attributable grounded text generation. InProceedings of the 62nd Annual Meeting of the Association for Computationa...
2024 arXiv
-
[516]
Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin
Association for Computational Linguistics. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024a. Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EM...
2024 arXiv
-
[2023]
Muye Huang, Han Lai, Xinyu Zhang, Wenjun Wu, Jie Ma, Lingling Zhang, and Jun Liu
RA VEN: in-context learning with retrieval aug- mented encoder-decoder language models.CoRR, abs/2308.07922. Muye Huang, Han Lai, Xinyu Zhang, Wenjun Wu, Jie Ma, Lingling Zhang, and Jun Liu. 2025. Evochart: A benchmark and a self-training approach towards real- world chart und...
2025 arXiv
-
[2024]
Robert Friel, Masha Belyi, and Atindriyo Sanyal
Colpali: Efficient document retrieval with vision language models.CoRR, abs/2407.01449. Robert Friel, Masha Belyi, and Atindriyo Sanyal
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.