Pith. sign in

REVIEW 3 major objections 7 minor 90 references

CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new Chinese multimodal financial benchmark shows GPT-4V scoring only 52% on objective chart questions, indicating current MLLMs cannot yet be trusted for financial chart analysis.

desk verdict Useful staged benchmark for Chinese multimodal finance, but the headline numbers rest on an unvalidated annotation step and an unreleased dataset. read the letter →

arxiv 2506.13055 v1 pith:Z6ZEUKQB submitted 2025-06-16 cs.CL

Jiangtong Li , Yiyun Zhu , Dawei Cheng , Zhijun Ding , Changjun Jiang This is my paper
classification cs.CL
keywords multimodallargelanguagemodelevaluationfinancialbenchmarkChinesefinancechartunderstandingvisualquestionansweringGPT-4Vstagedreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CFBenchmark-MM, a Chinese multimodal financial benchmark of 9,356 image-question pairs built from real financial research reports. It claims that current multimodal large language models, including GPT-4V, perform only moderately on financial chart analysis: 52% accuracy on objective questions and 38% on subjective questions when charts are provided. The authors argue this shows general-purpose MLLMs are not yet reliable for financial decision-making, and that the primary failure modes are misinterpretation of visual content and misunderstanding of financial concepts. The benchmark's value would be to guide domain-specific optimization of MLLMs for finance.

What carries the argument

The load-bearing mechanism is the three-step annotation pipeline followed by a staged evaluation protocol. In the pipeline, chart-analysis pairs are collected from real financial research reports; GPT-4 generates two questions per task category from the accompanying textual analysis; and manual examination filters out questions that cannot be answered from the chart alone. The staged evaluation (question-only, question+caption, question+image, question+image+caption) then isolates whether performance gains come from memory, textual grounding, or visual grounding, making the benchmark a diagnostic for multimodal financial capability.

What would settle it

Take a random sample of 200 benchmark questions, have two independent financial analysts answer them from the chart alone without the original textual analysis, and compare their answers to the paper's reference answers. If analysts cannot reliably reproduce the reference answers, or if inter-annotator agreement is low, the claim that questions are answerable from the chart alone — and therefore the benchmark's validity — fails.

Watch

Extended reading notes

Core claim

The central claim is that recognizing and reasoning over financial charts remains an unsolved problem for state-of-the-art MLLMs. The paper constructs CFBenchmark-MM from 2,339 verified charts and 9,356 question-answer pairs across five task types (arithmetic reasoning, statistical reasoning, structural reasoning, financial explanation, and financial knowledge), and evaluates 13 models in a staged protocol that separates inherent knowledge (question only), visual grounding (question + image), textual description (question + caption), and combined inputs. In the chart-inclusive setting, GPT-4V reaches 52% on objective multiple-choice questions and 38% on subjective short-answer questions; all other models score lower. The paper further identifies, through error analysis, that wrong answers mostly come from misreading charts and confusing financial concepts rather than from lack of domain knowledge.

Load-bearing premise

The benchmark is valid only if the manual examination truly removed every question whose answer cannot be derived from the chart alone, and if the GPT-4-generated reference answers are correct; neither is independently verified in the paper.

Editorial extensions

If this is right

  • If the benchmark accurately measures multimodal financial skill, then no current MLLM is reliable enough for automated financial chart analysis: the best model still fails nearly half of objective questions.
  • The staged evaluation implies that adding GPT-4V-generated captions helps open-source models but not GPT-4V itself, suggesting caption information is a subset of what the image already provides.
  • The error analysis implies that progress will require better visual alignment for charts and stronger financial-concept understanding, not merely larger language models.
  • The comparison of LLMs and their multimodal counterparts suggests that multimodal training can degrade pure-text financial knowledge, raising a trade-off for future financial MLLM development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not pursue: replace the charts with equivalent tabular data to separate chart-reading skill from numerical reasoning, since the current design does not isolate those two competences.
  • The gap between the Q+C and Q+I settings could be formalized as a measure of visual grounding quality, a diagnostic the paper reports but does not quantify.
  • If the benchmark is released without reference answers, its long-term value as a held-out testbed will depend on an independent expert audit of the ground truth, which the paper does not provide.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces CFBenchmark-MM, a Chinese-language multimodal financial benchmark with 9,356 image-question pairs built from 2,339 financial-report charts. Questions are generated by GPT-4 from the text analyses accompanying the charts, then manually screened. The authors evaluate 13 MLLMs under four input settings (Q, Q+C, Q+I, Q+I+C), using multiple-choice accuracy for objective tasks and GPT-4-based key-point matching for subjective tasks. The headline finding is that GPT-4V reaches 52% objective and 38% subjective accuracy in the Q+I setting, indicating that state-of-the-art MLLMs remain limited for financial chart analysis. The paper also reports instruction and language ablations, failure cases, and self-correction examples.

Significance. If the benchmark is valid, this is a useful resource: it uses real financial reports, covers five question types and five chart types, and proposes a staged evaluation protocol that separates textual and visual contributions. The failure-case analysis (Section 4.4 and Appendix D.2) is a strength and gives concrete evidence of the claimed error types. However, the central quantitative claims rest on unverified assumptions about question answerability and scorer reliability, and the benchmark is not released, so the reported 52%/38% figures cannot currently be independently checked. The significance is therefore conditional on strengthening the validity evidence.

major comments (3)
  1. [Section 3.1.3 / Appendix B.3] The manual examination does not certify that each question is answerable from the chart alone, and no inter-annotator agreement or rejection statistics are reported. Step 2 only verifies that the correct option is included among the choices; it does not check that the chart uniquely determines that option. Since Appendix A states that the benchmark will be released only after publication, the 52%/38% headline figures in Section 4.2 cannot currently be externally verified. If a nontrivial fraction of items are unanswerable from the chart, or if some reference answers are wrong, the Q+I scores are not a clean measure of chart-based reasoning. I ask the authors to report the number of items rejected at each manual-examination step, provide inter-annotator agreement for the examination, and release a substantial sample of the benchmark for review.
  2. [Section 3.2 / Table 8] The subjective scores are computed by GPT-4 as a scorer, but no validation of this scoring is provided. Because the reference answers were themselves generated by GPT-4 (Section 3.1.2), the reported 38% subjective score for GPT-4V may reflect an alignment between the generator and the scorer rather than the quality of the model's answer. The paper should report agreement between GPT-4 scoring and human scoring on a random sample, using at least two human annotators, and should consider an additional independent LLM scorer. Without this, the subjective numbers are not interpretable at face value.
  3. [Section 3.2 / Section 4.2] The captions used in the Q+C and Q+I+C conditions are generated by GPT-4V, the same model family as the strongest evaluated model, and the paper concludes that captions provide additional useful information to open-source models but not to GPT-4V, attributing this to the captions containing only part of the chart information. This is an uncontrolled comparison: the caption source is confounded with the model being evaluated, and no independent caption generator (e.g., human-written captions or captions from a different MLLM) is used. The claim about caption usefulness should be rephrased or supported by a condition with an independent caption source.
minor comments (7)
  1. [Section 4.2 / Table 8] The headline '52% accuracy and 38% score' should be defined explicitly; Table 8 does not show these aggregate values, and the GPT-4V row average is 46.7. Please report the objective and subjective sub-averages in the table or in the text.
  2. [Section 3.1.2] The numbers 2,500 collected pairs, 2,421 suitable pairs, and 2,339 valid charts should be reconciled; currently it is unclear how many pairs were rejected at each stage.
  3. [Section 4.1] There is a typo: '13 cutting-edged MLLMs' should read '13 cutting-edge MLLMs'.
  4. [Table 8 / Table 9] The model name 'mPLUG-0w1' appears to be a typo for 'mPLUG-Owl'.
  5. [Section 3.1.1] The screening step says 'Only when both individuals agree', but the paper does not report the level of agreement or the resolution of disagreements.
  6. [Section 3.2] The definition of the subjective score ('if three out of five key points are matched, the score is 0.6') implies a fixed key-point count that is not explained; please clarify how the number of key points is determined for each reference answer.
  7. [Section 4.3.2 / Table 4] The English-translation experiment translates instructions, questions, and answers using GPT-4; this means the translation itself is unvalidated, and the observed 1-3% differences may partly reflect translation quality. Please acknowledge this limitation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark scores are measurements against externally sourced charts with a human verification gate, not derivations from fitted inputs or self-citations.

full rationale

The paper's central claim is an empirical measurement: GPT-4V reaches 52% objective and 38% subjective accuracy on CFBenchmark-MM (Section 4.2, Table 8). This is not a derived prediction that reduces to its own inputs. The benchmark construction uses real financial charts from external research reports (Section 3.1.1), GPT-4 generates questions and reference answers from the accompanying textual analyses (Section 3.1.2), and human annotators manually verify chart-only answerability (Section 3.1.3). While the generation pipeline is GPT-4-based and subjective scoring also uses GPT-4 as a scorer, that is a validity and auditability concern rather than a circular derivation: the correct labels for objective questions are tied to chart content and verified by humans, and the reported accuracies are not defined in terms of the GPT-4 outputs. No load-bearing self-citations appear; the paper does not invoke any prior work by the same authors to justify its choices. The lack of inter-annotator agreement statistics and the delayed release (Appendix A) are reproducibility limitations, but they do not constitute circularity under the definitions used here. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on several domain assumptions about automated generation, manual verification, and automated scoring. No free parameters or invented theoretical entities are introduced.

assumptions (4)
  • domain assumption GPT-4-generated questions and answers from textual analyses are accurate and reflect the chart content.
    Section 3.1.2 uses GPT-4 to generate Q-A pairs from report analyses, and manual verification is assumed to filter errors.
  • domain assumption The manual examination by undergraduate students ensures questions are answerable from the chart alone.
    Section 3.1.3 describes the verification process but provides no inter-annotator agreement or statistical quality checks.
  • domain assumption GPT-4 as an automatic scorer for subjective answers provides scores comparable to human judgment.
    Subjective scoring in Section 4.1 is described as GPT-4 matching key points; no human agreement analysis is reported.
  • domain assumption The financial research reports used are representative of real-world financial chart usage.
    Data collection (Section 3.1.1) relies on publicly available reports, assumed to reflect practical financial contexts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model." pith.science (2026). https://pith.science/paper/Z6ZEUKQB

@misc{pith2026250613055,
  author       = {Pith},
  title        = {Pith review of: CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6ZEUKQB}},
  note         = {Machine review of arXiv:2506.13055}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have rapidly evolved with the growth of Large Language Models (LLMs) and are now applied in various fields. In finance, the integration of diverse modalities such as text, charts, and tables is crucial for accurate and efficient decision-making. Therefore, an effective evaluation system that incorporates these data types is essential for advancing financial application. In this paper, we introduce CFBenchmark-MM, a Chinese multimodal financial benchmark with over 9,000 image-question pairs featuring tables, histogram charts, line charts, pie charts, and structural diagrams. Additionally, we develop a staged evaluation system to assess MLLMs in handling multimodal information by providing different visual content step by step. Despite MLLMs having inherent financial knowledge, experimental results still show limited efficiency and robustness in handling multimodal financial context. Further analysis on incorrect responses reveals the misinterpretation of visual content and the misunderstanding of financial concepts are the primary issues. Our research validates the significant, yet underexploited, potential of MLLMs in financial analysis, highlighting the need for further development and domain-specific optimization to encourage the enhanced use in financial domain.

Figures

Figures reproduced from arXiv: 2506.13055 by the authors.

Figure 1
Figure 1. Two examples from CFBenchmark-MM. MLLMs. Additionally, our analysis of incorrect responses pinpoints that misinterpretation of visual content and misunderstanding of financial concepts are the primary issues. These findings empha￾size the importance of MLLMs to analyze multimodal content in financial contexts. Our contributions can be summarized as: • We introduce CFBenchmark-MM, a comprehensive bench￾mark with over… view at source ↗
Figure 2
Figure 2. The criteria we use when selecting data from financial research reports. We choose financial data that are clear, precise, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The illustration of benchmark evaluation system. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The distribution of different types of charts in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: We evaluate 13 cutting-edge MLLMs on CFBenchmark-MM with question and with (question, image) as input [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Failure cases in CFBenchmark-MM by GPT-4V with reasons and answers. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Ten examples from CFBenchmark-MM [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Failure cases in CFBenchmark-MM by GPT-4V with reasons and answers. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Self-correction cases in CFBenchmark-MM generated by GPT-4V. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 39 canonical work pages

  1. [5]

    Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, and Muhammad Abdul-Mageed. 2024. FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models.arXiv preprint arXiv:2402.10986(2024)

    arXiv 2024

  2. [1]

    Introducing the next generation of Claude

    2024. Introducing the next generation of Claude. https://www.anthropic.com/ news/claude-3-family

    2024

  3. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  4. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

    arXiv 2025

  5. [4]

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts.Advances in Neural Information Processing Systems35 (2022), 32897–32912

    2022

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  7. [7]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology15, 3 (2024), 1–45. CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

    2024

  8. [8]

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023. Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793(2023)

    arXiv 2023

Show all 90 references
  1. [9]

    Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, et al . 2023. DISC- FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning.arXiv:2310.15205(2023)

    2023 arXiv

  2. [10]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al

  3. [11]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instruct- blip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems36 (2023)

    2023

  4. [12]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems36 (2024)

    2024

  5. [13]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. 2024. InternLM- XComposer2: Mastering free-form text-image composition and comprehension in vision-language large model.arXiv preprint arXiv:24...

    2024 arXiv

  6. [14]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al . 2024. InternLM-XComposer2-4khd: A pioneering large vision-language model han- dling resolutions from 336 pixels to 4k hd.arXiv preprint arXiv...

    2024 arXiv

  7. [15]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. Glm: General language model pretraining with autoregressive blank infilling.arXiv preprint arXiv:2103.10360(2021)

    2021 arXiv

  8. [16]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv:2306.13394 [cs.CV]

    2024 arXiv

  9. [17]

    Ziliang Gan, Yu Lu, Dong Zhang, Haohan Li, Che Liu, Jian Liu, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, Rongjunchen Zhang, and Yong Dai. 2024. MME- Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning. arXiv:2411.03314 [cs.CV] https://arxiv.or...

    2024 arXiv

  10. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024 arXiv

  11. [19]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. InICLR

    2022

  12. [20]

    Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. 2023. Unified language-vision pretraining with dynamic discrete visual tokenization.arXiv preprint arXiv:2309.04669(2023)

    2023

  13. [21]

    Jimin Huang, Lingfei Qian, Xueqing Peng, Qianqian Xie, and Jordan W

    Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, Koduvayur Subbalakshmi, Guojun Xiong, Li et al. Jimin Huang, Lingfei Qian, Xueqing Peng, Qianqian Xie, and Jordan W. Suchow

  14. [22]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

    2023

  15. [23]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning. PMLR, 12888–12900

    2022

  16. [24]

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language repre- sentation learning with momentum distillation.Advances in neural information processing systems34 (2021), 9694–9705

    2021

  17. [25]

    Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, et al. 2021. Value: A multi-task benchmark for video-and-language understanding evaluation.arXiv preprint arXiv:2106.04632(2021)

    2021 arXiv

  18. [26]

    Yadong Li, Jun Liu, Tao Zhang, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, Chong Li, Yuanbo Fang, Dongdong Kuang, Mingrui Wang, Chenglin Zhu, Youwei Zhang, Hongyu Guo, Fengyu Zhang, Yuran Wang, Bowen Ding, Wei Song, Xu Li, Yuq...

    2025

  19. [27]

    Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al . 2021. Multibench: Mul- tiscale benchmarks for multimodal representation learning.arXiv preprint arXiv:2107.07502(2021)

    2021 arXiv

  20. [28]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023)

    2023

  21. [29]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. MMBench: Is Your Multi-modal Model an All-around Player? arXiv:2307.06281 [cs.CV] https://arxiv.org/abs/2307.06281

    2024 arXiv

  22. [30]

    Dakuan Lu, Hengkui Wu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng, Mengkun Han, Yingsi Xin, and Yanghua Xiao. 2023. Bbt-fin: Comprehensive construction of chinese financial domain pre-trained language model, corpus and benchmark.arXiv preprint arXiv:2302.09432(2023)

    2023 arXiv

  23. [31]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. 2024. DeepSeek-VL: towards real- world vision-language understanding.arXiv preprint arXiv:2403.05525(2024)

    2024 arXiv

  24. [32]

    Kostadin Mishev, Ana Gjorgjevikj, Irena Vodenska, Lubomir T Chitkushev, and Dimitar Trajanov. 2020. Evaluation of sentiment analysis in finance: from lexicons to transformers.IEEE access8 (2020), 131662–131682

    2020

  25. [33]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al

  26. [34]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training

    2018

  27. [35]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog1, 8 (2019), 9

    2019

  28. [36]

    In International conference on machine learning

    Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763

  29. [37]

    Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When flue meets flang: Benchmarks and large pre-trained language model for financial domain.arXiv preprint arXiv:2211.00083(2022)

    2022 arXiv

  30. [38]

    Ankur Sinha, Satishwar Kedas, Rishu Kumar, and Pekka Malo. 2022. SEntFiN 1.0: Entity-aware sentiment analysis for financial news.Journal of the Association for Information Science and Technology73, 9 (2022), 1314–1335

    2022

  31. [39]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research21, 140 (2020), 1–67

    2020

  32. [40]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971(2023)

    2023 arXiv

  33. [41]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079(2023)

    2023 arXiv

  34. [42]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805(2023)

    2023 arXiv

  35. [43]

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. 2024. The FinBen: An Holistic Financial Benchmark for Large Language Models.arXiv preprint arXiv:2402.12659(2024)

    2024 arXiv

  36. [44]

    Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2024. PIXIU: A Comprehensive Benchmark, In- struction Dataset and Large Language Model for Finance. InAdvances in Neural Information Processing Systems

    2024

  37. [45]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance.arXiv:2303.17564(2023)

    2023 arXiv

  38. [46]

    Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. FinGPT: Open- Source Financial Large Language Models.FinLLM at IJCAI(2023)

    2023

  39. [47]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding.Advances in neural information processing systems32 (2019)

    2019

  40. [48]

    Liang Xu, Lei Zhu, Yaotong Wu, and Hang Xue. 2024. SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applica- tions.arXiv preprint arXiv:2404.19063(2024)

    2024 arXiv

  41. [49]

    Dongyi Yi, Guibo Zhu, Chenglin Ding, Zongshu Li, Dong Yi, and Jinqiao Wang

  42. [50]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models.arXiv preprint arXiv:2306.13549(2023)

    2023 arXiv

  43. [51]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration.arXiv preprint arXiv:2311.04257 (2023)

    2023 arXiv

  44. [52]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. MMM...

    2024 arXiv

  45. [53]

    Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang, Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, et al. 2023. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models.arXiv preprint arXiv:2308.09975(2023)

    2023 arXiv

  46. [54]

    Xuanyu Zhang and Qing Yang. 2023. Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters. InProceedings of the 32nd ACM International Conference on Information and Knowledge Management. 4435–4439

    2023

  47. [55]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490(2023)

    2023 arXiv

  48. [59]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models.arXiv preprint arXiv:2304.10592(2023). CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large...

    2023 arXiv

  49. [60]

    无 法生成问题

    Incorporating considerations of security and robustness in the design of future benchmarks to facilitate safer and more reliable applications of MLLMs in financial domain. • MLLMs in finance: 1) Expanding the application of MLLMs within the financial domain by leveraging more ...

  50. [61]

    问题的验证：我们首先验证所提出的问题是否完全可以使用所 提供的多模态信息来回答。这包括检查问题是否直接关联到给定 的文本、图表或图像内容，以确保所有问题都能够凭借所给信息 找到答案。

  51. [62]

    客观问题的格式与准确性检查：对于客观类型的问题，我们细 致检查选项的格式是否正确，并确保所有提供的答案选项中包含 一个准确且无歧义的正确答案。这一步骤确保选择题的每个选项 都清晰、合适，并且正确答案在选项中明确标出。

  52. [63]

    主观问题的答案验证：对于主观问题，我们确认答案的准确 性，并要求答案以要点格式呈现。这意味着答案应简洁明了， 直接点出关键信息，并确保这些信息能够准确反映问题所需的内 容。

  53. [64]

    Question Verification: We first verify whether the questions posed can be entirely answered using the provided multimodal information

    标记与图表无关或信息不足的问题：我们将手动标记那些与图 表无直接关联或无法从图表中得到正确信息的问题。这包括识别 和标记那些题干与图表内容不匹配或图表无法提供足够信息以回 答问题的情况。 (1. Question Verification: We first verify whether the questions posed can be entirely answered using the provided multimodal information. This includes checking if the questions are ...

  54. [65]

    This step ensures that each option in multiple-choice questions is clear, appropri- ate, and the correct answer is distinctly marked within the options

    Format and Accuracy Check for Objective Questions: For objective- type questions, we meticulously examine whether the format of the options is correct and ensure that among all the provided answer op- tions, there is one accurate and unambiguous correct answer. This step ensur...

  55. [66]

    This means the answers should be concise and directly highlighting the key information and ensuring the information accurately reflects the content to address the question

    Verification of Answers for Subjective Questions: For subjective ques- tions, we confirm the accuracy of answers and require that the answers be presented in a bullet-point format. This means the answers should be concise and directly highlighting the key information and ensur...

  56. [67]

    环比上周回落 1.2个百分 点

    Marking Questions Irrelevant to Charts or Lacking Information: We mark questions that have no direct connection to the charts or cannot obtain information from the charts. This includes identifying and mark- ing instances where the question does not match the content of the ch...

  57. [68]

    原油在欧洲能源结构中的占比是33.5%。

  58. [69]

    煤炭在欧洲能源结构中的占比是12.2%。

  59. [70]

    Crude oil accounts for 33.5% of Europe's energy mix

    原油、天然气和煤炭在欧洲能源结构中的总 占比是33.5% + 25% + 12.2% = 70.7%。 (1. Crude oil accounts for 33.5% of Europe's energy mix

  60. [71]

    Natural gas accounts for 25% of Europe's energy mix

  61. [72]

    Coal accounts for 12.2% of Europe's energy mix

  62. [73]

    The total proportion of crude oil, natural gas, and coal in Europe's energy mix is 33.5% + 25% + 12.2% = 70.7%.) Subject:Financial Explanation 在军工板块的上市公司中，2022年的归母净利 润同比增长率与营业收入同比增长率相比有何 不同？(Among the listed companies in the military sector, what is the difference between the ...

    2022

  63. [74]

    2022年的营业收入同比增长率为11.68%，呈 现出稳定的增长态势。同期的归母净利润同比 增长率仅为0.09%，显示出净利润增速显著放 缓的现象。

  64. [75]

    The year-on- year growth rate of operating income in 2022 is 11.68%, showing a stable growth trend

    对比两者，可以看出营业收入的增长速度远 远超过了归母净利润的增长速度。(1. The year-on- year growth rate of operating income in 2022 is 11.68%, showing a stable growth trend. In the same period, net profit growth rate was only 0.09%, indicating a significant slowdown in net profit growth

    2022

  65. [76]

    Comparing the two, it can be seen that the growth rate of operating income far exceeds the growth rate of net profit.) Subject:Financial Knowledge 在提供的背景材料中，EPS代表的是什么金融 指标？ (What financial indicator does EPS represent in the background materials provided?)

  66. [77]

    EPS代表的是“每股收益”这一金融指标。 它是公司净利润与其发行的普通股股票数量之 间的比率，反映了每股普通股所享有的公司盈 利。

  67. [78]

    "Earnings Per Share

    EPS是投资者和分析师用来评估公司盈利能 力和比较不同公司之间业绩表现的重要指标之 一。 (1. EPS stands for the financial indicator ""Earnings Per Share"". It is the ratio between a company's net profit and the number of ordinary shares it has issued, reflecting the company's earnings enjoyed by each ordinary share

  68. [79]

    股票涨停”？ (Please explain what

    EPS is one of the important indicators used by investors and analysts to evaluate a company's profitability and compare performance between different companies.) Subject:Financial Knowledge 请解释什么是“股票涨停”？ (Please explain what "stock rising to the limit"?)

  69. [80]

    股票涨停是指股票的价格在交易日中 达到上涨的幅度限制，无法再继续上涨。

  70. [81]

    "Stock price rising to the limit

    当股票价格达到涨停板时，买入该股 票的订单将会堆积，因为卖出的价格不 能高于涨停价，导致交易暂停或者只能 在涨停价格上进行交易。 (1. ""Stock price rising to the limit"" refers to the situation where the stock price reaches the upper limit of increase during the trading day and cannot continue to rise

  71. [82]

    业务贯通”与“对内

    When the stock price reaches the rising limit, buy orders for the stock will accumulate because the selling price cannot be higher than the limit, resulting in trading being suspended or only conducted at the limit price.) Figure 7: Ten examples from CFBenchmark-MM CFBenchmark...

    2023

  72. [83]

    It ranked last among the Shenwan first-level sub-industries.) Subject Financial Explanation Question 在样本理财子公司中，基准利率上升的 理财子公司有多少家？ (How many wealth management subsidiaries in the sample experienced an increase in the benchmark interest rate?) Correct Answer 在样本理财子公司中，基准利率上升的 理财...

  73. [84]

    本周食品饮料板块指数整 体下跌了2.33个百分点。

  74. [85]

    This week, the Food and Beverage sector index overall decreased by 2.33 percentage points

    在申万一级子行业的排名 是第18。 (1. This week, the Food and Beverage sector index overall decreased by 2.33 percentage points

  75. [86]

    扣非每股收益”是指若不 考虑所得税影响，实际分 配给普通股东的每股收益。 在这张表格中，“扣非每 股收益

    It ranked 18th among the Shenwan first- level sub-industries.) GPT-4V Answer “扣非每股收益”是指若不 考虑所得税影响，实际分 配给普通股东的每股收益。 在这张表格中，“扣非每 股收益”显示了公司在 2020年、2021年和2022年 的每股盈利能力。 ("Non-GAAP earnings per share" refers to the earnings per share actually distributed to common shareholders if t...

    2020

  76. [87]

    金融主题ETF近一周的资金 流动情况是流入，且流入规 模较大，为+4.2亿元。

  77. [88]

    The capital flow situation for the Financial Theme ETF over the past week was an inflow, with a substantial amount of +420 million yuan

    新能源主题ETF近一周的资 金流动情况同样是流入，流 入规模也较大，为+3.5亿元。 (1. The capital flow situation for the Financial Theme ETF over the past week was an inflow, with a substantial amount of +420 million yuan

  78. [89]

    扣非每股收益”是什么意 思？ (What does

    The New Energy Theme ETF also experienced an inflow over the past week, with a significant amount of +350 million yuan.) GPT-4V Answer 券商主题ETF的近一周资金流入为3.4亿元新 能源主题ETF的近一周资金流出为9亿元。这 表明在这段时间内，投资者更多地关注了传 统金融领域，而不是新能源行业。(The Brokerage Theme ETF had a capital inflow of 340 million y...

    2016

  79. [90]

    在2016年至2021年间，口腔诊疗人次的复合增 速是9.8%。 (1. From 2016 to 2021, the compound annual growth rate of dental visits was 9.8%.) Answer1 (Failure) 根据图表，2016年到2019年的诊疗人次分别是： 32127409、35875646、40122671和44983173。 这些数据显示了这几年口腔诊疗服务的增长趋 势。 为了计算复合增速，我们需要考虑每年的增长 百分比。总体复合增速 = 平均增长率，平均增 长率 = (11.87 ...

    2016

  80. [2021]

    Finqa: A dataset of numerical reasoning over financial data.arXiv preprint arXiv:2109.00122(2021)

    2021 arXiv

  81. [2024]

    arXiv:2412.18174 [cs.CE] https://arxiv.org/abs/2412

    INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent. arXiv:2412.18174 [cs.CE] https://arxiv.org/abs/2412. 18174

    arXiv

  82. [2025]

    arXiv:2501.16688 [cs.CL] https://arxiv.org/abs/2501.16688

    MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark. arXiv:2501.16688 [cs.CL] https://arxiv.org/abs/2501.16688

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.