REVIEW 8 cited by
FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored, and their performance on complex tasks like financial agent remains unknown. This paper presents FinEval, a benchmark designed to evaluate LLMs' financial domain knowledge and practical abilities. The dataset contains 8,351 questions categorized into four different key areas: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge, and Financial Agent. Financial Academic Knowledge comprises 4,661 multiple-choice questions spanning 34 subjects such as finance and economics. Financial Industry Knowledge contains 1,434 questions covering practical scenarios like investment research. Financial Security Knowledge assesses models through 1,640 questions on topics like application security and cryptography. Financial Agent evaluates tool usage and complex reasoning with 616 questions. FinEval has multiple evaluation settings, including zero-shot, five-shot with chain-of-thought, and assesses model performance using objective and subjective criteria. Our results show that Claude 3.5-Sonnet achieves the highest weighted average score of 72.9 across all financial domain categories under zero-shot setting. Our work provides a comprehensive benchmark closely aligned with Chinese financial domain.
Forward citations
Cited by 8 Pith papers
-
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
A new benchmark shows LLMs' financial calculation accuracy collapses without explicit formulas and degrades further when they must generate multi-metric tables.
-
FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domain
FinGAIA is a 407-task Chinese financial agent benchmark where the best agent, ChatGPT DeepResearch, scores 48.9%, far below financial experts at 84.7%.
-
CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model
A 9,356-pair Chinese multimodal financial benchmark reveals that state-of-the-art multimodal LLMs, including GPT-4V, still score below 53% on objective and 39% on subjective financial chart tasks.
-
Agentar-Fin-R1: Enhancing Financial Intelligence through Domain Expertise, Training Efficiency, and Advanced Reasoning
Agentar-Fin-R1, an 8B and 32B financial LLM family, reports top scores on FinEval, FinanceIQ, and a new Finova benchmark while keeping general reasoning near its Qwen3 base.
-
FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios
A four-agent LLM pipeline trained with role-specific data improves human preference on comprehensive Chinese financial analysis tasks.
-
Enterprise Large Language Model Evaluation Benchmark
A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...
-
FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Design
A two-stage RL framework with length, image-selection, and adversarial rewards, trained on 89,378 ASP-built financial image-question pairs, improves multimodal reasoning over LMM-R1.
-
A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.
Discussion (0). Sign in to comment.