SMARTFinRAG: Interactive Modularized Financial RAG Benchmark

Yiwei Zha

REVIEW 4 major objections 7 minor 11 references

SMARTFinRAG: Interactive Modularized Financial RAG Benchmark

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SMARTFinRAG builds a modular, interactive benchmark for financial RAG systems and finds that answer quality varies sharply with retriever, model, and decoding settings.

desk verdict The platform is a genuinely useful modular financial RAG demo, but the empirical claims rest on a 10-question sample and one of the conclusions contradicts its own table. read the letter →

arxiv 2504.18024 v1 pith:O2VL7AUY submitted 2025-04-25 cs.CE cs.CLcs.IR

Yiwei Zha This is my paper · ORCID

classification cs.CEcs.CLcs.IR

keywords financialRAGretrieval-augmentedgenerationmodulararchitecturedocument-centricevaluationLLM-as-a-judgefaithfulnessrelevancyquestionanswering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SMARTFinRAG is a modular, interactive platform for benchmarking retrieval-augmented generation (RAG) systems on financial documents. The paper's central claim is that financial RAG quality is best measured by letting researchers swap pipeline components at runtime and by generating evaluation questions directly from newly ingested documents, rather than relying on static benchmarks. In experiments on 10-K filings from the 'Magnificent Seven' companies, configuration choices changed outcomes sharply: BM25 led on ranking metrics, GPT-4o led on answer faithfulness and relevancy, and optimal decoding settings differed by model. The paper concludes that empirical, configuration-level evaluation is necessary before deploying RAG systems in finance.

What carries the argument

The central object is a five-stage modular RAG pipeline: document ingestion, query preprocessing and routing, multi-strategy retrieval, LLM-based generation, and an evaluation engine, with each stage implemented as an independently replaceable component that can be swapped at runtime through a configuration system and web interface. The evaluation framework rests on two mechanisms: document-centric QA generation, where an LLM produces factual, numerical, and comparative questions from freshly ingested documents, and LLM-as-a-judge scoring, where a judge LLM assigns scalar faithfulness and relevancy scores to each answer. The modularity does the argument's work: by holding all other stages fixed, the experiments isolate how retriever choice, decoding parameters (temperature, top-p), and model backend move faithfulness and relevancy.

What would settle it

Run the response evaluation on all 317 QA pairs and compare the GPT-4 judge's faithfulness and relevancy scores against human financial analysts' ratings on the same query–context–response triples; if human-judge agreement is low or the full-set rankings of models and decoding settings differ from the 10-pair rankings, the paper's RQ2 and RQ3 conclusions would need revision.

Watch

Extended reading notes

Core claim

SMARTFinRAG's core discovery is that end-to-end RAG quality in finance cannot be read off from any single component: in the paper's controlled comparisons, the best retriever for ranking (BM25) was not the best for answer quality, and no model reached a faithfulness score above 0.60, leaving hallucination risk high across all configurations. This is demonstrated through a document-centric evaluation paradigm where an LLM writes question–answer pairs from ingested financial documents, retrieval is scored with hit rate, MRR, precision, recall, average precision, and NDCG, and generated answers are scored by a GPT-4-based judge on faithfulness and relevancy as scalar 0–1 values. The paper reports that vector and auto-merging retrievers produced identical numbers, that GPT-4o achieved the highest faithfulness (0.60) and relevancy (0.90) in the controlled model comparison, and that temperature and top-p effects were non-monotonic and model-dependent. Because only 10 of 317 generated QA pairs were used for response evaluation, the paper presents these findings as preliminary.

Load-bearing premise

The premise that the judge LLM's scalar faithfulness and relevancy scores are valid measures of financial answer quality, and that the LLM-generated QA pairs are correct test items, carries the entire response-quality comparison—yet the paper reports no human validation and used only 10 of 317 pairs for those scores.

Editorial extensions

If this is right

If the platform works as claimed, financial institutions can test RAG configurations on their own freshly ingested documents instead of relying on static QA benchmarks, making evaluations current and domain-specific.
The non-monotonic temperature and top-p effects imply that optimal decoding settings are model-specific and must be tuned empirically per deployment rather than inherited from defaults.
The low faithfulness ceiling (maximum 0.60) across models implies that even the best financial RAG configurations still need human review or additional grounding checks for high-stakes answers.
Because auto-merging retrieval returned the same scores as vector retrieval in the current setup, comparisons involving auto-merging currently reflect the vector retriever; a true hierarchical-merging implementation would be needed to test that strategy.
The statement that only 10 of 317 QA pairs were used for response evaluation makes the quantitative rankings indicative, not definitive; expanding the evaluation set is the paper's stated next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: the document-centric QA generator could be reused as a continuous regression suite, where each new financial filing automatically extends the test set and re-scores previously deployed configurations—an operational use the paper does not develop.
My inference: the absence of human validation for the LLM judge is testable; asking financial analysts to score the same 10 query–context–response triples and comparing agreement with the GPT-4 judge would directly bound how much of the RQ2/RQ3 conclusions rest on judge validity.
My inference: the finding that model size does not predict faithfulness suggests that financial-domain calibration rather than parameter count drives grounded answer quality; fine-tuning small models on financial corpora and re-running this benchmark would test that.
My inference: since vector and auto-merging retrievers behaved identically, the reported retriever comparison effectively has three distinct strategies rather than four; implementing genuine hierarchical merging could reveal whether structure-aware retrieval changes the rankings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a solid system/demo paper that addresses a real gap — interactive, modular financial RAG evaluation — but the experimental section is a long way from supporting the "significant performance variations" headline. The platform itself is the contribution, not the empirical findings.

What's new: SMARTFinRAG combines modular RAG components (BM25, dense, hybrid), runtime component swapping, document-centric QA generation from newly ingested filings, and a Streamlit UI that makes configuration easy. That combination is genuinely useful for practitioners who need to compare retrievers, models, and decoding settings on current documents. The code is on GitHub, the architecture is described in enough detail to be reproduced, and the paper is honest about several gaps (OCR, keyword expansion, local model integration).

What's done well: the modular design is thoughtfully structured; the evaluation engine logs traces, supports standard retriever metrics (hit rate, MRR, NDCG), and includes LLM-judge scores for faithfulness and relevancy. The related-work discussion correctly distinguishes static financial benchmarks like FinBen and PIXIU from dynamic RAG evaluation. The author also lists limitations in Section 6, which deserves credit.

Now the soft spots, in proportion. The first is the evaluation: response quality uses only 10 of 317 QA pairs, a single run per configuration, and no variance or significance tests. Scores like 0.15 vs 0.38 on faithfulness could reflect one query flipping. The non-monotonic temperature curves and model rankings in Tables 3–5 are plausible but not established. The second is a direct internal contradiction: the conclusion says hybrid retrievers "significantly outperform" standalone methods, but Table 2 shows BM25 with higher MRR and NDCG than Hybrid Fusion at every top-k setting. That needs fixing regardless of sample size. The third is the unvalidated LLM judge — no human comparison for the scalar faithfulness/relevancy scores — though the paper acknowledges this. Together these make the response-quality comparisons pilot data at best.

Who this is for: anyone building financial RAG systems who wants a quick way to ablate retrievers, prompts, and model settings on their own documents. It is not yet a benchmark you can trust for ranking models.

Recommendation: send it to a venue with a revision opportunity. The system deserves a serious referee, but it needs either a much larger evaluation or a heavy downplaying of the empirical claims to "preliminary observations," plus a fix to the Table 2 contradiction and ideally seeded repeated runs to show variance. If those land, it becomes a useful community resource.

Referee Report

4 major / 7 minor

Summary. SMARTFinRAG is presented as a modular, interactive benchmark and live-demo framework for financial RAG. The system supports runtime-swappable retrieval (BM25, dense vector, hybrid fusion, auto-merging), multiple LLM backends and decoding parameters, document ingestion and chunking, prompt strategies, and an LLM-as-judge evaluation module. The evaluation uses 10-K filings from the Magnificent 7 companies, generates 317 QA pairs from those documents, and reports retrieval metrics across retrievers and GPT-4-judge faithfulness/relevancy scores across decoding parameters and models. The abstract claims that the evaluation reveals significant performance variations across configurations, and the conclusion draws practical recommendations about retriever choice, temperature, and model selection. The manuscript is candid about its main limitations in Sections 4.2 and 6, but those limitations are directly load-bearing for the paper's central empirical claims.

Significance. The modular factory-based architecture and runtime-reconfigurable Streamlit interface are genuine contributions: they address a real gap in financial RAG tooling, and the document-centric QA generation is a reasonable strategy for a continuously updatable evaluation environment. The explicit limitation statements in Section 4.2 and Section 6 are also to the authors' credit. However, the empirical claims in the abstract and Section 6 go beyond what the current experiments can support. If the evaluation is expanded with more samples, repeated runs, uncertainty reporting, and human validation of the judge, and if the inconsistencies with the reported tables are corrected, the platform would be a useful resource for the community. In its current form, the central evidence for 'significant performance variations across configurations' is not statistically established.

major comments (4)

[§4.2, Tables 3–5, §6] The headline claim that the evaluation 'reveals significant performance variations across configurations' rests entirely on response-quality results computed from 10 of 317 QA pairs, with a single run per configuration. With n=10, a difference of 0.10 in faithfulness corresponds to roughly one query changing by one point, so the non-monotonic temperature curves in Table 3 and Figure 4, the top-p patterns in Table 4, and the model hierarchy in Table 5 could all be sampling noise or artifacts of the particular 10 selected pairs. No confidence intervals, repeated-seed runs, or statistical tests are reported. The paper should either provide uncertainty measures and confirm the patterns on a larger sample, or explicitly downgrade these results to illustrative observations; as written, the central empirical claim is not supported.
[§6 vs Table 2 and §4.3] The conclusion states that 'hybrid retrievers significantly outperform standalone lexical or dense methods,' but Table 2 shows the opposite: BM25 achieves higher MRR (0.701932 vs 0.684142), higher NDCG (0.735954 vs 0.719403), higher precision at every top-k, and higher hit rate at top-k=10 (0.839117 vs 0.826498) than Hybrid Fusion. The only sense in which Vector/Auto-Merging excel is hit rate and recall at k=10, and Section 4.3 itself already reports that BM25 achieves the best ranking metrics. This is a direct internal contradiction that must be corrected or substantiated with a comparison that actually supports the claim.
[§3.3.3, §4.2, §6] All response-quality scores in RQ2 and RQ3 come from a GPT-4 LLM-as-a-judge using scalar ratings, and Section 6 concedes that these scores lack categorical explanations and human validation. Since every generation-level conclusion depends on this judge, the paper should provide evidence that the judge's scores agree with human judgments, for example by reporting a validation sample with inter-annotator agreement or correlation with human ratings. Without such validation, the model comparisons in Tables 3–5 should be described only as 'GPT-4-judge scores' and cannot support general claims about faithfulness and relevancy.
[§3.3.1, §4.2] The evaluation paradigm generates QA pairs from the same ingested corpus that is later used as the retrieval index and as the relevance gold standard. This is an acceptable design for a continuously updating live benchmark, but it means the retrieval metrics measure in-corpus search ability rather than generalization to unseen queries, and faithfulness is judged against the very chunks that generated the question. The authors should state this limitation explicitly and, to support the benchmark claim, add a held-out set of human-authored or independently sourced questions whose answers are verified against documents outside the generation loop.

minor comments (7)

[§3.2.2, Table 2] The Auto-Merging subsection says its 'internal logic mirrors the hybrid strategy,' but Table 2 shows Auto-Merging and Vector with identical numbers; please clarify which implementation is actually used and align the text with the observed results.
[Figures 1–2] Figure 2 has the same caption as Figure 1 ('SMARTFinRAG pipeline. Modular components process raw financial documents into indexed chunks and support retrieval-augmented response generation with multi-faceted evaluation.'); the captions should describe the distinct contents of each figure.
[§6 vs Tables 3–5] The Limitations paragraph says 'only two LLMs were compared,' but Tables 3–5 report results for at least ten models; please rephrase to refer to the temperature experiment specifically or correct the count.
[Reference [6]] Reference [6] is listed as 'Author Names. Failsafeqa: A financial LLM benchmark for AI robustness, compliance, and hallucination risks. Blog Post, 2025.' This is an incomplete citation; provide authors, venue, and a stable identifier.
[§1, §3.2, §5, §6] There are several minor wording inconsistencies: 'reusableRAG' in Section 1 should be 'reusable RAG'; 'Y AML' in Section 3.2 should be 'YAML'; Section 3.1 and Section 3.2.2 list OCR as an implemented capability while Section 5 and Section 6 state that OCR remains a placeholder; and Table 5 includes GPT-4.1-nano, which is not in the supported model lists of Section 3.2.2 or Section 5.
[§4.4] The analysis states that 'higher temperatures (0.7) maximize faithfulness' for GPT-3.5-turbo, but Table 3 shows faithfulness 0.50 at 0.7 versus 0.38 at 0.3 and 0.30 at 0.5; while technically the maximum, the difference is within the noise range already discussed in the major comments, so the wording should be appropriately hedged.
[§4.5.1] The scatter-plot analysis in Section 4.5.1 describes clusters and rankings, but because each point is a single run on 10 QA pairs, the cluster interpretation should be labeled as provisional rather than definitive.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the evaluation loop is self-referential but not equivalent-to-input; the main risks are statistical fragility and interpretive inconsistency, not circularity.

full rationale

SMARTFinRAG makes no mathematical derivation claim whose output is identical to its input. The retrieval metrics (Section 3.3.2, Eqs. 1-7) are standard IR formulas applied to relevance judgments, and the response-quality metrics (Section 3.3.3, Eqs. 8-9) are LLM-judge scores. The evaluation corpus is self-generated: Section 3.3.1 creates QA pairs from ingested documents, Section 4.2 states that only 10 of 317 QA pairs were used for response evaluation, and Section 6 concedes that the metrics rely on scalar LLM-as-a-judge scores without categorical explanations or human validation. These are serious validity limitations: the reported comparisons rest on a very small sample, a single run per configuration, and a judge that may share biases with the generators. The conclusion that hybrid retrievers significantly outperform standalone methods is also contradicted by the paper's own Table 2, where BM25 achieves higher MRR and NDCG than Hybrid Fusion. However, fragility and internal contradiction are not circularity. No fitted parameter is renamed as a prediction, no load-bearing result depends on a self-citation (the paper contains no author self-citation chain), and no equation reduces to another by construction. The self-referential design, in which the same corpus generates the questions and supplies the retrieval targets, weakens external validity but does not make the reported scores equivalent to any input parameter by definition. The derivation chain is therefore self-contained rather than circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central contribution is a software platform, so no fitted parameters or invented physical entities are present. The key intellectual assumptions are about the validity of LLM-generated QA pairs and LLM-as-judge scoring, and about the representativeness of the very small evaluation subset.

assumptions (4)

domain assumption LLM-as-judge faithfulness and relevancy scores are valid measures of response quality without human validation or categorical explanation.
Section 3.3.3 defines Faithfulness as supported claims over total claims and Relevancy as an LLM scoring function, both assigned by a GPT-4 judge. Section 6 concedes no human validation and no categorical explanations.
domain assumption LLM-generated QA pairs from ingested documents form a correct and unbiased test set for retrieval and generation.
Section 3.3.1 generates factual, numerical, and comparative questions from document nodes and filters for quality and diversity. Section 4.2 mentions manual review of the 317 pairs, but only 10 are used for response evaluation.
domain assumption Relevance labels for retrieval metrics are derived from the same source chunks the questions were generated from.
Section 3.3.1 pairs each generated question with a reference answer from the source document, so retrieval evaluation targets the generating chunks. This can inflate retrieval scores and is not validated against external annotations.
domain assumption Averages over 10 QA pairs are representative enough to support conclusions about temperature, top-p, and model ranking.
Section 4.2: 'only a small subset (10 out of 317 QA pairs) was used for response evaluation.' Tables 3, 4, and the conclusions in Sections 4.4 and 4.5 rest on this sample.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SMARTFinRAG: Interactive Modularized Financial RAG Benchmark." pith.science (2026). https://pith.science/paper/O2VL7AUY

@misc{pith2026250418024,
  author       = {Pith},
  title        = {Pith review of: SMARTFinRAG: Interactive Modularized Financial RAG Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2VL7AUY}},
  note         = {Machine review of arXiv:2504.18024}
}

read the original abstract

Financial sectors are rapidly adopting language model technologies, yet evaluating specialized RAG systems in this domain remains challenging. This paper introduces SMARTFinRAG, addressing three critical gaps in financial RAG assessment: (1) a fully modular architecture where components can be dynamically interchanged during runtime; (2) a document-centric evaluation paradigm generating domain-specific QA pairs from newly ingested financial documents; and (3) an intuitive interface bridging research-implementation divides. Our evaluation quantifies both retrieval efficacy and response quality, revealing significant performance variations across configurations. The platform's open-source architecture supports transparent, reproducible research while addressing practical deployment challenges faced by financial institutions implementing RAG systems.

Figures

Figures reproduced from arXiv: 2504.18024 by the authors.

**Figure 2.** SMARTFinRAG pipeline. Modular components process raw financial documents into [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Retriever-level ranking metrics across top- [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Relationship between temperature settings and response quality metrics (faithfulness [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: Impact of top-p sampling values on response quality across different LLM families. Models [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

**Figure 6.** Figure 6: Response quality scatter plot across all LLM backends and parameter settings. Each point [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: SMARTFinRAG upload interface and configuration sidebar. Users can upload documents, [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

**Figure 8.** Figure 8: LLM and RAG parameter settings in SMARTFinRAG. Temperature, top- [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 9.** Figure 9: Evaluation interface of SMARTFinRAG. Supports QA generation, retriever testing, and [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages

[1]

Contextcheck: A framework for evaluating llm and rag systems

Addepto. Contextcheck: A framework for evaluating llm and rag systems. https://github. com/addepto/contextcheck, 2024

work page 2024
[2]

Finqa: A dataset of numerical reasoning over financial data

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, 2021

2021
[3]

Emerging trends: A gentle introduction to rag

Kenneth Ward Church, Jiameng Sun, Richard Yue, Peter Vickers, Walid Saba, and Raman Chandrasekar. Emerging trends: A gentle introduction to rag. Natural Language Engineering, pages 1–12, 2024. doi: 10.1017/S1351324924000044

doi:10.1017/s1351324924000044 2024
[4]

Financebench: A new benchmark for financial question answering

Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023

arXiv 2023
[5]

Fiqa: Financial question answering dataset

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Fiqa: Financial question answering dataset. In WWW ’18: Companion Proceedings of the The Web Conference 2018 , 2018

work page 2018
[6]

Failsafeqa: A financial llm benchmark for ai robustness, compliance, and hallucination risks

Author Names. Failsafeqa: A financial llm benchmark for ai robustness, compliance, and hallucination risks. Blog Post, 2025

work page 2025
[7]

Ragas: Automated evaluation of retrieval augmented generation

ES Shahul, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023

arXiv 2023
[8]

FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline

Kuldeep Singh, Simerjot Kaur, and Charese Smiley. Finqapt: Empowering financial decisions with end-to-end llm-driven question answering pipeline. arXiv preprint arXiv:2410.13959 , 2024. 20

work page Pith review arXiv 2024

Show all 11 references

[9]

Pixiu: A comprehensive benchmark, instruction dataset and large language model for finance

Qianqian Xie, Weiran Han, Zeyu Chen, Ruochen Xiang, Xiang Zhang, Yifei He, et al. Pixiu: A comprehensive benchmark, instruction dataset and large language model for finance. Advances in Neural Information Processing Systems , 36, 2023

2023
[10]

Finben: A holistic financial benchmark for large language models

Qianqian Xie, Weiran Han, Xiang Zhang, Yangning Lai, Minjia Peng, Alejandro Lopez-Lira, and Jing Huang. Finben: A holistic financial benchmark for large language models. arXiv preprint arXiv:2402.12659, 2024

2024 arXiv
[11]

Finarena: A human-agent collaboration framework for financial market analysis and forecasting

Congluo Xu, Zhaobin Liu, and Ziyang Li. Finarena: A human-agent collaboration framework for financial market analysis and forecasting. arXiv preprint arXiv:2503.02692, 2025. 21

2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Contextcheck: A framework for evaluating llm and rag systems

Addepto. Contextcheck: A framework for evaluating llm and rag systems. https://github. com/addepto/contextcheck, 2024

work page 2024

[2] [2]

Finqa: A dataset of numerical reasoning over financial data

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, 2021

2021

[3] [3]

Emerging trends: A gentle introduction to rag

Kenneth Ward Church, Jiameng Sun, Richard Yue, Peter Vickers, Walid Saba, and Raman Chandrasekar. Emerging trends: A gentle introduction to rag. Natural Language Engineering, pages 1–12, 2024. doi: 10.1017/S1351324924000044

doi:10.1017/s1351324924000044 2024

[4] [4]

Financebench: A new benchmark for financial question answering

Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023

arXiv 2023

[5] [5]

Fiqa: Financial question answering dataset

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Fiqa: Financial question answering dataset. In WWW ’18: Companion Proceedings of the The Web Conference 2018 , 2018

work page 2018

[6] [6]

Failsafeqa: A financial llm benchmark for ai robustness, compliance, and hallucination risks

Author Names. Failsafeqa: A financial llm benchmark for ai robustness, compliance, and hallucination risks. Blog Post, 2025

work page 2025

[7] [7]

Ragas: Automated evaluation of retrieval augmented generation

ES Shahul, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023

arXiv 2023

[8] [8]

FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline

Kuldeep Singh, Simerjot Kaur, and Charese Smiley. Finqapt: Empowering financial decisions with end-to-end llm-driven question answering pipeline. arXiv preprint arXiv:2410.13959 , 2024. 20

work page Pith review arXiv 2024