Pith. sign in

REVIEW 3 major objections 5 minor 30 references

FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FinNLI, a new benchmark for financial natural-language inference drawn from SEC filings, annual reports, and earnings calls, shows that the best current large language model reaches only 78.62% macro F1.

desk verdict A genuinely useful first financial NLI benchmark with a credible expert test set, but the fine-tuned PLM results are confounded by systematic label noise in the training data. read the letter →

arxiv 2504.16188 v1 pith:3GBBFI3Y submitted 2025-04-22 cs.CL cs.AIcs.LG

Jabez Magomere , Elena Kochkina , Samuel Mensah , Simerjot Kaur , Charese H. Smiley This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords financialnaturallanguageinferencebenchmarkdatasetdomainshiftlargemodelsspuriouscorrelationsexpertannotationreasoninggeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FinNLI is a benchmark of 21,304 premise–hypothesis pairs for natural-language inference in finance, paired with a 3,304-instance test set that finance experts annotated with almost perfect agreement. The paper's central claim is that financial NLI is a difficult, largely unsolved task: the best fine-tuned PLM reaches 74.57% macro F1 and the best LLM reaches 78.62%, while general-domain NLI models drop roughly 20 accuracy points when they move from MNLI to FinNLI. The authors attribute the drop to domain shift—financial jargon, numerical and temporal reasoning, and specialized knowledge—and argue the dataset's design, including z-filtering against spurious correlations and premise sampling from three real financial genres, makes the difficulty genuine rather than an artifact. They also report that finance-specific instruction-tuned LLMs perform worst, which suggests narrow instruction tuning does not transfer to a novel reasoning task. A reader should care because FinNLI provides a concrete yardstick for improving how models reason about financial text, with implications for risk assessment, reporting, and fact-checking.

What carries the argument

The load-bearing object is FinNLI itself, produced by a five-stage pipeline: multi-genre premise sampling; LLM hypothesis generation with role and writing-style conditioning; z-filtering to suppress spurious correlations; iterative prompt refinement driven by a general-domain NLI model and expert feedback; and three-expert annotation with majority voting. Z-filtering is the mechanism that defends the benchmark's difficulty claim: it rejects samples whose n-grams, lexical overlap, length ratios, or financial-term density correlate too strongly with labels, so models cannot solve the task by exploiting surface shortcuts. The expert-annotated test set anchors validity, giving every reported score a human-judged target.

What would settle it

Retrain the best fine-tuned model on FinNLI's training set after finance experts re-label every instance, and compare macro F1 on the same expert-annotated test set. If expert relabeling does not move scores materially, the LLM-label noise is benign and the plateau is genuine; if scores climb past the best LLM's 78.62%, part of the claimed difficulty is an artifact of training on labels experts reject.

Watch

Extended reading notes

Core claim

FinNLI is constructed by sampling premise sentences from SEC filings, annual reports, and earnings call transcripts, then prompting GPT-4 and Llama 3.1 70B to generate one entailment, one neutral, and one contradiction hypothesis per premise under randomized roles and writing styles. A z-filtering step removes instances whose task-independent features are strongly correlated with labels, and the final test set is annotated by three finance experts per instance, with majority labels as gold. The paper claims that the resulting benchmark is valid and hard: expert labels show a Fleiss κ of 88.31%, yet the best fine-tuned PLM (RoBERTa-Large) scores 74.57% macro F1 and the best LLM (Llama 3.1 70B) scores 78.62%. General-domain NLI models drop from roughly 90% accuracy on MNLI to about 70% on FinNLI, and finance-instruction-tuned FinMA models fall to 23.20% and 29.46%, below their base Llama-2 model. The authors' conclusion is that financial NLI remains an open problem: models over-infer implicit information, struggle with financial and mathematical reasoning, and most often mistake neutral hypotheses for entailment.

Load-bearing premise

The training and development sets treat Llama 3.1 70B's automatically generated labels as gold, yet finance experts agree with those labels only 76.84% of the time, so the benchmark's measured difficulty partly rests on the assumption that this label noise is random and harmless.

Editorial extensions

If this is right

  • General-domain NLI systems cannot be assumed to transfer to finance; the roughly 20-point accuracy drop from MNLI to FinNLI means domain-specific evaluation is a prerequisite for deploying NLI in financial applications.
  • Fine-tuned PLMs remain competitive with much larger models: RoBERTa-Large at 74.57% outperforms most evaluated LLMs, so task-specific fine-tuning is still a strong route for financial NLI.
  • Finance-specific instruction tuning can hurt rather than help: FinMA 7B and 30B score below their base Llama-2 model, implying that instruction-tuning data built for financial sentiment-style tasks does not generalize to novel reasoning tasks.
  • The plateau near 78.62% leaves clear headroom, and progress on FinNLI should carry over to downstream financial question answering, fact-checking, and risk assessment, since the authors frame NLI as underpinning those tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because training and development labels come from Llama 3.1 70B with only 76.84% agreement with expert annotators, part of the measured difficulty could reflect training-label noise; re-annotating the training set by experts would show whether the 74.57% fine-tuned PLM ceiling rises.
  • Beyond the paper: the two-generator test set invites a sharper generalization check—evaluating models on expert-written hypotheses never produced by an LLM—which would separate genuine financial reasoning from generation-pattern memorization.
  • Beyond the paper: the authors' observation that Llama 3.1 70B scores similarly on both generator subsets while fine-tuned RoBERTa prefers GPT-4-written instances suggests generator bias is partly a fine-tuning effect, a claim that could be tested by controlling for premise genre and hypothesis length.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FinNLI, a multi-genre financial natural language inference dataset containing 21,304 premise–hypothesis pairs drawn from annual reports, SEC filings, and earnings call transcripts. Hypotheses are generated by GPT-4 and Llama 3.1 70B under varied roles and writing styles, passed through z-filtering to mitigate spurious correlations, and a 3,304-instance test set is labeled by finance experts. The authors evaluate zero-shot general-domain NLI models, fine-tuned PLMs, and general-purpose and finance-domain LLMs, reporting that general-domain NLI models degrade sharply on FinNLI, that the best fine-tuned PLM (RoBERTa-Large) achieves 74.57% macro F1, that the best LLM (Llama 3.1 70B) achieves 78.62%, and that instruction-tuned financial LLMs underperform general-purpose models.

Significance. If the benchmark is sound, FinNLI fills a real gap: financial NLI is an underexplored task with clear downstream relevance, and the paper’s pipeline—real-world premises, multi-model generation, artifact filtering, and expert test annotation—is a plausible template for domain-specific NLI dataset construction. The paper earns credit for a transparent evaluation with standard deviations over three runs, a high expert inter-annotator agreement (Fleiss κ = 88.31%), an explicit error analysis, and confidence-based subgroup reporting. The central difficulty and domain-shift claims are, however, currently undercut by two design issues: the train/dev labels are LLM-generated with class-dependent noise, and the test set is adversarially sampled rather than representative. Both issues are fixable and do not invalidate the expert-annotated test set, but the paper’s headline quantitative claims need to be re-examined under those controls.

major comments (3)
  1. [§3.3, §4.1, Table 4]
  2. [§3.4, §6.1, abstract]
  3. [§4.2, data availability]
minor comments (5)
  1. [§6.1 vs Table 4]
  2. [Table 7, Appendix A.1]
  3. [§7.1]
  4. [Figure 3, Appendix G.1]
  5. [Appendix G.1 vs §6.2]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: expert-annotated test labels and independent external baselines support the benchmark claims.

full rationale

The paper's central claims are evaluated against a held-out test set with expert-annotated gold labels, not against the same labels used to construct the training signal. Section 3.5 describes three professional annotators per instance with majority-label gold assignment and reports a Fleiss kappa of 88.31%, so the test set is externally grounded. Fine-tuned PLMs are trained on Llama-generated train/dev labels, but their reported test scores are computed on the expert test set, and the difficulty conclusion is independently corroborated by zero-shot general-domain NLI models (BART-Large-MNLI and DeBERTa-V3-NLI) and by several LLM baselines that were not trained on FinNLI at all. The only self-reference is that Llama 3.1 70B generated a subset of test hypotheses and also achieved the best LLM score; Section 7.1 explicitly discloses this as 'self-preference bias' and reports that Llama performs consistently across both GPT-4-generated and Llama-generated test subsets, while the test set actually contains a majority of GPT-4-generated instances. The train/dev label-noise issue (Section 4.1: R4 agreement of 76.84%, per-class agreement from 66.52% for neutral to 90.86% for contradiction) is a genuine benchmark-quality limitation, but it is not a circular derivation: no parameter is fitted to test outcomes, no conclusion reduces to an input definition, and the paper's own limitations section acknowledges that synthetic data may contain biases and hallucinations. I find no load-bearing circular step in the claimed derivation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on dataset quality, which depends on hand-chosen pipeline parameters (z-filter settings, length filters) and assumptions about generation, filtering, and annotation reliability. No new theoretical entities are postulated.

free parameters (3)
  • z-filtering top k features = 20
    Hand-chosen in Stage 3; larger k removes more features with high label correlation, changing the composition of train/dev/test.
  • z-filtering sample batch size = 200
    Hand-chosen in Stage 3; the paper notes lower batch size leads to more removed samples and lower z-scores.
  • premise length filter = 10 to 100 words
    Appendix B; sentences outside this range are excluded, shaping premise length distribution.
assumptions (4)
  • domain assumption Publicly available financial documents (annual reports, SEC filings, earnings calls) are representative samples of financial text.
    Section 3.1; the benchmark's validity depends on this representativeness for finance.
  • domain assumption Finance experts provide reliable gold labels via majority vote.
    Section 3.5; Fleiss kappa is high (88.31%), but 27.33% low-confidence instances indicate residual ambiguity.
  • ad hoc to paper Z-filtering as implemented in Wu et al. (2022) reduces spurious correlations.
    Section 3.3; the paper applies it as a black-box and assumes the reported z-score reductions translate into fewer exploitable shortcuts.
  • ad hoc to paper The general-domain NLI model used for challenge sampling selects useful difficult examples.
    Section 3.4; the test set is sampled to include both agreed and disagreed instances, which shapes difficulty and may favor certain model behaviors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking." pith.science (2026). https://pith.science/paper/3GBBFI3Y

@misc{pith2026250416188,
  author       = {Pith},
  title        = {Pith review of: FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GBBFI3Y}},
  note         = {Machine review of arXiv:2504.16188}
}
read the original abstract

We introduce FinNLI, a benchmark dataset for Financial Natural Language Inference (FinNLI) across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts. Our dataset framework ensures diverse premise-hypothesis pairs while minimizing spurious correlations. FinNLI comprises 21,304 pairs, including a high-quality test set of 3,304 instances annotated by finance experts. Evaluations show that domain shift significantly degrades general-domain NLI performance. The highest Macro F1 scores for pre-trained (PLMs) and large language models (LLMs) baselines are 74.57% and 78.62%, respectively, highlighting the dataset's difficulty. Surprisingly, instruction-tuned financial LLMs perform poorly, suggesting limited generalizability. FinNLI exposes weaknesses in current LLMs for financial reasoning, indicating room for improvement.

Figures

Figures reproduced from arXiv: 2504.16188 by the authors.

Figure 1
Figure 1. An example of NLI in financial risk assess [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the FinNLI data generation pipeline. (1) We sample premises from real-world financial documents across multiple genres. (2) Hypothesis-label pairs are generated using multiple LLMs. (3) Z-filtering (Wu et al., 2022) removes spurious correlations. (4) The prompt is refined based on feedback from a general-domain NLI model and expert curation. (5) Finally, instances correctly predicted and misclassified by… view at source ↗
Figure 3
Figure 3. Average macro F1 scores (%) for various LLMs across different prompting setups evaluated on the FinNLI [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 22 canonical work pages

  1. [1]

    Label: neutral

    Premise: And then the actual funded balances that we get from various other lines Hypothesis: The funded balances obtained from various lines are expected to increase in the next fiscal quarter. Label: neutral

    work page

  2. [2]

    O’Reilly Media, Inc

    Semantic sensitivities and inconsistent pre- dictions: Measuring the fragility of nli models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 432–444. Mohaddeseh Bastan, Mihai Surdeanu, and Niranjan Bal- asubramanian. 2022. Bionli: Generating a biomedi- cal nli...

    work page 2022

  3. [3]

    Preprint, arXiv:2407.21783

    The llama 3 herd of models. Preprint, arXiv:2407.21783. Kristina Gligoric, Myra Cheng, Lucia Zheng, Esin Dur- mus, and Dan Jurafsky. 2024. NLP systems that can’t tell use from mention censor counterspeech, but teaching the distinction helps. In Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguist...

    arXiv 2024

  4. [4]

    Assign the Label : Based on your reasoning, choose the appropriate label: ‘entailment’, ‘neutral’, or ‘contradiction’

    work page

  5. [5]

    Explain Your Reasoning : Briefly describe the thought process that led to your decision. Here are a few examples: {examples} Given the premise and hypothesis below, determine the relationship between the sentences and return ‘entailment’, ‘neutral’, or ‘contradiction’: Premise: {premise} Hypothesis: {hypothesis} Reasoning: [Describe your reasoning here] L...

    work page

  6. [7]

    Preprint, arXiv:2404.13076

    Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambi- guity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83, Ann Arbor, Michigan. Association for Computat...

    arXiv 2005

  7. [9]

    A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim ...

    arXiv 2018

  8. [10]

    more difficult

    Knowledge-infused prompting: Assessing and advancing clinical text data generation with large language models. Preprint, arXiv:2311.00287. Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023a. Fingpt: Open-source financial large language models. FinLLM at IJCAI. Linyi Yang, Eoin Kenny, Tin Lok James Ng, Yi Yang, Barry Smyth, and Ruihai Dong. 2020. ...

    arXiv 2020

Show all 30 references
  1. [12]

    Hypothesis: The company has completed the remodeling of approximately 126 stores using the new store prototype

    Premise: Wet remember, I think about 126 stores that we’ve already remodeled on that new store prototype. Hypothesis: The company has completed the remodeling of approximately 126 stores using the new store prototype. Label: entailment

  2. [13]

    Hypothesis: We do not anticipate any losses from customers being unable to make their required payments, so no allowances for doubtful accounts are maintained

    Premise: We maintain allowances for doubtful accounts for estimated losses resulting from the inability of customers to make required payments. Hypothesis: We do not anticipate any losses from customers being unable to make their required payments, so no allowances for doubtfu...

  3. [14]

    Understand the Premise : Carefully read and comprehend the premise to identify the key information it provides

  4. [15]

    Consider if it adds new information, rephrases the premise, or predicts something based on the premise

    Analyze the Hypothesis : Read the hypothesis and determine what it claims or implies. Consider if it adds new information, rephrases the premise, or predicts something based on the premise

  5. [16]

    Does the hypothesis logically follow from the premise, contradict it, or is it unrelated?

    Compare and Reason : Compare the key information from the premise and hypothesis. Does the hypothesis logically follow from the premise, contradict it, or is it unrelated?

  6. [19]

    This means the relationship between the premise and the hypothesis should involve some form of mathematical deduction

    Numerical Reasoning: If the premise contains numbers, write hypotheses that require mathematical knowledge or quantitative reasoning. This means the relationship between the premise and the hypothesis should involve some form of mathematical deduction. Example: {examples}

  7. [20]

    The hypothesis should involve an inference that requires understanding dates and the relationship between different points in time

    Temporal Reasoning: If the premise includes dates, write hypotheses that require an understanding of time. The hypothesis should involve an inference that requires understanding dates and the relationship between different points in time. Example: {examples}

  8. [21]

    Example: {examples}

    Financial Knowledge: If the premise includes financial concepts, generate hypotheses that require familiarity with financial reasoning. Example: {examples}

  9. [22]

    likely,"

    Linguistic Knowledge: Try to write hypotheses that require linguistic understanding, such as antonyms, hypernyms, metaphors, or syntactic ambiguity. Example: {examples} Important Writing Considerations: • Write all hypotheses as clear, declarative sentences. Do not phrase them...

  10. [23]

    are not phrase fragments or nonsense, which would prevent us being able to assign a label to the relationship between the premise and the hypothesis

    Verify that both the premise and hypothesis are valid and meaningful sentencese.g. are not phrase fragments or nonsense, which would prevent us being able to assign a label to the relationship between the premise and the hypothesis. We have taken steps to preprocess the sample...

  11. [24]

    The hypothesis follows logically from the information contained in the premise

    Assign one label to each of hypothesis-premise pair.: Choose one of the three: • Entailment – A relationship between two statements where the truth of the premise guar- antees the truth the hypothesis. The hypothesis follows logically from the information contained in the prem...

  12. [25]

    Depends on how confident an annotator is while defining the relation between premise and hypothesis

    Indicate your level of confidence in the annotation: Low or High. Depends on how confident an annotator is while defining the relation between premise and hypothesis

  13. [26]

    provide the rationale for the decision in free text form explaining why the instance is difficult, and/or why the confidence is low

    [Optional] Provide Comments pertaining to annotating the instance, e.g. provide the rationale for the decision in free text form explaining why the instance is difficult, and/or why the confidence is low. Raise any other concerns you have in this field, e.g. in unlikely cases ...

  14. [27]

    Premise: Before joining FMC Technologies, Mr

    Incomplete Sentence: Sentence is visibly incomplete. Example: “Premise: Before joining FMC Technologies, Mr.”

  15. [28]

    Premise: $700.0 $613.7 $492.1 95 Rest of World 108.6 96.0 93.9

    Complex Notation: Excessive numbers or financial notation making it unreadable. Example: “Premise: $700.0 $613.7 $492.1 95 Rest of World 108.6 96.0 93.9”

  16. [29]

    Net Cruise Cost less fuel expense

    Document Title or Subtitle: Appears to be a section title instead of a standalone sentence. Example: “Net Cruise Cost less fuel expense.”

  17. [30]

    For further discussion of our annual impairment test, see the Critical Accounting Policies and Estimates section

    Reference to External Text: Refers to another document section without self-contained meaning. Example: “For further discussion of our annual impairment test, see the Critical Accounting Policies and Estimates section.” Edge Cases: • Labels Are Not Symmetrical: Premise: Girl a...

  18. [2018]

    Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107–112. Pengcheng He, Jianfeng Gao, a...

    2018 arXiv

  19. [2020]

    In Proceedings of the 29th ACM International Confer- ence on Information and Knowledge Management , CIKM ’20, page 3063–3070, New York, NY , USA

    Maec: A multimodal aligned earnings confer- ence call dataset for financial risk prediction. In Proceedings of the 29th ACM International Confer- ence on Information and Knowledge Management , CIKM ’20, page 3063–3070, New York, NY , USA. Association for Computing Machinery. X...

    2023 arXiv

  20. [2022]

    Transactions of the Associa- tion for Computational Linguistics, 10:1357–1374

    Investigating reasons for disagreement in natu- ral language inference. Transactions of the Associa- tion for Computational Linguistics, 10:1357–1374. Rasmus Jørgensen, Oliver Brandt, Mareike Hartmann, Xiang Dai, Christian Igel, and Desmond Elliott. 2023. Multifin: A dataset f...

    2023

  21. [2023]

    Nature, 623(7987):493–498

    Role play with large language models. Nature, 623(7987):493–498. Chaitanya Shivade et al. 2019. Mednli-a natural lan- guage inference dataset for the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. ...

    2019 arXiv

  22. [2024]

    Preprint, arXiv:2404.14219

    Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219. D Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063. Erik Arakelyan, Zhaoqi Liu, and Isabelle Augenstein

    2019 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.