Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Trained internal probes are the only confidence estimators whose scores can be thresholded into a controlled error rate on financial chart and document understanding, and the grounding-aware BICR alone also lowers its confidence when the…

desk verdict Solid empirical paper on confidence estimation for financial LVLMs; the central calibration-vs-ranking claim holds, but the sample-size inconsistency and single-judge correctness labels need fixing before I'd trust the deployment numbers. read the letter →

arxiv 2608.06532 v1 pith:MTYGZV6Y submitted 2026-08-06 cs.CL

Reza Khanmohammadi , Simerjot Kaur , Charese H. Smiley , Ivan Brugere , Mohammad M. Ghassemi This is my paper · ORCID
classification cs.CL
keywords confidenceestimationcalibrationvision-languagemodelsfinancialdocumentunderstandingselectivepredictiondeferralgroundingout-of-distributiontransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Confidence scores for financial vision-language models are only operationally useful if they can be thresholded into an error-controlled automation policy, and the paper argues that among seven estimators tested across five models, only two trained internal probes deliver that property. The inference-only baselines—verbalized self-report, self-probing, and prompt ensembles—rank correct above incorrect answers almost as well but are badly overconfident, with calibration error far above what a threshold can tolerate. The trained probes BICR and InternalInspector, transferred zero-shot from natural images to financial charts, tables, and documents, are the only estimators whose scores map to a controlled error rate, and BICR alone lowers its confidence when the model answers without using the figure. Read as deferral under an error budget, how much can be safely automated is set first by the base model's competence, so the strongest models can clear a real share of easy chart questions while nearly nothing survives a strict budget on the hardest document tasks. The paper is a caution to anyone who uses a raw confidence readout to decide whether a financial AI answer can be acted on.

What carries the argument

The argument is carried by two trained internal probes plus the selective-prediction framing. BICR (blind-image contrastive ranking) trains a confidence probe on the contrast between hidden representations under the real figure and under a blanked or deterministic random image, so its score encodes whether the answer used the figure; this is the only estimator here trained with a grounding signal. InternalInspector learns contrastively across attention, feed-forward, and activation states from all layers and supplies the other consistently well-calibrated score. Both are trained only on GQA natural images and applied to finance without adaptation, so the study measures out-of-distribution transfer. The deployment analysis uses bounded selective prediction: a risk-coverage curve is traced for each method and the safe yield is the largest automatable fraction at a 5% or 20% error budget, which is why calibration, not AUROC, is the property that makes a threshold meaningful.

What would settle it

An independent annotation of several hundred financial VQA answers from the same five models, deliberately oversampling numerical and OCR items, would settle the labeling question: if human labels disagree with the automated judge on a systematic class, every ECE, AUROC, and safe-yield figure shifts. The grounding claim can be tested separately by replacing first-token invariance with full-answer invariance on human-labeled ungrounded answers, and checking whether BICR's confidence drops specifically on those answers.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated in its own terms, is that the scarce property in confidence estimation for financial LVLMs is calibration, not ranking. Across four financial conditions, the inference-only baselines P(True), Self-Probing, and Prompt Ensemble produce competitive AUROC values (up to $0.774$ on the bilingual document condition) but ECE between $0.25$ and $0.58$, and Self-Probing labels as many as $93.8\%$ of its own errors with confidence above $0.8$, so no fixed threshold yields a controlled error rate. Trained internal probes, all trained only on natural-image GQA and evaluated on finance with no adaptation, hold the best calibration on every condition—BICR at ECE $8.6$ on FinMME and InternalInspector at $4.3$ on FinChart-Bench—making them the only estimators whose scores can be thresholded into a bounded-error deferral policy. On the subpopulation where replacing the chart with a random image leaves the first generated token unchanged, BICR alone drops its mean confidence sharply ($0.492$ versus $0.721$ for P(True) on FinMME) while remaining well calibrated, separating detected non-grounding from a fluent guess. The paper also finds that safe yield tracks base-model competence ($r = 0.68$), so the confidence layer adds deployable yield chiefly where the model is weak.

Load-bearing premise

The load-bearing premise is that the single automated judge used to label every answer correct or incorrect is accurate enough on the hard slices—especially numerically exact and OCR answers—to support all the calibration, ranking, and safe-yield numbers; the paper validates the judge on a 100-item audit with 94.8% agreement, which is a thin base for the strongest claims.

Editorial extensions

If this is right

  • A practitioner who gates automation on a confidence threshold should use BICR or InternalInspector, because only these two trained probes map a fixed cutoff to a controlled error rate across all four financial conditions.
  • Raw verbalized and logit self-reports should not be thresholded for financial deferral: they rank well but are overconfident enough that a cutoff at 0.8 can correspond to error rates far above budget.
  • Deferral yield is bounded by base-model competence, so confidence layers add automation mostly on weak models; the strongest models on easy chart conditions can clear substantial workload while even the best estimator clears almost nothing on the hardest document and Chinese-language conditions at a strict 5% budget.
  • Reliability must be reported per model and per task family: no estimator leads more than eight of twenty (model, condition) cells, and pooled bilingual numbers can create an apparent language robustness that disappears within individual models.
  • On the figure-invariant subpopulation, BICR's low, well-calibrated confidence identifies fluent answers produced without reading the chart, which is the failure mode most dangerous for a financial reviewer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the out-of-distribution gap is a lower bound would be to train BICR and InternalInspector on a modest slice of financial VQA data; if calibration improves further, the current transfer-only numbers understate what a deployment should expect from in-domain training.
  • The first-token-invariance diagnostic could be generalized to full-answer invariance; if the grounding signal holds under that stricter test, BICR-style scores could be used to flag ungrounded open-ended generations, the task family where every estimator studied here is weakest.
  • The safe-yield framing could be turned into a cost-of-error knob: a bank that assigns a dollar cost to an automated wrong answer can invert the risk-coverage curve to choose the automation threshold explicitly, rather than using a fixed 5% or 20% budget.
  • The bilingual composition-artifact result suggests that any pooled leaderboard, including the headline tables in this paper, should be read with per-model operating points in hand before deployment; the paper's own pooling shows how aggregation can hide both degradation and reachable yield.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper evaluates seven confidence estimators for financial vision-language models: three inference-only baselines (P(True), Self-Probing, Prompt Ensemble) and four trained internal probes (SAPLMA, CCPS, InternalInspector, BICR). The estimators are trained only on GQA natural-image data and evaluated without adaptation on four financial conditions from FinMME, FinChart-Bench, and the bilingual MME-Finance benchmark, across five open-weight LVLMs. The central claims are threefold: (1) the scarce property is calibration rather than ranking, with the trained probes BICR and InternalInspector being the only estimators whose scores can be thresholded into a controlled error rate; (2) confidence reliability is structured by model and task type, with no estimator dominating more than eight of twenty (model, condition) cells; and (3) safe-yield under bounded-error deferral is set first by base-model competence, with the confidence layer adding yield mostly where the model is weak. The paper also claims that BICR uniquely lowers confidence on figure-invariant answers, separating detected non-grounding from a fluent guess. The protocol is generally clean, and the paper includes several honest null results and limitation statements.

Significance. If the findings hold, this is a useful and timely contribution to selective prediction for financial LVLMs. The out-of-distribution protocol is well designed: all trainable probes are trained only on GQA and evaluated on unseen financial figures, with seed-averaging and per-cell significance tests. The framing in terms of bounded-error deferral, rather than aggregate accuracy, is appropriate for the stated deployment context. The paper also deserves credit for reporting a null result on FinChart-Bench for the grounding diagnostic, for using paired bootstrap and DeLong tests, and for openly acknowledging limitations. The main risk to the significance of the results is that every calibration, discrimination, and safe-yield number depends on binary correctness labels from a single gpt-5-mini judge validated on only 100 items, and the limitation statement itself concedes that the judge is imperfect for numerically exact answers. The BICR grounding finding also requires careful interpretation because BICR's training objective explicitly contrasts real and blanked figures, so part of the observed behavior is definitional rather than an emergent transfer result.

major comments (3)
  1. [Section 3, 'Correctness labels'] All calibration, discrimination, and safe-yield metrics in Tables 1, 3, and 4 are computed against binary correctness labels assigned by a single gpt-5-mini judge, validated by a 100-item audit stratified toward numerically-exact and OCR answers. That audit does not cover Chinese-language outputs, true/false items, captioning, or open-ended QA, which are substantial parts of the evaluation. Systematic judge error on any of those slices could shift the ECE, Brier, AUROC, and safe-yield values and could change which estimators appear thresholdable, because the headline claim that only BICR and InternalInspector support controlled error rates rests on these labels. The limitation statement's concession that the judge is 'imperfect for numerically exact answers' is also in tension with the audit's claim of no degradation on the numerical/OCR slice. Please report per-slice judge agreement on a larger stratified sample spanning models, languages, and task types, or re-run the key tables under multiple judges or a label-noise model.
  2. [Section 5 and Table 4, grounding analysis] The claim that BICR uniquely lowers its confidence on figure-invariant answers is partly definitional: BICR is trained by contrasting representations under the real figure against those under a blanked figure, so on the first-token-invariant subpopulation, where the model's output does not change when the figure is replaced, the estimator is expected by construction to produce a lower score. The paper's finding that this transfers to financial conditions is still informative, but the text should separate the definitional component from the transfer component. Please report how BICR's confidence correlates with an independent grounding measure, such as full-answer invariance or human judgments of whether the figure was used, and show calibration on that subset against a baseline that does not use blank-image contrast.
  3. [Tables 1 and 3] The pooled headline numbers in Table 1 and the safe-yield numbers in Table 3 are presented without confidence intervals or significance tests for the key differences. Some comparisons are close, for example the ECE difference between BICR and InternalInspector on MME-Finance-EN (13.8 vs. 12.4) and the AUROC differences on FinChart-Bench, and the conclusion that only the trained probes are thresholdable depends on the magnitude of these gaps. Please report bootstrap confidence intervals for the main ECE, AUROC, and safe-yield comparisons, or at least for the comparisons that drive the central claims.
minor comments (4)
  1. [Table 2] The DeLong p-values are printed as 0.0e+00 and 3.0e-06; values that underflow should be reported with an inequality such as p < 1e-300 rather than as exactly zero.
  2. [Table 4] The table includes unexplained 'verdict: SUPPORTED' and 'ref' entries; these should be removed or defined in the caption, since they are not part of a standard results table.
  3. [Figure 2] The per-task heatmap contains many small two-number cells that are difficult to read at print size; a larger figure or separate panels per task family would improve legibility.
  4. [Section 3, 'Correctness labels'] The sentence describing the 100-item audit says it was 'stratified toward numerically-exact and OCR answers' and that there was 'no degradation on the hard numerical/OCR slice relative to the rest,' but the size of the 'rest' is not stated; please specify how many non-numerical/OCR items were included.

Circularity Check

1 steps flagged · score 2.0 of 10

The main transfer evaluation is self-contained, but BICR's grounding back-off is a designed property presented as a finding.

  1. self definitional [Section 2 (Related Work, BICR description) and Section 5 (Deployment, figure-invariant subpopulation)]
    "BICR [17] additionally contrasts the representation under the real figure against one under a blanked figure, so its score reflects whether the answer used the image, the only estimator here trained with such a grounding signal. ... A thresholded BICR score is safe to act on because it backs off when the figure is ignored. Replacing each chart with a deterministic random image and isolating the figure-invariant subpopulation, the samples whose first generated token does not change under the swap and on which the model demonstrably ignores the figure"

    BICR's confidence is defined as a contrast between real and blanked/random-image representations, so on the figure-invariant subpopulation, which is itself defined by first-token invariance under the same image swap, the score is constructed to move low. The finding that BICR backs off on non-grounded answers is therefore a restatement of its training objective rather than an independent empirical law. The genuinely empirical part is that this designed behavior transfers out-of-distribution to financial charts and that the low confidence is calibrated on a majority-incorrect subpopulation; those parts are not circular, which keeps the overall circularity partial.

full rationale

The central evaluation is self-contained: five open-weight LVLMs and seven estimators are scored on FinMME, FinChart-Bench, and MME-Finance under a transfer-only protocol, with all trainable probes fit on GQA and frozen for finance. The main headline—inference baselines rank competitively but are too overconfident to threshold, while BICR and InternalInspector map to controlled error rates—is an empirical comparison against external benchmarks, so it does not reduce to its inputs. The one partial circularity is the grounding claim: BICR's score is defined by contrasting real versus blanked/random-image representations, and the figure-invariant subpopulation is defined by first-token invariance under that same swap, so BICR's lower confidence on non-grounded answers is a designed property, not an independent discovery. Its out-of-distribution transfer and calibration on a majority-incorrect subpopulation are still empirical and keep the circularity partial. The Section 7 limitation that the single gpt-5-mini judge is 'imperfect for numerically exact answers' is a correctness/validity concern, not a circularity, because the labels are external to the estimator equations; the same holds for the 100-item audit. Self-citations to prior work by the authors are not load-bearing here: BICR, CCPS, and the judge protocol are re-evaluated in this paper rather than assumed.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and fits no parameters to the financial test data. Its empirical claims rest on standard statistics, public benchmarks, a model judge, and a behavioral proxy for grounding. The only data-dependent choices are the per-model thresholds used to report safe yield, which are selected on the evaluation set itself.

free parameters (1)
  • per-model confidence threshold = chosen on the evaluation set
    Safe-yield operating points are selected per model on the same financial test data used to report yield, so reported automation fractions are in-sample optima rather than held-out estimates. The paper discloses this in Limitations.
assumptions (4)
  • domain assumption gpt-5-mini judge provides correct binary correctness labels
    All calibration and deferral metrics depend on these labels; validated only by a 100-item audit (Section 3, Correctness labels).
  • domain assumption First-token invariance under a deterministic random image identifies figure-grounding
    Used to define the figure-invariant subpopulation in Section 5; acknowledged as a proxy in Limitations.
  • domain assumption FinMME, FinChart-Bench, and MME-Finance gold answers are accurate
    The paper relies on benchmark labels as ground truth for correctness (Section 3).
  • standard math Standard statistical procedures (DeLong test, Friedman test, paired bootstrap) are validly applied
    Used for significance claims in Table 2 and Table 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding." pith.science (2026). https://pith.science/paper/MTYGZV6Y

@misc{pith2026260806532,
  author       = {Pith},
  title        = {Pith review of: Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MTYGZV6Y}},
  note         = {Machine review of arXiv:2608.06532}
}
read the original abstract

LVLMs are increasingly used to read financial charts, tables, and documents, where a single misread figure can move a decision and the most authoritative-looking answer is sometimes one the model produced without reading the exhibit. The operational question is therefore trust, not accuracy: which answers can be acted on, and which escalated to a reviewer. We evaluate seven confidence estimators, three inference-only and four trained internal probes, across five open-weight LVLMs and four conditions from three financial visual question-answering benchmarks, one bilingual; every probe is trained only on natural images and applied to finance without adaptation, so the results measure out-of-distribution transfer. Three findings hold. First, the scarce property is calibration, not ranking: the inference baselines rank correct above incorrect answers competitively but are badly overconfident, calibration error far above what a threshold can tolerate, and only the trained probes produce a thresholdable score. Second, reliability is structured rather than global, along two axes a practitioner can read directly: the best estimator shifts with both model and task, none leading more than eight of twenty (model, condition) cells, and a controlled bilingual contrast exposes an apparent language robustness as a composition artifact that dissolves once models are read one at a time. Third, cast as deferral under an error budget, how much can be safely automated is set first by the model's competence and only narrowed by its confidence, so deferral clears a real share of the easiest condition and almost none of the hardest, near zero at a strict 5% budget. Two trained probes carry the calibration a deferral policy needs, and among them only the grounding-aware one lowers its confidence on answers a model gives without using the figure, separating detected non-grounding from a fluent guess.

Figures

Figures reproduced from arXiv: 2608.06532 by the authors.

Figure 1
Figure 1. Reliability diagrams pooled across the five LVLMs (dashed line is perfect calibration). Marker size is proportional to [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Per financial task-type confidence quality (twelve largest native task and format categories pooled across the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Risk-coverage curves pooled across five LVLMs (lower is better). Dotted lines mark the 5% and 10% error budgets. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Attainable confidence separability rises with base [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 11 canonical work pages

  1. [1]

    Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When It’s Lying. arXiv:2304.13734 [cs.CL] https://arxiv.org/abs/2304.13734

    arXiv 2023

  2. [2]

    Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mo- han, Jianfeng He, Ming Jin, Chang-Tien Lu, and Lifu Huang. 2024. InternalInspec- tor 𝐼 2: Robust Confidence Estimation in LLMs through Internal States. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). ...

    doi:10.18653/v1/2024.findings-emnlp.751 2024

  3. [3]

    Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, and Muhammad Abdul-Mageed. 2024. FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models. InFindings of the Association for Computational Lin- guistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 1...

    doi:10.18653/v1/2024.findings-acl.774 2024

  4. [4]

    Debarpan Bhattacharya, Apoorva Kulkarni, and Sriram Ganapathy. 2025. FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suz...

    work page 2025

  5. [5]

    Yunkai Dang, Yifan Jiang, Yizhu Jiang, Anqi Chen, Wenbin Li, and Yang Gao. 2026. Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models. arXiv:2604.17274 [cs.CV] https://arxiv.org/abs/2604.17274

    arXiv 2026

  6. [6]

    Ran El-Yaniv and Yair Wiener. 2010. On the Foundations of Noise-free Selective Classification.J. Mach. Learn. Res.11 (Aug. 2010), 1605–1641

    work page 2010

  7. [7]

    Ziliang Gan, Dong Zhang, Haohan Li, Yang Wu, Xueyuan Lin, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, Rongjunchen Zhang, and Yong Dai. 2025. MME- Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning. InProceedings of the 33rd ACM International Conference on Multimedia (Dublin, Ireland)(MM ’25). Association for Computing Mac...

    arXiv 2025

  8. [8]

    Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. InProceedings of the 31st International Conference on Neural Information Processing Systems(Long Beach, California, USA)(NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 4885–4894

    work page 2017

Show all 34 references
  1. [9]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning - Volume 70(Sydney, NSW, Australia)(ICML’17). JMLR.org, 1321–1330

    2017

  2. [10]

    Dongxin Guo, Jikun Wu, and Siu Ming Yiu. 2026. FinGround: Detect- ing and Grounding Financial Hallucinations via Atomic Claim Verification. arXiv:2604.23588 [cs.AI] https://arxiv.org/abs/2604.23588

    2026 arXiv

  3. [11]

    Tiansheng Hu, Tongyan Hu, Liuyang Bai, Yilun Zhao, Arman Cohan, and Chen Zhao. 2025. FinTrust: A Comprehensive Benchmark of Trustworthiness Eval- uation in Finance Domain. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christ...

    2025 doi

  4. [12]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. 2019. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6693–6702. doi:10.1109/CVPR.2019.00686

    2019

  5. [13]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. FinanceBench: A New Benchmark for Financial Question Answering. arXiv:2311.11944 [cs.CL] https://arxiv.org/abs/2311.11944

    2023 arXiv

  6. [14]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran- Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tris- tan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav ...

    2022 arXiv

  7. [15]

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. 2024. Calibration-Tuning: Teaching Large Lan- guage Models to Know What They Don’t Know. InProceedings of the 1st Workshop on Uncertainty-A ware NLP (UncertaiNLP 2024), Raúl...

    2024

  8. [16]

    Ghassemi

    Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur, Charese Smiley, Ivan Brugere, Kundan S Thind, and Mohammad M. Ghassemi. 2026. How Reliable are Con- fidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains. InProceedings of the 19th Conferenc...

    2026

  9. [17]

    Smiley, Ivan Brugere, Kundan Thind, and Mohammad M

    Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur, Charese H. Smiley, Ivan Brugere, Kundan Thind, and Mohammad M. Ghassemi. 2026. Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking. arXiv:2605.10893 [cs.CL] https://arxiv.org/abs/2605.10893

    2026 arXiv

  10. [18]

    Ghassemi

    Reza Khanmohammadi, Erfan Miahi, Mehrsa Mardikoraem, Simerjot Kaur, Ivan Brugere, Charese Smiley, Kundan S Thind, and Mohammad M. Ghassemi. 2025. Calibrating LLM Confidence by Probing Perturbed Representation Stability. In Proceedings of the 2025 Conference on Empirical Method...

    2025 doi

  11. [19]

    Benjamin Kompa, Jasper Snoek, and Andrew L. Beam. 2021. Second Opinion Needed: Communicating Uncertainty in Medical Machine Learning.npj Digital Medicine4, 1 (2021), 4. doi:10.1038/s41746-020-00367-3

    2021 doi

  12. [20]

    Qing Li, Jiahui Geng, Chenyang Lyu, Derui Zhu, Maxim Panov, and Fakhri Karray. 2024. Reference-free Hallucination Detection for Large Vision-Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2024, Confidence Estimation for Financial Vision-Lan...

    2024

  13. [21]

    Lin Long, Changdae Oh, Seongheon Park, and Sharon Li. 2026. Under- standing Language Prior of LVLMs by Contrasting Chain-of-Embedding. arXiv:2509.23050 [cs.LG] https://arxiv.org/abs/2509.23050

    2026

  14. [22]

    Junyu Luo, Zhizhuo Kou, Liming Yang, Xiao Luo, Jinsheng Huang, Zhiping Xiao, Jingshu Peng, Chengzhong Liu, Jiaming Ji, Xuanzhe Liu, Sirui Han, Ming Zhang, and Yike Guo. 2025. FinMME: Benchmark Dataset for Financial Multi- Modal Reasoning Evaluation. InProceedings of the 63rd A...

    2025

  15. [23]

    Dong Shu, Haoyang Yuan, Yuchen Wang, Yanguang Liu, Huopu Zhang, Haiyan Zhao, and Mengnan Du. 2025. FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models. arXiv:2507.14823 [cs.CV] https: //arxiv.org/abs/2507.14823

    2025 arXiv

  16. [24]

    Mainak Singha. 2025. Detecting AI Hallucinations in Finance: An Information- Theoretic Method Cuts Hallucination Rate by 92%. arXiv:2512.03107 [cs.LG] https://arxiv.org/abs/2512.03107

    2025

  17. [25]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine- Tuned with Human Feedback. InProceedin...

    2023 doi

  18. [26]

    Xingqi Wang, Yiming Cui, Xin Yao, Shijin Wang, Guoping Hu, and Xiaoyu Qin

  19. [27]

    Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. 2022. Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly. InComputer Vision – ECCV 2022: 17th European Conference, Tel A viv, Israel, ...

    2022

  20. [28]

    Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim

  21. [29]

    Wenyi Xiao, Xinchi XU, and Leilei Gan. 2026. VL-Calibration: Decoupled Confi- dence Calibration for Large Vision-Language Models Reasoning. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Maria Liakata, Viviane...

    2026

  22. [30]

    InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.)

    Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Comp...

    2025 doi

  23. [31]

    Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. 2024. VL-Uncertainty: Detect- ing Hallucination in Large Vision-Language Model via Uncertainty Estimation. arXiv:2411.11919 [cs.CV] https://arxiv.org/abs/2411.11919

    2024 arXiv

  24. [32]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv:2306.13063 [cs.CL] https://arxiv.org/abs/ 2306.13063

    2024 arXiv

  25. [34]

    Thomas P Zollo and Richard Zemel. 2025. Confidence Calibration in Vision- Language-Action Models. arXiv:2507.17383 [cs.RO] https://arxiv.org/abs/2507. 17383

    2025

  26. [2025]

    arXiv:2509.17481 [cs.CV] https://arxiv.org/abs/2509.17481

    ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understanding. arXiv:2509.17481 [cs.CV] https://arxiv.org/abs/2509.17481

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.