Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States

T0 review · 3 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Among answers an LLM gives with full self-consistency, a linear probe on its residual stream still finds most of the wrong ones that surface signals miss.

desk verdict Clean multi-model result: residual probes keep 0.68–0.77 AUROC on confident FinQA errors while cheap output baselines collapse to ~0.55–0.63; controls are stronger than average for the subfield. read the letter →

arxiv 2607.11414 v1 pith:BULA2RWJ submitted 2026-07-13 cs.CL

Richard Zhe Wang This is my paper
classification cs.CL
keywords hallucinationdetectionlinearprobesfinancialQAresidualstreamself-consistencyFinTAT-QAtriage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In financial question answering, the most damaging mistakes are confident ones: the model answers with high certainty and is still wrong. This paper shows that those confident errors leave a usable trace in the model’s internal residual stream. A simple linear probe trained on mid-to-late-layer activations separates correct from incorrect answers even when the model’s own log-probabilities and self-assessment of correctness have collapsed toward chance. The advantage is largest on the harder FinQA benchmark, where 15–23 % of fully self-consistent answers are wrong, and it holds across three 8–9 B model families. The practical claim is that the same probe can rank answers for human review under a fixed review budget and catch substantially more errors than output-only methods, especially for models that over-commit.

What carries the argument

A logistic-regression probe that reads the mean residual-stream vector over the generated answer tokens at a fixed relative depth of two-thirds of the network; the probe is trained with document-grouped, class-balanced cross-validation against mechanically graded correctness.

What would settle it

Re-run the same probe-versus-baseline comparison after replacing the automatic grader with a fully human-audited label set (or after tightening the tolerance so that residual label noise falls below 1 %) and check whether the AUROC gap on the confident cell shrinks below statistical noise.

Watch

Extended reading notes

Core claim

Among answers on which eight temperature-0.7 resamples agree unanimously, a mean-pooled linear probe on the residual stream at two-thirds network depth recovers 0.68–0.77 AUROC for correctness on FinQA, while the best deployment-cheap output baselines (token log-probabilities or P(True)) fall to 0.55–0.63, a gain of roughly 0.13–0.15 AUROC that is replicated across Qwen3-8B, Llama-3.1-8B and Gemma-2-9B.

Load-bearing premise

The mechanical grader and the eight-sample self-consistency labels are accurate enough that residual noise of a few percent does not reverse the ranking of the probe against the output baselines.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether linear probes on residual-stream activations can detect confidently wrong answers in financial numeric QA better than deployment-cheap output signals. On FinQA and TAT-QA, with Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B, confidence is defined by 8/8 self-consistency of temperature-0.7 resamples and correctness by a mechanical grader (Eq. 1). Among unanimous confident answers on FinQA, 5–19% are wrong; a mean-pooled probe at 2/3 depth attains 0.68–0.77 AUROC while the best baselines (token log-probabilities, P(True)) fall to 0.55–0.63, a gain of +0.13 to +0.15 AUROC replicated across families (Table 3). The advantage shrinks or vanishes among unsure answers and does not transfer across the two benchmarks. Validity controls include document-grouped 5-fold CV, random-init placebos (~0.63 AUROC), layer-0 baselines, abstention ablation, and an LLM-judge audit of residual label noise. The authors position the probe as a low-cost triage component for human review under bounded capacity.

Significance. If the central comparison holds, the work supplies a concrete, deployment-relevant finding for high-stakes financial LLM systems: internal-state probes retain discriminative power precisely where output-level uncertainty signals collapse—among answers the model itself treats as certain. The multi-family replication, document-grouped CV, random-init placebo, and abstention ablation are stronger validity controls than are common in this literature and make the +0.13–0.15 AUROC gap on FinQA credible. The operational triage numbers (Table 5) and the simple transferable design rule (mean-pool at 2/3 depth) give the result practical value for self-hosted models. The contribution is incremental rather than foundational—linear correctness probing is established—but the confidence-stratified, finance-specific evaluation against fair baselines is new and useful.

major comments (3)
  1. §3, Eq. (1) and the LLM-judge audit: residual 4–6% label noise is acknowledged and argued to attenuate rather than invent AUROC gaps, but the paper never reports a noise-injection or label-flip sensitivity that would show the Table 3 ranking is preserved under the estimated noise rate. A short simulation (flip 5% of labels in both directions and recompute probe vs. best-baseline AUROC on the confident cell) would close this load-bearing premise.
  2. §4.2 / Table 3: approximate confidence intervals (±0.02 to ±0.04) are mentioned in the text for the key deltas, but no standard errors, bootstrap intervals, or paired tests appear in the tables. Given the modest wrong-answer counts in some cells (e.g., Llama FinQA confident-wrong n=329; TAT-QA cells even smaller), formal uncertainty quantification is needed before the cross-family claim can be treated as settled.
  3. §5.2 and the transfer result: the probe is task-specific (0.07–0.16 AUROC drop FinQA↔TAT-QA) and the manuscript evaluates only two numeric-QA benchmarks with prompted CoT and 8–9B models. The operational claim that probing is a ready triage mechanism for “financial applications” therefore rests on a narrow regime; either the claim should be scoped more tightly or a third task / larger model should be added to bound generality.
minor comments (5)
  1. Abstract and Table 1: the abstract states “15–23% are wrong on FinQA” among confident answers, while Table 1 reports reweighted population shares of 19%, 5%, and 14% for the three models. Align the abstract wording with the table (or clarify that 15–23% is a different conditional).
  2. Figure 1 caption and §5.1: precision/recall curves are population-reweighted; a brief note in the caption that prevalence is the reweighted confident-wrong rate would help readers interpret the dotted line.
  3. §4.1: the logistic-regression regularization C=0.1 and class-balancing are stated but not ablated; a one-sentence sensitivity note would strengthen the “fixed a priori” design claim.
  4. References and related work: Mirajkar (2025) is correctly distinguished, but a short explicit comparison of synthetic vs. real-filing transfer would make the novelty claim sharper.
  5. Typos / consistency: “placebos, which place the reference AUROC at ≈0.63” (§1 contribution 4) vs. “0.625” in Figure 2; pick one reported value.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: out-of-fold probe vs. fixed baselines on held-out documents; self-consistency is only a stratifier, not a training target or fitted prediction.

full rationale

This is a standard empirical ML evaluation paper, not a first-principles derivation. Correctness labels come from a mechanical grader (Eq. 1) against gold answers; behavioral confidence is an independent 8-sample self-consistency score (Eq. 2) used only to stratify the 2×2 grid, not as the probe’s training target. The linear probe (Eq. 4) is fit by class-balanced logistic regression on residual-stream activations and scored strictly out-of-fold under document-grouped CV, so reported AUROCs are not forced by construction from the fitted weights. Baselines (token log-probs, P(True)) are computed from outputs without using the probe’s labels. Validity controls (random-init placebo ≈0.63, layer-0 baseline, permuted-label control, abstention ablation, specialist-probe check) further separate surface features from the claimed internal signal. There are no self-citations by the sole author, no uniqueness theorems, no ansatz imported via citation, and no renaming of a known result as a new prediction. The central claim (probe advantage of +0.13–0.15 AUROC among 8/8-confident FinQA answers) is an empirical comparison, not a quantity that reduces to its inputs by definition.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical performance comparison; it rests on a small set of free design choices (regularization, layer depth, sample count, grading tolerance) and standard domain assumptions about self-consistency and linear probing rather than on newly postulated physical or mathematical entities.

free parameters (4)
  • logistic-regression C = 0.1
    ℓ2 regularization strength fixed at 0.1 for all probes; chosen by hand rather than cross-validated per fold.
  • probe layer depth = 2/3 depth
    Fixed a priori at ⌊2L/3⌋; the layer sweep later shows it is near-optimal but the number itself is a free design choice.
  • self-consistency sample size k = 8
    Number of temperature-0.7 resamples used both to define the confident stratum and as a non-deployable reference signal.
  • grading relative tolerance τ = 0.02
    2 % relative error under five scale conventions defines the binary correctness label that every AUROC is computed against.
assumptions (3)
  • domain assumption Self-consistency across k independent samples is a valid behavioral proxy for model confidence.
    Invoked in §3 to define the confident/unsure split; standard in the uncertainty literature but still an unproved modeling choice for financial numeric answers.
  • domain assumption Answer correctness is linearly decodable from residual-stream activations.
    Taken as a premise from prior probing work (§2) and used to justify the logistic probe architecture.
  • standard math Document-grouped cross-validation eliminates train–test leakage from shared source filings.
    Standard grouped CV; the paper quantifies the leakage that would otherwise inflate AUROC by up to 0.04 (§4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States." pith.science (2026). https://pith.science/paper/BULA2RWJ

@misc{pith2026260711414,
  author       = {Pith},
  title        = {Pith review of: Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BULA2RWJ}},
  note         = {Machine review of arXiv:2607.11414}
}
read the original abstract

Large language models (LLMs) in financial applications fail most consequentially when they are confidently wrong. Hedged, uncertain answers invite scrutiny, whereas confident errors silently degrade downstream decisions without warning. We ask how reliably such confidently wrong answers, or confident hallucinations, can be detected from a model's internal activations, and whether those activations carry information beyond its observable outputs. We train linear probes on the residual stream and evaluate them on two established question-answering (QA) benchmarks built from real filings, FinQA and TAT-QA. Behavioral confidence is measured as the agreement among eight resampled answers to the same question, and probe effectiveness is compared against baselines, such as token log-probabilities and the model's own True/False self-assessment of its answer. Our findings show that among confident answers, those for which all eight resamples agree, 15-23% are wrong on FinQA. There the probes have a significant advantage over baseline methods in detecting hallucinations, holding 0.68-0.77 AUROC while the best baselines fall to 0.55-0.63, across Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B. Our results suggest that probing can be a cost-effective triage mechanism for routing LLM answers to human review and quality control procedures in high-stakes financial applications.

Figures

Figures reproduced from arXiv: 2607.11414 by the authors.

Figure 1
Figure 1. Precision (left) and recall (right) for retrieving confident-wrong answers vs. review budget, among unanimously [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Probe AUROC by relative network depth (FinQA). Left: answer-conditioned probes (mean over generated tokens). [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 23 linked inside Pith

  1. [1]

    Aizierjiang Aiersilan. 2026. Hallucination Is Linearly Decodable from Mid-Layer Hidden States in Quantized LLMs.arXiv preprint arXiv:2606.02628(2026)

    arXiv 2026

  2. [2]

    Guillaume Alain and Yoshua Bengio. 2016. Understanding Intermediate Layers Using Linear Classifier Probes.arXiv preprint arXiv:1610.01644(2016)

    arXiv 2016

  3. [3]

    Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When It’s Lying. InFindings of the Association for Computational Linguistics: EMNLP

    2023

  4. [4]

    Yonatan Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Ad- vances.Computational Linguistics48, 1 (2022), 207–219. doi:10.1162/coli_a_00422

    doi:10.1162/coli_a_00422 2022

  5. [5]

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering Latent Knowledge in Language Models Without Supervision. InInternational Conference on Learning Representations (ICLR). arXiv:2212.03827

    arXiv 2023

  6. [6]

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. INSIDE: LLMs’ Internal States Retain the Power of Hallucina- tion Detection. InInternational Conference on Learning Representations (ICLR). arXiv:2402.03744

    arXiv 2024

  7. [7]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2021

  8. [8]

    Eunbyeol Cho, Yunseung Lee, Mirae Kim, Jeewon Yang, Youngjun Kwak, and Edward Choi. 2026. K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Finance.arXiv preprint arXiv:2605.29523(2026)

    arXiv 2026

Show all 38 references
  1. [9]

    Hoagy Cunningham, Jerry Wei, Zihan Wang, et al. 2026. Constitutional Classi- fiers++: Efficient Production-Grade Defenses against Universal Jailbreaks.arXiv preprint arXiv:2601.04603(2026)

    2026

  2. [10]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting Hallucinations in Large Language Models Using Semantic Entropy.Nature630, 8017 (2024), 625–630. doi:10.1038/s41586-024-07421-0

    2024 doi

  3. [11]

    Dongxin Guo, Jikun Wu, and Siu Ming Yiu. 2026. FinGround: Detecting and Grounding Financial Hallucinations via Atomic Claim Verification.arXiv preprint arXiv:2604.23588(2026). To appear, ACL 2026 Industry Track

    2026 arXiv

  4. [12]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.ACM Transactions o...

    2025 doi arXiv

  5. [13]

    Khizar Hussain and Murat Kantarcioglu. 2026. PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts.arXiv preprint arXiv:2605.17028(2026)

    2026 arXiv

  6. [14]

    Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz- Ziv, and Tomasz Kajdanowicz. 2025. The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs.arXiv preprint arXiv:2508.08285(2025)

    2025 arXiv

  7. [15]

    Saurav Kadavath, Tom Conerly, Amanda Askell, et al. 2022. Language Models (Mostly) Know What They Know.arXiv preprint arXiv:2207.05221(2022)

    2022 arXiv

  8. [16]

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs.arXiv preprint arXiv:2406.15927(2024)

    2024 arXiv

  9. [17]

    János Kramár, Joshua Engels, Zheng Wang, Bilal Chughtai, Rohin Shah, Neel Nanda, and Arthur Conmy. 2026. Building Production-Ready Probes for Gemini. arXiv preprint arXiv:2601.11516(2026)

    2026

  10. [18]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Lin- guistic Invariances for Uncertainty Estimation in Natural Language Generation. InInternational Conference on Learning Representations (ICLR). arXiv:2302.09664

    2023 arXiv

  11. [19]

    Mahesh Kumar, Bhaskarjit Sarmah, and Stefano Pasquali. 2026. FinReflectKG– HalluBench: GraphRAG Hallucination Benchmark for Financial Question An- swering Systems.arXiv preprint arXiv:2603.20252(2026)

    2026

  12. [20]

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen

  13. [21]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2305.11747

    2023 arXiv

  14. [22]

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wat- tenberg. 2023. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. InAdvances in Neural Information Processing Systems (NeurIPS). arXiv:2306.03341

    2023 arXiv

  15. [23]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large Language Models in Finance: A Survey. InProceedings of the 4th ACM International Confer- ence on AI in Finance (ICAIF)

    2023

  16. [24]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL). arXiv:2109.07958

    2022 arXiv

  17. [25]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. SelfCheckGPT: Zero- Resource Black-Box Hallucination Detection for Generative Large Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2303.08896

    2023 arXiv

  18. [26]

    Samuel Marks and Max Tegmark. 2023. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets.arXiv preprint arXiv:2310.06824(2023)

    2023 arXiv

  19. [27]

    Liar Circuits

    Soham Mirajkar. 2025. Dissecting the Ledger: Locating and Suppressing “Liar Circuits” in Financial Large Language Models.arXiv preprint arXiv:2511.21756 (2025)

    2025

  20. [28]

    Mulvey, H

    Yuqi Nie, Yaxuan Kong, Xiaowen Dong, John M. Mulvey, H. Vincent Poor, Qing- song Wen, and Stefan Zohren. 2024. A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges.arXiv preprint arXiv:2406.11903(2024)

    2024 arXiv

  21. [29]

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2025. LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. InInternational Conference on Learning Representations (ICLR). arXiv:2410.02707

    2025 arXiv

  22. [30]

    Adi Simhi, Jonathan Herzig, Itay Itzhak, Dana Arad, Zorik Gekhman, Roi Re- ichart, Fazl Barez, Gabriel Stanovsky, Idan Szpektor, and Yonatan Belinkov. 2025. HACK: Hallucinations Along Certainty and Knowledge Axes.arXiv preprint arXiv:2510.24222(2025)

    2025

  23. [31]

    Adi Simhi, Itay Itzhak, Fazl Barez, Gabriel Stanovsky, and Yonatan Belinkov. 2025. Trust Me, I’m Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer. InFindings of the Association for Computational Linguistics: EMNLP 2025. arXiv:2502.12964

    2025 arXiv

  24. [32]

    Mainak Singha. 2025. Detecting AI Hallucinations in Finance: An Information- Theoretic Method Cuts Hallucination Rate by 92%.arXiv preprint arXiv:2512.03107 (2025)

    2025

  25. [33]

    Likun Tan, Kuan-Wei Huang, and Kevin Wu. 2025. FRED: Financial Retrieval- Enhanced Detection and Editing of Hallucinations in Language Models.arXiv preprint arXiv:2507.20930(2025)

    2025 arXiv

  26. [34]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. InInternational Conference on Learning Representations (ICLR). arXiv:2203.11171

    2023 arXiv

  27. [35]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebas- tian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann

  28. [36]

    BloombergGPT: A Large Language Model for Finance.arXiv preprint arXiv:2303.17564(2023)

    2023 arXiv

  29. [37]

    Mengao Zhang, Jiayu Fu, Tanya Warrier, Yuwen Wang, Tianhui Tan, and Ke-wei Huang. 2025. FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance. InProceedings of the 6th ACM International Conference on AI in Finance (ICAIF). arXiv:2508.05201

    2025

  30. [38]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A Question An- swering Benchmark on a Hybrid of Tabular and Textual Content in Finance. InProceedings of the 59th Annual Meeting of the Association for C...

    2021

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.