Pith. sign in

REVIEW 4 major objections 4 minor 51 references

CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CausalAbstain claims that comparing the direct effect of a proposed answer with its feedback-mediated effect tells a multilingual LLM when to abstain, and that this causal filtering outperforms strong baselines on M-MMLU and M-Hellaswag.

desk verdict Useful empirical abstention method that improves multilingual QA, but the causal language overclaims and the per-language causal selection is not what drives the gains. read the letter →

arxiv 2506.00519 v2 pith:DUA5CPDF submitted 2025-05-31 cs.CL cs.AI

Yuxi Sun , Aoqi Zuo , Wei Gao , Jing Ma This is my paper
classification cs.CLcs.AI
keywords causalabstentionmultilingualLLMnaturaldirecteffecttotalindirectfeedbackselectionhallucinationmitigationstructuralmodelaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CausalAbstain is a training-free method for deciding when a multilingual LLM should abstain from answering. The paper's claim is that LLM-generated feedback in various languages is often unreliable, and that a causal comparison—the natural direct effect of the proposed answer on the abstention decision versus the total indirect effect mediated by feedback—identifies which feedback is worth using. On M-MMLU and M-Hellaswag, across three base models, the multilingual variant outperforms strong calibration-, prompting-, consistency-, and feedback-based baselines, with an average gain of 3.5% over the best competing method. If the claim is right, abstention decisions become more trustworthy in low-resource languages without retraining or threshold tuning.

What carries the argument

The load-bearing object is the structural causal model in the paper's Figure 2, with variables Q (question), A (proposed answer), F_i (feedback in language i), and D (abstention decision), and edges A→D and A→F_i→D. The natural direct effect NDE captures how strongly the answer alone drives the decision, while the total indirect effect TIE_i captures the additional influence mediated by feedback; both are computed in practice as Jensen-Shannon divergences between empirical decision distributions obtained by repeated prompting with and without feedback. A comparison NDE < TIE_i decides whether to admit feedback, and a majority vote over N×L feedback instances yields the final abstention decision.

What would settle it

Take a fixed set of question-answer pairs where the model's proposed answer A is held constant but the question Q is reworded into paraphrases or translated variants that do not change the correct answer; if the NDE/TIE comparison flips the abstention decision across these rewritings, then Q has a direct effect on the decision that the graph does not model, and the causal estimates are confounded.

Watch

Extended reading notes

Core claim

The central discovery is that abstention can be treated as a causal mediation problem: the proposed answer A causally influences the final decision D both directly and through the feedback F_i it generates in language i. By repeatedly sampling decisions with and without feedback and measuring the Jensen-Shannon divergence between the resulting distributions, the method estimates NDE (A→D) and TIE_i (A→F_i→D). When NDE ≥ TIE_i, feedback is judged unhelpful and ignored; when NDE < TIE_i, feedback is used, and multilingual decisions are combined by majority vote. The paper reports that this causal filtering raises abstention accuracy across languages and models, especially for low-resource languages, and that multilingual feedback helps more than native-language feedback alone.

Load-bearing premise

The load-bearing premise is the causal graph's assumption that the question Q has no direct effect on the feedback F_i or the decision D, so the only influence of Q runs through the proposed answer A; Section 2.1 states this explicitly, and if question wording changes the model's confidence or feedback independently of the answer, the computed NDE and TIE are confounded.

Editorial extensions

If this is right

  • If the causal comparison is doing the work, abstention can be improved without calibration sets or confidence thresholds, which makes the method easier to deploy across domains.
  • The method gives an interpretable reason for each abstention choice: it is driven by whether feedback shifts the decision distribution relative to the answer's direct effect.
  • Multilingual feedback should be preferred over native-language feedback for smaller models, since CAUSAL-MULTI outperforms CAUSAL-NATIVE on ChatGPT and Aya-13B while GPT-4o benefits less.
  • Aggregating votes across related languages is more robust than relying on any single language where TIE exceeds NDE, reducing the risk of low-resource language bias.
  • The same framework extends to any QA setting where feedback can be elicited, since it does not depend on language-specific training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The NDE/TIE comparison is essentially a decision-variance test; with N=3 iterations the JSD estimates are noisy, so a natural extension is to increase iterations or cluster semantically identical feedback before voting.
  • Editorial inference: If the graph were relaxed to include direct edges Q→F_i and Q→D, the procedure would need controlled direct effects; re-running the method on paraphrased questions would test whether those edges matter.
  • Editorial inference: Language relatedness could be replaced by a content-based similarity measure over the feedback itself, making the method adaptive to any language pool.
  • Editorial inference: The same causal filter could be applied to other auxiliary signals, such as retrieved passages or tool outputs, wherever the quality of the signal is uneven.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CausalAbstain, a training-free method for multilingual abstention in LLM question answering. The method first has the LLM generate feedback on its proposed answer in either the native language (CASUAL-NATIVE) or multiple related languages (CASUAL-MULTI), with N iterations per language. It then computes quantities labeled NDE and TIE from the empirical distributions of the LLM's binary True/False decisions with and without feedback (Eqs. 5-6), comparing them via Jensen-Shannon divergence. The decision rule is: if NDE >= TIE for all languages, vote over the no-feedback decisions; if there exists any language with TIE > NDE, vote over all N×L feedback decisions. Experiments on M-MMLU and M-Hellaswag with Aya-13B, ChatGPT, and GPT-4o report that CASUAL-MULTI achieves the best or second-best abstention accuracy in most settings, with ablations showing the importance of the feedback and the aggregation components.

Significance. If the causal claims were sound, the paper would make a notable contribution by giving an interpretable, training-free mechanism for filtering multilingual feedback in abstention decisions. The empirical gains over strong baselines such as MULTI-RELATED and CONFLICT are reported consistently across two datasets and five LLMs (including LLaMa and Phi in the appendix), and the authors provide open-source code and detailed case studies. The paper is also clearly written and the experimental setup is standard for the abstention literature. However, the central causal interpretation is not supported by the methodology: the quantities called NDE and TIE are not identified as causal effects, and the ablations show that the per-language TIE>NDE comparison does not itself select useful feedback. The empirical contribution may survive as a heuristic voting strategy, but the causal-framing contribution as stated does not.

major comments (4)
  1. [Sec. 2.2 and Sec. 3.2, Eqs. (3)-(6)] The paper defines NDE and TIE as differences of expectations over potential outcomes in a Pearl SCM (Eqs. 3-4), but then computes them in Eqs. (5)-(6) as Jensen-Shannon divergences between observed empirical distributions of LLM decisions. No identification argument is given, and JSD is not an expectation difference; it is a divergence between probability distributions. Consequently, the quantities called NDE and TIE do not measure the natural direct effect or total indirect effect of A on D, and the decomposition TE = NDE + TIE (Eq. 1) has no counterpart in the implemented computation. The baseline D0, modeled as a binomial with probability 0.5, is also introduced without justification. The causal interpretation is therefore not established by the presented equations.
  2. [Sec. 3.3 and Tables 2, 6] The claimed per-feedback causal selection is not implemented. The final CAUSAL-MULTI rule uses TIE_i only as a global existence gate: if any language has TIE_i > NDE, the method votes over all N×L feedback decisions; it does not restrict the vote to languages with TIE_i > NDE. The abstract's claim that the method identifies 'the most useful ones' among the feedback responses is thus not reflected in the algorithm. Moreover, the ablation setting labeled 'w/o aggregating over all languages' (Table 2, setting 4; Table 6, setting 4), which does implement the per-language TIE>NDE selection, underperforms CAUSAL-MULTI in every reported model/dataset combination (e.g., Aya-13B M-Hellaswag drops from .612 to .553; ChatGPT M-Hellaswag from .583 to .552; GPT-4o M-Hellaswag from .577 to .552). This is the paper's own evidence that the causal criterion is not the source of the gain; the gain comes from aggregating all languages once any language passes the gate.
  3. [Sec. 2.1, Fig. 2] The causal graph omits direct edges from Q to F_i and Q to D, with the stated justification that the question alone cannot trigger feedback or the decision. This assumption is questionable for the abstention task: the LLM's confidence in its answer, and hence its tendency to abstain, can plausibly depend on the question itself (e.g., question difficulty or language resource level). If Q has a direct effect on D or F_i, then the computed NDE and TIE are confounded and do not correspond to the path-specific effects in Fig. 2. The authors acknowledge this as a limitation, but it strengthens the concern that the causal quantities are not valid estimates of the intended effects.
  4. [Sec. 3.2, Eqs. (5)-(6); Sec. 4.1 'Answer Likelihood Distribution'] The estimand is not well-defined even as a nonparametric estimand. The distributions D(Q,A) and D(Q,A,F_i) are approximated by iterating the same LLM N times, but the paper does not state how randomness across iterations is controlled (e.g., temperature), nor why the softmax of average indicator values yields the probability of D=1. Because N is small (3), the JSD estimates are noisy and no confidence intervals or significance tests are reported. Improving significance testing would strengthen the empirical claims, but it would not repair the missing identification.
minor comments (4)
  1. [Abstract and throughout] The method name is written inconsistently as 'CausalAbstain', 'CASUAL-NATIVE', and 'CASUAL-MULTI'; the variant names should be corrected to 'CAUSAL-NATIVE' and 'CAUSAL-MULTI' throughout, including captions and tables.
  2. [Sec. 4.2, Table 1] The text states 'outperforms the strongest baseline in 4 out of 6 settings,' but from the table, CAUSAL-MULTI is best in 5 of 6 settings if one counts both M-MMLU and M-Hellaswag across the three models (the exception is ChatGPT M-Hellaswag, where CONFLICT achieves .595 vs .583). The claim should be reconciled with the table.
  3. [Sec. 5, Table 13] The decision rule description 'NDE not ≤ all TIE (NDE < TIE_du)' is confusing; the notation should be clarified to indicate that the gate is triggered by the existence of at least one language with TIE_i > NDE.
  4. [Appendix A, Table 8] The 'Related language settings' header is cut off in the provided text ('Related language sets are ...'), and the table lists repeated languages (e.g., 'Chinese' appears three times for zh); the authors should clarify whether these repetitions are intentional and whether the related languages are deduplicated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the NDE/TIE gate is a self-consistency heuristic that is not fitted to the evaluation labels, and the reported abstention accuracy comes from an independent external comparison.

full rationale

CausalAbstain is a training-free method: NDE and TIE are computed from the target LLM's own with- and without-feedback True/False verdicts (Eqs. 5-6), and the final abstention decision is a majority vote over one of those two verdict sets, selected by comparing NDE and TIE. The causal interpretation of these divergences is arguably a relabeling of a self-consistency heuristic, and the paper's own ablation setting 4 ("w/o aggregating over all languages") shows that using the per-language causal criterion without aggregation underperforms, which is a genuine concern about the claim that the method selects helpful feedback. However, this is a validity/overclaim issue rather than a circular derivation: no parameter is fitted to the benchmark labels, no load-bearing result is imported from a self-citation, and the reported Abstain Accuracy is evaluated against independent ground truth. The NDE/TIE comparison is not by construction equal to the abstention decision; it is one of several possible heuristics for choosing between two voting ensembles. The limitation section explicitly notes that the causal graph is a general framework and that latent variables could be added later, which is an acknowledged assumption rather than a circular step. Therefore, the paper's empirical claims are self-contained and do not reduce by construction to their inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a causal graph and an estimation strategy that are not derived from the data: the graph omits Q->D, the JSD implementation is not linked to true causal effects, the 0.5 baseline is arbitrary, and N and the related-language sets are tuned on validation data.

free parameters (3)
  • Baseline probability D0 = 0.5
    Hand-chosen binomial baseline used to compute NDE; not derived from data.
  • Number of iterations N = 3
    Tuned on a 100-instance held-out set (Appendix A, Figure 5) rather than fixed a priori.
  • Related language sets per language = chosen per language (Table 8)
    Selected using 100 held-out instances to determine the most suitable related languages, a form of validation-set fitting.
assumptions (4)
  • domain assumption The causal graph in Figure 2 has no edges from Q to D or Q to F_i.
    State in Section 2.1. If Q influences D directly, the measured effects are confounded.
  • ad hoc to paper Jensen-Shannon divergence between empirical decision distributions estimates the natural direct and total indirect effects.
    Eqs. (5)-(6) replace the expectation operators in Eqs. (3)-(4) with JSD without an identification proof.
  • ad hoc to paper D0, the no-answer baseline, is a binomial distribution with probability 0.5.
    Introduced in Section 2.2 after Eq. (3); arbitrary baseline that determines NDE magnitude.
  • domain assumption The LLM's decisions and feedback samples are exchangeable draws from a stable distribution.
    Implicit in using N i.i.d. samples to approximate Dhat.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention." pith.science (2026). https://pith.science/paper/DUA5CPDF

@misc{pith2026250600519,
  author       = {Pith},
  title        = {Pith review of: CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUA5CPDF}},
  note         = {Machine review of arXiv:2506.00519}
}
read the original abstract

Large Language Models (LLMs) often exhibit knowledge disparities across languages. Encouraging LLMs to \textit{abstain} when faced with knowledge gaps is a promising strategy to reduce hallucinations in multilingual settings. Current abstention strategies for multilingual scenarios primarily rely on generating feedback in various languages using LLMs and performing self-reflection. However, these methods can be adversely impacted by inaccuracies and biases in the generated feedback. To address this, from a causal perspective, we introduce \textit{CausalAbstain}, a method that helps LLMs determine whether to utilize multiple generated feedback responses and how to identify the most useful ones. Extensive experiments demonstrate that \textit{CausalAbstain} effectively selects helpful feedback and enhances abstention decisions with interpretability in both native language (\textsc{Casual-native}) and multilingual (\textsc{Causal-multi}) settings, outperforming strong baselines on two benchmark datasets covering encyclopedic and commonsense knowledge QA tasks. Our code and data are open-sourced at https://github.com/peachch/CausalAbstain.

Figures

Figures reproduced from arXiv: 2506.00519 by the authors.

Figure 1
Figure 1. Prior approaches often underperform beyond [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The causal graph of the question-answering [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The framework of CausalAbstain. We propose an abstention strategy to determine whether to adhere to or abstain from the proposed answer (i.e., the answer B) based on the feedback in the native language (CAUSAL￾NATIVE) and related languages (CAUSAL-MULTI), by leveraging natural direct effect (NDE) and total indirect effect (TIE) to assess the impact of feedback on the final abstention decision. We provide the English… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The distribution and accuracy rate of NDE [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The impact of language relatedness (different [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the accuracy of LLMs’ pro [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 10 canonical work pages

  1. [1]

    Amos Azaria and Tom Mitchell. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.68 The internal state of an LLM knows when it`s lying . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967--976, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.68 2023

  2. [2]

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023

    arXiv 2023

  3. [3]

    Hang Chen, Bingyu Liao, Jing Luo, Wenjing Zhu, and Xinyu Yang. 2024. https://doi.org/10.1109/tkde.2024.3352575 Learning a structural causal model for intuition reasoning in conversation . IEEE Transactions on Knowledge and Data Engineering, 36(7):3210–3223

    work page arXiv 2024

  4. [4]

    Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2024. https://arxiv.org/abs/2311.04155 Black-box prompt optimization: Aligning large language models without model training . Preprint, arXiv:2311.04155

    arXiv 2024

  5. [5]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. https://arxiv.org/abs/1808.09381 Understanding back-translation at scale . Preprint, arXiv:1808.09381

    arXiv 2018

  6. [6]

    Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2024 a . https://arxiv.org/abs/2305.09955 Knowledge card: Filling llms' knowledge gaps with plug-in specialized language models . Preprint, arXiv:2305.09955

    arXiv 2024

  7. [7]

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Orevaoghene Ahia, Shuyue Stella Li, Vidhisha Balachandran, Sunayana Sitaram, and Yulia Tsvetkov. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.239 Teaching LLM s to abstain across languages via multilingual feedback . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    work page doi:10.18653/v1/2024.emnlp-main.239 2024

  8. [8]

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.786 Don`t hallucinate, abstain: Identifying LLM knowledge gaps via multi- LLM collaboration . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    doi:10.18653/v1/2024.acl-long.786 2024

Show all 51 references
  1. [9]

    Yair Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. 2023. https://arxiv.org/abs/2310.00603 Faithful explanations of black-box nlp models using llm-generated counterfactuals . Preprint, arXiv:2310.00603

    2023 arXiv

  2. [10]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. https://doi.org/10.1145/3571730 Survey of hallucination in natural language generation . ACM Computing Surveys, 55(12):1–38

    2023 doi

  3. [11]

    Haitao Jiang, Lin Ge, Yuhe Gao, Jianian Wang, and Rui Song. 2024. https://arxiv.org/abs/2312.17122 Llm4causal: Democratized causal tools for everyone via large language model . Preprint, arXiv:2312.17122

    2024 arXiv

  4. [12]

    Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. 2024 a . https://arxiv.org/abs/2312.04350 Cladder: Assessing causal reasoning in language models . Pr...

    2024 arXiv

  5. [13]

    Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. 2024 b . https://arxiv.org/abs/2306.05836 Can large language models infer causation from correlation? Preprint, arXiv:2306.05836

    2024 arXiv

  6. [14]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  7. [15]

    Haoqiang Kang, Terra Blevins, and Luke Zettlemoyer. 2024. https://arxiv.org/abs/2402.10496 Comparing hallucination detection metrics for multilingual generation . Preprint, arXiv:2402.10496

    2024 arXiv

  8. [16]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://arxiv.org/abs/2302.09664 Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . Preprint, arXiv:2302.09664

    2023 arXiv

  9. [17]

    Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.eacl-main.241 Language generation models can cause harm: So what can we do about it? an actionable survey . In Proceedings of the 17th Confere...

    2023 doi

  10. [18]

    Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023 a . Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613

    2023 arXiv

  11. [19]

    Rossi, and Thien Huu Nguyen

    Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023 b . https://arxiv.org/abs/2307.16039 Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedbac...

    2023 arXiv

  12. [20]

    Zhongyang Li, Xiao Ding, Kuo Liao, Bing Qin, and Ting Liu. 2021. https://arxiv.org/abs/2107.09852 Causalbert: Injecting causal knowledge into pre-trained models with minimal supervision . Preprint, arXiv:2107.09852

    2021 arXiv

  13. [21]

    Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin

    Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. https://aclanthology.org/E17-2002/ URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors . In Proceedings of the 15th Conference of...

    2017

  14. [22]

    Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, Julian McAuley, Wei Ai, and Furong Huang. 2024 a . https://arxiv.org/abs/2403.09606 Large language models and causal inference in collaboration: A comprehe...

    2024 arXiv

  15. [23]

    Xin Liu, Muhammad Khalifa, and Lu Wang. 2024 b . https://arxiv.org/abs/2310.19208 Litcab: Lightweight language model calibration over short- and long-form responses . Preprint, arXiv:2310.19208

    2024 arXiv

  16. [24]

    Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. 2024. https://arxiv.org/abs/2407.16221 Do llms know when to not answer? investigating abstention abilities of large language models . Preprint, arXiv:2407.16221

    2024 arXiv

  17. [25]

    Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. https://arxiv.org/abs/2401.06855 Fine-grained hallucination detection and editing for language models . Preprint, arXiv:2401.06855

    2024 arXiv

  18. [26]

    R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5)

    2023 arXiv

  19. [27]

    Judea Pearl et al. 2000. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19

    2000

  20. [28]

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rockt \"a schel, and Sebastian Riedel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.200 KIL...

    2021 doi

  21. [29]

    Wenbo Shang and Xin Huang. 2024. A survey of large language models on generative graph analytics: Query, learning, and applications. arXiv preprint arXiv:2404.14809

    2024 arXiv

  22. [30]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057

    2022 arXiv

  23. [31]

    Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. 2023. https://arxiv.org/abs/2305.14628 Getting more out of mixture of language model reasoning experts . Preprint, arXiv:2305.14628

    2023 arXiv

  24. [32]

    Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.220 The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models . In Proceedings ...

    2023 doi

  25. [33]

    Mortensen

    Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. 2021. https://doi.org/10.18653/v1/2021.eacl-main.204 Cross-cultural similarity features for cross-lingual transfer learning of pragmatically motivated tasks . In Proceedings of the 16th Conference...

    2021 doi

  26. [34]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod...

    2023 doi

  27. [35]

    Neeraj Varshney and Chitta Baral. 2023. https://arxiv.org/abs/2305.01812 Post-abstention: Towards reliably re-attempting the abstained instances in qa . Preprint, arXiv:2305.01812

    2023 arXiv

  28. [36]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 a . https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . Preprint, arXiv:2203.11171

    2023 arXiv

  29. [37]

    Do, Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang, Yangqiu Song, Ginny Y

    Zhaowei Wang, Quyet V. Do, Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. 2023 b . https://arxiv.org/abs/2305.05191 Cola: Contextualized commonsense causal reasoning from the causal inference perspective . Preprint, arXiv:2...

    2023 arXiv

  30. [38]

    Junda Wu, Tong Yu, Xiang Chen, Haoliang Wang, Ryan Rossi, Sungchul Kim, Anup Rao, and Julian McAuley. 2024. https://doi.org/10.18653/v1/2024.acl-long.758 D e C o T : Debiasing chain-of-thought for knowledge-intensive tasks in large language models via causal intervention . In ...

    2024 doi

  31. [39]

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. https://arxiv.org/abs/2305.13300 Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts . Preprint, arXiv:2305.13300

    2024 arXiv

  32. [40]

    Jian Yang, Xinyu Hu, Gang Xiao, and Yulong Shen. 2024 a . A survey of knowledge enhanced pre-trained language models. ACM Transactions on Asian and Low-Resource Language Information Processing

    2024

  33. [41]

    Shuo Yang, Siwen Luo, and Soyeon Caren Han. 2025. Multimodal commonsense knowledge distillation for visual question answering (student abstract). In Proceedings of the AAAI conference on artificial intelligence, volume 39, pages 29545--29547

    2025

  34. [42]

    Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2024 b . https://arxiv.org/abs/2312.07000 Alignment for honesty . Preprint, arXiv:2312.07000

    2024 arXiv

  35. [43]

    Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. 2023. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296

    2023 arXiv

  36. [44]

    Chuanqi Zang, Hanqing Wang, Mingtao Pei, and Wei Liang. 2023. Discovering the real association: Multimodal causal reasoning in video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19027--19036

    2023

  37. [45]

    Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. 2024. https://arxiv.org/abs/2305.13669 The knowledge alignment problem: Bridging human and external knowledge for large language models . Preprint, arXiv:2305.13669

    2024 arXiv

  38. [46]

    Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.491 Don`t trust C hat GPT when your question is not in E nglish: A study of multilingual abilities and types of LLM s . In Proceedings of the 2023 Confere...

    2023 doi

  39. [47]

    Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023 b . Don't trust chatgpt when your question is not in english: a study of multilingual abilities and types of llms. arXiv preprint arXiv:2305.16339

    2023 arXiv

  40. [48]

    Shitian Zhao, Zhuowan Li, Yadong Lu, Alan Yuille, and Yan Wang. 2023. https://arxiv.org/abs/2312.06685 Causal-cog: A causal-effect look at context generation for boosting multi-modal language models . Preprint, arXiv:2312.06685

    2023 arXiv

  41. [49]

    Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. 2024. https://arxiv.org/abs/2309.17249 Batch calibration: Rethinking calibration for in-context learning and prompt engineering . Preprint, arXiv:2309.17249

    2024 arXiv

  42. [50]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.