Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TruthTorchLM is an open-source library that unifies over 30 methods for scoring LLM output truthfulness and benchmarks them on short- and long-form tasks.

desk verdict TruthTorchLM is a real, useful library with a fragile benchmark: treat Table 2 as illustrative until the auto-generated correctness labels are validated against humans and error bars are reported. read the letter →

arxiv 2507.08203 v1 pith:AWYPUMQ3 submitted 2025-07-10 cs.CL

Duygu Nur Yaldiz , Yavuz Faruk Bakman , Sungmin Kang , Alperen Öziş , Hayrettin Eren Yildiz , Mitash Ashish Shah , Zhiqi Huang , Anoop Kumar
show 4 more authors
Alfy Samuel Daben Liu Sai Praneeth Karimireddy Salman Avestimehr
This is my paper · ORCID
classification cs.CL
keywords truthfulnesspredictionhallucinationdetectionuncertaintyquantificationLLM-as-a-judgecalibrationlong-formfactualityAUROCopen-sourcelibrary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TruthTorchLM aims to make truthfulness prediction for large language models as easy as calling one function: it collects more than 30 existing scoring methods, called Truth Methods, behind a single interface for generation, evaluation, and calibration. The paper's empirical contribution is a head-to-head comparison of representative methods on open-domain question answering, grade-school math, and long-form biography generation. On short-form tasks, the supervised methods LARS and SAPLMA reach the highest AUROC, while on long-form biography generation Verbalized Confidence leads. If the library works as described, researchers can reproduce, extend, and compare truthfulness detectors without building each one from scratch, which matters for deploying LLMs in settings where wrong answers carry real cost.

What carries the argument

The load-bearing object is the TruthMethod base class: a post hoc scorer that takes generation-time information such as token ids, the model, and the tokenizer, and returns a truth value, with subclasses for LARS, semantic entropy, verbalized confidence, and dozens of other methods. Around it, the library builds a unified generation call, normalization and calibration utilities such as isotonic regression and min-max scaling that map raw scores to [0,1], and — for long-form text — decomposition methods that split generations into atomic claims plus Claim Check wrappers that score each claim individually. Tracked together, these pieces let one harness run, score, calibrate, and compare methods that otherwise live in separate codebases.

What would settle it

Run the released library on a fresh 500-question subset of the same short-form datasets and the same long-form prompts, but replace the automatic correctness labels with human annotation; if the AUROC leaderboard changes materially — for instance, if LARS and SAPLMA no longer top the short-form tasks — the evaluation claim is an artifact of the judge rather than a property of the methods.

Watch

Extended reading notes

Core claim

The central claim is that the entire landscape of post hoc truthfulness prediction — uncertainty estimation, sampling-based probing, supervised classifiers, and document-grounded checking — can be expressed through one abstraction, the Truth Method, and evaluated with one harness. The paper's evidence for that claim is the library itself plus a benchmark of representative methods: on the short-form question answering and math reasoning datasets, LARS and SAPLMA achieve the best AUROC among tested methods (0.861 and 0.834 for LARS and 0.850 and 0.815 for SAPLMA on an open-weight model, with LARS also leading on a closed API model), while Verbalized Confidence posts the best AUROC on the long-form biography dataset (0.698 for the open-weight model and 0.717 for the API model). A further finding is that long-form performance drops across the board, so claim-level decomposition and claim-check wrappers are needed to make methods usable there.

Load-bearing premise

The benchmark rankings assume the automatic correctness labels (an LLM judge for short answers, a search-based verifier for long answers) accurately reflect whether outputs are truly right or wrong.

Editorial extensions

If this is right

  • Any of the more than 30 methods can be invoked on a new model or dataset with the same generate-and-score call, so benchmark numbers across methods are directly comparable.
  • New truthfulness predictors can be added by implementing one standardized forward function and immediately inherit evaluation, calibration, and long-form support.
  • The benchmark gives a concrete selection rule: supervised methods such as LARS and SAPLMA are the strongest short-form choices when labels are available, while verbalized confidence is the strongest tested long-form predictor.
  • Long-form truthfulness can be scored claim by claim, so a reader can see which parts of a biography are unsupported rather than receiving one opaque number.
  • Calibrated scores in the unit interval make it possible to ensemble several truth methods into a single signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit is that the ranking's robustness is untested, because the correctness labels come from an automatic judge and a search-based verifier; a human-annotated subset of the same samples would tell whether the top methods hold.
  • The unified harness invites a larger-scale comparison across more models and datasets, and could be used to test whether the best method for each task shifts with model family.
  • Claim-level scoring plus wrappers suggests the same machinery could be applied to agentic or retrieval-augmented outputs, checking each retrieved fact or tool response as a claim.
  • If calibration is reliable, the library opens the door to meta-predictors that learn which truth method to trust per domain, something the paper only gestures at through ensembling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces TruthTorchLM, an open-source Python library that implements over 30 methods for predicting truthfulness of LLM outputs, with unified interfaces for generation, evaluation, calibration, and long-form claim-level prediction, and compatibility with HuggingFace and LiteLLM. The authors demonstrate the library by evaluating representative methods on TriviaQA, GSM8K, and FactScore-Bio using LLaMA-3-8B and GPT-4o-mini, reporting AUROC and PRR metrics. The empirical section uses GPT-4o-mini as an LLM judge for short-form correctness and the SAFE algorithm for long-form claim verification, without validating these automatic labels against human annotations.

Significance. If the library is as functional and comprehensive as claimed, it addresses a real gap: existing toolkits such as Guardrails and LM-Polygraph cover narrower subsets of truthfulness methods, whereas TruthTorchLM aims to unify uncertainty-based, supervised, document-grounded, and collaboration-based approaches under one interface. The code is publicly available, and the paper includes working code listings that demonstrate the library's core workflows. However, the empirical evaluation that supports the paper's comparative claims about method performance is weakened by reliance on unvalidated automatic labels, small long-form sample sizes, and missing statistical rigor. Given these issues, the empirical conclusions in Table 2 should be treated as preliminary until the authors provide validation of the label generation pipeline and uncertainty estimates.

major comments (4)
  1. [Section 4, Correctness Measure] The correctness labels for short-form answers are produced by GPT-4o-mini, which is also one of the two generators under evaluation, and no human agreement study or independent judge is reported. This creates a concrete risk that the labels are biased, which would systematically affect the AUROC/PRR values in Table 2 for all methods evaluated on GPT-4o-mini. Since the margins between top methods are sometimes small (e.g., LARS 0.861 vs. SAPLMA 0.850 on TriviaQA-LLaMA), the reported ranking may not be robust. The authors should validate the LLM judge against human annotations (e.g., report Cohen's kappa) or include an exact-match baseline on GSM8K as a sanity check, and report confidence intervals for the metrics.
  2. [Section 4, Experiments (Datasets)] The FactScore-Bio evaluation uses only 50 questions, and the extracted claims (1290 for GPT-4o-mini, 1764 for LLaMA-3-8B) are treated as independent samples for AUROC/PRR computation. Claims from the same generation are not independent, and with no clustering or bootstrap error bars, differences in Table 2 (e.g., VerbalizedConfidence PRR 0.460 vs. Eccentricity PRR 0.415 for LLaMA-3-8B) are likely within noise. The paper should report per-generation aggregated scores, confidence intervals, or a multilevel analysis to support the long-form conclusions.
  3. [Section 4, Correctness Measure] For long-form generations, the paper adopts the SAFE algorithm to estimate claim correctness via Google Search, but this algorithm's outputs are not validated against human annotations in the paper. Since SAFE is itself an LLM-based system, its accuracy and any systematic biases directly determine the validity of the FactScore-Bio results in Table 2. The authors should either validate SAFE on a human-labeled subset, report its precision/recall, or provide an alternative ground truth before drawing conclusions about long-form truthfulness prediction.
  4. [Section 4, Experiments] The evaluation does not report random seeds, generation temperatures, or the exact prompts used for generation and for the LLM judge, which prevents reproducibility of the reported numbers. Additionally, no error bars, confidence intervals, or significance tests are provided for any AUROC/PRR values in Table 2. Given the small margins between some methods, these omissions make it difficult to assess whether the reported rankings are meaningful or due to random variation. The authors should provide full experimental configuration details and quantify uncertainty (e.g., via multiple runs or bootstrapping).
minor comments (5)
  1. [Section 2.5] The section title 'Predicting Truthfullness in Long Form Generation' contains a typo; it should be 'Truthfulness' and 'Long-Form'. There is also a typo in the text: 'First, it generates a response ant this process' should read 'and this process'.
  2. [Section 4.1] In the discussion, 'Eccentiricity' is a typo for 'Eccentricity' in the sentence about sampling-based methods.
  3. [Table 2] The table formatting is difficult to read: some cells contain two numbers concatenated without a visible separator (e.g., '0.792 0.596' appears as a single run-on entry), and the header layout should be clarify which columns correspond to AUROC and PRR for each dataset.
  4. [Abstract and Section 2.1] The abstract claims 'over 30 truth methods', but Table 1 lists only a representative subset of 18. It would be helpful to provide a complete list or a reference to the repository's method index so readers can verify the count and scope.
  5. [References] The reference to 'Guardrails' is given as (guardrails-ai) without a formal citation or URL in the reference list; consider adding a proper citation. The YouTube link in the abstract is also unconventional and would be better placed in a footnote or supplementary materials.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TruthTorchLM is a tooling and benchmark contribution whose reported rankings are computed against externally grounded correctness labels, not forced by construction.

full rationale

The paper's central deliverables are a software library and a comparative evaluation of existing truthfulness-prediction methods. There is no derivation chain in which a prediction reduces to its own input. The correctness labels for short-form answers come from a GPT-4o-mini judge prompted with the question, generated answer, and reference answer (Section 4, Correctness Measure), and for long-form answers from the SAFE algorithm using Google Search (Section 4, Correctness Measure). These labels are external to the truth methods being scored, so the AUROC/PRR values in Table 2 are not identities or fitted parameters renamed as predictions. The paper does include self-citations: LARS (Yaldiz et al., 2025) and MARS (Bakman et al., 2024) were developed by the same authors, and Bakman et al. (2025) is cited for calibration and evaluation conventions. However, these citations are contextual rather than load-bearing: LARS and MARS are evaluated alongside the other methods against the same external labels, and no central claim depends on an unverified self-cited uniqueness theorem or on a self-cited ansatz. The unvalidated nature of the LLM-as-a-judge labels is a legitimate threat to the empirical rankings, but that is a correctness/validity concern, not circularity. The library's scope claims ('over 30 truth methods') are descriptive and not derived from any fitted quantity. Consequently, no circular step can be exhibited with equations or definitional reductions, and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's empirical claims rest entirely on the reliability of automatic labeling (LLM judge and SAFE), which is unvalidated. There are no fitted free parameters and no invented scientific entities; the only postulates are the standard metrics AUROC/PRR and the reliability of the correctness evaluators.

assumptions (3)
  • domain assumption LLM-as-a-judge correctness labels are accurate
    Section 4 Correctness Measure: binary correctness labels for short-form answers are assigned by prompting GPT-4o-mini; no human validation is reported.
  • domain assumption SAFE algorithm produces correct claim-level labels for FactScore-Bio
    Section 4: long-form correctness uses SAFE (Wei et al., 2024) with Google Search; the paper does not audit agreement with human labels.
  • domain assumption Claim decomposition yields self-contained, correctly parsed claims
    Section 2.5: decomposition quality is 'critical' for assessment but is only shown qualitatively in Table 4; no decomposition accuracy metric is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs." pith.science (2026). https://pith.science/paper/AWYPUMQ3

@misc{pith2026250708203,
  author       = {Pith},
  title        = {Pith review of: TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AWYPUMQ3}},
  note         = {Machine review of arXiv:2507.08203}
}
read the original abstract

Generative Large Language Models (LLMs)inevitably produce untruthful responses. Accurately predicting the truthfulness of these outputs is critical, especially in high-stakes settings. To accelerate research in this domain and make truthfulness prediction methods more accessible, we introduce TruthTorchLM an open-source, comprehensive Python library featuring over 30 truthfulness prediction methods, which we refer to as Truth Methods. Unlike existing toolkits such as Guardrails, which focus solely on document-grounded verification, or LM-Polygraph, which is limited to uncertainty-based methods, TruthTorchLM offers a broad and extensible collection of techniques. These methods span diverse tradeoffs in computational cost, access level (e.g., black-box vs white-box), grounding document requirements, and supervision type (self-supervised or supervised). TruthTorchLM is seamlessly compatible with both HuggingFace and LiteLLM, enabling support for locally hosted and API-based models. It also provides a unified interface for generation, evaluation, calibration, and long-form truthfulness prediction, along with a flexible framework for extending the library with new methods. We conduct an evaluation of representative truth methods on three datasets, TriviaQA, GSM8K, and FactScore-Bio. The code is available at https://github.com/Ybakman/TruthTorchLM

Figures

Figures reproduced from arXiv: 2507.08203 by the authors.

Figure 1
Figure 1. Overview of TruthTorchLM functionalities. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uncertainty Quantification for Retrieval-Augmented Reasoning

    cs.IR 2025-10 conditional novelty 6.0 of 10

    R2C perturbs reasoning states (paraphrasing, rethinking, validating) to score consistency, improving UQ AUROC by over 5% on average for retrieval-augmented reasoning.

Reference graph

Works this paper leans on

39 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Amos Azaria and Tom Mitchell. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.68 The internal state of an LLM knows when it ' s lying . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967--976, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.68 2023

  5. [5]

    Yavuz Bakman, Duygu Nur Yaldiz, Sungmin Kang, Tuo Zhang, Baturalp Buyukates, Salman Avestimehr, and Sai Praneeth Karimireddy. 2025. https://arxiv.org/abs/2506.01114 Reconsidering llm uncertainty estimation methods in the wild . Preprint, arXiv:2506.01114

    work page Pith review arXiv 2025

  6. [6]

    Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. https://doi.org/10.18653/v1/2024.acl-long.419 MARS : Meaning-aware response scoring for uncertainty estimation in generative LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    doi:10.18653/v1/2024.acl-long.419 2024

  7. [7]

    Neil Band, Tim G. J. Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Michael W Dusenberry, Ghassen Jerfel, Dustin Tran, and Yarin Gal. 2021. https://openreview.net/forum?id=jyd4Lyjr2iB Benchmarking Bayesian deep learning on diabetic retinopathy detection tasks . In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks ...

    work page 2021

  8. [8]

    https://github.com/BerriAI/litellm litellm

    BerriAI . https://github.com/BerriAI/litellm litellm

    work page

Show all 39 references
  1. [9]

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. https://openreview.net/forum?id=Zj12nzlQbz INSIDE : LLM s' internal states retain the power of hallucination detection . In The Twelfth International Conference on Learning Representations

    2024

  2. [10]

    I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023. Factool: Factuality detection in generative ai--a tool augmented framework for multi-task and multi-domain scenarios. arXiv preprint arXiv:2307.13528

    2023 arXiv

  3. [11]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [12]

    Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.778 LM vs LM : Detecting factual errors via cross examination . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12621--12640, ...

    2023 doi

  5. [13]

    Longchao Da, Tiejin Chen, Lu Cheng, and Hua Wei. 2024. https://arxiv.org/abs/2407.00994 Llm uncertainty quantification through directional entailment graph and claim level response augmentation . Preprint, arXiv:2407.00994

    2024 arXiv

  6. [14]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. https://doi.org/10.18653/v1/2024.acl-long.276 Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models ...

    2024 doi

  7. [15]

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. 2024. https://doi.org/10.18653/v1/2024.findings-acl.558 Fact-checking th...

    2024 doi

  8. [16]

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.41 LM -polygraph...

    2023 doi

  9. [17]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--630

    2024

  10. [18]

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024. https://doi.org/10.18653/v1/2024.acl-long.786 Don ' t hallucinate, abstain: Identifying LLM knowledge gaps via multi- LLM collaboration . In Proceedings of the 62nd Annual Meet...

    2024 doi

  11. [19]

    https://github.com/guardrails-ai/guardrails Guardrails

    guardrails-ai . https://github.com/guardrails-ai/guardrails Guardrails

  12. [20]

    Samworth

    Qiyang Han, Tengyao Wang, Sabyasachi Chatterjee, and Richard J. Samworth. 2017. https://api.semanticscholar.org/CorpusID:41805423 Isotonic regression in general dimensions . The Annals of Statistics

    2017

  13. [21]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  14. [22]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  15. [23]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://openreview.net/forum?id=VD-AYtP0dve Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations

    2023

  16. [24]

    Deren Lei, Yaxi Li, Mengya Hu, Mingyu Wang, Vincent Yun, Emily Ching, and Eslam Kamal. 2023. https://arxiv.org/abs/2310.03951 Chain of natural language inference for reducing large language model ungrounded hallucinations . Preprint, arXiv:2310.03951

    2023 arXiv

  17. [25]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  18. [26]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. https://openreview.net/forum?id=DWkJCSxKU5 Generating with confidence: Uncertainty quantification for black-box large language models . Transactions on Machine Learning Research

    2024

  19. [27]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  20. [28]

    Alexander V Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. https://openreview.net/forum?id=j2wCrWmgMX Kernel language entropy: Fine-grained uncertainty quantification for LLM s from semantic similarities . In The Thirty-eighth Annual Conference on Neural Informa...

    2024

  21. [29]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 GPT-4 Technical Report . Preprint, arXiv:2303.08774

    2023 arXiv

  22. [30]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  23. [31]

    Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan, Douwe Kiela, and Rebecca Qian. 2024. https://arxiv.org/abs/2407.08488 Lynx: An open source hallucination evaluation model . Preprint, arXiv:2407.08488

    2024 arXiv

  24. [32]

    Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2024. https://openreview.net/forum?id=LYx4w3CAgy LLM -check: Investigating detection of hallucinations in large language models . In The Thirty-eighth Annual Confe...

    2024

  25. [33]

    Liyan Tang, Philippe Laban, and Greg Durrett. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.499 M ini C heck: Efficient fact-checking of LLM s on grounding documents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818--88...

    2024 doi

  26. [34]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod...

    2023 doi

  27. [35]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Zixia Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V Le. 2024. https://openreview.net/forum?id=4M9f8VMt2C Long-form factuality in large language models . In The Thirty-eighth Annual Co...

    2024

  28. [36]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  29. [37]

    Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dimitrios Dimitriadis, Jieyu Zhao, and Salman Avestimehr. 2025. https://aclanthology.org/2025.findings-naacl.41/ Do not design, learn: A trainable scoring function for uncertainty estimat...

    2025

  30. [38]

    Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.299 LUQ : Long-text uncertainty quantification for LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244--526...

    2024 doi

  31. [39]

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2024. https://doi.org/10.18653/v1/2024.naacl-long.390 Knowing what LLM s DO NOT know: A simple yet effective self-detection method . In Proceedings of ...

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.