Pith. sign in

REVIEW 3 major objections 6 minor 47 references

How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that LLM-written explanations of SHAP tables can be scored automatically by extracting each feature's rank, sign, value, and assumption from the narrative, and that doing so reveals that LLMs often silently correct…

desk verdict A useful proof-of-concept framework for automated XAI narrative evaluation, but the headline sign self-correction finding needs better extraction validation before it can be trusted. read the letter →

arxiv 2412.10220 v1 pith:ZF2SVLDD submitted 2024-12-13 cs.CL cs.AI

Timour Ichmoukhamedov , James Hinns , David Martens This is my paper · ORCID
classification cs.CLcs.AI
keywords explainableAISHAPnarrativesLLMfaithfulnessevaluationextraction-basedmetricshallucinationdetectiontextembeddingsperplexitynarrativegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a fully automated way to grade the stories that LLMs write to explain model predictions from SHAP tables. The key mechanism is an extraction LLM that pulls out each feature's rank, sign, numeric value, and causal assumption from a narrative, then checks them against the original SHAP table. Across three tabular datasets and four LLMs, the metrics behave as expected: structured prompts improve faithfulness, embedding similarity matches the right human reference narrative more often than BLEURT, and when SHAP tables are manipulated, faithfulness scores collapse. A surprising result is that LLMs often 'correct' the manipulated sign of a feature, suggesting they sacrifice fidelity for plausibility. If the metrics are trustworthy, they provide a scalable substitute for human evaluation and expose a previously unnoticed hallucination failure mode.

What carries the argument

The extraction pipeline: a second LLM, always gpt-4o, reads the narrative and outputs a structured dictionary with, for each feature mentioned, a rank $r_j$, a sign $s_j\in\{-1,1\}$, a value $v_j$ (or null), and an assumption sentence (or null). Downstream metrics then score these extractions: rank agreement (RA), sign agreement (SA), and value agreement (VA) against the ground-truth SHAP table, perplexity for assumptions, and cosine similarity of narrative embeddings for human similarity. The extraction step is the load-bearing object: it converts free text into the structured quantities that every faithfulness score depends on.

What would settle it

Hand-annotate a fresh sample of narratives from a different generation model family and compare the extraction model's outputs to the human labels; if rank/sign/value agreement with the labels drops well below the roughly 98% reported, then the faithfulness scores in the comparison tables are not portable. Also rerun the manipulated-table experiment with a prompt instruction that the narrative must state the given signs verbatim; if sign agreement then returns near 100%, the self-correction is a style preference, not an irremovable bias.

Watch

Extended reading notes

Core claim

This paper claims that narrative quality can be measured automatically by decomposing an LLM-written explanation into an extractable dictionary of feature, rank, sign, value, and assumption, and scoring faithfulness against the source SHAP table, human similarity via embeddings, and assumption plausibility via perplexity. Using gpt-4o as the extractor, the authors report high validation accuracy (60/60 faulty narratives flagged, 1 false negative in 53). The central discovery is the sign self-correction phenomenon: when the SHAP table passed to a generation LLM has its feature signs inverted, the LLM writes a narrative whose implied signs agree with the manipulated table only about a quarter to a third of the time, far below the roughly 96% predicted from normal narratives. The authors interpret this as evidence that the models override the given explanation with their own internal knowledge, a form of hallucination that the faithfulness metrics bring into view.

Load-bearing premise

The entire faithfulness pipeline assumes the extraction model—gpt-4o in every experiment—correctly recovers rank, sign, value, and assumptions from a narrative, and that the small validation set (60 manipulated plus 60 human stories) is representative enough to guarantee this.

Editorial extensions

If this is right

  • The metrics can act as an automated validation layer before a narrative reaches a user, replacing or complementing slow, expensive human surveys.
  • Longer, structured generation prompts measurably improve rank and sign faithfulness across all tested models, giving a concrete lever for prompt design.
  • Because LLMs tend to silently correct implausible signs, a smooth-sounding narrative can misrepresent the underlying explanation; faithfulness scores on the narrative alone will not reveal that the original SHAP table was wrong.
  • Perplexity works for detecting deliberately irrational assumptions in isolation, but it does not robustly flag the assumptions inside real generated narratives, so it should not be used as a standalone plausibility check.
  • Modern embedding-based cosine similarity already matches the correct human narrative more reliably than BLEURT and can partially sense when a narrative was written from manipulated data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A deployed risk: if an upstream attribution contains errors (or a model has learned spurious correlations), an LLM narrator may 'fix' them into a more coherent but wrong story. The resulting text can sound more faithful than the model actually is, a failure that human readers, and possibly this metric suite, can miss.
  • The true-negative validation uses gpt-4o for both generating and extracting manipulated narratives; a shared model bias could inflate the reported extraction accuracy. A fairer test would hand-label narratives from several model families and measure extraction accuracy per family.
  • The extract-then-score design is generic enough to be embedded in agentic pipelines: regenerate the narrative until faithfulness thresholds are met, or apply the same extraction to other explanation formats such as LIME or counterfactual rules.
  • Embedding similarity could become a proper XAI narrative metric by fine-tuning a small regression head on top of embeddings to predict rank and sign agreement, rather than using raw cosine distance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a framework for automated evaluation of LLM-generated XAI narratives for tabular classification. It introduces an extraction LLM that recovers per-feature rank, sign, value, and assumptions from a narrative, and then computes faithfulness metrics (RA/SA/VA via Eq. 1). It also evaluates assumption quality via perplexity of extracted assumption sentences (Eq. 2) and human similarity via embedding cosine similarity, comparing against BLEURT. The authors validate the extraction model on 60 manipulated and 60 human narratives (Table I), validate perplexity on 47 manually modified assumptions (Fig. 5), and show that embedding-based matching outperforms BLEURT (Fig. 6). They then compare four LLMs and two prompt types, and report that when the SHAP table is manipulated (rank/sign inversion), sign accuracy stays around 0.30 instead of the chance-level ~0.04 they estimate, which they interpret as LLMs self-correcting implausible signs—a hallucination risk.

Significance. If the metrics are trustworthy, the framework provides a scalable, human-free evaluation pipeline for XAI narratives, with the extraction-based faithfulness scores and embedding similarity being concrete and reusable. The paper has several strengths: the faithfulness ground truth is an external SHAP table, no constants are fitted, per-dataset results are reported, and the authors include explicit caveats about limitations (e.g., Section IV A and the Conclusion on perplexity). The self-correction finding, if robust, is a practically important failure mode for XAI narratives. However, the current validation leaves exact extraction accuracy unmeasured and relies on small samples without statistical tests, so the strength of the central claims is not yet established. The framework is promising but needs additional validation before it can be regarded as ready for automated narrative quality control.

major comments (3)
  1. [Section IV A, Table I and Eq. (1)] The extraction model is validated only as a binary classifier over an 'any error' flag, but the downstream faithfulness metrics in Eq. (1) and Tables II–VI require exact per-feature rank/sign/value extraction. The 60/60 true-negative result does not establish exact-extraction accuracy: a narrative with multiple errors is correctly flagged even if only one error is recovered, and no human annotation of exact extractions is reported. Because gpt-4o is used both to generate and to extract the manipulated narratives, the measured SA of ~0.30 on manipulated tables (Table III) may reflect the extractor's parametric correction of implausible signs rather than the generator's self-correction. This is load-bearing for the central hallucination claim. Please add human-labeled exact extractions (or a second, independent extraction model) and report per-feature agreement for rank, sign, and value.
  2. [Section IV A, Fig. 5, Table III, Conclusion] The assumption validation uses 47 manually manipulated assumptions and shows a perplexity increase in that idealized setting, but the same metric behaves inconsistently on the manipulated narratives: in Table III, ΔPPL is negative for several models and iterations (e.g., llama-3-70b under both PPL(L) and PPL(M)), and the Conclusion concedes that 'perplexity does not appear to yield consistent behavior across models.' As stated, the claim that perplexity 'can be expected to work' for detecting unreasonable assumptions is not supported for the actual generation scenario. Please provide a validation on generated/manipulated narratives or temper the claim accordingly.
  3. [Section IV A and IV B] All quantitative comparisons rest on samples of 60 narratives (20 per dataset) and 47 assumptions, with no confidence intervals or significance tests; the ranges in Table III are min|max over four runs at T=0, which is not a statistical error estimate. Consequently, statements such as 'the long prompt consistently achieves a better performance' (Section IV B) and 'cosine similarity outperforms BLEURT' (42/60 vs 29/60, Fig. 6) are not established beyond descriptive comparison. Please add bootstrap confidence intervals or significance tests for the headline comparisons, especially the sign self-correction rates that motivate the hallucination finding.
minor comments (6)
  1. [Tables IV and V] Tables IV and V are identical in the manuscript, both titled 'Student'; one of them presumably should report the Fifa dataset. This needs to be corrected to support the per-dataset analysis.
  2. [Section IV A] The true-positive rate is derived by assuming that the false-positive rate is low, but the false-positive rate was measured on manipulated gpt-4o narratives, not on human-written narratives; please clarify why this transfer of an error-rate estimate is justified.
  3. [Eq. (1)] The denominator n - sum of phi entries is zero if all extracted values for a feature are phi; please specify the convention in that edge case (e.g., the term is skipped).
  4. [Section IV B, Table III] The 'naive expectation' of about 4% sign accuracy under the manipulation is not derived; please show the baseline calculation (e.g., assuming random sign generation or independent sign flips) so the reader can verify the claimed deviation.
  5. [Fig. 8 caption] The phrase 'filter out all features that occur less than once in all 20 Student instances' is unclear; please rephrase, for example as 'features with fewer than 16 occurrences in the pooled 320 narratives.'
  6. [Introduction and Fig. 6 caption] There are minor typos: 'give raise' should be 'give rise' (Introduction), and 'explanatizons' should be 'explanations' (Fig. 6 caption).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the faithfulness and similarity metrics are checked against external SHAP tables and human references, not against fitted outputs; only a minor non-load-bearing self-citation and a shared generator/extractor validation gap are present.

full rationale

The paper's load-bearing comparisons are self-contained. Faithfulness (RA/SA/VA, Eq. 1) compares gpt-4o extractions of rank, sign, and value against the original SHAP table, an external ground truth not derived from the extraction or generation models. No parameter is fitted to the target claim, and no equation in Sections III-IV defines a metric in terms of the quantity it is supposed to predict. The extraction-model validation (Section IV A) does use gpt-4o for both generation and extraction of the manipulated narratives and checks only a coarse any-error flag, so exact per-feature sign/rank extraction accuracy is not established by human labels; the paper itself concedes "more thorough experiments are needed." This is a measurement-validity risk that could affect the sign self-correction finding, but it is not a definitional reduction: the extracted sign is not defined by the generated sign. The assumptions perplexity is computed with independent small LLMs (Llama-3-8b, Mistral-7b) on extracted statements, and the paper explicitly addresses the concern of "first generating an assumption using an LLM and then measuring it relative to an LLM again"; the validation uses manually manipulated assumptions, so no target is fitted. The embedding-based human-similarity metric is validated against human-written reference narratives, again external. References [7] and [11] are prior work by the same group, but they supply only the narrative-generation recipe, datasets, and related work, not a load-bearing theorem or the metric equations. The paper's own caveats (Section IV A and the Section III warning that perplexity "has also been criticized") further show the claims are treated as empirical rather than assumed. Accordingly, the only circularity-adjacent feature is a minor, non-load-bearing self-citation, which puts this at score 2 rather than higher.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The framework depends on several unproven premises: SHAP tables are faithful to the model; the extraction LLM can reliably recover rank, sign, value, and assumptions; perplexity is a meaningful proxy for assumption reasonableness; and the researchers' own human narratives are a valid reference. The extraction validation partly uses the same model for generation and extraction, and the perplexity proxy is shown to be inconsistent on manipulated narratives, so these premises carry real risk.

assumptions (6)
  • domain assumption The SHAP explanations used as ground truth are themselves faithful to the prediction model.
    Stated in Section I: 'In the assumption that the SHAP table is itself fully faithful, this also captures the faithfulness of the narrative relative to the prediction model.' This is not validated.
  • domain assumption The extraction LLM (gpt-4o) can reliably extract rank, sign, value, and assumptions from narratives.
    The entire faithfulness pipeline depends on the extraction model. The paper validates it on a small sample (60 manipulated, 60 human narratives) but uses gpt-4o for both generation and extraction, which may share systematic biases.
  • domain assumption Perplexity computed with Llama-3-8B (and Mistral-7B) is a valid proxy for the reasonableness of assumptions.
    The paper notes perplexity has been criticized [27] and shows it is inconsistent on manipulated narratives, yet uses it as a downstream metric. This assumption is only partially supported.
  • domain assumption Randomly permuting the SHAP table before narrative generation produces narratives that are unfaithful in rank and/or sign.
    Used in Section IV A to validate the extraction model's true negative rate. The paper states 'we will assume that all narratives generated in this way indeed have inaccuracies', but does not verify each narrative.
  • domain assumption Human-written narratives (by the research team) are a suitable reference for human similarity.
    The human narratives are written by the authors' own team with the same prompt, so they may not represent typical end users or domain experts.
  • domain assumption The three datasets (Fifa, German Credit, Student) and 20 instances per dataset are representative of tabular classification XAI use cases.
    Used throughout; these are the same datasets as the authors' prior work [7], which limits independence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives." pith.science (2026). https://pith.science/paper/ZF2SVLDD

@misc{pith2026241210220,
  author       = {Pith},
  title        = {Pith review of: How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZF2SVLDD}},
  note         = {Machine review of arXiv:2412.10220}
}
read the original abstract

A rapidly developing application of LLMs in XAI is to convert quantitative explanations such as SHAP into user-friendly narratives to explain the decisions made by smaller prediction models. Evaluating the narratives without relying on human preference studies or surveys is becoming increasingly important in this field. In this work we propose a framework and explore several automated metrics to evaluate LLM-generated narratives for explanations of tabular classification tasks. We apply our approach to compare several state-of-the-art LLMs across different datasets and prompt types. As a demonstration of their utility, these metrics allow us to identify new challenges related to LLM hallucinations for XAI narratives.

Figures

Figures reproduced from arXiv: 2412.10220 by the authors.

Figure 1
Figure 1. FIG. 1. Overview of the workflow presented in the paper. First, a narrative based on a SHAP input table [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 2
Figure 2. It is quite general and in principle allows [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: FIG. 3 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 4
Figure 4. Figure 4: FIG. 4. Overview tables for SHAP faithfulness, reflecting the differences between the extracted quantities [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: FIG. 5. The increase in perplexity after manipu [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: FIG. 6. For every human written narrative out of 60 on the x-axis, the human similarity distance to 60 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: FIG. 7. To create a manipulated narrative for the experiments discussed in Sec. (IV B) an LLM is prompted [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: FIG. 8. The counts of various types of feature swaps that happen relative to the manipulated SHAP table for [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages

  1. [1]

    Proposing an automated framework with multiple metrics to evaluate narratives across the categories introduced above (Faithfulness, Human Similarity, Assump- tions)

    work page

  2. [2]

    Exploring and validating the behavior of the metrics on several proof-of-concept ex- periments to establish their trustworthi- ness

    work page

  3. [3]

    Flemish AI Research Program

    Applying the metrics to compare narra- tive generation across several datasets and LLMs and demonstrate how they allow identifying new challenges in LLMs for XAI. A schematic overview of our methodology is presented in Fig. 1. Most importantly, to achieve full automation we introduce an extrac- tion model that can extract various quantities of interest in...

    work page arXiv

  4. [4]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Confer- ence on Neural Information Processing Sys- tems, NIPS’17, page 4768–4777, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964

    work page 2017

  5. [5]

    ”why should i trust you?”: Explain- ing the predictions of any classifier

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ”why should i trust you?”: Explain- ing the predictions of any classifier. In Pro- ceedings of the 22nd ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining , KDD ’16, page 1135–1144, New York, NY, USA, 2016. Association for Com- puting Machinery. ISBN 9781450342322. doi: 10.1...

    arXiv 2016

  6. [6]

    A value for n-person games

    Lloyd S Shapley et al. A value for n-person games. 1953

    1953

  7. [7]

    The inadequacy of shapley values for explainabil- ity, 2023

    Xuanxiang Huang and Joao Marques-Silva. The inadequacy of shapley values for explainabil- ity, 2023. URL https://arxiv.org/abs/2302. 08160

    work page 2023

  8. [8]

    Ex- plainability is not a game

    Joao Marques-Silva and Xuanxiang Huang. Ex- plainability is not a game. Commun. ACM , 67(7):66–75, jul 2024. ISSN 0001-0782. doi: 10.1145/3635301. URL https://doi.org/10. 1145/3635301

    doi:10.1145/3635301 2024

Show all 47 references
  1. [9]

    Natural language explanations for machine learning classification decisions

    James Burton, Noura Al Moubayed, and Amir Enshaei. Natural language explanations for machine learning classification decisions. In 2023 International Joint Conference on Neu- ral Networks (IJCNN) , pages 1–9, 2023. doi: 10.1109/IJCNN54540.2023.10191637

    2023

  2. [10]

    Tell me a story! narrative-driven xai with large language models, 2023

    David Martens, James Hinns, Camille Dams, Mark Vergouwen, and Theodoros Evgeniou. Tell me a story! narrative-driven xai with large language models, 2023. URL https: //arxiv.org/abs/2309.17057

    2023 arXiv

  3. [11]

    Llms for xai: Future directions for explaining explanations, 2024

    Alexandra Zytek, Sara Pid` o, and Kalyan Veera- machaneni. Llms for xai: Future directions for explaining explanations, 2024. URL https: //arxiv.org/abs/2405.06064

    2024 arXiv

  4. [12]

    Natural lan- guage counterfactual explanations for graphs using large language models, 2024

    Flavio Giorgi, Cesare Campagnano, Fabrizio Silvestri, and Gabriele Tolomei. Natural lan- guage counterfactual explanations for graphs using large language models, 2024. URL https: //arxiv.org/abs/2410.09295

    2024 arXiv

  5. [13]

    Tagex- plainer: Narrating graph explanations for text- attributed graph learning models, 2024

    Bo Pan, Zhen Xiong, Guanchen Wu, Zheng Zhang, Yifei Zhang, and Liang Zhao. Tagex- plainer: Narrating graph explanations for text- attributed graph learning models, 2024. URL https://arxiv.org/abs/2410.15268

    2024 arXiv

  6. [14]

    Graphx- ain: Narratives to explain graph neural net- works, 2024

    Mateusz Cedro and David Martens. Graphx- ain: Narratives to explain graph neural net- works, 2024. URL https://arxiv.org/abs/ 2411.02540

    2024 arXiv

  7. [15]

    Faithful and plausible natu- ral language explanations for image classifi- cation: A pipeline approach

    Adam Wojciechowski, Mateusz Lango, and Ondrej Dusek. Faithful and plausible natu- ral language explanations for image classifi- cation: A pipeline approach. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Com- putational Linguist...

    2024 doi

  8. [16]

    In-context explainers: Harnessing llms for ex- plaining black box models, 2024

    Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. In-context explainers: Harnessing llms for ex- plaining black box models, 2024. URL https: //arxiv.org/abs/2310.05797

    2024 arXiv

  9. [17]

    Explaining ma- chine learning models with interactive natural language conversations using talktomodel

    Dylan Slack, Satyapriya Krishna, Himabindu Lakkaraju, and Sameer Singh. Explaining ma- chine learning models with interactive natural language conversations using talktomodel. Na- ture Machine Intelligence , 5(8):873–883, Aug

  10. [18]

    ME- TEOR: An automatic metric for MT evalu- ation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. ME- TEOR: An automatic metric for MT evalu- ation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss, editors, Pro- ceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Me...

    2005

  11. [19]

    Keane, Eoin M

    Mark T. Keane, Eoin M. Kenny, Eoin Delaney, and Barry Smyth. If only we had better coun- terfactual explanations: Five key deficits to rectify in the evaluation of counterfactual xai techniques. In Zhi-Hua Zhou, editor, Pro- ceedings of the Thirtieth International Joint Confer...

  12. [20]

    Do models explain them- selves? Counterfactual simulatability of natural language explanations

    Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen Mckeown. Do models explain them- selves? Counterfactual simulatability of natural language explanations. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nu...

  13. [21]

    Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for 18 Computation...

    2020 doi

  14. [22]

    BLEURT: Learning robust metrics for text generation

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, ed- itors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, pages ...

    2020 doi

  15. [23]

    The disagreement problem in ex- plainable machine learning: A practitioner’s perspective

    Satyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. The disagreement problem in ex- plainable machine learning: A practitioner’s perspective. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //openreview.net/fo...

    2024

  16. [24]

    F ActScore: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. F ActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors...

    2023 doi

  17. [25]

    PRobELM: Plausibility ranking evaluation for language models

    Moy Yuan, Eric Chamoun, Rami Aly, Chenxi Whitehouse, and Andreas Vlachos. PRobELM: Plausibility ranking evaluation for language models. In First Conference on Language Mod- eling, 2024. URL https://openreview.net/ forum?id=k8KS9Ps71d

    2024

  18. [26]

    A survey on natural language counterfactual generation, 2024

    Yongjie Wang, Xiaoqi Qiu, Yu Yue, Xu Guo, Zhiwei Zeng, Yuhong Feng, and Zhiqi Shen. A survey on natural language counterfactual generation, 2024. URL https://arxiv.org/ abs/2407.03993

    2024 arXiv

  19. [27]

    You can generate it again: Data-to-text generation with verification and correction prompting, 2023

    Xuan Ren and Lingqiao Liu. You can generate it again: Data-to-text generation with verification and correction prompting, 2023. URL https: //arxiv.org/abs/2306.15933

    2023 arXiv

  20. [28]

    The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey, 2024

    Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey, 2024. URL https: //arxiv.org/abs/2404.11584

    2024 arXiv

  21. [29]

    Sentence- bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence- bert: Sentence embeddings using siamese bert-networks. In Conference on Empiri- cal Methods in Natural Language Processing ,

  22. [30]

    Towards few-shot fact-checking via perplexity

    Nayeon Lee, Yejin Bang, Andrea Madotto, and Pascale Fung. Towards few-shot fact-checking via perplexity. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proce...

    2021

  23. [31]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, and et al. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  24. [32]

    Claude sonnet 3.5

    Anthropic. Claude sonnet 3.5. 2024. URL https://www.anthropic.com/news/ claude-3-5-sonnet

    2024

  25. [33]

    Perplexity from plm is unreliable for evaluating text quality, 2023

    Yequan Wang, Jiawen Deng, Aixin Sun, and Xuying Meng. Perplexity from plm is unreliable for evaluating text quality, 2023. URL https: //arxiv.org/abs/2210.05892

    2023 arXiv

  26. [34]

    Efficient estimation of word representations in vector space, 2013

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013. URL https://arxiv.org/abs/1301.3781

    2013 arXiv

  27. [35]

    Mistral large 2

    The Mistral AI Team. Mistral large 2

  28. [36]

    Zhang, Mark Har- man, and Meng Wang

    Shuyin Ouyang, Jie M. Zhang, Mark Har- man, and Meng Wang. An empirical study of the non-determinism of chatgpt in code gen- eration. ACM Trans. Softw. Eng. Methodol. , September 2024. ISSN 1049-331X. doi: 10.1145/3697010. URL https://doi.org/10. 1145/3697010. Just Accepted

    2024 doi

  29. [37]

    Nv-embed: Im- proved techniques for training llms as gener- alist embedding models, 2024

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Im- proved techniques for training llms as gener- alist embedding models, 2024. URL https: //arxiv.org/abs/2405.17428

    2024 arXiv

  30. [38]

    Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https://openreview.net/ foru...

    2024

  31. [39]

    Context-faithful prompting for large language models

    Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 14544–14556, Singapore, December ...

    2023 doi

  32. [40]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md

    2024

  33. [41]

    The llama 3 herd of mod- els, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. The llama 3 herd of mod- els, 2024. URL https://arxiv.org/abs/2407. 21783

    2024

  34. [45]

    Entity-based knowledge con- flicts in question answering

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge con- flicts in question answering. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on...

    2021 doi

  35. [2019]

    19 org/CorpusID:201646309

    URL https://api.semanticscholar. 19 org/CorpusID:201646309

  36. [2021]

    URL https: //doi.org/10.24963/ijcai.2021/609

    doi:10.24963/ijcai.2021/609. URL https: //doi.org/10.24963/ijcai.2021/609. Survey Track

    2021 doi

  37. [2023]

    doi:10.1038/s42256- 023-00692-8

    ISSN 2522-5839. doi:10.1038/s42256- 023-00692-8. URL https://doi.org/10.1038/ s42256-023-00692-8

    doi

  38. [2024]

    URL https://proceedings.mlr.press/ v235/chen24bl.html

  39. [2407]

    URL https://huggingface.co/ mistralai/Mistral-Large-Instruct-2407

    2024. URL https://huggingface.co/ mistralai/Mistral-Large-Instruct-2407

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.