Pith. sign in

REVIEW 3 major objections 4 minor 102 references

When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing uncertain claims with vaguer ones makes long-form LLM output more reliable without sacrificing much information.

desk verdict Solid framework, plausible mechanism, but the risk oracle likely flatters vague abstractions, so read the 27.73% as an upper bound until a stratified human check. read the letter →

arxiv 2602.11908 v3 pith:3TRSHRY6 submitted 2026-02-12 cs.AI cs.CLcs.LG

Shani Goren , Ido Galil , Ran El-Yaniv This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords selectiveabstractionlong-formgenerationfactualcorrectnessatomicclaimsuncertaintyestimationrisk-coveragetrade-offhallucinationmitigationinformationcontent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are fluent but unreliable in long-form writing, where a single misplaced detail can sink a biography or a legal summary. This paper argues that instead of refusing to answer when confidence is low, or chopping out whole sentences, a model should rewrite only the uncertain parts into vaguer, safer versions: 'born in Atchison, Kansas' becomes 'born in Kansas.' Across six open-source models and two long-form factuality benchmarks, this claim-level 'selective abstraction' consistently beats deletion-based baselines on the risk-coverage trade-off, improving the area under the curve by up to 27.73%. If the result holds, reliability and informativeness are not opposites but endpoints of a dial a user can turn.

What carries the argument

The core mechanism is the abstraction sequence: for each low-confidence atom, the model is prompted to produce a chain a^(0)=a, a^(1), ..., ⊤ where each step is a minimal generalization of the previous one (e.g., a precise birth date → month → year → decade), and confidence scores are elicited for the entire chain conditioned on the atom's own confidence. The selector then chooses the most specific chain element whose confidence exceeds a threshold θ; if none does, the atom is dropped. The risk-coverage curve is the evaluation instrument that demonstrates the mechanism's effect: risk falls smoothly as θ rises, and retained information is quantified via the entropy reduction of the set of ent

What would settle it

Run the same six models and benchmarks, but have independent human annotators (or a second fact-checking method grounded in a different knowledge source) label every atom in the original and abstracted outputs. If the area-under-the-risk-coverage advantage of atom-wise SA over deletion shrinks or vanishes under human labels, the claimed risk reduction was an artifact of the checker; if it persists, the effect is real.

Watch

Extended reading notes

Core claim

The central discovery is that uncertainty can be managed at the level of individual atomic claims, not whole responses or sentences. The paper defines Selective Abstraction, formalized through selective risk and coverage, and instantiates it as Atom-wise SA: decompose the generation into atomic claims (short, self-contained factual statements), elicit a verbalized confidence per atom, generate a chain of increasingly general rewrites (the abstraction sequence), and pick the most specific rewrite whose confidence clears a threshold; reconstruct text from the selected atoms. The risk of an output is the fraction of claims a fact-checking agent cannot support using an open encyclopedia; coverag

Load-bearing premise

The results stand on the assumption that the fact-checking agent's supported/unsupported labels are an unbiased measure of correctness—in particular, that vague abstractions are not systematically easier to label as supported simply because they are vague; only 102 claims were checked against human judgment.

Editorial extensions

If this is right

  • A user can dial reliability: raising the confidence threshold monotonically lowers factual risk while retaining a large fraction of the original information.
  • The method works with any confidence function; the paper shows verbalized self-confidence outperforms log-probability and P(True) baselines, suggesting better uncertainty models would further improve the trade-off.
  • Because atomization and reconstruction are prompt-based, the pipeline applies to any long-form domain, including non-factual generation such as forecasting, advice, and counterfactual reasoning, as the appendix demonstrates.
  • The conformal-style threshold selection lets a practitioner specify a target risk and receive a confidence threshold that achieves it with high probability on new prompts.
  • The observed cross-model consistency implies the benefit is not tied to one model family; hedging helps as long as the model produces confidence scores with some ranking ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test: apply the same abstraction idea in a retrieval-augmented setting, where the choice is not only 'how vague' but also 'what evidence to condition on'; the abstraction chain could be grounded in retrieved passages instead of self-confidence.
  • The information measure, which assumes a uniform prior over entities, likely underweights how surprising a rare-entity claim is; a frequency-weighted version would change coverage values and could alter which method wins at specific operating points.
  • If the fact-checking agent implicitly rewards vagueness, part of the reported risk reduction may be metric artifact; a human-annotation study on the abstracted outputs across all six models would settle this.
  • The abstraction chain itself is an audit trail: a reader can see exactly which details the model was unsure about and how it hedged, which could support human oversight in high-stakes writing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Selective Abstraction (SA), a framework for long-form generation that trades specificity for reliability by replacing low-confidence atomic claims with higher-confidence, less specific abstractions. It formalizes SA through selective risk and coverage, proposes atom-wise SA (atomize, elicit confidences, generate abstraction sequences, select the most specific abstraction above a threshold, reconstruct), and evaluates it on FactScore and LongFact-Objects with six open-source LLMs. Risk is measured by an LLM-based Wikipedia fact-checking agent; coverage is measured by an information-theoretic quantity computed from Wikidata counts via SPINACH. The main empirical claim is that atom-wise SA consistently improves the risk-coverage trade-off, with AURC improvements up to 27.73% over claim redaction. The paper also presents a conformal-inspired algorithm for selecting a confidence threshold to meet a target risk.

Significance. If the empirical claims hold, this is a useful and timely contribution: it gives a principled middle ground between full generation and binary abstention, at a finer granularity than prior selective generation work. The framing is clear, the pipeline is fully described with prompts, and the evaluation covers six models and two benchmarks while building on open resources. The comparison to redaction is fair in design, and the consistency of the direction of the effect across models is a genuine strength. The main risk to significance is external validity: the AURC gains are computed from labels produced by an LLM fact-checker and an approximate Wikidata-based information measure, and the validation of those proxies is currently thin. If the authors can close that gap, the paper would make a solid empirical contribution.

major comments (3)
  1. [Section 4.1, Appendix D.1] The risk labels are produced by a gpt-oss-120b agent instructed to mark SUPPORTED when Wikipedia contains clear supporting evidence and UNSUPPORTED otherwise. The reported human validation (F1=0.93, 102 claims, footnote 6) samples uniformly across the agent's own labels, but it is not stratified by abstraction level or specificity. Since SA inserts deliberately vague abstractions, and broad claims are easier to support on Wikipedia, label leniency toward vagueness would differentially lower SA's measured risk relative to redaction and inflate the AURC gains in Table 1. Please add human validation stratified by (i) original atoms vs abstractions and (ii) abstraction depth, reporting agreement separately for specific vs vague claims. This is the key external-validity check for the headline claim.
  2. [Table 1, Section 5] The main AURC comparison is reported as single numbers without variance. The evaluation uses only 36 FactScore and 76 LongFact prompts, and the improvements range from 5.41% to 27.73%. Without bootstrap confidence intervals over prompts or a paired significance test across thresholds, it is difficult to know whether, e.g., the Llama-3.3-70B-I gain of 6.48% is distinguishable from noise. Please report per-prompt AURC distributions, or at least standard errors or bootstrapped CIs, for the main table.
  3. [Section 4.2] Coverage, and therefore AURC, depends on the information measure I(c)=1−log|E_c|/log|E|, estimated by SPINACH queries over Wikidata. The paper acknowledges Wikidata sparsity and the uniform-prior assumption, but no validation of the natural-language-to-SPARQL conversion is reported. If the conversion systematically under- or over-counts |E_c| for abstracted predicates, the coverage values for SA relative to redaction shift and the AURC improvements change. Please include an error analysis of SPINACH on a sample of the predicates used, together with a sensitivity analysis of AURC to moderate perturbations of I(c).
minor comments (4)
  1. [Table 2] The header 'FACTSCORE∆RISK(%)' is missing spacing, and the table reports only averages. Reporting medians and per-model variability would help assess whether the matched-coverage risk gaps are consistent across models.
  2. [Appendix H] The theorem statement says P(|R(θ̂)−α|≤ϵ)≥1−δ, but ϵ is not explicitly defined in the statement; it is 'computed as a function' in the algorithm. Please spell out ϵ and clarify that R(θ̂)=P(θ_{n+1}>θ̂|D_cal) is a conditional probability over a single test claim, not the empirical risk plotted in Figure 4.
  3. [Section 2, References] There are minor typographical issues: '2In contrast' appears mid-sentence in the Related Work discussion, and reference [45] renders 'V olodya'. A final proofread would fix these.
  4. [Section 4.3] The anchor point at φ=0 with risk equal to the risk at the minimum observed coverage is a strong convention. Please justify it or show that the main conclusions are robust to alternative treatments of the low-coverage interval, such as linear interpolation from the highest-threshold operating point.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central AURC result is an empirical benchmark comparison, not a reduction to fitted inputs or a load-bearing self-citation chain.

full rationale

The paper's central claim—that atom-wise Selective Abstraction improves the risk-coverage trade-off—is supported by an end-to-end empirical pipeline evaluated on external benchmarks (FactScore, LongFact). Risk labels come from a Wikipedia-based fact-checking agent (Section 4.1), and coverage comes from a Wikidata-derived information measure (Section 4.2); neither is fitted to the test predictions, and the AURC numbers are measured, not derived from a fitted parameter. The abstraction-confidence conditioning described in Section 3 and Appendix C is a design choice that anchors abstraction confidences to the original atom confidence, but the paper does not use that conditioning to prove the risk reduction; it evaluates the resulting trade-off empirically. The self-citations to the authors' prior work on selective prediction and hierarchical selective classification ([7], [14]) are used for framing and conceptual alignment, not as load-bearing justification for the empirical results. The main validity concern—that the fact-checker may be systematically lenient toward vague abstractions—is a measurement-bias / external-validity risk, not a definitional circularity: it cannot be exhibited as an equation-level reduction of the paper's own derivation. Therefore, no circular step meeting the required evidentiary standard is present, and the honest finding is a low circularity score.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No numeric free parameters are fitted to the empirical results; the method is prompt-based and threshold is swept. The principal un-checked assumptions concern the reliability of the LLM-based risk and coverage measurements rather than invented physical or mathematical entities.

assumptions (6)
  • domain assumption Verbalized confidence scores reflect factual correctness of atoms and abstractions.
    The selection step assumes that κ_atom and κ_abs rank correct content above incorrect content. Table 4 reports AUROC 0.74 for verbal confidence on gpt-oss-120b/FactScore, so this is partially supported but not guaranteed.
  • domain assumption Agentic Wikipedia fact-checking labels are trustworthy ground truth for atom correctness.
    Risk is computed from SUPPORTED/UNSUPPORTED labels produced by a gpt-oss-120b agent searching Wikipedia (Section 4.1, Appendix D.1). Only 102 claims were human-validated, with F1 0.93; large-scale label bias is not ruled out.
  • domain assumption Uniform prior over entities in the information measure.
    I(c) = 1 − log|E_c|/log|E| assumes a uniform distribution over entities, acknowledged in Section 4.2 as a simplifying assumption with known limitations.
  • domain assumption Wikidata is a best-available proxy for real-world entity set sizes.
    Coverage is computed from Wikidata counts via SPINACH; the paper acknowledges Wikidata is sparse, so coverage numbers are approximate (Section 4.2).
  • domain assumption Calibration atoms are exchangeable with test atoms.
    Appendix H's conformal guarantee assumes exchangeability of A_cal and the test atom. Atoms drawn from the same prompt are not independent, and dependencies are ignored.
  • domain assumption Generated abstractions are logically implied by their source atoms.
    The abstraction prompt asks for generalizations that are true if the previous statement is true, but there is no verification that the model obeys this logical constraint (Appendix B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation." pith.science (2026). https://pith.science/paper/3TRSHRY6

@misc{pith2026260211908,
  author       = {Pith},
  title        = {Pith review of: When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TRSHRY6}},
  note         = {Machine review of arXiv:2602.11908}
}
read the original abstract

LLMs are widely used, yet they remain prone to factual errors that erode user trust and limit adoption in high-risk settings. One approach to mitigate this risk is to equip models with uncertainty estimation mechanisms that abstain when confidence is low. However, this binary "all-or-nothing" approach is excessively restrictive in long-form settings, often discarding valuable information. We introduce Selective Abstraction (SA), a framework that enables LLMs to trade specificity for reliability by selectively reducing the detail of uncertain content. We first formalize SA through the lenses of selective risk and coverage. We then propose Atom-wise Selective Abstraction, a claim-level instantiation that decomposes responses into atomic claims (short, self-contained statements each expressing a single fact) and replaces uncertain atoms with higher confidence, less specific abstractions. To evaluate this framework, we develop a novel end-to-end pipeline for open-ended generation that instantiates risk as factual correctness and measures coverage using an information-theoretic measure of retained information. Across six open-source models on the FactScore and LongFact-Objects benchmarks, atom-wise SA consistently outperforms existing baselines, improving the area under the risk-coverage curve (AURC) by up to 27.73% over claim removal, demonstrating that reducing specificity can boost accuracy and reliability while preserving most of their original meaning.

Figures

Figures reproduced from arXiv: 2602.11908 by the authors.

Figure 1
Figure 1. Left: Example abstraction sequence for atom￾wise selective abstraction (SA). Increasing the confidence threshold replaces low-confidence atoms with less spe￾cific, more reliable abstractions. Right: Example risk￾coverage curve comparing atom-wise SA to baselines. Model: gpt-oss-120b, dataset: FactScore. eration tasks [3, 13, 6]. Despite these advances, LLMs remain prone to factual errors, generating content that app… view at source ↗
Figure 2
Figure 2. An overview of the Selective Abstraction framework. The generated text is decomposed into atoms, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Qualitative example of atom-wise Selective Abstraction at two confidence thresholds: a higher threshold [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Risk-guided threshold selection on FactScore [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Risk–coverage curves obtained by instantiating [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Risk–coverage curves on FactScore with gpt-oss-120b, comparing verbalized confidence, [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Risk-coverage curves for atom-wise selective abstraction (ours) compared to redaction, Inline, and Self [PITH_FULL_IMAGE:figures/full_fig_p030_7.png]
Figure 8
Figure 8. Figure 8: Risk-coverage curves for atom-wise selective abstraction (ours) compared to redaction, Inline, and Self [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]
Figure 9
Figure 9. Figure 9: Risk-coverage curves on 36 FactScore entities comparing atom-wise selective abstraction (ours) compared to redaction, Inline, and Self-Revision for gpt-oss-120b (left) and gpt-oss-20b (right). We also report thresholded Inline, and Self-Revision evaluated at three thre…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

102 extracted references · 2 canonical work pages

  1. [1]

    So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V

    Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V . Le. Towards a human- like open-domain chatbot, 2020. URL https: //arxiv.org/abs/2001.09977

    arXiv 2020

  2. [2]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.CoRR, abs/2107.07511, 2021. URL https://arxiv. org/abs/2107.07511

    arXiv 2021

  3. [3]

    Introducing the claude 3 model family

    Anthropic. Introducing the claude 3 model family. https://www.anthropic.com/news/ claude-3-family, 2024

    2024

  4. [4]

    Semantic information.British Journal for the Philosophy of Science, 4(14):147–157, 1953

    Yehoshua Bar-Hillel and Rudolf Carnap. Semantic information.British Journal for the Philosophy of Science, 4(14):147–157, 1953. doi: 10.1093/bjps/iv. 14.147

    work page doi:10.1093/bjps/iv 1953

  5. [5]

    Synthese Library

    Rudolf Carnap.An Outline of a Theory of Semantic Information. Synthese Library. Springer, Dordrecht, 1964

    1964

  6. [6]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. URLhttps://arxiv.org/abs/2412.19437

    arXiv 2025

  7. [7]

    On the founda- tions of noise-free selective classification.J

    Ran El-Yaniv and Yair Wiener. On the founda- tions of noise-free selective classification.J. Mach. Learn. Res., 11:1605–1641, 2010. doi: 10.5555/ 1756006.1859904. URL https://dl.acm.org/ doi/10.5555/1756006.1859904

    arXiv 2010

  8. [8]

    Lm-polygraph: Un- certainty estimation for language models

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvi- gun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Bald- win, and Artem Shelmanov. Lm-polygraph: Un- certainty estimation for language models. In Yan- song Feng and Els Lefever, editors,Proceedings of the 2023 Conference on E...

    2023

Show all 102 references
  1. [9]

    Fact-checking the out- put of large language models via token-level uncer- tainty quantification

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the out- put of large language models via token-level unce...

    2024

  2. [10]

    Don’t hallucinate, abstain: Identifying LLM knowl- edge gaps via multi-llm collaboration

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying LLM knowl- edge gaps via multi-llm collaboration. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Me...

    2024

  3. [11]

    What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? InICLR, 2023

    Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? InICLR, 2023

    2023

  4. [12]

    Bias-reduced uncertainty estimation for deep neu- ral classifiers, 2019

    Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neu- ral classifiers, 2019. URL https://arxiv.org/ abs/1805.08206

    2019 arXiv

  5. [13]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261

    2025 arXiv

  6. [14]

    Hierar- chical selective classification, 2025

    Shani Goren, Ido Galil, and Ran El-Yaniv. Hierar- chical selective classification, 2025. URL https: //arxiv.org/abs/2405.11533

    2025 arXiv

  7. [15]

    Can language models be spe- cific? how?, 2023

    Jie Huang, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen mei Hwu. Can language models be spe- cific? how?, 2023. URL https://arxiv.org/ abs/2210.05159. 9

    2023 arXiv

  8. [16]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Trans

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Trans. Inf. Syst., 4...

    2025 doi

  9. [17]

    Optimized batch prompt- ing for cost-effective llms.Proceedings of the VLDB Endowment, 18(7):2172–2184, 2025

    Zhaoxuan Ji, Xinlu Wang, Zhaojing Luo, Zhongle Xie, and Meihui Zhang. Optimized batch prompt- ing for cost-effective llms.Proceedings of the VLDB Endowment, 18(7):2172–2184, 2025. doi: 10.14778/3734839.3734853

    2025

  10. [18]

    Survey of hallucination in natural language generation.ACM Comput

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Comput. Surv., 55(12):248:1–248:38, 2023. doi: 10.1145/3571730. URLhttps://doi.org/10.1145/3571730

    2023 doi

  11. [19]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading compre- hension. In Regina Barzilay and Min-Yen Kan, editors,Proceedings of the 55th Annual Meeting of the Association for Computational ...

    2017 doi

  12. [20]

    Language models (mostly) know what they know, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  13. [21]

    LLMs cannot (yet) match the specificity and simplicity of online commu- nities in long form question answering

    Kris-Fillip Kahl, Tolga Buz, Russa Biswas, and Gerard De Melo. LLMs cannot (yet) match the specificity and simplicity of online commu- nities in long form question answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computat...

    2024

  14. [22]

    Vem- pala, and Edwin Zhang

    Adam Tauman Kalai, Ofir Nachum, Santosh S. Vem- pala, and Edwin Zhang. Why language models hal- lucinate, 2025. URL https://arxiv.org/abs/ 2509.04664

    2025 arXiv

  15. [24]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language gen- eration

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language gen- eration. InThe Eleventh International Confer- ence on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net,

    2023

  16. [25]

    Generating with confidence: Uncertainty quantifi- cation for black-box large language models.Trans

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models.Trans. Mach. Learn. Res., 2024, 2024. URL https: //openreview.net/forum?id=DWkJCSxKU5

    2024

  17. [26]

    Semnani, Harold Tried- man, Jialiang Xu, Isaac Dan Zhao, and Monica S

    Shicheng Liu, Sina J. Semnani, Harold Tried- man, Jialiang Xu, Isaac Dan Zhao, and Monica S. Lam. SPINACH: sparql-based information nav- igation for challenging real-world questions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for ...

    2024

  18. [27]

    Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev

    Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation, 2023. URL https://arxiv....

    2023 arXiv

  19. [28]

    URL https://openreview.net/forum? id=VD-AYtP0dve

  20. [29]

    Passonneau

    Ani Nenkova and Rebecca J. Passonneau. Evaluat- ing content selection in summarization: The pyra- mid method. In Julia Hirschberg, Susan T. Du- mais, Daniel Marcu, and Salim Roukos, editors, Human Language Technology Conference of the North American Chapter of the Association ...

    2004

  21. [30]

    gpt-oss-120b & gpt-oss-20b model card,

    OpenAI. gpt-oss-120b & gpt-oss-20b model card,

  22. [31]

    findings-emnlp.938

    URL https://aclanthology.org/2024. findings-emnlp.938

    2024

  23. [32]

    Qwen3 technical report, 2025

    Qwen. Qwen3 technical report, 2025. URL https: //arxiv.org/abs/2505.09388

    2025 arXiv

  24. [33]

    Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, ed...

    2023

  25. [34]

    Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J. Liu. Out-of-distribution detection and selec- tive generation for conditional language models. In The Eleventh International Conference on Learning Representations, ICLR 2023,...

    2023

  26. [35]

    Using information content to eval- uate semantic similarity in a taxonomy.CoRR, abs/cmp-lg/9511007, 1995

    Philip Resnik. Using information content to eval- uate semantic similarity in a taxonomy.CoRR, abs/cmp-lg/9511007, 1995. URL http://arxiv. org/abs/cmp-lg/9511007

    1995 arXiv

  27. [36]

    Crowdsourcing lightweight pyramids for manual summary evalua- tion

    Ori Shapira, David Gabay, Yang Gao, Hadar Ro- nen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. Crowdsourcing lightweight pyramids for manual summary evalua- tion. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Confer...

    2019

  28. [37]

    Can knowledge graphs make large language models more trustworthy? an empirical study over open- ended question answering, 2025

    Yuan Sui, Yufei He, Zifeng Ding, and Bryan Hooi. Can knowledge graphs make large language models more trustworthy? an empirical study over open- ended question answering, 2025. URL https:// arxiv.org/abs/2410.08085

    2025 arXiv

  29. [38]

    Fact-checking complex claims with program-guided reasoning

    Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. Fact-checking complex claims with program-guided reasoning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, edi- tors,Proceedings of the 61st Annual Meeting of th...

    2023 doi

  30. [39]

    Lamda: Lan- guage models for dialog applications, 2022

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Q...

    2022 arXiv

  31. [41]

    A stitch in time saves nine: Detecting and mitigating hallucina- tions of llms by validating low-confidence gener- ation.CoRR, abs/2307.03987, 2023

    Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucina- tions of llms by validating low-confidence gener- ation.CoRR, abs/2307.03987, 2023. doi: 10.48550/ ARXIV .2307.03987. URLhttps://doi.org/10....

    2023 doi arXiv

  32. [42]

    Benchmarking uncertainty quantification methods 11 for large language models with lm-polygraph.Trans

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Ab- delrahman Boda Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking...

    2025 doi

  33. [43]

    Token- level density-based uncertainty quantification meth- ods for eliciting truthfulness of large language mod- els.CoRR, abs/2502.14427, 2025

    Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. Token- level density-based uncertainty quantification meth- ods for eliciting truthfulness of large language mod- els.CoRR, abs/2502.14427, 2025. doi: 10....

    2025 doi arXiv

  34. [44]

    Conditional validity of inductive conformal predictors.CoRR, abs/1209.2673, 2012

    Vladimir V ovk. Conditional validity of inductive conformal predictors.CoRR, abs/1209.2673, 2012

    2012 arXiv

  35. [45]

    The llama 3 herd of models.CoRR, abs/2407.21783, 2024

    Llama Team. The llama 3 herd of models.CoRR, abs/2407.21783, 2024. doi: 10.48550/ARXIV . 2407.21783. URL https://doi.org/10.48550/ arXiv.2407.21783

    2024 doi arXiv

  36. [46]

    Wikidata: a free collaborative knowledgebase.Commun

    Denny Vrandecic and Markus Kr¨otzsch. Wikidata: a free collaborative knowledgebase.Commun. ACM, 57(10):78–85, 2014. doi: 10.1145/2629489. URL https://doi.org/10.1145/2629489

    2014 doi

  37. [47]

    S. M. Towhidul Islam Tonmoy, S. M. Mehedi Za- man, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive sur- vey of hallucination mitigation techniques in large language models.CoRR, abs/2401.01313, 2024. doi: 10.48550/ARXIV .2401.01313. URL http...

    2024 doi arXiv

  38. [48]

    Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu

    Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification. In Amir Glober- sons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zh...

    2024

  39. [49]

    Nar- rowing the knowledge evaluation gap: Open- domain question answering with multi-granularity answers, 2024

    Gal Yona, Roee Aharoni, and Mor Geva. Nar- rowing the knowledge evaluation gap: Open- domain question answering with multi-granularity answers, 2024. URL https://arxiv.org/abs/ 2401.04695

    2024 arXiv

  40. [50]

    LUQ: long-text uncertainty quan- tification for llms

    Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. LUQ: long-text uncertainty quan- tification for llms. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024,...

    2024

  41. [51]

    Finding a bal- anced degree of automation for summary evaluation

    Shiyue Zhang and Mohit Bansal. Finding a bal- anced degree of automation for summary evaluation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceed- ings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pa...

    2021 doi

  42. [52]

    Machine-learning applications of algo- rithmic randomness

    V olodya V ovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algo- rithmic randomness. In Ivan Bratko and Saso Dze- roski, editors,Proceedings of the Sixteenth Inter- national Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 - 30...

    1999

  43. [54]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V . Le. Long-form factuality in large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ul- rich Paquet, J...

    2024

  44. [59]

    emnlp-main.299

    URL https://aclanthology.org/2024. emnlp-main.299

    2024

  45. [62]

    [Entity] [Fact]. ... # Examples ## Example 1 Input Text:Jennifer Coolidge is an American actress. Known for her work in the comedy genre, Coolidge is the recipient of several accolades, including a Golden Globe Award and two Primetime Emmy Awards. Atoms:

  46. [63]

    [Jennifer Coolidge] [is an American]

  47. [64]

    [Jennifer Coolidge] [is an actress]

  48. [65]

    [Jennifer Coolidge] [is known for her work in the comedy genre]

  49. [66]

    [Jennifer Coolidge] [has received several accolades]

  50. [67]

    [Jennifer Coolidge] [has received a Golden Globe Award]

  51. [68]

    ## Example 2 Input Text:Albert Einstein was a German-born theoretical physicist who is best known for developing the theory of relativity

    [Jennifer Coolidge] [has received two Primetime Emmy Awards]. ## Example 2 Input Text:Albert Einstein was a German-born theoretical physicist who is best known for developing the theory of relativity. Einstein also made important contributions to quantum mechanics. Atoms:

  52. [69]

    [Albert Einstein] [was born in Germany]

  53. [70]

    [Albert Einstein] [was a theoretical physicist]

  54. [71]

    [Albert Einstein] [is best known for developing the theory of relativity]

  55. [72]

    ## Example 3

    [Albert Einstein] [made important contributions to quantum mechanics]. ## Example 3 ... (more examples truncated) 18 Atoms Confidence Prompt # Role and Objective You are a careful and critical evaluator. Your task is to assess the factual correctness of each statement and prov...

  56. [73]

    - The confidence score must be a number between 0 and 100

    [Statement] Reasoning: [your reasoning here] Confidence: [score] - Use the same index numbers as in the list you are given. - The confidence score must be a number between 0 and 100. - Keep reasoning concise but informative — 1 to 3 sentences. - Do not repeat the statement in ...

  57. [78]

    # Review and Guidance Think step by step

    Assign a confidence score based on your belief that the statement is true. # Review and Guidance Think step by step. For each statement, explain your reasoning clearly and concisely before giving your final confidence score. 19 Abstraction Prompt # Role and Objective You are a...

  58. [79]

    Reasoning: (your reasoning here)

    [ENTITY] [FACT]. Reasoning: (your reasoning here)

  59. [80]

    Reasoning: (your reasoning here)

    [ENTITY] [GENERALIZED FACT]. Reasoning: (your reasoning here) ... K. STOP. Reasoning: (why no further generalization is meaningful) # Reasoning Steps

  60. [81]

    Start with the original sentence

  61. [82]

    Do not change the ENTITY

    Work only inside the FACT (the second bracket). Do not change the ENTITY

  62. [83]

    If the FACT contains multiple details, choose the one you are least confident about

  63. [84]

    Make the smallest logical generalization at each step

    Generalize that detail to a broader term. Make the smallest logical generalization at each step

  64. [85]

    The new statement must be logically implied by the previous one — that is, if the previous statement is true, the new one must also be true

  65. [86]

    The rest of the sentence must remain unchanged

    Modify only one detail per step. The rest of the sentence must remain unchanged

  66. [87]

    Keep the sentence format fixed: [ENTITY] [FACT]

  67. [88]

    For each abstraction, write 1–3 sentences of reasoning explaining why you chose that part to generalize, and why the generalization is valid

  68. [89]

    In that case, output ‘STOP‘ and explain why

    Continue until further abstraction would make the sentence trivial or uninformative. In that case, output ‘STOP‘ and explain why. # Generalization Examples - Date: ”March 3, 1920”→”March 1920”→”1920”→”1920s”→”20th century” - Location: ”Paris”→” ˆIle-de-France”→”France”→”Europe...

    1920

  69. [90]

    Reasoning: I’m less confident about the exact city than the country

    [Marie Curie] [was born in Warsaw, Poland]. Reasoning: I’m less confident about the exact city than the country

  70. [91]

    Reasoning: Warsaw is a city in Poland, so this is a valid generalization

    [Marie Curie] [was born in Poland]. Reasoning: Warsaw is a city in Poland, so this is a valid generalization

  71. [92]

    Reasoning: Poland is a European country

    [Marie Curie] [was born in Europe]. Reasoning: Poland is a European country

  72. [93]

    Reasoning: Further generalization would be too vague to retain meaning

    STOP. Reasoning: Further generalization would be too vague to retain meaning. ## Example 2 ... (more examples truncated) # Review and Guidance Think step by step. Each abstraction should be a clean, single-step generalization from the previous one, guided by your uncertainty. ...

  73. [94]

    - The confidence score must be a number between 0 and 100

    [Statement] Reasoning: [your reasoning here] Confidence: [score] - Use the same index numbers as in the list you are given. - The confidence score must be a number between 0 and 100. - Keep reasoning concise but informative — 1 to 3 sentences. - Do not repeat the statement in ...

  74. [95]

    Read the statement carefully

  75. [96]

    Consider whether it expresses a known fact, plausible claim, or questionable statement

  76. [97]

    Use your internal knowledge and reasoning to assess plausibility and factuality

  77. [98]

    Write a brief explanation of your reasoning

  78. [99]

    # Review and Guidance Think step by step

    Assign a confidence score based on your belief that the statement is true. # Review and Guidance Think step by step. For each statement, explain your reasoning clearly and concisely before giving your final confidence score. Reconstruction Prompt You are given a list of statem...

  79. [100]

    Use the tools toSEARCHandREADWikipedia (and only Wikipedia)

  80. [101]

    Gather relevant snippets from opened pages as evidence

  81. [102]

    SUPPORTED

    Decide whether the claim isSUPPORTEDorUNSUPPORTEDby Wikipedia. Label policy: • “SUPPORTED” only if Wikipedia contains clear supporting evidence. • “UNSUPPORTED” if (a) the claim is contradicted/refuted or (b) you cannot find supporting evidence. Output policy: • YouMUSTuse the...

  82. [103]

    • This is always the subject of the sentence

    Identify the entity at the beginning of the sentence. • This is always the subject of the sentence

  83. [104]

    people” • For places, use terms like “cities

    Classify the entity into a broad category. Use general terms only. • For people, always use “people” • For places, use terms like “cities”, “countries”, “rivers”, etc. • For objects, use terms like “books”, “paintings”, “technologies”, etc. • For abstract concepts, use terms l...

  84. [105]

    • Use the format: How many [pluralized broad category] are there?

    Broad Question: • Ask how many entities exist in that broad category. • Use the format: How many [pluralized broad category] are there?

  85. [106]

    • Keep the rest of the sentence as intact as possible

    Specific Question: • Replace the entity with its broad category. • Keep the rest of the sentence as intact as possible. • Rewrite it as a natural-sounding question starting with: How many [pluralized broad category]

  86. [107]

    French artists

    Do not use overly specific categories like “French artists” or “famous scientists.” Generalize the entity, not the predicate. # Examples ## Example 1 Input: Frida Kahlo was a Mexican painter. - Broad: How many people are there? - Specific: How many people are Mexican painters?...

  87. [108]

    These include sequence-level aggregates such as log-likelihood and (inverse) perplexity, as well as more local measures such as the minimum token log-probability

    Logprob-based scores8:We evaluate a range of confidence scores derived from token-level log-probabilities of the generated text. These include sequence-level aggregates such as log-likelihood and (inverse) perplexity, as well as more local measures such as the minimum token lo...

  88. [109]

    atoms” evaluates ranking over the original atom set, whereas “all

    P(True) [20] :elicits a binary truth judgment from the model for each generated atom (i.e., True or False) and aggregates the resulting probabilities to derive a confidence score as follows: Letp T (a)andp F (a)denote the probabilities assigned to tokens corresponding toTrueor...

  89. [111]

    findings-emnlp.111/

    URL https://aclanthology.org/2024. findings-emnlp.111/

    2024

  90. [152]

    URL https://aclanthology.org/ N04-1019/

    The Association for Computational Linguis- tics, 2004. URL https://aclanthology.org/ N04-1019/

    2004

  91. [531]

    Ice Bucket Challenge

    URL https://aclanthology.org/2021. emnlp-main.531/. 12 A Atom-wise Selective Abstraction Examples We provide qualitative examples of atom-wise selective abstraction (SA) across multiple models spanning both the FactScore and LongFact benchmarks. Each example shows the original...

    2021

  92. [2024]

    doi: 10.18653/v1/2024.findings-emnlp

    Association for Computational Linguis- tics. doi: 10.18653/v1/2024.findings-emnlp

    2024 doi

  93. [2025]

    URL https://arxiv.org/abs/2508. 10925

  94. [5262]

    Association for Computational Linguistics,

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.