Pith. sign in

REVIEW 4 major objections 6 minor 33 references

Beyond Facts: Evaluating Intent Hallucination in Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Complex multi-condition queries frequently cause LLMs to omit or misinterpret parts of the query, and this paper introduces a 20,068-item benchmark plus a constraint-weighted score to detect such 'intent hallucination' automatically.

desk verdict FAITHQA is a real benchmark, but its metric validation is circular and one of its main trend claims is contradicted by its own Table 4. read the letter →

arxiv 2506.06539 v1 pith:E55WZLYP submitted 2025-06-06 cs.CL cs.AI

Yijie Hao , Haofei Yu , Jiaxuan You This is my paper
classification cs.CLcs.AI
keywords intenthallucinationqueryalignmentLLMevaluationconstraintdecompositionbenchmarkretrieval-augmentedgenerationLLM-as-judgefactual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When a user packs several conditions into one query, current LLMs often answer only some of them—or answer conditions the user never stated—while remaining factually accurate. The paper names this failure 'intent hallucination' and argues it is a distinct, under-measured problem that standard factual checks miss. To study it, the authors build FAITHQA, a 20,068-problem benchmark spanning direct question answering, creative writing, and retrieval-augmented generation, and propose CONSTRAINTSCORE, which decomposes a query into weighted intent constraints and scores how many the response satisfies. Human validation indicates the score tracks human judgments far more closely than simple LLM-as-judge scoring (MSE 0.50 vs 4.72), suggesting automatic, query-centric intent evaluation is practical.

What carries the argument

The central object is the intent constraint: a short statement expressing one requirement—such as a location, time, subject, action, qualifier, or quantity—that a generation must address, classified as mandatory, important, or optional. The metric's mechanism is a two-step LLM pipeline: first, intent constraint mapping, which decomposes the query into these weighted constraint sets and includes a preliminary check that halts if needed external content is missing; second, intent constraint scoring, which asks an LLM to judge each constraint's satisfaction and computes an importance-weighted score $\frac{W_s}{W_t}\times 10$. The binary satisfaction function $S_\varphi(c,y)$ and the weighted normalization carry the argument, making the query itself rather than factual accuracy the evaluation target.

What would settle it

Ask independent human raters to score the same 1,000 responses holistically without seeing the constraint-decomposition rubric, and compare CONSTRAINTSCORE's mean squared error against those ratings; if the gap over the LLM-as-judge baseline narrows to non-significance, the claimed human alignment would not survive independent judgment.

Watch

Extended reading notes

Core claim

The paper establishes that intent hallucination is a real and frequent mode of LLM failure independent of factual accuracy: models either omit parts of the query or misinterpret it by responding to invented components, and the rate rises as query complexity grows and in long-context RAG settings. It further claims that CONSTRAINTSCORE, an automatic metric that decomposes the query into mandatory, important, and optional intent constraints and computes an importance-weighted satisfaction score normalized to 0-10, detects this failure more reliably than LLM-as-a-judge baselines, which systematically underestimate response quality. The headline evidence is a human evaluation of 1,000 prompt-response pairs in which CONSTRAINTSCORE's mean squared error against human scores was 0.50, compared with 4.72 for the baseline, supporting the paper's central assertion that query-centric, constraint-level evaluation is both feasible and better aligned with human judgment.

Load-bearing premise

The human ground truth was produced by asking annotators to apply the same constraint-decomposition rubric that CONSTRAINTSCORE uses, so the reported agreement could partly reflect both sides implementing the same method rather than independent human judgment.

Editorial extensions

If this is right

  • Intent hallucination is common even for state-of-the-art LLMs; on several FAITHQA tasks, the rate of perfectly constraint-satisfying responses falls below 50%.
  • Performance degrades as query complexity increases: Perfect rates decline from easy to hard constraint counts, and long RAG inputs produce more omission-style hallucinations.
  • Factual verification alone becomes a weaker detector as models scale, because larger models produce fewer factually wrong responses while still omitting query constraints.
  • CONSTRAINTSCORE offers an automatic query-centric evaluation that aligns with human judgment much more closely than zero-shot LLM-as-judge scoring.
  • The constraint-decomposition protocol can serve as a reusable evaluation approach for instruction-following and RAG faithfulness beyond the specific benchmark tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same constraint-decomposition idea could be turned into a training signal: penalizing omission or misinterpretation of decomposed constraints during fine-tuning might reduce intent hallucination, though the paper does not test this.
  • Because the metric is LLM-based, its reliability on queries with subtle or contradictory constraints may inherit the judge model's own biases; testing a subset of constraints with a non-LLM verifier would quantify that dependence.
  • The misinterpretation tasks suggest a testable extension: varying how 'inventable' a missing piece of context is could reveal whether models hallucinate missing evidence because of training-corpus priors or because of context-tracking failures.
  • FAITHQA's query-only and RAG splits could be used to measure whether retrieval noise amplifies omission or misinterpretation differently, a comparison the paper reports descriptively but does not isolate experimentally.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the concept of 'intent hallucination'—defined as a response omitting or misinterpreting constraints from a complex query—and proposes a benchmark FAITHQA (20,068 problems) covering open-answer, creative writing, and RAG-based evaluation tasks, alongside an automatic metric CONSTRAINTSCORE. CONSTRAINTSCORE decomposes a query into mandatory/important/optional intent constraints, uses an LLM to check each constraint against the response, and computes a weighted 0–10 score. The authors evaluate seven LLMs, reporting that intent hallucination is prevalent, worsens with more constraints, and that CONSTRAINTSCORE aligns more closely with human judgment than a self-consistent LLM-as-judge baseline (MSE 0.50 vs 4.72 on 1,000 sampled responses).

Significance. If the central claims were sound, the paper would contribute a useful query-centric perspective on non-factual hallucination, a large benchmark, and a practical automatic metric. The detailed prompts and appendices make the method reproducible, and the qualitative analyses of specific failure modes are informative. However, the claimed validation of CONSTRAINTSCORE rests on a circular human evaluation, and a key empirical conclusion is contradicted by the paper's own tables. These issues are load-bearing: they directly undermine the headline comparisons and the stated finding that intent hallucination reliably increases with query complexity.

major comments (4)
  1. [§7, Appendix C, Figure 5, Table 5] The human ground truth used to validate CONSTRAINTSCORE is circular. The evaluation rubric instructs annotators to decompose each query into mandatory/important/optional intent constraints and compute a weighted constraint score (Figure 5; the same categories and weights appear in the automatic method, §4.1–4.2). Thus the reported MSE gap (0.50 vs 4.72) measures how well GPT-4o reproduces a human who is forced through the same decomposition-and-scoring rubric, not how well either agrees with independent holistic human judgment. A human condition that elicits scores without exposing the constraint-rubric is required to break the circularity.
  2. [§7, Table 4] The claim that Perfect rates 'consistently decline' as the number of constraints increases is contradicted by Table 4: GPT-4o Tech Easy 0.42 vs Hard 0.53, and LLaMA3-70B improves from Easy to Hard on Culture (0.48→0.66) and History (0.52→0.63); Table 9 shows LLaMA3-70B Poem Easy = Hard = 0.51. The conclusion that greater query complexity reliably increases intent hallucination is therefore unsupported.
  3. [§6, Table 5] The baseline is not a 'pure LLM grading baseline' as claimed. The prompt template in Table 5 instructs the LLM to break the query into intent constraints with Mandatory/Important/Optional priority levels and to detect omission vs misinterpretation—the same constraint-based procedure used by CONSTRAINTSCORE. The comparison therefore does not isolate the value of constraint decomposition, and the statement that LLM judgers 'tend to provide biased evaluations' is not a fair characterization of the implemented baseline.
  4. [§7, Table 4] The interpretation of the Factual Verifiable Hallucination Rate is internally inconsistent. A lower Fact rate means fewer hallucinated responses are factually accurate, which makes factual checking easier, not harder; the paper claims the opposite ('becomes more challenging to detect hallucinations through factual checks as model size grows'). The caption's 'higher the better' also conflicts with the surrounding analysis.
minor comments (6)
  1. [Table 4 caption] The caption labels Fact as 'higher the better' while the text treats lower values as evidence that factual checks are less useful; this needs to be reconciled.
  2. [Appendix D.1, Table 5] The LLM-as-judge prompt does not explicitly instruct the model to output a numeric 1–10 score, although Section 6 says the baseline evaluates 'on a scale from 1 to 10'; the template should include this instruction.
  3. [Table 8] Table 8 uses the labels 'Minor Fabrication' and 'Major Fabrication' for what the rest of the paper calls 'Omission' and 'Misinterpretation'; the terminology should be made consistent.
  4. [Section 2] The claim that FAITHQA is 'the first' benchmark beyond factual verification is undermined by the cited InfoBench (Qin et al., 2024), which already evaluates instruction following via query decomposition; the novelty claim should be qualified.
  5. [Section 6] The sentence about the test set '150 randomly sampled questions... across every single category and difficulty' is ambiguous; Table 17 clarifies that the count varies (e.g., 300 for Creative Writing), and the text should state the per-cell sample sizes explicitly.
  6. [Table 4] The model name 'Llama3-70b' differs from 'LLaMA3-70B' used elsewhere; unify the capitalization.

Circularity Check

2 steps flagged · score 8.0 of 10

CONSTRAINTSCORE's central validation is self-referential: the human 'ground truth' in §7 is a human execution of CONSTRAINTSCORE's own decomposition rubric, so the headline MSE advantage does not establish alignment with independent human judgment.

  1. self definitional [Section 7 'Experimental Results' and Appendix C 'Human Evaluation']
    "The evaluation rubric for human annotators requires calculating the CONSTRAINTSCORE based on how well the response addresses each of the decomposed intent constraints. ... The MSE for CONSTRAINTSCORE is 0.50, which is significantly lower than the Baseline's MSE of 4.72. This result highlights that CONSTRAINTSCORE outperforms the Baseline and aligns more closely with human judgment."

    The human reference is computed by instructing annotators to apply the same constraint decomposition and weighted satisfaction calculation that CONSTRAINTSCORE automates, including the mandatory/important/optional categories from Section 4.1 and the weights from Section 4.2. The automatic metric is therefore validated against a human execution of itself, not against independent holistic human judgment. The reported MSE gap shows only that an LLM can approximate humans who are forced through the paper's own rubric; it does not show that the rubric captures how people naturally judge whether a response honors a query's intent.

  2. self definitional [Definition 3.2 and Abstract]
    "In that case, the actual response follows yh ∼Pθ(· |bC(q)), and the deviation between yh and the ideal response y∼Pθ(· |C(q))is defined asintent hallucination. ... By evaluating various LLMs on FAITHQA, we find that ... the phenomenon stems from omission or misinterpretation of LLMs."

    Definition 3.2 already defines intent hallucination as the model conditioning on a hallucinated constraint set bC(q) that differs from C(q) by replacement or omission. The abstract's 'finding' that intent hallucination stems from omission or misinterpretation is therefore entailed by the definition, not discovered empirically. The FAITHQA data is labeled with this taxonomy, so the benchmark cannot corroborate the taxonomy as an independent observation.

full rationale

The paper's benchmark and model-comparison results (Tables 3, 4, 9-11) are self-contained external evaluations and are not themselves circular. The circularity is concentrated in the metric-validation claim, which is the headline contribution. Section 7 states that human annotators are instructed to calculate CONSTRAINTSCORE from decomposed intent constraints; the automatic metric is then measured against those human scores. Because the human judgment variable is generated by the very rubric whose validity the experiment is meant to establish, the 0.50-versus-4.72 MSE comparison cannot support 'aligns more closely with human judgment' as an independent result. The omission/misinterpretation 'finding' is likewise a restatement of Definition 3.2. The hand-set weights αm=3, αi=2, αo=1 (Appendix G.1) are described as intuitive, and no weight-ablation table is provided, which further weakens the metric-construction story but is secondary to the self-referential validation. A holistic human-judgment condition, elicited without the paper's decomposition prompt, would be needed to break the loop and support the claimed superiority over LLM-as-judge baselines.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central contributions rest on hand-set weights, an assumed complete-coverage decomposition of query intent, and a circular human evaluation rubric. The two invented constructs, intent hallucination and intent constraint, are the paper's own definitions and are not validated against external measurements.

free parameters (3)
  • constraint weights (wm, wi, wo) = 3, 2, 1
    Hand-chosen in Appendix G.1 to reflect importance and counterbalance frequency; no ablation results are reported in the paper.
  • score bands = >=9 strong, 7-8 partial, <=7 major hallucination
    Arbitrary thresholds in Section 4.2 used to interpret CS values; no data-driven justification is provided.
  • difficulty split = Easy <=4 constraints, Hard >4 constraints
    Hand-set cutoff used to define dataset difficulty and to support the 'more constraints, more hallucination' claim; no empirical basis is given.
assumptions (4)
  • domain assumption Every intent of a query can be captured by decomposing it into location, time, subject, action, qualifiers, and quantity constraints, grouped into mandatory, important, and optional sets.
    Invoked in Definition 3.1 and Section 4.1 Step 3; the entire scoring framework depends on this complete-coverage assumption.
  • domain assumption A response that satisfies all decomposed intent constraints preserves the original query's meaning.
    Stated in Definition 3.1: the mapping 'ensures that C(q) captures all intent constraints needed to preserve the original meaning of q'.
  • ad hoc to paper Human annotators computing CONSTRAINTSCORE according to the provided rubric provide a valid ground truth for intent hallucination.
    Section 7 defines the human evaluation rubric as calculating the constraint-based score; using this as ground truth assumes the CS framework is the correct operationalization of intent hallucination.
  • domain assumption GPT-4o reliably extracts and scores intent constraints without query-dependent bias.
    GPT-4o is used for both constraint mapping and satisfaction in Section 4; no separate validation of the extraction step is reported beyond the circular human comparison.
invented entities (2)
  • Intent hallucination
    purpose: A proposed failure mode category for LLM outputs that omit or misinterpret parts of a user query, independent of factual accuracy.
    The concept is defined by the authors in Section 3.2 and its two sub-types are used to construct the benchmark; the human evaluation that would validate it uses the same definitional framework, so there is no external falsifiable handle.
  • Intent constraint
    purpose: The atomic unit of query meaning used to decompose queries and score responses.
    Defined in Definition 3.1 as the output of a mapping function C(q). The claim that these units capture all query meaning is assumed, not independently validated; the human rubric enforces the same decomposition, making the evidence self-referential.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Facts: Evaluating Intent Hallucination in Large Language Models." pith.science (2026). https://pith.science/paper/E55WZLYP

@misc{pith2026250606539,
  author       = {Pith},
  title        = {Pith review of: Beyond Facts: Evaluating Intent Hallucination in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E55WZLYP}},
  note         = {Machine review of arXiv:2506.06539}
}
read the original abstract

When exposed to complex queries containing multiple conditions, today's large language models (LLMs) tend to produce responses that only partially satisfy the query while neglecting certain conditions. We therefore introduce the concept of Intent Hallucination. In this phenomenon, LLMs either omit (neglecting to address certain parts) or misinterpret (responding to invented query parts) elements of the given query, leading to intent hallucinated generation. To systematically evaluate intent hallucination, we introduce FAITHQA, a novel benchmark for intent hallucination that contains 20,068 problems, covering both query-only and retrieval-augmented generation (RAG) setups with varying topics and difficulty. FAITHQA is the first hallucination benchmark that goes beyond factual verification, tailored to identify the fundamental cause of intent hallucination. By evaluating various LLMs on FAITHQA, we find that (1) intent hallucination is a common issue even for state-of-the-art models, and (2) the phenomenon stems from omission or misinterpretation of LLMs. To facilitate future research, we introduce an automatic LLM generation evaluation metric, CONSTRAINT SCORE, for detecting intent hallucination. Human evaluation results demonstrate that CONSTRAINT SCORE is closer to human performance for intent hallucination compared to baselines.

Figures

Figures reproduced from arXiv: 2506.06539 by the authors.

Figure 1
Figure 1. Examples of two types of intent hallucination (omission and misinterpretation). For omission, GPT-4o omits "particularly from Spain", leading to factually accurate yet hallucinated outputs. For misinterpretation, GPT-4o misinterprets the missing article as provided, which leads to hallucinated outputs. ing it into a series of short statements, each repre￾senting a specific requirement the generation must meet; and (… view at source ↗
Figure 2
Figure 2. CONSTRAINT SCORE calculation process. Despite both generations being factually accurate, Generation 1 is not ideal compared to Generation 2, as Generation 1 omits "the explorers must be from Spain". 4 Detecting Intent Hallucination Based on the definition of intent constraints and intent hallucinations, we introduce CONSTRAINT SCORE, a new evaluation metric that detects intent hallucination based on intent constrain… view at source ↗
Figure 3
Figure 3. Deviation distributions from human scores for Baseline (blue) and CONSTRAINT SCORE (red). Distributions are estimated using KDE. CONSTRAINT SCORE is more tightly centered around zero, indicating closer alignment with human evaluation, whereas base￾line shows a broader spread, reflecting higher error. choose Mean Squared Error (MSE) for perfor￾mance evaluation. The MSE for CONSTRAINT SCORE is 0.50, which is significa… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of violated Intent Constraints across evaluation scenarios in FAITHQA. LLMs frequently fail on subjects and actions (blue), especially in open-ended tasks like Creative Writing and Response Evaluation. Errors on fine-grained details like location, time, an…
Figure 5
Figure 5. Figure 5: Human Evaluation Webpage Screenshot. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 7 canonical work pages

  1. [1]

    Amos Azaria and Tom Mitchell. 2023. https://arxiv.org/abs/2304.13734 The internal state of an llm knows when it's lying . Preprint, arXiv:2304.13734

    arXiv 2023

  2. [2]

    Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. https://doi.org/10.18653/v1/P19-1080 Constrained decoding for neural NLG from compositional representations in task-oriented dialogue . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 831--844, Florence, Italy. Assoc...

    work page doi:10.18653/v1/p19-1080 2019

  3. [3]

    Meng Cao, Yue Dong, and Jackie Chi Kit Cheung. 2021. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. arXiv preprint arXiv:2109.09784

    arXiv 2021

  4. [4]

    Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. 2023. https://arxiv.org/abs/2310.00741 Felm: Benchmarking factuality evaluation of large language models . Preprint, arXiv:2310.00741

    arXiv 2023

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

  6. [6]

    Esin Durmus, He He, and Mona Diab. 2020. FEQA : A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Association for Computational Linguistics (ACL)

    work page 2020

  7. [7]

    Bairu Hou, Yang Zhang, Jacob Andreas, and Shiyu Chang. 2024. https://arxiv.org/abs/2406.06950 A probabilistic framework for llm hallucination detection via belief tree propagation . Preprint, arXiv:2406.06950

    arXiv 2024

  8. [8]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. https://arxiv.org/abs/2311.05232 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions . Preprint, arXiv:2311.05232

    arXiv 2023

Show all 33 references
  1. [9]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. https://doi.org/10.1145/3571730 Survey of hallucination in natural language generation . ACM Computing Surveys, 55(12):1–38

    2023 doi

  2. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  3. [11]

    Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation

    2018

  4. [12]

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. https://arxiv.org/abs/2305.11747 Halueval: A large-scale hallucination evaluation benchmark for large language models . Preprint, arXiv:2305.11747

    2023 arXiv

  5. [13]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

    2023 arXiv

  6. [14]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. https://arxiv.org/abs/2303.08896 Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models . Preprint, arXiv:2303.08896

    2023 arXiv

  7. [15]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2305.14251 Factscore: Fine-grained atomic evaluation of factual precision in long form text generation . Preprint, ...

    2023 arXiv

  8. [16]

    the moon is made of marshmallows

    Yifei Ming, Senthil Purushwalkam, Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2025. https://arxiv.org/abs/2410.03727 Faitheval: Can your language model stay faithful to context, even if "the moon is made of marshmallows" . Preprint, arXiv:2410.03727

    2025 arXiv

  9. [17]

    Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. https://arxiv.org/abs/2401.06855 Fine-grained hallucination detection and editing for language models . Preprint, arXiv:2401.06855

    2024 arXiv

  10. [18]

    Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2024. https://arxiv.org/abs/2305.15852 Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation . Preprint, arXiv:2305.15852

    2024 arXiv

  11. [19]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. https://arxiv.org/abs/2401.00396 Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models . Preprint, arXiv:2401.00396

    2024 arXiv

  12. [20]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  13. [21]

    Sameer Pradhan, Wayne Ward, Kadri Hacioglu, James H Martin, and Dan Jurafsky. 2005. Semantic role labeling using different syntactic views. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 581--588

    2005

  14. [22]

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. https://arxiv.org/abs/2401.03601 Infobench: Evaluating instruction following ability in large language models . Preprint, arXiv:2401.03601

    2024 arXiv

  15. [23]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  16. [24]

    Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.157 Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia . In Findings of the Association for Computational Linguist...

    2023 doi

  17. [25]

    Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/3c1e1fdf305195cd620c118aaa9717ad-Paper-Conference.pdf Llm-check: Investigating detection of halluc...

    2024

  18. [26]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . Preprint, arXiv:2203.11171

    2023 arXiv

  19. [27]

    Jinyang Wu, Feihu Che, Chuyuan Zhang, Jianhua Tao, Shuai Zhang, and Pengpeng Shao. 2024. https://arxiv.org/abs/2408.13533 Pandora's box or aladdin's lamp: A comprehensive analysis revealing the role of rag noise in large language models . Preprint, arXiv:2408.13533

    2024 arXiv

  20. [28]

    Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023. https://arxiv.org/abs/2310.06498 A new benchmark and reverse validation method for passage-level hallucination detection . Preprint, arXiv:2310.06498

    2023 arXiv

  21. [29]

    Jiawei Zhang, Chejian Xu, Yu Gai, Freddy Lecue, Dawn Song, and Bo Li. 2024 a . https://arxiv.org/abs/2404.02935 Knowhalu: Hallucination detection via multi-form knowledge based factual checking . Preprint, arXiv:2404.02935

    2024 arXiv

  22. [30]

    Fung, Jing Li, Manling Li, and Heng Ji

    Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R. Fung, Jing Li, Manling Li, and Heng Ji. 2024 b . https://arxiv.org/abs/2407.08039 Knowledge overshadowing causes amalgamated hallucination in large language models . Preprint, arXiv:2407.08039

    2024 arXiv

  23. [31]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. https://arxiv.org/abs/2305.11206 Lima: Less is more for alignment . Preprint, arXiv:2305.11206

    2023 arXiv

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.