Pith. sign in

REVIEW 3 major objections 5 minor 44 references

CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models produce unsupported claims 4.6 times more often when they infer, evaluate, or explain than when they restate facts; this paper builds the measurement standard, dataset, and detector that reveal the gap.

desk verdict A useful new dataset and framing of cognitive hallucination, but the headline percentages rest on circular auto-labeling evaluation and an unvalidated annotation boundary. read the letter →

arxiv 2505.20767 v4 pith:7KI72FAN submitted 2025-05-27 cs.CL cs.AI

Xiaqiang Tang , Jian Li , Keyu Hu , Du Nan , Xiaolong Li , Xi Zhang , Weigao Sun , Sihong Xie This is my paper
classification cs.CLcs.AI
keywords cognitivehallucinationfaithfulnessknowledge-groundeddialoguelegal-inspiredannotationframeworkdetectionLLMevaluationsentence-levelauto-labeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Faithfulness benchmarks for large language models mostly check whether a model's words match the supplied context—rephrasing is easy to verify. This paper argues that the harder, more consequential statements are 'cognitive' ones: inferences, explanations, and evaluations that go beyond the text. Drawing on how courts weigh circumstantial evidence, it defines three escalating standards of faithfulness (Rational, Grounded, Unequivocal) and builds a human-annotated dataset, CogniBench, that applies them sentence by sentence. The headline result is that LLMs hallucinate cognitive statements at a 64.8% rate versus 13.9% for factual ones—a 4.6x gap that fact-only benchmarks cannot see. The paper then scales the protocol with an LLM-judge auto-labeling pipeline into CogniBench-L and trains an 8B-parameter detector, CogniDet, that reaches 70.3% overall F1 in a single forward pass.

What carries the argument

The load-bearing device is the three-tier sequential decision framework adapted from legal treatment of circumstantial evidence. Annotators (or an LLM judge) classify each sentence by asking, in order: Is it rational (plausible, non-misleading speculation)? Is it grounded (logically supported by the context)? Is it unequivocal (admitting no reasonable alternative interpretation)? The sequence yields four categories—Misleading, Speculative, Reliable, and Unequivocal—and the paper shows that breaking the task into this ordered set of decisions raises inter-annotator agreement to 96.19% while cutting QA effort. A second mechanism is the auto-labeling pipeline: contrastive examples fix the boundary errors the LLM judge makes, sentence-marked formatting keeps annotation cheap, and majority voting over five independent GPT-4 judgments turns a single judge into a stable proxy for human annotation.

What would settle it

Re-annotate a random sample of several hundred sentences from CogniBench with a fresh team of annotators using only the published protocol; if the Speculative/Unequivocal split—and with it the overall 64.8% cognitive hallucination rate—does not reproduce within a few percentage points, the headline gap is an artifact of the annotation scheme. A complementary check: run the trained CogniDet detector on out-of-domain, high-stakes dialogues (for example, medical discharge or financial advisory text) and see whether its cognitive-hallucination F1 collapses below the 73.8% reported on CogniBench.

Watch

Extended reading notes

Core claim

The central claim is that LLM faithfulness cannot be assessed by factual consistency alone. The paper introduces a three-tier, legal-inspired standard for judging cognitive statements—statements that infer, explain, or evaluate beyond the provided context. A statement is Rational if it is plausible speculation, Grounded if it is logically derivable from the context, and Unequivocal if it is the only reasonable conclusion; the tier a statement satisfies determines whether it is Misleading, Speculative, Reliable, or Unequivocal. Using this protocol on knowledge-grounded multi-turn dialogues, the paper reports that 64.8% of cognitive statements are hallucinations (2.6% misleading, 62.2% speculative) compared with 13.9% for factual statements, that cognitive statements grow from about 15% to 50% of model output as conversations lengthen, and that existing hallucination detectors lose roughly 31% F1 on cognitive statements. The paper further claims that an automatic LLM-judge pipeline with contrastive prompting and five-way majority voting matches human annotation accuracy, and that a detector fine-tuned on the resulting 24k-dialogue corpus, CogniDet, achieves 70.3% overall F1 on CogniBench.

Load-bearing premise

The entire measurement rests on the boundary between a Speculative statement (plausible but unsupported) and an Unequivocal statement (the only reasonable conclusion): if different annotators, or the LLM judge, cannot reliably draw that line, then the 64.8% versus 13.9% hallucination gap is an artifact of the labeling scheme rather than a property of the models.

Editorial extensions

If this is right

  • Fact-only benchmarks systematically underestimate hallucination risk: cognitive statements hallucinate at 64.8% versus 13.9% for factual ones, a gap invisible to rephrasing checks.
  • Multi-turn evaluation is necessary: cognitive statements grow from about 15% of model output in early turns to about 50% in later turns, and both factual and cognitive hallucinations concentrate in longer dialogues.
  • Existing detectors degrade by about 31% F1 on cognitive statements, so hallucination detection needs training data that includes cognitive statements; CogniDet reaches 70.3% overall F1 in a single forward pass, versus 58.0% for the strongest prompting baseline tested.
  • The auto-labeling pipeline (contrastive prompting plus five-way majority vote) is claimed to match human annotation accuracy (82.2% overall F1), offering a low-cost proxy to audit newly released models without manual annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the cognitive hallucination gap is real, retrieval-augmented applications that rely on model reasoning—medical triage, financial advice, legal analysis—are much less trustworthy than their fact-rephrasing accuracy suggests, and deployment tests should measure cognitive faithfulness rather than citing accuracy alone.
  • The three-tier standard suggests a testable hierarchy for application design: creative assistants could be held to the Rational tier, helpful assistants to the Grounded tier, and high-stakes decision systems to the Unequivocal tier; one could validate whether model failures concentrate in the tier an application actually requires.
  • A natural extension the paper does not run: apply the auto-labeling pipeline to domain-specific corpora (medical, financial, legal) to test whether the cognitive hallucination gap widens when the context is dense with specialized terminology, where a model's inferences are harder to verify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a legal-inspired, three-tier faithfulness framework (Rational, Grounded, Unequivocal) for sentence-level assessment of 'cognitive statements' (inferences, evaluations, explanations) in knowledge-grounded dialogues. It introduces CogniBench, a human-annotated multi-turn dialogue benchmark built from GPT-4/RefGPT-generated dialogues over Wikipedia contexts, and CogniBench-L, a roughly 24k-dialogue extension produced by an LLM-based auto-labeling pipeline that uses contrastive prompts and five-way majority voting. The authors report three headline findings: cognitive statements become more frequent as dialogues lengthen; LLMs hallucinate 64.8% of cognitive statements versus 13.9% of factual statements; and an 8B detector (CogniDet) fine-tuned on CogniBench-L reaches 70.3% F1, outperforming existing hallucination detectors. The paper also releases the datasets, code/model, and detailed annotation prompts.

Significance. If the annotation boundary is reproducible, the benchmark fills a real gap: existing faithfulness datasets focus on verbatim factual consistency, and a tool for evaluating inferential claims would be useful both for benchmarking and for training low-cost detectors. The public release of the datasets and model, together with the detailed prompts in Appendix A.7, supports reproducibility and follow-up work. The reported 4.6x gap between cognitive and factual hallucination rates is a striking, falsifiable empirical claim. However, the central claims rest on two premises that are not yet established: the reliability of the Speculative/Unequivocal boundary, and the validity of the auto-labeling evaluation. As a resource, the paper is still valuable, but the headline statistics need to be made robust to these premises.

major comments (3)
  1. [Section 3.1, Table 3, Section 4.6] The auto-labeling prompts are refined using human annotations from CogniBench itself: Section 3.1 describes an 'Initial Prompting and Diagnosis' step in which LLM annotation errors are identified on CogniBench data and used to construct contrastive examples, and Table 3 then reports an 82.2% F1 for auto-labeling on the same CogniBench set. This is an in-sample estimate, and the claim that auto-labeling 'closely matches human annotation' is not supported by this evaluation. Please construct the contrastive examples on a development split and evaluate on a held-out split, or clearly label the 82.2% figure as a development-set estimate and validate it on independently annotated data. The concern propagates to Section 4.7, where auto-labeling is used as a proxy for human annotation to compare GPT-4, Gemini-Pro, and other models.
  2. [Section 2.2, Table 1, Figure 10, Section 4.4] The reproducibility of the three-tier labels is not established. Table 1 reports a 96.19% 'IAA' for the sequential decision framework, but no definition of the metric, no confidence interval, no chance-corrected statistic such as Cohen's kappa, and no description of the sample beyond the small QA-instance counts. Figure 10 shows two causal inferences that receive different labels: 'This policy helps prevent underage customers from accessing their products' is labeled Unequivocal, while 'Their success and reputation attract top talent...' is labeled Speculative, with annotation reasons that do not provide an operational rule separating the two cases. Since Section 4.4 counts Speculative statements as hallucinations, the 64.8% cognitive hallucination rate and the resulting 4.6x ratio are directly sensitive to this boundary. Please report per-category pairwise agreement, chance-corrected agreement, and an analysis of how the headline rates change under alternative boundary decisions.
  3. [Section 4.4, Figure 6] The headline 'hallucination rate' conflates categories with different epistemic status. Speculative statements are defined as plausible but ungrounded inferences (satisfying only Criterion 1), and labeling all such statements 'hallucinations' is a design choice rather than a measured fact. The abstract and conclusion present the 64.8% figure as an LLM deficiency, but it depends on counting rationally plausible inferences as hallucinations. Please report the four categories (Misleading, Speculative, Reliable, Unequivocal) separately, explicitly state that the 64.8% statistic uses the 'Rational but not Grounded' definition, and include a sensitivity analysis (e.g., counting only Misleading statements as hallucinations) so readers can separate the empirical finding from the definitional choice.
minor comments (5)
  1. [Section 4.1] The heading 'Matrices' should be 'Metrics'; also, the precision/recall formulas count words while the evaluation is described as sentence-level, so the unit of analysis should be clarified in the text.
  2. [Figure 10, Section A.5.4] Figure 10 contains a spelling error ('Anbreakthroughcame') and the figure is dense; please clean the text and consider enlarging the examples or splitting the figure for readability.
  3. [Table 5, Section A.9] The threshold values 2 and 3 used for majority voting are only defined in Appendix A.9; please define them when Table 5 is first cited in Section 4.6.
  4. [Abstract, Table 2] The text says CogniBench-L is '100 times larger' than CogniBench, but 24084/264 is approximately 91; please state the exact factor or write 'about two orders of magnitude.'
  5. [Figure 8, Section 4.7] Given the in-sample nature of the auto-labeling evaluation noted in Major Comment 1, the model-comparison numbers in Figure 8 should be accompanied by a caveat or by a validation of the auto-labeling proxy.

Circularity Check

2 steps flagged · score 6.0 of 10

Auto-labeling is evaluated on the same human annotations used to design its prompts, so the reported 82.2% F1 and the downstream CogniDet gains are partially forced by construction.

  1. fitted input called prediction [Section 3.1 (Contrastive and Formative Prompting); Section 4.6, Table 3; Appendix A.8.1]
    "We first prompt LLM with annotation examples from human annotators and ask the LLM to annotate statements accordingly. We then identify and collect common annotation errors. ... Auto-Labeling: We employ the auto-labeling pipeline Section 3 on human-labeled CogniBench data to assess the reliability of this method."

    The auto-labeling prompt is constructed by diagnosing errors on human-annotated examples and adding contrastive examples to fix exactly the boundary cases in that set. The pipeline's reliability is then measured by running it on the same human-labeled CogniBench data (Table 3 reports 82.20 overall F1; Table 5 ablations are on the same data) and presenting the result as a match to human annotators. Because the prompt was iteratively tuned on this very test set, the reported F1 measures prompt fitting rather than held-out annotation ability, so the 'reliable proxy' claim is forced by construction.

  2. fitted input called prediction [Section 3.1; Appendix A.8.1; Section 4.6, Table 3]
    "CogniDet: We fine-tune the Llama3 8B instruct model (Grattafiori et al., 2024) using the CogniBench-L dataset, which is generated via the auto-labeling pipeline outlined in Section 3."

    CogniBench-L is produced by the Section 3 auto-labeling prompts, and Section 3.1 states those prompts were built from human-annotator examples and refined by collecting errors on those examples. The same human-annotated CogniBench set is then used as the test set for CogniDet. No held-out split is reported, so the detector's 70.3% F1 and state-of-the-art claim are not independent of the labels that shaped its training data; the result is partly an artifact of the test-set-conditioned prompt design.

full rationale

The clearest circular step is the evaluation protocol for the auto-labeling pipeline. Section 3.1 tunes the prompt on human annotations from CogniBench, and Sections 4.6/A.8.1 evaluate that same pipeline on human-labeled CogniBench, so the claimed 'matches human annotators' accuracy' is not a held-out result. CogniDet inherits this issue because its training corpus is generated by that tuned prompt while its test set is the human annotations that informed the prompt. The 4.6x cognitive-vs-factual hallucination ratio is a separate matter: it is a definitional measurement under the paper's taxonomy (speculative plus misleading counted as hallucination), not a derivation that reduces to its own input, though its stability depends on the unverified reproducibility of the Rational/Grounded boundary. I do not count that as circularity. There is no load-bearing self-citation chain. Overall score 6: one empirical claim (auto-labeling reliability) and the related detector-ranking claim reduce by construction, while the benchmark's central descriptive statistics retain independent human-annotation content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claims rest on the annotation framework, which is defined by the authors and operationalized through vague legal metaphors. The auto-labeling pipeline introduces a tuned prompt configuration on the benchmark itself. No numeric model parameters are fitted to data in the traditional sense, but the prompt design and threshold choices act as free parameters that affect the reported performance.

free parameters (3)
  • Contrastive examples in auto-labeling prompts = Unknown; selected by iterative error diagnosis against the human labels
    Section 3.1 describes an iterative loop that uses human annotations from CogniBench to construct positive and negative prompt examples. This is a free parameter tuned on the evaluation set.
  • Majority-vote threshold = 2 or 3 (both reported)
    Section A.9 states that the threshold balances precision and recall and was chosen flexibly; it is a manual decision boundary.
  • Number of repeated samples = 5
    Section 3.2 fixes five repeated LLM prompts per instance without reporting a variance analysis that would justify this choice.
assumptions (4)
  • domain assumption Sentence-level classification into factual vs. cognitive statements is coherent and matches a real distinction in LLM outputs.
    The entire annotation protocol depends on this binary split (Section 2.1, Step 2). If the categories are not stable, all downstream labels are unstable.
  • ad hoc to paper A cognitive statement that is Rational but not Grounded is considered a hallucination for the benchmark's main statistics.
    Section 4.4 counts Speculative statements as cognitive hallucinations even though Section 2.2 says the Rational standard is sufficient for entertainment applications. This enforces one application standard on all data.
  • ad hoc to paper The three-tier legal-inspired criteria can be applied consistently by human annotators.
    The criteria rely on vague phrases like 'logically and reasonably drawn' and 'no alternative interpretations could reasonably exist' (Section 2.2). Figure 10 shows that near-identical inferences receive different labels.
  • domain assumption GPT-4-generated dialogues are representative of real-world LLM usage.
    All dialogues are generated by GPT-4 via RefGPT (Sections 2.1 and A.5.1). The empirical findings may reflect GPT-4's dialogue style rather than LLMs in general.
invented entities (2)
  • Cognitive statements as a formal annotation category
    purpose: To label sentences that extend beyond verbatim context, including inference, explanation, evaluation, and opinion (Section 2.1).
    The category is introduced by the authors and defined through examples. There is no external, independently validated standard for what counts as cognitive.
  • Three-tier faithfulness criteria (Rational, Grounded, Unequivocal)
    purpose: To assess cognitive statements with increasing rigor, inspired by legal standards for circumstantial evidence (Section 2.2).
    These are new constructs proposed by the paper. Their validity is asserted through a legal analogy, not established by independent evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models." pith.science (2026). https://pith.science/paper/7KI72FAN

@misc{pith2026250520767,
  author       = {Pith},
  title        = {Pith review of: CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KI72FAN}},
  note         = {Machine review of arXiv:2505.20767}
}
read the original abstract

Faithfulness hallucinations are claims generated by a Large Language Model (LLM) not supported by contexts provided to the LLM. Lacking assessment standards, existing benchmarks focus on "factual statements" that rephrase source materials while overlooking "cognitive statements" that involve making inferences from the given context. Consequently, evaluating and detecting the hallucination of cognitive statements remains challenging. Inspired by how evidence is assessed in the legal domain, we design a rigorous framework to assess different levels of faithfulness of cognitive statements and introduce the CogniBench dataset where we reveal insightful statistics. To keep pace with rapidly evolving LLMs, we further develop an automatic annotation pipeline that scales easily across different models. This results in a large-scale CogniBench-L dataset, which facilitates training accurate detectors for both factual and cognitive hallucinations. We release our model and datasets at: https://github.com/FUTUREEEEEE/CogniBench

Figures

Figures reproduced from arXiv: 2505.20767 by the authors.

Figure 1
Figure 1. Difference between “factual statement” and “cogni [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Dialogues in CogniBench on average contains three [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. (a)(b): Existing faithfulness assessment standards such as “Baseless” ( [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Auto-Labeling pipeline for hallucination detection. (a): We generate multi-turn, knowledge-grounded conversations [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Average number of factual statements decreases as [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Left: Distribution of Factual statements. Right: Distribution of Cognitive statements. LLMs are more likely to generate cognitive hallucinations (i.e., misleading and speculative statements) when producing cognitive state￾ments. statements in CogniBench according to th…
Figure 7
Figure 7. Figure 7: Top: Hallucinated statements increase with more conversation turns. Bottom: Distribution of hallucinated statements across dialogues. Cognitive hallucinations (speculative or misleading statements) tend to occur at the beginning or end of a dialogue, while factual hall…
Figure 8
Figure 8. Figure 8: Evaluation of factual and cognitive statement por [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Drawing parallels between legal domain evidence [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Examples of cognitive statements categorized according to the standards outlined in Section [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: CogniBench employs a uniform sampling across [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: CogniDet’s detection accuracy (F1) showing log [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Example of hallucination detection output from CogniDet, trained on CogniBench-L [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Example dialoge from CogniBench with RAGTruth ( [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 12 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Masha Belyi, Robert Friel, Shuai Shao, and Atindriyo Sanyal. 2024. Luna: An evaluation foundation model to catch language model hallucinations with high accuracy and low cost. arXiv preprint arXiv:2406.00975

    arXiv 2024

  3. [3]

    O'Reilly Media, Inc

    Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc."

    2009

  4. [4]

    Benjamin S Bloom et al. 1956. Taxonomy of. Educational Objectives

    work page 1956

  5. [5]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  6. [6]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  7. [7]

    Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. 2024. Large language model agent in financial trading: A survey. arXiv preprint arXiv:2408.06361

    arXiv 2024

  8. [8]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    arXiv 2022

Show all 44 references
  1. [9]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages...

    2024

  2. [10]

    Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Improved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2022

  3. [11]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--630

    2024

  4. [12]

    Bryan A Garner et al. 2004. Black's law dictionary

    2004

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  6. [14]

    Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. arXiv preprint arXiv:2106.13822

    2021 arXiv

  7. [15]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Informatio...

  8. [16]

    Kristiina Jokinen. 2024. The need for grounding in llm-based dialogue systems. In Proceedings of the Workshop: Bridging Neurons and Symbols for Natural Language Processing and Knowledge Graphs Reasoning (NeusymBridge)@ LREC-COLING-2024, pages 45--52

    2024

  9. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  10. [18]

    Wolters Kluwer Law et al. 2008. Criminal law. Wolters Kluwer

    2008

  11. [19]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  12. [20]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12

    2024

  13. [21]

    Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. 2024. Halludial: A large-scale benchmark for automatic dialogue-level hallucination evaluation. arXiv preprint arXiv:2406.07070

    2024 arXiv

  14. [22]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004--9017

    2023

  15. [23]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906--1919

    2020

  16. [24]

    Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. Fine-grained hallucination detection and editing for language models. arXiv preprint arXiv:2401.06855

    2024 arXiv

  17. [25]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332

    arXiv

  18. [26]

    New York State Unified Court System . n.d. https://www.nycourts.gov/judges/evidence/4-RELEVANCE/4-relevance.shtml Guide to new york evidence . Accessed: 2025-02-01

    2025

  19. [27]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2023. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. arXiv preprint arXiv:2401.00396

    2023 arXiv

  20. [28]

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. Ares: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2024

  21. [29]

    Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021. Questeval: Summarization asks for fact-based evaluation. arXiv preprint arXiv:2103.12693

    2021 arXiv

  22. [30]

    Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368

    2017 arXiv

  23. [31]

    Damien Sileo. 2024. https://aclanthology.org/2024.lrec-main.1361 tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu...

    2024

  24. [32]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1--8

    2025

  25. [33]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  26. [34]

    R v Villaroman. 2016. https://www.canlii.org/en/ R v villaroman . 2016 SCC 33 (CanLII), [2016] 1 SCR 1000, per Cromwell J, at para 55. Supreme Court of Canada

    2016

  27. [35]

    Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2024

  28. [36]

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526--1541

    2023

  29. [37]

    Wikimedia Foundation . 2025. English wikipedia dump. https://dumps.wikimedia.org/enwiki/. Accessed: 2025-05-27

    2025

  30. [38]

    Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2023. Are large language models really good logical reasoners? a comprehensive evaluation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.09841

    2023 arXiv

  31. [39]

    Dongjie Yang, Ruifeng Yuan, Yuantao Fan, Yifei Yang, Zili Wang, Shusen Wang, and Hai Zhao. 2023. Refgpt: Dialogue generation of gpt, by gpt, and for gpt. arXiv preprint arXiv:2305.14994

    2023 arXiv

  32. [40]

    Shunyu Yao, Qingqing Ke, Qiwei Wang, Kangtong Li, and Jie Hu. 2024. Lawyer gpt: A legal large language model with enhanced domain knowledge and reasoning capabilities. In Proceedings of the 2024 3rd International Symposium on Robotics, Artificial Intelligence and Information E...

    2024

  33. [41]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  34. [42]

    Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, et al. 2023. Characterglm: Customizing chinese conversational ai characters with large language models. arXiv preprint arXiv:2311.16832

    2023 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.