Pith. sign in

REVIEW 4 cited by

A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2104.08704 v2 pith:YGUKBHHU submitted 2021-04-18 cs.CL cs.AI

Tianyu Liu , Yizhe Zhang , Chris Brockett , Yi Mao , Zhifang Sui , Weizhu Chen , Bill Dolan This is my paper · ORCID
classification cs.CLcs.AI
keywords detectiondatasethallucinationtextapplicationscontentcreatefirst
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentence- or document-level detection may fail to provide the fine-grained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes (HAllucination DEtection dataSet). To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Neural Message-Passing on Attention Graphs for Hallucination Detection

    cs.LG 2025-09 conditional novelty 6.0 of 10

    CHARM trains graph neural networks on token-attention graphs built from LLM computational traces and outperforms prior hallucination detectors on five benchmarks at token and response level.

  2. Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Using hallucinated benchmark answers as rejected DPO pairs, ordered by an external fact-checker's grounding score, improves hallucination detection in 1B-3B Llama models.

  3. Retrieval Augmented Generation-based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps

    cs.CL 2025-05 reject novelty 3.0 of 10

    A RAG system over U.S. transportation cybersecurity statutes scores higher than vanilla chatbots on the authors' 59-question benchmark, but the benchmark gives the RAG system the source documents and withholds them fr...

  4. Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences

    cs.AI 2025-02 conditional novelty 3.0 of 10

    A guardrail pipeline combining detection, retrieval grounding, rule-based wrappers, and a repair model is reported to match OpenAI moderation and fix 80.7 percent of hallucinated HaluEval answers.

Pith tools