REVIEW 6 cited by
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.
Forward citations
Cited by 6 Pith papers
-
Reconsidering LLM Uncertainty Estimation Methods in the Wild
Most LLM uncertainty estimates degrade under distribution shift and adversarial prompts, but simple ensembling of scores at test time improves reliability.
-
Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
Tool-augmented LLM annotators improve agreement with ground-truth preferences on long-form factual and coding tasks, with mixed results on math, compared to standard LLM-as-a-judge baselines.
-
LRCTI: A Large Language Model-Based Framework for Multi-Step Evidence Retrieval and Reasoning in Cyber Threat Intelligence Credibility Verification
LRCTI uses an LLM to summarize threat reports, retrieve evidence in several rounds, and judge each claim credible or incredible, reporting strong F1 gains on CTI-200 and PolitiFact.
-
Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs
A small evaluation of Llama 3.1 shows factuality in school-level question answering degrades with decreasing language speaker count, though the statistical support is weakened by methodological issues.
-
HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMs
Modeling the full token-by-token trajectory of LLM hidden states with neural ODEs, CDEs, and SDEs improves hallucination detection by over 14% AUC on a constructed true/false benchmark, though gains shrink on QA datasets.
-
The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.
Discussion (0). Continue with ORCID to comment.