REVIEW 6 cited by
Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Self-detection for Large Language Models (LLMs) seeks to evaluate the trustworthiness of the LLM's output by leveraging its own capabilities, thereby alleviating the issue of output hallucination. However, existing self-detection approaches only retrospectively evaluate answers generated by LLM, typically leading to the over-trust in incorrectly generated answers. To tackle this limitation, we propose a novel self-detection paradigm that considers the comprehensive answer space beyond LLM-generated answers. It thoroughly compares the trustworthiness of multiple candidate answers to mitigate the over-trust in LLM-generated incorrect answers. Building upon this paradigm, we introduce a two-step framework, which firstly instructs LLM to reflect and provide justifications for each candidate answer, and then aggregates the justifications for comprehensive target answer evaluation. This framework can be seamlessly integrated with existing approaches for superior self-detection. Extensive experiments on six datasets spanning three tasks demonstrate the effectiveness of the proposed framework.
Forward citations
Cited by 6 Pith papers
-
SGIC: A Self-Guided Iterative Calibration Framework for RAG
SGIC feeds a model's own uncertainty scores back into its prompt for several calibration rounds and improves RAG accuracy on HotpotQA, NQ, and GSM8K.
-
Reconsidering LLM Uncertainty Estimation Methods in the Wild
Most LLM uncertainty estimates degrade under distribution shift and adversarial prompts, but simple ensembling of scores at test time improves reliability.
-
Large Language Models in the Task of Automatic Validation of Text Classifier Predictions
LLM-based annotators using token-probability thresholds, RAG, and reasoning fine-tuning matched or exceeded human annotator quality on a proprietary 250-class intent-validation task.
-
AGENT-X: Adaptive Guideline-based Expert Network for Threshold-free AI-generated teXt detection
AGENT-X is a zero-shot multi-LLM framework for AI-generated text detection that routes texts to guideline-specific agents and aggregates their calibrated confidences without threshold tuning.
-
Rehabilitation Exercise Quality Assessment and Feedback Generation Using Large Language Models with Prompt Engineering
A pre-trained LLM prompted with exercise-specific skeleton features can classify rehabilitation exercise quality with moderate accuracy and generate textual feedback without fine-tuning.
-
Towards Harmonized Uncertainty Estimation for Large Language Models
CUE combines a supervised correctness classifier with existing LLM uncertainty scores to improve indication, balance, and calibration, reporting AUROC and ECE gains across models and datasets.
Discussion (0). Sign in to comment.