REVIEW 5 cited by
Can AI Assistants Know What They Don't Know?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, AI assistants based on large language models (LLMs) show surprising performance in many tasks, such as dialogue, solving math problems, writing code, and using tools. Although LLMs possess intensive world knowledge, they still make factual errors when facing some knowledge intensive tasks, like open-domain question answering. These untruthful responses from the AI assistant may cause significant risks in practical applications. We believe that an AI assistant's refusal to answer questions it does not know is a crucial method for reducing hallucinations and making the assistant truthful. Therefore, in this paper, we ask the question "Can AI assistants know what they don't know and express them through natural language?" To answer this question, we construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets. Then we align the assistant with its corresponding Idk dataset and observe whether it can refuse to answer its unknown questions after alignment. Experimental results show that after alignment with Idk datasets, the assistant can refuse to answer most its unknown questions. For questions they attempt to answer, the accuracy is significantly higher than before the alignment.
Forward citations
Cited by 5 Pith papers
-
Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning
A reinforcement-learning method that rewards refusal over hallucination cuts LLM 'futile reasoning' from ~66-79% to 1-7% on Countdown while roughly preserving accuracy.
-
UCPO: Uncertainty-Aware Policy Optimization
UCPO uses decoupled advantage normalization and a dynamic uncertainty reward to train LLMs to express uncertainty, improving answer reliability (PAQ) but sometimes lowering F1.
-
UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions
A new bilingual benchmark ties unanswerable questions to Wikidata facts and shows that LLMs often store the relevant knowledge yet fail to use it to recognize unanswerability.
-
A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.
-
Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
A survey of LLM hallucination research that formalizes hallucination types and argues, via incompleteness and undecidability arguments, that hallucinations cannot be fully eliminated.
Discussion (0). Continue with ORCID to comment.