Pith. sign in

REVIEW 5 cited by

Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.03558 v1 pith:SEKQ6UAD submitted 2024-03-06 cs.CL

Yuhong Sun , Zhangyue Yin , Qipeng Guo , Jiawen Wu , Xipeng Qiu , Hui Zhao This is my paper · ORCID
classification cs.CL
keywords hallucinationunanswerablelanguagemathproblemwordapproachcalled
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are highly effective in various natural language processing (NLP) tasks. However, they are susceptible to producing unreliable conjectures in ambiguous contexts called hallucination. This paper presents a new method for evaluating LLM hallucination in Question Answering (QA) based on the unanswerable math word problem (MWP). To support this approach, we innovatively develop a dataset called Unanswerable Math Word Problem (UMWP) which comprises 5200 questions across five categories. We developed an evaluation methodology combining text similarity and mathematical expression detection to determine whether LLM considers the question unanswerable. The results of extensive experiments conducted on 31 LLMs, including GPT-3, InstructGPT, LLaMA, and Claude, demonstrate that in-context learning and reinforcement learning with human feedback (RLHF) training significantly enhance the model's ability to avoid hallucination. We show that utilizing MWP is a reliable and effective approach to assess hallucination. Our code and data are available at https://github.com/Yuki-Asuuna/UMWP.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Neural Message-Passing on Attention Graphs for Hallucination Detection

    cs.LG 2025-09 conditional novelty 6.0 of 10

    CHARM trains graph neural networks on token-attention graphs built from LLM computational traces and outperforms prior hallucination detectors on five benchmarks at token and response level.

  2. Investigating Symbolic Triggers of Hallucination in Gemma Models Across HaluEval and TruthfulQA

    cs.CL 2025-09 reject novelty 5.0 of 10

    Symbolic triggers like modifiers and named entities keep Gemma hallucination rates at 64-79% across model scales, suggesting larger models do not eliminate this failure mode.

  3. VisionTrap: Unanswerable Questions On Visual Data

    cs.CV 2025-07 conditional novelty 5.0 of 10

    VisionTrap shows that GPT-4o, GPT-4.1, Gemini Flash 2.5, and LLaVA tend to answer unanswerable visual questions rather than abstain, especially when given multiple-choice options.

  4. Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions

    cs.CL 2025-06 conditional novelty 3.0 of 10

    LLM robustness research is organized into adversarial robustness, out-of-distribution robustness, and evaluation, with an accompanying GitHub collection of papers.

  5. Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models

    cs.CL 2025-06 reject novelty 3.0 of 10

    A survey of LLM hallucination research that formalizes hallucination types and argues, via incompleteness and undecidability arguments, that hallucinations cannot be fully eliminated.

Pith tools