REVIEW 4 major objections 6 minor 1 cited by
Improving the Reliability of LLMs: Combining CoT, RAG, Self-Consistency, and Self-Verification
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Self-verification is the most reliable fix for LLM hallucination, the paper claims.
desk verdict Self-verification's reported advantage is an artifact of putting the ground-truth answer in the verification prompt. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the verification step in the self-verification pipeline: a second prompt receives the original query, the model's own generated answer, the ground-truth answer from the dataset, and the top five retrieved document chunks, and the model must decide whether the answer is hallucinated or correct. That step outperforms the paper's other interventions because it combines external evidence with an explicit judgement. The other load-bearing component is the self-consistency algorithm, which samples nine responses at varied temperatures, scores each against the reference answer with cosine similarity, applies a 0.5 threshold, and takes a majority vote on whether the output is hallucinated. The RAG pipeline, which embeds document chunks, retrieves the five most similar to the query, and feeds them to the model, supplies the external grounding for both retrieval-based and verification-based methods.
What would settle it
Run the same self-verification setup without the ground-truth answer in the prompt, keeping the query, generated answer, and retrieved documents. If FEVER accuracy and TruthfulQA MC2 drop to the level of RAG-plus-CoT or lower, the claimed advantage of self-verification depends on answer leakage rather than on independent checking.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that self-verification—a second pass in which the model receives its first answer, the ground-truth answer, and retrieved supporting documents and is asked to judge whether the answer is factually correct—is the most effective hallucination-mitigation strategy across the tested settings. The claim is comparative: self-verification outperforms baseline generation, chain-of-thought prompting, retrieval-augmented generation, their combination, and self-consistency voting, with LLaMA-2-7b slightly ahead of GPT-3.5-Turbo. On HaluEval, the best result is a RAG-plus-CoT setup reaching an 11 percent hallucination rate; on FEVER and TruthfulQA, self-verification tops the leaderboard. The paper presents these results as evidence that external grounding plus reflective checking, rather than reasoning alone, is what makes LLM outputs trustworthy.
Load-bearing premise
The load-bearing premise is that giving the model the correct answer inside the verification prompt measures its ability to verify; if the model is merely echoing that answer, the method's best performance is an experimental artifact.
Editorial extensions
If this is right
- Teams deploying LLMs for fact-oriented tasks should add a self-verification pass with retrieved evidence rather than relying on chain-of-thought prompting alone.
- Retrieval plus step-by-step reasoning is a close second and gives the lowest hallucination rate on HaluEval, so external grounding matters even when verification is not used.
- All five strategies beat the unmodified model, meaning even lightweight prompting is better than no mitigation at all.
- Self-verification is not model-specific: the pattern holds across GPT-3.5-Turbo, LLaMA-2-7b, and DeepSeek-R1, with LLaMA-2-7b slightly ahead.
- Because self-consistency requires nine samples per query and self-verification needs retrieval plus a second prompt, the accuracy gains come with added inference cost.
Reading between the lines
- A stricter test would remove the ground-truth answer from the verification prompt; if accuracy collapses, the reported edge of self-verification is leakage rather than checking.
- The comparison likely understates what a separate, stronger verifier model could do, since the paper uses the same LLM for generation and verification.
- The dependence on a hand-tuned cosine-similarity threshold of 0.5 and on top-5 retrieval suggests results could shift with different embeddings or retrieval granularity.
- A domain outside Wikipedia-style fact checking, such as medical or legal reasoning, would test whether verification still holds when retrieved documents are less authoritative.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper empirically compares hallucination-mitigation strategies for LLMs: baseline, chain-of-thought (CoT), retrieval-augmented generation (RAG), CoT+RAG, self-consistency, and self-verification. It evaluates GPT-3.5-Turbo, LLaMA-2-7b, and DeepSeek-R1 on 500 samples each from HaluEval, FEVER, and TruthfulQA, using hallucination rate, label accuracy, and a similarity-based MC2 proxy. The paper reports that self-verification achieves the best overall performance (e.g., roughly 90% accuracy on FEVER and 80% MC2 on TruthfulQA) and concludes in Section 6.2 that self-verification is the most robust approach. The experimental protocols are described in Sections 4.4-4.7, evaluation settings in Section 5, and results in Section 6.
Significance. If the comparison were valid, the paper would provide practically useful evidence about which mitigation stack to adopt, and the multi-model, multi-dataset setup is a reasonable starting point. The manuscript clearly describes its pipeline and identifies several engineering choices (e.g., retrieval counts, temperature, token limits) that matter in practice. However, the central empirical claim is currently not supported: the self-verification protocol gives the model the ground-truth answer, so the reported advantage of self-verification is an artifact of answer leakage rather than a measured property of the method. In addition, the lack of held-out tuning, small sample sizes, and absence of statistical uncertainty make the ranking unreliable. As it stands, the paper does not establish which method is most effective for reducing hallucination.
major comments (4)
- [Section 4.7, Figure 4] The verification prompt for self-verification explicitly includes the ground-truth answer from the dataset, along with the query, the model-generated answer, and retrieved documents. A model given the gold answer can label a response as correct simply by detecting that it matches the provided gold answer, so the FEVER accuracy around 90% and TruthfulQA MC2 around 80% in Section 6.1 do not measure an independent self-verification capability. The conclusion in Section 6.2 that 'Self-Verification had the best performance' is therefore an artifact of answer leakage rather than a property of the method. The authors should rerun the verification experiments with the gold answer removed from the verification prompt (e.g., verifying only against retrieved evidence) and report those numbers as the self-verification results.
- [Algorithm 1, Section 4.6; Section 5.1] The self-consistency decision in Algorithm 1 uses the ground-truth answer y in the cosine-similarity comparison, and the threshold tau=0.5 is said to be chosen 'through trial and error.' Section 5.1 also describes tuning the CoT prompt, the number of retrieved documents (2, 5, and 10), the temperature, and the max-token limit without specifying any train/validation split. If these choices were made on the same 500-sample evaluation sets used for the final numbers, the reported improvements in Section 6 may be optimistically biased and the method ranking may not generalize. Please state which hyperparameters were selected on held-out data and re-evaluate the final configurations on unseen test samples.
- [Section 4.2; Section 6.1] Only 500 samples per dataset are evaluated, and no error bars, confidence intervals, or statistical significance tests are reported. Because LLM outputs are stochastic, as the temperature sampling in Section 4.6 acknowledges, the small differences highlighted in Section 6.2 (e.g., 'LLaMA-2 slightly outperforming GPT-3.5-Turbo') may be sampling noise. The authors should report variability across independent runs or bootstrap confidence intervals, and run paired statistical tests for the central comparisons.
- [Section 5.3.3] The 'MC2' metric used for TruthfulQA is not the standard TruthfulQA MC2; it assigns each response a label by taking the highest cosine similarity to the sets of correct and incorrect reference answers and then computes accuracy. The claimed 'MC2 score of around 80%' in Section 6.1 should either be renamed (e.g., 'similarity-based truthfulness accuracy') or be computed with the official TruthfulQA evaluation script; otherwise the number is not comparable to MC2 results in the literature.
minor comments (6)
- [Title] The title as submitted ('Improving the Reliability of LLMs: Combining Chain-of-Thought Reasoning and Retrieval-Augmented Generation') does not match the arXiv title that includes self-consistency and self-verification; please make the titles consistent.
- [Section 4.3] The phrase 'for all dataset' should be 'for all datasets.'
- [Algorithm 1] The majority-vote rule does not specify a tie-breaking behavior when count_factual equals count_hallucinated.
- [Figure 5 caption] Figure 5 refers to 'our website' without providing a URL; for reproducibility, please include a publicly accessible repository or at least a stable URL.
- [References] Several references have incomplete author lists (e.g., Kojima et al., Wei et al., Liu et al., Singh and Kapoor, and Zhou et al.); please provide full author information or use standard bibliography formatting.
- [Section 4.2] The paper states that 500 samples per dataset were evaluated but does not describe the sampling procedure or whether a fixed random seed was used; please state how the subset was drawn.
Circularity Check
Self-verification's reported advantage is not identifiable because the verification prompt includes the ground-truth answer, allowing label leakage.
-
self definitional
[Section 4.7, Self Verification (Figure 4); claimed in Section 6.2]
"This second prompt is given the original query, the model's generated answer, the ground truth answer, and some supporting documents retrieved from a vector database similar to how it was done for RAG. The model used for generation is then asked to decide whether the generated answer is factually correct or hallucinated, based on the given evidence."
The verifier receives the gold answer as part of its input, so the 'self-verification' decision can be made by matching the generated answer to the provided correct answer. The reported FEVER accuracy ~90%, TruthfulQA MC2 ~80%, and the Section 6.2 conclusion that 'Self-Verification had the best performance' therefore measure a label-leaking classification task, not an independent verification capability. The best-method result is forced by the protocol rather than by any property of self-verification.
-
other
[Section 4.6, paragraph after Algorithm 1]
"We came up with a threshold tau = 0.5 through trial and error, if S(y_i_hat, y) > tau, the output is considered factually consistent; otherwise, it is classified as hallucinated."
The factual-versus-hallucinated label for each sampled answer is assigned by cosine similarity to ground truth y, and the threshold is tuned on the data. The reported HaluEval hallucination rate for self-consistency is thus a ground-truth-supervised classification of samples, not a measure of consistency among the model's own generations. Giving the method access to the answer key at inference time makes its apparent improvement over baselines an artifact of construction.
full rationale
The central claim of the paper is the Section 6.2 ranking that 'Self-Verification had the best performance.' That ranking is not identifiable from the reported experiments because Section 4.7 states that the verification prompt contains 'the ground truth answer' alongside the query, the model's generated answer, and retrieved documents. A verifier with the gold answer in its input can label the generated answer correct simply by matching the provided answer, so the FEVER and TruthfulQA scores for self-verification are confounded with label leakage. The same structural leak appears in Section 4.6, where the self-consistency criterion labels each response as factually consistent or hallucinated by thresholding cosine similarity to the ground truth. The resulting HaluEval rates are therefore not independent measurements of the mitigation strategy. No load-bearing self-citation chain is present; the circularity is in the experimental definitions themselves. Although self-verification may in principle be a useful technique, the evidence presented cannot distinguish its intrinsic effect from the effect of supplying the answer key, so the paper's headline comparison is forced by construction. Score 8 reflects that the best-method conclusion reduces to the leak in the protocol, not that every number in the paper is vacuous.
Assumptions & free parameters
free parameters (5)
- Similarity threshold tau =
0.5
- Number of retrieved documents k =
5
- Sampling temperature =
0.4
- Max token limit =
150
- Number of self-consistency samples n =
9
assumptions (5)
- domain assumption Cosine similarity between sentence embeddings of the generated answer and the reference answer is a valid measure of factual consistency.
- domain assumption Self-verification by the same model, given ground truth and retrieved documents, faithfully reflects the model's ability to detect its own hallucinations.
- domain assumption The ground truth labels in HaluEval, FEVER, and TruthfulQA are correct and the evaluation metrics (hallucination rate, label accuracy, MC2 via cosine similarity) capture the intended quality.
- domain assumption Wikipedia retrieval provides adequate grounding for TruthfulQA questions, even though the dataset's source URLs span many websites.
- domain assumption The 500-sample subset is representative of each full dataset.
Cite this review
Pith. "Pith review of Improving the Reliability of LLMs: Combining CoT, RAG, Self-Consistency, and Self-Verification." pith.science (2026). https://pith.science/paper/CLX7I4MY
@misc{pith2026250509031,
author = {Pith},
title = {Pith review of: Improving the Reliability of LLMs: Combining CoT, RAG, Self-Consistency, and Self-Verification},
year = {2026},
howpublished = {\url{https://pith.science/paper/CLX7I4MY}},
note = {Machine review of arXiv:2505.09031}
}
read the original abstract
Hallucination, where large language models (LLMs) generate confident but incorrect or irrelevant information, remains a key limitation in their application to complex, open-ended tasks. Chain-of-thought (CoT) prompting has emerged as a promising method for improving multistep reasoning by guiding models through intermediate steps. However, CoT alone does not fully address the hallucination problem. In this work, we investigate how combining CoT with retrieval-augmented generation (RAG), as well as applying self-consistency and self-verification strategies, can reduce hallucinations and improve factual accuracy. By incorporating external knowledge sources during reasoning and enabling models to verify or revise their own outputs, we aim to generate more accurate and coherent responses. We present a comparative evaluation of baseline LLMs against CoT, CoT+RAG, self-consistency, and self-verification techniques. Our results highlight the effectiveness of each method and identify the most robust approach for minimizing hallucinations while preserving fluency and reasoning depth.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
RoT: Enhancing Table Reasoning with Iterative Row-Wise Traversals
A training-free row-wise traversal prompt improves table question answering accuracy over long-CoT reasoning models on WikiTableQuestions and TableBench.
Reference graph
Works this paper leans on
-
[2]
Retrieval-augmented gen- eration for large language models: A survey.arXiv preprint arXiv:2312.10997. T. Kojima, Y . Tsuchiya, and A. Ogawa
-
[7]
Mitigating hal- lucination in retrieval-augmented chain-of-thought reasoning.arXiv preprint, arXiv:2303.08896. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian
-
[8]
arXiv preprint arXiv:2307.06435
A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435. S. Singh and A. Kapoor
-
[9]
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal
Hallucinations in open-domain question answering: Solutions through chain-of-thought reasoning.arXiv preprint, arXiv:2407.07071. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal
-
[11]
Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171. J. Wei, P. Wang, D. Schuurmans, M. Bosma, and D. Chen
-
[12]
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao
Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint, arXiv:2201.11903. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao
-
[13]
Large language models are better reasoners with self-verification.arXiv preprint arXiv:2212.09561. W. Zhou, H. Wang, and Z. Yu
-
[14]
Reduc- ing hallucinations in retrieval-augmented generation with chain-of-thought prompting.arXiv preprint, arXiv:2305.13534
Show all 13 references
-
[2018]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou
Fever: a large-scale dataset for fact extraction and verification.arXiv preprint arXiv:1803.05355. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou
-
[2021]
Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958. Y . Liu, W. Chen, and J. Gao
-
[2023]
Stephanie Lin, Jacob Hilton, and Owain Evans
Halueval: A large- scale hallucination evaluation benchmark for large language models.arXiv preprint arXiv:2305.11747. Stephanie Lin, Jacob Hilton, and Owain Evans
-
[2024]
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang
Factchd: Benchmarking fact-conflicting hallucina- tion detection.arXiv preprint arXiv:2310.12086. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang
-
[2025]
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen
Cot-rag: Integrating chain of thought and retrieval-augmented generation to en- hance reasoning in large language models.arXiv preprint arXiv:2504.13534. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.