Pith. sign in

REVIEW 9 cited by

When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.01297 v3 pith:2GKZM2IK submitted 2024-06-03 cs.CL

Ryo Kamoi , Yusen Zhang , Nan Zhang , Jiawei Han , Rui Zhang This is my paper · ORCID
classification cs.CL
keywords self-correctionllmsfeedbackresearchpriorquestionsstudiessurvey
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Self-correction is an approach to improving responses from large language models (LLMs) by refining the responses using LLMs during inference. Prior work has proposed various self-correction frameworks using different sources of feedback, including self-evaluation and external feedback. However, there is still no consensus on the question of when LLMs can correct their own mistakes, as recent studies also report negative results. In this work, we critically survey broad papers and discuss the conditions required for successful self-correction. We first find that prior studies often do not define their research questions in detail and involve impractical frameworks or unfair evaluations that over-evaluate self-correction. To tackle these issues, we categorize research questions in self-correction research and provide a checklist for designing appropriate experiments. Our critical survey based on the newly categorized research questions shows that (1) no prior work demonstrates successful self-correction with feedback from prompted LLMs, except for studies in tasks that are exceptionally suited for self-correction, (2) self-correction works well in tasks that can use reliable external feedback, and (3) large-scale fine-tuning enables self-correction.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification

    cs.AR 2026-03 unverdicted novelty 7.0 of 10

    FVRuleLearner retrieves learned operator-level reasoning rules to boost the functional correctness of LLM-generated SystemVerilog assertions by roughly 30 percentage points over simple prompting baselines.

  2. SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    An RL framework that trains vision-language models to self-correct captions via a scene-graph-based reward outperforms SFT and DPO on caption quality.

  3. Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A new error-attribution dataset and fine-tuned judge model that outputs score, error category, and feedback for LLM responses.

  4. Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B

    cs.CL 2026-07 accept novelty 5.5 of 10

    Self-inspection methods (Self-Refine, Reflexion, Best-of-N self-verify) lose to equal-token repeated sampling on math from 1.5B to 7B; no tested method reliably wins.

  5. LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Reference-free LLM judge scores failed to select better table-extraction outputs over eight regeneration iterations on FinTabNet and OmniDocBench; keeping the first output was safest.

  6. Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Reinforcement fine-tuning largely prevents catastrophic forgetting during continual post-training of a multimodal LLM, while supervised fine-tuning degrades both task and general performance.

  7. An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability

    cs.CL 2025-06 conditional novelty 5.0 of 10

    The reliability of LLM-as-a-Judge depends strongly on scoring rubrics and reference answers; sampling with averaging outperforms greedy decoding, and chain-of-thought reasoning adds little when rubrics are clear.

  8. Can LLMs $\textit{understand}$ Math? -- Exploring the Pitfalls in Mathematical Reasoning

    cs.CL 2025-05 reject novelty 5.0 of 10

    The paper proposes MAPLE, an LLM-judged metric for mathematical reasoning misalignment, but the metric's formula contradicts its stated behavior.

  9. Agentic AI for Commercial Insurance Underwriting with Adversarial Self-Critique

    cs.AI 2026-01 conditional novelty 4.0 of 10

    Adding an adversarial critic agent to a human-in-the-loop insurance underwriting AI reduced hallucinations by two-thirds and raised decision accuracy from 92% to 96% on 500 expert-validated cases.

Pith tools