{"id":"8da09be6-880e-4436-b638-7361e8139dec","arxiv_id":"2505.22823","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A self-critique and refinement framework with word-level feedback cuts unfaithfulness rates in LLM explanations by about 19 points on average, but gains may partly reflect the word-presence evaluation metric.","lead":"This paper tests whether large language models can improve the faithfulness of their own explanations by critiquing and refining them, with feedback that highlights important input words. Across three reasoning datasets and four LLMs, the method lowers the rate at which explanations miss the word that drove the prediction change, though the evaluation uses a narrow word-presence metric.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The counterfactual metric rewards echoing the fed-back word, and the paper's own random-word baseline (Table 12) shows word-selection quality does not drive the gains; the faithfulness claim is therefore not yet established.","rationale":"The paper is a well-structured empirical study with transparent reporting, including a random-word baseline that most authors would omit. The central claim is precisely stated with a specific number (36.02% vs 54.81%). My concern is not that the numbers are wrong under the reported metric, but that the metric is not a valid operationalization of the construct the paper claims to improve. The counterfactual test as implemented checks only whether the intervened word appears in the explanation; it does not check whether the explanation would let a reader predict the model's behavior, nor whether the cited reason is causally relevant. In the IWF condition, the refinement instruction explicitly tells the model to include the provided words, so when the intervened word is in the top-5, the model's inclusion of it is a compliance behavior, not evidence of self-assessment. The random baseline (Table 12) is the strongest internal evidence: random words achieve essentially the same unfaithfulness rate as IWF-Pmt and close to IWF-Attn, indicating that the specific attribution method is not the operative ingredient. The paper interprets this as 'robustness of IWF', but it also means the claimed novelty—attribution-based feedback—is not what drives the improvement. The length confound (Figure 6) compounds the problem: longer explanations are more likely to contain any target word, and the refinement process systematically lengthens explanations. The authors acknowledge the metric's narrowness in the Limitations section, which is commendable, but the evaluation section still treats the counterfactual test as the sole arbiter of faithfulness. The proposed concrete test—conditioning on whether the intervened word was in the feedback—would settle whether the gains are mechanical. If they are, the paper's conclusion 'LLMs can refine their explanations to better reflect their actual reasoning' would need to be substantially qualified. For these reasons, a CONDITIONAL verdict (pending such a test or a second metric) is appropriate; I do not move to REJECT because the paper's reported numbers appear honest and the framework might still improve faithfulness under a more robust metric.","tokens_in":23886,"tokens_out":8231,"duration_ms":77221,"concrete_test":"Reanalyze the existing counter instances from Table 1 for IWF-Attn. Split them into (A) instances where the intervened word is among the top-5 fed-back words and (B) instances where it is not. Compute post-refinement unfaithfulness rates for each subset and compare them with the corresponding Init-NLE rates on the same subsets. If the improvement over Init-NLE is almost entirely concentrated in subset (A), and subset (B) shows little or no improvement, the apparent faithfulness gain is an artifact of the refinement prompt's instruction to echo the fed-back word, not of the model assessing and revising its reasoning. Additionally, run a length-matched control: compare unfaithfulness rates of refined explanations against initial explanations of the same length (e.g., by truncating or via propensity-score matching on length).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the counterfactual test (Section 4.3, Eq. 10) measures whether an explanation reflects the model's actual reasoning. But the test only checks, via string matching, whether the explanation contains the intervened word. For IWF variants, the refinement prompt (Table 22) instructs the model to 'integrate the important words naturally', and the intervention word is itself part of the fed-back list whenever the attribution/prompt selector includes it. Thus a refined explanation can pass the test by mechanically echoing a supplied word rather than by reasoning about it. The paper's Appendix F.6 provides evidence for exactly this mechanism: a random-word baseline (Table 12) achieves unfaithfulness of 29.28% on e-SNLI, close to or better than IWF-Pmt (30.10%), IWF-Attn (27.69%), and IWF-IG (28.39%), and Table 13 shows the intervened word is in the top-5 for only 46.25% (Attn) and 34.00% (random) of counter instances. If faithfulness were being genuinely improved, the quality of selected words should matter; it does not. Figure 6 further shows a strong inverse length–unfaithfulness relationship with no length-controlled baseline, so longer refined explanations are more likely to contain the intervened word by chance. The paper's own Limitations section concedes that the counterfactual test 'reflects only one type of faithfulness criterion'. Without a second, semantically grounded faithfulness metric, the 18.79% absolute reduction does not establish that explanations better reflect the model's reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SR-NLE, an iterative self-critique and refinement framework that aims to improve the faithfulness of post-hoc natural language explanations (NLEs) generated by LLMs. The framework generates an initial NLE, then iteratively refines it using either free-form natural language feedback (NLF) or important-word feedback (IWF) derived from prompt-based, attention-based, or gradient-based attribution. Experiments on ComVE, ECQA, and e-SNLI with four instruction-tuned LLMs under 10B parameters evaluate faithfulness using counterfactual tests in which an explanation is considered faithful if it contains the intervened word by string matching. The best variant, IWF-Attn, reduces the average unfaithfulness rate from 54.81% to 36.02%, an absolute reduction of 18.79 percentage points.","tokens_in":24178,"tokens_out":7315,"duration_ms":75243,"significance":"If the claimed effect is genuine, the paper would make a practically useful contribution: an inference-time method for improving explanation faithfulness without fine-tuning or architectural changes, with released code and a broad experimental matrix (3 datasets x 4 models). The paper also includes useful ablations and an honest random-word baseline in Appendix F.6. However, the central claim is not yet established because the evaluation metric is aligned with the IWF feedback mechanism, no significance tests are provided, and the length-faithfulness correlation is uncontrolled. The paper's own Limitations section concedes that the counterfactual test reflects only one type of faithfulness criterion. The value of the paper therefore depends on whether the evaluation can be made more robust and whether the gains can be shown to be specific to informative feedback rather than to prompt-driven word echoing or length increases.","major_comments":[{"comment":"The counterfactual metric in Eq. (10) counts an explanation as faithful if it contains the intervened word, matched by string. For IWF variants, the refinement prompt (Appendix I, Table 22) instructs the model to \"integrate the important words naturally,\" and the intervened word is part of the supplied feedback whenever it appears in the top-N list. Thus a refined explanation can pass the test by echoing a fed-back word rather than by demonstrating the model's actual reasoning. This concern is supported by the paper's own random-word baseline: Table 12 shows that random word selection achieves an unfaithfulness of 29.28% on e-SNLI, close to IWF-Pmt (30.10%) and IWF-IG (28.39%), while Table 13 shows the intervened word is in the top-5 only 46.25% of the time for IWF-Attn and 34.00% for random. The Limitations section concedes that the counterfactual test \"reflects only one type of faithfulness criterion.\" Without an additional semantically grounded faithfulness metric, the reported absolute reduction of 18.79% does not establish that the explanations better reflect the model's reasoning.","section":null},{"comment":"Figure 6 shows a strong inverse relationship between explanation length and unfaithfulness across refinement rounds, but the paper provides no length-controlled baseline. Since longer explanations are more likely to contain the intervened word by chance, the observed improvement could be an artifact of the refinement prompts producing longer text. The claim in §5.3 that IWF methods \"optimize explanation content rather than merely increasing length\" requires a length-matched comparison or a length-normalized metric; as presented, the causal interpretation is not supported.","section":null},{"comment":"The abstract and §5.1 state that SR-NLE \"significantly reduces\" unfaithfulness, but the main results in Table 1 are reported without significance tests, confidence intervals, or standard deviations. Differences between IWF variants are small (e.g., 36.02% vs. 36.30% vs. 37.23%), and the random-word baseline in Appendix F.6 overlaps with these numbers. Statistical testing across the 12 model-dataset conditions is needed to support the comparative and \"significant\" claims.","section":null},{"comment":"The framework assumes that LLMs can identify and improve their own explanations when guided with appropriate prompts, but this assumption is not tested directly. The random-word baseline in Table 12 achieving near-IWF performance suggests that the model may be following a surface instruction to incorporate supplied words rather than evaluating which words are causally important for its prediction. A control that measures self-assessment ability independently of the feedback content, or a comparison with feedback containing genuinely irrelevant words in a semantically meaningful way, would strengthen this assumption.","section":null}],"minor_comments":[{"comment":"Several numerical entries are run together without spaces (e.g., \"24.3243.12\" in Table 1 under ECQA/Llama/IWF-Pmt), making the tables difficult to read.","section":null},{"comment":"The abstract claims the method requires \"no external supervision,\" but Appendix C uses GPT-4o to generate counterfactual interventions. Please clarify that this external model is used only for evaluation, not for the refinement process.","section":null},{"comment":"The comparison with G-TEX is acknowledged as not fully controlled, but the abstract and conclusion still describe SR-NLE as outperforming prior methods; either remove this comparison or keep the caveat in the summary statements.","section":null},{"comment":"Ablation studies are conducted on 100-instance subsets without confidence intervals; the choice of N=5 and other hyperparameters would be more convincing with error bars or a small repeat-seed analysis.","section":null}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the alignment between the evaluation metric and the IWF feedback. If the authors can provide a second, semantically grounded faithfulness metric and show significant gains over a random-word baseline and a length-controlled condition, I would be willing to accept the paper. As it stands, the current evidence does not support the central claim that the refined explanations better reflect the model's actual reasoning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes SR-NLE, a training-free self-refinement framework for improving the faithfulness of post-hoc natural language explanations, with feedback in the form of natural language or important words derived from feature attribution. The contribution is real as an application of self-refinement to a new target, and the empirical work is extensive: three datasets, four models, three refinement rounds, ablations, case studies, and released code. What it does well is transparency: the authors include a random-word baseline they did not have to include, and the limitations section concedes that the counterfactual test reflects only one type of faithfulness criterion.\n\nThe soft spot is central, not peripheral. The evaluation metric is a counterfactual test that counts an explanation as faithful if it contains the intervened word via string matching. For the IWF variants, the refinement prompt explicitly instructs the model to integrate the important words naturally, and the intervened word is itself in that list whenever the attribution or prompt selector includes it. So a refined explanation can pass the test by echoing a supplied word rather than by reasoning about it. The paper's own Table 12 shows a random-word baseline almost matching the IWF variants on e-SNLI, and Table 13 shows the intervened word is captured in the top-5 at rates similar to random selection. That is strong evidence that word-selection quality is not the driver; the iterative process itself and the increased explanation length (Figure 6) are confounds. There are no significance tests, and the G-TEX comparison is not apples-to-apples.\n\nI do not think the paper is dishonest or careless; it surfaces the very evidence that weakens its claim. But the claim that the explanations better reflect the model's actual reasoning is not established by the chosen metric. The strongest defensible statement is that SR-NLE reduces unfaithfulness as operationalized by the word-presence counterfactual test. That is still a useful finding, and the idea of using attribution feedback in self-refinement is worth exploring, but the paper needs a second, semantically grounded faithfulness metric and a length-controlled baseline before the central claim can stand.\n\nThis is for people working on explanation faithfulness and self-refinement in LLMs. It deserves a serious referee, because the framework is novel and the authors' transparency gives reviewers a tractable path to strengthening the evaluation.","headline":"A transparent but methodologically under-powered study: the self-refinement framework is new and the experiments are consistent, yet the faithfulness metric and the random-word baseline undermine the central claim.","tokens_in":24732,"tokens_out":2194,"would_cite":false,"duration_ms":23770,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can make their own post-hoc explanations more faithful to their actual reasoning by iteratively critiquing and refining them, cutting unfaithfulness from 54.81% to 36.02% without any fine-tuning.","keywords":["natural language explanations","faithfulness","self-refinement","self-critique","feature attribution","important word feedback","counterfactual test","large language models"],"falsifier":"Compare SR-NLE's refined explanations against a length-matched baseline that asks the model to rewrite without important-word feedback, or against a different faithfulness metric (e.g., simulatability or human agreement); if the unfaithfulness reduction disappears or reverses, the central claim is falsified.","tokens_in":23653,"feed_emoji":"🤖","tokens_out":10592,"duration_ms":96873,"temperature":0.7,"pith_summary":"The paper asks whether a large language model can make its own natural language explanations more faithful to its actual reasoning, without external supervision or training. The proposed framework, SR-NLE, runs an iterative loop in which the model critiques its explanation and rewrites it, guided either by free-text self-feedback or by a list of the input words that most influenced the prediction. Across three reasoning datasets and four instruction-tuned models, the strongest variant, attention-based important-word feedback, lowers the average unfaithfulness rate from 54.81% to 36.02% on a counterfactual faithfulness test. The authors conclude that LLMs can substantially improve the faithfulness of their own explanations, given the right kind of feedback.","feed_headline":"Self-critique cuts explanation unfaithfulness from 54.8% to 36.0%","feed_subtitle":"No fine-tuning or extra models needed: LLMs guided by important words make their own explanations more faithful.","key_machinery":"The central object is the important-word feedback (IWF) signal: a small set of input words that the model's own prediction depended on, extracted either by prompting the model to rank words or by computing feature attributions (attention weights or integrated gradients). Each round, the model receives the top-5 important words and is asked to integrate them into its explanation. This works by making the explanation mention the lexical evidence that actually changed the prediction, which is exactly what the counterfactual test checks. The iterative loop, running for up to three rounds, converts this guidance into progressively more faithful text, with attention-based scoring yielding the best results.","core_discovery":"The paper introduces SR-NLE and claims that an LLM can refine its own post-hoc explanations so they better reflect the reasoning that produced its prediction. The framework generates an initial explanation, then iterates between feedback and refinement; feedback can be natural-language self-critique or a list of the top-N most important input words identified by prompt-based, attention-based, or gradient-based attribution. Using a counterfactual test that judges an explanation faithful when it contains the intervened word, SR-NLE with attention-based important word feedback reduces the unfaithfulness rate from 54.81% to 36.02% on average across ComVE, ECQA, and e-SNLI with Falcon, Llama, Mistral, and Qwen models. The claim is that the improvement is driven by the iterative self-critique process itself and by feedback that points the model to the words that actually carried its decision.","pith_inferences":["Editorial inference: because prompt-based IWF nearly matches attribution-based IWF, the method likely transfers to API-only LLMs with no gradient or attention access; testing SR-NLE on a closed-weight model would verify this.","Editorial inference: the inverse relationship between explanation length and unfaithfulness (Figure 6) leaves open whether part of the gain is a verbosity effect; a length-matched baseline that rewrites without important-word feedback would separate the feedback mechanism from mere elaboration.","Editorial inference: the appendix finding that random important words still reduce unfaithfulness, and that the intervened word is only captured 30–46% of the time in top-5 selections, suggests the refinement loop itself carries much of the benefit; an even simpler feedback prompt (e.g., 'mention the word that changed your prediction') might capture a similar effect.","Editorial inference: a natural next test is to apply SR-NLE to chain-of-thought rationales or jointly generated explanations, which the authors explicitly leave open."],"forward_implications":["If SR-NLE is correct, explanation faithfulness can be improved at inference time with no fine-tuning, no architectural changes, and no extra models, using only the target LLM itself.","Prompt-based important-word feedback performs nearly as well as attribution-based feedback (37.23% vs 36.02% unfaithfulness), indicating that explicit access to internal attributions is not required.","Most of the improvement comes in the first refinement round, so two to three rounds offer a practical trade-off between faithfulness and compute.","Even the best variant leaves an average unfaithfulness rate of 36%, so self-refinement alone does not fully solve the faithfulness problem; further gains would require better feedback signals or training."],"supporting_citations":[{"why":"Defines the counterfactual faithfulness test and the unfaithfulness metric (string match of the intervened word) that all experiments use.","marker":"Atanasova et al. (2023)"},{"why":"Establishes the iterative self-refine paradigm that SR-NLE adapts to explanation faithfulness.","marker":"Madaan et al. (2023)"},{"why":"Shows attribution-derived important words can be computed directly from LLMs, the basis for IWF-Attr.","marker":"Bhan et al. (2024)"},{"why":"Shows LLMs can be prompted to score input word importance, the basis for IWF-Pmt.","marker":"Kroeger et al. (2023)"},{"why":"Defines integrated gradients, used as the gradient-based IWF-IG attribution.","marker":"Sundararajan et al. (2017)"},{"why":"G-Tex, the prior state-of-the-art faithfulness-improvement method that SR-NLE claims to outperform.","marker":"Yuan et al. (2025)"},{"why":"Self-consistency approach used to construct the SC-NLE baseline.","marker":"Wang et al. (2023b)"}],"fun_headline_variants":["Self-critique refines LLM explanations without fine-tuning","LLMs self-criticize to cut unfaithful explanations by 18.8 points","No tuning needed: self-critique makes explanations more faithful","AI self-critique boosts explanation fidelity across 4 LLMs","Self-critical feedback loops improve AI explanation trustworthiness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central claim rests on the counterfactual test's rule that an explanation is faithful if it contains the intervened word; if that string-match criterion does not capture whether the explanation reflects the model's actual reasoning, the measured reduction in unfaithfulness would not establish the claimed benefit.","fun_headline_variants_meta":{"raw":{"variants":["Self-critique refines LLM explanations without fine-tuning","LLMs self-criticize to cut unfaithful explanations by 18.8 points","No tuning needed: self-critique makes explanations more faithful","AI self-critique boosts explanation fidelity across 4 LLMs","Self-critical feedback loops improve AI explanation trustworthiness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1231,"prompt_tokens":969,"completion_tokens":262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":172}},"tokens_in":585,"tokens_out":262,"duration_ms":3456,"temperature":1.0,"reasoning_tokens":172,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:59:24.629107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare SR-NLE's refined explanations against a length-matched baseline that asks the model to rewrite without important-word feedback, or against a different faithfulness metric (e.g., simulatability or human agreement); if the unfaithfulness reduction disappears or reverses, the central claim is falsified.","supporting_citations":[{"cited_title":"Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations","cited_arxiv_id":"2402.12038","evidence_quote":"Shows attribution-derived important words can be computed directly from LLMs, the basis for IWF-Attr."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs can be prompted to score input word importance, the basis for IWF-Pmt."}],"review_version":1}