{"id":"2fe93519-b653-4951-89b9-6a77e70c9df9","arxiv_id":"2412.12527","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CDA is a training-free decoding method that weights parametric, contextual, and abstention distributions using null-prompt calibrated entropy, letting LLMs answer when they can and abstain when they cannot.","lead":"This paper introduces Contrastive Decoding with Abstention (CDA), a training-free decoding method that helps large language models answer a question when they have relevant knowledge and say they do not know when they do not. Across four models and three question-answering datasets, CDA beats prompt-only and entropy-based abstention baselines on both answer accuracy and abstention quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the paper's claim that CDA(-M) beats all baselines on F1_abs: in HotpotQA with LLAMA2-7B-Chat and MISTRAL-7B-Instruct, FSB and ENTROPY-first-token score higher than CDA-M.","rationale":"The reader's weakest_assumption concerns whether null-prompt entropy calibrates across models and domains. That is a legitimate generalization risk, but the paper has a more immediate, internal problem: the headline empirical claim is contradicted by the paper's own Table 1 before any external generalization is considered. Since the central claim in the reader's strongest_claim includes 'outperform all tested baselines on F1_abs... across three datasets and four LLMs,' and CDA-M does not beat FSB on HotpotQA/LLAMA2-7B (42.31 vs 53.82) or ENTROPY on HotpotQA/MISTRAL (56.67 vs 60.07), the claim as stated is false. This does not require reimplementation or speculation about prompt formats; it is a table-level check. The appropriate response is to condition acceptance on correcting the claim and providing per-cell analysis, which is the same CONDITIONAL verdict the reader reached but for a different reason. I also share the reader's concerns about Eq. 7 ambiguity and missing code, but those are secondary to a direct factual mismatch in the reported results.","tokens_in":32315,"tokens_out":14994,"duration_ms":134636,"concrete_test":"Recompute F1_abs for the HotpotQA row of LLAMA2-7B-Chat from Table 11: compare FSB (53.82), ACD-A (51.11), ABSTAIN (48.65), ENTROPY-first-token (45.11), CDA (42.41), and CDA-M (42.31). If max(baselines) > max(CDA, CDA-M), the claim 'outperforms all baselines on F1_abs' fails for that cell; repeat for MISTRAL-7B-Instruct HotpotQA (Table 13: ENTROPY 60.07 vs CDA-M 56.67). This table-only check settles whether the strongest claim needs to be revised to a per-metric or per-setting qualified claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest version of the central claim—that CDA and CDA-M outperform every baseline on F1_ans, F1_abs, and Reliability Score across all three datasets and all four models—is not supported by the paper's own results. Table 1 (full numbers in Tables 11 and 13) shows clear counterexamples on F1_abs. For HotpotQA with LLAMA2-7B-Chat, F1_abs is 53.82 for FSB, 51.11 for ACD-A, and 48.65 for ABSTAIN, while CDA and CDA-M score 42.41 and 42.31, respectively. For HotpotQA with MISTRAL-7B-Instruct, ENTROPY-first-token achieves 60.07, while CDA-M achieves 56.67. Thus the 'outperform all baselines on F1_abs' assertion is false as written; the abstract and conclusion overstate what the experiments establish. This is not a question of null-prompt calibration generalizing to unseen domains—it is a direct mismatch between the reported numbers and the paper's central empirical claim, and it materially changes the strength of the contribution.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Contrastive Decoding with Abstention (CDA and CDA-M), a training-free decoding method that combines parametric, contextual, and abstention output distributions. The weights for these distributions are derived from entropy differences between the actual input and a content-free null prompt, intended to measure how much relevant knowledge each source provides. The authors construct a controlled testbed with four knowledge-access scenarios and evaluate on NQ, HotpotQA, and TriviaQA with four instruction-tuned LLMs, reporting that CDA(-M) outperforms baselines on answerable-generation F1, abstention F1, and Reliability Score, including in a retrieval-augmented setting and against instruction-tuning baselines.","tokens_in":32562,"tokens_out":6406,"duration_ms":58854,"significance":"If the method works as described, it is a useful contribution: it offers a training-free way to combine abstention with contrastive decoding, with an intuitive null-prompt calibration idea. The experimental design is extensive in breadth—four models, three datasets, three seeds, ablations, a RAG setting, and a comparison with training-based abstention—and the paper includes concrete templates, hyperparameter values, and a computation-cost analysis. The central empirical claim, however, is overstated in the current text, and the core equations defining the weight construction are presented in a way that is internally inconsistent with the prose. These issues are fixable, but they affect the reproducibility and the exact strength of the claimed contribution.","major_comments":[{"comment":"The paper repeatedly states that CDA(-M) outperforms all baselines on F1_abs and Reliability Score, but the reported numbers contain direct counterexamples. In Table 1 (full results in Table 11), HotpotQA with LLAMA2-7B-Chat gives F1_abs = 42.41 for CDA and 42.31 for CDA-M, while FSB scores 53.82 and ACD-A scores 51.11; in HotpotQA with MISTRAL-7B-Instruct, ENTROPY-first-token achieves 60.07 while CDA-M achieves 56.67. The RAG results in Table 9 also show HotpotQA with LLAMA2-13B-Chat where FSB has RS = 45.28 while CDA-M has RS = 44.73, contradicting the caption of Figure 7. The abstract, Section 5.4, Section 6.5, and the table captions should be revised to state the actual pattern: CDA(-M) is best on most settings and metrics, but not all baselines on F1_abs and RS in every configuration.","section":"§5.4, Table 1; §6.5, Table 9"},{"comment":"The sign convention in Eq. (6) appears inconsistent with the surrounding text. The paper defines confidence as the additional information provided by the input relative to the null prompt, and lower entropy is standardly interpreted as higher confidence. Yet Eq. (6) sets r = max(H_input - H_null, 0) / H_null, which is positive only when the input distribution is more uncertain than the null prompt. As written, a knowledge source is upweighted when its entropy exceeds the null entropy, which is the opposite of the stated intuition. If the implementation actually uses H_null - H_input (or a ratio such as H_null / H_input), Eq. (6) must be corrected; otherwise the method as defined would down-weight the very sources the model is most confident about. This is load-bearing because all CDA weights derive from this quantity.","section":"§4.3, Eq. (6)"},{"comment":"Eq. (7) does not define a normalized set of weights as claimed. The printed formulas w_p = r_p / (r_p + r_c) * r_p and w_c = r_c / (r_p + r_c) * r_c give w_p + w_c = (r_p^2 + r_c^2) / (r_p + r_c), which is generally not equal to 1. Consequently w_a = 1 - w_p - w_c is not guaranteed to be nonnegative, and the interpretation of w_a as the abstention weight in Eq. (4) breaks. If the intended formulas are w_p = r_p / (r_p + r_c) and w_c = r_c / (r_p + r_c), the extra multiplication by r_p and r_c should be removed. Because Eq. (7) is the core weighting mechanism, this must be fixed before the method can be reproduced.","section":"§4.3, Eq. (7)"},{"comment":"The testbed labels parametric and contextual knowledge using the same model that is later evaluated. A sample is labeled P=1 only if the model itself answers consistently, and C=1 only if the model itself answers with the context; the same model's entropy then determines abstention behavior. This means the evaluation primarily measures whether CDA can exploit the model's self-consistency signal, not whether the model truly possesses or lacks external knowledge. The paper should discuss this construct-validity limitation more explicitly and, ideally, provide a small external validation (e.g., labels derived from a different model or human judgments) to show that the 'absent knowledge' scenarios correspond to genuinely unanswerable queries rather than to queries the model happens to fail consistently.","section":"§3.3–§3.5"}],"minor_comments":[{"comment":"The phrase 'where di it the ith token' contains a typo; it should read 'where d_i is the i-th token probability'.","section":"§4.3, Eq. (5)"},{"comment":"The sentence 'the model is considered to pose relevant parametric knowledge' should say 'possess'.","section":"§3.3"},{"comment":"The appendix title 'Experiential Setting Details' should be 'Experimental Setting Details'.","section":"Appendix B title"},{"comment":"The text says CDA uses templates from 'Table 3', but the templates appear in Figure 3; the cross-reference should be corrected.","section":"§B.1"},{"comment":"The caption 'CDA(-M) outperforms all the baselines across different metrics' is not supported by the table's own numbers; it should be rephrased to reflect that CDA(-M) is best on the majority of settings and metrics, with exceptions noted.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising, and the experimental breadth is a strength. The main blockers are the false universal-overperformance claims and the inconsistencies in Eqs. (6)–(7), which are likely typos but must be resolved for reproducibility. I do not see evidence of deliberate misreporting; the counterexamples are visible in the paper's own tables. If the equations are corrected and the claims are qualified, the paper could become a solid contribution to decoding-time abstention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, useful decoding-time abstention method with a clear new component — the abstention distribution plus null-prompt calibration — but the paper's headline claim is too strong. Their own Table 1 shows CDA-M losing to FSB and ACD-A on F1_abs in HotpotQA with Llama-2-7B, and to Entropy-first-token with Mistral-7B. The central mechanism still seems to hold in most cells, but the abstract and conclusion need softening.\n\nWhat's genuinely new: CDA extends context-aware contrastive decoding with an explicit abstention distribution and a null-prompt-based calibration that makes entropy comparisons across different prompts meaningful. The momentum variant stabilizes weights and helps in the noise-context case. The controlled testbed with four knowledge-access scenarios is a good tool, and the ablations — especially the w/o-calibration drop and the comparison with instruction-tuning — give real evidence that the calibration step is load-bearing. Experimental breadth is above average: four models, three datasets, three seeds, plus a RAG setting.\n\nSoft spots: The overclaim is the main one. Section 5.4 says CDA(-M) outperforms all baselines on F1_abs, but Tables 11 and 13 show clear counterexamples. That is a direct mismatch, not a nitpick. Second, Eq. 7 as printed multiplies the normalized share by the raw relevance again, so the weights don't sum to 1; likely a typo, but it should be corrected. Third, the testbed labels parametric and contextual knowledge by the same model that is later evaluated, so part of the evaluation is circular — this is a controlled testbed of model self-assessment, not an independent measure of knowledge availability. The paper acknowledges the controlled setup but should be more explicit about the circularity and its limits. Fourth, no code or data artifacts are released. The free parameters (eta, n, alpha) are reasonable, but only alpha gets a sensitivity analysis; eta and n are fixed.\n\nBottom line: For someone working on decoding-time abstention or RAG reliability, this is worth reading. The core idea is training-free, simple, and mostly well-demonstrated. I'd send it to review with a request to correct the overclaim and the equation typo, and to be more careful about the self-referential testbed. Not a desk reject — it just needs honest revision.","headline":"Solid training-free abstention method with a useful testbed, but the paper overclaims on F1_abs and needs a corrected Eq. 7 and more honest testbed framing.","tokens_in":33130,"tokens_out":2634,"would_cite":true,"duration_ms":23479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free decoding method, CDA, lets large language models answer when either parametric or contextual knowledge is available and abstain when neither is, by weighting three output distributions with null-prompt-calibrated entropy.","keywords":["contrastive decoding","abstention","question answering","uncertainty calibration","retrieval-augmented generation","knowledge gaps","decoding-time method","null-prompt entropy"],"falsifier":"Take a QA set where the model's parametric and contextual knowledge are known to be present or absent, run CDA with a null prompt that is not content-free (for example, a generic factual sentence in place of the placeholders), and check whether abstention F1 collapses toward the level of an always-answering baseline; if it does, the calibration step, not the mixture form, is carrying the result.","tokens_in":32098,"feed_emoji":"🤐","tokens_out":4802,"duration_ms":39872,"temperature":0.7,"pith_summary":"Large language models are expected to answer from two knowledge sources, the parameters learned in pretraining and the context supplied at inference time, but they have no built-in way to notice when neither source contains the answer. This paper proposes Contrastive Decoding with Abstention (CDA), a decoding-time method that estimates, at every step, how much each source contributes relative to a content-free null prompt and then blends parametric, contextual, and abstention distributions accordingly. The paper builds a controlled QA testbed in which the presence or absence of each knowledge source is known, and reports that CDA and its momentum variant CDA-M outperform all tested baselines on answer F1, abstention F1, and reliability score across three datasets and four instruction-tuned LLMs. The significance is that a model can learn to say \"unknown\" without any additional training.","feed_headline":"A decoding tweak teaches LLMs when to say \"unknown\"","feed_subtitle":"Null-prompt entropy weights decide when to answer from memory or context and when to abstain, with no extra training.","key_machinery":"The load-bearing object is the calibrated relevance ratio $r^p_t = \\max(H^p_t - \\bar{H}^p_t, 0)/\\bar{H}^p_t$ and its contextual analogue $r^c_t$. Here $H^p_t$ and $H^c_t$ are the entropies of the parametric and contextual output distributions at decoding step $t$, while $\\bar{H}^p_t$ and $\\bar{H}^c_t$ are the entropies of the same prompts with the actual question and context replaced by the placeholders \"[QUESTION]\" and \"[CONTEXT]\". These ratios are normalized into weights $w^p_t$ and $w^c_t$, with the abstention weight $w^a_t = 1 - w^p_t - w^c_t$, and the final distribution is $d^o_t = w^p_t d^p_t + w^c_t d^c_t + w^a_t d^a_t$. The mechanism turns the question of whether a knowledge source knows the answer into a relative-entropy comparison against the model's own baseline uncertainty.","core_discovery":"On its own terms, the paper establishes that decoding can be made to abstain by treating abstention as a third output distribution. CDA forms the final distribution as a weighted mixture of the parametric distribution, the contextual distribution, and an abstention distribution obtained from an explicit instruction to answer or else say \"Unknown\"; the weights are set by calibrating the entropy of each knowledge-conditioned distribution against the entropy of the same prompt with placeholders replacing the question and context. When a knowledge source adds information beyond the null prompt, it receives higher weight; when neither does, the abstention distribution dominates. In the paper's testbed, which separates answerable from unanswerable queries, this mechanism yields the best balance between correct answers and appropriate refusals among all compared methods, including training-based instruction tuning, and it generalizes to retrieval-augmented settings.","pith_inferences":["A natural extension the paper leaves implicit is to treat abstention as a controllable behavior per domain, for example by adjusting the abstention template or the momentum coefficient to trade false refusals against hallucinations.","The null-prompt calibration suggests a cheap diagnostic: the ratio $r$ could be computed once per query to predict whether a model will answer correctly, independent of any decoding scheme, turning the mechanism into a standalone confidence score.","A testable extension is to apply CDA to long-form generation or multi-hop reasoning where context is only partly relevant; the authors scope the testbed to short-form QA under a single-context assumption, and that boundary may not hold in wider settings.","Because CDA roughly doubles decoding cost, caching null-prompt entropies across queries with shared templates could cut the overhead in production use."],"forward_implications":["If CDA is correct, any instruction-tuned LLM can acquire abstention behavior at inference time, with no gradient updates, by running three forward passes per decoding step.","The calibrated ratio should transfer across questions and contexts within a model, and CDA-M's momentum smoothing suggests that stable weights over decoding steps improve answer reliability without sacrificing abstention.","In RAG pipelines, CDA offers a decoding-time safeguard: irrelevant retrieved contexts are downweighted, and unanswerable queries can terminate in an abstention response.","Compared with instruction tuning, which degrades out of domain, CDA is claimed to generalize across target datasets without retraining.","The controlled testbed makes the four knowledge-access scenarios explicit, so the same construction can be reused to evaluate any future abstention method under known knowledge availability."],"supporting_citations":[{"why":"Supplies the contrastive decoding basis of contrasting output distributions for text generation.","marker":"(Li et al., 2023)"},{"why":"Defines context-aware decoding (CAD), the template that CDA extends by adding abstention and adaptive weights.","marker":"(Shi et al., 2024b)"},{"why":"Provides adaptive contrastive decoding with entropy-based weights, the direct precursor for CDA's knowledge weighting.","marker":"(Kim et al., 2024b)"},{"why":"Supports entropy-based relevance estimation in retrieval-augmented settings, which CDA adapts to parametric and abstention distributions.","marker":"(Qiu et al., 2024)"},{"why":"Supplies the F1_ans and F1_abs metrics and the abstention-phrase detection used for evaluation.","marker":"(Kim et al., 2024a)"},{"why":"Defines the Reliability Score, the weighted accuracy-coverage metric that drives the reported comparison.","marker":"(Xu et al., 2024)"},{"why":"Motivates the calibration-before-use principle that underlies the null-prompt calibration step.","marker":"(Zhao et al., 2021)"},{"why":"Supports the idea that surface-form and input-conditioned biases distort raw probabilities, motivating relative entropy calibration.","marker":"(Holtzman et al., 2021)"},{"why":"Provides the entropy-based uncertainty formulation used to measure knowledge relevance.","marker":"(Malinin and Gales, 2021)"},{"why":"Supplies the MRQA benchmark that sources the NQ, HotpotQA, and TriviaQA testbed datasets.","marker":"(Fisch et al., 2019)"}],"fun_headline_variants":["No training needed: LLMs learn to abstain when clueless","Entropy-based decoding lets LLMs say 'I don't know'","CDA: a training-free way for LLMs to abstain when clueless","Using entropy to let LLMs skip questions they can't answer","A decoding trick that gives LLMs the option to abstain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if the entropy of a content-free null prompt, with placeholders in place of the question and context, reliably measures the model's intrinsic bias, so that the relative entropies $r^p_t$ and $r^c_t$ correctly indicate which knowledge source actually contains the answer.","fun_headline_variants_meta":{"raw":{"variants":["No training needed: LLMs learn to abstain when clueless","Entropy-based decoding lets LLMs say 'I don't know'","CDA: a training-free way for LLMs to abstain when clueless","Using entropy to let LLMs skip questions they can't answer","A decoding trick that gives LLMs the option to abstain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2304,"prompt_tokens":886,"completion_tokens":1418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1323}},"tokens_in":502,"tokens_out":1418,"duration_ms":10422,"temperature":1.0,"reasoning_tokens":1323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:59:27.048446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a QA set where the model's parametric and contextual knowledge are known to be present or absent, run CDA with a null prompt that is not content-free (for example, a generic factual sentence in place of the placeholders), and check whether abstention F1 collapses toward the level of an always-answering baseline; if it does, the calibration step, not the mixture form, is carrying the result.","supporting_citations":[],"review_version":1}