{"id":"dd87dede-2e79-4c5e-bc8f-7d881042fa28","arxiv_id":"2501.01059","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DAGCD improves context faithfulness by amplifying context tokens that attention marks as relevant, scaled by token-level uncertainty.","lead":"This paper introduces DAGCD, a decoding method that uses a language model's own attention scores to boost words from the provided context during generation, reducing answers that ignore the context. It reports large exact-match gains on seven open-book QA sets while keeping a single decoding pass.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Detector trained only at the first decoded token and on gold-answer labels is applied at every decoding step; no per-step validation supports Eq. 6.","rationale":"The reader's weakest assumption is essentially the same as the concern identified here: the detector's training signal is narrow (gold-answer tokens at the first generated token), yet DAGCD applies it broadly at every decoding step. My stress-test sharpens this into a concrete, falsifiable claim about per-step detector accuracy. If the detector only recognizes answer words from the first position, then the multi-token answer improvements that drive the headline EM gains are unexplained by the paper's stated mechanism. The paper provides cross-domain, prompt, and data-size validation of the detector, but all of that validation is performed in the training condition (first token after context, gold-answer labels). The stepwise application is a qualitatively different regime: after the first token is generated, the attention query position shifts, previously generated tokens enter the context, and the tokens that should be amplified include continuation tokens of a multi-word answer rather than the first answer word alone. No experiment in the paper characterizes this regime. This is not a claim that the method fails; it is a claim that the central mechanism is unverified exactly where the method is most needed. The proposed test isolates this by measuring detector AUC per decoding step on multi-token answers. If the detector maintains high AUC at later steps, the concern is resolved and the conditional verdict can be upgraded. If it does not, the paper's explanation for its own gains is unsupported, and the conditional verdict should remain or move toward reject. I keep the reader's CONDITIONAL verdict because the concern is addressable and the empirical headline numbers could survive even if the mechanism is mischaracterized; but the missing per-step validation is a genuine correctness risk for the central claim.","tokens_in":19119,"tokens_out":6835,"duration_ms":73065,"concrete_test":"On held-out MrQA samples with multi-token gold answers, recompute the LR detector's AUC at decoding steps t=1, 2, 3, and 4 using the same feature definitions and labeling gold-span context tokens as positive. If AUC at t>1 falls materially below the ~0.99 reported for t=1, the per-step application in Eq. 6 is unsupported; additionally report DAGCD's EM gain split by single-token versus multi-token answers to see whether the gains concentrate on the first-token regime.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the transfer of the Section 3.2 detector from its training condition to the per-token, per-step use in Eq. 6. The detector is trained on features extracted at the first generated token after context concatenation, with positive labels defined as context tokens that match the gold answer (Section 3.2). DAGCD then applies this same detector at every decoding step to identify 'utilized' context tokens and builds U_top from its outputs (Section 4.1). Nothing in Sections 2-4 or the appendices measures detector accuracy as a function of decoding step, nor for non-answer tokens that support multi-hop reasoning. If the learned signal is only 'gold answer word at position 1,' then for multi-token answers—common in SQuAD, NQ, and HotpotQA—later steps are out-of-distribution for the detector, and the reported 17.67% EM gains cannot be attributed to the stated mechanism. The empirical case therefore rests on an unverified generalization. A related weakness is that Eq. 6 only boosts context tokens already in the top-R of P; the paper never ablates the detector against a no-detector copy baseline (e.g., U_top built from raw attention ratios), leaving open that the detector is unnecessary. The first issue is the more fundamental, because it targets the mechanism that is supposed to make the method work.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynamic Attention-Guided Context Decoding (DAGCD), a single-pass decoding method aimed at reducing context faithfulness hallucinations in open-book QA. The authors first present analyses linking token-level uncertainty to unfaithful answers, then train a logistic-regression probing classifier on 'attention ratio' features to identify context tokens that are 'utilized,' where positive labels are context tokens matching the gold answer. At inference, DAGCD applies this detector at every decoding step to build a utilization distribution, restricts it to top-ranked tokens, and adds an entropy-scaled version of this distribution to the next-token probabilities (Eq. 6). Experiments across seven QA datasets and six LLAMA/Mistral models report consistent EM/F1 gains over greedy decoding and over CAD and COIECD baselines, with headline improvements of 17.67% EM on pretrained models and 2.25% on instruction-tuned models.","tokens_in":19464,"tokens_out":2997,"duration_ms":31176,"significance":"If the stated mechanism were established, DAGCD would be a useful lightweight contribution: it requires no second decoding pass, uses a small logistic-regression detector that transfers across models and datasets with reported AUC above 0.99, and is accompanied by ablations over training size, top-rank constraint, and scaling factor. The reproducible recipe (attention-ratio features, top-K head selection, entropy-scaled additive adjustment) is concrete and falsifiable. However, the significance is currently limited by a gap between the probing setup and the inference-time use of the detector: the classifier is trained only on first-token gold-answer-matching labels, yet it is applied at every decoding step to all context tokens. Until per-step and non-answer-token behavior is validated, the reported gains cannot be confidently attributed to the 'context utilization signal' that the paper claims to exploit. The paper is honest about alpha calibration difficulty and classifier robustness in its Limitations section, but those admitted limitations interact with the core evaluation and need to be addressed by additional experiments.","major_comments":[{"comment":"The detector is trained under conditions that differ from its inference-time use. Section 3.2 constructs positive labels as context tokens that match the gold answer, and the feature vectors are extracted at the first generated token after context concatenation. Section 4.1 then applies the same detector at every decoding step to all context tokens and builds the utilization distribution used in Eq. (6). Nothing in Sections 2–5 or the appendices measures detector accuracy or agreement as a function of decoding step, nor for tokens that support multi-hop reasoning without being the answer string. For multi-token answers, which are common in SQuAD, NQ, and HotpotQA, later decoding steps are out of distribution for the detector. The central claim—that DAGCD amplifies actually utilized context tokens—requires per-step validation, for example by reporting detector precision/recall at decoding positions 1, 2, 3, ... and on non-answer context tokens that appear in correct multi-token generations.","section":"§3.2, §4.1, Eq. (6)"},{"comment":"The paper never ablates the detector itself. DAGCD's adjustment is P' = P + α·H_norm(P)·U_top, where U_top is built from detector-filtered utilization scores. A natural control is to replace the detector output with a no-classifier baseline, e.g., U computed directly from raw attention ratios without thresholding, or with a random labeling of context tokens. The ablations in Figure 5 vary detector training data size, top-rank constraint, and α, but none removes the detector. Given that the positive training labels are gold-answer tokens, part of the EM gain could arise simply from amplifying any high-attention, top-ranked context token; without the no-detector control, the load-bearing claim that the learned classifier is necessary is unsupported.","section":"§4.2, §5.3"},{"comment":"The reported hyperparameter configuration is internally inconsistent. Section 5.1 states that α is set to 2 for pretrained models and 4 for instruction-tuned models. Appendix E.2 and Figure 13 report that on HotpotQA the optimal α for Mistral-7B is 5, while Mistral-7B-Instruct stabilizes only at α = 13. The abstract and Section 1 summarize gains as aggregate EM improvements of 17.67% and 2.25%, but it is unclear whether Table 1 uses the fixed values from Section 5.1 or per-model optimal values from the appendix. This distinction matters because selecting α per model from the test set would affect the fairness of the comparison to baselines. The authors should state explicitly which α values produced Table 1, and should present sensitivity results for all examined models, not only LLaMA2 and Mistral on HotpotQA.","section":"§5.1, Appendix E.2"},{"comment":"The interpretation of the probing result as evidence of a 'context utilization signal' is stronger than the labels support. Positive examples are defined as context tokens that match the gold answer, so the classifier learns to recognize answer-word-like tokens under open-book conditions, not to detect all tokens that contribute to reasoning or to faithful multi-token generation. The high AUC (>0.99) and cross-dataset generalization are consistent with this narrower reading. The paper should either soften the 'fundamental mechanism' claim (Section 3.3 and Conclusion) or add a probing evaluation with richer utilization labels, for example tokens that are copied into a correct answer span at the step they are generated, or tokens whose removal changes the model's prediction.","section":"§3.2, §4.1"}],"minor_comments":[{"comment":"The row label 'OURs' is a typo; it should read 'DAGCD' or 'Ours' consistently.","section":"§5.2, Table 1"},{"comment":"References for Meng et al. 2022a and 2022b are identical in content, as are the two Olsson et al. 2022 entries; these duplicate entries should be merged or disambiguated.","section":"References"},{"comment":"The adjusted distribution P' is not renormalized. Since the paper reports greedy decoding, argmax is unaffected by the missing normalization, but the method description and any sampling-based use would require renormalization; this should be stated explicitly.","section":"§4.3, Eq. (6)"},{"comment":"The prompt contains a space before the colon in 'information: {context}', which is likely intentional but should be checked for consistency with the prompt templates in Appendix F.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper crosses a fine line between 'probing analysis' and 'supervised learning on gold-answer tokens' when it claims that attention ratios encode a fundamental utilization signal. As a reviewer I would ask for the no-detector ablation and per-step detector diagnostics before endorsing the mechanism. The alpha inconsistency in the appendix versus the main text also needs to be resolved; it currently leaves room for test-set hyperparameter selection, which would weaken the comparison against CAD and COIECD. I do not see this as a reject: the method is simple, the baselines are reproduced, and the gains are large enough to merit further study. The revision should focus on closing the training-inference gap and clarifying the exact experimental configuration behind the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a usable, well-scoped decoding intervention with a genuinely nice probing result on attention ratios, but the central mechanism is under-validated at inference time and the hyperparameter story is messy. It deserves a serious referee round, not a desk reject.\n\nWhat's actually new: the attention-ratio feature, the LR detector that achieves >0.99 AUC across domains, and the single-pass entropy-scaled amplification. The cross-domain and cross-prompt probing is the strongest part of the paper; the result that 100 training samples suffice is interesting and should be reproducible. The method is lightweight, the code is provided, and comparing against CAD and COIECD under a unified prompt is the right baseline setup.\n\nWhere it gets soft. First, the detector's training data. The paper labels context tokens as 'utilized' when they match the gold answer, and the training samples are flipped-answer cases. It is not explicit about which decoding step the attention features come from; the closest description is the first-token analysis in Section 2.2. If the detector is trained only on first-token features and then applied at every decoding step, the transfer is unvalidated. The paper needs per-step detector accuracy or at least a mechanism analysis for multi-token answers. This is the main technical gap.\n\nSecond, the alpha inconsistency: the main text says alpha=2 for pretrained models and alpha=4 for instruction-tuned models, but Appendix E.2 reports optimal alpha=5 for Mistral-7B and alpha=13 for Mistral-7B-Instruct. That means the main-table numbers for Mistral were likely run at a suboptimal alpha. It does not kill the results, but it weakens the 'one setting per model type' claim and needs reconciliation.\n\nThird, there is no ablation against a pure attention-ratio copy baseline without the detector. Given the detector is trained on gold-answer labels, a no-detector version would clarify how much of the gain is just 'boost high-attention context words.' The circularity concern is not fatal—this is supervised learning, not circular derivation—but the 'fundamental utilization signal' language oversells what is at least partly an answer-word detector.\n\nThe limitations section is honest, and the citation pattern looks fine. Overall, a capable group with a plausible method and a real probing contribution, but the load-bearing generalization from probing to per-step decoding is not demonstrated. A serious referee should ask for the per-step analysis and the alpha reconciliation; with those, this could be a solid paper.\n\nRecommendation: send to peer review.","headline":"A solid, well-scoped decoding paper whose probing result is the strongest part; referee should push on per-step detector validity and alpha consistency.","tokens_in":19919,"tokens_out":3343,"would_cite":true,"duration_ms":31574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Single-pass decoding boosts context-faithful QA answers by 17.7%","keywords":["context faithfulness hallucination","attention ratio","context utilization signal","open-book question answering","retrieval-augmented generation","entropy-guided decoding","hallucination mitigation","single-pass decoding"],"falsifier":"Retrain the probing detector using labels taken from attention maps at the second or later generated tokens, or labeling non-answer support tokens as utilized. The paper's claim predicts AUC comparable to the reported >0.99; if the classifier instead falls toward chance, the signal is specific to first-token answer-word recognition and DAGCD's amplification has no demonstrated mechanism beyond the first generated token.","tokens_in":18957,"feed_emoji":"🎯","tokens_out":13210,"duration_ms":106267,"temperature":0.7,"pith_summary":"Large language models given a retrieved passage often answer from parametric knowledge instead of the passage, producing what the paper calls context faithfulness hallucinations. This paper argues that the model's own attention weights reveal which context tokens it is actually using, and that token-level uncertainty marks where it does not use them confidently. On this basis it proposes DAGCD, a single-pass decoding adjustment that adds an entropy-scaled boost to the probability of context tokens judged 'utilized' by a logistic-regression classifier over attention ratios. The paper reports that DAGCD outperforms greedy decoding across open-book QA datasets, improving exact match by 17.67% on pretrained models and 2.25% on instruction-tuned models, while keeping the computational complexity of greedy decoding.","feed_headline":"Single-pass decoding boosts context-faithful QA by 17.7%","feed_subtitle":"An attention-ratio detector amplifies context-grounded tokens in one pass, beating greedy and contrastive baselines.","key_machinery":"The load-bearing object is the attention ratio, defined for a context token $j$ in head $h$ of layer $l$ as $r^j_{l,h} = a^j_{l,h} / \\sum_{j \\in C} a^j_{l,h}$, the share of attention that token receives among all context tokens, which normalizes away attention-sink noise and cross-head magnitude differences. Aggregating these ratios across heads gives a feature vector per token; a logistic-regression classifier trained on those vectors, with ground-truth 'utilized' meaning the token matches the gold answer, becomes the Context Utilization Detector. At inference the classifier's learned coefficients weight the ratios to form utilization scores $s_j$, normalized into a utilization distribution $U$; a top-rank constraint keeps only tokens already ranked in the top-$R$ of the generation distribution, and the final adjustment $P' = P + \\alpha H_{\\mathrm{norm}}(P) \\cdot U_{\\mathrm{top}}$ amplifies those tokens in proportion to the current token-level normalized entropy. This mechanism carries the argument: it converts the claimed attention-ratio signal into a probability-mass shift within one decoding pass, using only top-10 attention heads and a 100-sample training set.","core_discovery":"The paper's central claim is that context faithfulness hallucinations in retrieval-augmented generation are not caused by the model ignoring the supplied context, but by its failing to prioritize context tokens it has already identified as relevant. The supporting observations are that wrong answers show higher uncertainty (average normalized entropy 0.36 vs 0.29 for correct answers, average maximum softmax probability 0.25 vs 0.41), and that in 66% of wrong cases the gold-answer token is ranked within the top 10 of the token-level distribution. A logistic-regression probe trained on per-head attention ratios classifies 'utilized' context tokens with AUC above 0.99 across six domains, even with only 100 training samples. DAGCD operationalizes this signal: at each decoding step it detects utilized context tokens from attention ratios, builds a utilization distribution over them, and adjusts the generation distribution as $P' = P + \\alpha H_{\\mathrm{norm}}(P) \\cdot U_{\\mathrm{top}}$, where $H_{\\mathrm{norm}}(P)$ is the normalized entropy and $\\alpha$ is a per-model constant. The paper reports that this adjustment outperforms greedy decoding, CAD, and COIECD on seven open-book QA datasets, with the largest gains on multi-hop reasoning and adversarial-swap settings.","pith_inferences":["If the attention-ratio signal is genuinely a utilization signal, the same classifier output should provide a per-token interpretability map of which retrieved sentences influenced a generation; that map could be tested as a diagnostic for RAG failures in long-context and tool-use settings.","The detector is trained on gold-answer-matching tokens at the first generated position; the claim that it detects utilization at every later step is an extrapolation that the paper does not test. Relabeling training data with tokens from later decoding positions, or with non-answer support tokens, would settle whether the signal is utilization or answer-word recognition.","The paper's own limitations flag that the scaling factor $\\alpha$ needs per-model calibration; a natural extension is to make $\\alpha$ self-tuning by deriving it from a running estimate of normalized entropy, which would remove the manual tuning step."],"forward_implications":["Because DAGCD reuses attention already computed during greedy decoding and needs no second forward pass, it maintains the theoretical time complexity of greedy decoding while reporting average EM gains of 17.67% on pretrained models and 2.25% on instruction-tuned models.","The largest reported gains are in settings where retrieved evidence is hardest to prioritize: multi-hop HotpotQA (18.80% EM on Mistral-7B) and adversarial NQ-swap (74.52% EM on Mistral-7B), suggesting the method corrects failures of evidence prioritization rather than generic decoding artifacts.","The detector transfers across domains with AUC above 0.99 when trained on 100 samples from HotpotQA, so applying DAGCD to a new open-book QA setup requires only a tiny calibration set rather than dataset-specific detector training.","On instruction-tuned models the absolute gains shrink but DAGCD still leads all baselines, implying fine-tuning already compresses much of the uncertainty signal the method exploits."],"supporting_citations":[{"why":"Supplies the Maximum Softmax Probability metric used to measure token-level confidence in the uncertainty analysis.","marker":"Hendrycks and Gimpel, 2017"},{"why":"Supplies the Normalized Entropy metric used to quantify token-level uncertainty.","marker":"Huang et al., 2023b"},{"why":"Establishes the relation between token-level uncertainty and hallucination that motivates the entropy-scaled boost.","marker":"Chuang et al., 2024b"},{"why":"CAD, the contrastive baseline that DAGCD must outperform and whose multi-pass overhead the method claims to remove.","marker":"Shi et al., 2024b"},{"why":"COIECD, the entropy-constrained baseline compared against; DAGCD extends its entropy motivation to a single pass.","marker":"Yuan et al., 2024"},{"why":"The copy-generator framework that motivates using attention weights to amplify context tokens during generation.","marker":"See et al., 2017"},{"why":"MrQA, the unified source of the six open-book QA datasets used for training the detector and evaluation.","marker":"Fisch et al., 2019"},{"why":"Provides the data-construction criterion for labeling: cases where the answer flips from incorrect to correct after context concatenation.","marker":"Meng et al., 2022b"},{"why":"NQ-Swap, the entity-swap dataset used to test robustness when context is deliberately conflicting.","marker":"Longpre et al., 2021"}],"fun_headline_variants":["Attention-guided decoding cuts context hallucinations by 17.7%","Uncertainty + attention: 17.7% more faithful LLM decoding","Single-pass attention detector: 17.7% fewer context hallucinations","Attention ratios reveal context use: decoding fix +17.7%","One-pass decoding with attention: 17.7% more context faithfulness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, which the paper's limitations section flags, is that a detector trained to label 'utilized' context tokens as those matching the gold answer at the first generated token continues to identify the context tokens that should be amplified at every later decoding step, including multi-token answers.","fun_headline_variants_meta":{"raw":{"variants":["Attention-guided decoding cuts context hallucinations by 17.7%","Uncertainty + attention: 17.7% more faithful LLM decoding","Single-pass attention detector: 17.7% fewer context hallucinations","Attention ratios reveal context use: decoding fix +17.7%","One-pass decoding with attention: 17.7% more context faithfulness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001812,"raw_usage":{"total_tokens":7114,"prompt_tokens":910,"completion_tokens":6204,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":6110}},"tokens_in":526,"tokens_out":6204,"duration_ms":43878,"temperature":1.0,"reasoning_tokens":6110,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:35:55.830248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the probing detector using labels taken from attention maps at the second or later generated tokens, or labeling non-answer support tokens as utilized. The paper's claim predicts AUC comparable to the reported >0.99; if the classifier instead falls toward chance, the signal is specific to first-token answer-word recognition and DAGCD's amplification has no demonstrated mechanism beyond the first generated token.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Maximum Softmax Probability metric used to measure token-level confidence in the uncertainty analysis."}],"review_version":1}