{"id":"83e7ffda-0928-4485-b960-89ba225a201f","arxiv_id":"2608.01930","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Evidence-bearing spans in a prior chain of thought act as a textual shortcut that competes with visual recomputation, and a training-free attention firewall that blocks this stale text improves VLM self-reflection.","lead":"This paper shows that when a vision-language model is asked to re-check an image after the image changed, it often reuses the evidence copied in its previous chain of thought instead of recomputing from the new image. The authors name this reuse a 'textual shortcut', diagnose which part of the old reasoning carries the effect, and introduce a training-free attention mask (FSAF) that blocks the old reasoning and improves visual updating.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evidence-span taxonomy (App. C.3) is broad enough that E–C may compare 'remove reasoning' vs 'remove filler'; content-specificity needs human-annotated span validation.","rationale":"The reader's weakest assumption matches the most load-bearing concern I can identify. The content-specificity claim—the paper's main conceptual contribution—is operationally defined by the E- rules, and those rules are broad enough that the E–C comparison may reduce to 'remove reasoning vs remove filler.' This is not an internal inconsistency, and it does not invalidate the broader finding that prior CoTs exert strong behavioral control: the prior-only PAR, matched support-withdrawal, neutral-history continuation, and FSAF results are supported by multiple independent design choices. But Tab. 2, Fig. 3, and Tab. 3 all depend on E-, so the strongest claim is not yet separated from the span heuristic. A human-annotation replication of the primary E–C contrast is the direct, concrete check that would settle this. I therefore keep the reader's CONDITIONAL verdict unchanged: the concern is a condition to test, not a demonstrated failure.","tokens_in":34452,"tokens_out":6224,"duration_ms":77849,"concrete_test":"Sample 150–200 prior CoTs from the 4,159-pair cohort. Have at least two annotators, blind to the E- rules, independently mark task-relevant evidence spans (observations, quantities, relations, deductions). Measure sentence-level agreement with E-; then recompute the primary E–C contrast (Tab. 2) using the human spans. If model-equal ΔVUR/ΔPAR and margin shifts replicate within ~5 pp / 0.5 nats, the taxonomy is not the driver. If the effect collapses or reverses, the content-specificity claim is an artifact of the rule scheme.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that evidence-bearing content, not generic context, carries prior-CoT control—rests entirely on the E- span definition in Appendix C.3. The rules mark a sentence as evidence-bearing if it contains a number/symbol, an option letter, a visual-lexicon term, or a reasoning connective such as 'because/so/answer/value'. On VS-Bench math CoTs this is nearly coextensive with 'all substantive sentences'; the leftover non-evidence context is largely meta-commentary. The length-matched C control therefore removes filler, while E removes the actual reasoning. The large E–C effects in Tab. 2 and Fig. 3 may thus demonstrate that removing task-relevant reasoning weakens prior-answer control, not that a specifically 'evidence-bearing' textual category is the carrier. Tab. 3's order-disruption result has the same dependence on E-. The paper explicitly calls the rules a high-recall operational decomposition and disclaims uniqueness (App. C.3), but provides no validation against human causal annotation or an alternative span definition. This does not threaten the existence of prior-CoT influence—the prior-only, support-withdrawal, and neutral-history controls are independent—but it does undercut the strongest content-specificity conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates why vision-language models (VLMs) fail to revise their answers when the image changes, and what they reuse instead of recomputing. Using 800 matched image pairs from VS-Bench, the authors build a counterfactual protocol in which the model first produces a coherent incorrect prior CoT from a counterfactual image, then reflects under the current image. Across 16 VLMs they report that removing evidence-bearing spans (E-) from the prior CoT shifts answer preference more than length-matched non-evidence removal or final-answer removal, that reordering evidence weakens prior control, and that a correct reflected answer does not retire the shortcut: under support withdrawal, recovered responses relapse more than direct controls. They propose FSAF, a training-free attention-mask intervention that blocks fresh computation from attending to the prior CoT, and report improvements in visual update rate and reduction of prior-answer rate across five Qwen VLMs.","tokens_in":34780,"tokens_out":5903,"duration_ms":63154,"significance":"If the findings hold, the paper makes a solid empirical contribution: it identifies a specific textual carrier of prior-state influence, shows that answer correctness is not a certificate of evidence-state revision, and provides a simple, effective mitigation. Strengths include the matched-pair design, the breadth of models, the dose-response analysis, and the unusually complete protocol documentation with a claim–evidence index, exact prompts, and checkpoint IDs. The main risk is that the 'evidence-bearing' span definition (App. C.3) is a broad lexical proxy for substantive reasoning; without validation against human annotations or an alternative span operationalization, the content-specificity conclusion is not fully established. The paper's broader diagnostic results—prior-CoT influence, residual dependence after recovery, and the FSAF effect—do not rest on that taxonomy and are well supported.","major_comments":[{"comment":"The operational definition of E- is a fixed lexical rule list (numbers/symbols, option letters, visual lexicon, reasoning connectives). On VS-Bench mathematical CoTs this is nearly coextensive with 'all substantive sentences'; the C control removes the leftover meta-commentary. The large E–C effects therefore demonstrate that removing task-relevant reasoning weakens prior-answer control, but not specifically that the 'evidence-bearing' category is the causal carrier. Since Tab. 2 and Fig. 3 are the main content-specificity evidence, please validate the E- decomposition against human causal annotations or a second operationalization (e.g., parse-based factual claims vs. discourse markers), and report the agreement. This is load-bearing for the paper's central characterization.","section":"Appendix C.3, Tab. 2, Fig. 3"},{"comment":"The order-disruption result (D vs NE) also depends entirely on the same E- taxonomy. If the taxonomy over-captures reasoning content, the D–NE contrast is not a clean test of 'evidence organization' but rather of reordering meaningful vs. filler sentences. The authors should provide a human-annotated subset or a second independent span definition and check whether the D–NE effect is robust to the choice of decomposition.","section":"Tab. 3, App. C.3"},{"comment":"The headline FSAF improvements are reported without confidence intervals, and the rule-based evaluator and the 235B judge disagree on 26.42% of VUR/PAR labels in the FSAF evaluation. Please report paired bootstrap CIs for VUR/PAR/m, and ideally validate the semantic judge on a human-annotated sample. The magnitude of the reported effect makes it likely that the main conclusion stands, but the precision of the estimate and the judge's reliability are currently unquantified.","section":"Tab. 7, App. F.5"}],"minor_comments":[{"comment":"The phrase 'weakening current-image support' suggests a visual manipulation, but the support-withdrawal condition deletes textual evidence-bearing spans while keeping the image. Please clarify in the abstract/main text.","section":"Abstract and App. B.4"},{"comment":"The ranking rule for dose-response weights option cues by 2x; the rationale for this weighting is not given. Please justify or show that results are insensitive to the ranking.","section":"App. C.3"},{"comment":"The 1,800-pair FSAF cohort is unbalanced across models (267–486 per model); consider reporting per-model CIs and the pooled estimate with model as a random effect.","section":"Tab. 7"},{"comment":"The eligibility filter (direct correctness on both images) is necessary but reduces the pool to 4,159/12,800 model-pairs; a brief discussion of how this selection might shape the conclusions would be helpful.","section":"App. H.1"},{"comment":"The semantic judge is itself a Qwen-family VLM; a human-annotated sanity check on a random subset would strengthen confidence in the VUR/PAR labels.","section":"A.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-organized and unusually transparent about protocols and limitations. The main risk is the unvalidated E- taxonomy for the content-specificity claim; this is fixable within a revision by adding human annotation or an alternative span definition. I would not reject, but the central characterization needs this support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. Worth your time. The core finding is that when a VLM fails to recompute from a changed image, the prior chain of thought often acts as a textual shortcut: the model reuses old reasoning instead of deriving from the current image. The matched counterfactual setup on VS-Bench, run across 16 models, is careful and convincing. Removing the prior reasoning from context shifts answers toward the current image far more than removing the final-answer span or length-matched filler. Order matters too: reversing the evidence-bearing sentences weakens the shortcut. They also show that a correct answer does not retire the shortcut: after recovery, withdrawing current-image support makes the prior answer resurface much more on history-conditioned paths than on direct ones. That is a clean, nontrivial result.\n\nThe mitigation, FSAF, is a simple attention mask blocking a fresh re-query and subsequent generation from attending to the prior assistant trace. It is training-free, practical, and consistent across five Qwen models, raising VUR by about 18 points and cutting prior-answer rate from 39% to under 4%. I could see this being used beyond the benchmark.\n\nThe main soft spot is the operational definition of 'evidence-bearing' in Appendix C.3. The rules mark sentences containing numbers, option letters, visual terms, or reasoning connectives. On math-like CoTs, that captures almost every substantive sentence, leaving the length-matched control to remove mostly meta-commentary. So the headline E–C contrast may simply show that removing real reasoning matters more than removing filler, not that a special content category is the carrier. The paper disclaims uniqueness in the appendix, but the abstract and title overstate the specificity. This does not sink the paper: the prior-CoT influence, order effect, retirement failure, and FSAF result all rest on comparisons that do not depend on this taxonomy. Still, the content-specificity claim should be softened or backed with human span annotations.\n\nSmaller issues: Table 7 has no confidence intervals (the ablation table does), FSAF is tested only on one model family, and the semantic judge disagrees with the rule evaluator on 26% of labels—though they sensibly use the stronger judge for main claims.\n\nOverall: a serious piece of work. The central phenomenon is real, the controlled comparisons are careful, and the intervention is convincing. The evidence-taxonomy caveat is worth addressing but does not change the main conclusion. If you work on VLM evaluation or self-correction, read it. I would send it to peer review.","headline":"Solid, careful VLM self-reflection diagnosis with a clean training-free fix; the 'evidence-bearing' label is broader than it looks, but the core shortcut and FSAF results hold.","tokens_in":35240,"tokens_out":3417,"would_cite":true,"duration_ms":34571,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model that fails to recompute after an image changes is often reusing evidence-bearing reasoning from its prior chain of thought as a textual shortcut, and this shortcut can be suppressed by a training-free attention mask.","keywords":["vision-language models","self-reflection","chain-of-thought","textual shortcut","attention mask","visual recomputation","counterfactual evaluation","FSAF"],"falsifier":"Run the same removal and reordering experiments with human-annotated evidence spans instead of the rule-based taxonomy: if length-matched random or non-evidence removal produces the same VUR/PAR shifts as evidence removal, the content-specificity claim fails. Alternatively, apply FSAF while replacing the prior CoT with a length-matched neutral history of the same role structure: if the VUR gain persists despite no stale evidence being present, the effect is not specifically about stale evidence access.","tokens_in":34374,"feed_emoji":"👁️","tokens_out":4935,"duration_ms":46327,"temperature":0.7,"pith_summary":"When a vision-language model is asked to re-examine an image after the visual evidence has changed, it often fails to recompute the answer from the current image. This paper argues that the reason is a textual shortcut: the model's own prior chain of thought, written when the old image was present, remains in context and its evidence-bearing sentences (numbers, option letters, visual and reasoning terms) steer the new answer. Controlled removal shows this effect is content-specific: deleting those spans changes answer preference and free-generation outcomes more than deleting length-matched non-evidence text or the final-answer span, and the control weakens gradually as more stale evidence is removed. Correcting the answer does not retire the shortcut; a matched stress test shows the prior answer resurges when current support is withdrawn. The paper introduces Fresh-State Attention Firewall (FSAF), a training-free mask that prevents the fresh re-query from attending to the old chain of thought, raising the visual-update rate from 35.28% to 53.61% and lowering the prior-answer rate from 39.22% to 3.67% across five Qwen VLMs.","feed_headline":"Blocking stale reasoning lifts VLM visual updates from 35% to 54%","feed_subtitle":"A training-free attention mask stops fresh answers from reading the prior chain of thought, cutting stale-answer reuse to under 4%.","key_machinery":"The central machinery is a matched counterfactual comparison: a question paired with two visually similar images that imply different answers, with the model first producing a coherent prior CoT on the counterfactual image. A hand-written, outcome-independent taxonomy (Appendix C.3) marks sentences as evidence-bearing if they contain numbers, option letters, visual-lexicon terms, or mathematical/reasoning terms; removal and reordering of these spans, plus the answer-preference margin m(S) = log p(y|S) − log p(y⁻|S) and the VUR/PAR semantic outcomes, isolate what carries prior control. The mitigation is FSAF, an additive attention mask applied at every language layer that sets logits to −∞ fo","core_discovery":"Evidence-bearing reasoning in a prior chain of thought is not inert context; it is an organized, reusable path to an answer that competes behaviorally with recomputation from the current image. The paper establishes this by a matched counterfactual setup in which the model first answers a paired image correctly, then reflects under the changed image with its old chain of thought in context. Removing or reordering the evidence-bearing sentences of that chain shifts the model's preference and its generated answer back toward the current image, while length-matched non-evidence removal and final-answer removal do not. A correct reflected answer does not certify that the shortcut is gone: withdr","pith_inferences":["If the taxonomy of 'evidence-bearing' content is what carries the effect, a similar rule-based decomposition could be used to build automated detectors for when a model is relying on stale text rather than the current stimulus.","The finding that reordering evidence weakens control suggests the shortcut is not just a bag of cues but an ordered derivation; a testable extension is whether inserting contradictions or uncertainty markers in the prior CoT would disrupt it more than deletion.","The reported 26.42% disagreement between the rule evaluator and the 235B semantic judge implies the effect sizes may be judge-sensitive; an editorial extension would be to re-run the FSAF evaluation with a second independent judge to bound that sensitivity.","The idea that fresh computation must be protected from stale textual reuse could transfer to other domains, such as code editing or tool-use agents, where a prior reasoned plan lingers in context."],"forward_implications":["If the claim is right, VLM self-reflection systems should treat the prior reasoning trace as a state to be quarantined, not merely as context to be prompted over.","A correct answer following reflection is not evidence that the model has revised its evidence state; evaluation protocols should stress-test recovered answers before trusting them.","The shortcut's strength depends on how much stale evidence remains and how it is organized, implying that summarising or compressing old reasoning may be less effective than blocking it.","Training-free intervention can yield large gains on updating behavior without retraining, suggesting that accessibility control is a practical lever for reliability.","Extensions of the same protocol could audit other forms of persistent context for shortcut behaviour."],"fun_headline_variants":["Attention firewall boosts VLM visual updates from 35% to 54%","Fresh-state mask raises VLM visual updates, slashes stale answer reuse","VLM self-reflection improved by blocking stale reasoning in prior CoT","Training-free firewall lifts VLM visual updates to 54%, cuts stale answers"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the hand-written rule taxonomy for 'evidence-bearing' sentences (numbers, option letters, visual and reasoning terms) correctly identifies the spans whose presence causes prior-answer control; if those rules misclassify the causal spans, the content-specificity result could be an artifact of the rule scheme rather than a property of the models.","fun_headline_variants_meta":{"raw":{"variants":["Attention firewall boosts VLM visual updates from 35% to 54%","Fresh-state mask raises VLM visual updates, slashes stale answer reuse","VLM self-reflection improved by blocking stale reasoning in prior CoT","Training-free firewall lifts VLM visual updates to 54%, cuts stale answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2584,"prompt_tokens":807,"completion_tokens":1777,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1697}},"tokens_in":551,"tokens_out":1777,"duration_ms":13648,"temperature":1.0,"reasoning_tokens":1697,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:04:59.903407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same removal and reordering experiments with human-annotated evidence spans instead of the rule-based taxonomy: if length-matched random or non-evidence removal produces the same VUR/PAR shifts as evidence removal, the content-specificity claim fails. Alternatively, apply FSAF while replacing the prior CoT with a length-matched neutral history of the same role structure: if the VUR gain persists despite no stale evidence being present, the effect is not specifically about stale evidence access.","supporting_citations":[],"review_version":1}