{"id":"c9be4278-b7ec-49b5-b951-e406fa4880fb","arxiv_id":"2412.18404","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A modified gradient-attribution method on frozen CLIP, using negative attributions, localizes misaligned words in image-text pairs and yields a fine-grained alignment score.","lead":"The paper shows that by stripping the ReLU step from a standard attention-explanation method, the negative gradients inside CLIP can point to words in a caption that do not match the image. The authors package this into a detector called CLIP4DM plus a score F-CLIPScore, and show it beats other zero-shot models on several misalignment benchmarks while running much faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed threshold epsilon is not calibrated across CLIPScore regimes; paper's own low-score analysis and Rich-HF epsilon switch undermine the universal decision rule.","rationale":"After reading the paper in good faith, the strongest claim is that removing ReLU from GAE lets negative token attributions flag misaligned words and that a single threshold epsilon yields SOTA zero-shot dense misalignment detection. The paper is honest and well evaluated; the mechanism is plausible and the code is public. However, the load-bearing step is Eq. (10), and evidence that this step is universally valid is missing. The paper itself shows the threshold is not stable: Rich-HF requires a different epsilon, and the F-CLIPScore analysis in Appendix D demonstrates the low-CLIPScore failure mode, where gradients spread and few tokens cross epsilon, making obviously misaligned pairs look aligned. Figure 13a confirms the attribution signal is weakest exactly in the low-alignment groups. This is not merely an outside-consensus issue; it is an internal inconsistency between the universal decision rule and the documented behavior. The concrete test of cross-benchmark threshold transfer and per-score-bin AUC would settle whether the signal is calibrated. Since the reader already assigned CONDITIONAL, this concern does not move the verdict: the paper should be accepted conditionally, with the claims narrowed to high-CLIPScore regimes and a clearly specified threshold-selection protocol.","tokens_in":22430,"tokens_out":4982,"duration_ms":47631,"concrete_test":"Recompute FOIL and nocaps-FOIL localization using the Rich-HF-tuned epsilon (-0.00001) instead of the FOIL-tuned -0.00005, and report LA/AP; if LA drops materially, the headline numbers are threshold-dependent. Independently, split nocaps-FOIL by CLIPScore quartiles and compute the AUC of word-level attributions for true vs. false words within each quartile; if AUC varies substantially across quartiles, a single fixed epsilon cannot be the right decision rule, and the paper should either normalize attributions or use a score-dependent threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central decision rule Eq. (10) mis(w_j)=1 if w_j < epsilon assumes a single threshold separates misaligned from aligned words across all inputs. The paper provides no normalization or score-dependent calibration, and its own evidence shows the assumption fails. In Rich-HF (Table 5), the best F1 requires epsilon=-0.00001, while the default for FOIL/nocaps-FOIL is -0.00005; changing epsilon in that range swings Rich-HF F1 from 0.427 to 0.314. More directly, Appendix D (Figure 15 and surrounding text) states that when CLIPScore is extremely low, gradients distribute across tokens so few fall below epsilon, causing F-CLIPScore to assign high alignment to clearly misaligned captions (e.g., 'A car an two men...'). Figure 13a shows attribution and F-CLIPScore correlations are worst precisely in the low ground-truth alignment group, which is the regime a misalignment detector must handle. Thus the negative-attribution signal is not a calibrated, scale-invariant indicator; its separation depends on the global score, text length, and dataset. The reported state-of-the-art LA numbers therefore rest on benchmark-specific threshold selection, not on a uniform signal, and the 'drop-in replacement' claim for F-CLIPScore is contradicted by the paper's own low-score analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLIP4DM, a zero-shot method for dense misalignment detection between images and text. The method computes per-token attribution scores by removing the ReLU from the gradient-weighted attention aggregation of GAE (Eq. 8), so that negative attribution values can flow. Word-level attributions are thresholded (Eq. 10) to flag misaligned words, and F-CLIPScore (Eq. 11) aggregates these into a global alignment score. The authors evaluate on FOIL, nocaps-FOIL, HAT, SeeTRUE-Feedback, Rich-HF, MMVP, and SugarCrepe, reporting state-of-the-art zero-shot localization accuracy (e.g., 0.836 LA on FOIL with ViT-H/14) and competitive global metrics, while also providing ablations of the attribution formulation, layer choice, and F-CLIPScore components. The central claim is that negative gradients of individual text tokens in a frozen CLIP model indicate misalignment, and that thresholding these attributions yields a general-purpose detector.","tokens_in":22701,"tokens_out":3815,"duration_ms":35586,"significance":"If the central heuristic holds, the paper contributes a cheap, training-free dense misalignment detector that avoids the heavy inference cost of foundation-model pipelines and the annotation cost of fine-tuned approaches. The experimental breadth is a genuine strength: the method is evaluated on multiple benchmarks spanning natural and generated images/text, single and multiple misalignments, and object, attribute, relation, and action errors. The ablations isolate the contribution of removing ReLU, the choice of layers, and the composition of F-CLIPScore, and the code is publicly released. The qualitative analyses honestly document both strengths (entity-level objects, intangible objects) and limitations (backgrounds, small objects, adjectives). However, the significance is conditional: the universal decision rule in Eq. (10) depends on a threshold that is tuned per benchmark and fails in a regime the paper itself identifies as important.","major_comments":[{"comment":"The central decision rule mis(w_j)=1 if w_j < epsilon assumes a single threshold separates misaligned from aligned words across all inputs. Yet the paper uses different thresholds on different benchmarks: epsilon=-0.00005 for FOIL/nocaps-FOIL (Section Experiments) and epsilon=-0.00001 for Rich-HF (Tables 5 and 6). Table 5 shows that changing epsilon from -0.00001 to -0.00005 on Rich-HF swings F1 from 0.427 to 0.314, with recall dropping from 0.516 to 0.231. This demonstrates that the reported state-of-the-art numbers rest on benchmark-specific threshold selection, not on a scale-invariant signal. The paper should either derive a principled way to set epsilon (e.g., input-dependent normalization or a statistical criterion) or explicitly scope the claims to the tuned setting; without this, the 'uniform signal' interpretation is not supported.","section":"Token Aggregation and F-CLIPScore (Eq. 10), Table 5"},{"comment":"The paper's own analysis undermines the drop-in replacement claim for F-CLIPScore. In Appendix D, the authors state that when CLIPScore is extremely low, gradients distribute across tokens so that few fall below epsilon, causing F-CLIPScore to assign high alignment to clearly misaligned captions (e.g., 'A car an two men...' is ranked top 3% by F-CLIPScore). Figure 13a shows that F-CLIPScore's Pearson correlation with ground-truth alignment is worst in the [0.0, 0.2) group, which is precisely the regime a misalignment detector must handle. The manuscript suggests applying F-CLIPScore selectively to samples with typical CLIPScore values, which is a severe qualification of the general 'drop-in replacement' claim. The authors should quantify what fraction of real-world inputs fall in the failing regime and either fix the metric or clearly state the restricted applicability.","section":"Appendix D and Figure 13"},{"comment":"The paper's core premise—that negative entries of R_l^h carry a uniform semantic signal for misalignment—is an empirical heuristic without a mechanistic derivation. Table 8 shows that using both positive and negative gradients outperforms retaining only negative gradients, which weakens the sign-specific interpretation: if negative gradients alone are the misalignment signal, it is unclear why discarding positive gradients hurts performance. The authors should provide a diagnostic (e.g., distributions of attribution values for aligned vs. misaligned words across score bins, or a per-sample analysis) to show that the negative-gradient signal is not confounded by gradient scale or by interactions with positive gradients. Without such evidence, the claim that negative gradients 'indicate misalignment' is only supported by benchmark aggregates under tuned thresholds.","section":"Allowing Negative Gradient Flow (Eq. 8), Ablation Table 8"}],"minor_comments":[{"comment":"The section heading 'Comparsion with Baselines' is misspelled; it should be 'Comparison with Baselines'.","section":"Analysis (typo)"},{"comment":"The column labels 'Dense Misalign' and 'Global Misalign' are confusing: it is not immediately clear that they refer to the type of evaluation (dense detection vs. global score) rather than the number of misaligned words. Please clarify in the caption or use more descriptive names.","section":"Table 1 and Table 11"},{"comment":"The epsilon values are run into the model names (e.g., 'Ours (ViT-H/14)epsilon=-0.00001'); add a space or present epsilon as a separate column for readability.","section":"Table 5"},{"comment":"The example 'A car an two men standing in front of it' contains a grammatical error that should be corrected if it is quoted verbatim; otherwise please mark it as a transcription of the dataset example.","section":"Appendix D, Figure 15"},{"comment":"The related-work section would benefit from a brief discussion of whether the negative-gradient hypothesis has any precedent in gradient-based explanation methods beyond GAE, since the paper frames this as a novel interpretation.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with broad evaluation and a clear ablation structure, and the code release is a plus. The main concern is that the central decision rule relies on a threshold that is tuned per benchmark and fails in the low-CLIPScore regime, which the authors themselves document. I believe this is fixable within the manuscript's scope by adding a calibration or a principled threshold-selection method, and by tightening the claims to match the evidence. I would not reject on the basis of the heuristic being empirical, since the paper is honest about its limitations; but the current presentation overstates the universality of the signal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper does something genuinely new and mostly backs it up. Removing the ReLU from the GAE attribution rule and treating negative gradients as misalignment indicators is a simple trick that works surprisingly well on FOIL, nocaps-FOIL, and SeeTRUE-Feedback, and the method is fast, zero-shot, and built on a frozen CLIP. The ablations are honest and useful, and the code is public. This is a practical tool worth having.\n\nWhat is actually new: prior CLIP XAI work used positive-only relevance to find relevant image regions; this paper is the first to use negative attributions to localize misaligned words in text. The F-CLIPScore aggregation is also new, and it does improve global alignment correlation over plain CLIPScore on several benchmarks. The evaluation spans five dense-misalignment datasets plus MMVP and SugarCrepe, which is broader than most papers in this space. The efficiency argument is real: orders of magnitude faster than ALOHa and the VLM baselines.\n\nThe soft spots are mostly about calibration. The core signal is an empirical heuristic, not derived from CLIP's contrastive objective, and threshold epsilon is tuned per benchmark: -0.00005 for FOIL/nocaps-FOIL but -0.00001 on Rich-HF, with F1 swinging from 0.427 to 0.314 over that range. That matters. Also, the paper itself shows that when CLIPScore is extremely low, gradients spread across tokens and F-CLIPScore assigns high alignment to obviously bad captions (e.g., \"A car an two men...\"). The authors acknowledge this and suggest excluding low-score outliers, but the prose still calls F-CLIPScore a drop-in replacement for CLIPScore. That claim is too strong and should be narrowed.\n\nNone of this sinks the central localization claim. The negative-attribution signal clearly carries real information, and the paper is unusually candid about its failure modes: backgrounds, small objects, adjectives, and low-CLIPScore inputs. The per-benchmark threshold tuning is a legitimate concern for any production use, but it is not circularity and not a fatal flaw. A revised version that fixes the threshold selection protocol and qualifies the global-score claim would be solid.\n\nWho this is for: anyone building hallucination detectors, caption-filtering pipelines, or fine-grained reward models for VLMs. It deserves a serious referee; I would send it to review with the expectation of a conditional accept.","headline":"Removing ReLU from CLIP attribution exposes a usable misalignment signal; the paper's global-score claims outrun its calibration.","tokens_in":23233,"tokens_out":1599,"would_cite":true,"duration_ms":16331,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Removing the ReLU from CLIP's attention attributions makes negative gradients expose misaligned caption words, giving state-of-the-art zero-shot dense misalignment detection and a CLIPScore replacement that better tracks human alignment…","keywords":["dense misalignment detection","CLIP interpretability","gradient attribution","negative relevance","F-CLIPScore","zero-shot evaluation","text-to-image alignment","object hallucination"],"falsifier":"Take a held-out set of captions where each misaligned word is known, and plot the distribution of the word-level attribution $w_j$ for misaligned versus aligned words per layer group; if the two distributions do not separate below a fixed epsilon consistently across domains, or if the optimal epsilon varies by more than the reported search range when tuned per dataset, the central heuristic is falsified.","tokens_in":22240,"feed_emoji":"🔎","tokens_out":5966,"duration_ms":48987,"temperature":0.7,"pith_summary":"This paper tries to establish that a frozen CLIP model, without any training or extra modules, can detect word-level mismatches between an image and its caption: after removing the ReLU activation from the standard GAE attribution rule, a text token whose attribution falls below a small negative threshold reliably marks a word that does not match the image. The authors show this across five benchmarks spanning natural and generated captions, objects, attributes, and relations, reporting state-of-the-art zero-shot localization accuracy and an F-CLIPScore aggregate that beats plain CLIPScore for detecting global mismatches. The motivation is practical: prior dense misalignment detectors rely on large vision-language models or human-annotated fine-tuning, while this method needs only one frozen CLIP forward-grad-backward pass, running tens of times faster than foundation-model pipelines.","feed_headline":"Frozen CLIP flags misaligned words via negative gradients","feed_subtitle":"Removing one ReLU turns CLIP gradients into zero-shot word-level misalignment detection.","key_machinery":"The load-bearing object is the modified GAE attribution rule $R^h_l = \\nabla A^h_l \\odot A^h_l$, where $\\nabla A^h_l$ is the gradient of the CLIP score with respect to an attention map and $A^h_l$ is the attention map itself; removing the ReLU lets negative values flow. Word attributions $w_j$ are obtained by averaging token-level attributions across layers and along the [EOS] row, and a word is predicted misaligned when $w_j$ falls below a single fixed threshold $\\epsilon$. F-CLIPScore then combines the global CLIP score with the summed negative misaligned attributions, acting as a drop-in replacement for CLIPScore.","core_discovery":"The paper's central claim is that the negative entries of the gradient-attention product $R^h_l = \\nabla A^h_l \\odot A^h_l$ carry a consistent semantic signal: for a misaligned caption, the text tokens that contradict the image receive proportionally negative attribution, so the decision rule $\\mathrm{mis}(w_j)=1$ iff $w_j < \\epsilon$ picks out exactly those words. The claim rests on removing the ReLU that prior relevance-propagation methods apply to gradients, which had discarded negative values as noise. Supported by ablations showing full-gradient attributions outperform negative-only variants, the paper asserts that both positive and negative gradients are needed and that averaging attribution maps across the final layers preserves this signal. It further claims that aggregating only the negative misaligned attributions into $(1-\\mathrm{score})\\cdot\\sum_j \\mathrm{mis}(w_j)\\cdot w_j$ yields a global score, F-CLIPScore, that better correlates with human alignment judgments than the plain CLIP similarity score.","pith_inferences":["Where the paper stops with a fixed epsilon, a natural extension is to calibrate the threshold per domain or per caption-length bucket; the benchmark-wide CLIPScores vary, so a per-domain epsilon might lift performance on very low-score inputs.","The same 'negative attribution' reading could transfer to other contrastive dual encoders built on attention, not just CLIP, and to other tasks like VQA alignment or retrieval reranking where a token-level mismatch is the failure mode.","The documented failure cases (backgrounds, small objects, adjectives) suggest the signal is biased toward salient foreground nouns; combining the attribution map with a vision backbone that is stronger on small objects, or with an upweighted token-level prior, is a testable path to fix the bias.","F-CLIPScore's degraded behavior at very low CLIPScore, where gradients spread across tokens, implies that applying it to noisy alt-text (as opposed to well-aligned generated captions) would need a fallback or a gating function on the global score."],"forward_implications":["Misalignment labels can be extracted from any frozen CLIP model in one backward pass, removing the need for reference captions, object detectors, or fine-tuned reward models.","F-CLIPScore improves global image-text alignment estimation over CLIPScore, especially on hard negatives where added words inflate plain similarity, giving a cheap upgrade for captioning and text-to-image evaluation.","The method scales with backbone quality: larger CLIP variants and pretraining on larger alt-text corpora improve both localization accuracy and misalignment classification, pointing to continued gains from better CLIP checkpoints.","Because the same detector can run at tens of frames per second, it could be used as real-time feedback for generation loops or large-scale data cleaning rather than post-hoc analysis."],"supporting_citations":[{"why":"Supplies the GAE attribution rule that the paper modifies by removing the ReLU.","marker":"Chefer, Gur, and Wolf 2021a"},{"why":"Introduces the CLIP model whose frozen text and image encoders are used for attribution.","marker":"Radford et al. 2021"},{"why":"Defines CLIPScore, the baseline global metric that F-CLIPScore is built to replace.","marker":"Hessel et al. 2021"},{"why":"Provides the FOIL benchmark of single-word caption mismatches used for localization evaluation.","marker":"Shekhar et al. 2017"},{"why":"Contributes the nocaps-FOIL and HAT benchmarks and the ALOHa baseline that the paper compares against.","marker":"Petryk et al. 2024"},{"why":"Supplies the SeeTRUE-Feedback test set for textual misalignment detection across diverse domains.","marker":"Gordon et al. 2024"},{"why":"Provides the Rich-HF dataset with word-level misalignment labels and human alignment scores used for F-CLIPScore correlation analysis.","marker":"Liang et al. 2024"},{"why":"Gives the OpenCLIP implementation that trains the ViT-H/14 backbone used for the best results.","marker":"Cherti et al. 2023"}],"fun_headline_variants":["CLIP's negative gradients expose word-image mismatches","Drop the ReLU: CLIP gradients find misaligned text","Zero-shot misalignment spotting from frozen CLIP","Negative gradient signs flag CLIP's biggest misses","Turn CLIP into a dense misalignment detector"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim breaks down if negative attribution values do not carry a uniform 'this word is wrong' signal across layers and domains, since the method relies on one fixed threshold epsilon to separate bad words from good words across all benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["CLIP's negative gradients expose word-image mismatches","Drop the ReLU: CLIP gradients find misaligned text","Zero-shot misalignment spotting from frozen CLIP","Negative gradient signs flag CLIP's biggest misses","Turn CLIP into a dense misalignment detector"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1370,"prompt_tokens":1016,"completion_tokens":354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":278}},"tokens_in":632,"tokens_out":354,"duration_ms":3856,"temperature":1.0,"reasoning_tokens":278,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:42:08.602768+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of captions where each misaligned word is known, and plot the distribution of the word-level attribution $w_j$ for misaligned versus aligned words per layer group; if the two distributions do not separate below a fixed epsilon consistently across domains, or if the optimal epsilon varies by more than the reported search range when tuned per dataset, the central heuristic is falsified.","supporting_citations":[{"cited_title":"W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"Introduces the CLIP model whose frozen text and image encoders are used for attribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CLIPScore, the baseline global metric that F-CLIPScore is built to replace."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FOIL benchmark of single-word caption mismatches used for localization evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the nocaps-FOIL and HAT benchmarks and the ALOHa baseline that the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SeeTRUE-Feedback test set for textual misalignment detection across diverse domains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Rich-HF dataset with word-level misalignment labels and human alignment scores used for F-CLIPScore correlation analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the OpenCLIP implementation that trains the ViT-H/14 backbone used for the best results."}],"review_version":1}