{"id":"82a27ea4-cc73-4d95-a3ed-a4b41bf40f2d","arxiv_id":"2411.12713","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A training-free decoding method that splits images into complementary SAM-based parts and adaptively contrasts token distributions to reduce hallucinations in vision-language models.","lead":"This paper proposes CATCH, a training-free decoding method that reduces hallucinations in vision-language models by splitting the input image into complementary segments and adaptively contrasting token probabilities from each segment. It reports consistent gains on POPE, MME, and CHAIR benchmarks with LLaVA-1.5 and InstructBLIP.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CATCH's reported gains may be a conservative-response artifact: POPE precision rises while recall falls, and CHAIR rewards fewer object mentions; no control separates visual correction from a simple 'answer less' strategy.","rationale":"The reader's verdict is CONDITIONAL, and my analysis supports keeping it. The paper's method is plausible, training-free, and the consistency of gains across LLaVA-1.5 and InstructBLIP is real evidence that something systematic is happening. However, the systematic effect could be a conservative response bias rather than the proposed visual-defect correction. The reader identified the Eq. 6 selection rule as the weakest assumption, but I see the more load-bearing issue at the evaluation level: POPE precision/recall trade-offs and CHAIR's sensitivity to object-mention counts allow the reported hallucinations reductions to be achieved by omitting uncertain content. The paper even acknowledges the lower yes ratio. This is not an internal contradiction that forces rejection, but it is a concrete confound that code release and a matched control could settle. Therefore the verdict remains CONDITIONAL, pending the control experiment and code.","tokens_in":11827,"tokens_out":6899,"duration_ms":75091,"concrete_test":"Run CATCH and a matched control that uses the same base LVLM but replaces the adaptive contrast with a fixed downward bias on uncertain tokens, calibrated to match CATCH's POPE 'yes' ratio and CHAIR caption length. If the control matches CATCH's POPE F1 and CHAIR_S/I scores, then the reported gains are consistent with conservatism rather than visual-defect correction. Additionally, report CHAIR object-level recall (fraction of ground-truth objects mentioned) and mean caption length for CATCH versus baseline; if CATCH mentions fewer real objects and produces shorter captions, the hallucination reduction is partly avoidance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CATCH removes hallucinations by correcting a visual information bottleneck. For that claim to hold, the contrastive correction must specifically restore visual evidence rather than merely make the model more cautious. The reported evidence does not separate these alternatives. On POPE, CATCH consistently trades recall for precision: for example, with LLaVA on GQA-adversarial, recall drops from 93.13 to 91.47 while precision rises from 61.68 to 71.72; with LLaVA on A-OKVQA-popular, recall drops from 91.67 to 90.67 while precision rises from 70.51 to 80.90. The paper itself attributes the precision gain to 'a lower yes response ratio compared to the baseline.' On CHAIR, the metrics also reward fewer object mentions: CHAIR_S is hallucinated objects divided by all mentioned objects, and CHAIR_I is the fraction of captions containing any hallucinated object, so a model that omits uncertain objects scores better even if it has not improved visual perception. The paper reports neither object-level recall of ground-truth objects nor caption length. Thus the main benchmarks cannot distinguish 'fewer false claims' from 'a conservative answer strategy that would reduce hallucinations without any visual correction.' If the latter is the case, the information-bottleneck narrative is not supported by the evidence. A secondary ambiguity is that the selection rule in Eq. 6 assumes JSD-to-nonvisual ranks visual relevance, and Eq. 8 uses zd in its second branch even when the selected decoupled image is zr, making the exact algorithm hard to reconstruct. The load-bearing gap is the absence of a control for response conservatism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CATCH, a training-free decoding-time method to reduce hallucinations in large vision-language models (LVLMs). It first uses SAM to decouple the input image into two complementary views (dual and residual), introduces a non-visual text-only input, and at each token step computes Jensen-Shannon divergences between the output distributions of the non-visual input and the original/decoupled views. Based on these divergences, it selects one decoupled view and either contrastively subtracts its logits from the original (hallucination-existence case) or adds them to the original (diversity-insufficient case). CATCH is evaluated on POPE, MME Hallucination, and CHAIR using LLaVA-1.5 and InstructBLIP baselines, reporting consistent gains over the baselines and over prior contrastive-decoding methods. The paper also presents analyses (Figs. 3 and 5) aimed at motivating the visual-defect/information-bottleneck narrative and the claim that CATCH delays cumulative hallucinations.","tokens_in":12122,"tokens_out":5280,"duration_ms":47873,"significance":"If the central claim holds, CATCH would be a valuable, generally applicable, training-free intervention that reduces object hallucination in LVLMs without requiring task-specific data or fine-tuning. The paper has real strengths: it evaluates on two model families across three standard benchmarks, reports internally consistent numbers, and includes an interesting empirical observation (Fig. 1) that decoupling the image changes ground-truth versus hallucinated token probabilities. The method is simple enough to reproduce. However, the causal narrative is not yet established by the evidence: the benchmark results do not separate improvement due to genuine visual correction from a conservative response bias, and the information-bottleneck formulation is presented as a motivation rather than derived or verified. Because the manuscript's core claim rests on this distinction, the significance of the contribution currently depends on additional analysis rather than on the reported numbers alone.","major_comments":[{"comment":"There is an inconsistency between the selection rule in Eq. (6) and the decoding rule in Eq. (8). Eq. (6) selects the decoupled image z from {z_d, z_r} based on comparing d(z_d, z_n) and d(z_r, z_n); when z_r is selected, the text in §3.4 says the 'decoupled image' distribution is used for both contrastive subtraction and contrastive enhancement. However, Eq. (8) conditions the first branch on d(z_d, z_n) ≥ d(v, z_n) rather than on d(z, z_n) ≥ d(v, z_n), and the second branch uses logit_θ(y | z_d, x) even when Eq. (6) selected z_r. As written, the algorithm is ill-defined for the case z = z_r and the second branch never uses the selected decoupled image. Please correct Eq. (8) to use the selected z in both branches and to compare d(z, z_n) with d(v, z_n), or explicitly state if the design intentionally always uses z_d in the enhancement branch and justify that choice.","section":"§3.4, Eq. (8)"},{"comment":"The information bottleneck objective in Eq. (2), min[I(v; z) − I(z; y)], is never optimized in the paper. The text states that SAM is used to 'optimize this objective,' but no derivation is given showing that selecting the top M objects by area as z_d (and masking them to form z_r) minimizes I(v; z) − I(z; y), nor is any approximation or justification provided. As it stands, the IB formulation is a post hoc narrative rather than a derivation of CVD. Please either derive the connection between Eq. (2) and the SAM-based decoupling, or reframe the contribution to present CVD as a heuristic motivated by (but not derived from) the information bottleneck principle.","section":"§3.2, Eq. (2)"},{"comment":"The reported improvements are not sufficient to distinguish 'visual correction' from a conservative-response artifact. On POPE, CATCH consistently increases precision while decreasing recall; for example, LLaVA on GQA-adversarial shows precision rising from 61.68 to 71.72 while recall falls from 93.13 to 91.47, and LLaVA on A-OKVQA-popular shows precision rising from 70.51 to 80.90 while recall falls from 91.67 to 90.67. The paper itself attributes the precision gain to a 'lower yes response ratio' (§2). On CHAIR, the metrics reward fewer object mentions, so a model that omits uncertain objects can improve regardless of visual perception. To support the claim that CATCH specifically restores visual evidence, please add controls: report object-level recall of ground-truth objects on CHAIR, caption length, and a POPE analysis that either matches the baseline's yes-rate through rejection sampling or compares against a constant-'no' baseline. Without such controls, the benchmark results are also compatible with a strategy that simply makes the model more conservative.","section":"§2, Table 1 and Fig. 4"},{"comment":"The selection rule in Eq. (6) assumes that the view with the larger Jensen-Shannon divergence from the non-visual input is the one whose visual content is most relevant to the next token. This is a load-bearing premise, because selecting the wrong view would make the contrastive subtraction or addition amplify the wrong signal. The supporting evidence in Fig. 3 uses a masking procedure ('key visual features') that is not described in the same terms as the actual CVD (which selects top M objects by area), and it is only demonstrated on a random subset of MSCOCO, not on the heterogeneous images in POPE, MME, and CHAIR. Please either provide a per-benchmark validation of the selection rule or add an analysis (e.g., a token-prediction probe) showing that the larger-JSD view is indeed the one that improves next-token probability for ground-truth tokens.","section":"§3.3, Eq. (6)"}],"minor_comments":[{"comment":"The definitions of CHAIR_S and CHAIR_I in the text are incorrect. The paper describes CHAIR_S as 'the proportion of hallucinated sentences relative to all sentences,' but CHAIR_S is the fraction of hallucinated object instances among all mentioned object instances; similarly, CHAIR_I is the fraction of captions containing any hallucinated object, not 'the proportion of hallucinated objects relative to all generated objects.' Please correct these descriptions.","section":"§2, CHAIR definitions"},{"comment":"The notation p_θ(y_t | z, v, x) on the left-hand side of Eq. (8) does not match the right-hand side, which conditions on z in the first branch and on z_d in the second branch. Please make the conditioning consistent after fixing the branch logic.","section":"§3.4, Eq. (8)"},{"comment":"There are several typographical errors: 'Constrastive' in the Keywords, 'Implementtation' in §3.5, 'showm' in §2, and a duplicated affiliation number 4 in the author list. These should be corrected in the final version.","section":"Throughout"},{"comment":"The figure caption states that five random seeds were averaged, but no error bars or per-seed variance are shown. Please include error bars or report the variance, since the MME subtasks contain only 60 questions each and are thus susceptible to seed noise.","section":"Fig. 4"},{"comment":"The hyperparameters α, β, and the object-count ratio M/N are fixed by default (α = 1.2, β = 3, M = N * 0.05), but no sensitivity analysis is reported. Since the method's performance is likely sensitive to these values, please report an ablation over at least α and β, and over M/N for a representative subset.","section":"§3.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not yet ready for acceptance. The most pressing issue is the inconsistency in Eq. (8), which is a concrete technical error that must be corrected. More fundamentally, the evaluation as presented does not rule out the possibility that the improvements reflect a shift toward more conservative yes/no responses and shorter, object-sparse captions rather than a genuine recovery of visual evidence. The authors should be asked to provide the controls suggested in the major comments (object-level recall on CHAIR, caption length, yes-rate matching on POPE, and a selection-rule validation). The information-bottleneck framing also needs either a derivation or a clear demotion to motivational heuristic. I would not reject the paper on the current evidence because the core idea is plausible and the empirical results are suggestive, but the claims as written outrun the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the method itself is new in the contrastive-decoding literature: SAM-based decoupling into complementary views, a non-visual input to screen which view to use, and adaptive token-level contrast (subtract or add) — that combination is not in VCD, M3ID, HALC, or RITUAL. Second, the evaluation does not support the paper's own story. The benchmarks reward caution: POPE counts yes/no correctness and CATCH raises precision while lowering recall (e.g., LLaVA on GQA adversarial: recall 93.13 to 91.47, precision 61.68 to 71.72), and the paper itself credits the gain to 'a lower yes response ratio.' CHAIR's metrics are ratios over mentioned objects, so omitting objects also looks good. There is no control that separates 'the decoupled image restored visual evidence' from 'the model just answers no less often.' That is a load-bearing gap.\n\nWhat is genuinely good: the approach is training-free, works on two different LVLMs, and the gains on POPE and CHAIR are large. The Fig. 3 analysis — masked images produce distributions close to the non-visual input — is compelling motivation. The method is also not circular: it is evaluated on external benchmarks and does not restate its target result.\n\nThe soft spots beyond the confound. The Information Bottleneck equation (2) is never optimized; SAM segmentation is asserted as the implementation without a derivation. Eq. (8) uses zd in the second branch even when the selected decoupled image is zr — an inconsistency that makes the algorithm ambiguous. RITUAL appears in Table 1 but is not cited in the text. No code is released, so all numbers are self-reported. None of these are fatal, but they add uncertainty.\n\nBottom line: this is a plausible method and the empirical results are probably real in the narrow sense that the benchmark scores improve. But the paper's central claim — that hallucinations come from a visual information bottleneck and CATCH fixes it — is not supported by the current evidence. A serious referee should engage, but the revision needs a conservative-baseline control (e.g., simply downweighting 'yes' or truncating mentions), object-level recall on CHAIR, and code release. Without that, I wouldn't rely on the numbers.\n\nRecommendation: send to peer review, but it needs major revision before acceptance.","headline":"CATCH is a novel training-free decoding method with big benchmark gains, but the reported numbers cannot distinguish visual correction from a conservative response shift, so the mechanistic claim is unproven.","tokens_in":12721,"tokens_out":2417,"would_cite":false,"duration_ms":24201,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CATCH is a training-free decoding-time method that reduces hallucinations in large vision-language models by contrastively comparing SAM-decoupled complementary image views against a text-only baseline.","keywords":["hallucination mitigation","large vision-language models","contrastive decoding","information bottleneck","visual defect","Segment Anything Model (SAM)","decoding-time intervention","object hallucination"],"falsifier":"Construct or find an image where a large, salient object (for example, a bright red ball) is present but irrelevant, while a small, easily missed object (for example, a key on the floor) is the ground-truth object the caption must mention for the next token. Segment the image so the ball forms the dual image and the key is in the residual; if the ball's distribution is farther from the text-only prior than the residual's, NVS selects the ball, and CATCH will suppress or fail to boost the correct token. A statistical version: on 1,000 random MSCOCO images, mask the ground-truth region and measure whether the view with the larger Jensen-Shannon divergence from the non-visual input gives a higher probability to the ground-truth token than the other view does; a null or negative correlation would refute the load-bearing selection rule.","tokens_in":2227,"feed_emoji":"🖼️","tokens_out":2129,"duration_ms":109434,"temperature":0.7,"pith_summary":"Large vision-language models (LVLMs) often invent objects or attributes that are not in the image, and the authors argue the root cause is a 'visual defect': the whole image overloads the model's vision-language alignment, compressing away fine-grained features and pushing the decoder onto linguistic priors. CATCH intervenes at decoding time, without any retraining or task-specific data, by segmenting the image with SAM into two complementary views (foreground-object view and residual view), running a text-only version of the same prompt, and using Jensen-Shannon divergence to decide which view carries the evidence for the next token. Depending on that comparison, it either contrastively subtracts the original (hallucination-rich) distribution or adds the decoupled view to restore diversity. On POPE, MME, and CHAIR benchmarks, CATCH reports consistent gains over two base models, including up to 8.07 accuracy points on POPE, 16% total-score improvement on MME, and a 45.8% relative reduction on CHAIR hallucination metrics. The importance is that a decoder-side correction, not more data or a bigger visual encoder, may be sufficient to curb a substantial share of LVLM hallucination.","feed_headline":"Training-free decoding cuts vision-language hallucinations by 45%","feed_subtitle":"It contrasts decoupled image views against a text-only baseline to suppress hallucinated tokens.","key_machinery":"The mechanics rest on the complementarity of two views plus a divergence-based selection rule. CVD builds $z_d$ and $z_r$ so that $z_d + z_r = v$ in pixel space, each carrying a simplified subset of the visual features; the non-visual input $z_n$ serves as a language-prior reference. At each token step, the model runs multiple forward passes (original $v$, the two views, and $z_n$), and NVS selects the view with the larger Jensen-Shannon divergence from $z_n$ as the decoupled image $z$ (Equation 6). ATCD then applies Equation (8): if the chosen view's divergence from the non-visual input is greater than the original image's divergence, it samples from $\\mathrm{softmax}[\\alpha\\,\\mathrm{logit}(y|z,x) - \\mathrm{logit}(y|v,x)]$ to subtract hallucinated concepts; otherwise it samples from $\\mathrm{softmax}[\\beta\\,\\mathrm{logit}(y|v,x) + \\mathrm{logit}(y|z,x)]$ to add diversity. The whole mechanism is training-free and is reported to be robust across two LVLMs and three benchmarks.","core_discovery":"The central claim is that hallucinations in LVLMs emerge from a vision-language alignment bottleneck rather than only from statistical or language priors. The paper shows that when the visual input is decoupled into simplified complementary parts, the probability of the ground-truth token rises while the hallucinated token's probability falls, implying that extraneous visual features actively interfere with fine-grained perception. CATCH formalizes this with three components: Complementary Visual Decoupling (CVD) uses SAM to split the image into a dual image $z_d$ (top $M$ objects) and a residual image $z_r$ (the rest), and introduces a non-visual input $z_n$ that contains only the text prompt; Non-Visual Screening (NVS) computes $D_{\\mathrm{JS}}(z_d,z_n)$ and $D_{\\mathrm{JS}}(z_r,z_n)$ and selects whichever view is more divergent as the decoupled image $z$; Adaptive Token-level Contrastive Decoding (ATCD) then compares $D_{\\mathrm{JS}}(z,z_n)$ with $D_{\\mathrm{JS}}(v,z_n)$ to choose between subtractive and additive contrastive decoding. The authors report that this procedure reduces hallucination across object-existence (POPE), attribute (MME), and open-ended captioning (CHAIR) benchmarks on both LLaVA-1.5 and InstructBLIP, and that it delays cumulative hallucinations from roughly 40% of the generated sequence to about 80%.","pith_inferences":["The paper's selection rule equates 'more divergent from a text-only prior' with 'more relevant to the next token,' but that equivalence is not proven for all images; a natural stress test is an image with a salient but irrelevant object that the language prior favors, where the rule could pick the wrong view and CATCH would amplify the hallucination.","The information-bottleneck framing implies that any operation that increases the mutual information between the visual evidence and the next token at decoding time—not just SAM-based decoupling—should produce a similar effect; testing CATCH against simple crops, magnification, or attention-weighted masks would isolate the role of the complementary split.","The method requires several forward passes per token, so its practical cost is not reported; if the selection rule proves correct, a lightweight approximation of the JSD ranking could make the approach faster without losing the core contrastive mechanism.","The paper analyzes cumulative hallucinations by measuring when the output distribution collapses to the non-visual distribution; a stronger test would use oracle ground-truth tokens to verify that the selected view actually lifts the probability of the correct next token at that position."],"forward_implications":["Without any training or data collection, CATCH can be applied to an already deployed LVLM to reduce object hallucinations in both short-answer and open-ended generation.","The same mechanism—splitting the image into complementary views and anchoring on a text-only prior—generalizes to different visual question-answering tasks, as the paper shows on POPE, MME, and CHAIR.","By shifting the onset of cumulative hallucinations from ~40% to ~80% of the sequence, CATCH reduces error propagation in long captions, making it useful for tasks that require detailed descriptions.","Because CATCH improves not only existence but also count, position, and color hallucinations, the mechanism addresses attribute-level fabrication, not just object presence.","The method is model-agnostic across the two decoders tested (LLaVA-1.5 and InstructBLIP), suggesting it targets a structural failure mode of LVLM decoding rather than one specific training scheme."],"supporting_citations":[{"why":"Supplies the Segment Anything Model that produces the dual and residual image views used throughout CATCH.","marker":"[20]"},{"why":"The visual contrastive decoding work that CATCH directly extends and the principal baseline compared on POPE.","marker":"[18]"},{"why":"LLaVA-1.5, one of the two LVLMs on which CATCH is evaluated for hallucination reduction.","marker":"[12]"},{"why":"InstructBLIP, the other LVLM baseline used in all three benchmark evaluations.","marker":"[22]"},{"why":"POPE, the binary object-existence benchmark that provides the main accuracy and F1 comparisons.","marker":"[24]"},{"why":"CHAIR, the open-ended captioning benchmark whose metrics quantify cumulative hallucination reduction.","marker":"[25]"},{"why":"MME hallucination subtasks that test existence, count, position, and color hallucinations.","marker":"[26]"},{"why":"MSCOCO, used for the masking analysis in Fig. 3 that motivates the divergence-ranking selection rule and for CHAIR evaluation sampling.","marker":"[21]"},{"why":"M3ID, a contrastive decoding baseline that CATCH improves upon in the POPE results.","marker":"[19]"}],"fun_headline_variants":["Training-free contrastive decoding cuts hallucinations 45% in LVLMs","Decouple visual inputs to reject hallucinated tokens in LVLMs","Contrastive decoding with decoupled views halves LVLM hallucinations","Adaptive token contrast cuts hallucination by contrasting decoupled image views","Decoupling visuals and screening text cues curbs LVLM hallucinations"],"cache_read_input_tokens":14720,"weakest_assumption_plain":"At each decoding step, the complementary image view whose output distribution is farthest from the text-only, non-visual input is assumed to be the one containing the visual evidence most relevant to the next token; if that ranking is wrong, the contrastive subtraction or addition targets the wrong view and can reinforce the hallucination.","fun_headline_variants_meta":{"raw":{"variants":["Training-free contrastive decoding cuts hallucinations 45% in LVLMs","Decouple visual inputs to reject hallucinated tokens in LVLMs","Contrastive decoding with decoupled views halves LVLM hallucinations","Adaptive token contrast cuts hallucination by contrasting decoupled image views","Decoupling visuals and screening text cues curbs LVLM hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2977,"prompt_tokens":1038,"completion_tokens":1939,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":654,"tokens_out":1939,"duration_ms":13745,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:13:01.956525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or find an image where a large, salient object (for example, a bright red ball) is present but irrelevant, while a small, easily missed object (for example, a key on the floor) is the ground-truth object the caption must mention for the next token. Segment the image so the ball forms the dual image and the key is in the residual; if the ball's distribution is farther from the text-only prior than the residual's, NVS selects the ball, and CATCH will suppress or fail to boost the correct token. A statistical version: on 1,000 random MSCOCO images, mask the ground-truth region and measure whether the view with the larger Jensen-Shannon divergence from the non-visual input gives a higher probability to the ground-truth token than the other view does; a null or negative correlation would refute the load-bearing selection rule.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"The visual contrastive decoding work that CATCH directly extends and the principal baseline compared on POPE."},{"cited_title":"Advances in neural information processing systems 36 (2024)","cited_arxiv_id":null,"evidence_quote":"LLaVA-1.5, one of the two LVLMs on which CATCH is evaluated for hallucination reduction."},{"cited_title":"In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 16 2014, Proceedings, Part V 13, pp","cited_arxiv_id":null,"evidence_quote":"MSCOCO, used for the masking analysis in Fig. 3 that motivates the divergence-ranking selection rule and for CHAIR evaluation sampling."}],"review_version":1}