{"id":"222be55f-83e8-4aee-8078-0d62a9c7c45c","arxiv_id":"2504.15619","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AdaViP reduces object hallucination in multimodal LLMs by training on image-edited preference pairs and adaptively weighting vision and language preferences.","lead":"This paper introduces AdaViP, a training method that creates negative examples by removing key objects from images with vision models, then fine-tunes multimodal language models while balancing visual and text preferences. The authors report large reductions in object hallucination on several benchmarks, surpassing prior DPO-based alignment methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CLIP-selected removal step—the core of vision-based preference construction—is never validated; if it removes objects that are not load-bearing for yw, the preference signal is noise and the claimed mechanism is unsupported.","rationale":"The reader's weakest assumption identified the same core vulnerability: the CLIP-selected removal may not correspond to the elements yw depends on, and no human validation or failure analysis is provided. I agree this is the most load-bearing concern for the mechanism claim. I add a sharper causal test: a random-removal ablation would directly determine whether the selection strategy, rather than the adaptive loss or the mere presence of augmented image pairs, is responsible for the gains. The loss derivation itself is sound, and I found no internal inconsistency in Eqs. 10-13. The reporting issues the reader noted (baselines sourced from external papers, no error bars, the F1 discrepancy between Tables 1 and 2) are real but secondary: they weaken verification of the exact SOTA margin without identifying a specific flaw in the method. Consequently, the appropriate verdict remains CONDITIONAL: the paper is plausible and internally consistent, but the central mechanism rests on an unvalidated assumption, and a single targeted ablation could either confirm or undermine it.","tokens_in":13611,"tokens_out":10225,"duration_ms":103650,"concrete_test":"Retrain AdaViP with the adaptive loss but choose Il uniformly at random from the candidate set C (Section 3.2, Eq. 8) instead of by the CLIP score in Eq. 9, keeping all other hyperparameters, data, and training steps identical. Compare Object HalBench Non-Rsp/Non-Men and AMBER discriminative F1 against the CLIP-selected version. If random selection matches or exceeds CLIP-selected performance, the claim that strategic key-element selection drives the gains is falsified; if random selection is materially worse, the CLIP assumption is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AdaViP achieves SOTA trustworthiness because it (1) constructs vision-based preference pairs by removing key visual elements and (2) adaptively balances vision- and language-based preferences. The loss derivation in Eqs. 10-13 is internally consistent: Eq. 11 maximizes log P(D++ first) under a Plackett-Luce model, and the gradient in Eq. 12 has the correct softmax weights. The vulnerability is upstream, in data construction. Section 3.2 (Eq. 9) selects the rejected image Il as the candidate whose summed CLIP score against the sentence-decomposed preferred response yw is lowest. This assumes CLIP similarity faithfully ranks which removed object is semantically load-bearing for yw. The paper provides no human validation, no failure analysis of RAM tagging, GroundingDINO detection, SAM segmentation, LaMa inpainting, or CLIP ranking, and no comparison against random removal. Figure 3 shows selected successes only. If the cascade removes an object that is irrelevant to yw, the triple (Q, Il, yw) teaches the model to distrust yw for a change that does not affect its truth, injecting noise into the preference signal. Table 2's ablation does not isolate the selection mechanism: it compares DPO, equal-weight vision+language, and adaptive weighting, but never random vs. CLIP-selected removals, nor adaptive weighting without vision pairs. Thus the claim that strategic key-element removal is responsible for the gains is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaViP, a preference-alignment method for multimodal LLMs that augments conventional language-based DPO with vision-based preference pairs. The vision-based rejected sample is constructed by cascading RAM, GroundingDINO, SAM, and LaMa to detect, segment, and remove objects from the preferred image, then selecting via CLIP the candidate removal least similar to the preferred response. The training loss unifies the preferred, vision-rejected, and language-rejected triples under a Plackett-Luce-style objective, yielding adaptive weights for the two rejected samples. Experiments on LLaVA-1.5 7B/13B report lower object hallucination on Object HalBench, AMBER, and MMHal-Bench than several published baselines, including values above GPT-4V on Object HalBench.","tokens_in":13905,"tokens_out":8396,"duration_ms":72309,"significance":"The loss derivation is internally consistent: Equation (11) follows from the Plackett-Luce model in Equation (10), and the gradient in Equation (12) and the weights in Equation (13) are correct algebraic consequences. The adaptive weighting requires no additional tuned hyperparameter, and the approach builds on off-the-shelf visual foundation models, making it a practical recipe if the empirical claims hold. If the reported numbers are reproduced, AdaViP would be a strong method for reducing object hallucination in open-source MLLMs. However, the evidence does not yet validate the CLIP-based removal mechanism, and the state-of-the-art comparisons are not re-run under identical conditions, so the significance is conditional on additional experiments.","major_comments":[{"comment":"The construction of the rejected image Il rests on the assumption that CLIP similarity to the sentence-decomposed preferred response yw ranks removals by semantic load-bearingness. The paper provides no human validation, no failure analysis of the RAM/GroundingDINO/SAM/LaMa cascade, and no comparison against random removal; Figure 3 shows only selected successful cases. If the cascade removes an object that yw does not depend on, the resulting triple (Q, Il, yw) is a noisy preference, and the central claim that strategic key-element removal drives the gains is unsupported. Please add (i) a quantitative precision/recall study of the removal cascade and CLIP ranking, and (ii) an ablation comparing CLIP-selected removals against random removals and a no-removal control.","section":"§3.2, Eq. (9)"},{"comment":"The ablation does not isolate the two claimed components. It compares DPO, equal-weight vision+language DPO, and adaptive weighting, but there is no row with adaptive weighting while omitting vision pairs, nor a row with vision pairs built from random removals. Consequently, the improved F1 and accuracy under '+Adaptive' could be due entirely to the weighting scheme, and the table cannot establish that CLIP-selected object removal contributes. Please add the two missing rows.","section":"§4.2, Table 2"},{"comment":"The state-of-the-art claim rests on baseline numbers 'sourced from [8,30,37]' rather than re-run under the same evaluation protocol, and no standard deviations or multiple seeds are reported. Given the large reported margins, please re-evaluate at least the closest baselines (RLAIF-V, V-DPO, MDPO) on Object HalBench and AMBER, and report variance. Also reconcile Table 2's base LLaVA-1.5-7B overall F1 of 74.7 with Table 1's 77.7 for the same model and AMBER Discriminative metric.","section":"§4.1 and Table 1"}],"minor_comments":[{"comment":"The abstract says AdaViP achieves '93.7% and 96.4% reductions in response-level and mentioned-level hallucination,' but Table 1 reports non-hallucination rates; please rephrase to avoid equating a rate with a reduction.","section":"Abstract"},{"comment":"Equation (1) has an unbalanced parenthesis in sigma(r*(yw|Q,I) - (r*(yl|Q,I)); insert the missing closing parenthesis.","section":"Eq. (1)"},{"comment":"The base LLaVA-1.5-7B row reports an overall F1 of 74.7, while Table 1 reports 77.7 for the same model and AMBER Discriminative F1; please clarify which subset or metric is used in each table.","section":"Table 2"},{"comment":"The variable n is reused for the number of RAM categories in Equation (5) and the number of decomposed sentences in Equation (9); use a separate symbol such as m for the sentence count.","section":"§3.2, Eq. (9)"},{"comment":"The bullet claiming 'over 70%' improvement on AMBER Generative is ambiguous: the reported CHAIR drop from 3.5 to 1.8 is about 49%, while the Cog. drop from 1.0 to 0.3 is 70%; please specify which generative metric is meant.","section":"§4.3"},{"comment":"Figure 5 needs axis labels, a legend, and a description of what each curve averages over; as printed, the weight and reward dynamics are hard to reproduce. Also fix the typo 'essboard' in Figure 3.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2504.15619. First, the method is a sensible next step over V-DPO/MDPO/POVID: instead of corrupting images randomly, it deletes specific objects with a RAM + GroundingDINO + SAM + Big LaMa pipeline, then uses CLIP to pick the most damaging removal, and trains with a three-outcome Plackett–Luce loss that automatically weights vision vs. language preferences. The loss derivation in Eqs. 11–13 is algebraically consistent, and the ablations show the adaptive weighting matters. Second, the central mechanism is not actually validated. The choice of which object to remove is made by CLIP similarity to the preferred response, and the paper gives no human check, no failure analysis, and no comparison against random removal. So we cannot yet tell whether 'strategic key-element removal' drives the gains or whether any image perturbation would do. That is the real soft spot.\n\nWhat the paper does well: the idea of unifying the two preference types in one Plackett–Luce model is clean, and the gradient analysis is honest. The reported gains over DPO are large (Object HalBench 80→94 non-hallucination for 7B), so there is likely something real here, even if exact numbers should be treated with caution. I checked the math; it holds.\n\nWhere it is soft: no error bars, most baselines are cited from other papers rather than re-run, and the abstract's 'reductions' should be 'non-hallucination rates'—the numbers are percentages of correct output, not reductions in errors. There is also a numeric inconsistency: Table 1 gives LLaVA-1.5-7B AMBER Discriminative F1 as 77.7, Table 2 gives 74.7 for the same setting. These are reporting issues, not fraud, but they need fixing before publication.\n\nThe stress test's concern is fair and lands: the CLIP-selected removal step is the load-bearing assumption for the vision preference pairs, and it is the least-examined part. The ablation in Table 2 does not isolate selection (random vs. CLIP) or the adaptive loss without vision pairs. So the paper's mechanism claim is under-supported, though not contradicted.\n\nBottom line: this is a paper for people working on MLLM alignment and hallucination mitigation. It deserves a serious referee. I would send it to review with a request for targeted revisions: validate the removal selection, add error bars, re-run at least one strong baseline, fix the abstract, and reconcile the tables. It is a solid incremental contribution in need of cleaning, not a desk reject.","headline":"A sensible next step over V-DPO/MDPO with a clean adaptive DPO loss, but the central removal-selection mechanism is unvalidated and the reporting needs fixes; deserves a serious referee.","tokens_in":14452,"tokens_out":2972,"would_cite":true,"duration_ms":26007,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdaViP claims that training multimodal LLMs to prefer intact images over images with key objects removed sharply cuts hallucination, beating GPT-4V on Object HalBench.","keywords":["multimodal large language models","hallucination mitigation","preference optimization","direct preference optimization","vision-based preference pairs","adaptive preference weighting","object removal","trustworthiness"],"falsifier":"A human study on a random sample of constructed pairs would settle it: if annotators frequently find that the selected removed object is not mentioned or implied by the preferred response, or if replacing similarity-based selection with random removal selection leaves most of the hallucination gains intact, then the selection mechanism is not carrying the result.","tokens_in":13400,"feed_emoji":"🖼️","tokens_out":7836,"duration_ms":69439,"temperature":0.7,"pith_summary":"The paper argues that preference alignment for multimodal LLMs has been too language-centric: contrasting a good answer against a bad answer for the same image does not force the model to look carefully at the image. AdaViP therefore creates a second kind of rejected sample by removing a key visual object from the image while keeping the preferred answer, so the model must learn which visual details the answer depends on. A cascade of visual foundation models proposes candidate removals, and a vision-language similarity model selects the removal that most lowers similarity to the preferred response. The training loss then treats the preferred sample as beating both the language-based and vision-based rejected samples, with weights on the two rejection types set adaptively from their implicit rewards. The authors report that this lowers object hallucination to the point where open 7B and 13B models outperform GPT-4V on the Object HalBench non-hallucination rate.","feed_headline":"Vision-based DPO cuts object hallucination below GPT-4V","feed_subtitle":"Removing key objects from images trains multimodal LLMs to notice what answers depend on.","key_machinery":"The load-bearing mechanism is the constructed vision-based preference pair plus an adaptive three-way DPO objective. In the construction stage, an image tagging model lists object categories, an open-set detector localizes them, a segmentation model produces masks, and an inpainting model removes each object to create candidate images; a vision-language similarity model scores each candidate against the preferred response decomposed into sentences, and the lowest-scoring candidate becomes the rejected image (Eq. 9). In the optimization stage, the loss (Eq. 11) is a Plackett-Luce-style softmax over the preferred sample versus both rejected samples, so the weight given to the language-based and vision-based rejected samples is $\\omega_{+-} = \\mathrm{Sigmoid}(\\hat{r}(D^{+-})-\\hat{r}(D^{-+}))$ and $\\omega_{-+} = \\mathrm{Sigmoid}(\\hat{r}(D^{-+})-\\hat{r}(D^{+-}))$, respectively, where $\\hat{r}$ is the implicit DPO reward. This weighting is what prevents the easily distinguishable vision-rejected samples from dominating training.","core_discovery":"On the paper's own terms, the claim is that hallucination in multimodal LLMs can be reduced by making visual context a first-class citizen of preference optimization. Given a preferred triple of question, image, and preferred response, AdaViP constructs a rejected triple with the same question and response but an image from which a key object has been removed; the removed object is chosen as the one that most lowers the similarity between the perturbed image and the sentence-decomposed preferred response. The model is trained with a DPO-style loss in which the preferred sample must beat both the language-rejected sample and the vision-rejected sample, and the relative weights of the two rejection types are set by a sigmoid over their implicit reward differences. In the paper's experiments this produces Object HalBench non-hallucination rates of 93.7% response-level and 96.4% mention-level for a 7B model, and 95.4% and 97.7% for a 13B model, above GPT-4V's 86.4% and 92.7%; the same recipe also improves AMBER and MMHal-Bench scores.","pith_inferences":["The paper does not test whether the similarity-based selector is the bottleneck; a stronger scorer or human-verified removals could plausibly harden the negatives further or, conversely, expose that the training loss is doing most of the work.","The constructed pairs are essentially hard negatives for visual grounding, so the same data could serve as a probing task: asking a trained model what changed between the original and rejected image would directly measure whether the model has learned to track the removed object.","The adaptive weighting scheme should transfer to other alignment settings where one rejection type is much easier than another, such as safety versus helpfulness, although the paper demonstrates it only for vision versus language."],"forward_implications":["If the paper is right, object hallucination in multimodal LLMs can be attacked by changing the preference data rather than the architecture: no new model components are needed, only constructed image perturbations and a modified loss.","The adaptive weighting is necessary: the paper reports that a fixed equal-weight combination of vision and language preferences drops AMBER discriminative accuracy from 64.8% to 51.8%, while the adaptive loss raises it to 79.9%.","During training the model relies more on vision-based preferences early and shifts toward language preferences later, implying that the visual signal is learned first and then integrated with language preferences.","The gains transfer from a 7B to a 13B model of the same family, so the mechanism is not tied to one model size.","Because AdaViP's 7B and 13B variants exceed GPT-4V on Object HalBench non-hallucination rates, the paper's approach is a candidate route for open-source models to close the trustworthiness gap with proprietary systems."],"supporting_citations":[{"why":"Supplies the DPO objective and implicit reward definition that AdaViP extends to three outcomes.","marker":"[23]"},{"why":"Provides the language-based preference dataset that AdaViP uses as the preferred and language-rejected samples.","marker":"[37]"},{"why":"A vision-guided DPO baseline that AdaViP compares against on hallucination benchmarks.","marker":"[32]"},{"why":"A baseline combining vision and language preferences with fixed weights, which AdaViP's adaptive strategy improves on.","marker":"[30]"},{"why":"Provides the vision-language similarity model used to select the rejected image with the lowest score against the preferred response.","marker":"[22]"},{"why":"Provides the image tagging step that lists object categories to be removed.","marker":"[41]"},{"why":"Provides open-set detection that localizes the tagged objects with bounding boxes.","marker":"[17]"},{"why":"Provides the segmentation masks used to isolate each object before removal.","marker":"[11]"},{"why":"Provides the inpainting model that removes the segmented objects while preserving the rest of the image.","marker":"[28]"},{"why":"Provides the Plackett-Luce ranking model that justifies unifying vision and language preferences in one softmax loss.","marker":"[21]"}],"fun_headline_variants":["Adaptive vision DPO cuts object hallucination 93.7%","Removing key objects trains MLLMs to avoid hallucinations","AdaViP: Vision-aware preference optimization beats GPT-4V","Object removal in DPO slashes hallucination rates","Multimodal LLMs taught to see before they answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the candidate image whose similarity to the preferred answer is lowest is the one whose removed object the answer actually depends on; if that ranking is wrong, the rejected image teaches the model to distrust an irrelevant visual change.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive vision DPO cuts object hallucination 93.7%","Removing key objects trains MLLMs to avoid hallucinations","AdaViP: Vision-aware preference optimization beats GPT-4V","Object removal in DPO slashes hallucination rates","Multimodal LLMs taught to see before they answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3182,"prompt_tokens":931,"completion_tokens":2251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2166}},"tokens_in":547,"tokens_out":2251,"duration_ms":14073,"temperature":1.0,"reasoning_tokens":2166,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:22:18.726263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human study on a random sample of constructed pairs would settle it: if annotators frequently find that the selected removed object is not mentioned or implied by the preferred response, or if replacing similarity-based selection with random removal selection leaves most of the hallucination gains intact, then the selection mechanism is not carrying the result.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Supplies the DPO objective and implicit reward definition that AdaViP extends to three outcomes."},{"cited_title":"V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization","cited_arxiv_id":null,"evidence_quote":"A vision-guided DPO baseline that AdaViP compares against on hallucination benchmarks."},{"cited_title":"Resolution-robust large mask inpainting with fourier convolutions","cited_arxiv_id":null,"evidence_quote":"Provides the inpainting model that removes the segmented objects while preserving the rest of the image."},{"cited_title":"The analysis of permutations","cited_arxiv_id":null,"evidence_quote":"Provides the Plackett-Luce ranking model that justifies unifying vision and language preferences in one softmax loss."}],"review_version":1}