Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Defending LVLMs Against Vision Attacks through Partial-Perception Supervision

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DPS, a black-box, training-free defense, counteracts LVLM vision attacks by prompting the model with descriptions of its own cropped views, exploiting the confidence gap between attacked and clean inputs.

desk verdict Plausible training-free defense with real empirical results, but the mechanism story and standard-performance numbers are thinner than the headline ASR tables. read the letter →

arxiv 2412.12722 v2 pith:XLBJDASJ submitted 2024-12-17 cs.CV cs.AIcs.CR

Qi Zhou , Tianlin Li , Qing Guo , Dongxia Wang , Yun Lin , Yang Liu , Jin Song Dong This is my paper · ORCID
classification cs.CVcs.AIcs.CR
keywords LVLMsecurityvisionattackdefensepartial-perceptionsupervisionweak-to-stronglearningadversarialrobustnessjailbreakblack-boximagecropping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large vision-language model under attack can be talked out of its wrong answer by its own responses to cropped copies of the same image. The proposed method, DPS (Defense through Partial-Perception Supervision), is black-box and training-free: a partial-perception pass describes a crop of the image, and that description is injected as a prompt when the model re-answers the original question on the full image. The defense exploits an asymmetry the authors observe: attacked inputs make the model less confident, so it defers to the partial observation, while on clean inputs it stays confident and ignores the injected description. Across six attack datasets on three commercial LVLMs, DPS cuts the average attack success rate by 76.3%, and with a safety-checker extension (LS-DPS) jailbreak success rates fall to 0.02–0.10 while general MM-Vet scores stay close to the undefended model.

What carries the argument

The load-bearing mechanism is a two-stage prompting loop. First, Part-Perc models answer the original question on cropped copies (center, random, and adaptive crops) and their descriptions are summarized into a supervisory message. Second, the Full-Perc model receives the original image, the original question, and that message, and is asked to re-analyze before answering. Appendix C formalizes the effect as prior injection: if $\varphi(T,T_c)$ measures semantic consistency between a candidate answer $T$ and the cropped-view text $T_c$, then the defended distribution weights each candidate by $\varphi(T,T_c)$, which suppresses attack targets that the crop text contradicts. The LS-DPS variant appends a safety-awareness instruction and an LLM-based safety checker that rewrites harmful answers.

What would settle it

Take an attack whose cue survives all three cropping strategies (e.g., place the typographic label in the image center) and measure DPS's attack success rate on RTA-100; if attack success stays near the 0.94–1.00 undefended range, the crop-sensitivity premise fails. Separately, compute the paper's confidence proxy (inverse standard deviation over varied question prefixes) on attacked versus clean images; if attacked inputs are not less confident, the supervision step has no footing.

Watch

Extended reading notes

Core claim

The paper's central claim is that a weaker perceptual model can supervise a stronger one at inference time. DPS generates three cropped views (center, random, and adaptive), asks the model to describe each crop, summarizes those descriptions, and then prompts the full-image model to re-analyze the original image with that summary in hand. The mechanism is an inferred confidence gap: vision attacks make the LVLM unsure, so the supervisory text shifts its answer; clean images leave it confident, so the supervision is overridden. The authors report that this reduces misleading-task attack success rates to 0.24–0.58 across six datasets (average 0.31 on Qwen-VL-Plus), reduces jailbreak success rates to 0.04–0.10 under LS-DPS, and keeps standard MM-Vet performance close to vanilla (e.g., 54.8 vs 57.3 on Qwen-VL-Plus and 66.8 vs 69.3 on GPT-4o-Mini).

Load-bearing premise

The defense works only if cropping removes the attack's misleading cue while keeping enough of the image for a useful description, and if an attacked model becomes less confident than on clean input so it follows the cropped description instead of its own full-image reading.

Editorial extensions

If this is right

  • Deployment requires no training and no access to model weights: DPS only needs the ability to crop images, call the LVLM, and prompt it twice, so it works on closed commercial models.
  • Stronger defense comes from more partial views: the paper reports that increasing the number of crops from 1 to 5 on the Self-Gen dataset monotonically lowers attack success, so the supervision signal aggregates.
  • The method extends to jailbreak safety with a prompt change plus a plug-in LLM checker, and LS-DPS keeps attack success at 0.02–0.10 while preserving standard performance.
  • Because it only changes responses when the model is already uncertain, DPS does not drag down clean-image quality the way voting-based SmoothVLM does; MM-Vet totals remain close to vanilla.
  • If the attack survives cropping, the defense offers no correction (noted in Appendix D.7.4), so DPS's protection is bounded by the crop-sensitivity of the attack.
  • LLM-Secured DPS and the safety-aware prompt are separable components, so the safety gains on jailbreak datasets can be combined with other cropping-based defenses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit in the paper but not stated: the same confidence asymmetry suggests a detector; a response that flips when a neutral description is injected could be flagged as attacked, turning DPS into a low-cost attack classifier alongside its corrective role.
  • Implicit in the paper but not stated: adaptive attackers could defeat DPS by placing typographic or noise cues in every cropped region, or by crafting perturbations that keep the model confident; the paper's own failure-case discussion points toward this, and the fix would need detection rather than correction.
  • Implicit in the paper but not stated: the weak-supervises-strong pattern may generalize beyond attacks; any auxiliary perceptual channel (depth maps, OCR transcripts, second views) could be fed as supervision to steady an LVLM that is uncertain, though the paper only demonstrates cropping.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DPS (Defense through Partial-Perception Supervision), a black-box, training-free defense for large vision-language models against misleading and jailbreak-style vision attacks. DPS first generates descriptions from one or more cropped views of the input image (the 'Part-Perc' model), then prompts the full-image model ('Full-Perc') with these partial-perception descriptions as supervisory information while re-answering the original question. The authors claim that attacked inputs make the model less confident, so it follows the partial supervision, while clean inputs leave the model confident enough to ignore it. A safety-aware variant, LS-DPS, adds an LLM-based safety checker. The evaluation covers six datasets, three commercial LVLMs (Qwen-VL-Plus, GPT-4o-Mini, Gemini-1.5-Flash) plus the open-source Qwen2.5-VL-32B, several baselines, ablations of cropping strategies, and an efficiency comparison. The paper reports an average attack success rate reduction of 76.3% across the three commercial models and close-to-vanilla standard performance on MM-Vet, with statistical significance tests only for the open-source model.

Significance. If the empirical claims hold, DPS is a notable practical contribution: it requires no training, no gradient access, and works through standard API prompting, which is attractive for deployed LVLMs. The paper also provides a broad set of comparisons against six baselines on both misleading and jailbreak datasets and makes its code available. The t-tests and standard-deviation bars for the open-source model, the ablations over cropping strategies, and the efficiency analysis are strengths. However, the central mechanism rests on a behavioral asymmetry that is not quantitatively established, and the theoretical appendix does not provide the promised justification. The commercial-model results lack error bars, and the standard-performance claim is reported for LS-DPS rather than plain DPS. These gaps do not necessarily invalidate the empirical findings, but they make the current manuscript's central claims stronger than the evidence supports.

major comments (5)
  1. [Section 3.2 and Appendix C, Eqs. (3)-(6)] The central premise that attacked inputs accept partial-perception supervision while clean inputs ignore it is supported only qualitatively in Figure 2, and the 'theoretical justification' in Appendix C does not fill the gap. Equations (3)-(6) introduce a semantic-consistency function phi(T, T_c) without defining it or connecting it to the prompting mechanism, and the key inequality P_defense(T_clean | x', T_c) > P_defense(T_target | x', T_c) is simply asserted. The confidence measure defined in the Section 3.2 footnote (inverse standard deviation of responses to varied prefixes) is never reported for clean versus attacked inputs. The paper should either provide a real derivation with a defined phi and explicit assumptions, or present quantitative measurements of the model's answer-change rate on clean inputs, the acceptance rate of the injected supervision on attacked inputs, and the correctness rate of the partial descriptions. Without one of these, the mechanism is an untested behavioral assumption rather than an explained defense.
  2. [Tables 1 and 2] The headline reduction claims rest on single-run ASR values for Qwen-VL-Plus, GPT-4o-Mini, and Gemini-1.5-Flash, with no error bars, confidence intervals, or significance tests. The t-tests in Table 10 cover only Qwen2.5-VL-32B and only five datasets. Since black-box API models show run-to-run stochasticity, the reported differences (e.g., DPS ASR 0.24 vs. baseline 0.48 for Qwen on RTA-100) could be within sampling noise. The authors should either report multiple runs with variance for the API-model tables or explicitly state that the numbers are single evaluations and discuss the implications for the strength of the claims.
  3. [Table 4 and Section 5.3] The standard-performance claim is not backed for plain DPS. Table 4 reports LS-DPS, which includes an additional LLM safety checker, yet Section 5.3's text says 'DPS' has minimal impact on standard performance. Even if LS-DPS scores are close to vanilla, that does not establish that DPS leaves clean inputs' original responses unchanged: MM-Vet may still award similar scores to altered but correct answers. The paper should report the MM-Vet score for plain DPS, and ideally also a clean-input answer-change rate or edit-distance statistic between vanilla responses and DPS responses, to substantiate the 'confidently maintaining its original response' premise.
  4. [Appendix D.7.4 and Tables 1-2] The paper admits in Appendix D.7.4 that when cropping fails to eliminate the attack, DPS is ineffective, but it never quantifies how often this occurs across the six datasets. The residual ASR values in Tables 1 and 2 could therefore be dominated by cropping failures rather than by the supervision mechanism. To support the claim that partial-perception supervision itself contributes beyond simple crop-based smoothing, the authors should provide a failure decomposition: among residual ASR cases, how often the partial description was wrong, how often the full model ignored a correct partial description, and how often the crop retained the attack cue. Without this breakdown, the mechanism's contribution is confounded with the well-known sensitivity of attacks to cropping.
  5. [Appendix D.6, Table 13] The comparison with MirrorCheck optimizes its detection threshold on the test data 'to achieve its theoretical upper-bound performance in the wild.' This is test-set leakage and makes the comparison unfair: a defense that is threshold-tuned on the test set is not being evaluated in the same way as LS-DPS, which is used without such tuning. If the intent is to show an oracle upper bound, that should be stated clearly and the threshold should not be used to conclude LS-DPS is superior. A validation-set threshold should be reported for a fair comparison.
minor comments (5)
  1. [Eq. (2)] The summation in Eq. (2) uses D_k, but the set is defined as D_vet; this is likely a typo and should be corrected to match the definition.
  2. [Section 5.5] The text says 'The experimental results are provided in the Appendix 7' but the relevant section is Appendix D.4; the reference should be fixed.
  3. [Section 4.1] The claim that 'questions are not involved in the defense method' is contradicted by the Step 1 and Step 2 prompts in Section 4.2, both of which include the original question. The black-box property (no gradient or internal access) is unaffected, but the wording should be corrected.
  4. [Figure 2] The figure contains the typo 'Qriginal Question' in Adv-Case 5 and Adv-Case 7, and the explanation of the '[Original Question]' placeholder in Clean-Case 1 could be made clearer.
  5. [Table 4] The header of Table 4 should explicitly state that the reported DPS row is LS-DPS, not plain DPS, and Section 5.3 should avoid using 'DPS' when the table reports LS-DPS. This will prevent readers from attributing the standard-performance results to the method without the safety checker.

Circularity Check

1 steps flagged · score 1.0 of 10

Empirical DPS results are self-contained; only the Appendix C 'theoretical justification' restates the mechanism as a formal assertion rather than deriving it.

  1. other [Appendix C, Eqs. (5)-(6)]
    "We have ϕ(T, Tc) ∈ [0, 1] representing the semantic consistency between the candidate text T and Tc. ... and we have Pdefense(Tclean|x′, Tc) > Pdefense(Ttarget|x′, Tc), that is, the text from partial image provide additional information and increase the probability of Tclean significantly."

    The appendix is presented as a theoretical justification, but Eqs. (5)-(6) define the defended probability by multiplying the attacked-image probability with an unmeasured semantic-consistency factor phi, and then assert the desired inequality. The superiority of Tclean over Ttarget under defense is exactly the effect DPS must establish; it is not derived from a model of confidence or from the actual prompting mechanism, but is assumed by assigning phi(Tclean,Tc) a larger value than phi(Ttarget,Tc). Since phi is a free placeholder and no relation between phi and the LVLM's behavior is given, the 'derivation' restates the mechanism rather than proving it.

full rationale

The paper's main claim is an empirical one: DPS reduces attack success rate on six external datasets across several LVLMs, with no parameter fitted to the test data and no reliance on a self-citation chain. The defense pipeline (crop, describe, supervise, answer) is evaluated directly against SmoothVLM, MLLM-Protector, ECSO, IVA, and Warning baselines, so the reported ASR reductions are not forced by construction. The clean-input preservation claim is supported only by qualitative cases in Figure 2, and the paper never reports a quantified clean-input answer-change rate or decomposes residual ASR between cropping failures and ignored supervision; these are evidence gaps that weaken the behavioral premise (and are partly acknowledged in Appendix D.7.4), but they are not circularity. The same model acting as both Part-Perc and Full-Perc supervisor is an analogy to weak-to-strong learning, not a circular derivation. The only circular-adjacent passage is Appendix C, where phi is defined and the desired inequality is asserted rather than proven; because no reported number depends on this appendix, the overall circularity score is 1.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on three assumptions that the paper does not prove: crop-sensitivity of vision attacks, confidence asymmetry between clean and attacked inputs, and a post-hoc probability-reweighting model with an undefined consistency function. The default crop count (3) and the specific crop-strategy combination are hand-selected design choices, ablated but not derived. No new entities are introduced.

free parameters (2)
  • number_of_partial_crops = 3 (default)
    The default uses three partial image copies (center, random, adaptive cropping). Ablation in Table 7 shows defense improves from 1 to 5 crops on Self-Gen, so the choice of 3 is a hand-selected hyperparameter affecting the reported ASR.
  • cropping_strategy_set = {center, random, adaptive}
    The combination of three cropping strategies is a design choice. Ablation Table 3 shows each strategy's contribution varies by dataset, and the choice is not derived from first principles.
assumptions (3)
  • domain assumption Vision attacks against LVLMs are sensitive to image cropping: cropping disrupts the adversarial cue while preserving enough semantics for partial description.
    Adopted from Sun et al. 2024 and Zhang et al. 2024a; demonstrated qualitatively in Figure 2 (Adv-Case 2). The failure mode in Appendix D.7.4 occurs when cropping does not eliminate the attack, confirming this is an assumption, not a guarantee.
  • domain assumption LVLMs show high confidence on clean inputs and reduced confidence under attack, so injected textual supervision is accepted only under attack.
    Section 3.2 defines confidence as inverse standard deviation over prompt variations and shows a handful of qualitative examples (Clean-Case 3/4 vs Adv-Case 3/4). No quantitative confidence measurements are reported across the six datasets, yet the entire mechanism depends on this asymmetry.
  • ad hoc to paper The probability reweighting in Appendix C is a valid model of prompting, with an unspecified semantic consistency function phi.
    Appendix C assumes a semantic-consistency weight phi(T, Tc) multiplies output probabilities and asserts without proof that it raises the clean answer probability above the target. phi is never defined, and the inequality P_defense(T_clean|x',Tc) > P_defense(T_target|x',Tc) is stated as fact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Defending LVLMs Against Vision Attacks through Partial-Perception Supervision." pith.science (2026). https://pith.science/paper/XLBJDASJ

@misc{pith2026241212722,
  author       = {Pith},
  title        = {Pith review of: Defending LVLMs Against Vision Attacks through Partial-Perception Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLBJDASJ}},
  note         = {Machine review of arXiv:2412.12722}
}
read the original abstract

Recent studies have raised significant concerns regarding the vulnerability of Large Vision Language Models (LVLMs) to maliciously injected or perturbed input images, which can mislead their responses. Existing defense methods show that such vision attacks are sensitive to image modifications especially cropping, using majority voting across responses of modified images as corrected responses. However, these modifications often result in partial images and distort the semantics, which reduces response quality on clean images after voting. Instead of directly using responses from partial images for voting, we investigate using them to supervise the LVLM's responses to the original images. We propose a black-box, training-free method called DPS (Defense through Partial-Perception Supervision). In this approach, the model is prompted using the responses generated by a model that perceives only a partial image. With DPS, the model can adjust its response based on partial image understanding when under attack, while confidently maintaining its original response for clean input. Our findings show that the weak model can supervise the strong model: when faced with an attacked input, the strong model becomes less confident and adjusts its response based on the weak model's partial understanding, effectively defending against the attack. With clean input, it confidently maintains its original response. Empirical experiments show our method outperforms the baseline, cutting the average attack success rate by 76.3% across six datasets on three popular models.

Figures

Figures reproduced from arXiv: 2412.12722 by the authors.

Figure 1
Figure 1. The Comparison of DPS and SmoothVLM a ‘weak-to-strong’ phenomenon. This phenomenon demon￾strates that, while weaker models (e.g., an LLM without cer￾tain necessary information) underperform stronger models (e.g., the same LLM with full information access functions) in generalization and other capabilities, they can still col￾laborate to guide and supervise stronger models, enhancing their performance. We draw an ana… view at source ↗
Figure 2
Figure 2. Illustration of the sensitivity and confidence when facing vision attacks, and the preliminary exploration for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The framework of DPS and LLM-Secured DPS [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparing the standard performance of different defense methods on the MM-Vet benchmark [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Examples Showcase Across Six Datasets A. Datasets We provide a detailed introduction to the following datasets used in the experiment, which include the following datasets: Challenging Misleading Datasets: RTA-100 (Azuma & Matsui, 2023) and MultiTrust Misleading Datase…
Figure 6
Figure 6. Figure 6: Standard Performance with Qwen2.5-VL-32B [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: The Warning Prompt enhances the standard performance of Qwen-VL-Plus [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Training-free Alignment of Large Language Models

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.

Reference graph

Works this paper leans on

30 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [3]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923,

    arXiv

  3. [4]

    Cao, Y ., Xing, Y ., Zhang, J., Lin, D., Zhang, T., Tsang, I., Liu, Y ., and Guo, Q

    URL https://arxiv.org/abs/ 2312.09390. Cao, Y ., Xing, Y ., Zhang, J., Lin, D., Zhang, T., Tsang, I., Liu, Y ., and Guo, Q. Scenetap: Scene-coherent ty- pographic adversarial planner against vision-language models in real-world environments,

    arXiv

  4. [6]

    And in Table 10, we demonstrate the statistical significance using a t-test, with p-values consistently less than 0.05, confirming their significance

    The proposed methods, DPS and LS-DPS, consistently achieve the best performance against misleading and jailbreak attacks. And in Table 10, we demonstrate the statistical significance using a t-test, with p-values consistently less than 0.05, confirming their significance. Here, the t-test is performed between DPS and the baseline with the best performance...

    work page 2018

  5. [7]

    Du, X., Ghosh, R., Sim, R., Salem, A., Carvalho, V ., Law- ton, E., Li, Y ., and Stokes, J. W. Vlmguard: Defending vlms against malicious prompts via unlabeled data. arXiv preprint arXiv:2410.00296,

    arXiv

  6. [8]

    B., and Mor- datch, I

    Du, Y ., Li, S., Torralba, A., Tenenbaum, J. B., and Mor- datch, I. Improving factuality and reasoning in lan- guage models through multiagent debate. arXiv preprint arXiv:2305.14325,

    arXiv

  7. [9]

    Mirrorcheck: Efficient adversarial defense for vision-language models

    Fares, S., Ziu, K., Aremu, T., Durasov, N., Tak´aˇc, M., Fua, P., Nandakumar, K., and Laptev, I. Mirrorcheck: Efficient adversarial defense for vision-language models. arXiv preprint arXiv:2406.09250,

    arXiv

  8. [10]

    Figstep: Jailbreaking large vision- language models via typographic visual prompts

    Gong, Y ., Ran, D., Liu, J., Wang, C., Cong, T., Wang, A., Duan, S., and Wang, X. Figstep: Jailbreaking large vision- language models via typographic visual prompts. arXiv preprint arXiv:2311.05608,

    arXiv

Show all 30 references
  1. [11]

    Huang, Y ., Liang, L., Li, T., Jia, X., Wang, R., Miao, W., Pu, G., and Liu, Y

    URL https:// arxiv.org/abs/2402.03749. Huang, Y ., Liang, L., Li, T., Jia, X., Wang, R., Miao, W., Pu, G., and Liu, Y . Perception-guided jail- break against text-to-image models. Proceedings of the AAAI Conference on Artificial Intelligence , 39(25): 26238–26247, Apr

    arXiv

  2. [12]

    doi: 10.1609/aaai.v39i25. 34821. URL https://ojs.aaai.org/index. php/AAAI/article/view/34821. Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rockt¨aschel, T., and Perez, E. Debating with more per- suasive llms lea...

    doi

  3. [13]

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A

    URL https://arxiv.org/abs/2402.06782. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y ., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015–4026,

    arXiv

  4. [14]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000

  5. [15]

    One prompt word is enough to boost adversarial robustness for pre- trained vision-language models

    Li, L., Guan, H., Qiu, J., and Spratling, M. One prompt word is enough to boost adversarial robustness for pre- trained vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24408–24419, 2024a. Li, M., Li, L., Yin, Y ...

    arXiv

  6. [16]

    Interpreting and mitigating hallucination in mllms through multi-agent debate

    Lin, Z., Niu, Z., Wang, Z., and Xu, Y . Interpreting and mitigating hallucination in mllms through multi-agent debate. arXiv preprint arXiv:2407.20505,

    arXiv

  7. [17]

    Com- promising embodied agents with contextual backdoor attacks, 2024a

    Liu, A., Zhou, Y ., Liu, X., Zhang, T., Liang, S., Wang, J., Pu, Y ., Li, T., Zhang, J., Zhou, W., Guo, Q., and Tao, D. Com- promising embodied agents with contextual backdoor attacks, 2024a. URL https://arxiv.org/abs/ 2408.02882. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visua...

    arXiv

  8. [20]

    Qraitem, M., Tasnim, N., Teterwak, P., Saenko, K., and Plummer, B. A. Vision-llms can fool themselves with self-generated typographic attacks. arXiv preprint arXiv:2402.00626,

    arXiv

  9. [21]

    Safe- guarding vision-language models against patched visual prompt injectors

    Sun, J., Wang, C., Wang, J., Zhang, Y ., and Xiao, C. Safe- guarding vision-language models against patched visual prompt injectors. arXiv preprint arXiv:2405.10529,

    arXiv

  10. [22]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530,

    arXiv

  11. [23]

    Drivevlm: The conver- gence of autonomous driving and large vision-language models

    11 Defending LVLMs Against Vision Attacks Through Partial-Perception Supervision Tian, X., Gu, J., Li, B., Liu, Y ., Hu, C., Wang, Y ., Zhan, K., Jia, P., Lang, X., and Zhao, H. Drivevlm: The conver- gence of autonomous driving and large vision-language models. arXiv preprint ...

    arXiv

  12. [24]

    T., Jia, X., Sun, W., Wu, C., Chen, J., Hu, X., Li, Y ., Wang, X., Zhang, N., Tuan, L

    Wang, K., Zhang, G., Zhou, Z., Wu, J., Yu, M., Zhao, S., Yin, C., Fu, J., Yan, Y ., Luo, H., Lin, L., Xu, Z., Lu, H., Cao, X., Zhou, X., Jin, W., Meng, F., Mao, J., Wang, Y ., Wu, H., Wang, M., Zhang, F., Fang, J., Qu, W., Liu, Y ., Liu, C., Zhang, Y ., Li, Q., Guo, C., Qin, Y...

    arXiv

  13. [26]

    URL https://arxiv.org/abs/2407. 13647. Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2308.02490,

    arXiv

  14. [27]

    Jailguard: A uni- versal detection framework for llm prompt-based attacks

    Zhang, X., Zhang, C., Li, T., Huang, Y ., Jia, X., Hu, M., Zhang, J., Liu, Y ., Ma, S., and Shen, C. Jailguard: A uni- versal detection framework for llm prompt-based attacks. arXiv preprint arXiv:2312.10766, 2024a. Zhang, Y ., Huang, Y ., Sun, Y ., Liu, C., Zhao, Z., Fang, Z....

    arXiv

  15. [28]

    RTA-100 is a real-world typographic attack dataset, in which the handwritten tag from incorrect classes is placed next to the objects in the image

    and MultiTrust Misleading Dataset (Zhang et al., 2024b). RTA-100 is a real-world typographic attack dataset, in which the handwritten tag from incorrect classes is placed next to the objects in the image. Whereas, the MultiTrust Misleading dataset contains challenging, visuall...

    2024

  16. [29]

    Both MM-safetyBench and HADES are datasets for evaluating LVLM in safety-critical scenarios

    and HADES (Li et al., 2024d). Both MM-safetyBench and HADES are datasets for evaluating LVLM in safety-critical scenarios. They incorporate jailbreak images, which are generated using diffusion models, and these images are then enhanced with specific typographical additions. O...

    2024

  17. [2012]

    Mllm-protector: Ensuring mllm’s safety without hurting performance

    Pi, R., Han, T., Xie, Y ., Pan, R., Lian, Q., Dong, H., Zhang, J., and Zhang, T. Mllm-protector: Ensuring mllm’s safety without hurting performance. arXiv preprint arXiv:2401.02906,

    arXiv

  18. [2018]

    Feature squeezing: Detecting adversarial examples in deep neural networks

    Xu, W., Evans, D., and Qi, Y . Feature squeezing: Detecting adversarial examples in deep neural networks. In Pro- ceedings 2018 Network and Distributed System Security Symposium. Internet Society,

    2018

  19. [2022]

    M., Vedaldi, A., Zisserman, A., and Jawahar, C

    Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pp. 3498–3505. IEEE,

    2012

  20. [2023]

    Qwen-vl: A frontier large vision- language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966,

    arXiv

  21. [2024]

    S., and Guo, Q

    Chung, N., Gao, S., Vu, T.-A., Zhang, J., Liu, A., Lin, Y ., Dong, J. S., and Guo, Q. Towards transferable attacks against vision-llms in autonomous driving with typogra- phy. arXiv preprint arXiv:2405.14169,

    arXiv

  22. [2025]

    Cheng, H., Xiao, E., Gu, J., Yang, L., Duan, J., Zhang, J., Cao, J., Xu, K., and Xu, R

    URL https: //arxiv.org/abs/2412.00114. Cheng, H., Xiao, E., Gu, J., Yang, L., Duan, J., Zhang, J., Cao, J., Xu, K., and Xu, R. Unveiling typographic deceptions: Insights of the typographic vulnerability in large vision-language model. arXiv. org,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.