arxiv: 2604.16515 · v1 · submitted 2026-04-15 · 💻 cs.CV · cs.CR· cs.LG

Recognition: unknown

Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations

Jiachen Qian , Zhaolu Kang

Authors on Pith no claims yet

Pith reviewed 2026-05-10 13:56 UTC · model grok-4.3

classification 💻 cs.CV cs.CRcs.LG

keywords multimodal agentsadversarial attacksvisual perturbationsprice constraintsCLIP encodershallucinationrobustness

0 comments

The pith

Imperceptible visual changes let multimodal agents override price text and select expensive options.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that multimodal agents processing screenshots for shopping tasks can be led to ignore clear price information through tiny, human-imperceptible image changes. This stems from Visual Dominance Hallucination, in which the visual encoder dominates over textual evidence because of gaps in how CLIP-style models align the two modalities. The authors introduce PriceBlind, a white-box attack that uses a Semantic-Decoupling Loss to steer the image embedding toward low-cost visual anchors while keeping pixel changes minimal. When the claim holds, agents handling financial decisions become vulnerable to producing irrational choices such as buying overpriced items despite explicit price constraints in the text. The work also reports that stronger vision encoders and a verify-then-act check can lower the attack rate, though these steps reduce performance on unaltered inputs.

Core claim

Multimodal agents exhibit Visual Dominance Hallucination in which imperceptible visual perturbations on screenshots override conflicting textual price evidence and produce irrational purchase selections. PriceBlind exploits the modality gap inside CLIP-based encoders by optimizing a Semantic-Decoupling Loss that aligns the perturbed image embedding with low-cost value-associated anchors while preserving pixel-level fidelity. On the E-ShopBench benchmark the method reaches roughly 80 percent attack success in white-box settings; a simplified coordinate-selection protocol yields 35-41 percent transfer success against GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet. Robust encoders and verify-the

What carries the argument

Visual Dominance Hallucination, the tendency of CLIP-based vision-language encoders to let small image changes dominate conflicting text, exploited by PriceBlind's Semantic-Decoupling Loss that pulls embeddings toward low-cost anchors.

If this is right

Price-constrained agents can be induced to violate their limits at high rates when the attacker has white-box access to the vision encoder.
The perturbations transfer across frontier multimodal models under a single-turn coordinate-selection protocol, achieving 35-41 percent success.
Switching to robust vision encoders lowers attack success substantially.
Adding a verify-then-act defense further reduces success rates but introduces a measurable drop in clean-task accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same visual override effect may appear in other agent tasks that combine screenshots with textual rules, such as form-filling or navigation under budget limits.
Multimodal agents may benefit from explicit cross-modal consistency checks that compare visual and textual signals before acting.
Improving vision-language alignment during pretraining could shrink the modality gap that PriceBlind exploits, offering a path to broader robustness.

Load-bearing premise

That Visual Dominance Hallucination is a stable property of the modality gap in CLIP encoders and not an artifact of the particular shopping benchmark or the tested model families.

What would settle it

Apply the same PriceBlind perturbations to a multimodal model whose vision encoder is not CLIP-based and measure whether attack success rate falls near zero while clean accuracy stays comparable.

Figures

Figures reproduced from arXiv: 2604.16515 by Jiachen Qian, Zhaolu Kang.

**Figure 2.** Figure 2: Schematic of the PriceBlind framework. We employ an Ensemble-DI-FGSM strategy attacking multiple open-source surrogates. A key component is the Semantic Decoupling Regularizer (Ldec), which nudges the target-item embedding towards pre-computed low-cost anchor centroids in the surrogate visual embedding spaces. anchor bank. The Semantic Decoupling Loss minimizes the cosine distance between the adversarial … view at source ↗

**Figure 3.** Figure 3: Grad-CAM saliency visualization. In the benign case (Top), the model highlights the OCR price regions. Under PriceBlind attack (Bottom), saliency shifts away from the price text towards the manipulated visual features [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: JPEG robustness curves. PriceBlind maintains [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

read the original abstract

The rapid proliferation of Multimodal Large Language Models (MLLMs) has enabled mobile agents to execute high-stakes financial transactions, but their adversarial robustness remains underexplored. We identify Visual Dominance Hallucination (VDH), where imperceptible visual cues can override textual price evidence in screenshot-based, price-constrained settings and lead agents to irrational decisions. We propose PriceBlind, a stealthy white-box adversarial attack framework for controlled screenshot-based evaluation. PriceBlind exploits the modality gap in CLIP-based encoders via a Semantic-Decoupling Loss that aligns the image embedding with low-cost, value-associated anchors while preserving pixel-level fidelity. On E-ShopBench, PriceBlind achieves around 80% ASR in white-box evaluation; under a simplified single-turn coordinate-selection protocol, Ensemble-DI-FGSM transfers with roughly 35-41% ASR across GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet. We also show that robust encoders and Verify-then-Act defenses reduce ASR substantially, though with some clean-accuracy trade-off.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Visual perturbations override prices in MLLM agents on E-ShopBench at 80% white-box ASR, but the effect may be tied to the benchmark and simplified protocol rather than a general CLIP gap.

read the letter

Colleague, this paper shows that small visual changes to screenshots can make multimodal agents ignore price text and pick higher-cost options. They name the issue Visual Dominance Hallucination and build PriceBlind around a Semantic-Decoupling Loss that pulls the CLIP image embedding toward low-price anchors while limiting pixel changes. The white-box results on E-ShopBench hit around 80% attack success rate, which is the strongest part of the evidence. They also run transfer tests with Ensemble-DI-FGSM under a single-turn coordinate-selection setup and get 35-41% ASR on GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet. Including defense checks with robust encoders and a Verify-then-Act step is helpful, since it shows mitigation is possible even if clean accuracy drops a bit. The softer area is the claim that this comes from a broad modality gap in CLIP encoders. The results could instead reflect E-ShopBench screenshot layouts, price rendering, or the narrow single-turn protocol. Without tests on varied image styles, full multi-turn agent loops, or other benchmarks, it is hard to tell how much the finding applies to realistic financial agents. The transfer numbers are modest, so the practical threat level is still unclear. The work engages with existing adversarial MLLM literature in a straightforward way and avoids circular definitions. It is aimed at people studying robustness for multimodal agents in e-commerce or finance. The idea and the empirical starting point are solid enough to send for peer review, though referees will likely ask for more generality checks and full protocol details with error bars.

Referee Report

2 major / 1 minor

Summary. The paper claims to discover Visual Dominance Hallucination (VDH) in MLLMs, where visual adversarial perturbations override textual price evidence in screenshots, leading to irrational agent decisions. PriceBlind uses a Semantic-Decoupling Loss to align image embeddings with low-cost anchors while maintaining fidelity, achieving ~80% white-box ASR on E-ShopBench and 35-41% transfer ASR to GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet under a simplified single-turn coordinate-selection protocol. Defenses are shown to reduce ASR with some accuracy trade-off.

Significance. Should the empirical findings prove robust and the VDH effect generalize, the work would be significant in exposing modality-specific vulnerabilities in multimodal agents for financial tasks. The transfer attack results across commercial models underscore practical risks, and the proposed loss function offers a targeted approach to exploiting CLIP encoder gaps. However, the current presentation limits the ability to fully assess its contribution to the field.

major comments (2)

[Abstract] The abstract reports specific ASR percentages but supplies no experimental details, baselines, error bars, or full protocol. The central claim of effective bypassing cannot be verified from the given text, and transfer results are limited to a simplified protocol.
[Experiments] The assumption that VDH is a robust, general phenomenon driven by the modality gap in CLIP encoders is not adequately supported; the results could be artifacts of the E-ShopBench benchmark, price rendering, or the simplified decision protocol. Additional ablation studies on varied layouts and multi-turn interactions are needed to substantiate the broader implications.

minor comments (1)

[Abstract] Consider clarifying the definition of 'imperceptible' perturbations and providing quantitative measures of visual fidelity (e.g., PSNR or LPIPS scores) to support the stealth claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight opportunities to strengthen the presentation and empirical support for Visual Dominance Hallucination (VDH). We will revise the manuscript accordingly, with targeted improvements to the abstract and additional experiments to address concerns about generality.

read point-by-point responses

Referee: [Abstract] The abstract reports specific ASR percentages but supplies no experimental details, baselines, error bars, or full protocol. The central claim of effective bypassing cannot be verified from the given text, and transfer results are limited to a simplified protocol.

Authors: We agree that the abstract, due to its brevity, does not include full experimental details or error bars, which are instead provided in the main body (Sections 4.1–4.3 and Tables 1–3). The protocol is explicitly described as a simplified single-turn coordinate-selection setup to enable controlled evaluation of the attack. In the revision, we will expand the abstract to briefly note the E-ShopBench benchmark, white-box and transfer settings, and the single-turn limitation, while directing readers to the full protocol and results for verification. revision: partial
Referee: [Experiments] The assumption that VDH is a robust, general phenomenon driven by the modality gap in CLIP encoders is not adequately supported; the results could be artifacts of the E-ShopBench benchmark, price rendering, or the simplified decision protocol. Additional ablation studies on varied layouts and multi-turn interactions are needed to substantiate the broader implications.

Authors: We acknowledge that the current evaluation is centered on E-ShopBench with specific price renderings and a single-turn protocol, which isolates the VDH effect but limits claims of full generality. The transfer results to GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet provide some evidence that the modality gap in CLIP-based encoders contributes beyond the benchmark. In the revised manuscript, we will include new ablation studies on varied layouts, alternative price renderings, and different screenshot styles. We will also add discussion of multi-turn extensions, including preliminary experiments where feasible, to better support that VDH arises from the modality gap rather than protocol artifacts. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical attack with independent loss and benchmark evaluation

full rationale

The paper identifies VDH via observation of MLLM behavior on screenshots, introduces a new Semantic-Decoupling Loss to target the CLIP modality gap, and measures ASR directly on E-ShopBench plus transfer settings. No equations, definitions, or self-citations reduce the reported success rates to fitted inputs or tautological constructions; the central results are protocol-specific empirical outcomes rather than derivations that collapse to the method's own assumptions.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the existence of a modality gap in CLIP encoders and the realism of the E-ShopBench benchmark. No free parameters are explicitly fitted in the abstract. VDH is introduced as a new descriptive entity without independent falsifiable evidence beyond the attack results.

axioms (1)

domain assumption CLIP-based encoders exhibit a modality gap exploitable by semantic alignment to low-cost anchors
Invoked to justify the Semantic-Decoupling Loss design.

invented entities (1)

Visual Dominance Hallucination (VDH) no independent evidence
purpose: To name and explain the observed override of text price evidence by visual cues
New term coined in the abstract to frame the vulnerability; no external evidence provided.

pith-pipeline@v0.9.0 · 5500 in / 1367 out tokens · 28113 ms · 2026-05-10T13:56:08.008866+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages · 1 internal anchor

[1]

Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

Dissecting adversarial robustness of multi- modal LM agents. InICLR. Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. 2019. Im- proving transferability of adversarial examples with input diversity. InCVPR. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting u...

work page internal anchor Pith review arXiv 2019
[2]

Visual Simplicity:Products with minimal branding, simple packaging
[3]

Buy the item strictly under $50

Product Separation:Avoid direct overlap in exact products or screenshots used in E- ShopBench I.2 Category Distribution • Office supplies (pens, notebooks, folders): 150 images • Basic kitchenware (plastic containers, uten- sils): 150 images • Generic household items (cleaning supplies, storage boxes): 200 images I.3 Visual Characteristics These items typ...