REVIEW 2 major objections 5 minor 27 references
Multimodal GUI agents often form textual state beliefs from the serialized structure (DOM or accessibility tree) rather than from the rendered pixels, even when they perceive the pixels correctly, and a single such mis-sourced belief can do
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Across four open-weight models and three OpenAI models, textual state beliefs of GUI agents follow a conflicting structured value on 30–75% of probes where the same model reads the pixels correctly, so agent state beliefs are frequently structure-sourced, not pixel-grounded.
T0 review reviewed 2026-08-02 challenge →
load-bearing objection Strong synthetic core, but the natural zero-edit prevalence claim rests on model-annotated gold unless Appendix J is clarified; worth a serious referee with a human re-audit demanded. the 2 major comments →
Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central discovery is that a GUI agent's textual state belief is often a copy of the serialized structure, not a reading of the pixels, even when the pixels are perceived correctly. The paper defines the Perception-Fusion Gap (PFG) as the fraction of probes where a model answers correctly from pixels alone yet, when the same screenshot is paired with a conflicting structural value, resolves the belief toward the structure. Across four open-weight models and three API models, PFG is positive on web-text probes (0.30–0.75), and a stricter crop-verified variant leaves the gap intact. The effect extends to natural zero-edit divergences: on stale snapshots from live pages, models follow the ou
What carries the argument
The central object is the Perception-Fusion Gap (PFG), the fraction of probes in which a model perceives the correct pixel value in isolation yet answers with the conflicting structural value when both channels are present. It is computed from paired single-channel interventions: the structure-swap condition edits only the serialized structure while keeping the screenshot untouched, and the pixels-only condition verifies perception. A strict variant (PFGstrict) re-verifies perception on a tight crop of the target region to rule out page-context artifacts. The paired interventions allow causal attribution of the belief to the structure channel.
Load-bearing premise
The load-bearing premise is that the ground-truth interface state is always defined by the pixels—the value a human sees on screen—so that following the structure under conflict is always an error; if the structure is sometimes the authoritative channel (hidden state, accessibility-only content, or a stale screenshot), the Perception-Fusion Gap would overstate the belief-source problem.
What would settle it
Take a set of real GUI states where the serialized structure contains a value that is not rendered but is the semantically correct state (e.g., an input field's value attribute after a script update, or an aria-label that is the intended accessible name). If humans, shown only the pixels, cannot determine the correct value but the structure provides it, count those probes. If a model follows the structure and humans judge it correct, the pixel-gold assumption fails and PFG would flag correct behavior as a fusion failure. Alternatively, a direct falsifier of the dependency claim: find any off-t
If this is right
- If correct, a GUI agent can have a correct visual read and still act on a structurally copied wrong value; the live episodes show one such first-step belief leads to task failure in at least 97% of multi-step cases.
- The effect is strongest for textual state; non-text identity like icons and widget types remains largely pixel-bound in capable models, so the mechanism is not pure OCR or string matching.
- Specialist coordinate-emitting agents stay visually grounded under conflict, while structure-indexed agents are hijacked, showing the failure follows the action channel rather than the model class.
- Prompt-level cues that work at the belief level fail at the action level; certificate checks buy safety only through refusals; a training-free consistency gate that re-grounds on pixels is the only compared mitigation that lowers both hijack and task error.
- Scaling does not close the gap: one larger model repaired visibility grounding on ghost nodes yet worsened text-value deference.
Where Pith is reading between the lines
- A testable extension the paper leaves implicit: if a model is trained or prompted to output an action coordinate rather than an element ID, the same belief conflict should produce far less structure-following; the paper's action-format sweep already hints at this, and a controlled trial would make it a design principle.
- The pixel-gold assumption implies that any deployment where the structure is the authoritative channel (hidden state, accessibility-only content, or a screenshot that is itself stale) would penalize correct structure-following behavior; a 'structure-gold' variant of the benchmark would map where the two interpretations diverge.
- The severity analysis suggests natural conflicts are at least as harsh as synthetic ones; one could mine a larger corpus of stale snapshots with timestamps to estimate how often production agents actually face such conflicts per hour of operation.
- The consistency gate's success at re-grounding on pixels suggests a general architecture: maintain a belief plus a source tag, and when a conflict is detected, recover by searching the pixels for the goal; this could generalize beyond GUI agents to any multimodal system with redundant input channels.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a diagnostic for whether multimodal GUI agents' textual state beliefs are sourced from rendered pixels or from serialized structure (DOM/accessibility tree). It builds a 735-probe benchmark with paired single-channel interventions, including structure-swap conflicts, natural zero-edit divergences mined from live websites, and stale mobile-tree nodes, and scores all answers by deterministic forced choice. The central metric, the Perception-Fusion Gap (PFG), measures the fraction of probes a model perceives correctly in a pixels-only condition yet resolves toward a conflicting structural value. Across four open-weight and three API models, the paper reports positive PFG values on web text (0.30–0.75), a crop-verified strict variant within 0.05, structure-following rates of 0.38–0.88 on stale natural web snapshots, white-box ablation localizing the effect to a single copied structural value, live action and multi-step failure propagation, and a four-way mitigation comparison in which only a consistency gate reduces both hijack and task error.
Significance. If the result holds, this is a genuinely useful measurement of belief provenance rather than another end-to-end task-success benchmark. The design has real strengths: paired interventions control one channel at a time; PFG is an interventional measurement with no fitted parameters; the strict crop-verified variant guards the perception-conditioning loophole; cluster-robust confidence intervals exclude zero for every model; the human baseline and label-noise sensitivity support the construct; and the white-box ablation with random-token controls is a clean causal test. Reproducibility is also a strength: the core claims are established on four fixed open-weight checkpoints, with deterministic parsing, temperature-0 decoding, and released run manifests. The paper is transparent about several limitations, but one of those limitations interacts with a headline real-world claim and needs to be addressed before acceptance.
major comments (2)
- [Benchmark Construction (Gold-label audit); Table 3; Discussion and Limitations] The zero-edit real-world leg is load-bearing for the abstract's 'up to 0.88' claim, but its gold labels are not human-verified. The Gold-label audit describes 'two independent annotators' without saying 'human', and the Limitations state that 'the expanded families use model annotators under the same two-annotator pixel-only protocol; the decisive mobile-stale subset and the human baseline rest on human annotation.' Thus the 182-probe web-natural audit-consensus subset, including the 42 stale probes behind Table 3, likely has model-annotated gold. If the model annotators share the tested models' difficulty with small ticker text or their structural bias, the consensus filter can select a biased subset and inflate structure-following. Please either report a human two-annotator audit for the web-natural family (at minimum the 42 stale probes) or remove/downgrade the 'up to 0.88' claim from
- [Problem Formulation; Eq. (1)] The metric defines gold strictly by pixels and labels any structure-following answer as a fusion failure. This is a deliberate and transparent modeling choice, and the human baseline shows the convention is reasonable for visible text. However, because the paper's real-world framing suggests deployment errors, the main text should more explicitly state that PFG is a diagnostic under the pixel-gold convention, and that in cases where the structure is the only authoritative channel (for example, accessibility-only content or a stale screenshot), structure-following would not be a failure. The current discussion in the Problem Formulation is one sentence; a short paragraph in the Discussion would prevent over-interpretation.
minor comments (5)
- [Appendix J vs. Discussion and Limitations] Please disambiguate 'annotator' throughout: state explicitly which annotation steps use humans and which use model annotators. The current wording creates an internal ambiguity about the web-natural gold labels.
- [Table 3 caption] The caption should state that the 182-probe audit-consensus subset is based on model annotators under the pixel-only protocol unless a human audit is added. This is necessary for readers to assess the stale-row estimates.
- [Table 2] Use consistent notation for the strict variant (pfg^strict or pfg_str) across tables and the text; the current 'pfg str' is easy to misread.
- [Abstract and Contributions] The abstract says '225 zero-edit divergences,' but the headline natural results are computed on the 182-probe audit-consensus subset. Please state this distinction in the abstract or contributions so the reader is not misled about the sample size behind the 0.38–0.88 estimate.
- [Experiments (Gradient attribution)] The gradient attribution result (0.028 vs. 0.034) is suggestive but not strong on its own; the text already relies mainly on the behavioral and white-box ablations. Consider softening the wording or presenting the attribution as corroborative rather than independent evidence.
Circularity Check
No significant circularity: PFG is an interventional measurement with no fitted parameters, and the only self-citation is non-load-bearing.
full rationale
The central derivation chain is not circular. PFG (Eq. 1) is computed directly from deterministic forced-choice answers under paired single-channel interventions; the perception-correct subset C is defined from the same model's pixels-only read, and the stricter crop-verified variant closes the residual loophole. No quantity entering PFG is fitted from the data being explained, and no 'prediction' is equivalent by construction to an input: the structure-swap gold is rule-based on the rendered text, and the web-natural leg is scored on an audit-consensus subset with a human baseline showing sfr≈0.03. The explicit stance that 'gold is always defined by the pixels' is an operationalization of the construct, not a circular derivation. The only self-citation is to the authors' certificate-check paper, used as a mitigation baseline ('certificate check in the spirit of action-level defenses (Zhang, Zheng, and Yang 2026)'), and it is not load-bearing evidence for the main effect. The Limitations note that 'the expanded families use model annotators under the same two-annotator pixel-only protocol; the decisive mobile-stale subset and the human baseline rest on human annotation' is a real validity caveat for the zero-edit natural leg, but it concerns gold-label verification, not a definitional reduction of the metric to its inputs. The score of 2 reflects only this minor self-citation and the acknowledged annotation caveat; there is no circular step in the derivation of the Perception-Fusion Gap.
Axiom & Free-Parameter Ledger
free parameters (4)
- audit-consensus scoring subsets =
web-natural 182/225; mobile-stale 114/250; desktop 46/60
- material divergence threshold in live mining =
no containment relation after normalization
- fixed page-update delay before screenshot =
not reported numerically
- distractor selection for structure-swap edits =
on-page string of the same broad type
axioms (6)
- domain assumption Pixel gold: ground truth is the value a human sees on screen, not the DOM/accessibility tree value.
- domain assumption Paired single-channel interventions identify causation: only one channel changes per condition.
- domain assumption Forced-choice JSON answers are a faithful measure of the agent's belief.
- domain assumption The four open-weight and three OpenAI models are representative of current GUI agents.
- domain assumption Human pixel audits are a valid gold standard.
- domain assumption Non-zero gradient attribution on visual tokens means visual evidence is processed.
Cite this review
Pith. "Pith review of Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure." pith.science (2026). https://pith.science/paper/IIYWYJR3
@misc{pith2026260704334,
author = {Pith},
title = {Pith review of: Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure},
year = {2026},
howpublished = {\url{https://pith.science/paper/IIYWYJR3}},
note = {Machine review of arXiv:2607.04334}
}
read the original abstract
Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a document object model or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing benchmarks score task success, element grounding, or attack resistance and do not ask whether that belief is drawn from the pixels. We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors, and measure it with paired single-channel interventions over 735 probes spanning real web, mobile, and desktop interfaces, of which 225 are zero-edit divergences mined from live production websites, all scored by deterministic forced choice with no model judge. Our central metric is the Perception-Fusion Gap (PFG), the fraction of probes a model perceives correctly yet resolves toward structure under conflict; a stricter variant that re-verifies perception on a tight crop of the target region leaves the gap intact. Across models from four vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and on unedited stale snapshots from live pages the same models follow the outdated structure on up to 0.88 of probes. A white-box ablation traces the textual effect to a single copied structural value, and gradient attribution shows the visual evidence is processed yet overridden. In live multi-step environments, one mis-sourced belief at the first step compounds into task failure with a self-recovery rate of at most 0.03. Comparing four mitigations on identical probes, prompt-level cues fail at the action level, certificate checks buy safety with refusals, and a training-free consistency gate is alone in reducing both hijack and task error. Visual state reliance thus gives a measurable diagnostic of whether agent state beliefs are visually grounded.
Figures
Reference graph
Works this paper leans on
-
[3]
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. InACL. ArXiv:2401.10935. Deka, B.; et al
-
[6]
WebVoyager: Building an End-to- End Web Agent with Large Multimodal Models. InACL. ArXiv:2401.13919. Hong, W.; Wang, W.; Lv, Q.; Xu, J.; Yu, W.; Ji, J.; Wang, Y.;Wang,Z.;Dong,Y.;Ding,M.;andTang,J.2024. CogA- gent: A Visual Language Model for GUI Agents. InCVPR. ArXiv:2312.08914. Jia,H.;Liao,J.;Zhang,X.;Xu,H.;Xie,T.;Jiang,C.;Yan,M.; Liu,S.;Ye,W.;andHuang,F...
Pith/arXiv arXiv 2024
-
[7]
VisualWebArena: Evaluating Mul- timodal Agents on Realistic Visual Web Tasks. InACL. ArXiv:2401.13649. Kuntz, T.; Duzan, A.; Zhao, H.; Croce, F.; Kolter, Z.; Flam- marion, N.; and Andriushchenko, M
-
[8]
arXiv preprint arXiv:2506.14866
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents. arXiv preprint arXiv:2506.14866. Li, G.; et al
-
[9]
ScreenSpot-Pro: GUI Grounding for ProfessionalHigh-ResolutionComputerUse.arXivpreprint arXiv:2504.07981. Lin, K. Q.; Li, L.; Gao, D.; Yang, Z.; Wu, S.; Bai, Z.; Lei, W.; Wang, L.; and Shou, M. Z
-
[10]
ShowUI: One Vision- Language-Action Model for GUI Visual Agent. InCVPR. ArXiv:2411.17465. Liu,C.;Xu,Z.;Wei,Q.;Wu,J.;Zou,J.;Wang,X.E.;Zhou, Y.;andLiu,S.2025.MoreThinking,LessSeeing?Assessing Amplified Hallucination in Multimodal Reasoning Models. arXiv preprint arXiv:2505.21523. Liu, E. Z.; Guu, K.; Pasupat, P.; Shi, T.; and Liang, P
Pith/arXiv arXiv 2025
-
[12]
Lu, Y.; Yang, J.; Shen, Y.; and Awadallah, A
Visual Confused Deputy: Exploiting and Defend- ing Perception Failures in Computer-Using Agents.arXiv preprint arXiv:2603.14707. Lu, Y.; Yang, J.; Shen, Y.; and Awadallah, A
-
[13]
Lu, Y.; Zhao, M.; Ju, T.; Yan, Z.; Ma, X.; Guo, Y.; Ding, D.; Liu, G.; and Zhang, Z
Om- niParser for Pure Vision Based GUI Agent.arXiv preprint arXiv:2408.00203. Lu, Y.; Zhao, M.; Ju, T.; Yan, Z.; Ma, X.; Guo, Y.; Ding, D.; Liu, G.; and Zhang, Z
-
[14]
Muryn, V.; Sumyk, M.; Hirna, M.; Garkot, S.; and Sham- rai, M
EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environ- mental Injection Attacks.arXiv preprint arXiv:2505.14289. Muryn, V.; Sumyk, M.; Hirna, M.; Garkot, S.; and Sham- rai, M
-
[15]
Screen2AX: Vision-Based Approach for Au- tomatic macOS Accessibility Generation.arXiv preprint arXiv:2507.16704. Park, J.; and Shim, H
-
[16]
Grounding Driving VLA via Inverse Kinematics.arXiv preprint arXiv:2605.21061. Qin, Y.; et al
-
[17]
UI-TARS: Pioneering Automated GUI Interaction with Native Agents.arXiv preprint arXiv:2501.12326. Rahman, M. A.; Rahman, M. A.; Samin, N. H.; Arean, A. I. H.; and Noshin, J. A
-
[18]
Step-Level Visual Ground- ingFaithfulnessPredictsOut-of-DistributionGeneralization in Long-Horizon Vision-Language Models.arXiv preprint arXiv:2603.06828. Rawles, C.; et al
-
[19]
AndroidWorld: A Dynamic Bench- marking Environment for Autonomous Agents.arXiv preprint arXiv:2405.14573. Schiepanski, T. M.; and Piël, N
-
[20]
arXiv preprint arXiv:2508.04412
Beyond Pixels: Ex- ploring DOM Downsampling for LLM-Based Web Agents. arXiv preprint arXiv:2508.04412. Takeshita, M.; Kawada, T.; Ohashi, T.; Kitada, S.; and Iy- atomi, H
-
[21]
Wu,Q.;Cheng,K.;Yang,R.;Zhang,C.;Yang,J.;Jiang,H.; Mu, J.; Peng, B.; Qiao, B.; Zheng, L.; et al
WebInject: Prompt Injection Attack to Web Agents.arXiv preprint arXiv:2505.11717. Wu,Q.;Cheng,K.;Yang,R.;Zhang,C.;Yang,J.;Jiang,H.; Mu, J.; Peng, B.; Qiao, B.; Zheng, L.; et al. 2025a. GUI- Actor: Coordinate-Free Visual Grounding for GUI Agents. arXiv preprint arXiv:2506.03143. Wu, Z.; Wu, Z.; Xu, F.; Wang, Y.; Sun, Q.; Jia, C.; Cheng, K.;Ding,Z.;Chen,L.;...
-
[22]
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environ- ments. InNeurIPS. ArXiv:2404.07972. Xu, Y.; Wang, Z.; Wang, J.; Lu, D.; Xie, T.; Saha, A.; Sa- hoo, D.; Yu, T.; and Xiong, C
-
[23]
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. InICML. ArXiv:2412.04454. Yuan, Z.; et al
-
[24]
Zhang, G.; Zheng, H.; and Yang, H
Scaling Computer-Use Grounding via UserInterfaceDecompositionandSynthesis.arXivpreprint arXiv:2505.12370. Zhang, G.; Zheng, H.; and Yang, H
-
[25]
Zheng, B.; Gou, B.; Kil, J.; Sun, H.; and Su, Y
Hallucination as Exploit: Evidence-Carrying Multimodal Agents.arXiv preprint arXiv:2605.19192. Zheng, B.; Gou, B.; Kil, J.; Sun, H.; and Su, Y
-
[26]
GPT- 4V(ision)isaGeneralistWebAgent,ifGrounded. InICML. ArXiv:2401.01614. Technical Appendix This appendix carries the complete experimental protocols and results behind the main text. Appendix A details the interventions, Appendix B the construction pipeline includ- ing the live-site mining and the severity distributions, Ap- pendix C the full per-model ...
-
[27]
err” is the text-swap / stale-node action error, “SF
Why we do not report a saliency ratio.We also com- puted a structure-over-pixel attribution ratio, the structure shareofsummedandper-tokengradient×inputandsquared- gradient (Fisher) attribution over the image and structure token spans. This quantity is unreliable here: image tokens, whicharecontinuouspost-vision-towerfeatures,andtextto- kens,whichareembed...
2024
-
[2018]
ReinforcementLearningonWebInterfacesUsingWorkflow- Guided Exploration. InICLR. ArXiv:1802.08802. Liu, X.; He, B.; Liu, X.; Luo, A.; Zhang, H.; and Chen, H
-
[2023]
Mind2Web: Towards a Generalist Agent for the Web. InNeurIPS. ArXiv:2306.06070. Gou,B.;etal.2025.NavigatingtheDigitalWorldasHumans Do: Universal Visual Grounding for GUI Agents. InICLR. ArXiv:2410.05243. Guo, M.; Jiao, Q.; Shi, Z.; Quan, Y.; Zhang, B.; Li, D.; Che, L.; Xu, W.; Liu, S.; Liu, Z.; Kapadia, M.; Pavlovic, V.; Liu, J.; Wang, M.; Shi, Y.; Metaxas...
Pith/arXiv arXiv 2025
-
[2024]
GUI-World: A Video Benchmark and DatasetforMultimodalGUI-OrientedUnderstanding.arXiv preprint arXiv:2406.10819. Cheng, K.; et al
-
[2025]
VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents.arXiv preprint arXiv:2506.02456. Chen, D.; et al
-
[2026]
He, H.; Yao, W.; Ma, K.; Yu, W.; Dai, Y.; Zhang, H.; Lan, Z.; and Yu, D
MemEye: A Visual-Centric Evaluation Framework for Mul- timodal Agent Memory.arXiv preprint arXiv:2605.15128. He, H.; Yao, W.; Ma, K.; Yu, W.; Dai, Y.; Zhang, H.; Lan, Z.; and Yu, D
This paper was first reviewed by deepseek-v4-flash on August 2, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.