Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Computer-use models can see desktop screens but cannot reliably reconstruct how and why the screen changed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:45 UTC pith:ALLL3IXF

load-bearing objection A useful diagnostic benchmark with honest reporting; the 37% filter and unreleased data are the real conditions, not fatal flaws. the 4 major comments →

arxiv 2607.26041 v2 pith:ALLL3IXF submitted 2026-07-28 cs.AI cs.CV

Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?

Abhishek Pillai , Samir Kumar Nayak , Yuan Chen This is my paper
classification cs.AI cs.CV
keywords computer-use agentsGUI state-transition understandingtemporal orderingaction-effect predictionbefore-after pairsvisual groundingbenchmarkdesktop automation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Desktop-Delta Bench (DDB) is a benchmark designed to isolate one capability that end-to-end agent scores hide: whether a computer-use model can reconstruct the causal, task-relevant transition produced by an action. It contains 2,013 human-verified samples from office workflows on a Linux desktop, split between two offline tasks: ordering three screenshots by chronology (sometimes with one decoy screenshot from another recording) and inferring which of five action families, with what payload, produced a before-after pair. Across eight model families, ordering accuracy tops out near 65%, far below saturation, and adding the task instruction helps reject decoys but not order correctly. Single-action results show the bottleneck is identifying the action, not locating it: click F1 reaches 0.96 while drag F1 is 0.76. The authors position DDB as a diagnostic layer between single-frame grounding and final task success, with the goal of steering fixes toward verification, reliability, and recovery.

Core claim

The paper's central claim is that current computer-use agents lack a measurable, model-level ability to understand GUI state transitions, and that this gap is invisible to end-task and single-frame benchmarks. DDB supplies the missing measurement: 2,013 human-reviewed instances from multi-app Linux desktop trajectories, scored offline. In temporal ordering, models must pick the correct chronology among shuffled screenshots and optionally flag a cross-trajectory decoy; the best exact-match rates are 65.1% (no decoy) and 65.7% (decoy), with a presentation-order copying bias behind many errors. In single-action reconstruction, models must name the action family and its structured payload from o

What carries the argument

The central object is Desktop-Delta Bench (DDB), an offline benchmark whose unit of analysis is the 'delta'—the difference between desktop observations. Its two tasks are (1) 3-frame temporal ordering over six output permutations (twelve when a decoy may be present), which forces transitive consistency across pairwise relations and models stale or poisoned context, and (2) single-action reconstruction from before/after frames, scored separately for family, non-spatial payload, and continuous coordinates. The key design choice is the asynchronous observation model: frames are sampled across unobserved intermediate states, and human review keeps only samples with one defensible answer, discard

Load-bearing premise

The load-bearing premise is that every retained DDB instance has a single correct, human-verifiable answer; the paper reports discarding 37.3% of raw candidates as ambiguous or multi-action during human review, so the benchmark measures only unambiguous desktop transitions.

What would settle it

Give an optical-flow or pixel-difference baseline the same before-after pairs and ordering triples: if it matches or beats the best vision-language model, DDB is measuring low-level screen change, not causal transition understanding. Alternatively, re-annotate a random subsample of the 2,013 instances with fresh reviewers; if disagreement approaches the model error rates, part of the reported gap is label noise.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the ordering results hold, current desktop agents cannot reliably reject stale or unrelated observations, so context poisoning is a live failure mode even when the final task sometimes succeeds.
  • Task context improves decoy identification by 6.9 percentage points while lowering non-decoy exact match by 2.2 points, so richer prompting helps trajectory membership, not chronology.
  • Click localization is close to solved (F1 0.96) while drag recognition is not (0.76), so the next bottleneck is action-family and payload recovery, not coordinate grounding.
  • An offline, human-verified benchmark of this kind lets developers attribute failures to family, payload, or spatial error without running full agent rollouts.
  • The measured presentation-order echo (two models produce A→B→C in 37.7–45.6% of failures) implies that some models are substantially guessing the displayed label order.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If real desktop transitions are often ambiguous or delayed, DDB's filtered samples may set an upper bound on model skill; a version with longer inter-frame gaps or multi-causal changes would likely show larger gaps.
  • The small effect of reasoning settings (≤4.5 percentage points) suggests the ordering bottleneck is perceptual rather than deliberative; training on synthetic or recorded transition pairs may help more than scaling inference.
  • A cheap check: if a pixel-difference or optical-flow baseline matched the best vision-language models on DDB, the benchmark would be capturing low-level visual change rather than semantic transition understanding.
  • The decoy task models a single poisoning event; extending DDB to multiple decoys or to K>3 frames would test whether errors compound as context accumulates over a long horizon.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Desktop-Delta Bench (DDB), an offline step-level benchmark for evaluating whether computer-use models understand desktop GUI state transitions. DDB contains 2,013 human-verified instances from novel Ubuntu desktop recordings across ~15 applications and 50 task domains. It defines two tasks: temporal ordering of three screenshots (463 instances, of which 105 include a cross-trajectory decoy) and single-action reconstruction from before/after frame pairs (1,550 instances, with five action families and structured payloads). The authors evaluate 8 model families over 48 configurations and report that ordering is far from saturated (best non-decoy and decoy exact match 65.1% and 65.7%, respectively), that task context improves decoy identification but not chronology, that closed-source models outperform open-source ones by a large margin, and that action-family inference is harder than spatial localization (click F1 0.96 vs. drag F1 0.76). The paper positions DDB as a diagnostic layer between GUI grounding and end-to-end task success.

Significance. If the benchmark's validity holds, DDB fills a genuine gap: existing benchmarks largely score end-task success or single-frame grounding, whereas DDB isolates transition-level understanding with controlled, human-reviewed Linux desktop trajectories. Strengths of the paper include the targeted failure dimensions (state verification, source tracking, context-aware control), the decoy design that does not cue the model, the structured payload space separating action family from spatial/parameter reconstruction, and the careful reporting of per-configuration results in the appendix. The comparison with EvoGUI is informative and appropriately scoped. However, the external-validity claims currently rest on a filtering step that discards 37.3% of raw candidates as ambiguous, and the evaluation protocol uses best-of-three sampling at temperature 1 without variance estimates. These issues are load-bearing for the central claim that DDB diagnoses how models handle the asynchronous, noisy transitions that motivate the benchmark. The paper is a useful contribution to the community, but the validity of its headline quantitative comparisons needs to be established more rigorously.

major comments (4)
  1. [§4.3, §3.2–3.3] The benchmark's external validity rests on each retained instance having a unique, defensible gold label. Section 4.3 reports that 1,196 of 3,209 raw candidates (37.3%) were discarded as ambiguous or multi-action, with single-action candidates reduced from 2,732 to 1,550 (43.3% discard). No inter-annotator agreement, adjudication statistics, or characterization of the discarded instances is provided; the text describes only that 'every candidate mined by one collector, receives 1 of 3 statuses from the second.' The paper's own motivation (Sections 3.2–3.3) stresses delayed, occluded, and concurrently caused transitions, yet the filtering removes exactly those cases. If models behave differently on the excluded ambiguous or multi-cause transitions, the headline gaps (click vs. drag F1, ordering exact match, closed/open gap) may not generalize to the real desktop conditions the benchmark t
  2. [§6.1] The evaluation protocol decodes at temperature 1 and retains the best of up to 3 retries per sample. Under stochastic decoding, best-of-3 is an order statistic, not a typical model output, and it will systematically inflate accuracy relative to single-sample or greedy decoding. No confidence intervals or standard errors are reported, so the precise magnitudes in the headline comparisons (e.g., 65.1% vs. 65.7%, 22.9 pp closed/open gap, 0.96 vs. 0.76 click/drag F1) are not established. This also affects claims like 'reasoning affects performance minimally (all |Δ| ≤ 4.5 pp)' in §6.2.1. Please report average results over multiple seeds (or greedy decoding) with variance estimates, and clarify how the best-of-3 rule interacts with the reported ties, e.g., Gemini 3.6 Flash's identical pairwise scores in Table A1.
  3. [§4.3, Appendix A.4] The gold-label review is not independent of the recorded action. The review interfaces described in A.4 expose 'gold chronology, decoy assignment, explanation...' and 'normalized action overlays, the structured target' to the reviewer. When the reviewer decides whether a pair or triple is 'unambiguous,' they necessarily know the action that produced the transition. This can bias the retained set toward transitions whose effect is obvious given the known action, and it does not validate that a human viewing only the frames would agree on a single answer. Please conduct a blind annotation study (annotators see only the frames, not the action labels), report agreement, and discuss whether the retained instances are uniquely identifiable from the visual evidence alone. This is important because the benchmark's central claim is that each instance has exactly one defensible answer.
  4. [§6.3, Table A2-a] Some per-family comparisons rest on very small samples. For example, scroll has only 33 gold instances (2.1% of 1,550) and text entry 135, so the reported scroll F1 values (e.g., 0.70 for Gemini 3.6 Flash vs. 0.34 for GLM-4.6V) are high-variance estimates. The macro-F1 primary metric is appropriate, but the paper draws conclusions about per-family ordering (e.g., 'text is next... key command is unsaturated') without confidence intervals. Please provide bootstrap or exact binomial confidence intervals for the per-family metrics, or explicitly flag small-sample families as exploratory.
minor comments (5)
  1. [Throughout] Table numbering is inconsistent: 'Table 2' is used both for the payload specification (§3.2) and for the single-action results (§6.3). Please renumber or disambiguate.
  2. [§3.1] The formal state definition and rendering equation are missing from the text as rendered—only the prose descriptions remain. Please ensure the equations are included in the final version.
  3. [§6.1 vs. Appendix B.3] Section 6.1 says all models except Kimi-K2.6 have reasoning enabled, but Table B3 lists GLM-4.6V as 'Reasoning off.' Please reconcile this discrepancy.
  4. [§2.3] Minor typographical issue: 'DB differs on 6 axes' should be 'DDB differs on 6 axes.' Also consider rephrasing the patent-priority comparison to EvoGUI, which is not a scientific argument.
  5. [§1] The statement about a pending U.S. provisional patent in the contributions section is unusual for a benchmark paper. It would be cleaner to move it to a footnote or acknowledgements.

Circularity Check

0 steps flagged

No significant circularity: DDB labels come from recorded events and human review, not from model outputs; no self-citation chain or fitted parameter forces any headline result.

full rationale

I walked the paper's derivation chain. DDB's gold labels are defined from recorded input events plus human review (§4.2–4.3: 'Gold labels derive from recorded events'; 'Every candidate mined by one collector, receives 1 of 3 statuses from the second'), never from VLM outputs or fitted parameters. The two tasks are prediction problems with fixed human-verified targets; the reported baselines are computed from the label space, not fitted to model performance. No equation in §§3–5 equates a predicted quantity with an input by construction. The only filtering step (37.3% of candidates discarded as ambiguous) is a data-validity/generalizability concern, not a circular reduction, because the retained labels still come from independent recorded events. The paper contains no load-bearing self-citation: the closest reference (UI-Vision [6], Nayak et al. 2025) is external prior work and is not used to derive DDB's claims. The empirical results — decoy-condition accuracy, presentation-order echo, click-vs-drag gaps — are measured outputs, not definitions. The stated limitation about lacking long-horizon context accumulation also does not undermine the independence of the step-level results.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The benchmark introduces no physical entities and fits no free parameters. It relies on human-verified ground truth, the sufficiency of 1-second frame windows, the uniqueness of ordering in filtered triples, and correct event-frame synchronization. These are domain assumptions inherited by any screen-transition benchmark.

axioms (5)
  • domain assumption Human review yields a unique, correct gold answer for every retained DDB instance.
    Sections 3.2/4.3: ambiguous or multi-action candidates were excluded (3,209→2,013). If real transitions are often ambiguous, the benchmark is an idealized subset and reported gaps are upper bounds on real transition understanding.
  • domain assumption A 1-second before/after screenshot window contains enough visible evidence to identify the canonical action family and payload.
    Section 3.2/4.2: pairs whose after-frame did not unambiguously reflect the recorded action were removed during human review; this assumes the retained pairs are visually resolvable.
  • domain assumption Ordering triples separated by many unobserved intermediate actions have a unique chronological order inferable from visible high-level state.
    Section 3.3: triples are sampled from action-mapped positions, not consecutive frames; ambiguous triples were discarded, assuming high-level state progression is sufficient.
  • domain assumption Recorded pynput events and 10 fps frame captures are correctly synchronized and mapped to gold labels.
    Section 4.2: alignment pipeline is described but no independent synchronization validation or error rate is reported.
  • standard math Standard combinatorics (1/12 vs 1/6 baselines) and macro-F1/AUC metric definitions are correctly applied.
    Section 5 uses standard formulas; no formal verification is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 15933 in / 17643 out tokens · 153550 ms · 2026-08-01T00:45:38.334446+00:00 · methodology

0 comments
read the original abstract

Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action- crucial for rejecting stale observations, verifying progress, and recovering from failure. This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning. We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB trajectories targets 3 failure dimensions- state verification, source tracking, and context-aware control- through 2 complementary tasks: 463 3-frame temporal-ordering instances, including 105 with a cross-trajectory decoy, and 1,550 before-after pairs labeled from 5 actions + its payload. We evaluate 8 closed and open-source model families across 32 ordering and 16 single-action settings, observing consistent gaps. Ordering remains unsaturated: best non-decoy and decoy exact-match rates are 65.1% and 65.7%. Task context improves decoy identification by 6.9 percentage points but reduces non-decoy exact match by 2.2 points; error analysis reveals systematic copying of the presented A-B-C order. Single-action results show that inferring the action family is harder than locating it: click F1 is 0.96 vs, 0.76 for drag, while recognized drags are generally localized well. DDB, thus, complements end-to-end benchmarks by filling the missing diagnostic layer between GUI grounding and final task success, enabling targeted improvements to desktop CUA verification, reliability, and recovery.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 3 canonical work pages · 2 internal anchors

  1. [3]

    This shows DDB is far from saturated

    Methodical analysis and validation: 8 frontier closed and open-source models- with leading reported agent-level results on CUA benchmarks- evaluated across 48 configurations (ordering: 8 models × 2 inference settings × 2 prompt conditions, action: 8 models × 2 settings). This shows DDB is far from saturated. A U.S. provisional patent covering the DDB mode...

    2026

  2. [4]

    Figure 3: Application distribution of the 463 ordering triples across the 3 failure dimensions, placed by focused app for sample

    Individual recordings details with its dimension map are in Table A3 in Appendix A. Figure 3: Application distribution of the 463 ordering triples across the 3 failure dimensions, placed by focused app for sample. Figure 4: Action distribution of single-action reconstruction task. The 1,550 single-action samples are dominated by clicks (70.8%), followed b...

    2013

  3. [5]

    https://www.anthropic.com/news/claude-sonnet-5

    (June 30, 2026). https://www.anthropic.com/news/claude-sonnet-5

    2026

  4. [6]

    OSWorld-G evaluates fine-grained screenshot-conditioned manipulation [12]

    scores next-action prediction from 8,227 queries across 83 desktop applications [Nayak et al., 2025]. OSWorld-G evaluates fine-grained screenshot-conditioned manipulation [12]. All operate on single screenshots [20]. A predicted point can be spatially correct while the predicted event is semantically wrong- confusing right-click with left-click, selection...

    2025

  5. [7]

    EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

    Yang, Yaohan, et al. “EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding.” arXiv.Org, 19 July 2026, https://arxiv.org/abs/2607.17050v1

    Pith/arXiv arXiv 2026

  6. [9]

    OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. arXiv:2606.29537. https://doi.org/10.48550/arXiv.2606.29537

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.29537

  7. [10]

    AgentHijack

    measures agent efficiency relative to human step counts and completion times on the same tasks [Abhyankar et al., 2025]. AgentHijack

    2025

  8. [11]

    In International Conference on Learning Representations (ICLR 2026)

    Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness. In International Conference on Learning Representations (ICLR 2026). https://openreview.net/forum?id=9W4bPRsEIT

    2026

  9. [12]

    In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267, 45817–45851

    UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267, 45817–45851. https://proceedings.mlr.press/v267/nayak25a.html

    2025

  10. [14]

    In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267, 4874–4910

    Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267, 4874–4910. https://proceedings.mlr.press/v267/bonatti25a.html

    2025

  11. [15]

    arXiv:2604.27776

    WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments. arXiv:2604.27776. https://doi.org/10.48550/arXiv.2604.27776

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.27776

  12. [16]

    arXiv:2506.16042

    OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents. arXiv:2506.16042. https://doi.org/10.48550/arXiv.2506.16042

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.16042

  13. [17]

    arXiv:2402.17553

    OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web. arXiv:2402.17553. https://doi.org/10.48550/arXiv.2402.17553

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.17553

  14. [18]

    target”) yields an action-mapped trajectory through demonstrations: where 𝑜!is the event-aligned pre-action frame and õ!

    web trajectories spanning 120 domains: temporal ordering (K = 3, 4, 5), inverse action/value prediction over click/type/select, and contrastive 1-step successor discrimination. DB differs on 6 axes: • Data source: EvoGUI re-mines existing Mind2Web / WebLINX browser trajectories. DDB collects controlled Linux desktop recordings across 50 task domains. • En...

    1920

  15. [19]

    arXiv:2505.13227

    Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis. arXiv:2505.13227. https://doi.org/10.48550/arXiv.2505.13227

    doi:10.48550/arxiv.2505.13227

  16. [20]

    arXiv:2505.12493

    GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning. arXiv:2505.12493. https://doi.org/10.48550/arXiv.2505.12493

    doi:10.48550/arxiv.2505.12493

  17. [21]

    arXiv:2602.17365

    Computer-Using World Model. arXiv:2602.17365. https://doi.org/10.48550/arXiv.2602.17365

    doi:10.48550/arxiv.2602.17365

  18. [22]

    In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026)

    AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026). https://doi.org/10.1145/3770855.3817494

    arXiv 2026

  19. [23]

    arXiv:2606.15673

    Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking. arXiv:2606.15673. https://doi.org/10.48550/arXiv.2606.15673

    work page doi:10.48550/arxiv.2606.15673

  20. [24]

    In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), PMLR 235, 33007–33056

    WebLINX: Real-World Website Navigation with Multi-Turn Dialogue. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), PMLR 235, 33007–33056. https://proceedings.mlr.press/v235/lu24e.html

    2024

  21. [25]

    H Company. n.d. Agent Loop. H Tech Hub. Retrieved July 27, 2026, from https://hub.hcompany.ai/agent-loop

    2026

  22. [26]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 9313–9332

    SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 9313–9332. https://doi.org/10.18653/v1/2024.acl-long.505

    doi:10.18653/v1/2024.acl-long.505 2024

  23. [27]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 881–905

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 881–905. https://doi.org/10.18653/v1/2024.acl-long.50

    doi:10.18653/v1/2024.acl-long.50 2024

  24. [28]

    arXiv:2406.03679

    On the Effects of Data Scale on UI Control Agents. arXiv:2406.03679. https://doi.org/10.48550/arXiv.2406.03679

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.03679

  25. [29]

    In International Conference on Learning Representations (ICLR 2024)

    WebArena: A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations (ICLR 2024). https://doi.org/10.48550/arXiv.2307.13854

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.13854 2024

  26. [30]

    Locally hosted models are run in BF16 precision with vLLM inference [32]

    (73.1%), and GLM-4.6V [31]. Locally hosted models are run in BF16 precision with vLLM inference [32]. All in the same configuration receive the same prompt and JSON parsing evaluator, decoded at temperature 1 (recommended). All are tested with reasoning enabled (provider managed/native) except Kimi-K2.6, in 2 settings each: medium/high for closed-source, ...

    2048

  27. [31]

    (July 9, 2026)

    GPT-5.6: Frontier Intelligence that Scales with Your Ambition. (July 9, 2026). https://openai.com/index/gpt-5-6/

    2026

  28. [32]

    Anthropic. 2026a. Claude Opus 4.8. (May 28, 2026). https://www.anthropic.com/news/claude-opus-4-8

    2026

  29. [33]

    In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023), 611–626

    Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023), 611–626. https://doi.org/10.1145/3600006.3613165

    arXiv 2023

  30. [34]

    In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), 8387–8401

    Attacking Vision-Language Computer Agents via Pop-ups. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), 8387–8401. https://doi.org/10.18653/v1/2025.acl-long.411 APPENDIX The appendix provides the specifics underlying the main results. Appendix A reports all evaluated configurations, analyzes temporal o...

    doi:10.18653/v1/2025.acl-long.411 2025

  31. [42]

    Closed-source models use medium/high provider controls (accessed between July 15-July 25, 2026); open-source models use their stated reasoning mode with 2,048- and 4,096-token output budgets (2K, 4K). Model / component Reasoning configuration Reasoning setting(s) Ordering permutation - Seed 42 GPT-5.6-Sol Reasoning effort Medium; High Claude Opus 4.8 Thin...

    2026

  32. [2023]

    In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks

    Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks. https://arxiv.org/abs/2306.06070

    Pith/arXiv arXiv 2023

  33. [2024]

    In Advances in Neural Information Processing Systems 37 (NeurIPS 2024)

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://doi.org/10.48550/arXiv.2404.07972

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.07972 2024

  34. [2025]

    arXiv:2504.07981

    ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. arXiv:2504.07981. https://doi.org/10.48550/arXiv.2504.07981

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.07981

  35. [2026]

    AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

    AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions. In Proceedings of the 43rd International Conference on Machine Learning (ICML 2026). https://doi.org/10.48550/arXiv.2605.25707

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.25707 2026