What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities

Bingchen Miao; Juncheng Li; Kaihang Pan; Mengze Li; Minghe Gao; Qifan Yu; Siliang Tang; Wei Ji; Wendong Bu; Yang Wu

REVIEW 4 major objections 6 minor 46 references

What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniBench auto-generates 36,076 graph-structured virtual-agent tasks with controllable complexity, evaluates ten capabilities, and shows that fine-tuning on its graph trajectories improves agents on external benchmarks.

desk verdict A serious benchmark artifact with a real but addressable validity gap: the load-bearing LLM-synthesized evaluators are only indirectly validated. read the letter →

arxiv 2506.08933 v1 pith:37SVAJFU submitted 2025-06-10 cs.CV

Wendong Bu , Yang Wu , Qifan Yu , Minghe Gao , Bingchen Miao , Zhenkui Zhang , Kaihang Pan , Yunfei Li

show 5 more authors

Mengze Li Wei Ji Juncheng Li Siliang Tang Yueting Zhuang

This is my paper

classification cs.CV

keywords virtualagentsGUIbenchmarktaskgraphscontrollablecomplexitymultimodallargelanguagemodelscapabilityevaluationsyntheticsynthesis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing virtual-agent benchmarks fail on three counts: task complexity is fixed and uncontrolled, manual annotation limits scale and scenario coverage, and evaluation is coarse-grained. To fix this, the authors propose OmniBench, a self-generating pipeline that composes small subtasks into task graphs so complexity can be specified in advance along five dimensions: dependency, instruction, knowledge, hierarchy, and branch. They then evaluate agents with two graph-based metrics, Coverage Rate and Logical Consistency, across ten capability dimensions. The central empirical claim is that fine-tuning two open-source GUI agents on OmniBench's graph-structured trajectories raises their success rates on AndroidControl and OmniAct above the original OS-Atlas and UGround models, and that adding task intents improves planning for both open and closed models. If correct, this would make large-scale agent evaluation cheap and automated, and would point to graph-structured data as a better training signal than manually annotated linear demonstrations.

What carries the argument

The task graph $G=(S,R)$ has nodes as subtasks with explicit input and output resource lists, and edges as dependencies formed when one subtask's output matches another's input. Five complexity dimensions are read directly from graph topology: edge count (dependency), node count (instruction), number of application categories (knowledge), depth (hierarchy), and width (branch), each binned into easy, medium, or hard. A bottom-up synthesis pipeline explores 49 applications to propose subtasks, iteratively synthesizes trajectories and evaluation functions through cross-verification, composes subtasks into graphs under extracted task intents, and validates that the summarized instruction preserves the graph's dependency structure. The evaluator tracks each node as Completed, Evaluating, or Waiting, and defines Coverage Rate as depth-weighted progress and Logical Consistency as the agent's coherence score divided by the maximum coherence over all topological orders.

What would settle it

Take a random sample of tasks across all 20 scenarios, have independent human annotators score agent trajectories with the same rubric, and compare their scores against OmniEval's Coverage Rate and Logical Consistency on a sample larger than the reported 50 trajectories; also manually audit a few hundred synthesized evaluation functions by replaying subtask executions. If agreement drops far below the reported correlation, or if the evaluation functions systematically accept unfinished subtasks, the benchmark's scores and the graph-versus-chain gap would not be reliable.

Watch

Extended reading notes

Core claim

The paper's central claim is that representing a GUI task as a directed acyclic graph of subtasks, instead of a linear step sequence, makes task complexity composable and measurable, and that this representation is not just a benchmarking device: training on graph-structured trajectories transfers to external benchmarks, improving AndroidControl and OmniAct success rates over the original OS-Atlas and UGround models. A second claim is that mainstream agents collapse in the presence of graph structure: even GPT-4o reaches only 20.5% on graph-structured tasks while humans reach 80.1%, and instruction-order sensitivity drops after graph-trajectory fine-tuning. The paper also reports that adding task intent to prompts raises planning scores by up to 8.9 points for a closed-source model, and that the weakest capabilities across all agents are Subtask Identification and Long Instruction Following.

Load-bearing premise

The evaluation functions synthesized by a code LLM from the 11 hand-designed APIs must correctly judge when each subtask is completed, and the 90.7% human acceptance rate measured on sampled tasks must generalize to the full set of 36,076 tasks.

Editorial extensions

If this is right

If OmniBench's pipeline is right, future GUI-agent benchmarks can be produced at scale without manual trajectory annotation, with task complexity dialed in before synthesis.
Fine-tuning with graph-structured trajectories should be added to the GUI-agent training recipe, since it transfers to external benchmarks like AndroidControl and OmniAct.
Task intent becomes a cheap, plug-and-play lever: adding it to prompts improves planning scores by up to 8.9 points for closed-source models and around 1.2-1.7 points in fine-tuning.
Graph-structured tasks are a harder and more diagnostic test than chain-structured tasks, since even strong agents lose roughly 15 points when the same nodes and edges are arranged non-linearly.
The capability profiles, especially weak Subtask Identification and Long Instruction Following, give concrete targets for the next generation of virtual agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reusable core of this work is probably the 11 system-level APIs plus the cross-verification protocol, not the 36k task count; a smaller benchmark with audited evaluation functions could be nearly as informative.
The expression-order sensitivity result implies that future agent leaderboards should randomize instruction phrasing, otherwise rankings may reflect prompt order rather than agent ability.
The graph-versus-chain performance gap suggests that real-world office workflows, which are naturally parallel and branching, may be systematically underrepresented by datasets built from linear demonstrations.
A testable extension is to use the five-dimensional complexity decomposition as a diagnostic protocol for comparing agents before and after interventions such as long-context training or reinforcement learning.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Candid take: this is a real contribution. The bottom-up synthesis pipeline that composes 36k DAG tasks with five controllable complexity dimensions and ten capability axes is exactly what the GUI-agent evaluation subfield is missing. The paper is clear about the mechanism, the quality-control ablations are useful, and the human-acceptance numbers (90.7% on sampled graphs) suggest the task synthesis itself is not garbage. The graph-based metrics (CR and LC) are sensible, and the observation that graph-structured tasks crush even GPT-4o (20.5% vs 80.1% human) is a finding worth knowing.

The main thing to worry about is the evaluation functions. They are generated by Claude-3.5-Sonnet from 11 predefined APIs, and every CR/LC score, capability ranking, and graph-vs-chain comparison in Tables 4, 5, and Figure 7 depends on those functions. The paper's validation is indirect: the 90.7% human acceptance rate is about task graphs, not evaluator verdicts; the 50-trajectory correlation with human scores is episode-level; and the cross-verification loop uses trajectories from the same model family that later gets evaluated, so co-adaptation is a real risk. The concrete failure mode—an eval function that checks for a 'save' click and file existence but not the correct content or directory—is plausible. That said, the correlation with human judgment on 50 trajectories is some evidence the metrics aren't wildly off. Still, a direct validation of per-subtask verdicts on a few hundred instances would go a long way, and the paper should state this limitation openly.

The fine-tuning claims are weaker. The gains on AndroidControl and OmniAct are 0.2–0.7 points, which is within noise; there are no significance tests. And the abstract says training on graph-structured data 'more efficiently guides agents compared to manually annotated data,' but no manual-annotation baseline appears in the experiments—they compare to the base models, not to a model trained on manual data. That claim is overstated as written.

The novelty claim ('only scalable benchmark that defines composable task complexity using graphs to evaluate multiple essential capabilities') is a bit strong given CRAB and TaskBench already use graphs, but the combination of automated synthesis, five complexity dimensions, and ten capabilities is genuinely new as a package.

Bottom line: worth a serious referee. If the release includes the artifacts and evaluation functions so others can audit them, this becomes a standard benchmark. The eval-function validation and the manual-baseline comparison should be requested before acceptance. I'd bring it to reading group and cite it if I worked on GUI agents.

Referee Report

4 major / 6 minor

Summary. The paper introduces OmniBench, a graph-structured virtual-agent benchmark built with an automated bottom-up synthesis pipeline. It claims to generate 36,076 tasks across 20 scenarios and 49 applications, with five controllable complexity dimensions and ten capability-oriented test dimensions. The companion OmniEval framework scores agents with two graph-based metrics, Coverage Rate (CR) and Logical Consistency (LC), using subtask evaluation functions automatically synthesized by a code LLM from 11 hand-designed system-level APIs. The paper reports a 90.7% human acceptance rate on sampled tasks, a 50-trajectory correlation between OmniEval and human evaluation, and evaluations of 12 models including humans, open/closed-source MLLMs, and GUI agents. It also fine-tunes OS-Atlas and UGround on graph-structured trajectories and reports small improvements on AndroidControl and OmniAct, and concludes that graph-structured training data is more efficient than manually annotated data.

Significance. If the central claims hold, OmniBench would be a useful and timely resource: the automatic synthesis pipeline is described in enough detail to be reproducible, the graph-based metrics are a genuine step beyond binary success-rate evaluation, and the multidimensional capability breakdown addresses a real gap in virtual-agent benchmarking. The manuscript's strengths include the detailed appendix with prompts and data schemas, the inclusion of a human-acceptance quality check, and an explicit human-correlation validation of the proposed metrics. However, the validity of every OmniBench score depends on the correctness of the synthesized per-subtask evaluation functions, and that assumption is not directly validated. In addition, the headline claim about training efficiency relative to manual annotation is not supported by the experiments as presented. These issues are addressable in revision, but they are load-bearing for the paper's main conclusions.

major comments (4)

[Section 3.2 / Appendix B.2 / C.3] The correctness of the Code-LLM-synthesized subtask evaluation functions is the load-bearing input to every CR and LC score, but the evidence presented does not validate those functions directly. Table 3 measures human acceptance of sampled task graphs, not evaluator verdicts; Figure 5 correlates OmniEval with human scores on only 50 whole trajectories, which can agree in aggregate even with large per-subtask errors; and Appendix B.2's cross-verification loop uses trajectories produced by the same model family (Claude-3.5-Sonnet and GPT-4o) that is later evaluated, so trajectories and evaluators can co-adapt to shared blind spots. I request a direct per-subtask audit: sample evaluation functions, run them on agent trajectories spanning clear successes and failures, and compare each verdict with human judgment, reporting per-subtask precision/recall and inter-annotator agreement. Without this, the rankings in Tables 4 and 5 and the chain-vs-graph comparison in Figure 7 are not secured.
[Abstract / Section 5.3 (Tables 6 and 7)] The claim that training on graph-structured data 'can more efficiently guide agents compared to manually annotated data' is not demonstrated by the experiments. Tables 6 and 7 compare Omni-OS-Atlas and Omni-UGround with the original OS-Atlas and UGround checkpoints; those baselines are not a matched manual-annotation training condition, so the comparison is confounded by dataset, recipe, and data quantity. I ask for a controlled experiment that fine-tunes the same backbone on (a) OmniBench graph-structured trajectories and (b) an equivalent amount of manually annotated or chain-structured trajectories, with the same budget and hyperparameters, or the claim should be removed or weakened.
[Tables 6 and 7] The reported external-benchmark improvements are small: average success-rate gains of 0.46 and 0.73 points for Omni-OS-Atlas and 0.40 and 0.30 for Omni-UGround, and some individual metrics decrease (e.g., AndroidControl-Low Grounding for OS-Atlas falls from 83.76 to 83.51 in Table 6). No confidence intervals, multiple seeds, or significance tests are reported. These magnitudes are within typical run-to-run variation for fine-tuned agents, so the conclusion that graph-structured trajectories 'significantly enhance' action reasoning is not yet supported. Please report means and variances over repeated runs or a significance test.
[Section 4.2 / Table 12] The construction of the 10 capability test sets needs clarification. The table constrains combinations of complexity dimensions, but it is not stated whether a task satisfying several hard constraints (e.g., hard Dependency plus hard Hierarchy plus hard Branch) is assigned to multiple capability sets, nor how many tasks each set contains. Some assignments are also counterintuitive: 'Long Instruction Following' is constrained on Hierarchy and Branch but not on Instruction complexity. Without a disjoint-assignment rule and per-capability task counts, the capability-wise scores in Table 4 are difficult to interpret as measuring distinct capabilities.

minor comments (6)

[Section 3.4 / Figure 3] The figure legend contains the typo 'Defendency'; also, the label 'Branching' should be aligned with the term 'Branch Complexity' used in the text.
[Section 4.1] The maximum number of steps N in the graph evaluator is never specified; please state the default value and whether the results are sensitive to it.
[Table 12 / Appendix D.1] Please state in the main text that empty stars in Table 12 mean 'unconstrained', not 'low', since the caption alone is easy to misread.
[Figure 7] The y-axis label reads 'accuracy', while the surrounding text and Section 4 use Coverage Rate; clarify which metric is plotted and whether the human value of 80.1 is CR or SR.
[Section 2] The claim that OmniBench is 'the only scalable benchmark' should be qualified by a precise definition of 'scalable', since CRAB and TASKBENCH also use graph structures.
[Section 3.2 / Appendix C.2] Minor copyedits: 'rule-based, The synthesis' should be 'rule-based, the synthesis', and Appendix C.2's example contains 'new a Text Document'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims rest on external transfer and independent human checks; the same-source evaluator risk is a validity concern, not a definitional reduction.

full rationale

OmniBench's load-bearing claims are (i) that the synthesized 36k-task graph dataset is usable for agent evaluation, (ii) that OmniEval's CR/LC scores align with human judgment, and (iii) that fine-tuning on OmniBench trajectories transfers to external benchmarks such as AndroidControl and OmniAct. None of these reduces to the paper's own inputs by construction. The subtask evaluation functions are generated from 11 predefined APIs by Claude-3.5-Sonnet, but the paper does not define an agent's score as the output of those functions in a way that makes the benchmark's conclusions tautological; the functions are checked against sampled human acceptance ratings and a 50-trajectory correlation with human scores (Figure 5). The graph-versus-chain comparison, the capability rankings, and the complexity-level tables are downstream of those evaluator verdicts rather than fitted parameters renamed as predictions. No parameter is fitted to the target result, and no uniqueness theorem from the authors' prior work is invoked to force the graph-based design. Self-citations appear in the bibliography and related-work discussion, but they are not load-bearing for the central empirical claims. The main non-circular validity risk is that LLM-synthesized evaluation functions may not perfectly capture semantic subtask completion, and the human-acceptance and correlation checks are based on samples; that is a correctness concern, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark's central claims rest on five domain assumptions: the DAG representation of tasks, the reliability of LLM-generated trajectories, the correctness of LLM-generated evaluation functions, the validity of GPT-4o as a consistency validator, and the generalization of human acceptance ratings. The hand-set complexity thresholds and depth-based weights are free parameters that shape which tasks are labeled hard and which capability is being measured.

free parameters (4)

Complexity level thresholds = Easy/Medium/Hard: edges <=1, 2-3, >=4; nodes <=2, 3-4, >=5; app categories <=1, 2-3, >=4; depth <=2, 3-4, >=5; width…
Hand-chosen cutoffs in Table 2 define which tasks count as easy, medium, or hard for each of the five complexity dimensions, and these buckets directly determine the 10 capability test sets in Section 4.2.
Depth-based weight in Coverage Rate = w(s_i) = d(s_i) / sum_j d(s_j)
Section 4.1 defines CR with depth-based weights without a principled derivation; changing the weighting scheme changes every reported CR score.
Maximum steps N in graph evaluator = Not reported
Section 4.1 states that if the agent does not complete any subtask within N steps the entire task fails, but the value of N is never given, making the exact evaluation protocol under-specified.
Set of 11 system-level APIs = 11 hand-designed APIs (check_mouse_clicks, check_keyboard_types, check_file_exists, check_text_exists_via_ocr, etc.)
The expressiveness of the evaluation functions is capped by these hand-designed APIs (Appendix B.2, C.3), so any subtask whose completion cannot be expressed through them will be mis-scored.

assumptions (5)

domain assumption Real-world digital tasks can be modeled as DAGs of subtasks with typed input and output resources.
Section 3.1 defines the task graph G={S, R}; the entire synthesis pipeline, complexity metrics, and graph evaluator assume this decomposition faithfully represents how humans do office, web, and mobile work.
domain assumption Advanced MLLMs can generate correct, diverse subtask trajectories and execution traces.
Section 3.2 and B.2 use Claude-3.5-Sonnet and GPT-4o to synthesize the trajectories; if the base generation is biased or wrong, the 295K demonstrations inherit those errors.
domain assumption Code-LLM-composed evaluation functions built from 11 hand-designed APIs correctly decide subtask completion.
Section 3.2 and B.2, Figure 10; every CR and LC score in Tables 4 and 5 depends on these generated functions being correct.
domain assumption GPT-4o's consistency validator can infer the true dependency structure from a task instruction.
Section 3.2 and B.4; instructions are accepted or re-summarized based on whether GPT-4o's inferred dependencies match the ground-truth graph, so the entire instruction set is filtered by GPT-4o's own inference ability.
domain assumption Human acceptance by three trained annotators on sampled tasks generalizes to the full 36k dataset.
Section 3.3 reports 90.7% acceptance on 400-sample ablations, but the paper does not provide confidence intervals or inter-annotator agreement, and does not establish that the sample is representative of all scenarios and complexity levels.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities." pith.science (2026). https://pith.science/paper/37SVAJFU

@misc{pith2026250608933,
  author       = {Pith},
  title        = {Pith review of: What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/37SVAJFU}},
  note         = {Machine review of arXiv:2506.08933}
}

read the original abstract

As multimodal large language models (MLLMs) advance, MLLM-based virtual agents have demonstrated remarkable performance. However, existing benchmarks face significant limitations, including uncontrollable task complexity, extensive manual annotation with limited scenarios, and a lack of multidimensional evaluation. In response to these challenges, we introduce OmniBench, a self-generating, cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity through subtask composition. To evaluate the diverse capabilities of virtual agents on the graph, we further present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities. Our synthesized dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91\% human acceptance rate. Training on our graph-structured data shows that it can more efficiently guide agents compared to manually annotated data. We conduct multidimensional evaluations for various open-source and closed-source models, revealing their performance across various capabilities and paving the way for future advancements. Our project is available at https://omni-bench.github.io/.

Figures

Figures reproduced from arXiv: 2506.08933 by the authors.

**Figure 1.** Overview of OmniBench, a systematic benchmark with five-dimensional task complexity and bottom-up automatic task synthesis for generating structured task graphs. It evaluates ten virtual agent capabilities using high-quality graph-based data, ensuring scalable and realistic task evaluation. based benchmark, OmniBench. It dynamically synthesizes tasks with controllable complexity based on a bottomup pipeline. OmniB… view at source ↗

**Figure 2.** Bottom-up task synthesis pipeline [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Statistics of OmniBench mensional evaluation framework. In this section, we first introduce a graph-based evaluator with two novel metrics for finegrained and diverse evaluation (Section 4.1). Then, we describe the construction of test tasks designed to evaluate 10 distinct capabilities by constraining task complexity (Section 4.2). 4.1 Graph-based Evaluator Currently, most benchmarks still evaluate agents in a co… view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Comparison of mainstream virtual agent evaluation strategies with the evaluation strategy we propose. Additionally, to fully leverage the potential of the graph-based evaluator, we have designed two novel graph-based metrics. Traditional metrics fail to evaluate interm…

**Figure 5.** Figure 5: Correlation between Coverage Rate and Logical Consistency with Human Evaluation. Capability Boundaries of Mainstream Agents. As detailed in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Performance comparison of various models on chainstructured tasks and graph-structured tasks [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Distribution of five major errors in 100 failure instances for each model. An asterisk (*) indicates that the agent uses GPT4o as the planner. 6 Conclusion In conclusion, we introduced OmniBench, a graph-based benchmark that addresses the limitations of existing eval…

**Figure 9.** Figure 9: Overview of the 20 defined task scenarios. The scenarios are derived from combinations of application categories, where each box represents a task scenario with associated application icons. sis. The Subtask Synthesis stage includes two steps: Subtask Discovery and Ite…

**Figure 10.** Figure 10: An overview of how predefined system-level APIs are composed into evaluation functions using Code LLMs. Cross-Verification. We design a cross-verification algorithm to optimize synthesized subtasks. Specifically, the algorithm performs N iterations, where an MLLM and…

**Figure 11.** Figure 11: An overview of the constraint relationships among environmental resources. Task Composition with Consistent Intent. To construct meaningful task graphs, we first curate a subtask pool by filtering out low-quality samples through a cross-verification process. A na¨ıv…

**Figure 12.** Figure 12: A task example in OmniBench. D Details of OmniEval In this section, we introduce the details of OmniEval, including the design of ten essential capabilities, the experimental setup for evaluation, and implementation details during the evaluation. D.1 Capability Desig…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages

[1]

Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

arXiv 2023
[2]

Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., and Plummer, B. A. A dataset for interactive vision-language navigation with unknown command feasibility. In European Conference on Computer Vision, pp.\ 312--328. Springer, 2022

work page 2022
[3]

Spider2-v: How far are multimodal agents from automating data science and engineering workflows? arXiv preprint arXiv:2407.10956, 2024

Cao, R., Lei, F., Wu, H., Chen, J., Fu, Y., Gao, H., Xiong, X., Zhang, H., Mao, Y., Hu, W., et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? arXiv preprint arXiv:2407.10956, 2024

arXiv 2024
[4]

Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24185--24198, 2024

work page 2024
[5]

Seeclick: Harnessing gui grounding for advanced visual gui agents

Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

arXiv 2024
[6]

Mind2web: Towards a generalist agent for the web

Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024

work page 2024
[7]

Dysen-vdm: Empowering dynamics-aware text-to-video diffusion with llms

Fei, H., Wu, S., Ji, W., Zhang, H., and Chua, T.-S. Dysen-vdm: Empowering dynamics-aware text-to-video diffusion with llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7641--7653, 2024 a

work page 2024
[8]

Video-of-thought: Step-by-step video reasoning from perception to cognition

Fei, H., Wu, S., Ji, W., Zhang, H., Zhang, M., Lee, M.-L., and Hsu, W. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning, 2024 b

work page 2024

Show all 46 references

[9]

Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing

Fei, H., Wu, S., Zhang, H., Chua, T.-S., and Yan, S. Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing. 2024 c

2024
[10]

Enhancing video-language representations with structural spatio-temporal alignment

Fei, H., Wu, S., Zhang, M., Zhang, M., Chua, T.-S., and Yan, S. Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024 d

2024
[11]

Generalist virtual agents: A survey on autonomous agents across digital platforms

Gao, M., Bu, W., Miao, B., Wu, Y., Li, Y., Li, J., Tang, S., Wu, Q., Zhuang, Y., and Wang, M. Generalist virtual agents: A survey on autonomous agents across digital platforms. arXiv preprint arXiv:2411.10943, 2024 a

2024 arXiv
[12]

De-fine: Decomposing and refining visual programs with auto-feedback

Gao, M., Li, J., Fei, H., Pang, L., Ji, W., Wang, G., Lv, Z., Zhang, W., Tang, S., and Zhuang, Y. De-fine: Decomposing and refining visual programs with auto-feedback. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, pp.\ 7649–7657, New York, NY, ...

2024
[13]

Benchmarking multimodal cot reward model stepwise by visual program, 2025

Gao, M., Liu, X., Yue, Z., Wu, Y., Chen, S., Li, J., Tang, S., Wu, F., Chua, T.-S., and Zhuang, Y. Benchmarking multimodal cot reward model stepwise by visual program, 2025. URL https://arxiv.org/abs/2504.06606

2025 arXiv
[14]

Iris: Breaking gui complexity with adaptive focus and self-refining

Ge, Z., Li, J., Pang, X., Gao, M., Pan, K., Lin, W., Fei, H., Zhang, W., Tang, S., and Zhuang, Y. Iris: Breaking gui complexity with adaptive focus and self-refining. arXiv preprint arXiv:2412.10342, 2024

2024 arXiv
[15]

Navigating the digital world as humans do: Universal visual grounding for gui agents

Gou, B., Wang, R., Zheng, B., Xie, Y., Chang, C., Shu, Y., Sun, H., and Su, Y. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024

2024 arXiv
[16]

Cogagent: A visual language model for gui agents

Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14281--14290, 2024

2024
[17]

Hu, S., Ouyang, M., Gao, D., and Shou, M. Z. The dawn of gui agent: A preliminary case study with claude 3.5 computer use. arXiv preprint arXiv:2411.10323, 2024

2024 arXiv
[18]

P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

2024 arXiv
[19]

P., Russak, M., Koh, J

Kapoor, R., Butala, Y. P., Russak, M., Koh, J. Y., Kamble, K., AlShikh, W., and Salakhutdinov, R. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pp.\ 161--178. Springer, 2025

2025
[20]

Y., Lo, R., Jang, L., Duvvur, V., Lim, M

Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024

2024 arXiv
[21]

Li, J., Wang, X., Tang, S., Shi, H., Wu, F., Zhuang, Y., and Wang, W. Y. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12123--12132, 2020 a

2020
[22]

Fine-grained semantically aligned vision-language pre-training

Li, J., He, X., Wei, L., Qian, L., Zhu, L., Xie, L., Zhuang, Y., Tian, Q., and Tang, S. Fine-grained semantically aligned vision-language pre-training. Advances in neural information processing systems, 35: 0 7290--7303, 2022

2022
[23]

Fine-tuning multimodal llms to follow zero-shot demonstrative instructions

Li, J., Pan, K., Ge, Z., Gao, M., Ji, W., Zhang, W., Chua, T.-S., Tang, S., Zhang, H., and Zhuang, Y. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations, 2023 a

2023
[24]

Variational cross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding

Li, J., Tang, S., Zhu, L., Zhang, W., Yang, Y., Chua, T.-S., Wu, F., and Zhuang, Y. Variational cross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (10): 0 1...

2023
[25]

Mapping natural language instructions to mobile ui action sequences

Li, Y., He, J., Zhou, X., Zhang, Y., and Baldridge, J. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776, 2020 b

2005 arXiv
[26]

Q., Li, L., Gao, D., Yang, Z., Wu, S., Bai, Z., Lei, W., Wang, L., and Shou, M

Lin, K. Q., Li, L., Gao, D., Yang, Z., Wu, S., Bai, Z., Lei, W., Wang, L., and Shou, M. Z. Showui: One vision-language-action model for gui visual agent. arXiv preprint arXiv:2411.17465, 2024

2024 arXiv
[27]

Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

Lu, Q., Shao, W., Liu, Z., Meng, F., Li, B., Chen, B., Huang, S., Zhang, K., Qiao, Y., and Luo, P. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024

2024 arXiv
[28]

Boosting virtual agent learning and reasoning: A step-wise, multi-dimensional, and generalist reward model with benchmark, 2025

Miao, B., Wu, Y., Gao, M., Yu, Q., Bu, W., Zhang, W., Li, Y., Tang, S., Chua, T.-S., and Li, J. Boosting virtual agent learning and reasoning: A step-wise, multi-dimensional, and generalist reward model with benchmark, 2025. URL https://arxiv.org/abs/2503.18665

2025 arXiv
[29]

Self-supervised meta-prompt learning with meta-gradient regularization for few-shot generalization

Pan, K., Li, J., Song, H., Lin, J., Liu, X., and Tang, S. Self-supervised meta-prompt learning with meta-gradient regularization for few-shot generalization. arXiv preprint arXiv:2303.12314, 2023

2023 arXiv
[30]

Towards unified multimodal editing with enhanced knowledge collaboration

Pan, K., Fan, Z., Li, J., Yu, Q., Fei, H., Tang, S., Hong, R., Zhang, H., and Sun, Q. Towards unified multimodal editing with enhanced knowledge collaboration. Advances in Neural Information Processing Systems, 37: 0 110290--110314, 2024 a

2024
[31]

I3: I ntent-i ntrospective retrieval conditioned on i nstructions

Pan, K., Li, J., Wang, W., Fei, H., Song, H., Ji, W., Lin, J., Liu, X., Chua, T.-S., and Tang, S. I3: I ntent-i ntrospective retrieval conditioned on i nstructions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retriev...

2024
[32]

Auto-encoding morph-tokens for multimodal llm

Pan, K., Tang, S., Li, J., Fan, Z., Chow, W., Yan, S., Chua, T.-S., Zhuang, Y., and Zhang, H. Auto-encoding morph-tokens for multimodal llm. arXiv preprint arXiv:2405.01926, 2024 c

2024 arXiv
[33]

Generative multimodal pretraining with discrete diffusion timestep tokens

Pan, K., Lin, W., Yue, Z., Ao, T., Jia, L., Zhao, W., Li, J., Tang, S., and Zhang, H. Generative multimodal pretraining with discrete diffusion timestep tokens. arXiv preprint arXiv:2504.14666, 2025 a

2025 arXiv
[34]

Unlocking aha moments via reinforcement learning: Advancing collaborative visual comprehension and generation

Pan, K., Wu, Y., Bu, W., Shen, K., Li, J., Wang, Y., Li, Y., Tang, S., Xiao, J., Wu, F., et al. Unlocking aha moments via reinforcement learning: Advancing collaborative visual comprehension and generation. arXiv preprint arXiv:2506.01480, 2025 b

2025
[35]

Androidinthewild: A large-scale dataset for android device control

Rawles, C., Li, A., Rodriguez, D., Riva, O., and Lillicrap, T. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024

2024
[36]

Scribeagent: Towards specialized web agents using production-scale workflow data

Shen, J., Jain, A., Xiao, Z., Amlekar, I., Hadji, M., Podolny, A., and Talwalkar, A. Scribeagent: Towards specialized web agents using production-scale workflow data. arXiv preprint arXiv:2411.15004, 2024

2024 arXiv
[37]

Taskbench: Benchmarking large language models for task automation

Shen, Y., Song, K., Tan, X., Zhang, W., Ren, K., Yuan, S., Lu, W., Li, D., and Zhuang, Y. Taskbench: Benchmarking large language models for task automation. arXiv preprint arXiv:2311.18760, 2023

2023 arXiv
[38]

Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents

Wang, L., Deng, Y., Zha, Y., Mao, G., Wang, Q., Min, T., Chen, W., and Chen, S. Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents. arXiv preprint arXiv:2406.08184, 2024 a

2024 arXiv
[39]

Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv prepr...

2024 arXiv
[40]

NE x T - GPT : Any-to-any multimodal LLM

Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. NE x T - GPT : Any-to-any multimodal LLM . In Proceedings of the International Conference on Machine Learning, pp.\ 53366--53397, 2024 a

2024
[41]

P., et al

Wu, Z., Wu, Z., Xu, F., Wang, Y., Sun, Q., Jia, C., Cheng, K., Ding, Z., Chen, L., Liang, P. P., et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024 b

2024 arXiv
[42]

J., Cheng, Z., Shin, D., Lei, F., et al

Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024

2024 arXiv
[43]

Crab: Cross-environment agent benchmark for multimodal language model agents

Xu, T., Chen, L., Wu, D.-J., Chen, Y., Zhang, Z., Yao, X., Xie, Z., Chen, Y., Liu, S., Qian, B., et al. Crab: Cross-environment agent benchmark for multimodal language model agents. arXiv preprint arXiv:2407.01511, 2024 a

2024 arXiv
[44]

Aguvis: Unified pure vision agents for autonomous gui interaction

Xu, Y., Wang, Z., Wang, J., Lu, D., Xie, T., Saha, A., Sahoo, D., Yu, T., and Xiong, C. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024 b

2024 arXiv
[45]

Webshop: Towards scalable real-world web interaction with grounded language agents

Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 0 20744--20757, 2022

2022
[46]

F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al

Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

arXiv 2023

[2] [2]

Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., and Plummer, B. A. A dataset for interactive vision-language navigation with unknown command feasibility. In European Conference on Computer Vision, pp.\ 312--328. Springer, 2022

work page 2022

[3] [3]

Spider2-v: How far are multimodal agents from automating data science and engineering workflows? arXiv preprint arXiv:2407.10956, 2024

Cao, R., Lei, F., Wu, H., Chen, J., Fu, Y., Gao, H., Xiong, X., Zhang, H., Mao, Y., Hu, W., et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? arXiv preprint arXiv:2407.10956, 2024

arXiv 2024

[4] [4]

Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24185--24198, 2024

work page 2024

[5] [5]

Seeclick: Harnessing gui grounding for advanced visual gui agents

Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

arXiv 2024

[6] [6]

Mind2web: Towards a generalist agent for the web

Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024

work page 2024

[7] [7]

Dysen-vdm: Empowering dynamics-aware text-to-video diffusion with llms

Fei, H., Wu, S., Ji, W., Zhang, H., and Chua, T.-S. Dysen-vdm: Empowering dynamics-aware text-to-video diffusion with llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7641--7653, 2024 a

work page 2024

[8] [8]

Video-of-thought: Step-by-step video reasoning from perception to cognition

Fei, H., Wu, S., Ji, W., Zhang, H., Zhang, M., Lee, M.-L., and Hsu, W. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning, 2024 b

work page 2024