Pith. sign in

REVIEW 3 major objections 5 minor 53 references

SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SmartAgent's three-step thought chain aims to make embodied agents personalize recommendations.

desk verdict New benchmark and task formulation, but the central COUT claim is contradicted by the paper's own ablation: end-to-end training beats the two-stage pipeline on the paper's main personalization metric. read the letter →

arxiv 2412.07472 v3 pith:OMZSJMOJ submitted 2024-12-10 cs.AI

Jiaqi Zhang , Chen Gao , Liyuan Zhang , Yong Li , Hongzhi Yin This is my paper · ORCID
classification cs.AI
keywords Chain-of-User-ThoughtembodiedagentpersonalizedrecommendationGUInavigationlargevision-languagemodelSmartSpotbenchmarkimplicitpreferenceexplicitrequirement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that embodied agents, systems that perceive and act on a user interface, can be trained to personalize by reasoning in three explicit steps: deciding the next GUI action, stating the user's underlying requirement in words, and then answering yes or no to each candidate item. It introduces Chain-of-User-Thought (COUT) as a general paradigm for this progressive reasoning, and SmartAgent as an implementation built on a vision-language model. To test the idea, the authors built SmartSpot, a benchmark of 144 GUI episodes across seven life-service channels, with annotations for both actions and user preferences. The central claim is that this is the first system to achieve full-stage embodied personalized reasoning, from basic screen operation to implicit item preference.

What carries the argument

The central mechanism is the Chain-of-User-Thought pipeline, instantiated as a two-stage training regime. In the embodied stage, a Perceiver model, initialized from the SeeClick GUI agent, maps screenshots, instructions, and eight past actions to the next GUI action; a Reasoner then condenses that action history into a short textual 'underlying requirement.' In the personalization stage, the same Perceiver reads item-pool screenshots and outputs 'Yes' or 'No' per item. The explicit intermediate Thought #2 is the load-bearing connector: it turns ambiguous instructions into concrete constraints that narrow the item pool before recommendation.

What would settle it

On a larger held-out version of SmartSpot, train the same backbone with and without the explicit Thought #2 intermediate text, keeping training compute and data identical; if end-to-end training matches or exceeds the two-stage Imp.Acc across channels, the claim that COUT's intermediate underlying-requirement reasoning improves item recommendation would be refuted. Similarly, an ablation that removes Thought #2 at inference and still gets the same Imp.Acc would show the explicit requirement is not load-bearing.

Watch

Extended reading notes

Core claim

SmartAgent is the first agent to couple GUI navigation, explicit user requirement inference, and implicit item recommendation in a single embodied reasoning chain. The paper's core claim is that a two-stage training pipeline, first teaching a Perceiver model to emit GUI actions (Thought #1) and a Reasoner to summarize the user's underlying requirement (Thought #2), then teaching the same Perceiver to output 'Yes' or 'No' recommendations (Thought #3), enables personalized reasoning that task-only embodied agents lack. The authors report that SmartAgent reaches 0.64 Element Accuracy and 0.50 Step Success Rate on SmartSpot, predicts 71% of explicit underlying requirements correctly, and recommends the correct items 24% of the time. They also report competitive GUI-grounding results on ScreenSpot and second-best pure-vision results on Mind2Web, which they interpret as evidence that adding personalization does not destroy basic embodied skills.

Load-bearing premise

The load-bearing premise is that inserting an explicit 'underlying requirement' text between GUI actions and item selection improves personalized recommendation compared with training the same model end-to-end on the same data; the paper's own Table 5 reports end-to-end Imp.Acc of 0.31 versus 0.24 for two-stage, so that premise is not supported by the reported numbers.

Editorial extensions

If this is right

  • If COUT works, personal assistants on phones and web platforms can be trained on ambiguous queries without golden action trajectories, allowing diverse user paths as supervision.
  • The explicit underlying-requirement text gives a human-readable explanation for why an item was recommended, which could support debugging and user trust.
  • The same two-stage recipe could transfer to other cyber environments such as e-commerce, travel booking, and hotel selection wherever GUI access to an item pool is followed by selection.
  • The zero-shot result on the held-out MEDICINE channel suggests that COUT agents can be dropped into new channels with little or no fine-tuning, which matters for cold-start personalization.
  • Combining GUI behavior signals like clicks and scrolls with item content could improve recommendation in scenarios where back-end item sequences are unavailable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported Table 5 comparison suggests that the intermediate underlying-requirement text may not actually improve item recommendation accuracy over end-to-end training; a reader might infer that COUT's benefit lies in interpretability or step-wise supervision rather than raw Imp.Acc.
  • The benchmark's small scale of 144 episodes and over 1,400 steps means the headline numbers are early evidence; an extension would be to build a larger, multi-platform version to test whether the two-stage advantage, or lack of it, persists.
  • If the explicit requirement text is what enables zero-shot transfer on MEDICINE, then a testable extension is to measure how recommendation quality degrades when Thought #2 is corrupted or removed at inference.
  • The paper's framing suggests that user-oriented diversity, not task completion, is the ultimate objective; a further implication is that evaluation metrics for embodied agents should include preference alignment, not just action accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes Chain-of-User-Thought (COUT), a personalized reasoning paradigm for embodied GUI agents in which the agent first navigates a GUI (Thought #1), then summarizes the user's explicit underlying requirement (Thought #2), and finally recommends items from the resulting pool (Thought #3). The authors instantiate COUT in SmartAgent, a Qwen-VL-based model trained with LoRA in two stages, and introduce SmartSpot, a 144-episode benchmark built from Meituan GUI trajectories with annotations for instructions, underlying requirements, and item recommendations. The evaluation reports embodied action metrics (Ele.Acc, SSR) and personalized metrics (Exp.Acc, Imp.Acc) on SmartSpot, plus transfer results on ScreenSpot and Mind2Web.

Significance. The paper addresses a real gap: embodied agents that handle ambiguous user queries and personalize item recommendations are rarely studied, and the SmartSpot benchmark with explicit annotations for embodied actions and user preferences is a potentially useful community resource. The authors also evaluate on standard GUI benchmarks (ScreenSpot, Mind2Web), which is good practice. However, the central claim that COUT's explicit intermediate reasoning improves personalized recommendation is not supported by the reported experiments: the end-to-end baseline in Table 5 obtains a higher Imp.Acc than the proposed two-stage pipeline, and the absolute values of the key personalized metric are low. As submitted, the evidence does not substantiate the 'first full-stage embodied personalized reasoning' claim.

major comments (3)
  1. [Section 6.6, Table 5] The ablation reported in Table 5 undercuts the central claim of the paper. End-to-End training without the explicit underlying-requirement step achieves Imp.Acc 0.31, while the proposed Two-stage COUT pipeline achieves 0.24; End-to-End is also higher on Ele.Acc (0.67 vs. 0.64) and SSR (0.53 vs. 0.50). Since Imp.Acc is the paper's own metric for implicit personalized recommendation, the reported data do not support the claim that the COUT intermediate improves personalized reasoning and in fact suggest the opposite. The explanation in Section 6.6 (overfitting to long episodes) is post hoc, and no confidence intervals, number of random seeds, or per-channel breakdowns are reported. This is a load-bearing issue for the paper's main contribution.
  2. [Section 6.1, Table 2; Section 4.2] The claim of 'first full-stage embodied personalized reasoning' is evaluated only on the self-constructed SmartSpot benchmark. There is no external personalized benchmark, and the baselines in Table 2 report no Exp.Acc or Imp.Acc values, so no comparison supports the personalized claims. The absolute performance is low: SmartAgent's overall Imp.Acc is 0.24, and zero-shot Imp.Acc in Table 4 is 0.14. Additionally, the benchmark construction in Section 4.2 relies on annotator-generated intention seeds and the authors' own definitions of explicit and implicit preferences, but no inter-annotator agreement or validation of these definitions is provided. The generalizability of the task formulation therefore remains unverified.
  3. [Section 6.5, Table 4] The zero-shot results are described as showing that SmartAgent 'surprisingly exceeds' full-stage fine-tuning on Exp.Acc (0.77 vs. 0.71) and as evidence of 'preliminary zero-shot reasoning capability.' However, the same table reports zero-shot Imp.Acc of 0.14, which is far below the 50% chance level of the binary Yes/No recommendation action under a balanced class assumption. No statistical tests or confidence intervals are provided, and the overfitting explanation is speculative. The evidence does not support the claim of robust zero-shot personalized reasoning.
minor comments (5)
  1. [Section 4.1, Table 1] The dataset contains only 144 episodes and roughly 1,400 steps total, with 10-20 episodes per channel. Results on such a small benchmark are likely noisy; please report the train/validation/test split and per-channel standard errors or bootstrap confidence intervals.
  2. [Section 5.2] The distinction between the Perceiver and the Reasoner is underspecified: it is unclear whether both are initialized from the same Qwen-VL weights, whether the Reasoner is fine-tuned or frozen, and how Thought #2 is fed into the personalized stage.
  3. [Section 6, Metrics] Exp.Acc is defined only as 'semantic similarity' between the predicted underlying requirement and ground truth, but no similarity measure, threshold, or aggregation procedure is specified, making the reported 0.71 value difficult to interpret or reproduce.
  4. [Section 6.3, Appendix Table 6] On Mind2Web, SmartAgent is below SeeClick on most metrics (for example, Cross-Task SSR 20.6 vs. 25.5), so the statement that it achieves 'second place generally' should be qualified to the specific category and metric.
  5. [Section 6.4, Figure 5] The case study in Figure 5 contains apparent typos (e.g., a flight duration '23.5h' and a price ￥710 for a 7-hour route), which should be corrected before publication.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: SmartAgent's COUT derivation and evaluation are empirically grounded, with only minor non-load-bearing self-citations.

full rationale

The paper's central derivation chain is: define COUT as a three-thought reasoning paradigm, construct the SmartSpot benchmark to instantiate it, train SmartAgent in two stages to produce those thoughts, and evaluate on SmartSpot plus external benchmarks. No equation or construction makes a predicted quantity equal to an input by definition. Equation (1) merely defines the action space conditioned on task goal and user preference; it is not derived from the evaluation metrics. The benchmark is author-created, but the agent is trained and tested on it in a standard supervised manner, and the paper also reports results on the external ScreenSpot and Mind2Web benchmarks, so the evaluation is not self-referential. The metrics (Ele.Acc, SSR, Exp.Acc, Imp.Acc) are computed against held-out ground-truth actions and annotated requirements, not against fitted constants. Notably, the paper's own ablation in Table 5 shows end-to-end training outperforming the proposed two-stage COUT pipeline on Imp.Acc (0.31 vs 0.24), which is the opposite of what a circular argument would force; this is evidence that the reported numbers are empirical rather than constructed. The zero-shot MEDICINE evaluation uses a held-out channel, and all self-citations (e.g., EmbodiedCity, NineRec, on-device recommender systems) appear in related-work or implementation context and are not load-bearing for the paper's core claims. Therefore, no circular step meets the quoted-evidence standard, and any circularity concern is limited to the usual in-domain benchmark caveat, not to a derivation that reduces to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new mathematical objects or physical entities. It relies on standard machine learning assumptions about pretrained models and the validity of its self-built benchmark. The key dependencies are the representativeness of SmartSpot and the suitability of semantic similarity as an accuracy measure.

assumptions (3)
  • domain assumption The Qwen-VL backbone, initialized from SeeClick, provides sufficiently strong GUI grounding and multimodal understanding for the personalized reasoning tasks.
    The method relies on the pretrained LVLM to process screenshots and instructions; Section 5.3 states the training starts from SeeClick base model, but no independent verification of this capability is provided.
  • domain assumption Semantic similarity between the predicted and ground-truth 'underlying requirement' is a valid measure of explicit preference accuracy.
    Exp.Acc is computed via semantic similarity without specifying the exact model or threshold, implying that text similarity captures the correctness of the inferred user need.
  • domain assumption The SmartSpot dataset, with 144 episodes collected by experienced annotators from Meituan, is representative of real-world embodied personalized tasks.
    The benchmark is self-constructed and small; Section 4.2 describes generating instruction pairs from intention seeds and annotator GUI operations, but no external validation of representativeness is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World." pith.science (2026). https://pith.science/paper/OMZSJMOJ

@misc{pith2026241207472,
  author       = {Pith},
  title        = {Pith review of: SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OMZSJMOJ}},
  note         = {Machine review of arXiv:2412.07472}
}
read the original abstract

Recent advances in embodied agents with multimodal perception and reasoning capabilities based on large vision-language models (LVLMs), excel in autonomously interacting either real or cyber worlds, helping people make intelligent decisions in complex environments. However, the current works are normally optimized by golden action trajectories or ideal task-oriented solutions toward a definitive goal. This paradigm considers limited user-oriented factors, which could be the reason for their performance reduction in a wide range of personal assistant applications. To address this, we propose Chain-of-User-Thought (COUT), a novel embodied reasoning paradigm that takes a chain of thought from basic action thinking to explicit and implicit personalized preference thought to incorporate personalized factors into autonomous agent learning. To target COUT, we introduce SmartAgent, an agent framework perceiving cyber environments and reasoning personalized requirements as 1) interacting with GUI to access an item pool, 2) generating users' explicit requirements implied by previous actions, and 3) recommending items to fulfill users' implicit requirements. To demonstrate SmartAgent's capabilities, we also create a brand-new dataset SmartSpot that offers a full-stage personalized action-involved environment. To our best knowledge, our work is the first to formulate the COUT process, serving as a preliminary attempt towards embodied personalized agent learning. Our extensive experiments on SmartSpot illuminate SmartAgent's functionality among a series of embodied and personalized sub-tasks. We will release code and data upon paper notification at https://github.com/tsinghua-fib-lab/SmartAgent.

Figures

Figures reproduced from arXiv: 2412.07472 by the authors.

Figure 1
Figure 1. The Chain-of-User-Thought (COUT) reasoning paradigm. The red line shows a sequence of GUI actions, while the blue line illustrates our three-step thought process. In Thought #1, according to a user’s instruction, an agent performs GUI actions to search for an item pool. In Thought #2 with seeing the pool, the agent reasons underlying requirements behind the original instruction, as implied by the previous actions. I… view at source ↗
Figure 2
Figure 2. The full-stage embodied personalized capabilities of our proposed SmartAgent, range from basic environment cognition to advanced user personal intention reasoning. We quantitatively evaluate SmartAgent’s functionality in comprehensive embodied and personalized sub-tasks, in￾cluding GUI Grounding, Autonomous GUI Operation, Un￾derlying Reasoning, Personalized Recommendation, and Zero-shot Reasoning, as illustrated in … view at source ↗
Figure 3
Figure 3. Two-stage training paradigm of SmartAgent. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of methods on SmartSpot. SmartAgent performs comparable and even better with GUI specialist model and gen￾eral LLM in all scenarios. Notably, in the more complex scenarios TRAVEL1 and TRAVEL2, which involve longer episodes, SmartAgent consistently shows exce…
Figure 5
Figure 5. Figure 5: Case study for embodied personalized reasoning. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 25 canonical work pages

  1. [51]

    Android in the 11 zoo: Chain-of-action-thought for gui agents

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the 11 zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713, 2024. 4

    arXiv 2024

  2. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 7

    arXiv 2023

  3. [2]

    Haptic perception us- ing optoelectronic robotic flesh for embodied artificially in- telligent agents

    Jose A Barreiros, Artemis Xu, Sofya Pugach, Narahari Iyen- gar, Graeme Troxell, Alexander Cornwell, Samantha Hong, Bart Selman, and Robert F Shepherd. Haptic perception us- ing optoelectronic robotic flesh for embodied artificially in- telligent agents. Science Robotics, 7(67):eabi6745, 2022. 2, 4

    work page 2022

  4. [3]

    Seeclick: Har- nessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yan- tao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Har- nessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024. 4, 7, 8

    arXiv 2024

  5. [4]

    An image dataset for benchmarking recommender systems with raw pixels

    Yu Cheng, Yunzhu Pan, Jiaqi Zhang, Yongxin Ni, Aixin Sun, and Fajie Yuan. An image dataset for benchmarking recommender systems with raw pixels. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), pages 418–426. SIAM, 2024. 4

    work page 2024

  6. [5]

    Mobile-bench: An evaluation benchmark for llm-based mo- bile agents

    Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, et al. Mobile-bench: An evaluation benchmark for llm-based mo- bile agents. arXiv preprint arXiv:2407.00993, 2024. 4

    arXiv 2024

  7. [6]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024. 2, 4, 7, 8

    work page 2024

  8. [7]

    Palm- e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm- e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 2

    arXiv 2023

Show all 53 references
  1. [8]

    Workarena: How capable are web agents at solv- ing common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Is- sam H Laradji, Manuel Del Verme, Tom Marty, L ´eo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solv- ing common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024. 2

    2024 arXiv

  2. [9]

    A survey of embodied ai: From simulators to research tasks

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230–244, 2022. 2

    2022

  3. [10]

    A social path to human- like artificial intelligence

    Edgar A Du ´e˜nez-Guzm´an, Suzanne Sadedin, Jane X Wang, Kevin R McKee, and Joel Z Leibo. A social path to human- like artificial intelligence. Nature Machine Intelligence , 5 (11):1181–1188, 2023. 2

    2023

  4. [11]

    Exploring adapter-based transfer learning for rec- ommender systems: Empirical studies and practical insights

    Junchen Fu, Fajie Yuan, Yu Song, Zheng Yuan, Mingyue Cheng, Shenghui Cheng, Jiaqi Zhang, Jie Wang, and Yun- zhu Pan. Exploring adapter-based transfer learning for rec- ommender systems: Empirical studies and practical insights. In Proceedings of the 17th ACM International Conf...

    2024

  5. [12]

    Simulating human society with large language model agents: City, social media, and economic system

    Chen Gao, Fengli Xu, Xu Chen, Xiang Wang, Xiangnan He, and Yong Li. Simulating human society with large language model agents: City, social media, and economic system. In Companion Proceedings of the ACM on Web Conference 2024, pages 1290–1293, 2024. 2

    2024

  6. [13]

    Embodiedcity: A benchmark platform for embodied agent in real-world city environment

    Chen Gao, Baining Zhao, Weichen Zhang, Jinzhu Mao, Jun Zhang, Zhiheng Zheng, Fanhang Man, Jianjie Fang, Zile Zhou, Jinqiang Cui, et al. Embodiedcity: A benchmark platform for embodied agent in real-world city environment. arXiv preprint arXiv:2410.09604, 2024. 4

    arXiv

  7. [14]

    Recommendation as language process- ing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5)

    Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language process- ing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, pages 299–315, 2022. 4

    2022

  8. [15]

    Deepfm: a factorization-machine based neural network for ctr prediction

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr prediction. arXiv preprint arXiv:1703.04247,

    arXiv

  9. [16]

    We- bvoyager: Building an end-to-end web agent with large mul- timodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. We- bvoyager: Building an end-to-end web agent with large mul- timodal models. arXiv preprint arXiv:2401.13919, 2024. 4

    2024 arXiv

  10. [17]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281– 1...

    2024

  11. [18]

    Bridging language and items for retrieval and recommendation

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation. arXiv preprint arXiv:2403.03952, 2024. 4

    2024 arXiv

  12. [19]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2021. 7

    2021

  13. [20]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023. 4, 6

    2023 arXiv

  14. [21]

    Self-attentive se- quential recommendation

    Wang-Cheng Kang and Julian McAuley. Self-attentive se- quential recommendation. In 2018 IEEE international con- ference on data mining (ICDM) , pages 197–206. IEEE,

    2018

  15. [22]

    Lan- guage models can solve computer tasks

    Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Lan- guage models can solve computer tasks. Advances in Neural Information Processing Systems, 36, 2024. 2, 4

    2024

  16. [23]

    On the limits of evaluating embodied agent model generalization using validation sets

    Hyounghun Kim, Aishwarya Padmakumar, Di Jin, Mohit Bansal, and Dilek Hakkani-Tur. On the limits of evaluating embodied agent model generalization using validation sets. arXiv preprint arXiv:2205.09249, 2022. 2

    2022 arXiv

  17. [24]

    Matrix fac- torization techniques for recommender systems

    Yehuda Koren, Robert Bell, and Chris V olinsky. Matrix fac- torization techniques for recommender systems. Computer, 42(8):30–37, 2009. 4

    2009

  18. [25]

    Exploring the upper limits of text- based collaborative filtering using large language models: Discoveries and insights

    Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. Exploring the upper limits of text- based collaborative filtering using large language models: Discoveries and insights. arXiv preprint arXiv:2305.11700,

  19. [26]

    Mapping natural language instructions to mobile ui action sequences

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776, 2020. 4

    2005 arXiv

  20. [27]

    Coco-agent: A comprehensive cognitive mllm agent for smartphone gui automation

    Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. Coco-agent: A comprehensive cognitive mllm agent for smartphone gui automation. arXiv preprint arXiv:2402.11941v3, 2024. 4

    2024 arXiv

  21. [28]

    Gaia: a benchmark for general ai assistants

    Gr ´egoire Mialon, Cl´ementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023. 4

    2023 arXiv

  22. [29]

    Scalable recommendation of wikipedia articles to editors using representation learning

    Oleksii Moskalenko, Denis Parra, and Diego Saez- Trumper. Scalable recommendation of wikipedia articles to editors using representation learning. arXiv preprint arXiv:2009.11771, 2020. 4

    2009 arXiv

  23. [30]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024. 2, 3, 4

    2024 arXiv

  24. [31]

    Androidinthewild: A large- scale dataset for android device control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large- scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024. 2, 4, 5

    2024

  25. [32]

    Bpr: Bayesian personalized rank- ing from implicit feedback

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized rank- ing from implicit feedback. arXiv preprint arXiv:1205.2618,

    arXiv

  26. [33]

    World of bits: An open-domain plat- form for web-based agents

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernan- dez, and Percy Liang. World of bits: An open-domain plat- form for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR, 2017. 4

    2017

  27. [34]

    Behav- ior: Benchmark for everyday household activities in virtual, interactive, and ecological environments

    Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Mart´ın-Mart´ın, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behav- ior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In ...

    2022

  28. [35]

    Towards general computer con- trol: A multimodal agent for red dead redemption ii as a case study

    Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. Towards general computer con- trol: A multimodal agent for red dead redemption ii as a case study. arXiv preprint arXiv:2403.03186, 2024. 2, 4

    2024 arXiv

  29. [36]

    Enabling conversational interaction with mobile ui using large language models

    Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17, 2023. 4

    2023

  30. [37]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024. 4

    2024 arXiv

  31. [38]

    Llmrec: Large language models with graph augmentation for recommendation

    Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Llmrec: Large language models with graph augmentation for recommendation. In Proceedings of the 17th ACM Interna- tional Conference on Web Search and Data Mining , pag...

    2024

  32. [39]

    A survey on large language models for recommendation

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Heng- shu Zhu, Qi Liu, et al. A survey on large language models for recommendation. World Wide Web, 27(5):60, 2024. 4

    2024

  33. [40]

    Osworld: Benchmark- ing multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmark- ing multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024. 4

    2024 arXiv

  34. [41]

    Ur- ban generative intelligence (ugi): A foundational platform for agents in embodied city environment

    Fengli Xu, Jun Zhang, Chen Gao, Jie Feng, and Yong Li. Ur- ban generative intelligence (ugi): A foundational platform for agents in embodied city environment. arXiv preprint arXiv:2312.11813, 2023. 2

    2023 arXiv

  35. [42]

    Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation

    An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Lin- jie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562, 2023. 4

    2023 arXiv

  36. [43]

    Octopus: Embodied vision-language programmer from environmental feedback

    Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. In European Conference on Computer Vision , pages 20–38. Springer,

  37. [44]

    Appagent: Multimodal agents as smartphone users

    Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771 ,

    arXiv

  38. [45]

    On- device recommender systems: A comprehensive survey

    Hongzhi Yin, Liang Qu, Tong Chen, Wei Yuan, Ruiqi Zheng, Jing Long, Xin Xia, Yuhui Shi, and Chengqi Zhang. On- device recommender systems: A comprehensive survey. arXiv preprint arXiv:2401.11441, 2024. 4

    2024 arXiv

  39. [46]

    A simple convolutional generative network for next item recommendation

    Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joe- mon M Jose, and Xiangnan He. A simple convolutional generative network for next item recommendation. In Pro- ceedings of the twelfth ACM international conference on web search and data mining, pages 582–590, 2019. 4

    2019

  40. [47]

    Federated unlearning for on-device rec- ommendation

    Wei Yuan, Hongzhi Yin, Fangzhao Wu, Shijie Zhang, Tieke He, and Hao Wang. Federated unlearning for on-device rec- ommendation. In Proceedings of the sixteenth ACM inter- national conference on web search and data mining , pages 393–401, 2023. 4

    2023

  41. [48]

    Where to go next for recommender systems? id-vs

    Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? id-vs. modality-based recommender models revisited. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development...

    2023

  42. [49]

    You only look at screens: Multimodal chain-of-action agents

    Zhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309.11436, 2023. 4

    2023 arXiv

  43. [50]

    Ninerec: A benchmark dataset suite for evaluating transfer- able recommendation

    Jiaqi Zhang, Yu Cheng, Yongxin Ni, Yunzhu Pan, Zheng Yuan, Junchen Fu, Youhua Li, Jie Wang, and Fajie Yuan. Ninerec: A benchmark dataset suite for evaluating transfer- able recommendation. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2024. 4

    2024

  44. [52]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. In The Twelfth International Confer- ence on Learning Representations, 2023. 4 12

    2023

  45. [53]

    More Results This section shows SmartAgent’s performance results on autonomous GUI operation on the benchmark Mind2Web in Table 6. Table 6. Results of Autonomous GUI operation on Mind2Web. Methods Pure Visual Cross-Task Cross-Website Cross-Domain Ele.Acc Op.F1 SSR Ele.Acc Op.F...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.