Pith. sign in

REVIEW 4 major objections 8 minor 1 cited by

DeepShop: A Benchmark for Deep Research Shopping Agents

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DeepShop, a benchmark of 150 evolved live-Amazon shopping queries, finds that the best current web agent completes only 32% of tasks and the best deep-research systems only 30%.

desk verdict DeepShop is a real advance in shopping-agent benchmarks, but the headline numbers mix 30-query and 150-query runs and the task-answerability audit is too thin to trust exact rates. read the letter →

arxiv 2506.02839 v1 pith:POLFLWVM submitted 2025-06-03 cs.IR cs.AI

Yougang Lyu , Xiaoyu Zhang , Lingyong Yan , Maarten de Rijke , Zhaochun Ren , Xiuying Chen This is my paper
classification cs.IRcs.AI
keywords webagentsonlineshoppingbenchmarke-commercequerycomplexityevolutiondeepresearchfine-grainedevaluationsearchfilterssortingpreferences
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeepShop is a benchmark built to test whether web agents and deep-research systems can handle shopping queries the way real buyers phrase them: with several product attributes, search filters, and sorting preferences combined in one request. The paper's core claim is that existing benchmarks use queries that are too simple and deterministic, so they overstate agent ability. Using 150 tasks evolved from 50 real user queries across Books, Electronics, Home, Fashion, and Sports, evaluated live on Amazon, DeepShop reports that retrieval-only systems score below 8%, the strongest web agent reaches 32% holistic success, and the strongest deep-research systems reach 30%. The paper argues that these low scores trace to specific failures in filtering, sorting, grounding, replanning, and hallucination, and that the benchmark can guide agents toward realistic deployment.

What carries the argument

The machinery of DeepShop is query evolution plus decomposed evaluation. Starting from 50 real shopping seed queries, GPT-4o is prompted to generate diverse variants across five categories and then to iteratively add product attributes, search filters, or sorting preferences over five rounds, producing 600 queries from which a balanced, manually verified set of 150 is selected. The evaluation pipeline then splits each query into three subqueries, attribute, filter, and sort, asks GPT-4o to judge each subquery as success or not from screenshots and final answers, and combines those judgments by a rule so that holistic success requires every explicitly requested component to pass. This decomposition is what lets the paper attribute failures to particular capability gaps rather than to overall task difficulty.

What would settle it

Take the 150 DeepShop tasks on a fixed day and have two independent human raters, who do not know the agent outputs, check each task for a unique, currently available matching Amazon product; if a substantial fraction, say more than 10%, are ambiguous, unverifiable, or unsolvable, then the benchmark's agent failure rates would be inflated by task artifacts.

Watch

Extended reading notes

Core claim

On its own terms, DeepShop establishes that complex, realistic shopping queries are not yet solved by any system class. The benchmark's 150 tasks are generated by GPT-4o diversity and complexity evolution from 50 seed queries, manually verified, and balanced across five product categories; each query decomposes into product-attribute, search-filter, and sorting-preference subqueries. Fine-grained evaluation by GPT-4o, with 80–86% agreement against human judges, plus rule-based holistic success, shows that simple RAG fails because it cannot interact with the website, web agents improve through interaction but rarely satisfy all three constraint types at once, and deep-research systems handle attributes and sorting relatively well but fall short on filters. The headline numbers are 32.00% holistic task success for the best web agent, Browser Use, and 30.00% for both commercial deep-research systems, with performance degrading steadily from easy to hard queries.

Load-bearing premise

The benchmark's validity rests on the assumption that its 150 GPT-4o-generated, manually revised tasks are realistic, unambiguous, and answerable on Amazon at evaluation time; if many tasks lack a unique correct product or become unsolvable as the catalog changes, the low success rates would measure task artifacts rather than agent ability.

Editorial extensions

If this is right

  • Retrieval-augmented generation cannot solve multi-constraint shopping queries: without web interaction, even attributes score 7.33% and filters and sorting score below 6%, so any shopping assistant needs an interactive component.
  • Progress among web agents comes from combining HTML structure with visual grounding, but satisfying attributes, filters, and sorting simultaneously remains the bottleneck; the best web agent still fails two-thirds of tasks.
  • Deep-research systems show that long-horizon reasoning helps on attributes and sorting, yet search filters remain their weakest dimension, which points to dedicated filtering actions and verification on product detail pages as a development target.
  • Because task success falls as query complexity rises, evaluation on easy queries alone will overestimate agent readiness for real shopping; DeepShop's difficulty levels provide a way to measure that gap.
  • The specific error patterns, poor grounding of small widgets, failure to replan after failed searches, limited action spaces, and hallucinated product and return-policy details, give concrete targets for agent design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • [Editorial inference] If the fine-grained subquery decomposition is reliable, it could be reused as a partial-credit reward signal for training shopping agents, since it identifies which of attribute, filter, or sort constraints an agent already satisfies.
  • [Editorial inference] Because DeepShop runs on the live Amazon catalog, its numbers are time-dependent; a natural extension is to re-run the benchmark periodically or pin catalog snapshots to separate agent capability from product-availability drift.
  • [Editorial inference] The close parity between purpose-built web agents and general deep-research systems suggests that high-level planning and constraint tracing, rather than browser manipulation alone, may be the limiting factor; a testable follow-up would give a web agent the deep-research system's planning module and measure the gain.
  • [Editorial inference] The paper's human–GPT agreement rates of 80–86% leave room for evaluation noise; a stricter protocol using majority voting over several independent GPT-4o judgments could sharpen the benchmark's headline scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper introduces DeepShop, a benchmark for evaluating web shopping agents on complex, multi-constraint queries over live Amazon. Starting from 50 seed queries taken from Mind2Web-Live and WebVoyager, the authors use GPT-4o to generate 50 diversity-evolved queries and then apply five rounds of attribute, filter, and sorting complexity evolution, yielding a 600-query pool from which a balanced 150-query subset is selected (30 queries per category across Books, Electronics, Home, Fashion, and Sports). Each query is annotated with product-attribute, search-filter, and sorting-preference subqueries, and assigned to easy, medium, or hard difficulty by the number of evolution rounds. The paper evaluates a simple RAG baseline, four web agents (Agent-E, SeeAct, WebVoyager, Browser Use), and two deep research systems (Gemini Deep Research and OpenAI Deep Research) on the live site, using GPT-4o automatic judgments for web agents and manual judgments for deep research systems. The headline finding is low overall task success (e.g., Browser Use 32.00%, both deep research systems 30.00%), with particularly poor performance on filters and sorting preferences.

Significance. DeepShop addresses a genuine gap: existing online shopping benchmarks largely use short deterministic queries, while realistic shopping tasks involve multiple attributes, filters, and sorting preferences. The paper's strengths are that agents are evaluated in a live environment rather than a static snapshot; the evaluation separates three fine-grained dimensions from holistic success; a human-agreement check is included; and the dataset and code are released. The error analysis, including grounding failures, limited action spaces, and hallucination in deep research systems, is a useful contribution. If the answerability and evaluation-validity concerns raised in the major comments are resolved, DeepShop would be a valuable and much-needed testbed for the web-agent community. The present evidence supports the qualitative conclusion that current systems struggle with complex shopping queries, but the precise numerical ranking across system families is not yet established.

major comments (4)
  1. [§3.5, §5.1, Table 2] The central interpretation that agents fail on DeepShop presupposes that every one of the 150 tasks is answerable and unambiguous on live Amazon at evaluation time. Section 3.5 only says that tasks were 'manually verified and, if necessary, revised' to 'confirm that the answers are available on the corresponding website'; it reports no verification criteria, no revision counts or examples, no inter-annotator agreement, and no protocol for re-checking after catalog changes. Since even a small fraction of broken or drifted tasks would materially lower the reported 16–32% task-success rates, the paper must document the verification process, provide a per-task answerability audit (ideally at evaluation time), and either exclude, repair, or explicitly flag tasks whose ground truth disappears from the live catalog.
  2. [§3.6, §A.2, Table 2] The headline comparison mixes evaluation protocols and sample sizes: web agents and the RAG baseline are scored by GPT-4o on all 150 queries, while both deep research systems are manually scored on only 30 randomly sampled queries (Appendix A.2), and the main table reports no confidence intervals or significance tests across these groups. The paper's claim that Browser Use (32.00%) and the deep research systems (30.00%) perform comparably is therefore not supported by the reported evidence. The authors should evaluate the deep research systems on the same 150-query set or report interval estimates and an appropriate significance test, and should state how the 'first recommended item' rule in Appendix A.2 affects the comparison.
  3. [§3.3, §3.6, §5.1] Same-source bias is not analyzed. GPT-4o generates the query evolutions, powers every web-agent baseline, and serves as the automatic judge; the human-agreement check covers only 50 WebVoyager trajectories and reports aggregate agreement without inter-annotator reliability or a breakdown by model, difficulty, or category. Because the evaluator may be systematically lenient or strict toward outputs of its own model family, the paper should report a confusion-matrix analysis of GPT-4o versus human judgments, ideally on trajectories from several baselines, and discuss the direction of any disagreement.
  4. [§3.4, §5.3, Figure 5(b)] The easy/medium/hard split is defined by the number of complexity-evolution rounds, but the paper does not validate that this grouping corresponds to perceived or ground-truth difficulty independent of agent performance. Queries produced by repeated 'add 10 to 20 words' rewriting (Figure 10) may become redundant or internally contradictory, and Figure 5(b) then partly measures artifacts of query construction. Provide human difficulty ratings or a task-level consistency check for the 150 queries, and report examples of queries at each complexity level in the main text or appendix.
minor comments (8)
  1. [§3.3, §3.4] The paper alternates between 'GPT4-o' and 'GPT-4o'; please use a single consistent spelling throughout.
  2. [§3.5, Figure 4] The term 'DeepShop hard' is used in Figure 4 before it is defined; introduce the definition of the hard subset before presenting these results.
  3. [§3.4] The transition from 100 queries after diversity evolution to 600 queries after five complexity-evolution rounds is implicit; state explicitly that the pool consists of six complexity levels (seed plus five evolutions) over 100 base queries.
  4. [Table 2] It is unclear whether the bold/underline formatting is unique: both deep research systems show 30.00% task success but only one is bold. Clarify the formatting rule or add a note.
  5. [§2 and References] The text refers to 'Mind2Web-Live [34]', but reference [34] is WebCanvas; align the citation or add the correct Mind2Web-Live reference.
  6. [§A.2] Report the number and background of human annotators and whether all 50 trajectories were double-annotated; the current description mentions 'a human annotator' without specifying inter-annotator agreement.
  7. [§3.1] The POMDP tuple omits the reward function R and observation function Ω; either include them or state explicitly that they are not used in the evaluation protocol.
  8. [§5.2, Figure 5(a)] The claim that OpenAI deep research 'fails entirely in both' Fashion and Sports should be qualified by the sample size of the 30-query subset used for deep research systems, since a 0% result on a small sample has wide uncertainty.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DeepShop's central claims rest on external live-web agent runs and human-validated evaluation, not on a derivation that reduces to its own inputs.

full rationale

DeepShop is an empirical benchmark rather than a derivation chain, so the circularity patterns based on self-definition or fitted-input prediction do not apply. The central claims—that RAG, web agents, and deep research systems achieve low success rates on complex shopping queries—are obtained by executing those systems against a live external environment (Amazon) and are not derived from the benchmark's construction. Query generation uses GPT-4o, and the evaluation judge is also GPT-4o, which creates a same-source bias worth noting as a validity concern, but it is not a circular reduction by construction: task success is not defined as 'GPT-4o approves its own output' in a way that forces the reported numbers. The paper separately reports human agreement rates of 84%, 80%, 82%, and 86% on 50 trajectories, providing an independent anchor for the evaluation. The manual verification of task answerability in Section 3.5 is unquantified, but that is a benchmark-quality limitation, not a circular argument. The many self-citations appear in Related Work and Limitations as pointers to prior work or future directions and are not load-bearing for the main empirical claims. No equation, metric, or fitted parameter is shown to be equivalent to another by construction. Therefore the paper is self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

DeepShop does not derive scientific constants or make parameter-free predictions; its central claim depends on design choices and modeling assumptions. The dominant assumptions are that GPT-4o generates realistic and answerable shopping queries, that such products exist on Amazon, and that GPT-4o's screenshot-based judgment is a reliable proxy for human success. The free parameters are the hand-chosen difficulty thresholds, the number of evolution rounds, and the agent step limit, each of which shapes the reported success rates.

free parameters (3)
  • Difficulty thresholds = easy (0-1), medium (2-3), hard (4-5) rounds
    The number of complexity-evolution rounds used to bin tasks into three levels is chosen by hand; no evidence shows these thresholds correspond to actual difficulty (Section 5.3).
  • Evolution rounds T = 5
    T=5 complexity evolution iterations define the 600-query pool and thus the difficulty distribution of the benchmark; this is an arbitrary design choice (Section 3.4).
  • Agent step limit = 15
    All web agents are capped at 15 steps per task, which materially affects success rates on multi-constraint queries that require many interactions (Appendix C.2).
assumptions (4)
  • domain assumption GPT-4o-generated queries are semantically coherent and representative of real shopping intents
    The entire benchmark is built by prompting GPT-4o to rewrite/evolve 50 seed queries; if the model introduces artifacts or unrealistic phrasings, the benchmark measures LLM quirks rather than real shopping behavior (Sections 3.2-3.4).
  • domain assumption Products matching all constraints exist and can be found on Amazon at evaluation time
    Success is defined by finding a product satisfying all constraints; the authors manually verify answerability for the final 150 tasks, but the live catalog changes and the verification is not described in detail (Section 3.5).
  • domain assumption GPT-4o's screenshot-based judgments approximate human judgment for task success
    The fine-grained and holistic metrics rely on GPT-4o judging screenshots; agreement with human judges is 80-86% on a 50-trajectory sample, leaving a nontrivial error rate that could shift reported scores (Section 3.6, Appendix A.2).
  • standard math The POMDP formulation of web shopping tasks is adequate
    The task formulation follows prior work [11,34] and is not challenged in the paper; it is a standard modeling choice (Section 3.1).
invented entities (1)
  • DeepShop benchmark dataset independent evidence
    purpose: The 150-task evaluation set plus the query evolution pipeline used to create it
    The dataset is released on HuggingFace under CC BY 4.0 (Appendix F), providing an external artifact for re-use, though the live Amazon environment prevents independent verification of answerability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepShop: A Benchmark for Deep Research Shopping Agents." pith.science (2026). https://pith.science/paper/POLFLWVM

@misc{pith2026250602839,
  author       = {Pith},
  title        = {Pith review of: DeepShop: A Benchmark for Deep Research Shopping Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/POLFLWVM}},
  note         = {Machine review of arXiv:2506.02839}
}
read the original abstract

Web agents for online shopping have shown great promise in automating user interactions across e-commerce platforms. Benchmarks for assessing such agents do not reflect the complexity of real-world shopping scenarios, as they often consist of overly simple queries with deterministic paths, such as "Find iPhone 15." Real shopping scenarios are inherently more layered, involving multi-dimensional product attributes, search filters, and user-specific sorting preferences. To address this gap, we introduce DeepShop, a benchmark designed to evaluate web agents in complex and realistic online shopping environments. DeepShop comprises three key components. (1) Query diversity evolution: Starting from real user queries, we generate diverse queries across five popular online shopping domains. (2) Query complexity evolution: We further evolve these queries to increase complexity, considering product attributes, search filters, and sorting preferences, and classify them into three levels: easy, medium, and hard, based on the number of evolutions. (3) Fine-grained and holistic evaluation: We propose an automated evaluation framework that assesses agent performance in terms of fine-grained aspects (product attributes, search filters, and sorting preferences) and reports the overall success rate through holistic evaluation. We conduct a systematic evaluation of retrieval-augmented generation (RAG) methods, web agents, and deep research systems. Results show that RAG struggles with complex queries due to its lack of web interaction, while other methods face significant challenges with filters and sorting preferences, leading to low overall success rates. We also perform cross-category, complexity-based evaluations and error analyses to support the advancement of deep research shopping agents.

Figures

Figures reproduced from arXiv: 2506.02839 by the authors.

Figure 1
Figure 1. DeepShop evaluates agents on realistic and complex shopping queries with fine-grained, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Running examples of diversity and complexity evolution in DeepShop. Complexity [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Product category distribution after query diversity evolution. To mitigate this, we construct a balanced subset of 150 queries from our 600-query pool, systematically selecting 30 queries each from five major categories: Books, Elec￾tronics, Home, Fashion, and Sports. Following previous work [11, 12], we manually verify each generated task and, if necessary, revise it to ensure high quality and con￾firm that the ans… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Analysis of query complexity evolution. product attributes, as depicted in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Detailed analysis of performance across different product categories and query complexity. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Prompts for GPT-4o evaluation 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Instructions for human evaluation. final actions.5 Browser Use[29] is an open-source web agent framework that combines visual understanding with HTML structure parsing to support robust web navigation and interaction.6 • Deep research systems: Since we cannot strictly …
Figure 8
Figure 8. Figure 8: Prompts for deep research systems. 5 https://github.com/MinorJerry/WebVoyager 6 https://github.com/browser-use/browser-use 7 https://blog.google/products/gemini/google-gemini-deep-research/ 8 https://openai.com/index/introducing-deep-research/ 17 [PITH_FULL_IMAGE:figu…
Figure 9
Figure 9. Figure 9: Prompts for query diversity evolution. I want you act as a Prompt Rewriter for web shopping. Your objective is to rewrite a given prompt into a more complex version to make those web shopping agents a bit harder to handle. But the rewritten prompt must be reasonable an…
Figure 10
Figure 10. Figure 10: Prompts for query complexity evolution. C.2 Implementation Details We evaluate open-source agents—Agent-E, SeeAct, WebVoyager, and Browser Use—within real￾time web environments. Agent-E, SeeAct, and Browser Use are executed via Playwright, while WebVoyager leverages S…
Figure 11
Figure 11. Figure 11: Prompts for complexity evolution strategies. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Limited grounding ability of web agents. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Illustration of web agent’s failure to reassess and replan. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Illustration of the web agent’s failure to apply the price filter during task execution. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Illustration of the web agent’s failure to learn from execution. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Illustration of hallucination errors in the OpenAI deep research system. The system returns [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Detailed view of the first returned product link. Although the task specifies size Medium, [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory

    cs.CV 2026-08 conditional novelty 6.0 of 10

    FocusMem factorizes latent GUI memory into a role-aware content basis, a state-conditioned readout, and a trust gate, improving frozen-policy web-agent success rates over fixed latent-memory baselines on five benchmarks.

Reference graph

Works this paper leans on

63 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Agent-e: From autonomous web navigation to foundational design principles in agentic systems

    Abuelsaad, T., Akkil, D., Dey, P., Jagmohan, A., Vempaty, A., and Kokku, R. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032, 2024

    arXiv 2024

  2. [2]

    Large language models empowered personalized web agents

    Cai, H., Li, Y ., Wang, W., Zhu, F., Shen, X., Li, W., and Chua, T. Large language models empowered personalized web agents. CoRR, abs/2410.17236, 2024

    arXiv 2024

  3. [3]

    Chatshop: Interactive information seeking with language agents

    Chen, S., Wiseman, S., and Dhingra, B. Chatshop: Interactive information seeking with language agents. arXiv preprint arXiv:2404.09911, 2024

    arXiv 2024

  4. [4]

    I., Fetahu, B., and Malmasi, S

    Chen, Z., Choi, J. I., Fetahu, B., and Malmasi, S. Identifying high consideration e-commerce search queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: EMNLP 2024 - Industry Track, Miami, Florida, USA, November 12-16, 2024, pp. 563–572. Association for Computational Linguistics, 2024

    work page 2024

  5. [5]

    Mind2web: Towards a generalist agent for the web

    Deng, X., Gu, Y ., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y . Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023

    work page 2023

  6. [6]

    S., and Gur, I

    Furuta, H., Lee, K., Nachum, O., Matsuo, Y ., Faust, A., Gu, S. S., and Gur, I. Multimodal web navigation with instruction-finetuned foundation models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2024

    work page 2024

  7. [7]

    Confu- cius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum

    Gao, S., Shi, Z., Zhu, M., Fang, B., Xin, X., Ren, P., Chen, Z., Ma, J., and Ren, Z. Confu- cius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18030–18038, 2024

    work page 2024

  8. [8]

    Try deep research and Gemini 2.0 flash experimental

    Gemini. Try deep research and Gemini 2.0 flash experimental. Gemini Blog, 2025. URL https://blog.google/products/gemini/google-gemini-deep-research/

    work page 2025

Show all 63 references
  1. [9]

    Navigating the digital world as humans do: Universal visual grounding for gui agents

    Gou, B., Wang, R., Zheng, B., Xie, Y ., Chang, C., Shu, Y ., Sun, H., and Su, Y . Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  2. [10]

    V ., Safdari, M., Matsuo, Y ., Eck, D., and Faust, A

    Gur, I., Furuta, H., Huang, A. V ., Safdari, M., Matsuo, Y ., Eck, D., and Faust, A. A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  3. [11]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    He, H., Yao, W., Ma, K., Yu, W., Dai, Y ., Zhang, H., Lan, Z., and Yu, D. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  4. [12]

    OpenWebV oyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization

    He, H., Yao, W., Ma, K., Yu, W., Zhang, H., Fang, T., Lan, Z., and Yu, D. OpenWebV oyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. arXiv preprint arXiv:2410.19609, 2024. 10

    2024 arXiv

  5. [13]

    Query reformulation in e-commerce search

    Hirsch, S., Guy, I., Nus, A., Dagan, A., and Kurland, O. Query reformulation in e-commerce search. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 1319–1...

    2020

  6. [14]

    P., Zhao, D., Bonatti, R., and Koishida, K

    Jang, L., Li, Y ., Ding, C., Lin, J., Liang, P. P., Zhao, D., Bonatti, R., and Koishida, K. Video- WebArena: Evaluating long context multimodal agents with video understanding web tasks. arXiv preprint arXiv:2410.19100, 2024

    2024 arXiv

  7. [15]

    P., Littman, M

    Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artif. Intell., 101(1-2):99–134, 1998

    1998

  8. [16]

    Agent-centric information access

    Kanoulas, E., Eustratiadis, P., Li, Y ., Lyu, Y ., Pal, V ., Poerwawinata, G., Qiao, J., and Wang, Z. Agent-centric information access. arXiv preprint arXiv:2502.19298, 2025

    2025 arXiv

  9. [17]

    Language models can solve computer tasks

    Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023

    2023

  10. [18]

    Y ., Lo, R., Jang, L., Duvvur, V ., Lim, M

    Koh, J. Y ., Lo, R., Jang, L., Duvvur, V ., Lim, M. C., Huang, P., Neubig, G., Zhou, S., Salakhut- dinov, R., and Fried, D. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computationa...

    2024

  11. [19]

    L., Yao, S., Chen, Y ., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y ., and Tang, J

    Lai, H., Liu, X., Iong, I. L., Yao, S., Chen, Y ., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y ., and Tang, J. AutoWebGLM: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, ...

    2024

  12. [20]

    InfiGUI-R1: Advancing multimodal gui agents from reactive actors to deliberative reasoners

    Liu, Y ., Li, P., Xie, C., Hu, X., Han, X., Zhang, S., Yang, H., and Wu, F. InfiGUI-R1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504.14239, 2025

    2025 arXiv

  13. [21]

    H., Kasner, Z., and Reddy, S

    Lù, X. H., Kasner, Z., and Reddy, S. Weblinx: Real-world website navigation with multi-turn dialogue. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    2024

  14. [22]

    Improving legal judgment prediction through reinforced criminal element extraction

    Lyu, Y ., Wang, Z., Ren, Z., Ren, P., Chen, Z., Liu, X., Li, Y ., Li, H., and Song, H. Improving legal judgment prediction through reinforced criminal element extraction. Inf. Process. Manag., 59(1):102780, 2022

    2022

  15. [23]

    Multi- defendant legal judgment prediction via hierarchical reasoning

    Lyu, Y ., Hao, J., Wang, Z., Zhao, K., Gao, S., Ren, P., Chen, Z., Wang, F., and Ren, Z. Multi- defendant legal judgment prediction via hierarchical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2198–22...

    2023

  16. [24]

    Feature-level debiased natural language understanding

    Lyu, Y ., Li, P., Yang, Y ., de Rijke, M., Ren, P., Zhao, Y ., Yin, D., and Ren, Z. Feature-level debiased natural language understanding. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial ...

    2023

  17. [25]

    Knowtuning: Knowledge-aware fine-tuning for large language models

    Lyu, Y ., Yan, L., Wang, S., Shi, H., Yin, D., Ren, P., Chen, Z., de Rijke, M., and Ren, Z. Knowtuning: Knowledge-aware fine-tuning for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA,...

    2024

  18. [26]

    MACPO: weak-to-strong alignment via multi-agent contrastive preference optimization

    Lyu, Y ., Yan, L., Wang, Z., Yin, D., Ren, P., de Rijke, M., and Ren, Z. MACPO: weak-to-strong alignment via multi-agent contrastive preference optimization. CoRR, abs/2410.07672, 2024

    2024 arXiv

  19. [27]

    Cognitive biases in large language models for news recommendation

    Lyu, Y ., Zhang, X., Ren, Z., and de Rijke, M. Cognitive biases in large language models for news recommendation. CoRR, abs/2410.02897, 2024. 11

    2024 arXiv

  20. [28]

    GAIA: a benchmark for general AI assistants

    Mialon, G., Fourrier, C., Wolf, T., LeCun, Y ., and Scialom, T. GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  21. [29]

    and Žuni ˇc, G

    Müller, M. and Žuni ˇc, G. Browser use = state of the art web agent, 2024. URL https: //browser-use.com/posts/sota-technical-report

    2024

  22. [30]

    WebGPT: Browser-assisted question-answering with human feedback

    Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V ., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. WebGPT: Browser-assisted question-answering with human feedba...

    2021 arXiv

  23. [31]

    D., Xie, Z., Kim, S., Zhang, R., Yu, T., Tanjim, M

    Nguyen, D., Chen, J., Wang, Y ., Wu, G., Park, N., Hu, Z., Lyu, H., Wu, J., Aponte, R., Xia, Y ., Li, X., Shi, J., Chen, H., Lai, V . D., Xie, Z., Kim, S., Zhang, R., Yu, T., Tanjim, M. M., Ahmed, N. K., Mathur, P., Yoon, S., Yao, L., Kveton, B., Nguyen, T. H., Bui, T., Zhou, ...

    2024

  24. [32]

    S., et al

    Ning, L., Liang, Z., Jiang, Z., Qu, H., Ding, Y ., Fan, W., Wei, X.-y., Lin, S., Liu, H., Yu, P. S., et al. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. arXiv preprint arXiv:2503.23350, 2025

    2025 arXiv

  25. [33]

    Introducing deep research

    OpenAI. Introducing deep research. OpenAI Blog, 2025. URL https://openai.com/index/ introducing-deep-research/

    2025

  26. [34]

    WebCanvas: Benchmarking web agents in online environments

    Pan, Y ., Kong, D., Zhou, S., Cui, C., Leng, Y ., Jiang, B., Liu, H., Shang, Y ., Zhou, S., Wu, T., et al. WebCanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373, 2024

    2024 arXiv

  27. [35]

    Information discovery in e-commerce

    Ren, Z., He, X., Yin, D., and de Rijke, M. Information discovery in e-commerce. Found. Trends Inf. Retr., 18(4-5):417–690, 2024

    2024

  28. [36]

    From pixels to UI actions: Learning to follow instructions via graphical user interfaces

    Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K. From pixels to UI actions: Learning to follow instructions via graphical user interfaces. In Advances in Neural Information Processing Systems 36: Annual Conference on N...

    2023

  29. [37]

    Learning to use tools via cooperative and interactive agents

    Shi, Z., Gao, S., Chen, X., Feng, Y ., Yan, L., Shi, H., Yin, D., Ren, P., Verberne, S., and Ren, Z. Learning to use tools via cooperative and interactive agents. arXiv preprint arXiv:2403.03031, 2024

    2024 arXiv

  30. [38]

    Generate-then-ground in retrieval- augmented generation for multi-hop question answering

    Shi, Z., Sun, W., Gao, S., Ren, P., Chen, Z., and Ren, Z. Generate-then-ground in retrieval- augmented generation for multi-hop question answering. arXiv preprint arXiv:2406.14891, 2024

    2024 arXiv

  31. [39]

    Tool learning in the wild: Empowering language models as automatic tool agents

    Shi, Z., Gao, S., Yan, L., Feng, Y ., Chen, X., Chen, Z., Yin, D., Verberne, S., and Ren, Z. Tool learning in the wild: Empowering language models as automatic tool agents. In Proceedings of the ACM on Web Conference 2025, pp. 2222–2237, 2025

    2025

  32. [40]

    Direct retrieval-augmented optimization: Synergizing knowledge selection and language models

    Shi, Z., Yan, L., Sun, W., Feng, Y ., Ren, P., Ma, X., Wang, S., Yin, D., de Rijke, M., and Ren, Z. Direct retrieval-augmented optimization: Synergizing knowledge selection and language models. arXiv preprint arXiv:2505.03075, 2025

    2025 arXiv

  33. [41]

    Iterative self-incentivization empowers large language models as agentic searchers

    Shi, Z., Yan, L., Yin, D., Verberne, S., de Rijke, M., and Ren, Z. Iterative self-incentivization empowers large language models as agentic searchers. arXiv preprint arXiv:2505.20128, 2025

    2025 arXiv

  34. [42]

    A taxonomy of queries for e-commerce search

    Sondhi, P., Sharma, M., Kolari, P., and Zhai, C. A taxonomy of queries for e-commerce search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pp. 1245–1248. ACM, 2018

    2018

  35. [43]

    M., Chang, Y ., Nadaf, M., and Iyyer, M

    Song, Y ., Thai, K., Pham, C. M., Chang, Y ., Nadaf, M., and Iyyer, M. BEARCUBS: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.07919, 2025. 12

    2025 arXiv

  36. [44]

    R., Yao, S., Narasimhan, K., and Griffiths, T

    Sumers, T. R., Yao, S., Narasimhan, K., and Griffiths, T. L. Cognitive architectures for language agents. Trans. Mach. Learn. Res., 2024, 2024

    2024

  37. [45]

    H., Kallumadi, S., Murdock, V ., and de Rijke, M

    Tsagkias, M., King, T. H., Kallumadi, S., Murdock, V ., and de Rijke, M. Challenges and research opportunities in ecommerce search and recommendations. SIGIR Forum, 54(1):2:1–2:23, 2020

    2020

  38. [46]

    GUI agents with foundation models: A comprehensive survey

    Wang, S., Liu, W., Chen, J., Gan, W., Zeng, X., Yu, S., Hao, X., Shao, K., Wang, Y ., and Tang, R. GUI agents with foundation models: A comprehensive survey. CoRR, abs/2411.04890, 2024

    2024 arXiv

  39. [47]

    A cooperative multi- agent framework for zero-shot named entity recognition

    Wang, Z., Zhao, Z., Lyu, Y ., Chen, Z., de Rijke, M., and Ren, Z. A cooperative multi- agent framework for zero-shot named entity recognition. In Proceedings of the ACM on Web Conference 2025, pp. 4183–4195, 2025

    2025

  40. [48]

    Z., Gandhi, A., Neubig, G., and Fried, D

    Wang, Z. Z., Gandhi, A., Neubig, G., and Fried, D. Inducing programmatic skills for agentic tasks. arXiv preprint arXiv:2504.06821, 2025

    2025 arXiv

  41. [49]

    Wiz- ardLM: Empowering large language models to follow complex instructions

    Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wiz- ardLM: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  42. [50]

    H., Gou, B., Song, D., Sun, H., and Su, Y

    Xue, T., Qi, W., Shi, T., Song, C. H., Gou, B., Song, D., Sun, H., and Su, Y . An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025

    2025

  43. [51]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  44. [52]

    J., Malaviya, C., Bogin, B., Press, O., and Berant, J

    Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., and Berant, J. Assistantbench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, Novembe...

    2024

  45. [53]

    Large language model-brained GUI agents: A survey

    Zhang, C., He, S., Qian, J., Li, B., Li, L., Qin, S., Kang, Y ., Ma, M., Liu, G., Lin, Q., Rajmohan, S., Zhang, D., and Zhang, Q. Large language model-brained GUI agents: A survey. arXiv preprint arXiv:2411.18279, 2024

    2024 arXiv

  46. [54]

    Towards personalized and semantic retrieval: An end-to-end solution for e-commerce search via embed- ding learning

    Zhang, H., Wang, S., Zhang, K., Tang, Z., Jiang, Y ., Xiao, Y ., Yan, W., and Yang, W. Towards personalized and semantic retrieval: An end-to-end solution for e-commerce search via embed- ding learning. In Proceedings of the 43rd International ACM SIGIR conference on research ...

    2020

  47. [55]

    Variational reasoning over incomplete knowledge graphs for conversational recommendation

    Zhang, X., Xin, X., Li, D., Liu, W., Ren, P., Chen, Z., Ma, J., and Ren, Z. Variational reasoning over incomplete knowledge graphs for conversational recommendation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, Singapor...

    2023

  48. [56]

    Towards empathetic conversational recommender systems

    Zhang, X., Xie, R., Lyu, Y ., Xin, X., Ren, P., Liang, M., Zhang, B., Kang, Z., de Rijke, M., and Ren, Z. Towards empathetic conversational recommender systems. In Proceedings of the 18th ACM Conference on Recommender Systems, RecSys 2024, Bari, Italy, October 14-18, 2024, pp....

    2024

  49. [57]

    Towards empathetic conversational recommender systems

    Zhang, X., Xie, R., Lyu, Y ., Xin, X., Ren, P., Liang, M., Zhang, B., Kang, Z., de Rijke, M., and Ren, Z. Towards empathetic conversational recommender systems. arXiv preprint arXiv:2409.10527, 2024

    2024 arXiv

  50. [58]

    MMInA: Benchmarking multihop multimodal internet agents

    Zhang, Z., Tian, S., Chen, L., and Liu, Z. MMInA: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.09992, 2024

    2024 arXiv

  51. [59]

    Improving sequential recommenders through counterfactual augmentation of system exposure

    Zhao, Z., Ren, Z., Yang, J., Yan, Z., Wang, Z., Yang, L., Ren, P., Chen, Z., de Rijke, M., and Xin, X. Improving sequential recommenders through counterfactual augmentation of system exposure. arXiv preprint arXiv:2504.13482, 2025. 13

    2025 arXiv

  52. [60]

    Gpt-4v(ision) is a generalist web agent, if grounded

    Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y . Gpt-4v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    2024

  53. [61]

    Y ., Jin, X., Wang, Z

    Zheng, B., Fatemi, M. Y ., Jin, X., Wang, Z. Z., Gandhi, A., Song, Y ., Gu, Y ., Srinivasa, J., Liu, G., Neubig, G., et al. SkillWeaver: Web agents can self-improve by discovering and honing skills. arXiv preprint arXiv:2504.07079, 2025

    2025 arXiv

  54. [62]

    A Navy Flower

    Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y ., Fried, D., Alon, U., and Neubig, G. WebArena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, ...

    2024

  55. [2024]

    OpenReview.net, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.