Pith. sign in

REVIEW 3 major objections 3 minor 3 cited by

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces MM-BrowseComp, a 224-question benchmark for multimodal browsing agents, and shows that even the strongest tested model, OpenAI o3 with tools, reaches only 29.02% accuracy, suggesting current agents lack native multimoda

desk verdict A useful multimodal extension of BrowseComp whose headline accuracy figure is only as good as the benchmark-construction details the abstract does not disclose. read the letter →

arxiv 2508.13186 v1 pith:CL3OXFOQ submitted 2025-08-14 cs.CL cs.AIcs.CV

Shilong Li , Xingyuan Bu , Wenjie Wang , Jiaheng Liu , Jun Dong , Haoyang He , Hao Lu , Haozhe Zhang
show 16 more authors
Chenchen Jing Zhen Li Chuanhao Li Jiayi Tian Chenchen Zhang Tianhao Peng Yancheng He Jihao Gu Yuanxing Zhang Jian Yang Ge Zhang Wenhao Huang Wangchunshu Zhou Zhaoxiang Zhang Ruizhe Ding Shilei Wen
This is my paper · ORCID
classification cs.CLcs.AIcs.CV
keywords multimodalbrowsingagentswebagentbenchmarkvisualreasoningdeepsearchevaluationdependencychecklist
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MM-BrowseComp is a benchmark of 224 hand-crafted questions that require AI browsing agents to understand images in the query itself and to extract and reason about multimodal content embedded in webpages. The paper's central claim is that current state-of-the-art models, including OpenAI o3 with tools, perform poorly on this benchmark, achieving only 29.02% accuracy, which the authors interpret as evidence that these models lack native multimodal reasoning. Each question comes with a verified checklist, so the benchmark can pinpoint where along the search-and-reasoning path the multimodal dependency breaks.

What carries the argument

The central object is the benchmark itself: a dataset of 224 hand-crafted questions with verified checklists. The checklist is the key mechanism: for each question, it records the intended answer and the multimodal dependencies along the search/reasoning path, allowing fine-grained analysis of whether failure is due to perception, retrieval, or reasoning.

What would settle it

A reader could test whether independent human researchers, given the same browsing tools and queries, can reproduce the verified checklists' ground truth; if humans disagree on a substantial fraction of the 224 questions, then the low model scores would not cleanly indicate a capability gap.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that multimodal browsing is a distinct and currently unsolved capability: when retrieval and reasoning require processing images or videos, even frontier models with browsing tools fail most of the time. The authors establish this by hand-crafting 224 questions that cannot be answered from text alone, attaching verified checklists to each, and measuring the accuracy of several state-of-the-art models.

Load-bearing premise

The 224 hand-crafted questions and their verified checklists are a representative and correctly specified sample of multimodal browsing tasks; if the questions are ambiguous or the checklists are arbitrary, the low accuracy would not reflect a real capability gap.

Editorial extensions

If this is right

  • If the benchmark reflects real tasks, current agents cannot be trusted for open-ended web research that involves images or videos.
  • The verified checklists make it possible to localize where multimodal reasoning fails, guiding targeted improvements in agent architectures.
  • Models that claim native multimodal capabilities can be objectively measured against a hard, realistic test.
  • Text-only agents are insufficient for a large class of browsing queries, implying that multimodal input and output pipelines are necessary for deep search.
  • The benchmark's fine-grained labels could support error analysis and training data generation for multimodal browsing agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to break down failure modes by checklist step (e.g., locating the right image vs. interpreting it) to see whether perception or reasoning dominates the 29.02% accuracy ceiling.
  • The benchmark could be adapted to test agents that act on live video content directly rather than only on static frames or transcripts.
  • The paper's claim about lacking 'native multimodal reasoning' suggests that merely appending vision encoders to text agents may not close the gap; architectural changes that intertwine visual and textual reasoning would be needed.
  • The 224 questions are hand-crafted and may be harder than typical real-world multimodal browsing, but if they are also ambiguous, the low accuracy could overstate the capability gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces MM-BrowseComp, a benchmark of 224 hand-crafted multimodal browsing questions with verified checklists, designed to test agents that must retrieve and reason over images and videos in addition to text. The abstract reports that even OpenAI o3 with tools achieves only 29.02% accuracy, concluding that current models have suboptimal multimodal browsing capabilities and lack native multimodal reasoning. The present review is based on the abstract alone because the full text was not available.

Significance. If the benchmark construction and evaluation are sound, this addresses a genuine gap: existing browsing benchmarks such as BrowseComp are text-centric, while real web browsing is multimodal. The idea of providing a verified checklist per question for fine-grained analysis of multimodal dependencies is a useful methodological contribution. However, the abstract supplies no evidence that the 224 questions are valid, the checklists are reliable, or the evaluation is replicable. The headline accuracy therefore cannot yet be interpreted as a robust measurement of a capability gap.

major comments (3)
  1. [Abstract] The central accuracy claim (o3 with tools: 29.02%) is reported without error bars, confidence intervals, number of runs, or any measure of variance. With only 224 questions, the binomial standard error is about 3 percentage points, so differences of a few points between models are not meaningful unless the protocol provides per-run or per-model variance. The abstract should state how many runs were performed and how the point estimate was aggregated.
  2. [Abstract] No information is given about how the 224 questions were constructed, selected, or validated. It is not stated whether annotators agreed on the questions' multimodal necessity, whether pilot testing was done to remove ambiguities, or whether any questions were discarded for being answerable from text alone. Without such evidence, the low accuracy could be an artifact of question design rather than a genuine multimodal capability gap. The full paper needs to describe the construction protocol, inclusion criteria, and inter-annotator agreement.
  3. [Abstract] The notion of a 'verified checklist' is load-bearing for both the fine-grained analysis and the overall accuracy numbers, but the abstract does not explain how checklists were produced or validated, nor how a model's answer was graded against them. The checklist grading may contain subjective judgment about what counts as a necessary reasoning step or a correct answer. The paper should report the grading procedure, whether graders were blinded, and inter-grader reliability.
minor comments (3)
  1. [Abstract] The phrase 'suboptimal multimodal capabilities' is vague; the paper should specify the comparison baseline or the expected level of performance being assumed.
  2. [Abstract] The term 'native multimodal reasoning' is not defined. It could mean end-to-end training on interleaved image-text data, architectural integration of vision and language, or the absence of explicit tool-mediated vision. A precise definition would help readers interpret the claim.
  3. [Abstract] The abstract should indicate whether the evaluation was conducted by humans, by an LLM judge, or by an automated comparison, since this materially affects the reliability of the reported 29.02% accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detected; benchmark measurement is not a derivation from its inputs.

full rationale

The available manuscript text is the abstract only. MM-BrowseComp is presented as a hand-crafted benchmark of 224 questions with verified checklists, followed by an evaluation of existing models. There is no derivation chain, fitted parameter called a prediction, or self-citation that carries the central claim. The claim that top models achieve 29.02% accuracy is an empirical measurement, not a quantity constructed from those models' outputs. Benchmark validity concerns (ambiguity, checklist objectivity, protocol fairness) would be correctness or external-validity risks, not circularity, and the abstract provides insufficient detail to establish any specific reduction of the evaluation to its inputs. No circular step can be quoted or exhibited, so the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

No free parameters or invented entities are evident from the abstract. The benchmark relies on the assumption that the question set and checklists are representative and unbiased measures of multimodal browsing skill.

assumptions (1)
  • domain assumption The 224 hand-crafted questions and their verified checklists are valid instruments for measuring multimodal browsing ability.
    The abstract asserts that the benchmark evaluates multimodal retrieval and reasoning, but does not provide evidence of question validation, inter-annotator agreement, or bias testing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents." pith.science (2026). https://pith.science/paper/CL3OXFOQ

@misc{pith2026250813186,
  author       = {Pith},
  title        = {Pith review of: MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CL3OXFOQ}},
  note         = {Machine review of arXiv:2508.13186}
}
read the original abstract

AI agents with advanced reasoning and tool use capabilities have demonstrated impressive performance in web browsing for deep search. While existing benchmarks such as BrowseComp evaluate these browsing abilities, they primarily focus on textual information, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 224 challenging, hand-crafted questions specifically designed to assess agents' multimodal retrieval and reasoning capabilities. These questions often incorporate images in prompts, and crucial information encountered during the search and reasoning process may also be embedded within images or videos on webpages. Consequently, methods relying solely on text prove insufficient for our benchmark. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp reveals that even top models like OpenAI o3 with tools achieve only 29.02\% accuracy, highlighting the suboptimal multimodal capabilities and lack of native multimodal reasoning in current models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MMShopBench, built from real multimodal shopping logs, shows even the best agent satisfies the full request in under two-thirds of cases, and fine-tuning on 900 real-log trajectories substantially closes the gap to pr...

  2. DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories

    cs.CV 2026-02 conditional novelty 6.5 of 10

    The paper reframes image retrieval as agentic exploration over personal visual histories and shows the best tested multimodal agent scores only 28.7 exact match on its new DISBench benchmark.

  3. UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.

Reference graph

Works this paper leans on

55 extracted references · 27 canonical work pages · cited by 3 Pith papers

  1. [1]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. A...

    arXiv 2025

  2. [2]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Dama...

    arXiv 2025

  3. [3]

    Deepresearch bench: A comprehensive benchmark for deep research agents

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. 2025. URL https://api.semanticscholar.org/CorpusID:279391682

    work page 2025

  4. [4]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108--24118, 2025

    work page 2025

  5. [5]

    Gaia benchmark: Results public dataset, 2025

    GAIA. Gaia benchmark: Results public dataset, 2025. URL https://huggingface.co/datasets/gaia-benchmark/results_public

    work page 2025

  6. [6]

    Gemini Deep Research

    Google. Gemini Deep Research . Google blog, 2024. URL https://gemini.google/overview/deep-research/?hl=en

    work page 2024

  7. [7]

    Gemini 2.5

    Google. Gemini 2.5 . Google blog, 2025 a . URL https://blog.google/technology/google-deepmind/google-gemini-updates-io-2025/

    work page 2025

  8. [8]

    Google. Gemini . 2025 b . URL https://gemini.google.com/

    work page 2025

Show all 55 references
  1. [9]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  2. [10]

    Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation

    Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, and Guohao Li. Owl: Optimized workforce learning for general multi-agent assistance in real-wo...

    2025

  3. [11]

    OpenAI Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mkadry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alexander Kirillov, Alex Nichol, Alex P...

    2024 arXiv

  4. [13]

    Mmsearch: Benchmarking the potential of large models as multi-modal search engines

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, Yu Liu, Chunyuan Li, and Hongsheng Li. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. ArXiv, abs/2409.12959, 2024 b . URL...

    2024 arXiv

  5. [14]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  6. [15]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787--798, 2014

    2014

  7. [16]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024 a

    2024 arXiv

  8. [17]

    Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790, 2024 b

    2024 arXiv

  9. [19]

    Websailor: Navigating super-human reasoning for web agent

    Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592, 2025 b

    2025 arXiv

  10. [20]

    Graphreader: Building graph-based agent to enhance long-context abilities of large language models

    Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, et al. Graphreader: Building graph-based agent to enhance long-context abilities of large language models. arXiv preprint arXiv:2406.14550, 2024 c

    2024 arXiv

  11. [21]

    Search-o1: Agentic search-enhanced large reasoning models

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025 c

    2025 arXiv

  12. [22]

    Dream: Disentangling risks to enhance safety alignment in multimodal large language models

    Jianyu Liu, Hangyu Guo, Ranjie Duan, Xingyuan Bu, Yancheng He, Shilong Li, Hui Huang, Jiaheng Liu, Yucheng Wang, Chenchen Jing, et al. Dream: Disentangling risks to enhance safety alignment in multimodal large language models. arXiv preprint arXiv:2504.18053, 2025

    2025 arXiv

  13. [23]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer, 2024

    2024

  14. [24]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    2023 arXiv

  15. [25]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  16. [26]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209, 2021

    2021

  17. [27]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub \`e n Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697--1706, 2022

    2022

  18. [28]

    Llama 4 Herd

    Meta. Llama 4 Herd . Meta blog, 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  19. [29]

    Gaia: a benchmark for general ai assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023

    2023

  20. [30]

    Microsoft Copilot

    Microsoft. Microsoft Copilot . Microsoft blog, 2024. URL https://copilot.microsoft.com/

    2024

  21. [31]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  22. [32]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. OpenAI blog, 2024. URL https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  23. [33]

    OpenAI. Chatgpt. 2025 a . URL https://chatgpt.com/

    2025

  24. [34]

    Introducing deep research

    OpenAI. Introducing deep research. OpenAI blog, 2025 b . URL https://openai.com/index/introducing-o3-and-o4-mini/

    2025

  25. [35]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. OpenAI blog, 2025 c

    2025

  26. [36]

    Introducing gpt-4.1 in the api

    OpenAI. Introducing gpt-4.1 in the api. OpenAI blog, 2025 d . URL https://openai.com/index/gpt-4-1/

    2025

  27. [37]

    Training powerful llm agents with end-to-end reinforcement learning, 2025

    Jie Ouyang, Ruiran Yan, Yucong Luo, Mingyue Cheng, Qi Liu, Zirui Liu, Shuo Yu, and Daoyu Wang. Training powerful llm agents with end-to-end reinforcement learning, 2025. URL https://github.com/0russwest0/Agent-R1

    2025

  28. [38]

    Introducing perplexity deep research

    Perplexity.AI. Introducing perplexity deep research. Perplexity.AI blog, 2025. URL https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research

    2025

  29. [39]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  30. [40]

    Hal: Gaia leaderboard, 2025

    Princeton University SAgE Group. Hal: Gaia leaderboard, 2025. URL https://hal.cs.princeton.edu/gaia

    2025

  31. [41]

    Humanity's last exam leaderboard, 2025

    Scale AI . Humanity's last exam leaderboard, 2025. URL https://scale.com/leaderboard/humanitys_last_exam

    2025

  32. [42]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025

    2025 arXiv

  33. [43]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 arXiv

  34. [44]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37: 0 95095--95169, 2024

    2024

  35. [45]

    Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents

    Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents. arXiv preprint arXiv:2502.18017, 2025

    2025 arXiv

  36. [46]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. ArXiv, abs/2411.04368, 2024. URL https://api.semanticscholar.org/CorpusID:273877483

    2024 arXiv

  37. [47]

    Browsecomp: A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Is abella Fulford, Hyung Won Chung, Alexandre Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. ArXiv, abs/2504.12516, 2025. URL https://api.semantic...

    2025 arXiv

  38. [48]

    Webdancer: Towards autonomous information seeking agency

    Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Webdancer: Towards autonomous information seeking agency. 2025 a . URL https://api.semanticscholar.org/CorpusID:278959248

    2025

  39. [49]

    Mmsearch-r1: Incentivizing lmms to search

    Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. Mmsearch-r1: Incentivizing lmms to search. arXiv preprint arXiv:2506.20670, 2025 b

    2025 arXiv

  40. [50]

    Grok 3 beta — the age of reasoning agents

    x.AI. Grok 3 beta — the age of reasoning agents. x.AI blog, 2025. URL https://x.ai/news/grok-3

    2025

  41. [51]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  42. [52]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 0 20744--20757, 2022 a

    2022

  43. [53]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022 b . URL https://api.semanticscholar.org/CorpusID:252762395

    2022 arXiv

  44. [54]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  45. [55]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025

    2025 arXiv

  46. [56]

    Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese. ...

    2025 arXiv

  47. [57]

    deepflowio/deepflow

    Li Zhuofeng, JIN Jie, XIANG Yang, JIN XiaoFeng, YUAN Chao, LIU Chao, KANG Xiang, YE WeiQiang, Jiping Yin, Song Zhen, LIU Lvqiao, Lin Huanchao, Dexian YI, jianchang , LU Yao, kylewangchina , Zheng Ya, Jiawei , Nie RunJie, deepflow-lifei , duandaa , oldduckruirui , ZHANG Shu Xin...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.