Pith. sign in

REVIEW 4 major objections 7 minor 47 references

Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Agent-RewardBench tests whether multimodal models can judge each step an agent takes, and finds that even the strongest model is only about 62% accurate.

desk verdict A genuinely new step-level agent reward benchmark with a solid construction pipeline, but the downstream-correlation claim rests on five data points and an unspecified scoring conversion; conditionally worth refereeing. read the letter →

arxiv 2506.21252 v1 pith:E7EG2ABU submitted 2025-06-26 cs.CL cs.AI

Tianyi Men , Zhuoran Jin , Pengfei Cao , Yubo Chen , Kang Liu , Jun Zhao This is my paper
classification cs.CLcs.AI
keywords Agent-RewardBenchrewardmodelingmultimodalagentsstep-levelevaluationperceptionplanningsafetyMLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to measure how well multimodal large language models can act as reward models for agents: given an image, a task step, and two candidate actions, can the model tell which one is better? It introduces Agent-RewardBench, 1,136 pairwise, step-level comparisons spanning perception, planning, and safety across seven real-world agent scenarios, with responses sampled from ten different multimodal models. The central finding is that current MLLMs are weak reward judges: the best black-box model, gemini-1.5-pro, reaches only 61.6% accuracy overall, and safety judgment is much worse than perception or planning. The paper also reports that benchmark accuracy strongly predicts success on VisualWebArena under reward-guided A* search (Pearson r = 0.981, p = 0.003), so there is a concrete reason to care: choosing a reward model by this benchmark should improve downstream agent search.

What carries the argument

The load-bearing object is the pairwise comparison protocol: each sample pairs a chosen step-level response with a rejected one, and the reward model must pick the better answer from an image, a question, and two candidates. The benchmark construction pipeline samples responses from ten multimodal models, pairs positive and negative step outcomes, filters for difficulty by running three small models with order-swapped positions (Pixtral-12B, llava-onevision-qwen2-7b, InternVL2-8B), and then manually verifies the remaining pairs. Each pair is evaluated twice with order swapped to remove positional bias, and accuracy is averaged across perception, planning, and safety dimensions.

What would settle it

Fresh expert annotators would re-label a random sample of, say, 200 Agent-RewardBench pairs using the original instructions, and the new labels would be compared against the benchmark's chosen/rejected assignments. If agreement is low (for example, below 90%) or if recomputing model accuracies and the VisualWebArena correlation with the relabeled subset changes the reported 0.981 correlation materially, the central claim that the benchmark measures reward modeling ability would be falsified.

Watch

Extended reading notes

Core claim

The paper claims that Agent-RewardBench is the first benchmark for evaluating reward modeling in multi-step agent tasks, measuring a model's ability to score individual steps across perception, planning, and safety. On its 1,136 manually filtered pairwise comparisons, even the strongest black-box MLLM reaches only 61.6% overall accuracy, safety accuracy is strikingly low (GPT-4o 39.2%, Claude-3.5-Sonnet 22.4%), and open-source models near 7B parameters are close to chance in several dimensions. The benchmark's headline validity result is that model accuracy on Agent-RewardBench correlates with VisualWebArena success under A* search (Pearson r = 0.981, p = 0.003), supporting the claim that accurate reward modeling directly improves agent search performance.

Load-bearing premise

The benchmark's validity rests on the 1,136 gold labels being correct, especially the positive labels, which are inherited from source datasets and only manually filtered when a rejected answer is clearly better; no inter-annotator agreement is reported.

Editorial extensions

If this is right

  • Reward-model selection for agent systems can be driven by a single benchmark score: accuracy on Agent-RewardBench tracks VisualWebArena success under A* search with r = 0.981.
  • Even frontier models are far from reliable safety judges (GPT-4o 39.2%, Claude-3.5-Sonnet 22.4%), so safety-specific reward training is a concrete gap.
  • Open-source models around 7-11B parameters are near chance in several dimensions (Llama-3.2-11B: 53.5% perception, 50.6% planning), suggesting specialized agent-reward training could move them substantially.
  • Step-level evaluation lets developers identify which type of step—perception, planning, or safety—a reward model fails on, rather than judging only final task success.
  • The small-model difficulty filter generalizes to larger models, so the benchmark's discrimination should persist across model scales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported downstream correlation is computed on one environment (VisualWebArena) with 100 sampled points and one policy model; treating Agent-RewardBench as a universal selection signal would require similar checks on other agent environments and policies.
  • Because positive labels are inherited from source datasets, a wrong source annotation could silently penalize a reward model that judges correctly; an independent re-annotation study is a natural next step.
  • If training reward models directly on Agent-RewardBench pairs improves VisualWebArena success, it would close the loop between benchmark and training; that is testable.
  • The low safety scores suggest that reward-guided search could endorse unsafe actions; deployment tests should add adversarial safety probes beyond pop-up attacks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes Agent-RewardBench, a benchmark of 1,136 pairwise preference instances for evaluating multimodal LLMs as step-level reward models in agent tasks. The benchmark covers perception, planning, and safety across seven scenarios (web, mobile, desktop, driving, Minecraft, virtual house, travel planning), constructed by sampling responses from multiple models, filtering by difficulty using three small MLLMs, and manual review. Experiments on 12 MLLMs report that even the best black-box model, gemini-1.5-pro, achieves only 61.6% accuracy. The paper further reports a Pearson correlation of 0.981 between benchmark accuracy and VisualWebArena success under A* search, claiming that the benchmark is relevant to downstream agent performance.

Significance. If the benchmark's validity and downstream correlation hold, it addresses a real gap: there is no standard way to choose reward models for multimodal agents, and step-level reward evaluation across perception, planning, and safety is a useful contribution. The resource is timely, covers scenarios missing from prior reward benchmarks, and the authors release code. The main strengths are the breadth of scenarios, the step-level pairwise design, and the position-swapped evaluation to mitigate order bias. However, the central downstream-relevance claim is currently supported by a fragile five-point correlation, and the difficulty-filtering step introduces a circularity with the evaluated model set. These issues must be resolved before the benchmark's main claims can be accepted.

major comments (4)
  1. [Section 5.1 / Figure 5] The claim that Agent-RewardBench 'demonstrates strong relevance to downstream tasks' rests on a single Pearson correlation (r=0.981, p=0.003) computed from five reward models, one policy model (GPT-4o-2024-08-06), and a 100-task sample of VisualWebArena. With n=5, the coefficient is highly sensitive to individual points; the paper reports no leave-one-out analysis, no confidence interval, and no repeated-search variance. Moreover, the text does not specify how the pairwise preference judgment used in Section 4.1 is converted to the scalar node scores (v) in the A* search illustrated in Figure 4. If the conversion (e.g., a different prompt or a normalization scheme) is not exactly the same protocol as the benchmark evaluation, the reported correlation may reflect the conversion procedure rather than reward-model quality. I request (i) a precise statement of the pairwise-to-scalar mapping, (ii) leave-one-out and bootstrap robustness checks, and (iii) a scatter plot with model labels so readers can assess leverage.
  2. [Section 3.2 / Table 2] The difficulty-control step in Section 3.2 uses Pixtral-12B-2409, llava-onevision-qwen2-7b-ov-hf, and InternVL2-8B to select pairs whose accuracy is in the 'middle or difficult' range, and the same three models then appear in the evaluated models in Table 2. This creates a circular dependency: the benchmark's item selection is partially optimized with respect to these models, so their scores (and, in principle, the difficulty calibration) are not independent measurements. The paper should either exclude the filtering models from the main results table, or report the main conclusions with and without them to show that the ranking of other models (e.g., gemini-1.5-pro at 61.6%) is unaffected.
  3. [Section 3.2 / Limitations] The Limitations section states that 'the positive labels remain faithful to the annotations in the data source,' and Section 3.2 describes manual verification as removing cases where the rejected answer is correct and the chosen answer is wrong. This means the gold labels themselves are never independently verified, and no inter-annotator agreement is reported for the manual screening. If any source annotation (e.g., a bounding box, a next action, or a safety judgment) is incorrect, the benchmark will penalize a reward model for correctly identifying the true best action. Please report the number of pairs removed at each filtering stage, the number of annotators who reviewed each pair, and agreement statistics (e.g., Cohen's kappa or Fleiss' kappa) on a double-annotated subset.
  4. [Section 3.2 / Section 5.2] The difficulty-selection criterion is stated inconsistently: Section 3.2 says pairs are selected if their accuracy lies in the 'middle or difficult' range among 10 candidates, while Section 5.2 says 'we primarily select medium-difficulty samples.' These are different sets if 'difficult' means high accuracy rather than medium. Please specify the exact threshold or percentile rule, and clarify whether both medium and hard pairs are included in the final 1,136 samples.
minor comments (7)
  1. [Abstract] The sentence 'there is no clear on how to select reward models for agents' is grammatically incomplete; consider 'there is no established methodology for selecting reward models for agents.'
  2. [Section 5.1] The phrase 'And no search methods only have 34% acc.' is a sentence fragment and unclear; please rewrite to state the no-search baseline accuracy and how it compares to the A* results.
  3. [Section 5.1 / Figure 5] The caption 'It shows high correlation' is uninformative; specify what is plotted, name the five models, and add the no-search baseline to the figure.
  4. [Table 3 / Table 2] Table 3 lists Llama-3.2-90B-Vision-Instruct among evaluated models, but Table 2 reports no results for this model; clarify whether it was evaluated and, if not, remove it from Table 3.
  5. [General] The paper repeatedly states that responses are sampled from 10 models, while Table 2 reports 12 models and Table 3 lists 13; align these counts and explain the discrepancy.
  6. [Section 3.2] The pairing procedure says 'we sample 10 pairs' per query but does not specify whether the same positive response is reused across pairs or sampled independently; please specify to aid reproducibility.
  7. [Section 3.1] The construction of the 100 web-safety pop-up samples is described only by reference to Zhang et al. (2024b); please include the exact prompt or attack template and the selection criteria.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and its downstream correlation are grounded in external data and independent evaluation.

full rationale

Agent-RewardBench is a benchmark-construction paper whose central claims are that current MLLMs are weak agent reward models and that benchmark accuracy correlates with downstream VisualWebArena A* search performance. Neither claim reduces to the benchmark's inputs by construction. The gold labels are inherited from external datasets with a manual verification pass, so the evaluation is grounded outside the paper. The downstream correlation is an external validation using VisualWebArena, a separate benchmark, and is therefore not self-referential; the small sample size (n=5) and the unspecified pairwise-to-scalar conversion are statistical and protocol concerns, not circularity. The difficulty filtering in Section 3.2 uses Pixtral-12B, llava-onevision-qwen2-7b, and InternVL2-8B to select medium/hard pairs, and Table 2 reports scores for these same models; this creates a calibration artifact for those three rows, but the paper's difficulty-generalization analysis in Section 5.2 explicitly excludes the filtering models, and the headline challenge claims rest on non-filter models such as gemini-1.5-pro, GPT-4o, and Claude 3.5 Sonnet. Self-citations such as Jin et al. (2024) and Men et al. (2024) appear only as related-work context and do not carry the load of the benchmark's validity. Overall, the paper's argument is self-contained against external benchmarks, and I find no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No fitted numeric model parameters appear in the paper. The load-bearing design choices are curation heuristics: the difficulty target, the per-category budgets, and the pairing count. There are no invented physical or mathematical entities; the benchmark is an evaluation artifact, not a postulated entity.

free parameters (3)
  • difficulty selection rule = pairs with small-model accuracy in the middle or difficult range among 10 candidates per query
    Chosen by hand in Section 3.2; the medium-difficulty target defines the dataset and therefore affects all model rankings in Table 2.
  • per-category sample budget = ranges from 82 (embody safety) to 259 (web planning), total 1,136
    Hand-selected counts in Section 3.1; category weights affect the dimension averages and totals reported in Table 2.
  • pairs per query = 10
    The number of candidate pairs sampled per query in Section 3.2 is a design choice that bounds the evaluation pool.
assumptions (4)
  • domain assumption Responses from 10 sampled MLLMs are representative of the real-world distribution of agent step responses.
    Section 3.2 Response Generation builds Sr by sampling 5 black-box and 5 white-box models; this assumes those 10 models cover the relevant response space.
  • domain assumption The three filtering small models' accuracy ranks difficulty in a way that generalizes to other models.
    Section 3.2 Difficulty Control and Section 5.2: the benchmark keeps pairs that Pixtral-12B, LLaVA-OneVision-7B, and InternVL2-8B find medium-difficult, an assumption tested only on averaged held-out models.
  • domain assumption Source dataset annotations are correct labels for the chosen responses.
    Limitations section: 'the positive labels remain faithful to the annotations in the data source.'
  • domain assumption Pairwise comparison accuracy is a valid proxy for a reward model's utility in guided search.
    Section 5.1 assumes benchmark accuracy predicts A* search success with GPT-4o as policy model; this is the correlation claim, not established by construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents." pith.science (2026). https://pith.science/paper/E7EG2ABU

@misc{pith2026250621252,
  author       = {Pith},
  title        = {Pith review of: Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7EG2ABU}},
  note         = {Machine review of arXiv:2506.21252}
}
read the original abstract

As Multimodal Large Language Models (MLLMs) advance, multimodal agents show promise in real-world tasks like web navigation and embodied intelligence. However, due to limitations in a lack of external feedback, these agents struggle with self-correction and generalization. A promising approach is to use reward models as external feedback, but there is no clear on how to select reward models for agents. Thus, there is an urgent need to build a reward bench targeted at agents. To address these challenges, we propose Agent-RewardBench, a benchmark designed to evaluate reward modeling ability in MLLMs. The benchmark is characterized by three key features: (1) Multiple dimensions and real-world agent scenarios evaluation. It covers perception, planning, and safety with 7 scenarios; (2) Step-level reward evaluation. It allows for the assessment of agent capabilities at the individual steps of a task, providing a more granular view of performance during the planning process; and (3) Appropriately difficulty and high-quality. We carefully sample from 10 diverse models, difficulty control to maintain task challenges, and manual verification to ensure the integrity of the data. Experiments demonstrate that even state-of-the-art multimodal models show limited performance, highlighting the need for specialized training in agent reward modeling. Code is available at github.

Figures

Figures reproduced from arXiv: 2506.21252 by the authors.

Figure 1
Figure 1. An illustration of SFT, reward guided training and reward guided searching methods. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of Agent-RewardBench. It evaluates the reward ability of perception, planning, and safety. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Response distribution in Agent-RewardBench. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: An illustration of A*. The A* search algo [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Correlation between Agent-RewardBench and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Error types in negative samples. Specifically, we annotate 50 data samples and identify six primary error categories: "Understand￾ing error" refers to the model’s deviation in under￾standing the current task. "Missing steps" means that the model has omitted some steps …
Figure 6
Figure 6. Figure 6: Difficulty control. The accuracy of other models on these difficulty levels is then averaged to assess dataset perfor￾mance, excluding the models used for filtering. The results, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Manual Verification Platform. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 7 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. 2024. Pixtral 12b. arXiv preprint arXiv:2410.07073

    arXiv 2024

  4. [4]

    Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Yaohang Li, Xing Luo, Chenyu Yi, and Alex Kot. 2024. Benchlmm: Benchmarking cross-style visual capability of large multimodal models. In European Conference on Computer Vision, pages 340--358. Springer

    work page 2024

  5. [5]

    Pengfei Cao, Tianyi Men, Wencan Liu, Jingwen Zhang, Xuzhao Li, Xixun Lin, Dianbo Sui, Yanan Cao, Kang Liu, and Jun Zhao. 2025. Large language models for planning: A comprehensive and systematic survey. arXiv preprint arXiv:2505.19683

    arXiv 2025

  6. [6]

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023 a . Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915

    arXiv 2023

  7. [7]

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. 2023 b . Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4-vision and beyond. arXiv preprint arXiv:2310.02071

    arXiv 2023

  8. [8]

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. 2024 a . Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain. arXiv preprint arXiv:2402.15527

    arXiv 2024

Show all 47 references
  1. [9]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024 b . How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101

    2024

  2. [10]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

    2024 arXiv

  3. [11]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024 a . Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

    2024

  4. [12]

    Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. 2024 b . Ai agents under threat: A survey of key security challenges and future pathways. ACM Computing Surveys

    2024

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148--166. Springer

    2024

  7. [15]

    Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, and Yu Su. 2024. Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559

    2024 arXiv

  8. [16]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  9. [17]

    Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Rag-rewardbench: Benchmarking reward models in retrieval augmented generation for preference alignment. arXiv preprint arXiv:2412.13746

    2024 arXiv

  10. [18]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024 a . Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649

    2024 arXiv

  11. [19]

    Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. 2024 b . Tree search for language model agents. arXiv preprint arXiv:2407.01476

    2024

  12. [20]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  13. [21]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  14. [22]

    Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. 2024 b . Vlrewardbench: A challenging benchmark for vision-language generative reward models. arXiv preprint arXiv:2411.17451

    2024 arXiv

  15. [23]

    Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. 2025. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.02584

    2025 arXiv

  16. [24]

    Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Unlocking the future: Exploring look-ahead planning mechanistic interpretability in large language models. arXiv preprint arXiv:2406.16033

    2024 arXiv

  17. [25]

    Lu Qiu, Yuying Ge, Yi Chen, Yixiao Ge, Ying Shan, and Xihui Liu. 2024. Egoplan-bench2: A benchmark for multimodal large language model planning in real-world scenarios. arXiv preprint arXiv:2412.04447

    2024 arXiv

  18. [26]

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. 2024. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573

    2024 arXiv

  19. [27]

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502

    2024 arXiv

  20. [28]

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. 2024. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723

    2024 arXiv

  21. [29]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  22. [30]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  23. [31]

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560

    2023 arXiv

  24. [32]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  25. [33]

    Qiucheng Wu, Handong Zhao, Michael Saxon, Trung Bui, William Yang Wang, Yang Zhang, and Shiyu Chang. 2024. Vsp: Assessing the dual challenges of perception and reasoning in spatial planning tasks for vlms. arXiv preprint arXiv:2407.01863

    2024 arXiv

  26. [34]

    Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024 a . Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622

    2024 arXiv

  27. [35]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. 2024 b . Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972

    2024 arXiv

  28. [36]

    Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2024. Agent lumos: Unified and modular training for open-source language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  29. [37]

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2020

  30. [38]

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823

    2023 arXiv

  31. [39]

    Yuanzhao Zhai, Tingkai Yang, Kele Xu, Feng Dawei, Cheng Yang, Bo Ding, and Huaimin Wang. 2024. Enhancing decision-making for llm agents via step-level q-value models. arXiv preprint arXiv:2409.09345

    2024 arXiv

  32. [40]

    Min Zhang, Xian Fu, Jianye Hao, Peilong Han, Hao Zhang, Lei Shi, Hongyao Tang, and Yan Zheng. 2024 a . Mfe-etp: A comprehensive evaluation benchmark for multi-modal foundation models on embodied task planning. arXiv preprint arXiv:2407.05047

    2024 arXiv

  33. [41]

    Yanzhe Zhang, Tao Yu, and Diyi Yang. 2024 b . Attacking vision-language computer agents via pop-ups. arXiv preprint arXiv:2411.02391

    2024 arXiv

  34. [42]

    Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. 2024 c . Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. arXiv preprint arXiv:2408.15978

    2024 arXiv

  35. [43]

    Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, et al. 2024 a . Rmb: Comprehensively benchmarking reward models in llm alignment. arXiv preprint arXiv:2410.09893

    2024 arXiv

  36. [44]

    Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Anderson Compalas, Dawn Song, and Xin Eric Wang. 2024 b . Multimodal situational safety. arXiv preprint arXiv:2410.06172

    2024 arXiv

  37. [45]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

    2023 arXiv

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.