Pith. sign in

REVIEW 4 major objections 6 minor 158 references

MageBench: Bridging Large Multimodal Models to Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MageBench shows current LMMs severely lack vision-in-the-chain reasoning: only a few models barely beat random acting, and all fall far below human level.

desk verdict A well-engineered benchmark with a real finding, but the headline claim about random baselines is overbroad and needs scoping before this is publication-ready. read the letter →

arxiv 2412.04531 v1 pith:I52GPJNE submitted 2024-12-05 cs.CV cs.AIcs.LG

Miaosen Zhang , Qi Dai , Yifan Yang , Jianmin Bao , Dongdong Chen , Kai Qiu , Chong Luo , Xin Geng
show 1 more author
Baining Guo
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords multimodalagentsvision-in-the-chainreasoningLMMevaluationbenchmarkvisualfeedbackSokobanplanningspatialwebUIreconstructionembodiedAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MageBench introduces a benchmark of 483 lightweight scenarios across three environments, WebUI page reconstruction, Sokoban puzzles, and a football game, designed to test whether large multimodal models can reason when visual feedback arrives continuously during decision-making, a pattern the authors call vision-in-the-chain (ViC). The paper claims that current LMMs, both open and closed, perform near or below random-acting baselines in the online settings, and that even the strongest product-level models stay far below human-level performance. The intended upshot is that today's multimodal models are not yet reliable agents for robotics or other tasks that require adapting plans to what the eyes see, because the missing capability is not language reasoning but visual-feedback-driven reasoning, visual imagination, and spatial planning.

What carries the argument

The load-bearing object is the vision-in-the-chain (ViC) reasoning paradigm, defined as decision-making in which image observations enter the reasoning chain at every step and plans must be revised from visual feedback. MageBench instantiates ViC through three lightweight but planning-heavy environments, WebUI (rebuild a webpage from a description and screenshots), Sokoban (push boxes in minimal steps), and Football (control the ball handler to score), and through two baseline agents: a Global planner that sees only the opening image and an Online planner that receives fresh frames and acts. The scoring machinery compares model performance against random and idle baselines and human reference performance, using a historical-best reward in Sokoban, a dense composite football reward, and a human-calibrated Atomic Element Similarity score for WebUI.

What would settle it

Check whether a policy that only moves in a circle or repeats a single action scores above random in Sokoban or Football; if such a trivial policy earns a high historical-best or dense reward, the metric does not isolate the claimed reasoning abilities. Alternatively, if a model trained purely on ViC-style image-action traces reaches human-level under the same unified prompt on all 483 scenarios, the claim that current models lack the underlying capability would be overturned.

Watch

Extended reading notes

Core claim

The paper's central claim is that current large multimodal models severely lack vision-in-the-chain reasoning: the ability to keep looking, update a plan, and act in a loop, as opposed to reasoning over a fixed image with text-only chain-of-thought. MageBench operationalises this with 483 tasks in three environments, and under a unified agent protocol the evaluation shows that in the online, feedback-driven setting only two product-level models beat the random-acting baseline, while all models remain far below the human reference. It further finds that models do not improve when given rendered visual feedback to revise code, do not benefit from longer action and observation memory, and score poorly on best-of-N in Sokoban, which the paper interprets as evidence of absent visual imagination and think-ahead ability.

Load-bearing premise

The conclusion rests on the assumption that the three hand-designed metrics (Sokoban historical-best reward, dense football reward, and web Atomic Element Similarity) actually measure planning, feedback integration, and imagination, rather than rewarding lucky early progress or instruction-following artifacts.

Editorial extensions

If this is right

  • If current models truly lack ViC reasoning, then gains on static image QA and text chain-of-thought benchmarks will not transfer to multimodal agent applications; agent capability must be measured with interactive visual feedback.
  • The WebUI results imply that relying on browser-rendered feedback for self-correction does not currently help models, so engineering agents that assume models can debug their own visual output will underdeliver.
  • The best-of-N results suggest football-style tasks may be amenable to reinforcement learning because sampling many trajectories yields strong improvements, while Sokoban-style spatial planning may require mechanism-level training changes rather than more sampling.
  • Models' failure to improve with longer action and observation memory indicates a bottleneck in interleaved image-text long-context processing, not just in planning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible extension not made in the paper is that ViC failures will show up in any human-in-the-loop or robotics setting where the model must ground each decision in a fresh camera view; the qualitative clean-table and book-stacking probes in the appendix point in this direction.
  • The benchmark's metric design suggests a testable refinement: if the football reward is meant to measure tactical value, then an independent human ranking of the same decision trajectories should correlate with the dense reward; a mismatch would indicate the reward rewards intermediate proxies rather than winning play.
  • The paper leaves open whether ViC is a data problem or an architecture problem; a natural follow-up is to train a model on interleaved image-action traces in these same environments and see whether the human gap closes without architectural change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MageBench, a multimodal agent benchmark with three lightweight environments (WebUI, Sokoban, Football) designed to evaluate the "vision-in-the-chain" (ViC) reasoning abilities of large multimodal models (LMMs). The authors define two agent settings, Global (planning from the initial observation) and Online (interactive planning with per-step visual feedback), and evaluate 14 models, alongside idle/random baselines and human performance. Their headline finding is that only a few product-level models outperform random action selection, and all are far below human level, which they interpret as strong evidence that current LMMs lack ViC-type reasoning, visual imagination, and interleaved image-text long-context handling. The paper also reports Best-of-N scaling curves, error statistics, memory ablations, and qualitative generalization studies to robotics and structured visual generation. Code and data are promised to be released.

Significance. If the conclusions are robust, MageBench would be a useful, lightweight, reasoning-oriented benchmark for evaluating LMMs as agents, filling a gap between VQA-style benchmarks and heavy agent environments. The ViC formulation is a helpful conceptual contribution, and the careful engineering—repeated runs, random/idle/human baselines, and human-preference-calibrated WebUI metric—are strengths. The release of code and data would lower the barrier for the community. However, the central claim depends critically on the design of the three evaluation metrics; the Sokoban and Football rewards are manually constructed and the random baseline margins are small, so the paper's headline conclusions need sensitivity checks before they can be taken at face value.

major comments (4)
  1. [Appendix A.2.3] The Sokoban evaluation uses the historical best cumulative reward (R = max_t sum R(tau) - R_best + 100) rather than a final-state or completion-based metric. Because early rewards are retained even if progress is later undone, a random policy that happens to push a box onto a target in the first few steps keeps that +4.5, while a planner that solves the level after many steps receives the same incremental credit. This likely inflates the random baseline (46.61) relative to a final-state metric, compressing the gap to GPT-4o (53.03) and Gemini (51.84). Since the claim in Sec. 5.1 that "only GPT-4o and Gemini performed better than the random baseline" in Sokoban-Online is load-bearing for the paper's central conclusion, please provide an ablation with an alternative metric (e.g., number of boxes on targets at trajectory end, or completion rate) to show the conclusion is not an artifact of the historical-max reward.
  2. [Sec. 3.3, Eq. (1); Table 1] The Football reward is a manually weighted sum of six sub-rewards with fixed coefficients, and the random baseline (17.33) is within the reported δ = ±2.2 of Gemini's score (18.33). Under that precision, Gemini cannot be said to outperform random in Football; only GPT-4o (21.20) is clearly above the baseline. Nevertheless, Sec. 5.1 states that "only GPT-4o and Gemini performed better than the random baseline" in Football. This overclaim should be corrected, and the authors should report proper statistical tests (e.g., paired bootstrap) for the Football comparison. Additionally, the sensitivity of the conclusion to the hand-chosen λ values should be assessed (e.g., by re-evaluating with a range of weights or a secondary win-rate metric).
  3. [Abstract and Sec. 5.1] The headline statement "only a few product-level models are better than random acting" is not true for the WebUI environment, where the random baseline is 0.00 and every evaluated model scores above 0. If the intended claim is restricted to Sokoban and Football, the abstract and Sec. 5.1 should say so explicitly; otherwise, this overgeneralization misrepresents the results and weakens the credibility of the central claim. Please revise the phrasing to match the actual data in Table 1.
  4. [Sec. 5.1 and Fig. 5] The conclusion that models "failed to boost the result with browser's rendering feedback" is based on comparing WebUI-Global (generate all code at once) with WebUI-Online (modify the Global output using rendered images). These settings differ in more than the presence of visual feedback: the Online task involves code modification and a more complex multi-turn interaction, which may independently affect performance. Although the HD-CoT-NoIMG condition in Fig. 5 provides a no-vision control, the main Table 1 comparison (Global vs. Online) is not fully controlled. Please clarify whether the Online scores are computed from the same initial code for all models, and discuss the potential confound between task structure and visual feedback.
minor comments (6)
  1. [Fig. 1] There is a spelling typo: "enviroments" should be "environments".
  2. [Sec. 1] The phrase "to assist humen" contains a typo ("humen" should be "humans").
  3. [Appendix E.1.4 and E.3.1] The word "javascipt" appears in prompt templates; it should be "javascript".
  4. [Sec. 6 vs. Abstract] The abstract states "We release our code and data," but Sec. 6 says "we will release the code and data as open-source in the near future." Please make these statements consistent, and indicate the exact release status.
  5. [Table 1 and Table 2] The abbreviation "IFE" is defined in the Table 1 caption, but Table 2 uses "IA" (invalid actions). Use consistent terminology (e.g., "IA" vs. "IFE") and define all abbreviations at first use.
  6. [Fig. 5] The gray-shaded areas are described as "regions of variance" but no definition is given. Specify whether these are standard deviations, confidence intervals, or min-max ranges.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MageBench's conclusions are empirical benchmark results, and its metrics are anchored to external human preference or prior work rather than to the models being evaluated.

full rationale

The paper's load-bearing claims do not reduce by construction to their inputs. MageBench is an empirical benchmark: the three environments and metrics are defined independently of the models under test. The Sokoban reward is taken from DeepMind's work (scaled 5x), with a historically-best cumulative reward minus the BFS-optimal trajectory reward; the random and idle baselines are computed from actual random policies, not fitted to model outputs. The Football dense reward uses fixed lambda values and game-state features, and the WebUI AES weights are tuned with PSO to human preference rankings, which is an external anchor; the reported 93% agreement with human preferences is a validation statistic, not a predicted model score. No parameter is fitted to model outputs and then reported as a prediction. The 'ViC' concept is defined by the paper, but the claim that current LMMs lack ViC reasoning is an empirical result from the benchmark comparisons, not a tautology. Self-citations (e.g., refs. [58] and [89]) appear only as background for VQA and in-context learning and are not load-bearing. The appendix's post-hoc justification for the historical-best Sokoban reward is a metric-validity concern (it may inflate the random baseline), but it is not a circular step: the metric still measures task progress and is applied uniformly to baselines, models, and humans. The paper's own stated limitations (small number of environments, standardized agent setup) limit generality but do not make the derivation circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on the assumptions that the three environments are representative of agent reasoning, that the hand-crafted metrics measure the intended capabilities, and that the small human sample is an adequate 'human-level' baseline. The paper provides no formal argument for these assumptions, only qualitative selection criteria and some validation experiments (e.g., 93% agreement for the WebUI metric).

free parameters (4)
  • WebUI AES attribute weights and exponent = Not disclosed
    The attribute weights and element-space exponent in the AES metric are tuned via PSO to match human preference rankings (Appendix A.1.3). These weights directly determine model scores and rankings, so they are fitted parameters rather than fixed, theoretically derived constants.
  • Football reward coefficients = lambda1=16, lambda2=20, lambda3=40, lambda4=20, lambda5=400, lambda6=100, T=400, beta=10, epsilon=1
    The six lambda weights, time horizon T, and the pass-reward constants (beta, epsilon) in Eq. (1) and Appendix A.3.3 are hand-chosen numbers that define the reward shaping. Different choices could change which models perform well, so these are free parameters of the evaluation.
  • Sokoban reward magnitudes and historical-best modification = +4.5, -5.5, +54.5, -0.5 (scaled 5x from DeepMind), plus subtract R_best and add 100
    The reward weights are scaled from a prior work, and the decision to use historical-best cumulative reward instead of final reward is justified post hoc to avoid output-length artifacts (Appendix A.2.3). This choice affects the metric and is a free modeling decision.
  • Auto-rendering distance threshold delta = Unspecified
    The auto-rendering algorithm for football (Appendix A.3.4) uses a preset threshold delta to decide when to return control to the player. The value is not given in the paper; it affects when the model is consulted and could influence the evaluation trajectory.
assumptions (4)
  • domain assumption The three selected environments (WebUI, Sokoban, Football) are representative of the core agent capabilities (knowledge/engineering, spatial planning, interaction).
    Invoked in Sec. 3 selection criteria, where the authors justify the choice based on qualitative reasoning about 'representativeness on reasoning' and 'visual feedback', without formal proof that these environments cover the intended capability space.
  • domain assumption The hand-designed metrics (AES, Sokoban reward, football reward) validly measure the intended capabilities (visual intelligence, imagination, feedback-driven planning).
    The paper provides some validation (e.g., 93% human agreement for WebUI, auto-rendering invariance for football), but the metrics are not proven to isolate the targeted cognitive skills from reward artifacts or task-specific priors.
  • domain assumption Human-level performance from a few PhD students is an adequate and representative baseline for 'human-level'.
    Sec. 5.1 states that human annotators were 'several PhD candidates with strong reasoning abilities'; the paper does not report the number of participants or inter-subject variability, yet uses this as the human-level reference.
  • domain assumption The auto-rendering algorithm does not materially alter the evaluation results.
    Appendix A.3.4 provides an experiment with an AI bot showing similar rewards with and without auto-rendering, but the threshold delta is unspecified and the algorithm changes the points at which the model is consulted, which could subtly affect model behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MageBench: Bridging Large Multimodal Models to Agents." pith.science (2026). https://pith.science/paper/I52GPJNE

@misc{pith2026241204531,
  author       = {Pith},
  title        = {Pith review of: MageBench: Bridging Large Multimodal Models to Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I52GPJNE}},
  note         = {Machine review of arXiv:2412.04531}
}
read the original abstract

LMMs have shown impressive visual understanding capabilities, with the potential to be applied in agents, which demand strong reasoning and planning abilities. Nevertheless, existing benchmarks mostly assess their reasoning abilities in language part, where the chain-of-thought is entirely composed of text.We consider the scenario where visual signals are continuously updated and required along the decision making process. Such vision-in-the-chain reasoning paradigm is more aligned with the needs of multimodal agents, while being rarely evaluated. In this paper, we introduce MageBench, a reasoning capability oriented multimodal agent benchmark that, while having light-weight environments, poses significant reasoning challenges and holds substantial practical value. This benchmark currently includes three types of environments: WebUI, Sokoban, and Football, comprising a total of 483 different scenarios. It thoroughly validates the agent's knowledge and engineering capabilities, visual intelligence, and interaction skills. The results show that only a few product-level models are better than random acting, and all of them are far inferior to human-level. More specifically, we found current models severely lack the ability to modify their planning based on visual feedback, as well as visual imagination, interleaved image-text long context handling, and other abilities. We hope that our work will provide optimization directions for LMM from the perspective of being an agent. We release our code and data at https://github.com/microsoft/MageBench.

Figures

Figures reproduced from arXiv: 2412.04531 by the authors.

Figure 1
Figure 1. Overview of the MageBench. MageBench is a multi-modality agent benchmark as well as a lightweight and fast platform for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The difference between vision-in-the-chain reasoning [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An overview of WebUI and its evaluation. LMM Agents [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (28 more)
Figure 4
Figure 4. Figure 4: The segment from the Germany vs. Mexico match in the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: WebUI-Online results. Different line styles repre [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: WebUI error construction. Each part of the pie graph is [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The complete evaluation pipeline of WebUI. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Details of different difficulties of the environments. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: An overview of initial scene setting in the football en [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Left: Diagram illustrating the calculation of Passing Ball Reward. Right: Diagram illustrating the optimal shooting angle and opponent distance. receives a ball), instead of start passing. We only consider successful passes, as unsuccessful passes have already been pe…
Figure 12
Figure 12. Figure 12: Left: Illustration of our auto-rendering algorithm. P1 and P2 represent the trajectories of two players on our team, while O1, O2, and O3 represent the trajectories of three opponent players. All moments shown in light color indicate that they are automatically render…
Figure 13
Figure 13. Figure 13: Effectiveness of auto-rendering technique. It reduce [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: The initial scene for Clean-up table task, which is care [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Task description for Book Hanoi tower task. [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: The Sokoban-Online, Football-Online result on MageBench-mini of the 3 models, and the last frame of their performance on [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Initial scene for Book Hanoi Tower task. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: GPT-4o on Clean-up table. # Analyze The current observation is too dark to see the books or regions clearly. # Action Put book in region A to region B # Analyze There are two books visible on the table, one in region B and one in region C. Region A is currently empty.…
Figure 19
Figure 19. Figure 19: GPT-4o on Book Hanoi Tower. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Task description for webpage generation (left and middle) and data visualization (right). [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Task description for image to SVG task. of 1.5. Similarly, LLaMA-3.2-90B and GPT-4o scored be￾tween 0.5 and 1 point. The last two models continued to perform poorly in this task. The aforementioned analysis indicates that our results can be robustly generalized to oth…
Figure 22
Figure 22. Figure 22: The cross-domain results demonstrate that our WebUI environment can robustly generalize to other engineering-level structured [PITH_FULL_IMAGE:figures/full_fig_p028_22.png]
Figure 23
Figure 23. Figure 23: A sample web page is provided where, in this example, clicking the small container button below will inject the corresponding [PITH_FULL_IMAGE:figures/full_fig_p030_23.png]
Figure 24
Figure 24. Figure 24: An example of preprocessed HTML file. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_24.png]
Figure 25
Figure 25. Figure 25: An example of task description document. [PITH_FULL_IMAGE:figures/full_fig_p031_25.png]
Figure 26
Figure 26. Figure 26: An example python file for interaction. Please adapt the code according to the current rendered screen shot of previous implementation, with this output format: # Analyze Your Analyze # Regenerate re-generated code If some of the files (e.g., javascript) do not need t…
Figure 27
Figure 27. Figure 27: GPT-4o’s output text(Upper image) and screen shot of GPT-4o’s implementation(Lower image). This implementation has a [PITH_FULL_IMAGE:figures/full_fig_p033_27.png]
Figure 28
Figure 28. Figure 28: LLaMA-3.2-Vision-Instruct’s output text(Upper image) and screen shot of LLaMA-3.2-Vision-Instruct’s implementation(Lower [PITH_FULL_IMAGE:figures/full_fig_p034_28.png]
Figure 29
Figure 29. Figure 29: An example of GPT-4o’s performances on Sokoban-Global settings, failed case (Left) and success case (Right). [PITH_FULL_IMAGE:figures/full_fig_p036_29.png]
Figure 30
Figure 30. Figure 30: An example of GPT-4o’s and Qwen2-VL-72B’s performances on Sokoban-Global settings. [PITH_FULL_IMAGE:figures/full_fig_p036_30.png]
Figure 31
Figure 31. Figure 31: An example of GPT-4o’s performances on Sokoban-Online settings with max-loop=20. [PITH_FULL_IMAGE:figures/full_fig_p037_31.png]
Figure 32
Figure 32. Figure 32: An example of GPT-4o’s performances on Football-Online settings. [PITH_FULL_IMAGE:figures/full_fig_p037_32.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

158 extracted references · 13 canonical work pages

  1. [1]

    Build with claude, 2024. 3, 6

    2024

  2. [2]

    Hello gpt-4o, 2024. 3, 6

    2024

  3. [3]

    Grok-2 beta release, 2024. 3

    2024

  4. [4]

    Introducing openai o1-preview, 2024. 2

    2024

  5. [5]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 6

    arXiv 2024

  6. [6]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2, 3

    arXiv 2023

  7. [7]

    Few- shot training llms for project-specific code- summarization

    Toufique Ahmed and Premkumar Devanbu. Few- shot training llms for project-specific code- summarization. In ASE, 2022. 2

    2022

  8. [8]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022. 2, 3

    2022

Show all 158 references
  1. [9]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al. Language models are few-shot learners. In NeurIPS, 2023. 2, 3

    2023

  2. [10]

    Chateval: Towards better llm-based evalu- ators through multi-agent debate

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evalu- ators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023. 3

    2023 arXiv

  3. [11]

    Q-ground: Image qual- ity grounding with large multi-modality models

    Chaofeng Chen, Sensen Yang, Haoning Wu, Liang Liao, Zicheng Zhang, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Q-ground: Image qual- ity grounding with large multi-modality models. In ACM MM, 2024. 2

    2024

  4. [12]

    Can vlms play action role-playing games? take black myth wukong as a study case

    Peng Chen, Pi Bu, Jun Song, Yuan Gao, and Bo Zheng. Can vlms play action role-playing games? take black myth wukong as a study case. arXiv preprint arXiv:2409.12889, 2024. 2, 3, 4

    2024 arXiv

  5. [13]

    Internvl: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, et al. Internvl: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks. In CVPR, 2024. 6

    2024

  6. [14]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob De- vlin, Maarten Bosma, Gaurav Mishra, et al. Palm: Scaling language modeling with pathways. JMLR,

  7. [15]

    Nvlm: Open frontier-class multimodal llms

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuol- ing Yang, Zihan Liu, Jon Barker, Tuomas Rinta- maki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402, 2024. 6

    2024 arXiv

  8. [16]

    Mobile- bench: An evaluation benchmark for llm-based mo- bile agents

    Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, et al. Mobile- bench: An evaluation benchmark for llm-based mo- bile agents. arXiv preprint arXiv:2407.00993, 2024. 4

    2024 arXiv

  9. [17]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. NeurIPS, 2024. 3

    2024

  10. [18]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, et al. A survey on in-context learning. In EMNLP, 2024. 3

    2024

  11. [19]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 2, 3, 6

    2024 arXiv

  12. [20]

    How far are we from agi

    Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. How far are we from agi. arXiv preprint arXiv:2405.10313, 2024. 3

    2024 arXiv

  13. [21]

    Blink: Multi- modal large language models can see but not per- ceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multi- modal large language models can see but not per- ceive. arXiv preprint arXiv:2404.12390, 2024. 3

    2024 arXiv

  14. [22]

    Multimodal web navigation with instruction-finetuned foundation models

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yu- taka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305.11854, 2023. 3

    2023 arXiv

  15. [23]

    Cantor: Inspiring multimodal chain-of-thought of mllm

    Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, et al. Cantor: Inspiring multimodal chain-of-thought of mllm. In ACM MM, 2024. 2, 3

    2024

  16. [24]

    Formalizing properties of agents

    Richard Goodwin. Formalizing properties of agents. JLC, 1995. 2

    1995

  17. [25]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 2, 3

    2017

  18. [26]

    Large language model based 9 multi-agents: A survey of progress and challenges

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based 9 multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024. 2

    2024 arXiv

  19. [27]

    Chatllm network: More brains, more intelligence

    Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence. arXiv preprint arXiv:2304.12998, 2023. 3

    2023 arXiv

  20. [28]

    Sapien: affective virtual agents pow- ered by large language models

    Masum Hasan, Cengiz Ozel, Sammy Potter, and Ehsan Hoque. Sapien: affective virtual agents pow- ered by large language models. In ACIIW, 2023. 3

    2023

  21. [29]

    Olympiadbench: A challenging bench- mark for promoting agi with olympiad-level bilin- gual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, et al. Olympiadbench: A challenging bench- mark for promoting agi with olympiad-level bilin- gual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024. 3

    2024 arXiv

  22. [30]

    Metagpt: Meta program- ming for multi-agent collaborative framework

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, et al. Metagpt: Meta program- ming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. 3

    2023 arXiv

  23. [31]

    Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning

    Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842, 2023. 2, 6

    2023 arXiv

  24. [32]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. 4

    2022 arXiv

  25. [33]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 2, 3

    2019

  26. [34]

    Tablevqa-bench: A visual question answering bench- mark on multiple table domains

    Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. Tablevqa-bench: A visual question answering bench- mark on multiple table domains. arXiv preprint arXiv:2404.19205, 2024. 3

    2024 arXiv

  27. [35]

    Large lan- guage models are zero-shot reasoners

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large lan- guage models are zero-shot reasoners. In NeurIPS,

  28. [36]

    Google re- search football: A novel reinforcement learning en- vironment

    Karol Kurach, Anton Raichuk, Piotr Sta ´nczyk, Michał Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Car- los Riquelme, Damien Vincent, et al. Google re- search football: A novel reinforcement learning en- vironment. In AAAI, 2020. 5

    2020

  29. [37]

    Learning the user’s deeper preferences for multi-modal recommendation systems

    Fei Lei, Zhongqi Cao, Yuning Yang, Yibo Ding, and Cong Zhang. Learning the user’s deeper preferences for multi-modal recommendation systems. TOMM,

  30. [38]

    Seed- bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench: Benchmarking multimodal large language models. In CVPR, 2024. 2, 3

    2024

  31. [39]

    Camel: Commu- nicative agents for" mind" exploration of large lan- guage model society

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Commu- nicative agents for" mind" exploration of large lan- guage model society. NeurIPS, 2023. 3

    2023

  32. [40]

    Mmro: Are multimodal llms eligible as the brain for in-home robotics? arXiv preprint arXiv:2406.19693, 2024

    Jinming Li, Yichen Zhu, Zhiyuan Xu, Jindong Gu, Minjie Zhu, Xin Liu, Ning Liu, Yaxin Peng, Feifei Feng, and Jian Tang. Mmro: Are multimodal llms eligible as the brain for in-home robotics? arXiv preprint arXiv:2406.19693, 2024. 3

    2024 arXiv

  33. [41]

    Ap- pagent v2: Advanced agent for flexible mobile inter- actions

    Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Ap- pagent v2: Advanced agent for flexible mobile inter- actions. arXiv preprint arXiv:2408.11824, 2024. 4

    2024

  34. [42]

    Mcu: A task-centric framework for open- ended agent evaluation in minecraft

    Haowei Lin, Zihao Wang, Jianzhu Ma, and Yitao Liang. Mcu: A task-centric framework for open- ended agent evaluation in minecraft. arXiv preprint arXiv:2310.08367, 2023. 4

    2023 arXiv

  35. [43]

    Visual spatial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. ACL, 2023. 3

    2023

  36. [44]

    Improved baselines with visual instruction tun- ing

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tun- ing. In CVPR, 2024. 6

    2024

  37. [45]

    Llava- next: Improved reasoning, ocr, and world knowl- edge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuan- han Zhang, Sheng Shen, and Yong Jae Lee. Llava- next: Improved reasoning, ocr, and world knowl- edge, 2024. 6

    2024

  38. [46]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS,

  39. [47]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xu- anyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023. 3

    2023 arXiv

  40. [48]

    Visualagentbench: Towards large multimodal models as visual foundation agents

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yi- fan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327, 2024. 3, 4

    2024 arXiv

  41. [49]

    Mmbench: Is your multi-modal model an all-around player? In ECCV, 2025

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2025. 2

    2025

  42. [50]

    Artificial empathy in marketing inter- actions: Bridging the human-ai gap in affective and social customer experience

    Yuping Liu-Thompkins, Shintaro Okazaki, and Hairong Li. Artificial empathy in marketing inter- actions: Bridging the human-ai gap in affective and social customer experience. JAMS, 2022. 3

    2022

  43. [51]

    Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525 ,

    arXiv

  44. [52]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022. 2, 3

    2022

  45. [53]

    Math- vista: Evaluating mathematical reasoning of foun- dation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Math- vista: Evaluating mathematical reasoning of foun- dation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. 3

    arXiv

  46. [54]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022. 3

    2022

  47. [55]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In CVPR,

  48. [56]

    Adaptive machine transla- tion with large language models

    Yasmin Moslem, Rejwanul Haque, John D Kelle- her, and Andy Way. Adaptive machine transla- tion with large language models. arXiv preprint arXiv:2301.13294, 2023. 2

    2023 arXiv

  49. [57]

    Mobile- flow: A multimodal llm for mobile gui agent

    Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. Mobile- flow: A multimodal llm for mobile gui agent. arXiv preprint arXiv:2407.04346, 2024. 4

    2024 arXiv

  50. [58]

    Learnable in-context vector for visual question answering

    Yingzhe Peng, Chenduo Hao, Xu Yang, Jiawei Peng, Xinting Hu, and Xin Geng. Learnable in-context vector for visual question answering. arXiv preprint arXiv:2406.13185, 2024. 2

    2024 arXiv

  51. [59]

    Sum- marization is (almost) dead

    Xiao Pu, Mingqi Gao, and Xiaojun Wan. Sum- marization is (almost) dead. arXiv preprint arXiv:2309.09558, 2023. 2

    2023 arXiv

  52. [60]

    Virtualhome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018. 4

    2018

  53. [61]

    Imagination-augmented agents for deep reinforcement learning

    Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, et al. Imagination-augmented agents for deep reinforcement learning. NeurIPS, 2017. 2, 5

    2017

  54. [62]

    Code generation with alphacodium: From prompt engineering to flow engineering

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500, 2024. 3

    2024 arXiv

  55. [63]

    A systematic survey of prompt engineering in large lan- guage models: Techniques and applications

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large lan- guage models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024. 3

    2024 arXiv

  56. [64]

    Image captioning for effective use of language models in knowledge-based visual question answering

    Ander Salaberria, Gorka Azkune, Oier Lopez de Lacalle, Aitor Soroa, and Eneko Agirre. Image captioning for effective use of language models in knowledge-based visual question answering. ESA,

  57. [65]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. NeurIPS, 2024. 3, 6

    2024

  58. [66]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Hao- tian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023. 3

    2023 arXiv

  59. [67]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M Dai, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2, 3, 6

    2023 arXiv

  60. [68]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  61. [69]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  62. [70]

    Tool-lmm: A large multi-modal model for tool agent learning

    Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Zhengxin Li, et al. Tool-lmm: A large multi-modal model for tool agent learning. arXiv preprint arXiv:2401.10727, 2024. 3

    2024 arXiv

  63. [71]

    V oyager: An open-ended embod- ied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar. V oyager: An open-ended embod- ied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. 3

    2023 arXiv

  64. [72]

    Qcap- tion: Video captioning and q&a through fusion of large multimodal models

    Jiale Wang, Gee Wah Ng, Lee Onn Mak, Randall Cher, Ng Ding Hei Ryan, and Davis Wang. Qcap- tion: Video captioning and q&a through fusion of large multimodal models. In FUSION, 2024. 2

    2024

  65. [73]

    Large language models for robotics: Opportuni- ties, challenges, and perspectives

    Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, et al. Large language models for robotics: Opportuni- ties, challenges, and perspectives. arXiv preprint arXiv:2401.04334, 2024. 2

    2024 arXiv

  66. [74]

    Mobile-agent: Autonomous multi-modal mobile de- vice agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile de- vice agent with visual perception. arXiv preprint arXiv:2401.16158, 2024. 2, 4 11

    2024 arXiv

  67. [75]

    Document-level machine translation with large lan- guage models

    Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. Document-level machine translation with large lan- guage models. arXiv preprint arXiv:2304.02210 ,

    arXiv

  68. [76]

    Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents

    Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents. arXiv preprint arXiv:2406.08184, 2024. 4

    2024 arXiv

  69. [77]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, et al. A survey on large language model based autonomous agents. FCS, 2024. 2

    2024

  70. [78]

    Qwen2-vl: Enhancing vision-language model’s per- ception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, et al. Qwen2-vl: Enhancing vision-language model’s per- ception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 6

    2024 arXiv

  71. [79]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 2, 3

    2022 arXiv

  72. [80]

    Chain-of-thought prompting elicits rea- soning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits rea- soning in large language models. NeurIPS, 2022. 2, 3

    2022

  73. [81]

    mchartqa: A univer- sal benchmark for multimodal chart question answer based on vision-language alignment and reasoning

    Jingxuan Wei, Nan Xu, Guiyong Chang, Yin Luo, BiHui Yu, and Ruifeng Guo. mchartqa: A univer- sal benchmark for multimodal chart question answer based on vision-language alignment and reasoning. arXiv preprint arXiv:2404.01548, 2024. 3

    2024 arXiv

  74. [82]

    Intel- ligent agents: Theory and practice

    Michael Wooldridge and Nicholas R Jennings. Intel- ligent agents: Theory and practice. KER, 1995. 2

    1995

  75. [83]

    A glance at in-context learning

    Yongliang Wu and Xu Yang. A glance at in-context learning. FCS, 2024. 3

    2024

  76. [84]

    Dettoolchain: A new prompting paradigm to unleash detection ability of mllm

    Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. Dettoolchain: A new prompting paradigm to unleash detection ability of mllm. arXiv preprint arXiv:2403.12488, 2024. 3

    2024 arXiv

  77. [85]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023. 2, 3

    2023 arXiv

  78. [86]

    Large multimodal agents: A survey

    Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large multimodal agents: A survey. arXiv preprint arXiv:2402.15116, 2024. 3

    2024 arXiv

  79. [87]

    A paradigm shift in machine translation: Boosting translation perfor- mance of large language models

    Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation perfor- mance of large language models. arXiv preprint arXiv:2309.11674, 2023. 2

    2023 arXiv

  80. [88]

    Auto-gpt for online decision making: Benchmarks and additional opinions

    Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224, 2023. 3

    2023 arXiv

  81. [89]

    Exploring diverse in-context configurations for image captioning

    Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. 2024. 2, 3

    2024

  82. [90]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. NeurIPS, 2022. 3, 4

    2022

  83. [91]

    React: Synergizing reasoning and acting in language mod- els

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language mod- els. arXiv preprint arXiv:2210.03629, 2022. 3, 6

    2022 arXiv

  84. [92]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 6

    2024 arXiv

  85. [93]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 2

    2023 arXiv

  86. [94]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. 6

    2024 arXiv

  87. [95]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 2

    2023 arXiv

  88. [96]

    Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, , et al. Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024. 3

    2024

  89. [97]

    Star: Self-taught reasoner bootstrapping reasoning with reasoning

    Eric Zelikman, YH Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In NeurIPS, 2024. 2

    2024

  90. [98]

    Transporter networks: Rearranging the visual world for robotic manipula- tion

    Andy Zeng, Pete Florence, Jonathan Tompson, Ste- fan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, et al. Transporter networks: Rearranging the visual world for robotic manipula- tion. In CoRL, 2021. 3, 4

    2021

  91. [99]

    Large language models for robotics: A survey

    Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226, 2023. 2

    2023

  92. [100]

    12 Prompting large language model for machine trans- lation: A case study

    Biao Zhang, Barry Haddow, and Alexandra Birch. 12 Prompting large language model for machine trans- lation: A case study. 2023. 2

    2023

  93. [101]

    Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs

    Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs. arXiv preprint arXiv:2401.02582, 2024. 2, 3

    2024 arXiv

  94. [102]

    Geoe- val: benchmark for evaluating llms and multi-modal models on geometry problem-solving.arXiv preprint arXiv:2402.10104, 2024

    Jiaxin Zhang, Zhongzhi Li, Mingliang Zhang, Fei Yin, Chenglin Liu, and Yashar Moshfeghi. Geoe- val: benchmark for evaluating llms and multi-modal models on geometry problem-solving.arXiv preprint arXiv:2402.10104, 2024. 3

    2024 arXiv

  95. [103]

    Internlm-xcomposer-2.5: A versa- tile large vision language model supporting long- contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, et al. Internlm-xcomposer-2.5: A versa- tile large vision language model supporting long- contextual input and output. arXiv preprint arXiv:2407.03320, 2024. 6

    2024 arXiv

  96. [104]

    Opt: Open pre- trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, et al. Opt: Open pre- trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 2

    2022 arXiv

  97. [105]

    Benchmarking large language models for news summarization

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. ACL, 2024. 2

    2024

  98. [106]

    Automatic chain of thought prompt- ing in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493, 2022. 2

    2022 arXiv

  99. [107]

    Multimodal chain- of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain- of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023. 2, 3

    2023 arXiv

  100. [108]

    Agent as cerebrum, controller as cerebellum: Implementing an embodied lmm-based agent on drones

    Haoran Zhao, Fengxing Pan, Huqiuyue Ping, and Yaoming Zhou. Agent as cerebrum, controller as cerebellum: Implementing an embodied lmm-based agent on drones. arXiv preprint arXiv:2311.15033 ,

    arXiv

  101. [109]

    Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in lan- guage models

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in lan- guage models. 2023. 2, 3

    2023

  102. [110]

    Language agent tree search unifies reasoning acting and planning in language models

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406,

    arXiv

  103. [111]

    Image-of-thought prompting for visual reasoning refinement in mul- timodal large language models

    Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang. Image-of-thought prompting for visual reasoning refinement in mul- timodal large language models. arXiv preprint arXiv:2405.13872, 2024. 2, 3

    2024 arXiv

  104. [112]

    Webarena: A realistic web environ- ment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, et al. Webarena: A realistic web environ- ment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. 2, 3, 4

    2023 arXiv

  105. [113]

    Visual in-context learning for large vision-language models

    Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. Visual in-context learning for large vision-language models. arXiv preprint arXiv:2402.11574, 2024. 3

    2024 arXiv

  106. [114]

    Ghost in the minecraft: Generally capable agents for open-world environments via large language mod- els with text-based knowledge and memory

    Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, et al. Ghost in the minecraft: Generally capable agents for open-world environments via large language mod- els with text-based knowledge and memory. arXiv preprint arXiv:2305.17144, 2023. 3, 4

    2023 arXiv

  107. [115]

    Multichartqa: Benchmark- ing vision-language models on multi-chart problems

    Zifeng Zhu, Mengzhao Jia, Zhihan Zhang, Lang Li, and Meng Jiang. Multichartqa: Benchmark- ing vision-language models on multi-chart problems. arXiv preprint arXiv:2410.14179, 2024. 3 13 A. Dataset details A.1. WebUI A.1.1. Web page collection and prepossessing We searched and ...

    2024 arXiv

  108. [116]

    The resources used in the website construction and the places where they are used (including image url, long texts, external CSS links, etc.). 4. Specify the class/id/- name of the elements that can be interacted with, This will be used for automated interaction testing. An ex...

  109. [117]

    It is not possible to push two boxes side by side at the same time from the side-by-side direction

  110. [118]

    The game can go into a lose-lose state, for example if you push the box into a corner where the corner is not the target area

    The red brick pattern is the wall. The game can go into a lose-lose state, for example if you push the box into a corner where the corner is not the target area. In this case, you should restart the game. A.2.3. Evaluation Our evaluation metrics are adapted from those proposed...

    2018

  111. [119]

    Belgium | 2018 FIFA World Cup; 69:32

    Brazil vs. Belgium | 2018 FIFA World Cup; 69:32

    2018

  112. [120]

    Mexico | 2018 FIFA World Cup; 82:11

    Germany vs. Mexico | 2018 FIFA World Cup; 82:11

    2018

  113. [121]

    Germany | 2014 FIFA World Cup; 12:19

    Brazil vs. Germany | 2014 FIFA World Cup; 12:19

    2014

  114. [122]

    Germany | 2014 FIFA World Cup; 26:55 R2

    Brazil vs. Germany | 2014 FIFA World Cup; 26:55 R2

    2014

  115. [123]

    Croatia | 2018 FIFA World Cup Final; 46:53

    France vs. Croatia | 2018 FIFA World Cup Final; 46:53

    2018

  116. [124]

    Japan | 2018 FIFA World Cup; 4:01

    Belgium vs. Japan | 2018 FIFA World Cup; 4:01

    2018

  117. [125]

    Mexico | 2018 FIFA World Cup; 53:08

    Germany vs. Mexico | 2018 FIFA World Cup; 53:08

    2018

  118. [126]

    Argentina | 2018 FIFA World Cup; 29:28 R3

    France vs. Argentina | 2018 FIFA World Cup; 29:28 R3

    2018

  119. [127]

    Belgium | 2018 FIFA World Cup; 14:13

    Brazil vs. Belgium | 2018 FIFA World Cup; 14:13

    2018

  120. [128]

    Belgium | 2018 FIFA World Cup; 24:36

    Brazil vs. Belgium | 2018 FIFA World Cup; 24:36

    2018

  121. [129]

    Mexico | 2018 FIFA World Cup; 13:11

    Germany vs. Mexico | 2018 FIFA World Cup; 13:11

    2018

  122. [130]

    Argentina | 2018 FIFA World Cup; 47:25 R4

    France vs. Argentina | 2018 FIFA World Cup; 47:25 R4

    2018

  123. [131]

    Mexico | 2018 FIFA World Cup; 62:58

    Germany vs. Mexico | 2018 FIFA World Cup; 62:58

    2018

  124. [132]

    Argentina | 2018 FIFA World Cup; 18:06

    France vs. Argentina | 2018 FIFA World Cup; 18:06

    2018

  125. [133]

    Spain | 2018 FIFA World Cup; 82:02

    Portugal vs. Spain | 2018 FIFA World Cup; 82:02

    2018

  126. [134]

    Germany | 2014 FIFA World Cup; 26:23 R5

    Brazil vs. Germany | 2014 FIFA World Cup; 26:23 R5

    2014

  127. [135]

    Belgium | 2018 FIFA World Cup; 6:22

    Brazil vs. Belgium | 2018 FIFA World Cup; 6:22

    2018

  128. [136]

    Argentina | 2018 FIFA World Cup; 42:08

    France vs. Argentina | 2018 FIFA World Cup; 42:08

    2018

  129. [137]

    Argentina | 2018 FIFA World Cup; 64:47

    France vs. Argentina | 2018 FIFA World Cup; 64:47

    2018

  130. [138]

    Spain | 2018 FIFA World Cup; 31:49 R6

    Portugal vs. Spain | 2018 FIFA World Cup; 31:49 R6

    2018

  131. [139]

    Japan | 2018 FIFA World Cup; 8:02

    Belgium vs. Japan | 2018 FIFA World Cup; 8:02

    2018

  132. [140]

    Spain | 2018 FIFA World Cup; 62:08

    Portugal vs. Spain | 2018 FIFA World Cup; 62:08

    2018

  133. [141]

    Germany | 2014 FIFA World Cup; 40:46

    Brazil vs. Germany | 2014 FIFA World Cup; 40:46

    2014

  134. [142]

    Brazil | 2010 FIFA World Cup; 16:55 R7

    Netherlands vs. Brazil | 2010 FIFA World Cup; 16:55 R7

    2010

  135. [143]

    Mexico | 2018 FIFA World Cup; 39:49

    Germany vs. Mexico | 2018 FIFA World Cup; 39:49

    2018

  136. [144]

    Argentina | 2018 FIFA World Cup; 00:52

    France vs. Argentina | 2018 FIFA World Cup; 00:52

    2018

  137. [145]

    Spain | 2018 FIFA World Cup; 21:25

    Portugal vs. Spain | 2018 FIFA World Cup; 21:25

    2018

  138. [146]

    Spain | 2018 FIFA World Cup; 64:22 R8

    Portugal vs. Spain | 2018 FIFA World Cup; 64:22 R8

    2018

  139. [147]

    Croatia | 2018 FIFA World Cup Final; 0:18

    France vs. Croatia | 2018 FIFA World Cup Final; 0:18

    2018

  140. [148]

    Belgium | 2018 FIFA World Cup; 85:34

    Brazil vs. Belgium | 2018 FIFA World Cup; 85:34

    2018

  141. [149]

    Argentina | 2018 FIFA World Cup; 61:53

    France vs. Argentina | 2018 FIFA World Cup; 61:53

    2018

  142. [150]

    Spain | 2018 FIFA World Cup; 18:43 R9

    Portugal vs. Spain | 2018 FIFA World Cup; 18:43 R9

    2018

  143. [151]

    Japan | 2018 FIFA World Cup; 20:53

    Belgium vs. Japan | 2018 FIFA World Cup; 20:53

    2018

  144. [152]

    Mexico | 2018 FIFA World Cup; 24:29

    Germany vs. Mexico | 2018 FIFA World Cup; 24:29

    2018

  145. [153]

    Spain | 2018 FIFA World Cup; 26:31

    Portugal vs. Spain | 2018 FIFA World Cup; 26:31

    2018

  146. [154]

    Germany | 2014 FIFA World Cup; 01:51 A.3.2

    Brazil vs. Germany | 2014 FIFA World Cup; 01:51 A.3.2. System prompt psys The system prompt of football environment is shown as fol- low: You’re going to play a football game. The character you control wears a yellow jersey with a blue name above his head. Your goal is to atta...

    2014

  147. [155]

    Black Myth: Wukong,

    In Figure 12, the automatic rendering corresponds to the periods from T = 1to T = 6and from T = 14to T = 18. It is worth noting that within the maximum 10 frames of automatic rendering, the player is defaulted to move in a certain direction, which might prevent the player from...

  148. [156]

    Speech Coding and Synthesis

    from the three aforementioned generation tasks. Overall, the qualitative analysis results are highly correlated with the WebUI scores. We elaborate on this correlation in the fol- lowing. Firstly, regarding web page generation, the Claude model not only perfectly meets the req...

  149. [157]

    # Regenerate re-generate the files (html, css or javascript) that need to be changed

    which parts of the previous code need to change and why. # Regenerate re-generate the files (html, css or javascript) that need to be changed. If some of the files (e.g., javascript) do not need to be changed, just write ’*javascript do not need to change *’ and do not generat...

  150. [158]

    en"> <head> <meta charset=

    If not, which parts of the previous code need to change and why. # Regenerate re-generate the files (html, css or javascript) that need to be changed. If some of the files (e.g., javascript) do not need to be changed, just write ’*javascript do not need to change *’ and do not...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.