REVIEW 7 cited by
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent research on Large Language Models (LLMs) has led to remarkable advancements in general NLP AI assistants. Some studies have further explored the use of LLMs for planning and invoking models or APIs to address more general multi-modal user queries. Despite this progress, complex visual-based tasks still remain challenging due to the diverse nature of visual tasks. This diversity is reflected in two aspects: 1) Reasoning paths. For many real-life applications, it is hard to accurately decompose a query simply by examining the query itself. Planning based on the specific visual content and the results of each step is usually required. 2) Flexible inputs and intermediate results. Input forms could be flexible for in-the-wild cases, and involves not only a single image or video but a mixture of videos and images, e.g., a user-view image with some reference videos. Besides, a complex reasoning process will also generate diverse multimodal intermediate results, e.g., video narrations, segmented video clips, etc. To address such general cases, we propose a multi-modal AI assistant, AssistGPT, with an interleaved code and language reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate LLMs with various tools. Specifically, the Planner is capable of using natural language to plan which tool in Executor should do next based on the current reasoning progress. Inspector is an efficient memory manager to assist the Planner to feed proper visual information into a specific tool. Finally, since the entire reasoning process is complex and flexible, a Learner is designed to enable the model to autonomously explore and discover the optimal solution. We conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving state-of-the-art results. Moreover, showcases demonstrate the ability of our system to handle questions far more complex than those found in the benchmarks.
Forward citations
Cited by 7 Pith papers
-
Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents
Lucid shows that imperceptible image perturbations can make multimodal agents misremember past events with 61.6% poisoning and 58.4% injection success.
-
ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory
A robotic agent operating system with source-grounded graph memory and split-wise self-evolution improves long-horizon embodied task success and memory QA scores over baseline controllers.
-
Personalize Your Large Vision-language Models With In-context Prompt Tuning
ICPT converts a few reference images of a personalized concept into an adaptive-length visual prompt plus a label embedding, letting a frozen LVLM add and reason about multiple concepts on the fly.
-
MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
A new one-million-sample multimodal agent tuning dataset with GPT-4o-generated rationales, reflection, and tool/RAG calls is shown to improve fine-tuned models, though training/eval benchmark overlap is not addressed.
-
A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.
-
Efficiently Enhancing General Agents With Hierarchical-categorical Memory
EHC, a memory-augmented tool-use agent that retrieves category-specific past experiences, outperforms the CLOVA baseline on GQA, NLVR2, MagicBrush editing, and image tagging without updating model parameters.
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
Discussion (0). Sign in to comment.