Pith. sign in

REVIEW 7 cited by

ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.02052 v5 pith:QTKDZWKQ submitted 2024-10-02 cs.CL cs.CV

Xiao Yu , Baolin Peng , Vineeth Vajipey , Hao Cheng , Michel Galley , Jianfeng Gao , Zhou Yu This is my paper · ORCID
classification cs.CLcs.CV
keywords searchagentsr-mctsgpt-4olearningexploratoryexplorestate
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Autonomous agents have demonstrated significant potential in automating complex multistep decision-making tasks. However, even state-of-the-art vision-language models (VLMs), such as GPT-4o, still fall short of human-level performance, particularly in intricate web environments and long-horizon tasks. To address these limitations, we present ExACT, an approach to combine test-time search and self-learning to build o1-like models for agentic applications. We first introduce Reflective Monte Carlo Tree Search (R-MCTS), a novel test time algorithm designed to enhance AI agents' ability to explore decision space on the fly. R-MCTS extends traditional MCTS by 1) incorporating contrastive reflection, allowing agents to learn from past interactions and dynamically improve their search efficiency; and 2) using multi-agent debate for reliable state evaluation. Next, we introduce Exploratory Learning, a novel learning strategy to teach agents to search at inference time without relying on any external search algorithms. On the challenging VisualWebArena benchmark, our GPT-4o based R-MCTS agent achieves a 6% to 30% relative improvement across various tasks compared to the previous state-of-the-art. Additionally, we show that the knowledge and experience gained from test-time search can be effectively transferred back to GPT-4o via fine-tuning. After Exploratory Learning, GPT-4o 1) demonstrates the ability to explore the environment, evaluate a state, and backtrack to viable ones when it detects that the current state cannot lead to success, and 2) matches 87% of R-MCTS's performance while using significantly less compute. Notably, our work demonstrates the compute scaling properties in both training - data collection with R-MCTS - and testing time. These results suggest a promising research direction to enhance VLMs' capabilities for agentic applications via test-time search and self-learning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NeuroSymActive: Differentiable Neural-Symbolic Reasoning with Active Exploration for Knowledge Graph Question Answering

    cs.CL 2026-02 unverdicted novelty 6.0 of 10

    NeuroSymActive combines soft-unification symbolic modules, a neural path evaluator, and Monte-Carlo-style active exploration to reach strong answer accuracy on KGQA benchmarks while cutting graph lookups and model cal...

  2. WebGuard: Building a Generalizable Guardrail for Web Agents

    cs.AI 2025-07 conditional novelty 6.0 of 10

    WebGuard introduces an action-level risk dataset for web agents and shows that a fine-tuned 7B model improves risk-prediction accuracy from about 38% to 80% and high-risk recall from 20% to 76%, still below deployment...

  3. Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.

  4. LLM-First Search: Self-Guided Exploration of the Solution Space

    cs.AI 2025-06 conditional novelty 6.0 of 10

    LLM-First Search, where the model itself decides whether to continue or backtrack during reasoning, outperforms MCTS, BestFS, and ToT-BFS on harder Countdown and Sudoku tasks while using fewer tokens.

  5. AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery

    cs.CR 2025-05 conditional novelty 6.0 of 10

    Fake 'Close AD' ads make VLM web agents click them over 60% of the time, and near 100% in some settings.

  6. BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism

    cs.CL 2025-05 conditional novelty 6.0 of 10

    BacktrackAgent adds a trained error detector and a rewriter to GUI agents, improving task success on Mobile3M and Auto-UI benchmarks.

  7. Read Quietly, Think Aloud: Decoupling Comprehension and Reasoning in LLMs

    cs.CL 2025-07 conditional novelty 4.0 of 10

    Masking loss on early training tokens and injecting embeddings from an auxiliary reader improves LLM accuracy on several reasoning benchmarks.

Pith tools