REVIEW 8 cited by
Go-Explore: a New Approach for Hard-Exploration Problems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, which is the dominant method to improve performance on hard-exploration domains. To address this shortfall, we introduce a new algorithm called Go-Explore. It exploits the following principles: (1) remember previously visited states, (2) first return to a promising state (without exploration), then explore from it, and (3) solve simulated environments through any available means (including by introducing determinism), then robustify via imitation learning. The combined effect of these principles is a dramatic performance improvement on hard-exploration problems. On Montezuma's Revenge, Go-Explore scores a mean of over 43k points, almost 4 times the previous state of the art. Go-Explore can also harness human-provided domain knowledge and, when augmented with it, scores a mean of over 650k points on Montezuma's Revenge. Its max performance of nearly 18 million surpasses the human world record, meeting even the strictest definition of "superhuman" performance. On Pitfall, Go-Explore with domain knowledge is the first algorithm to score above zero. Its mean score of almost 60k points exceeds expert human performance. Because Go-Explore produces high-performing demonstrations automatically and cheaply, it also outperforms imitation learning work where humans provide solution demonstrations. Go-Explore opens up many new research directions into improving it and weaving its insights into current RL algorithms. It may also enable progress on previously unsolvable hard-exploration problems in many domains, especially those that harness a simulator during training (e.g. robotics).
Forward citations
Cited by 8 Pith papers
-
TopoExplore: Topological Discrimination for Archive-Based Exploration
Topology-aware cell selection on the visited-set occupancy grid yields 1.52× median entry speedup over Go-Explore by bonus-ing only strict entrances of live voids and ignoring sealed regions.
-
Behavioral Exploration: Learning to Explore via In-Context Adaptation
A coverage-conditioned behavioral cloning policy adapts in-context to its own history, making robots explore new expert-like behaviors online without online reinforcement learning.
-
Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
Data-driven adaptive exploration achieves O(N^{3/4}) regret in continuous-time linear-quadratic reinforcement learning, matching fixed-schedule methods and extending them to zero initial states.
-
Reinforcement Learning via Implicit Imitation Guidance
A reinforcement learning method that learns a state-dependent covariance from expert-policy action differences and uses it as exploration noise, improving sample efficiency on sparse-reward continuous control tasks.
-
Designing Pin-pression Gripper and Learning its Dexterous Grasping with Online In-hand Adjustment
A new active pin-array gripper combined with a curriculum reinforcement learning policy achieves substantially higher grasp success on unseen objects than fixed-jaw or passive grippers.
-
TextAtari: 100K Frames Game Playing with Language Agents
TextAtari is a text-based Atari benchmark for language agents; 7-8B LLMs stay below 10% of human scores in over 90% of tested conditions, and knowledge injection helps more than chain-of-thought.
-
TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.
-
First Return, Entropy-Eliciting Explore
FR3E improves LLM math reasoning by finding high-entropy tokens in a trajectory and doing extra rollouts from those points to shape rewards.
Discussion (0). Continue with ORCID to comment.