REVIEW 6 cited by
Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Fine-tuning is a widespread technique that allows practitioners to transfer pre-trained capabilities, as recently showcased by the successful applications of foundation models. However, fine-tuning reinforcement learning (RL) models remains a challenge. This work conceptualizes one specific cause of poor transfer, accentuated in the RL setting by the interplay between actions and observations: forgetting of pre-trained capabilities. Namely, a model deteriorates on the state subspace of the downstream task not visited in the initial phase of fine-tuning, on which the model behaved well due to pre-training. This way, we lose the anticipated transfer benefits. We identify conditions when this problem occurs, showing that it is common and, in many cases, catastrophic. Through a detailed empirical analysis of the challenging NetHack and Montezuma's Revenge environments, we show that standard knowledge retention techniques mitigate the problem and thus allow us to take full advantage of the pre-trained capabilities. In particular, in NetHack, we achieve a new state-of-the-art for neural models, improving the previous best score from $5$K to over $10$K points in the Human Monk scenario.
Forward citations
Cited by 6 Pith papers
-
Latent Activation Editing: Inference-Time Refinement of Learned Policies for Safer Multirobot Navigation
Editing a frozen RL policy's latent activations at inference time, using a collision world model, cuts collisions by about 90% on a curated set of hard multirotor scenarios and on real Crazyflies.
-
Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning
SGFT uses a simulation-trained value function to guide real-world exploration via potential-based reward shaping and short-horizon objectives, substantially improving fine-tuning sample efficiency.
-
MaestroMotif: Skill Design from Artificial Intelligence Feedback
MaestroMotif uses LLM-generated rewards and code policies to train and combine five NetHack skills, solving complex text-specified tasks without task-specific RL training.
-
Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
Warm-start RL (WSRL) fine-tunes offline-pretrained RL agents online with no offline data retention, using 5,000 warm-up rollouts from the frozen pre-trained policy followed by standard high-UTD SAC.
-
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
A benchmark of six long-horizon game environments shows current LLMs and VLMs struggle on hard tasks and often do worse when given images.
-
From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning
Pretraining a navigation foundation model on videos and then fine-tuning only a residual attention module with reinforcement learning improves success rate and collision avoidance compared to behavior cloning or super...
Discussion (0). Continue with ORCID to comment.