REVIEW 4 major objections 6 minor 33 references
Fine-Tuning without Performance Degradation
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A value-driven schedule can fine-tune offline policies without early performance collapse.
desk verdict Useful, careful empirical paper on offline-to-online fine-tuning degradation, with a genuinely new FQE-based schedule, but the title overclaims and the FQE reliability gap needs real work. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the jump-start switching rule: an episode is split at a guide step h, with a fixed guide policy trained by InAC controlling the first h actions and an online exploration policy trained by SAC controlling the rest. The new piece is the automatic schedule: Fitted Q Evaluation (FQE) regresses Bellman targets to estimate the value of the current mixed policy, and the algorithm decreases h only when the FQE estimate of the current policy is at least as high as the FQE estimate of the initial policy, with the tolerance set to zero. The guide decrement is fixed at $\Delta = 2T/j$, with $T$ the episode horizon and $j$ a fixed number of episodes. This machinery converts the question 'how risky is exploration right now?' into a value comparison that can be computed from the growing replay buffer, without requiring access to the deployment environment for tuning.
What would settle it
Take a near-expert task, freeze the FQE critic after its offline training so it never sees online data, and run AJS with the schedule permitted to shrink h on vft >= vinit; if returns still degrade below the initial policy, the FQE comparison itself is too optimistic, and the claimed safety comes from training on online data rather than from the switching rule.
Extended reading notes
Core claim
The central discovery is that a policy that switches from a stable guide policy to a fast online learner part-way through an episode can be scheduled without any hand-tuned threshold. The guide step h starts at the episode horizon, so the agent first acts like the offline policy; after each episode, the algorithm asks Fitted Q Evaluation whether the current mixed policy (guide for h steps, then SAC) has estimated value at least as high as the initial policy, and only then shrinks h by a fixed decrement. Because the comparison is made with off-policy estimates rather than noisy windowed returns, the tolerance of the original Jump-Start method can be set to zero, which removes the deployment-environment sweep. The paper also updates the guide policy itself with InAC rather than freezing it, which improves stability when the entropy setting is changed. Across Expert, Medium-Expert, and Medium datasets in three MuJoCo environments, AJS shows roughly SAC-level final improvement with only InAC-level degradation, and it expands exploration faster when the offline policy is worse.
Load-bearing premise
Everything rests on the off-policy value estimate ranking the current mixed policy correctly against the initial policy; if FQE is biased or noisy in the first episodes, the schedule either lets exploration in too soon or keeps the agent on the guide policy too long.
Editorial extensions
If this is right
- If AJS works as reported, offline-to-online fine-tuning no longer needs a deployment-environment sweep for the jump-start tolerance: a default zero threshold plus an off-policy estimate replaces it.
- Conservative offline learners can be used as the guide policy without freezing them; updating the guide policy by InAC keeps the stability the method relies on.
- Exploration accelerates when the offline policy is weak and stays conservative when it is near-optimal, because the FQE comparison reflects actual performance gains.
- The degradation/improvement trade-off is not inevitable: a method can reach SAC-like final improvement while holding worst-case performance close to the conservative baseline, at least on the tested tasks.
Reading between the lines
- A likely extension is to make the switch state-dependent rather than time-dependent: shrink h only in states where FQE error is small, which would reduce risk in safety-critical task regions.
- The safety of the schedule is only as strong as the FQE estimate; early in fine-tuning the estimates are trained on a buffer that is mostly offline data, and ranking the mixed policy against the initial policy may carry systematic bias that the experiments do not isolate.
- Another testable consequence is that a method that directly estimates the value difference between the two policies, rather than comparing two separate FQE fits, would reduce variance and allow even faster schedules without degradation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the problem of performance degradation during online fine-tuning of policies learned offline. It first provides empirical evidence that common offline-to-online methods either suffer from a sharp early performance drop (e.g., SAC fine-tuning) or learn too slowly (e.g., InAC, IQL, AWAC). It then proposes Automatic Jump-Start (AJS), which extends the Jump-Start RL framework by using Fitted Q Evaluation (FQE) to estimate the performance of the current jump-start policy and automatically decide when to increase the number of exploration steps. Experiments on D4RL MuJoCo environments (HalfCheetah, Hopper, Walker2d with Expert, Medium-Expert, and Medium datasets) compare AJS against InAC, SAC, PEX, and several JSRL variants, reporting that AJS balances stability and improvement without tuning the JSRL tolerance threshold.
Significance. If the empirical findings hold, AJS is a practically relevant contribution to offline-to-online reinforcement learning: it removes a deployment-time hyperparameter sweep that Jump-Start requires, which is important for real-world applications where tuning in the deployment environment is infeasible. The paper ships a reasonably thorough empirical study with 15 seeds per setting, 95% bootstrap confidence intervals, nine environment/dataset combinations, and comparisons against several relevant baselines. The idea of using an off-policy value estimator to schedule exploration is promising and goes beyond simple sliding-window return thresholds. However, the central safety mechanism of AJS is not yet rigorously justified: the FQE-based comparison that drives the schedule is susceptible to estimator drift and is not well-defined for the non-stationary jump-start policy as written. These issues need to be addressed before the central claim can be accepted.
major comments (4)
- [§5.2, Algorithm 1] The trigger for increasing exploration compares vinit, computed once with the initially trained FQE, against vft, computed later with an FQE that is retrained on a growing online buffer. Because Fζ drifts as the buffer and the evaluated policy change, an increase in vft can reflect estimator drift rather than genuine improvement of the jump-start policy over the initial policy. The paper provides no calibration check for this comparison and no ablation in which the baseline is re-estimated with the current FQE; without such a control, the safety guarantee of AJS is not established.
- [Algorithm 2, Algorithm 3] FQE's Bellman target (Algorithm 3) samples the next action a'∼πe(s') for a stationary policy, but the evaluated policy πjs in Algorithm 2 switches from πη to πϕ at step h and therefore depends on the current time index t. The Q-function and the performance estimate vft are not well-defined unless the time index or remaining guide steps are included in the state representation, since a state s' reached at different times in the episode leads to different next-action distributions. As written, the value being estimated is ambiguous, which directly affects the decision rule in Algorithm 1.
- [Algorithm 1, Section 13] The paper omits the values of k (initial FQE training iterations) and j (episodes over which the guide step is reduced), and the FQE retraining schedule and network details are not reported. Algorithm 1 also uses T both for episode length and as the number of FQE training iterations passed to OPETraining, which is confusing. Since the headline claim is that AJS avoids environment-specific hyperparameter tuning, these new parameters must be specified and their sensitivity analyzed; otherwise the algorithm is not reproducible and it is unclear whether the good results depend on hidden tuning.
- [Title and §6.1, Figure 7] The title promises 'Fine-Tuning without Performance Degradation,' but the paper's own degradation metric in Figure 7 shows AJS has a nonzero average degradation, only slightly lower than SAC and slightly higher than InAC. The abstract's weaker claim of 'significantly reduces performance degradations' is supported, but the title and the phrase 'without performance degradation' overstate the result. Please revise the title and any equivalent claims to match the evidence.
minor comments (6)
- [Abstract] The abstract contains a typo: 'duringfine-tuning' should be 'during fine-tuning'.
- [§3.2] The sentence 'we see exactly the performance degradation we say above' should read 'we saw above'.
- [§5.2] The text 'For ASJ, the evaluated policy' should be 'For AJS', and the performance estimate formula has an unmatched bracket and an undefined A; please clarify the notation.
- [§6.3] The text contains 'The ASJ agent is able to quickly learn' and other instances of 'ASJ' that should be 'AJS'.
- [Figure 4 and elsewhere] The environment name 'HalfCheetach' is misspelled in several places; it should be 'HalfCheetah'.
- [§13.2] The statement 'The online buffer size was initialized to the offline dataset size to get access to all data for offline training' is ambiguous about whether the buffer is later allowed to grow; please clarify.
Circularity Check
No load-bearing circularity: the FQE-based guide-step schedule is an online control signal, and the paper's performance claims are validated by external environment rollouts.
full rationale
The paper's chain is empirical rather than derivational: Section 3 documents performance degradation, Section 4 shows conservative algorithms avoid degradation but learn slowly, and Section 5 introduces Automatic Jump Start, whose guide step h is reduced only when the FQE estimate vft reaches the fixed initial estimate vinit. This FQE comparison is an algorithmic feedback signal, not a fitted parameter renamed as a prediction; the claimed properties of AJS—low degradation and fast fine-tuning—are evaluated by actual returns on D4RL against SAC, PEX, InAC, and JSRL variants, so the empirical conclusions are grounded outside the FQE estimate itself. The only author self-citation is InAC (Xiao et al., 2023), used as an off-the-shelf offline algorithm and additionally characterized by this paper's own experiments; it is not invoked as a uniqueness theorem and does not force the AJS outcome. No equation in the paper reduces a prediction to its input by construction, and no fitted value is reported as a predicted performance. The self-referential loop of using an estimate of the current policy to decide how much to explore is a control design choice, not a circular derivation, and the paper includes an ablation (JSRL+SAC+InAC) that isolates the FQE mechanism. Accordingly, there is no significant circularity.
Assumptions & free parameters
free parameters (4)
- j (episodes for guide-step reduction)
- k (initial FQE training iterations)
- vinit comparison slack =
0
- FQE retraining schedule
assumptions (4)
- domain assumption The fine-tuning environment is the same MDP that generated the offline dataset.
- domain assumption FQE converges to an accurate estimate of the value of the mixed jump-start policy.
- domain assumption InAC fine-tuning remains stable and does not cause severe degradation, so updating the guide policy with InAC is safe.
- standard math Standard deep RL assumptions hold, including sufficient function approximation and convergence of the critic.
Cite this review
Pith. "Pith review of Fine-Tuning without Performance Degradation." pith.science (2026). https://pith.science/paper/BSAZXSVB
@misc{pith2026250500913,
author = {Pith},
title = {Pith review of: Fine-Tuning without Performance Degradation},
year = {2026},
howpublished = {\url{https://pith.science/paper/BSAZXSVB}},
note = {Machine review of arXiv:2505.00913}
}
read the original abstract
Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during \emph{fine-tuning} is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.
Figures
Figures from the paper (19 more)
Reference graph
Works this paper leans on
-
[1]
Better fine-tuning by reducing representational collapse
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations, 2021
work page 2021
-
[2]
Uncertainty-based offline reinforcement learning with diversified q-ensemble
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In Advances in neural information processing systems, 2021
work page 2021
-
[3]
Efficient online reinforcement learning with offline data
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp.\ 1577--1594. PMLR, 2023
work page 2023
-
[4]
Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
V \' ctor Campos, Pablo Sprechmann, Steven Hansen, Andre Barreto, Steven Kapturowski, Alex Vitvitskyi, Adria Puigdomenech Badia, and Charles Blundell. Beyond fine-tuning: Transferring behavior in reinforcement learning. arXiv preprint arXiv:2102.13515, 2021
work page Pith review arXiv 2021
-
[5]
D4rl: Datasets for deep data-driven reinforcement learning, 2020
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020
2020
-
[6]
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. Pmlr
-
[7]
Soft actor-critic algorithms and applications
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018
arXiv 2018
-
[8]
Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning
Ryan Julian, Benjamin Swanson, Gaurav Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. In Proceedings of the 2020 Conference on Robot Learning, 2020
work page 2020
Show all 33 references
-
[9]
Offline reinforcement learning with implicit q-learning
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022
2022
-
[10]
Conservative q-learning for offline reinforcement learning
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020
2020
-
[11]
Batch policy learning under constraints
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, 2019
2019
-
[12]
Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In 5th Annual Conference on Robot Learning, 2021
2021
-
[13]
PROTO : Iterative policy regularized offline-to-online reinforcement learning
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. PROTO : Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305.15669, 2023
2023 arXiv
-
[14]
Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions
Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions. arXiv preprint arXiv:2303.17396, 2023
2023 arXiv
-
[15]
Mildly conservative q-learning for offline reinforcement learning
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Thirty-sixth Conference on Neural Information Processing Systems, 2022
2022
-
[16]
What happens to BERT embeddings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to BERT embeddings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020
2020
-
[17]
AWAC : Accelerating online reinforcement learning with offline datasets
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC : Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020
2006 arXiv
-
[18]
Cal- QL : Calibrated offline RL pre-training for efficient online fine-tuning
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal- QL : Calibrated offline RL pre-training for efficient online fine-tuning. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023
2023
-
[19]
Peters, Sebastian Ruder, and Noah A
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, 2019
2019
-
[20]
Lifelong generative modeling
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling. Neurocomputing, 404: 0 381--400, 2020
2020
-
[21]
Improving language models fine-tuning with representation consistency targets
Anastasia Razdaibiedina, Vivek Madan, Zohar Karnin, Ashish Khetan, and Vishaal Kapoor. Improving language models fine-tuning with representation consistency targets. arXiv:2205.11603v1, 2022
2022 arXiv
-
[22]
Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine
Laura Smith, J. Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In International Conference on Robotics and Automation, 2022
2022
-
[23]
Hybrid RL : Using both offline and online data can make RL efficient
Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL : Using both offline and online data can make RL efficient. In International Conference on Learning Representations, 2023
2023
-
[24]
Jump-start reinforcement learning
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Jos\' e phine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-start reinforcement learning. In International Conference on Machine Learning, 2023
2023
-
[25]
Unifying task specification in reinforcement learning
Martha White. Unifying task specification in reinforcement learning. In International Conference on Machine Learning, 2017
2017
-
[26]
Principal component analysis
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2 0 (1): 0 37--52, 1987
1987
-
[27]
The in-sample softmax for offline reinforcement learning
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. In International Conference on Learning Representations, 2023
2023
-
[28]
Policy expansion for bridging offline-to-online reinforcement learning
Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. In International Conference on Learning Representations, 2023
2023
-
[29]
Revisiting few-sample BERT fine-tuning
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations, 2021
2021
-
[30]
Improving offline-to-online reinforcement learning with q-ensembles
Kai Zhao, Yi Ma, Jinyi Liu, HAO Jianye, Yan Zheng, and Zhaopeng Meng. Improving offline-to-online reinforcement learning with q-ensembles. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023
2023
-
[31]
Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. arXiv:2210.13846, 2022
2022 arXiv
-
[32]
A closer look at how fine-tuning changes BERT
Yichu Zhou and Vivek Srikumar. A closer look at how fine-tuning changes BERT . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022
2022
-
[33]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.