{"id":"613f3b03-a33d-45f5-a531-764b959c7e67","arxiv_id":"2505.00913","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Automatic Jump Start uses Fitted Q Evaluation to adapt the Jump-Start exploration schedule, reducing fine-tuning performance degradation without tuning a tolerance threshold.","lead":"The paper shows that many offline-to-online reinforcement learning algorithms either lose performance or learn slowly when fine-tuned online, and introduces Automatic Jump Start (AJS), an algorithm that uses off-policy performance estimates to decide when to let an exploration policy take control. AJS reduces the early fine-tuning performance drop while retaining most of the learning speed of Soft Actor-Critic.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's FQE comparison uses a fixed vinit against a re-trained estimator, so estimator drift, not policy improvement, can trigger premature exploration; a re-estimated-baseline ablation would settle it.","rationale":"The paper's stated contribution is an automatic schedule that expands exploration only when an OPE estimate says the current jump-start policy is at least as good as the initial policy. Everything the algorithm promises—low degradation with fast fine-tuning—rests on that comparison being correctly calibrated. My reading of Algorithm 1 found a concrete reason to doubt calibration: vinit is never recomputed after Fζ is retrained. This is more specific than 'FQE may be biased': even an unbiased FQE can have different bias at two checkpoints, and the decision threshold moves with the estimator. The paper also omits the FQE training budget k, the schedule denominator j, and any validation of FQE estimates against returns, all of which the reader flagged. I do not claim the empirical results are fabricated or that AJS fails; the 15-seed curves across nine D4RL settings are real evidence. The concern is that the mechanism's safety margin is unverified, and the title overclaims. The proposed ablation isolates exactly this margin. Because the reader's CONDITIONAL verdict already requires such validation, I recommend no change to the verdict, while sharpening the reason the condition is needed.","tokens_in":21847,"tokens_out":6943,"duration_ms":78046,"concrete_test":"Run AJS on the same 9 D4RL settings with one ablation: after every OPETraining call, recompute vinit by evaluating the fixed initial InAC guide policy with the current Fζ, and reduce h only when vft >= this re-estimated vinit. Compare the h schedule and the worst-return degradation metric to the paper's AJS. If the schedule or degradation changes materially, the original stale-baseline comparison is responsible; if not, the concern does not land. A complementary value check: record FQE's vft and the Monte Carlo return of πjs at each decision point and report the fraction of decisions where FQE reports vft>=vinit while true return is below p0.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing component of AJS is the guide-step decision in Algorithm 1: h is reduced only when vft >= vinit, where vinit is computed once as Fζ(S0,A0) after initial FQE training and vft is computed later as Fζ(S0,a~πjs(S0,0,h)). In between, Fζ is repeatedly retrained via OPETraining(Fζ,B,πjs,T) on a buffer that grows online, so the two quantities are produced by different estimators. FQE bias and scale can drift as the buffer and the evaluated policy change; a rising vft can therefore reflect estimator drift rather than genuine improvement of the jump-start policy over the initial policy. The paper gives no calibration check for this comparison and omits k, j, and FQE architectural details that would let a reader bound the error. Relatedly, FQE's Bellman target assumes a stationary policy, but πjs switches from InAC to SAC at step h, so the value being estimated is not well-defined unless Q is augmented with the time index or remaining guide steps; no such augmentation is described. If the comparison is miscalibrated, h can shrink prematurely and re-introduce degradation, or stay too large and sacrifice the fast learning that motivates AJS.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the problem of performance degradation during online fine-tuning of policies learned offline. It first provides empirical evidence that common offline-to-online methods either suffer from a sharp early performance drop (e.g., SAC fine-tuning) or learn too slowly (e.g., InAC, IQL, AWAC). It then proposes Automatic Jump-Start (AJS), which extends the Jump-Start RL framework by using Fitted Q Evaluation (FQE) to estimate the performance of the current jump-start policy and automatically decide when to increase the number of exploration steps. Experiments on D4RL MuJoCo environments (HalfCheetah, Hopper, Walker2d with Expert, Medium-Expert, and Medium datasets) compare AJS against InAC, SAC, PEX, and several JSRL variants, reporting that AJS balances stability and improvement without tuning the JSRL tolerance threshold.","tokens_in":22063,"tokens_out":5494,"duration_ms":54932,"significance":"If the empirical findings hold, AJS is a practically relevant contribution to offline-to-online reinforcement learning: it removes a deployment-time hyperparameter sweep that Jump-Start requires, which is important for real-world applications where tuning in the deployment environment is infeasible. The paper ships a reasonably thorough empirical study with 15 seeds per setting, 95% bootstrap confidence intervals, nine environment/dataset combinations, and comparisons against several relevant baselines. The idea of using an off-policy value estimator to schedule exploration is promising and goes beyond simple sliding-window return thresholds. However, the central safety mechanism of AJS is not yet rigorously justified: the FQE-based comparison that drives the schedule is susceptible to estimator drift and is not well-defined for the non-stationary jump-start policy as written. These issues need to be addressed before the central claim can be accepted.","major_comments":[{"comment":"The trigger for increasing exploration compares vinit, computed once with the initially trained FQE, against vft, computed later with an FQE that is retrained on a growing online buffer. Because Fζ drifts as the buffer and the evaluated policy change, an increase in vft can reflect estimator drift rather than genuine improvement of the jump-start policy over the initial policy. The paper provides no calibration check for this comparison and no ablation in which the baseline is re-estimated with the current FQE; without such a control, the safety guarantee of AJS is not established.","section":"§5.2, Algorithm 1"},{"comment":"FQE's Bellman target (Algorithm 3) samples the next action a'∼πe(s') for a stationary policy, but the evaluated policy πjs in Algorithm 2 switches from πη to πϕ at step h and therefore depends on the current time index t. The Q-function and the performance estimate vft are not well-defined unless the time index or remaining guide steps are included in the state representation, since a state s' reached at different times in the episode leads to different next-action distributions. As written, the value being estimated is ambiguous, which directly affects the decision rule in Algorithm 1.","section":"Algorithm 2, Algorithm 3"},{"comment":"The paper omits the values of k (initial FQE training iterations) and j (episodes over which the guide step is reduced), and the FQE retraining schedule and network details are not reported. Algorithm 1 also uses T both for episode length and as the number of FQE training iterations passed to OPETraining, which is confusing. Since the headline claim is that AJS avoids environment-specific hyperparameter tuning, these new parameters must be specified and their sensitivity analyzed; otherwise the algorithm is not reproducible and it is unclear whether the good results depend on hidden tuning.","section":"Algorithm 1, Section 13"},{"comment":"The title promises 'Fine-Tuning without Performance Degradation,' but the paper's own degradation metric in Figure 7 shows AJS has a nonzero average degradation, only slightly lower than SAC and slightly higher than InAC. The abstract's weaker claim of 'significantly reduces performance degradations' is supported, but the title and the phrase 'without performance degradation' overstate the result. Please revise the title and any equivalent claims to match the evidence.","section":"Title and §6.1, Figure 7"}],"minor_comments":[{"comment":"The abstract contains a typo: 'duringfine-tuning' should be 'during fine-tuning'.","section":"Abstract"},{"comment":"The sentence 'we see exactly the performance degradation we say above' should read 'we saw above'.","section":"§3.2"},{"comment":"The text 'For ASJ, the evaluated policy' should be 'For AJS', and the performance estimate formula has an unmatched bracket and an undefined A; please clarify the notation.","section":"§5.2"},{"comment":"The text contains 'The ASJ agent is able to quickly learn' and other instances of 'ASJ' that should be 'AJS'.","section":"§6.3"},{"comment":"The environment name 'HalfCheetach' is misspelled in several places; it should be 'HalfCheetah'.","section":"Figure 4 and elsewhere"},{"comment":"The statement 'The online buffer size was initialized to the offline dataset size to get access to all data for offline training' is ambiguous about whether the buffer is later allowed to grow; please clarify.","section":"§13.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the empirical study is well-structured, but the core FQE comparison that drives AJS needs either a theoretical guarantee, a calibration check, or an ablation with a re-estimated baseline. The missing hyperparameters and the non-stationary-policy issue are fixable with a careful revision, but they are load-bearing for the central claim. I would encourage the editor to request a revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a genuinely useful empirical paper, not a breakthrough. The new algorithm, AJS, is a thoughtful modification of Jump-Start RL that replaces the threshold epsilon with an FQE-based comparison, and the experiments mostly back up the central claim: it degrades less than SAC and learns faster than conservative offline algorithms. The title overpromises, though, and there is a real soft spot in how the FQE comparison is set up.\n\nWhat's new: the FQE-driven adaptive schedule for the guide-step h. The ablation against JSRL+SAC+InAC is the right control—it isolates the FQE mechanism from the InAC guide update, showing that AJS's balance comes from the value comparison, not just from updating the guide policy. The degradation survey across PEX, PROTO, IQL, AWAC, and ensemble variants is careful and reproducible in structure: 15 seeds, 95% bootstrap confidence intervals, nine environment/dataset pairs. Figure 7 is convincing that AJS sits where it claims to sit.\n\nThe soft spots. First, the title says 'without Performance Degradation,' but AJS still degrades—just less. That's an overclaim, not a fatal flaw. Second, the paper omits key constants: j, k, FQE training iterations, architecture details. Without those, replication is guesswork. Third, and more substantive: the FQE comparison in Algorithm 1 uses a vinit computed once, then compares it against a vft from a repeatedly retrained estimator. The stress-test note is right that estimator drift can trigger premature exploration, and the paper gives no calibration check, no error analysis, no ablation with a re-estimated baseline. Also, FQE's Bellman target assumes a stationary policy, but the evaluated πjs switches from guide to exploration at step h; unless Q is augmented with the time index, the value being estimated is not well-defined. The paper doesn't mention this. These concerns sit at the load-bearing point of the algorithm.\n\nOn the positive side, the omission of CAL-QL and Balanced Replay is a gap but not fatal; the paper is positioned as a practical study, and the comparison to JSRL variants is the relevant one.\n\nMy take: this deserves a serious referee. The empirical evidence for AJS is real, and the idea is worth engaging with. The authors should be pushed to provide error bounds on FQE or soften the safety claims, and to release code and all hyperparameters. Who is it for: people working on offline-to-online RL, especially deployment-oriented settings. It would be a solid paper at a main conference after revision.","headline":"Useful, careful empirical paper on offline-to-online fine-tuning degradation, with a genuinely new FQE-based schedule, but the title overclaims and the FQE reliability gap needs real work.","tokens_in":22608,"tokens_out":4705,"would_cite":true,"duration_ms":44579,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A value-driven schedule can fine-tune offline policies without early performance collapse.","keywords":["offline-to-online reinforcement learning","fine-tuning","performance degradation","Automatic Jump-Start","off-policy evaluation","Fitted Q Evaluation","InAC","SAC"],"falsifier":"Take a near-expert task, freeze the FQE critic after its offline training so it never sees online data, and run AJS with the schedule permitted to shrink h on vft >= vinit; if returns still degrade below the initial policy, the FQE comparison itself is too optimistic, and the claimed safety comes from training on online data rather than from the switching rule.","tokens_in":21609,"feed_emoji":"🤖","tokens_out":6142,"duration_ms":57488,"temperature":0.7,"pith_summary":"Fine-tuning an offline-learned reinforcement learning policy usually costs a performance collapse in the first phase of online interaction, and the remedies designed to prevent it either learn too slowly or still degrade. The paper claims that both problems can be avoided by one mechanism: start each episode by following a fixed guide policy, hand control to a fast online learner only for the final steps, and let the number of guide steps shrink automatically when an off-policy value estimate says the mixed policy is at least as good as the initial one. The resulting Automatic Jump-Start algorithm needs no per-environment threshold sweep, and the experiments on standard D4RL MuJoCo tasks show it reaches a similar final improvement to SAC while keeping degradation close to the level of the most conservative baselines. A sympathetic reader should take away that the hard part of fine-tuning is not exploration itself but deciding when exploration is safe, and that this decision can be delegated to a learned value comparison.","feed_headline":"A new algorithm fine-tunes policies without the usual performance drop","feed_subtitle":"It replaces a hand-tuned jump-start threshold with a learned value comparison, so deployment needs no environment sweeps.","key_machinery":"The carrying object is the jump-start switching rule: an episode is split at a guide step h, with a fixed guide policy trained by InAC controlling the first h actions and an online exploration policy trained by SAC controlling the rest. The new piece is the automatic schedule: Fitted Q Evaluation (FQE) regresses Bellman targets to estimate the value of the current mixed policy, and the algorithm decreases h only when the FQE estimate of the current policy is at least as high as the FQE estimate of the initial policy, with the tolerance set to zero. The guide decrement is fixed at $\\Delta = 2T/j$, with $T$ the episode horizon and $j$ a fixed number of episodes. This machinery converts the question 'how risky is exploration right now?' into a value comparison that can be computed from the growing replay buffer, without requiring access to the deployment environment for tuning.","core_discovery":"The central discovery is that a policy that switches from a stable guide policy to a fast online learner part-way through an episode can be scheduled without any hand-tuned threshold. The guide step h starts at the episode horizon, so the agent first acts like the offline policy; after each episode, the algorithm asks Fitted Q Evaluation whether the current mixed policy (guide for h steps, then SAC) has estimated value at least as high as the initial policy, and only then shrinks h by a fixed decrement. Because the comparison is made with off-policy estimates rather than noisy windowed returns, the tolerance of the original Jump-Start method can be set to zero, which removes the deployment-environment sweep. The paper also updates the guide policy itself with InAC rather than freezing it, which improves stability when the entropy setting is changed. Across Expert, Medium-Expert, and Medium datasets in three MuJoCo environments, AJS shows roughly SAC-level final improvement with only InAC-level degradation, and it expands exploration faster when the offline policy is worse.","pith_inferences":["A likely extension is to make the switch state-dependent rather than time-dependent: shrink h only in states where FQE error is small, which would reduce risk in safety-critical task regions.","The safety of the schedule is only as strong as the FQE estimate; early in fine-tuning the estimates are trained on a buffer that is mostly offline data, and ranking the mixed policy against the initial policy may carry systematic bias that the experiments do not isolate.","Another testable consequence is that a method that directly estimates the value difference between the two policies, rather than comparing two separate FQE fits, would reduce variance and allow even faster schedules without degradation."],"forward_implications":["If AJS works as reported, offline-to-online fine-tuning no longer needs a deployment-environment sweep for the jump-start tolerance: a default zero threshold plus an off-policy estimate replaces it.","Conservative offline learners can be used as the guide policy without freezing them; updating the guide policy by InAC keeps the stability the method relies on.","Exploration accelerates when the offline policy is weak and stays conservative when it is near-optimal, because the FQE comparison reflects actual performance gains.","The degradation/improvement trade-off is not inevitable: a method can reach SAC-like final improvement while holding worst-case performance close to the conservative baseline, at least on the tested tasks."],"supporting_citations":[{"why":"Introduces Jump-Start RL, the guide/exploration switching mechanism AJS builds on.","marker":"Uchendu et al., 2023"},{"why":"Supplies Fitted Q Evaluation, the off-policy estimator used to set the guide step.","marker":"Le et al., 2019"},{"why":"Supplies InAC, used for the offline initialization and as the guide policy with its stability-preserving updates.","marker":"Xiao et al., 2023"},{"why":"Supplies SAC, the fast online exploration and learning component and the main performance-degradation baseline.","marker":"Haarnoja et al."},{"why":"Supplies the D4RL datasets and normalized-return protocol used in all experiments.","marker":"Fu et al., 2020"},{"why":"Policy Expansion, the stability-focused baseline that also keeps a fixed offline policy.","marker":"Zhang et al., 2023"},{"why":"PROTO, the KL-penalty baseline used to show that conservative fine-tuning learns slowly.","marker":"Li et al., 2023"},{"why":"IQL, the offline baseline whose fine-tuning variant appears in the stable-but-slow comparison and in a Jump-Start variant.","marker":"Kostrikov et al., 2022"}],"fun_headline_variants":["AJS fine-tunes with minimal performance degradation","Adaptive Jump Start: faster fine-tuning, less loss","No more early fine-tuning drop with AJS","AJS: fine-tuning without the initial slump","AJS: remove hand-tuned sweeps in fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the off-policy value estimate ranking the current mixed policy correctly against the initial policy; if FQE is biased or noisy in the first episodes, the schedule either lets exploration in too soon or keeps the agent on the guide policy too long.","fun_headline_variants_meta":{"raw":{"variants":["AJS fine-tunes with minimal performance degradation","Adaptive Jump Start: faster fine-tuning, less loss","No more early fine-tuning drop with AJS","AJS: fine-tuning without the initial slump","AJS: remove hand-tuned sweeps in fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1488,"prompt_tokens":914,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":498}},"tokens_in":530,"tokens_out":574,"duration_ms":6478,"temperature":1.0,"reasoning_tokens":498,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:31:22.267961+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a near-expert task, freeze the FQE critic after its offline training so it never sees online data, and run AJS with the schedule permitted to shrink h on vft >= vinit; if returns still degrade below the initial policy, the FQE comparison itself is too optimistic, and the claimed safety comes from training on online data rather than from the switching rule.","supporting_citations":[{"cited_title":"Jump-start reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Introduces Jump-Start RL, the guide/exploration switching mechanism AJS builds on."}],"review_version":1}