{"id":"a52e4cab-e243-43d7-9c66-7ba8a18724b9","arxiv_id":"2412.03767","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Hyper makes curiosity-driven RL less sensitive to the exploration bonus weight by separating repositioning from exploration, with a claimed polynomial sample-complexity bound.","lead":"This paper proposes Hyper, an RL algorithm that alternates a repositioning phase driven by a learned exploitation policy with a curiosity-driven exploration phase, aiming to make performance robust to the curiosity bonus weight beta. The authors provide a sample-complexity bound under a linear MDP assumption and report maze and locomotion experiments showing Hyper tolerates a wider range of beta than standard curiosity-driven RL.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem B.1's p-independent sample complexity is not established: the proof drops a 1/p factor in the final regret-to-PAC conversion.","rationale":"The paper's stated contribution is a provably efficient, β-robust algorithm with a p-independent sample-complexity guarantee. That guarantee is exactly what Theorem B.1 formulates, and the proof of Theorem B.14 is the only support. The final inequality in that proof is algebraically incorrect: a factor 1/p is discarded. Since p is a free hyperparameter in (0,1), this is not a constant-factor issue. This is more load-bearing than the theory-experiment mismatch emphasized by the reader: even for the clean linear Algorithm 1 analyzed in the appendix, the stated complexity is unproven. The reader did correctly identify the 1/p waiting-time argument in Eq. (32)-(33) as fragile, so there is partial overlap; but the dropped 1/p in the final conversion is a sharper, self-contained defect. No additional concerns are needed to support rejection. The empirical results are suggestive but do not rescue the theoretical claim, and the lack of code release limits verification.","tokens_in":26337,"tokens_out":14125,"duration_ms":129089,"concrete_test":"Recompute the last two lines of the proof of Theorem B.14: substitute the bound for ∑_{k∈I}(…) into the total regret expression, keeping the factor 1/p. Then convert regret to sample complexity by solving (C/p)·sqrt(d^3 H^3 T) ≤ ε·(T/H). If the result is O(d^3 H^4/(p^2 ε^2)) rather than O(d^3 H^4/ε^2), Theorem B.1 needs either a p-dependent bound or a restriction to constant p.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the proof of Theorem B.14, the total regret is upper bounded by (C'/p)·sqrt(d^3 H^3 T ι^2) and then declared equal to O-tilde(sqrt(d^3 H^3 T ι^2)), silently dropping the factor 1/p. This is not a harmless constant: p ∈ (0,1) is arbitrary, and the theorem claims a p-independent sample complexity O-tilde(d^3 H^4/ε^2). Carrying the 1/p through the regret-to-PAC conversion gives O(d^3 H^4/(p^2 ε^2)) total steps, which can be arbitrarily worse as p → 0. The earlier step Eq. (32)-(33) also replaces the per-episode sum by (1/p) times the sum over L_k=0 episodes; even granting that renewal argument, the 1/p must remain in the final bound. Thus Theorem B.1—the paper's central 'any p' guarantee—does not follow from the supplied proof. The reader's concern about the empirical Algorithm 2 (per-step updates, decaying p) is a separate mismatch, but this dropped 1/p already breaks the theorem for the provable Algorithm 1.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hyper, a two-policy exploration algorithm that interleaves a repositioning phase, driven by an exploitation policy, with a curiosity-driven exploration phase, with the goal of making curiosity-based RL robust to the intrinsic-reward coefficient beta. The theoretical section claims a provably efficient version, Linear-UCB-Hyper, with sample complexity O~(d^3 H^4 / epsilon^2) for any truncation probability p in (0,1) under linear MDP assumptions, and the empirical section evaluates the practical version (Algorithm 2), with TD3 and Disagreement, on navigation and locomotion tasks, reporting better beta-robustness than baselines.","tokens_in":26577,"tokens_out":7925,"duration_ms":80498,"significance":"If the theorem were correct, the paper would make a valuable contribution by combining decoupled exploitation/exploration with a provable sample-efficiency guarantee that is robust to the key hyperparameter p, and the empirical robustness analysis would be a useful practical addition. The paper also contains a clear exposition of the beta-sensitivity problem and a reasonable experimental design with multiple environments and baselines. However, the theoretical guarantee is the paper's central advertised contribution, and the supplied proof has a load-bearing gap in the treatment of the truncation probability p. The empirical results are suggestive but do not compensate for this, because the evaluated Algorithm 2 differs from the provable Algorithm 1 in update cadence and p schedule. The paper is not in a publishable state as it stands.","major_comments":[{"comment":"The proof drops the 1/p factor introduced at Eq. (33). After bounding the sum over fully-exploratory episodes by C' * sqrt(d^3 H^3 T iota^2), the paper concludes total regret <= (C'/p) * sqrt(d^3 H^3 T iota^2) = O~(sqrt(d^3 H^3 T iota^2)). This equality is false as an asymptotic statement for arbitrary p in (0,1): p is an algorithm hyperparameter, not a fixed constant, and carrying 1/p through the regret-to-PAC conversion gives O~(d^3 H^4 / (p^2 epsilon^2)) total steps, which can be arbitrarily worse as p -> 0. Theorem B.1's p-independent claim therefore does not follow from the supplied proof.","section":"Appendix B, Theorem B.14 (last display)"},{"comment":"The symbol p is used simultaneously as the geometric truncation probability and as the confidence parameter: Lemma B.8 states P(E) >= 1 - p/2 with p the truncation probability, while Theorem B.14 concludes 'with probability at least 1 - delta' and defines beta using log(2dT/delta). The proof never relates the geometric p to the confidence delta, so the high-probability statement of Theorem B.1 is not well-formed for arbitrary p. The two roles of p must be separated before the theorem can be evaluated.","section":"Appendix B, Lemmas B.8-B.9 and Theorem B.14"},{"comment":"The paper claims that Algorithm 1 and Algorithm 2 'differ only in function approximation,' but they also differ in a way that matters for the proof: the theoretical analysis (Eq. (32)-(33)) requires policy weights to be updated only on episodes with L_k = 0 and uses a fixed geometric p, whereas Algorithm 2 updates the policies at every environment step and linearly decays p (lines 14-20). The theorem therefore does not cover the algorithm that is actually evaluated in the experiments, so the empirical results cannot be interpreted as validating the theoretical efficiency claim.","section":"Section 4 and Algorithm 2"}],"minor_comments":[{"comment":"The informal theorem in the main text is too vague: it states only 'polynomially' and omits the dependence on p, which is the central quantity in question. The formal statement should appear in the main text or be referenced with its exact p-dependence.","section":"Section 4, Theorem 4.2"},{"comment":"The assignment for the pessimistic weight uses 'checkQ*_{h+1}' which appears to be a typo; it should be the maximum over actions of the pessimistic Q function, matching Algorithm 3.","section":"Algorithm 1, line 25"},{"comment":"The proof refers to 'Lemma B.1' when bounding the elliptic potential; the referenced lemma does not exist in the appendix. The citation should be to Lemma C.1 or C.2.","section":"Appendix B, Lemma B.6 proof"},{"comment":"The derivation from H beta iota sqrt(2 d K p) to the final C' sqrt(d^3 H^3 T iota^2) involves an unchecked exponent in H: with T = K H, the displayed expression should be checked carefully, since the powers of H appear inconsistent across the displayed inequalities.","section":"Appendix B, Eq. (47)"}],"recommendation":"reject","confidential_remarks":"The paper's empirical section is reasonably thorough and the beta-robustness observation is interesting, but the central theoretical result is not supported by the proof: the dropped 1/p factor is a load-bearing error that invalidates the claimed p-independent sample complexity. The mismatch between the provable algorithm and the evaluated implementation compounds the problem. This is not a case where a small revision would restore the main claim; the theorem would need to be restated with an explicit 1/p dependence, which would substantially weaken the advertised contribution, or the algorithm and experiments would need to be reworked to match a corrected analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical core is worth a look, but the paper's central theoretical claim, Theorem B.1, is not established. In the proof of Theorem B.14, the regret bound comes out as (C'/p)·sqrt(d^3 H^3 T ι^2), and the next line declares this is O-tilde(sqrt(d^3 H^3 T ι^2)). That drop of 1/p is load-bearing: p ∈ (0,1) is arbitrary, and carrying it through the PAC conversion gives O(d^3 H^4/(p^2 ε^2)) total steps, which is not p-independent. So the advertised \"any truncation probability\" guarantee does not follow from the supplied proof. The reader's other concern, that Algorithm 2 updates every step and decays p while Algorithm 1 and the theorem update only on episodes with L_k=0, is real but secondary; the dropped 1/p already breaks the theorem for the provable version.\n\nWhat is genuinely good here is the warm-up analysis in Section 3. The tabular example showing how the proper β range shifts with reward scales is clear and useful, and the repositioning-and-exploration idea, with the bounded geometric distribution, is a sensible engineering contribution beyond the existing decoupling papers [18,19,39]. The empirical β-sensitivity comparison across several tasks is also informative, though five seeds and no code release limit how much weight it can carry. The proof up to the renewal step is mostly standard LSVI-UCB machinery; Lemmas B.10–B.13 look like the usual optimistic/pessimistic decomposition. The problem is concentrated in the renewal sum and the final conversion, which is exactly where the paper goes wrong.\n\nI would not cite this paper as a proof of p-robust sample efficiency, and the theorem as written should not be taken at face value. That said, the paper is not incoherent, and the empirical recipe may survive a corrected analysis. I would send it to peer review rather than desk-reject: a serious referee could push the authors to either prove a p-dependent bound, or restrict the theorem to a fixed p that matches the evaluated algorithm, and the empirical contribution is worth publishing if the theory is made honest.","headline":"The empirical recipe is plausible and the warm-up analysis is nice, but the central 'any p' sample-complexity theorem has a dropped 1/p factor that breaks the stated guarantee.","tokens_in":27129,"tokens_out":3022,"would_cite":false,"duration_ms":34681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that splitting each episode into a repositioning phase and an exploration phase makes curiosity-driven reinforcement learning sample-efficient and largely insensitive to the curiosity coefficient.","keywords":["reinforcement learning","curiosity-driven exploration","hyperparameter robustness","exploration-exploitation trade-off","linear MDP","sample complexity","repositioning and exploration","decoupled policies"],"falsifier":"Run the neural Hyper algorithm exactly as in the paper but replace per-step updates with updates only on episodes with $L_k=0$ and freeze $p$; if final performance in a sparse-reward maze or locomotion task changes by more than run-to-run variance, the theorem's preconditions are doing real work. Conversely, a single environment with a narrow proper-$\\beta$ window where Hyper still collapses for large $\\beta$ would refute the robustness claim.","tokens_in":26125,"feed_emoji":"🫟","tokens_out":7177,"duration_ms":65787,"temperature":0.7,"pith_summary":"Curiosity-driven reinforcement learning adds an intrinsic reward scaled by a coefficient $\\beta$, and the right value of $\\beta$ varies wildly with the environment: too small and the agent settles for suboptimal states, too large and it over-explores forever. The paper argues that the real difficulty is not the coefficient itself but the coupling of exploration and exploitation in a single policy, and it proposes Hyper, which splits each episode into a repositioning phase run by an exploitation policy and an exploration phase run by a curiosity-driven policy. The paper claims that this split, together with a bounded geometric distribution over the repositioning length, keeps exploration persistent without letting intrinsic rewards corrupt exploitation. It proves that a linear-function-approximation version of Hyper obtains an $\\epsilon$-optimal policy in $\\widetilde{O}(d^3 H^4/\\epsilon^2)$ samples, and reports experiments showing Hyper matches or beats baselines while being far less sensitive to $\\beta$.","feed_headline":"Hyper splits episodes to make curiosity-driven RL robust to β","feed_subtitle":"A repositioning phase before exploration keeps sample efficiency while removing the need to tune the curiosity weight.","key_machinery":"The load-bearing object is the repositioning-and-exploration mechanism. In each episode a length $L_k$ is drawn from a bounded geometric distribution with parameter $p$; the first $L_k$ steps follow the exploitation policy, and the remaining $H - L_k$ steps follow the curiosity-driven exploration policy. In the provable version the exploration policy is built from an optimistic UCB value function and a pessimistic lower-bound value function on a linear MDP, whose transition and reward are linear in a known feature map $\\phi$. The proof's sample-complexity bound follows because policy weights change only on episodes with $L_k = 0$, so the expected number of episodes between updates is $1/p$, which inflates the regret by at most $1/p$. In the practical version the same two-phase structure is used with neural-network value functions, a decay schedule for $p$, and per-step updates.","core_discovery":"The central discovery claimed is that a repositioning-and-exploration cycle decouples task learning from curiosity exploration: before exploring, the agent follows its current exploitation policy for a geometrically distributed number of steps, then switches to the exploration policy for the rest of the episode. This makes the exploration policy start from states that the exploitation policy judges promising, so the exploration bonus cannot dominate the task reward even when $\\beta$ is large. The theoretical version maintains optimistic and pessimistic Q-functions with UCB-style bonuses, and it updates its weights only on episodes whose repositioning length is zero. Theorem B.1 states that with any fixed truncation probability $p \\in (0,1)$, this Linear-UCB-Hyper finds an $\\epsilon$-optimal exploitation policy in $\\widetilde{O}(d^3 H^4/\\epsilon^2)$ steps with high probability. The paper reads this as showing that Hyper is simultaneously sample-efficient and robust to the curiosity coefficient, the two properties that usually trade against each other.","pith_inferences":["The proof and the evaluated algorithm are not the same object: Theorem B.1 requires a fixed $p$ and updates only on episodes with $L_k=0$, while the practical algorithm decays $p$ and updates every step. If that gap is not closed, the sample-complexity guarantee should be attributed to a restricted variant, and the robustness evidence for the practical method remains empirical.","A natural testable extension is to run the neural version exactly as described in the paper but freeze $p$ and update only after full-exploration episodes; if performance barely changes, the mismatch is benign, and if it collapses, the theorem's precondition is genuinely load-bearing.","The bounded geometric distribution is behaving like a temporally extended option that moves the agent to a promising region before exploring, so one could connect Hyper to option-based exploration and ask whether learning the switch policy, rather than fixing $p$, improves robustness further.","The theory's dependence on $\\beta$ is hidden in constants: the bonus scale enters only as a logarithmic factor in the confidence bound, so the robustness claim is asymptotic, while the practical gain is that large $\\beta$ no longer destabilizes training, which the proof does not quantify."],"forward_implications":["If Theorem B.1 is right, curiosity-driven exploration can be made robust to $\\beta$ without sacrificing sample efficiency, so practitioners can set $\\beta$ large to encourage exploration and rely on the repositioning phase to keep exploitation stable.","The decoupling claim implies that exploitation learning can be trained without intrinsic rewards while still using exploration data, as long as repositioning aligns the two data distributions, which directly addresses the distribution-shift failure of simple decoupled agents.","Because the mechanism does not require resettable environments, it transfers the idea of returning to promising states, previously only practical when episodes can be reset, to general non-resettable reinforcement learning settings.","The theorem gives an explicit polynomial sample bound of $\\widetilde{O}(d^3 H^4/\\epsilon^2)$ for linear MDPs, so the method is certified to be sample-efficient in the worst case, not merely empirically.","The bounded geometric distribution prevents probability mass from piling up at the full horizon, keeping the repositioning phase useful in environments with short episodes."],"supporting_citations":[{"why":"Supplies the linear-MDP value-iteration construction, the realizability and optimism lemmas, and the covering-number bounds on which Theorem B.1 relies.","marker":"[17]"},{"why":"Supplies the self-normalized concentration inequality and determinant-ratio bounds used to control the bonus terms in the regret proof.","marker":"[40]"},{"why":"Provides the feature-map linear MDP assumption adopted in Assumption 4.1.","marker":"[22]"},{"why":"Supplies the UCB-style Q-learning baseline used to demonstrate the sensitivity of curiosity methods to the coefficient $\\beta$.","marker":"[15]"},{"why":"Supplies the ensemble-disagreement intrinsic reward used as the curiosity module in the empirical comparisons.","marker":"[12]"},{"why":"Supplies the off-policy actor-critic learning algorithm on which Hyper and the baselines are implemented.","marker":"[23]"},{"why":"Supplies the decoupling baseline that Hyper extends with the repositioning phase to avoid distribution shift.","marker":"[18]"},{"why":"Supplies the fixed-target intrinsic reward used in the discrete-grid comparison with an option-based method.","marker":"[10]"},{"why":"Establishes the confidence-bound exploration principle that motivates the UCB intrinsic reward.","marker":"[13]"},{"why":"Provides the reset-based exploration paradigm that Hyper generalizes to settings where the environment cannot be reset.","marker":"[30]"}],"fun_headline_variants":["Curiosity RL gets beta-robust via episode splitting","Splitting episodes makes RL exploration hyperparameter-free","Decouple task and curiosity: RL without beta tuning","Hyper: robust RL exploration without tuning curiosity weight","Reposition before explore: RL that ignores curiosity weight"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof needs the geometric repositioning length to be drawn with a fixed probability $p$ and the value-function weights to be updated only on episodes that skip repositioning entirely, whereas the version tested in the experiments decays $p$ and updates after every step; if that mismatch is not repaired, the stated sample complexity does not cover the algorithm that produced the reported results.","fun_headline_variants_meta":{"raw":{"variants":["Curiosity RL gets beta-robust via episode splitting","Splitting episodes makes RL exploration hyperparameter-free","Decouple task and curiosity: RL without beta tuning","Hyper: robust RL exploration without tuning curiosity weight","Reposition before explore: RL that ignores curiosity weight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1145,"prompt_tokens":875,"completion_tokens":270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":194}},"tokens_in":491,"tokens_out":270,"duration_ms":2981,"temperature":1.0,"reasoning_tokens":194,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:09:10.787178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the neural Hyper algorithm exactly as in the paper but replace per-step updates with updates only on episodes with $L_k=0$ and freeze $p$; if final performance in a sparse-reward maze or locomotion task changes by more than run-to-run variance, the theorem's preconditions are doing real work. Conversely, a single environment with a narrow proper-$\\beta$ window where Hyper still collapses for large $\\beta$ would refute the robustness claim.","supporting_citations":[{"cited_title":"Provably efficient reinforcement learning with linear function approximation","cited_arxiv_id":null,"evidence_quote":"Supplies the linear-MDP value-iteration construction, the realizability and optimism lemmas, and the covering-number bounds on which Theorem B.1 relies."},{"cited_title":"Improved algorithms for linear stochastic bandits","cited_arxiv_id":null,"evidence_quote":"Supplies the self-normalized concentration inequality and determinant-ratio bounds used to control the bonus terms in the regret proof."},{"cited_title":"Sample-optimal parametric q-learning using linearly additive features","cited_arxiv_id":null,"evidence_quote":"Provides the feature-map linear MDP assumption adopted in Assumption 4.1."},{"cited_title":"Self-supervised exploration via disagreement","cited_arxiv_id":null,"evidence_quote":"Supplies the ensemble-disagreement intrinsic reward used as the curiosity module in the empirical comparisons."},{"cited_title":"Decoupling exploration and exploitation in reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the decoupling baseline that Hyper extends with the repositioning phase to avoid distribution shift."}],"review_version":1}