{"id":"1872243d-cb4a-4e0d-8877-c78d91166c79","arxiv_id":"2505.13768","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A unified framework proves that augmenting any confidence-based online RL algorithm with offline data yields order-optimal suboptimality-gap and regret bounds, with a new concentrability coefficient that separates the coverage needs of gap versus regret minimization.","lead":"This paper presents a simple hybrid reinforcement learning recipe: take any confidence-based online RL algorithm and feed it both an offline dataset and online data at every step. The authors prove suboptimality-gap and regret bounds with a new coverage coefficient, and show that minimizing regret calls for offline data covering suboptimal policies, not just the optimal one.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1's proof (Appendix B, Theorem B.1, step (b)) asserts U(pi*|D_{t-1}) <= U(pi_t|D_{t-1}) from UCB optimality; this does not follow, so the O~(1/sqrt(N1)) online component of the sub-optimality gap bound is unsupported as written.","rationale":"I agree with the reader's diagnosis. The paper's central contribution is a unified hybrid RL framework with a claimed sub-optimality gap that combines offline and online sample sizes harmonically. The proof of Theorem 3.1 hinges on comparing the uncertainty of the (unknown) optimal policy to that of the policy actually executed by the UCB oracle. The UCB selection rule does not imply such an inequality; it compares upper confidence indices, not uncertainties, and the gap Vhat^{pi_t} - Vhat^{pi*} can be positive. I verified this by constructing a two-arm instance where the offline data covers only the suboptimal arm: the UCB algorithm plays the suboptimal arm because its estimated mean is high and its uncertainty is tiny, while the optimal arm's uncertainty remains large, so the asserted inequality fails. This is not an internal inconsistency in the algorithm's implementation but a derivation gap in the main upper bound; the theorem could be repaired by adding an assumption or by a different argument, but as written the stated bound is not established. The regret bound (Theorem 3.2) is on firmer ground: it uses Uhat(pi_t) directly and combines the eluder sum with the offline concentrability term, so it does not inherit this defect. I also note the abstract's 'outperforms any pure online or offline algorithm' is stronger than what the theorems show even if the proof gap is fixed; the results provide improvement in certain coverage regimes and no asymptotic degradation otherwise, not universal dominance. The experimental section provides illustrative support and checks the qualitative separation, but does not compare against prior hybrid RL methods and ships no code, which is a secondary weakness for an empirical validation of a theory paper. Overall, the reader's CONDITIONAL verdict remains appropriate: the central sub-optimality-gap claim needs a corrected proof or an explicit additional assumption before acceptance. No change to the verdict is needed.","tokens_in":26251,"tokens_out":10175,"duration_ms":97368,"concrete_test":"Construct a 2-arm linear contextual bandit with d=2, phi(arm1)=e1, phi(arm2)=e2, theta*=e1, and arm2 mean reward 0.99; let the offline dataset D0 consist of N0=100 pulls of arm2 only. Run Algorithm 1 with LinUCB (Eqs. 5-6) for N1=1. With beta=1, Vhat(arm1)=0, Uhat(arm1)=1, Vhat(arm2)=0.99, Uhat(arm2)=1/sqrt(101)≈0.099, so UCB chooses arm2 while Uhat(arm1)>Uhat(arm2). This instance satisfies the paper's assumptions, so the asserted inequality in step (b) is false. Then check whether Theorem B.1 can be re-derived without step (b); if the O~(1/sqrt N1) term cannot be recovered, the theorem needs an additional assumption (e.g., Vhat^{pi_t} <= Vhat^{pi*} or a coverage condition on pi*).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is in the proof of Theorem 3.1 (Appendix B, Theorem B.1). To show Uhat(pi* | D0∪D_N1) = O(C_Alg/sqrt(N1)), the proof averages Uhat(pi* | D_{t-1}) and invokes step (b): Uhat(pi* | D_{t-1}) <= Uhat(pi_t | D_{t-1}), claimed to follow from optimality of pi_t. But pi_t = argmax_pi (Vhat^pi + Uhat^pi) only implies Vhat^{pi_t} + Uhat^{pi_t} >= Vhat^{pi*} + Uhat^{pi*}, equivalently Uhat(pi*) - Uhat(pi_t) <= Vhat^{pi_t} - Vhat^{pi*}. The right-hand side is not guaranteed non-positive; Vhat^{pi_t} can exceed Vhat^{pi*}. Neither the confidence-interval definition (Section 3.1) nor the Eluder-type condition (Definition 3.3, which only bounds Uhat of executed pi_t) prevents this. If step (b) fails, the average bound on Uhat(pi*) collapses to the offline-only term O(C_Alg sqrt(C(pi*|rho)/N0)), and the harmonic-mean combination O~(1/sqrt(N0/C+N1)) in Theorem 3.1 is not established. Since the abstract's claim of universal improvement over pure online learning relies on the online component, this is central. Theorem 3.2's regret proof does not use step (b) and is not affected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a generic hybrid reinforcement-learning framework in which any confidence-based online RL algorithm is run with the offline dataset included at every online update. The main theoretical claims are: (i) the sub-optimality gap of the final pessimistic policy is O~(C_Alg / sqrt(N0/C(pi*|rho) + N1)); and (ii) the online regret is O~(C_Alg sqrt(N1) sqrt(N1/(N0/C(pi^-|rho)+N1))), where C(pi|rho) is a new concentrability coefficient defined as the squared ratio of uncertainty levels. The paper also proves matching lower bounds and instantiates the framework in tabular MDPs and linear contextual bandits, with experiments supporting the predicted separation between coverage requirements for gap minimization and regret minimization.","tokens_in":26585,"tokens_out":18049,"duration_ms":177276,"significance":"If the upper bounds are established, the paper gives an appealing unified template: offline data acts as an effective additional sample budget, and the results are order-optimal with respect to the stated lower bounds. The regret theorem, the lower bounds, and the coverage-separation insight are valuable and appear coherent. The paper also benefits from concrete instantiations with tabular MDPs and linear bandits and from experimental validation. However, the significance is currently conditional: the proof of the sub-optimality gap theorem contains an unjustified inference, and the new concentrability coefficient is defined in terms of the very uncertainty quantities being bounded, so its interpretation as an independent coverage measure needs clarification.","major_comments":[{"comment":"The step labelled (b) in the proof of Theorem B.1, namely Uhat(pi*|D_{t-1}) <= Uhat(pi_t|D_{t-1}) 'due to the optimality of pi_t', is not justified. Since pi_t maximizes Vhat + Uhat, UCB optimality only implies Vhat(pi*) + Uhat(pi*) <= Vhat(pi_t) + Uhat(pi_t), equivalently Uhat(pi*) - Uhat(pi_t) <= Vhat(pi_t) - Vhat(pi*). The right-hand side need not be non-positive, and neither the confidence-interval definition in Section 3.1 nor the Eluder-type condition in Definition 3.3 (which bounds only the executed policies pi_t) rules this out. This step is what converts the average of Uhat(pi*|D_{t-1}) into the Eluder bound, so the O~(C_Alg/sqrt(N1)) online component of Theorem 3.1 is not established as written. The regret proof in Theorem B.2 does not use this step and appears unaffected. A possible repair is to use UCB selection to show V* - V(pi_t) <= 2 Uhat(pi_t|D_{t-1}) and then compare the final LCB against the LCB of a visited policy, but the authors should supply such an argument or add an explicit assumption.","section":"Appendix B, Theorem B.1"},{"comment":"The offline component of the sub-optimality gap bound is largely definitional. Since C(pi*|rho) = (U(pi*)/U(rho))^2 and U(rho) = O~(1/sqrt(N0)), the bound O~(C_Alg sqrt(C(pi*|rho)/N0)) is just O~(C_Alg U(pi*)), i.e., the target policy's own offline estimation error. Thus the theorem's offline term partly restates the new coefficient rather than relating C(pi|rho) to an independent data-coverage quantity such as a density-ratio concentrability. The paper should either prove concrete inequalities comparing C(pi|rho) with standard concentrability coefficients in the examples (the short arguments in Section 4 only give inequalities in the opposite direction) or explicitly state that C(pi|rho) is a normalization of the target policy's minimax estimation error. Without this, the claim that the paper introduces a new and meaningful concentrability coefficient is weakened.","section":"Definition 3.2 and proof of Theorem 3.1"},{"comment":"The proofs in Theorem B.1 and Theorem B.2 use the monotonicity property that Uhat(pi|D) <= Uhat(pi|D') whenever D' is a subset of D, but this property is not stated among the assumptions on the oracle algorithm. For a generic 'confidence-based algorithm', enlarging the dataset does not automatically shrink the confidence bound; the definition of the oracle in Section 3.1 only guarantees that Uhat is an upper confidence bound. This monotonicity should be made an explicit assumption of the framework, or verified for each instantiation, since both main upper bounds rely on it.","section":"Section 3.1 and Appendix B"}],"minor_comments":[{"comment":"The stated sub-optimality gap uses C(pi^{-epsilon}|rho) in the denominator, but for gap minimization the relevant coefficient is C(pi*|rho); the same symbol appears to be copied from the regret statement.","section":"Corollary 4.2"},{"comment":"The uncertainty level is denoted both U_M*(pi) and U_M(pi) at different places; please use one notation consistently and define all arguments in the minimax definition of Alg0.","section":"Definition 3.1"},{"comment":"The remark writes C(pi^{-epsilon}|rho) = 0, but Definition 3.2 states that C(pi|rho) is in [1, infinity]; the boundary case should be reconciled or the definition should allow zero.","section":"Remark 3.2"},{"comment":"The lower-bound formulas are typeset ambiguously: e.g., Omega(1/sqrt(N0/C(pi*|rho)+N1)) should be written with clear parentheses so that the denominator is N0/C(pi*|rho) + N1, not 1/sqrt(N0) divided by C plus N1.","section":"Theorem 5.1"},{"comment":"The text says Figure 1(a) and (f) list C(pi^{-epsilon}|rho), but the tables in the figure appear to list C(pi*|rho); please make the caption and text consistent.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The central gap in the proof of Theorem 3.1 is real but appears fixable: the standard optimism argument V* - V(pi_t) <= 2 Uhat(pi_t|D_{t-1}), combined with a comparison between the final LCB and the LCB of past executed policies, may recover the online component without the invalid monotonicity of Uhat(pi*) against Uhat(pi_t). The definitional character of C(pi|rho) should also be addressed, as it affects how the contribution is framed. I would be willing to re-review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a genuinely useful framework paper and I would send it to referees, but the proof of Theorem 3.1 has a load-bearing gap. The regret bound and the separation insight are likely to be influential; the sub-optimality gap upper bound, as written, is not established.\n\nWhat is new and good. The reduction is clean: take any confidence-based online RL algorithm, feed it the union of offline data and all online trajectories so far, select policies by UCB, and for gap minimization output the LCB policy. The new concentrability coefficient C(pi|rho) = (U(pi)/U(rho))^2 based on a minimax uncertainty level is well-defined and results in tighter bounds for tabular MDPs and linear bandits. The main conceptual contribution is the separation between coverage of pi* for gap minimization and coverage of sub-optimal policies for regret minimization. That is a real insight and is supported by the lower bounds. The regret upper bound (Theorem 3.2) appears sound: the proof only needs the eluder condition on the executed policies, and the calculations check out. The lower bounds are coherent and match the regret upper bound. The literature coverage is fine, with relevant concurrent work cited.\n\nThe soft spot. In Appendix B, Theorem B.1, step (b) asserts U(pi*|D_{t-1}) <= U(pi_t|D_{t-1}) 'due to the optimality of pi_t'. This does not follow from UCB maximization. pi_t maximizes Vhat + Uhat, so the inequality only gives Vhat(pi_t)+Uhat(pi_t) >= Vhat(pi*)+Uhat(pi*). Nothing prevents Uhat(pi*) from being larger than Uhat(pi_t). Neither the confidence-ball definition nor the eluder-type condition (which bounds the sum of Uhat of the executed policies) helps. Without step (b), the averaged Uhat(pi*) term does not vanish as O(1/sqrt(N1)), and Theorem 3.1 reduces to an offline-only bound. That is critical because the abstract's claim of universal improvement over pure online learning rests on this step.\n\nMinor issues. The abstract says 'outperforms any pure online or offline algorithm alone', but the theorems only show no asymptotic degradation and improvement when the offline data has good coverage. The experiments compare only against pure UCB, not prior hybrid methods, and no code is shipped. Both are secondary to the proof gap.\n\nWho this is for: researchers in hybrid RL theory who will find the framework and the regret/coverage separation useful. I would give it a serious referee, with the clear expectation that the authors either supply a valid argument for the missing step or soften Theorem 3.1 accordingly. The paper's core idea is good enough to warrant that effort.","headline":"Clean and insightful framework for hybrid RL, but the sub-optimality gap upper bound has a proof gap that makes the main theorem unproven as written.","tokens_in":27150,"tokens_out":4484,"would_cite":true,"duration_ms":38194,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68Q32","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that augmenting any confidence-based online RL algorithm with an offline dataset yields order-optimal sub-optimality gap and regret, and that the right offline data differs by objective.","keywords":["hybrid reinforcement learning","offline-to-online RL","confidence-based algorithms","concentrability coefficient","uncertainty level","regret minimization","sub-optimality gap","linear contextual bandits"],"falsifier":"Run Algorithm 1 on a two-state tabular MDP or a two-arm linear bandit with an offline dataset chosen so that after some round $t$ the UCB-selected policy has $\\hat{U}^{\\pi_t}_{\\mathrm{Alg}}(D_{t-1}) < \\hat{U}^{\\pi^*}_{\\mathrm{Alg}}(D_{t-1})$ while both uncertainty bounds remain valid upper bounds. That single inequality contradicts the proof's step (b) in Theorem B.1; if it occurs with non-negligible probability, the claimed $\\tilde{O}(1/\\sqrt{N_1})$ online component of the sub-optimality gap bound does not follow from the stated assumptions.","tokens_in":26006,"feed_emoji":"🎯","tokens_out":17187,"duration_ms":147717,"temperature":0.7,"pith_summary":"Hybrid reinforcement learning—combining a fixed offline dataset with online interaction—has many ad hoc recipes; this paper claims one generic recipe already attains the best possible rates. The recipe is to take any confidence-based online algorithm that outputs a value estimate and a high-probability uncertainty bound, rerun it on the pooled dataset $D_0 \\cup D_{t-1}$ at every episode, play the upper-confidence-bound policy, and, for gap minimization, return the lower-confidence-bound policy at the end. The proved sub-optimality gap is $\\tilde{O}(C_{\\mathrm{Alg}}/\\sqrt{N_0/C(\\pi^*|\\rho)+N_1})$ and the regret is $\\tilde{O}(C_{\\mathrm{Alg}}\\sqrt{N_1}\\sqrt{N_1/(N_0/C(\\pi^-_\\epsilon|\\rho)+N_1)})$, with matching lower bounds on a two-arm bandit instance, so offline data helps exactly according to a new concentrability coefficient $C(\\pi|\\rho)$ and never asymptotically hurts. A practically useful consequence is a separation: gap minimization wants offline data that covers the optimal policy, while regret minimization wants coverage of sub-optimal policies. The authors instantiate the framework for tabular MDPs and linear contextual bandits and report experiments consistent with these predictions.","feed_headline":"Offline data plus online learning hits the optimal rate","feed_subtitle":"A unified recipe augments any confidence-based RL algorithm with logged data and matches lower bounds for both gap and regret.","key_machinery":"The load-bearing identity is the harmonic-mean shortcut $\\min\\{C(\\pi^*|\\rho)/N_0, 1/N_1\\} \\le 2/(N_0/C(\\pi^*|\\rho)+N_1)$ used at the end of Theorem 3.1's proof, which lets the offline uncertainty term and the online Eluder term be merged into a single effective-sample-size rate. The objects feeding that identity are the uncertainty level $U_{M^*}(\\pi)$, the concentrability coefficient $C(\\pi|\\rho)$, and the Eluder-type condition $\\sum_{t=1}^{N_1}\\hat{U}^{\\pi_t}_{\\mathrm{Alg}}(D_{t-1})^2 \\le C_{\\mathrm{Alg}}^2$, which plays the role of a pigeonhole or elliptical-potential lemma controlling how much exploration can cost. Together they express offline and online statistical difficulty in the same units, so the two bounds can be compared and combined.","core_discovery":"On its own terms, the paper's central claim is that confidence-based online RL needs no new algorithmic machinery to benefit from offline data: initialize with $D_0$, keep it in the training set, act optimistically, and finish pessimistically. Defining $U_{M^*}(\\pi)$ as the minimax expected error of the best oracle in estimating $V^\\pi_{M^*}$ from $N_0$ draws under the behavior policy $\\rho$, and $C(\\pi|\\rho)=(U_{M^*}(\\pi)/U_{M^*}(\\rho))^2$, the paper proves (Theorem 3.1) that the final policy's gap is $\\tilde{O}(C_{\\mathrm{Alg}}/\\sqrt{N_0/C(\\pi^*|\\rho)+N_1})$ and (Theorem 3.2) that the cumulative regret is $\\tilde{O}(C_{\\mathrm{Alg}}\\sqrt{N_1}\\sqrt{N_1/(N_0/C(\\pi^-_\\epsilon|\\rho)+N_1)})$ whenever the oracle satisfies an Eluder-type condition. Theorem 5.1 gives matching $\\tilde{\\Omega}$ lower bounds on a two-arm linear contextual bandit, establishing order-optimality. The conceptual discovery is the coefficient: $N_0$ offline trajectories under $\\rho$ act like $N_0/C(\\pi|\\rho)$ virtual online episodes, and which coefficient matters flips depending on whether the learning goal is a final near-optimal policy or low cumulative regret.","pith_inferences":["A natural extension the paper does not pursue is to optimize the offline data-collection policy under a budget: the two coefficients make the tradeoff explicit, and a mixture policy that covers both $\\pi^*$ and a spread of $\\epsilon$-suboptimal policies should dominate either extreme; this is testable in the paper's own linear-bandit setup.","The proof gap in the appendix suggests the rate may still be true but needs one more argument: replacing the asserted inequality $\\hat{U}(\\pi^*|D_{t-1})\\le \\hat{U}(\\pi_t|D_{t-1})$ with a variance-weighted or exploration-conditioned bound would likely preserve the claimed order while making the proof self-contained.","For practitioners using deep RL, where exact confidence sets are unavailable, the framework suggests a concrete heuristic: train the value network and an epistemic-uncertainty estimate on $D_0 \\cup D_{t-1}$ at every round, act by UCB, and output by LCB; the rate guarantee would not transfer formally, but the effective-sample-size interpretation gives a principled reason to expect similar qualitati","The lower-bound instance is a two-arm linear bandit, so the order-optimality claim has not been tested against richer MDP structure; constructing a tabular MDP instance where the gap and regret coefficients genuinely differ would strengthen the separation beyond the current simulation evidence."],"forward_implications":["A behavior policy $\\rho$ with good coverage of the optimal policy makes the hybrid sub-optimality gap behave like an online learner with $N_0/C(\\pi^*|\\rho)+N_1$ episodes, so the gap strictly improves over both pure online and pure offline whenever the coefficient is finite.","For regret, the relevant coefficient is $C(\\pi^-_\\epsilon|\\rho)$, the worst concentrability among policies that are at least $\\epsilon$ sub-optimal; offline data that explores sub-optimal policies yields a constant-factor speed-up over pure online, while a dataset of only optimal trajectories yields no asymptotic regret improvement.","The two lower bounds show that the dependence on $N_0$, $N_1$, and the concentrability coefficients is order-tight, so the simple plug-in design cannot be improved asymptotically by more elaborate hybrid algorithms.","In tabular MDPs the bounds become $\\tilde{O}(\\sqrt{H^4|X||A|/(N_0/C(\\pi^*|\\rho)+N_1)})$ for the gap and the corresponding regret; in linear contextual bandits they become $\\tilde{O}(d\\sqrt{1/(N_0/C+N_1)})$ and $\\tilde{O}(d\\sqrt{N_1}\\sqrt{N_1/(N_0/C+N_1)})$, giving explicit dimension dependence.","The separation result translates to data-collection guidance: for a final-policy objective, collect offline data concentrated on good actions; for a regret objective, collect a broad exploratory dataset, and a single dataset cannot simultaneously optimize both unless it covers both kinds of policies."],"supporting_citations":[{"why":"It supplies the tabular MDP estimator and the $\\beta/\\sqrt{N}$ uncertainty bound whose squared sum satisfies the Eluder-type condition, instantiating the framework in tabular MDPs.","marker":"[Azar et al., 2017]"},{"why":"It supplies the Lin-UCB confidence ellipsoid and the elliptical potential lemma used to define the uncertainty function and verify the Eluder-type condition in linear contextual bandits.","marker":"[Abbasi-Yadkori et al., 2011]"},{"why":"It provides the two-arm linear bandit hard instance and the estimation-error-to-regret reduction on which the matching lower bounds of Theorem 5.1 are built.","marker":"[He et al., 2022]"},{"why":"It is the earlier hybrid RL analysis with a single-policy concentrability coefficient against which the paper compares its new coefficient and claims tighter rates.","marker":"[Xie et al., 2021b]"},{"why":"It is the earlier hybrid RL algorithm for tabular MDPs with partial coverage that the unified framework simplifies and matches or improves.","marker":"[Li et al., 2023]"},{"why":"It is the earlier hybrid RL algorithm for linear MDPs whose bounds and assumptions the paper compares with in its main table.","marker":"[Wagenmaker and Pacchiano, 2023]"},{"why":"It is the earlier regret analysis for hybrid RL with general function approximation that serves as the regret baseline in the paper's comparison.","marker":"[Tan and Xu, 2024]"},{"why":"It analyzes multi-armed bandits with offline data, and the paper notes its regret bound order-wise matches theirs when specialized to the multi-armed bandit case.","marker":"[Cheung and Lyu, 2024]"}],"fun_headline_variants":["Unified hybrid RL matches lower bounds for gap and regret","Offline data boosts online RL to optimal rates","One recipe: add offline data to any confidence-based RL","Hybrid RL achieves optimal sub-optimality and regret"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of the sub-optimality gap bound assumes that, whenever the algorithm picks the UCB-optimistic policy, that policy's estimation uncertainty about the optimal policy is at least as large as the optimal policy's own uncertainty, i.e. $\\hat{U}(\\pi^*|D_{t-1})\\le \\hat{U}(\\pi_t|D_{t-1})$; the argument derives this from the policy being UCB-optimal, but maximizing an estimated value plus uncertainty bound does not by itself force that inequality.","fun_headline_variants_meta":{"raw":{"variants":["Unified hybrid RL matches lower bounds for gap and regret","Offline data boosts online RL to optimal rates","One recipe: add offline data to any confidence-based RL","Hybrid RL achieves optimal sub-optimality and regret"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2742,"prompt_tokens":1119,"completion_tokens":1623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":735,"completion_tokens_details":{"reasoning_tokens":1558}},"tokens_in":735,"tokens_out":1623,"duration_ms":12335,"temperature":1.0,"reasoning_tokens":1558,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:12:08.659244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on a two-state tabular MDP or a two-arm linear bandit with an offline dataset chosen so that after some round $t$ the UCB-selected policy has $\\hat{U}^{\\pi_t}_{\\mathrm{Alg}}(D_{t-1}) < \\hat{U}^{\\pi^*}_{\\mathrm{Alg}}(D_{t-1})$ while both uncertainty bounds remain valid upper bounds. That single inequality contradicts the proof's step (b) in Theorem B.1; if it occurs with non-negligible probability, the claimed $\\tilde{O}(1/\\sqrt{N_1})$ online component of the sub-optimality gap bound does not follow from the stated assumptions.","supporting_citations":[{"cited_title":"Leveraging offline data in online reinforcement learning","cited_arxiv_id":null,"evidence_quote":"It is the earlier hybrid RL algorithm for linear MDPs whose bounds and assumptions the paper compares with in its main table."}],"review_version":1}