{"id":"82f945e4-7dd1-4938-936c-b34d7183a130","arxiv_id":"2508.21314","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Regularized agent-state-based Q-learning converges almost surely to the fixed point of a regularized MDP induced by the behavioral policy.","lead":"This paper shows that Q-learning for partially observable environments, using an agent state and policy regularization, converges to a known limit defined by the exploration policy. It is useful because many practical reinforcement learning algorithms rely on exactly this combination of ideas.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proof of Theorem 1 is incomplete: X2 convergence relies on a one-sided bound on the regularization noise, and the periodic proof has a phase-index error.","rationale":"The reader's weakest assumption was Assumption 2 (existence and full support of the limiting distribution). That is a legitimate scope limitation, but it does not threaten the truth of the conditional statement; it only limits applicability. The more load-bearing issue is that the main proof, as written, has a technical gap in the X2 component: the conclusion that the sup norm decreases does not follow from the one-sided bound on the regularization noise. This gap is fixable with a standard Lipschitz argument, and the same repair would likely fix the periodic case after correcting the phase-index error. Because the gap is concrete and internal, it justifies the reader's CONDITIONAL verdict; it does not warrant rejection, since the underlying result appears plausible and the flaws are repairable. Hence the reader's verdict stands unchanged, and my agreement is partial because I identify a different primary concern than Assumption 2.","tokens_in":17370,"tokens_out":18752,"duration_ms":178659,"concrete_test":"Re-derive part c of Appendix A using the two-sided Lipschitz inequality |Ω*(q)-Ω*(q')| ≤ ∥q-q'∥∞. Verify that the contraction bound ∥X2_{t+1}∥ ≤ (1 - α_t(1-γ))∥X2_t∥ holds and that Possibility (i)'s contradiction argument goes through. For Appendix B, correct the phase index so that γ∥Q^{Jℓ+1K}_t − Q^{Jℓ+1K}_μ∥ = γ∥Δ^{Jℓ+1K}_t∥, and check whether the L-step product contraction (as in [12]) recovers convergence of all phases simultaneously. If either step fails, the theorem is unproven as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the proof of Theorem 1 (Appendix A, part c), inequality (11c) establishes only U2_t(z,a) ≤ κ∥X2_t∥. The next step concludes X2_{t+1}(z,a) < ∥X2_t∥ and treats this as decay of the sup norm, but sup-norm decrease requires |U2_t(z,a)| ≤ κ∥X2_t∥, or at least a matching lower bound. When X2_t(z,a) is negative, the one-sided bound does not control |X2_{t+1}(z,a)|; the subsequent comparison argument also uses ∥U2_t∥ < κB(0), which is not implied by the one-sided inequality. This is internally repairable via the 1-Lipschitz property of Ω* (for entropy/KL regularization, |Ω*(q)-Ω*(q')| ≤ ∥q-q'∥∞), but as written the proof is invalid. The same defect appears in Appendix B for the periodic case, and there is an additional index error: the text writes γ∥Q^{Jℓ+1K}_t − Q^{Jℓ+1K}_μ∥ = γ∥Δ^ℓ_t∥, whereas the left side equals γ∥Δ^{Jℓ+1K}_t∥. Thus the phase coupling in the contraction argument is not established. These are internal proof gaps, not just citation gaps, and they directly affect the central convergence claims.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies regularized agent-state-based Q-learning (RASQL) in POMDPs, where the Q-table is indexed by a model-free agent state rather than a belief state, and the policy is entropy- or KL-regularized. The main result, Theorem 1, states that under a learning-rate condition (Assumption 1) and a stationarity/full-support condition on the behavior policy (Assumption 2), the RASQL iteration (2) converges almost surely to Q_μ, the unique fixed point of the regularized Bellman operator (4) built from the behavior policy's stationary distribution. Theorem 2 gives an analogous result for a periodic variant, RePASQL. The numerical section illustrates convergence on a small POMDP with the agent state equal to the observation. The paper also correctly notes that the limit depends on the exploration policy and is not guaranteed to be optimal.","tokens_in":17692,"tokens_out":4729,"duration_ms":49367,"significance":"If the result holds, it is a useful contribution: it extends the known convergence theory of agent-state-based Q-learning to entropy/KL-regularized iterations, which are widely used in practice but lack theoretical guarantees in POMDPs. The characterization of the limit via the stationary distribution of the behavior policy is explicit and falsifiable, and the periodic extension is natural. The numerical example is limited but supports the claimed qualitative behavior. However, the proof as written contains load-bearing gaps in the treatment of the regularized error component and in the periodic phase coupling, so the advertised convergence is not established by the current manuscript.","major_comments":[{"comment":"The proof of convergence of X^2_t establishes only a one-sided upper bound: U^2_t(z,a) ≤ κ∥X^2_t∥. From this the manuscript concludes X^2_{t+1}(z,a) < ∥X^2_t∥ and hence ∥X^2_{t+1}∥ < ∥X^2_t∥. This is not valid: for a negative component X^2_t(z,a), the displayed inequality is trivial and does not control |X^2_{t+1}(z,a)|. A large negative U^2_t(z,a) can make the sup norm increase. The later step that (11c) implies ∥U^2_t∥ < κB(0) also requires a two-sided bound. The missing two-sided bound follows from the 1-Lipschitz property of Ω* for entropy/KL regularization, but this is neither stated nor used. Thus the central X^2 convergence argument is incomplete as written.","section":"Appendix A, part c (Eq. (11c))"},{"comment":"In the periodic proof, the display reads γ∥Q^{Jℓ+1K}_t − Q^{Jℓ+1K}_μ∥ = γ∥∆^ℓ_t∥. This is a phase-index error: the left-hand side involves the phase-(ℓ+1) Q-function, while ∆^ℓ_t is the phase-ℓ error. The equality does not hold in general, and it is load-bearing because it is used to connect the regularization error to the current phase's contraction argument. The proof can likely be repaired by working with the sup norm over all phases or by explicitly tracking the phase indices, but the current argument is invalid.","section":"Appendix B, part c"},{"comment":"The convergence of the X^0_t and X^1_t components is asserted by reference to external results ([12], [41], and 'proposition 4 from [12]') rather than proved. The hypotheses of those theorems are not stated, and the verification that they hold here is only partial; in particular, the measurability and step-size conditions for the periodic Markov chain result are not checked in the text. Since these components are essential to both Theorem 1 and Theorem 2, the proof is incomplete at this juncture.","section":"Appendix A, parts a-b; Appendix B, parts a-b"}],"minor_comments":[{"comment":"The phrase 'mild technical conditions' overstates the content of Assumption 2. For a general agent-state update φ and an arbitrary exploration policy μ, existence of a limiting distribution with positive mass on every (z,a) is not automatic. The theorem is conditional on Assumption 2, but the paper does not discuss conditions on φ, μ, or the POMDP under which Assumption 2 holds; this limits the advertised scope.","section":"Section IV, Assumption 2"},{"comment":"Notation: 'Fix (z◦, a◦) ∈ ×Z × A' should read '∈ Z × A'. The extra multiplication symbol appears in both appendices.","section":"Appendix A, part b and Appendix B, part b"},{"comment":"The equation labels in the chain ending in (11c) contain a duplicated '(e)' label; the labels should be renumbered for clarity.","section":"Appendix A, part c"},{"comment":"The numerical experiments are a useful sanity check but report only median and quartiles for a single POMDP with z_t = y_t. Since the theory covers general agent-state updates, an example with a nontrivial φ (e.g., finite-memory or recurrent update) would strengthen the empirical claim.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The proof gaps are serious but appear repairable within the scope of the paper: the X^2 bound can be fixed by proving/using the two-sided Lipschitz property of Ω*, and the periodic phase error can be fixed by using a global norm over phases. The external citations for X^0 and X^1 also need to be substantiated. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The problem is well chosen: agent-state Q-learning with regularization is exactly what people run in practice, and there was no convergence guarantee before. The characterization of the limit—a regularized MDP built from the behavior policy's stationary distribution—is the natural target, and the paper states it cleanly. The authors are also honest that they do not prove optimality; they only show convergence to a policy-dependent fixed point.\n\nThe proof strategy is standard and sensible: split the error into reward, transition, and regularization components. But the details in the regularization component (X2) are not right in the current draft. Inequality (11c) gives an upper bound on U2_t, not a two-sided bound. Then X2_{t+1} = (1-α)X2_t + αU2_t; an upper bound on U2_t doesn't control the sup norm when X2_t can be negative. The conclusion ||X2_{t+1}|| < ||X2_t|| simply doesn't follow. This is repairable: for entropy/KL regularization, Ω* is 1-Lipschitz in sup norm, so |U2_t| ≤ γ||Qt−Qμ||, and the argument would go through if that were stated. But as written, the proof has a gap.\n\nThe periodic case has a more serious index error. In the key bound, they write γ||Q^{Jℓ+1K}_t − Q^{Jℓ+1K}_μ|| = γ||Δℓ_t||. That's false: Δℓ_t is defined as Qℓ_t − Qℓ_μ, not the Jℓ+1K phase, so the left side equals γ||Δ^{Jℓ+1K}_t||. The subsequent decomposition into X^{ℓ,i}_t components is then misaligned. This may be fixable by working with the cycle, but as written Theorem 2's proof does not hold.\n\nThe citations for X0 and X1 are under-specified—conditions of [12] and [41] aren't stated—and Assumption 2's full-support condition is real and can fail for natural agent-state updates. The numerical experiment is one toy POMDP, no code, no learning-rate schedule; illustrative at best.\n\nNet: the central claim is plausible and likely true, but the current manuscript has two concrete gaps in the main proofs. I'd want to see a corrected version before treating Theorem 1 and 2 as established. Still, the paper is worth refereeing; a careful referee could push the authors to fix these issues, and the contribution would then be solid.","headline":"Plausible and likely repairable, but the proof as written has two real gaps: a one-sided bound in X2 and a phase-index error in the periodic extension.","tokens_in":18155,"tokens_out":4495,"would_cite":false,"duration_ms":42871,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent-state regularized Q-learning provably converges in POMDPs","keywords":["POMDP","Q-learning","agent state","policy regularization","entropy regularization","stochastic approximation","periodic policies","convergence analysis"],"falsifier":"Run tabular RASQL on a small finite POMDP where the behavior policy has full support on (z,a), and compare the long-run Q-table to the fixed point Q_mu of (4) computed by value iteration; a mismatch beyond Monte Carlo error would falsify Theorem 1. Equivalently, using a behavior policy whose stationary distribution has zero mass on some reachable (z,a) should make the corresponding entries fail to settle, confirming that the full-support assumption is load-bearing.","tokens_in":17285,"feed_emoji":"🤖","tokens_out":6213,"duration_ms":61833,"temperature":0.7,"pith_summary":"Regularized agent-state-based Q-learning (RASQL) is the common practical recipe of combining a recurrent or frame-stacked \"agent state\" with an entropy or KL regularizer in partially observable environments. This paper proves that, under standard stochastic-approximation and exploration assumptions, the RASQL update converges almost surely to a well-defined limit. The limit is not the optimal POMDP value function; it is the unique fixed point of a regularized Bellman operator built from a fictitious MDP whose rewards and transitions are the true model averaged under the stationary distribution of the fixed exploratory policy. The paper gives the same treatment for a periodic-policy variant (RePASQL), and small numerical experiments match the predicted limits.","feed_headline":"Agent-state regularized Q-learning provably converges in POMDPs","feed_subtitle":"Its limit is a regularized Bellman fixed point averaged under the exploration policy's stationary distribution.","key_machinery":"The load-bearing object is the averaged-MDP construction: from the limiting distribution of the behavior policy, define average rewards r_mu(z,a) and transitions P_mu(z'|z,a), then build the regularized Bellman operator whose unique fixed point Q_mu is the announced limit. The proof decomposes the error Q_t - Q_mu into three components: reward-averaging noise, transition-averaging noise, and the residual from using the current Q-table inside the regularized target. A contraction-type comparison with factor gamma(1+1/C)<1 drives the residual component to zero, while the averaging components vanish by stochastic-approximation results and the law of large numbers for the underlying Markov chain","core_discovery":"On the paper's own terms, the central discovery is Theorem 1: for any POMDP, agent-state update rule, and strongly convex policy regularizer, if the behavior policy makes the joint process (state, observation, agent state, action) converge to a limiting distribution with full support over every (z,a), then the Q-table update converges almost surely to Q_mu, the unique fixed point of the regularized Bellman operator (4). Q_mu depends on that limiting distribution: rewards are averaged as r_mu(z,a) = E[r(S,a)|z] and transitions as P_mu(z'|z,a) = P(Z_{t+1}=z' | z,a) under the stationary process. Theorem 2 extends the same convergence result to periodic policies with period L, with limits Q^ell_","pith_inferences":["A natural reading is that the result exposes an off-policy bias: in regularized agent-state Q-learning, the behavior policy is not just a data-collection device but part of the objective, since the limit depends on its stationary distribution.","The full-support assumption is genuinely load-bearing; for recurrent or frame-stacking agent-state updates, reachability of every (z,a) is not automatic, so exploration design should target the induced occupation distribution, not merely coverage.","The same averaging construction plausibly carries over to continuous agent states or function approximation whenever empirical occupation converges, though the paper does not pursue that extension.","The periodic-policy variant suggests that scheduling exploration as a periodic policy could be used deliberately to shape the limiting Q-table and the final regularized policy."],"forward_implications":["If Theorem 1 is right, entropy- or KL-regularized Q-learning over agent states can be run in any POMDP satisfying the assumptions and will converge almost surely instead of oscillating.","The converged Q-table, and the stochastic policy it induces, are explicitly computable from the model plus the behavior policy, so the limit is testable in simulation.","The limit generally is not the optimal agent-state policy; it inherits the exploration policy's stationary distribution, so changing exploration changes the learned value function.","Regularization yields a stochastic greedy policy at the limit, which can beat deterministic stationary policies in partially observed control, extending a known benefit of agent-state Q-learning.","The periodic variant RePASQL converges to a periodic fixed point, covering policies that change from step to step, with the period-1 case recovering the stationary result."],"supporting_citations":[{"why":"Supplies the theory of regularized MDPs and the Legendre-Fenchel-transform Bellman operator whose fixed point is the convergence target.","marker":"[23]"},{"why":"Provides the periodic agent-state Q-learning analysis and the periodic Markov chain result that the RePASQL proof borrows.","marker":"[12]"},{"why":"Establishes convergence of finite-memory Q-learning for POMDPs, a predecessor argument reused for one error component.","marker":"[11]"},{"why":"Analyzes Q-learning in non-Markovian environments, the setting shared by agent-state updates.","marker":"[14]"},{"why":"Supplies the stochastic approximation theorems used to show the averaging error components vanish.","marker":"[41]"},{"why":"Shows stochastic stationary policies can outperform deterministic stationary policies for POMDPs, motivating the regularization benefit.","marker":"[40]"}],"fun_headline_variants":["Regularized agent-state Q-learning converges in POMDPs","For any POMDP, regularized agent-state Q-learning converges","Agent-state Q-learning with regularization provably converges in POMDPs","Mild conditions suffice: regularized agent-state Q-learning converges in POMDPs","Periodic policies too: regularized agent-state Q-learning converges in POMDPs"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The central assumption is that the fixed behavior policy makes the joint process of state, observation, agent state, and action converge to a limiting distribution that visits every (z,a) pair infinitely often; for arbitrary agent-state updates such as recurrent networks this is not guaranteed and can fail.","fun_headline_variants_meta":{"raw":{"variants":["Regularized agent-state Q-learning converges in POMDPs","For any POMDP, regularized agent-state Q-learning converges","Agent-state Q-learning with regularization provably converges in POMDPs","Mild conditions suffice: regularized agent-state Q-learning converges in POMDPs","Periodic policies too: regularized agent-state Q-learning converges in POMDPs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002696,"raw_usage":{"total_tokens":10110,"prompt_tokens":713,"completion_tokens":9397,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":9308}},"tokens_in":457,"tokens_out":9397,"duration_ms":66455,"temperature":1.0,"reasoning_tokens":9308,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:23:44.233569+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run tabular RASQL on a small finite POMDP where the behavior policy has full support on (z,a), and compare the long-run Q-table to the fixed point Q_mu of (4) computed by value iteration; a mismatch beyond Monte Carlo error would falsify Theorem 1. Equivalently, using a behavior policy whose stationary distribution has zero mass on some reachable (z,a) should make the corresponding entries fail to settle, confirming that the full-support assumption is load-bearing.","supporting_citations":[{"cited_title":"A theory of regularized Markov decision processes,","cited_arxiv_id":null,"evidence_quote":"Supplies the theory of regularized MDPs and the Legendre-Fenchel-transform Bellman operator whose fixed point is the convergence target."},{"cited_title":"Periodic agent-state based Q- learning for POMDPs,","cited_arxiv_id":null,"evidence_quote":"Provides the periodic agent-state Q-learning analysis and the periodic Markov chain result that the RePASQL proof borrows."},{"cited_title":"Convergence of finite memory Q learning for POMDPs and near optimality of learned policies under filter stability,","cited_arxiv_id":null,"evidence_quote":"Establishes convergence of finite-memory Q-learning for POMDPs, a predecessor argument reused for one error component."},{"cited_title":"Reinforcement learning in non-Markovian environments,","cited_arxiv_id":null,"evidence_quote":"Analyzes Q-learning in non-Markovian environments, the setting shared by agent-state updates."},{"cited_title":"Gradient-based algorithms for zeroth- order optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the stochastic approximation theorems used to show the averaging error components vanish."},{"cited_title":"Learning without state- estimation in partially observable Markovian decision processes,","cited_arxiv_id":null,"evidence_quote":"Shows stochastic stationary policies can outperform deterministic stationary policies for POMDPs, motivating the regularization benefit."}],"review_version":1}