Corrected heavy-ball Q-learning with convergence and acceleration guarantees is derived via switched linear system and joint spectral radius analysis, extended to linear function approximation.
Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
abstract
Q-value iteration (Q-VI) is usually analyzed through the \(\gamma\)-contraction of the Bellman operator. This argument proves convergence to \(Q^*\), but it gives only a coarse account of when the induced greedy policy becomes optimal. We study discounted Q-VI as a switching system and focus on the practically optimal solution set (POSS), the set of \(Q\)-functions whose tie-broken greedy policies are optimal. The main result shows that Q-VI reaches the optimal action class in finite time by entering an invariant tube around \(\mathcal X_1=Q^*+\operatorname{span}(\mathbf 1)\), which is contained in the POSS. For every \(\varepsilon>0\), the distance to \(\mathcal X_1\) satisfies an exponential bound with rate \((\bar\rho+\varepsilon)^k\), where \(\bar\rho\) is the joint spectral radius of the projected switching family restricted to directions transverse to \(\mathcal X_1\). When \(\bar\rho<\gamma\), this transverse convergence is faster than the classical contraction rate. The analysis separates fast policy identification from the subsequent convergence to \(Q^*\), which may still be governed by the all-ones mode. We also give spectral and graph-theoretic conditions under which the strict inequality \(\bar\rho<\gamma\) holds or fails.
years
2026 2verdicts
UNVERDICTED 2representative citing papers
Deflated Q-value iteration admits a projected switching-system model whose joint spectral radius can be strictly smaller than the discount factor, yielding a sharper convergence characterization while leaving the greedy policy sequence unchanged.
citing papers explorer
-
Heavy-Ball Q-Learning with Residual Weighting Correction
Corrected heavy-ball Q-learning with convergence and acceleration guarantees is derived via switched linear system and joint spectral radius analysis, extended to linear function approximation.
-
Switching-Geometry Analysis of Deflated Q-Value Iteration
Deflated Q-value iteration admits a projected switching-system model whose joint spectral radius can be strictly smaller than the discount factor, yielding a sharper convergence characterization while leaving the greedy policy sequence unchanged.