Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Unrolling Dynamic Programming via Graph Filters

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BellNet unrolls dynamic programming into a cascade of learnable nonlinear graph filters, approximating optimal value functions and policies in a fraction of the iterations needed by classical policy and value iteration, and transferring…

desk verdict BellNet gives a clean graph-filter unrolling of policy iteration with correct math, but its core speedup claim rests on an unproven self-referential training objective and a transfer test that only checks symmetry. read the letter →

arxiv 2507.21705 v1 pith:BIMJPN5U submitted 2025-07-29 cs.AI

Sergio Rozada , Samuel Rey , Gonzalo Mateos , Antonio G. Marques This is my paper · ORCID
classification cs.AI MSC 90C4068T07
keywords BellNetalgorithmunrollinggraphsignalprocessingfiltersdynamicprogrammingpolicyiterationBellmanerrortransferability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the iterative steps of dynamic programming, specifically policy iteration, can be unrolled into a trainable deep network it calls BellNet, where each layer is a nonlinear graph filter acting on the rewards of a Markov decision process. By interpreting the transition probability matrix as the adjacency matrix of a weighted directed graph, the authors show that policy evaluation is a matrix polynomial filter and that policy improvement is a softmax nonlinearity. They train the filter coefficients to minimize the Bellman error from random value-function initializations, and report that on a cliff-walking grid the learned model matches or beats truncated policy and value iteration with significantly fewer layers. They also report that the trained model transfers to a mirrored grid environment without retraining.

What carries the argument

The key object is the nonlinear graph-filter cascade: each BellNet layer computes a reduced-order polynomial of the current policy's transition matrix applied to the reward, plus a bias term from the previous value estimate, then applies a softmax to obtain the next policy. This relies on the identity that policy evaluation is a graph filter, $q^{(k+1)} = \sum_{j=0}^{k-1} \gamma^j (P_\pi)^j r + \gamma^k (P_\pi)^k q^{(0)}$, and on the Cayley–Hamilton argument that the infinite filter has an equivalent finite degree $K \le |S||A|$ (or $K \le |S|$ when $P_\pi$ is diagonalizable). The learnable coefficients $h^{(l)}$ adapt this filter to minimize Bellman error, and weight sharing across layers enables reuse beyond training depth for transfer.

What would settle it

Train BellNet on the cliff-walking grid as described, then evaluate it on a structurally different MDP such as a larger grid with different cliff placements or a random sparse transition graph with long effective horizons; if the normalized value error in Eq. (9) stays high or grows with problem size, the few-iteration and transferability claims fail beyond the demonstrated environment. A second check: run the Eq. (8) optimization from several diverse random value-function initializations and verify that the resulting Bellman residual consistently decreases toward zero; if some initializations fail to reduce the residual, the 'regardless of q-bar' claim is false.

Watch

Extended reading notes

Core claim

The central claim is that an unrolled, parametric version of policy iteration — BellNet — can be represented as a cascade of nonlinear graph filters, and that this representation yields an effective DP solver with fewer required iterations than classical methods. Concretely, the paper derives that the value function for a fixed policy is a finite-order polynomial of the transition matrix applied to the reward vector, $q^\pi = \sum_{j=0}^{K} \bar{h}_j (P_\pi)^j r$, with the order bounded by the Cayley–Hamilton theorem, so each exact policy-evaluation step is a graph filter. BellNet stacks $L+1$ such filters, each followed by a row-wise softmax, and learns the filter coefficients by minimizing the squared Bellman error of its own previous output, $\| r + \gamma P_{\Pi[n]} q[n] - \Phi(\bar{q}, H) \|_2^2$. The paper further claims that this learned model outperforms truncated policy iteration in recovering the optimal value function on a grid-world, and that weight-shared variants transfer across environment modifications without retraining.

Load-bearing premise

The training objective in Eq. (8) minimizes the Bellman error of the network's own previous outputs, and the paper assumes this self-referential objective drives the model to the optimal value function and policy from any random initialization, but no convergence proof is supplied.

Editorial extensions

If this is right

  • If BellNet's few-iteration claim holds beyond the cliff-walking grid, DP problems with large state-action spaces could be solved with substantially fewer repeated sweeps, cutting inference cost.
  • The graph-filter view gives an explicit handle on complexity: the filter order $K$ and unrolling depth $L$ directly trade off approximation quality versus computation, potentially allowing users to pick a desired accuracy regime.
  • Weight-shared BellNet blocks can be reused beyond their training depth, suggesting a path to scalable inference on larger or modified MDPs where retraining is prohibitive.
  • Because the method uses the known transition matrix rather than sampled data, it sits between model-based DP and model-free RL, offering a data-free way to learn acceleration across related MDPs.
  • Finite-order exactness from the Cayley–Hamilton bound means the representation is not a heuristic truncation: an exact policy evaluation is recoverable with a sufficiently high filter order, unifying value and policy iteration as special cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The self-referential Bellman-error training in Eq. (8) resembles a fixed-point learning problem; if it converges, BellNet could be viewed as a learned amortized DP solver that compresses many classical iterations into one forward pass, an interpretation the paper hints at but does not formalize.
  • The transfer result to a mirrored grid suggests the learned filters capture structural features of grid navigation (avoid cliffs, head toward goal) rather than memorizing one reward map; a testable extension would be to transfer across different grid sizes or obstacle layouts, which the paper does not run.
  • Annealing the softmax temperature $\tau$ during inference could recover exactly the greedy max of policy improvement, potentially making BellNet's policy provably improving while retaining differentiability during training; this is an extension the paper leaves implicit.
  • The graph-filter formulation may extend beyond finite tabular MDPs to graph-structured state spaces where the transition matrix is sparse, since the polynomial filter only needs matrix-vector products — a scalability path the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes BellNet, an unrolled dynamic-programming architecture in which truncated policy evaluation is represented as a cascade of graph filters with learnable coefficients and row-wise softmax policy improvement. The authors show that the fixed-policy Bellman recursion corresponds to a polynomial of the transition-probability matrix (Prop. 1), define a training objective that minimizes the Bellman error between the model output and a self-generated target, and report experiments on a cliff-walking grid comparing BellNet with truncated value and policy iteration and testing transfer to a mirrored grid. The paper claims that BellNet approximates optimal policies in a fraction of the iterations required by classical DP methods.

Significance. If the main claims held, the paper would make a useful conceptual contribution: expressing unrolled DP as a nonlinear graph-filter cascade (Eq. (7) and Prop. 1) gives a compact, transferable representation and connects DP to a mature GSP toolbox. The authors provide reproducible code, identify the bias term from truncated evaluation, and correctly note that value iteration corresponds to K=1 and policy iteration to L=K=∞. The significance is currently conditional, however, because the central speedup claim rests on an unproven convergence property of the self-referential training objective and on a single grid-world experiment whose transfer test is an isomorphic symmetry.

major comments (3)
  1. [Section IV, Eq. (8), item (c)] The training loss minimizes ||r + γP_{Π[n]}q[n] − Φ(¯q,H)||², where {q[n], Π[n]} = Φ(¯q, H[n]). The target is a Bellman target of the current model policy, not the Bellman optimality target from Eq. (3); the two coincide only at the optimal fixed point. Since Φ has finite depth L and filter order K, the Bellman optimality fixed point generally lies outside Φ's range, so each update is a projected iteration with no proved contraction, monotonicity, or fixed-point property. The assertion in Section IV(c) that BellNet is 'initialized with an arbitrary VF ¯q and trained to converge to the optimal VF and policy regardless of ¯q' is therefore unsupported. Because every numerical advantage in Fig. 2 is mediated by coefficients learned from this objective, this missing convergence/fixed-point analysis is load-bearing. Please provide a theorem with explicit assumptions, or substantially reframe the claim and add an empirical convergence study over many q¯ initializations.
  2. [Section V, Test case 3 and Fig. 3] The transfer experiment uses a mirrored grid. Mirroring a grid-world induces a permutation of states and actions, and graph filters are permutation-equivariant by construction (Section IV, Transferability). The successful transfer in Fig. 2c is therefore essentially a check of equivariance, not evidence that BellNet generalizes to genuinely different MDP topologies, sizes, or reward structures. To support contribution C3, evaluate on non-isomorphic environments (e.g., different grid dimensions, modified cliff positions, or changed rewards) and report error distributions for those environments.
  3. [Section V, Fig. 2a–2c and Eq. (9)] The 'fraction of iterations' comparison is not fully specified. Each BellNet layer applies K+1 matrix-polynomial (graph filter) operations plus a softmax, while the baselines are counted in 'policy improvement steps' or 'policy evaluation updates'; this does not compare like with like. In addition, BellNet is trained on the same grid used in evaluation, and the training cost is not included in the claimed speedup. Please state the per-iteration operation counts, clarify whether reported numbers are inference-only comparisons, and either include wall-clock times or give a complexity table.
minor comments (5)
  1. [Proposition 1 proof] The equality rank(Pπ) = rank(P) = |S| requires additional assumptions (e.g., P has full column rank and the policy-induced factor has full row rank); as written it does not follow for an arbitrary MDP. The Cayley–Hamilton bound K ≤ |S||A| is sufficient for the proposition, so the diagonalizable claim should be stated with the needed assumptions.
  2. [Section IV, Eq. (8)] The notation 'argmin_H' conflicts with item (a), which says the coefficients are updated via gradient descent. Please specify whether H[n+1] is the exact minimizer or one (or several) gradient steps on the loss.
  3. [Section IV, Eq. (7)] The term h_{K+1}(Pπ)^{K+1}q(0) uses an extra coefficient, but the dimensions and the scalar/vector status of the coefficients are not defined consistently across Eqs. (6)–(8). Please define all coefficient arrays explicitly.
  4. [Fig. 2 caption] The caption does not say what is randomized over the 15 realizations (initial VF, training seed, or both), and the baselines appear deterministic; please clarify whether the interquartile range applies only to BellNet or also to the classical baselines.
  5. [Fig. 1 caption and legend] The figure is not self-contained: it does not define the meaning of the two parallel graph-filter branches, the role of q(0), or the notation Π(l), q(l). Please expand the caption so the schematic can be read without returning to the text.

Circularity Check

0 steps flagged · score 2.0 of 10

BellNet's derivation is self-contained: the Eq. (8) target is an environment-driven Bellman target, and the speedup and transfer claims are grounded by independent comparisons against q* and a no-retraining experiment, so no prediction reduces by construction to fitted inputs.

full rationale

No circular step is present in the paper's derivation chain. The training objective in Eq. (8) is self-referential in the sense that the target r + gamma P_{Pi[n]} q[n] uses the current model outputs q[n] and Pi[n], but that target is not the model output itself; it is the Bellman target of the current softmax policy computed from the known MDP quantities r and P. Minimizing the discrepancy drives the model toward a fixed point of that policy's Bellman operator, not toward a quantity defined by the fitted parameters. The evaluation metric in Eq. (9) compares against q* computed independently via policy iteration, and Test case 3 evaluates the learned coefficients on a modified grid without retraining, which provides external grounding outside the fitted values. Proposition 1, the finite-order graph-filter representation of the value function, is a standard Cayley-Hamilton result proved in the paper and is not a self-citation or an input-output tautology. The self-citations in the reference list, such as [21], [24], [25], [29], and [30], are background references on graph-filter design and low-rank reinforcement learning; none of them carries the load of the speedup or transferability claims. The assertion in Section IV(c) that BellNet converges to the optimal value function and policy regardless of the initial qbar is unproven, but that is a correctness gap rather than circularity: even if the projected Bellman-error update had a suboptimal fixed point, the failure would be an empirical/convergence issue, not a reduction of the reported predictions to the training inputs. No equation in the paper defines the predicted quantity in terms of the fitted parameters, and no fitted parameter is renamed as a prediction. The mild self-reference in Eq. (8) is standard bootstrapping for TD-style learning and does not by construction force the reported results, so the paper warrants a low circularity score rather than an accusation of circular derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The architecture rests on standard DP fixed-point theory, the Cayley-Hamilton theorem, and spectral radius facts about stochastic matrices. The two ad hoc premises are the convergence of the self-referential Bellman-error training and the adequacy of softmax policy improvement. No new physical or mathematical entities are introduced.

free parameters (3)
  • BellNet filter coefficients h^{(l)}_j = learned, values not reported
    Optimized by minimizing Bellman error in Eq. (8); these coefficients are the core approximation mechanism.
  • Softmax temperature tau = not specified
    Controls how closely the softmax policy approximates greedy improvement; no value, schedule, or sensitivity analysis is given.
  • Unrolling depth L and filter order K = L in {3,5,10}, K in {5,10,15}
    Chosen per experiment; the complexity and fidelity tradeoff claimed by the paper depends directly on these manually selected hyperparameters.
assumptions (5)
  • standard math Cayley-Hamilton theorem
    Used in Proposition 1 to bound the degree of the finite graph-filter representation of the value function.
  • standard math Convergence of the Neumann series (I - gamma P_pi)^{-1} for gamma < 1
    Needed for Eq. (6); follows from the spectral radius of the stochastic matrix P_pi and gamma < 1.
  • standard math Classical DP convergence results for policy and value iteration
    Background theory cited from [3]; the paper relies on these results to define the fixed-point problems being unrolled.
  • ad hoc to paper Self-referential Bellman-error minimization converges to the Bellman optimality fixed point
    Assumed in Section IV Learning; no proof is provided, and the loss target uses the model's own output.
  • ad hoc to paper Softmax policy improvement preserves sufficient optimality
    The greedy max in Eq. (2) is replaced with a row-wise softmax; the paper does not analyze how temperature affects the attained fixed point.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unrolling Dynamic Programming via Graph Filters." pith.science (2026). https://pith.science/paper/BIMJPN5U

@misc{pith2026250721705,
  author       = {Pith},
  title        = {Pith review of: Unrolling Dynamic Programming via Graph Filters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BIMJPN5U}},
  note         = {Machine review of arXiv:2507.21705}
}
read the original abstract

Dynamic programming (DP) is a fundamental tool used across many engineering fields. The main goal of DP is to solve Bellman's optimality equations for a given Markov decision process (MDP). Standard methods like policy iteration exploit the fixed-point nature of these equations to solve them iteratively. However, these algorithms can be computationally expensive when the state-action space is large or when the problem involves long-term dependencies. Here we propose a new approach that unrolls and truncates policy iterations into a learnable parametric model dubbed BellNet, which we train to minimize the so-termed Bellman error from random value function initializations. Viewing the transition probability matrix of the MDP as the adjacency of a weighted directed graph, we draw insights from graph signal processing to interpret (and compactly re-parameterize) BellNet as a cascade of nonlinear graph filters. This fresh look facilitates a concise, transferable, and unifying representation of policy and value iteration, with an explicit handle on complexity during inference. Preliminary experiments conducted in a grid-like environment demonstrate that BellNet can effectively approximate optimal policies in a fraction of the iterations required by classical methods.

Figures

Figures reproduced from arXiv: 2507.21705 by the authors.

Figure 1
Figure 1. BellNet schematic. A cascade of learnable graph filters and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Evaluation of BellNet across different scenarios. We report the median error of the estimated [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Cliff walking environment (top) and its mirrored version [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 31 canonical work pages

  1. [1]

    E. V . Denardo, Dynamic Programming: Models and Applications , Courier Corporation, 2012

    work page 2012

  2. [2]

    M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, 2014

    work page 2014

  3. [3]

    Bertsekas, Dynamic Programming and Optimal Control: Volume I , vol

    D. Bertsekas, Dynamic Programming and Optimal Control: Volume I , vol. 4, Athena Scientific, 2012

    work page 2012

  4. [4]

    Learning fast approximations of sparse coding,

    K. Gregor and Y . LeCun, “Learning fast approximations of sparse coding,” in Int. Conf. Mach. Learning , 2010, pp. 399–406

    work page 2010

  5. [5]

    Graph unrolling networks: Interpretable neural networks for graph signal denoising,

    S. Chen, Y . C. Eldar, and L. Zhao, “Graph unrolling networks: Interpretable neural networks for graph signal denoising,” IEEE Trans. Signal Process., vol. 69, pp. 3699–3713, 2021

    work page 2021

  6. [6]

    Graph signal processing: Overview, challenges, and ap- plications,

    A. Ortega, P. Frossard, J. Kova ˇcevi´c, J. M. F. Moura, and P. Van- dergheynst, “Graph signal processing: Overview, challenges, and ap- plications,” Proc. IEEE, vol. 106, no. 5, pp. 808–828, 2018

    2018

  7. [7]

    Graph signal processing: History, development, impact, and outlook,

    G. Leus, A. G. Marques, J. M. F. Moura, A. Ortega, and D. I. Shuman, “Graph signal processing: History, development, impact, and outlook,” IEEE Signal Process. Mag. , vol. 40, no. 4, pp. 49–60, 2023

    work page 2023

  8. [8]

    Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing,

    V . Monga, Y . Li, and Y . C. Eldar, “Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing,” IEEE Signal Process. Mag., vol. 38, no. 2, pp. 18–44, 2021

    work page 2021

Show all 40 references
  1. [9]

    Robust stochastically- descending unrolled networks,

    S. Hadou, N. NaderiAlizadeh, and A. Ribeiro, “Robust stochastically- descending unrolled networks,” IEEE Trans. Signal Process. , vol. 72, pp. 5484–5499, 2024

    2024

  2. [10]

    Graph filters for signal processing and machine learning on graphs,

    E. Isufi, F. Gama, D. I. Shuman, and S. Segarra, “Graph filters for signal processing and machine learning on graphs,” IEEE Trans. Signal Process., vol. 72, pp. 4745–4781, 2024

    2024

  3. [11]

    Value iteration networks,

    A. Tamar, Y . Wu, G. Thomas, S. Levine, and P. Abbeel, “Value iteration networks,” in Conf. Neural Inform. Process. Syst. , 2016, vol. 29

    2016

  4. [12]

    Generalized value iteration networks: Life beyond lattices,

    S. Niu, S. Chen, H. Guo, C. Targonski, M. Smith, and J. Kova ˇcevi´c, “Generalized value iteration networks: Life beyond lattices,” in AAAI Conf. Artificial Intell. , 2018, vol. 32

    2018

  5. [13]

    Graph neural induction of value iteration,

    A. Deac, P.-L. Bacon, and J. Tang, “Graph neural induction of value iteration,” arXiv preprint arXiv:2009.12604 , 2020

    2009 arXiv

  6. [14]

    On solving MDPs with large state space: Exploitation of policy structures and spectral properties,

    L. Liu, A. Chattopadhyay, and U. Mitra, “On solving MDPs with large state space: Exploitation of policy structures and spectral properties,” IEEE Trans. Commun. , vol. 67, no. 6, pp. 4151–4165, 2019

    2019

  7. [15]

    Policy sampling and interpolation for wireless networks: A graph signal processing approach,

    L. Liu and U. Mitra, “Policy sampling and interpolation for wireless networks: A graph signal processing approach,” in Proc. IEEE Global Commun. Conf., 2019, pp. 1–6

    2019

  8. [16]

    Reduced dimension policy iteration for wireless network control via multiscale analysis,

    M. Levorato, S. Narang, U. Mitra, and A. Ortega, “Reduced dimension policy iteration for wireless network control via multiscale analysis,” in Proc. IEEE Global Commun. Conf. , 2012, pp. 3886–3892

    2012

  9. [17]

    Stability properties of graph neural networks,

    F. Gama, J. Bruna, and A. Ribeiro, “Stability properties of graph neural networks,” IEEE Trans. Signal Process., vol. 68, pp. 5680–5695, 2020

    2020

  10. [18]

    Graph neural networks: Architec- tures, stability, and transferability,

    L. Ruiz, F. Gama, and A. Ribeiro, “Graph neural networks: Architec- tures, stability, and transferability,” Proc. IEEE , vol. 109, no. 5, pp. 660–682, 2021

    2021

  11. [19]

    Trans- ferability of spectral graph convolutional neural networks,

    R. Levie, W. Huang, L. Bucci, M. Bronstein, and G. Kutyniok, “Trans- ferability of spectral graph convolutional neural networks,” J. Mach. Learning Res., vol. 22, no. 272, pp. 1–59, 2021

    2021

  12. [20]

    Learning by transference: Training graph neural networks on growing graphs,

    J. Cervino, L. Ruiz, and A. Ribeiro, “Learning by transference: Training graph neural networks on growing graphs,” IEEE Trans. Signal Process., vol. 71, pp. 233–247, 2023

    2023

  13. [21]

    Re- designing graph filter-based GNNs to relax the homophily assumption,

    S. Rey, M. Navarro, V . M. Tenorio, S. Segarra, and A. G. Marques, “Re- designing graph filter-based GNNs to relax the homophily assumption,” in IEEE Int. Conf. Acoust., Speech and Signal Process. , 2025, pp. 1–5

    2025

  14. [22]

    A manifold perspective on the statistical generalization of graph neural networks,

    Z. Wang, J. Cervino, and A. Ribeiro, “A manifold perspective on the statistical generalization of graph neural networks,” in Int. Conf. Learn. Representations, 2025

    2025

  15. [23]

    Dynamic programming,

    R. Bellman, “Dynamic programming,” Science, vol. 153, no. 3731, pp. 34–37, 1966

    1966

  16. [24]

    Robust graph filter identification and graph denoising from signal observations,

    S. Rey, V . M. Tenorio, and A. G. Marques, “Robust graph filter identification and graph denoising from signal observations,” IEEE Trans. Signal Process., vol. 71, pp. 3651–3666, 2023

    2023

  17. [25]

    Optimal graph-filter design and applications to distributed linear network operators,

    S. Segarra, A. G. Marques, and A. Ribeiro, “Optimal graph-filter design and applications to distributed linear network operators,” IEEE Trans. Signal Process., vol. 65, no. 15, pp. 4117–4131, 2017

    2017

  18. [26]

    Least-squares policy iteration,

    M. G. Lagoudakis and R. Parr, “Least-squares policy iteration,” J. Mach. Learning Res., vol. 4, no. Dec, pp. 1107–1149, 2003

    2003

  19. [27]

    A tutorial on linear function approximators for dynamic programming and reinforcement learning,

    A. Geramifard et al., “A tutorial on linear function approximators for dynamic programming and reinforcement learning,” Foundations and Swerves® in Machine Learning , vol. 6, no. 4, pp. 375–451, 2013

    2013

  20. [28]

    FLAMBE: Structural complexity and representation learning of low rank MDPs,

    A. Agarwal, S. Kakade, A. Krishnamurthy, and W. Sun, “FLAMBE: Structural complexity and representation learning of low rank MDPs,” in Conf. Neural Inform. Process. Syst. , 2020, vol. 33, pp. 20095–20107

    2020

  21. [29]

    Tensor and matrix low- rank value-function approximation in reinforcement learning,

    S. Rozada, S. Paternain, and A. G. Marques, “Tensor and matrix low- rank value-function approximation in reinforcement learning,” IEEE Trans. Signal Process., vol. 72, pp. 1634–1649, 2024

    2024

  22. [30]

    Solving finite-horizon MDPs via low-rank tensors,

    S. Rozada, J. L. Orejuela, and A. G. Marques, “Solving finite-horizon MDPs via low-rank tensors,” arXiv preprint arXiv:2501.10598 , 2025

    2025 arXiv

  23. [31]

    Kernel-based reinforcement learning,

    D. Ormoneit and S. Sen, “Kernel-based reinforcement learning,” Mach. Learn., vol. 49, no. 2, pp. 161–178, 2002

    2002

  24. [32]

    Nonparametric Bellman mappings for value iteration in distributed reinforcement learning,

    Y . Akiyama and K. Slavakis, “Nonparametric Bellman mappings for value iteration in distributed reinforcement learning,” arXiv preprint arXiv:2503.16192, 2025

    2025

  25. [33]

    R. A. Horn and C. R. Johnson, Matrix Analysis, Cambridge University Press, 2012

    2012

  26. [34]

    R. S. Sutton, Reinforcement Learning: An Introduction , A Bradford Book, 2018

    2018

  27. [35]

    Algorithmic survey of parametric value function approximation,

    M. Geist and O. Pietquin, “Algorithmic survey of parametric value function approximation,” IEEE Trans. Neural Netw. Learning Syst., vol. 24, no. 6, pp. 845–867, 2013

    2013

  28. [36]

    A generalized Kalman filter for fixed point approximation and efficient temporal-difference learning,

    D. Choi and B. Van Roy, “A generalized Kalman filter for fixed point approximation and efficient temporal-difference learning,”Discrete Event Dyn. Syst. , vol. 16, no. 2, pp. 207–239, 2006

    2006

  29. [37]

    TD convergence: An optimization perspective,

    K. Asadi, S. Sabach, Y . Liu, O. Gottesman, and R. Fakoor, “TD convergence: An optimization perspective,” in Conf. Neural Inform. Process. Syst., 2023, vol. 36, pp. 49169–49186

    2023

  30. [38]

    Simplifying neural networks by soft weight-sharing,

    Steven J. Nowlan and Geoffrey E. Hinton, “Simplifying neural networks by soft weight-sharing,” Neural Comput. , vol. 4, no. 4, pp. 473–493, 1992

    1992

  31. [39]

    OpenAI Gym,

    G. Brockman et al., “OpenAI Gym,” arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  32. [40]

    Gymnasium: A standard interface for reinforcement learning environments,

    M. Towers et al., “Gymnasium: A standard interface for reinforcement learning environments,” arXiv preprint arXiv:2407.17032 , 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.