{"id":"4ae96004-0b21-4b77-8dad-4c58d94efd93","arxiv_id":"2507.21705","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BellNet learns graph-filter coefficients for truncated policy iteration and approximates optimal policies in fewer steps than classical DP on grid-world tasks.","lead":"This paper introduces BellNet, a network that \"unrolls\" policy iteration into a few learnable layers, treating transition probabilities as edges of a graph. In small grid-world tests, BellNet reaches near-optimal policies in fewer iteration rounds than classical value and policy iteration and transfers to a modified grid without retraining.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BellNet's central speedup claim depends on the unproven convergence of the Eq. (8) self-referential Bellman-error update to the optimal value function; absent that, the Fig. 2 results may only show per-instance curve fitting on one grid-world.","rationale":"The mathematical core is sound: Eqs. (5)-(7) correctly rewrite truncated policy evaluation as graph-filter recursions, Proposition 1 is a valid Cayley-Hamilton degree reduction, and the softmax cascade is a faithful unrolling of policy iteration at infinite depth. The released code and internally consistent trends are real supporting evidence. The load-bearing weakness is the training objective: Eq. (8) minimizes a self-referential Bellman error against the current model's policy, and no argument shows the projected sequence reaches the optimal value function. Because the model class is finite-depth and finite-order, the projection can bias the fixed point, so the Fig. 2 speedups may be instance-specific rather than a general DP acceleration property. The transfer experiment is also weak since the mirrored grid is graph-isomorphic to the training grid, which only demonstrates permutation equivariance; but the Eq. (8) gap is more central because it underlies the primary claim. A randomized multi-MDP stress test with multiple qbar initializations would settle whether the learned operator approximates Bellman optimality or merely fits one environment. This supports the reader's CONDITIONAL verdict without changing it.","tokens_in":8787,"tokens_out":10704,"duration_ms":131900,"concrete_test":"Train BellNet with the paper's settings on 20 randomly generated grid-world MDPs of the same size (random cliff, start, and goal placements), each from 10 random qbar initializations. Report the median normalized Bellman optimality residual ||qhat - (r + gamma P_{Pi_hat} qhat)||/||qhat|| and the Eq. (9) error against a policy-iteration ground truth. If the residual is not near zero or the final error varies materially with qbar, the Eq. (8) objective is not converging to q* and the headline claim must be restricted to per-instance fitting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV defines the learning rule as H[n+1] = argmin_H || r + gamma P_{Pi[n]} q[n] - Phi(qbar, H) ||^2, with {q[n], Pi[n]} = Phi(qbar, H[n]). The target is the Bellman target of the current softmax policy Pi[n], not the Bellman optimality target r + gamma P_{v*} q* from Eq. (3); the two coincide only at the optimal fixed point. Because Phi is restricted to L+1 layers and order-K graph filters, exact minimization generally does not land in the model's range, so each update is a projected version of policy/value iteration. The paper asserts (Section IV, item (c)) that BellNet converges to the optimal VF and policy regardless of qbar, but gives no contraction, monotonicity, or fixed-point analysis. If the projected operator has a suboptimal fixed point, the architecture is fitting the training MDP and the 'fraction of iterations' claim does not carry to other problems. This gap is load-bearing because every reported advantage in Fig. 2 is mediated by the learned coefficients from this objective.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BellNet, an unrolled dynamic-programming architecture in which truncated policy evaluation is represented as a cascade of graph filters with learnable coefficients and row-wise softmax policy improvement. The authors show that the fixed-policy Bellman recursion corresponds to a polynomial of the transition-probability matrix (Prop. 1), define a training objective that minimizes the Bellman error between the model output and a self-generated target, and report experiments on a cliff-walking grid comparing BellNet with truncated value and policy iteration and testing transfer to a mirrored grid. The paper claims that BellNet approximates optimal policies in a fraction of the iterations required by classical DP methods.","tokens_in":8977,"tokens_out":7253,"duration_ms":91602,"significance":"If the main claims held, the paper would make a useful conceptual contribution: expressing unrolled DP as a nonlinear graph-filter cascade (Eq. (7) and Prop. 1) gives a compact, transferable representation and connects DP to a mature GSP toolbox. The authors provide reproducible code, identify the bias term from truncated evaluation, and correctly note that value iteration corresponds to K=1 and policy iteration to L=K=∞. The significance is currently conditional, however, because the central speedup claim rests on an unproven convergence property of the self-referential training objective and on a single grid-world experiment whose transfer test is an isomorphic symmetry.","major_comments":[{"comment":"The training loss minimizes ||r + γP_{Π[n]}q[n] − Φ(¯q,H)||², where {q[n], Π[n]} = Φ(¯q, H[n]). The target is a Bellman target of the current model policy, not the Bellman optimality target from Eq. (3); the two coincide only at the optimal fixed point. Since Φ has finite depth L and filter order K, the Bellman optimality fixed point generally lies outside Φ's range, so each update is a projected iteration with no proved contraction, monotonicity, or fixed-point property. The assertion in Section IV(c) that BellNet is 'initialized with an arbitrary VF ¯q and trained to converge to the optimal VF and policy regardless of ¯q' is therefore unsupported. Because every numerical advantage in Fig. 2 is mediated by coefficients learned from this objective, this missing convergence/fixed-point analysis is load-bearing. Please provide a theorem with explicit assumptions, or substantially reframe the claim and add an empirical convergence study over many q¯ initializations.","section":"Section IV, Eq. (8), item (c)"},{"comment":"The transfer experiment uses a mirrored grid. Mirroring a grid-world induces a permutation of states and actions, and graph filters are permutation-equivariant by construction (Section IV, Transferability). The successful transfer in Fig. 2c is therefore essentially a check of equivariance, not evidence that BellNet generalizes to genuinely different MDP topologies, sizes, or reward structures. To support contribution C3, evaluate on non-isomorphic environments (e.g., different grid dimensions, modified cliff positions, or changed rewards) and report error distributions for those environments.","section":"Section V, Test case 3 and Fig. 3"},{"comment":"The 'fraction of iterations' comparison is not fully specified. Each BellNet layer applies K+1 matrix-polynomial (graph filter) operations plus a softmax, while the baselines are counted in 'policy improvement steps' or 'policy evaluation updates'; this does not compare like with like. In addition, BellNet is trained on the same grid used in evaluation, and the training cost is not included in the claimed speedup. Please state the per-iteration operation counts, clarify whether reported numbers are inference-only comparisons, and either include wall-clock times or give a complexity table.","section":"Section V, Fig. 2a–2c and Eq. (9)"}],"minor_comments":[{"comment":"The equality rank(Pπ) = rank(P) = |S| requires additional assumptions (e.g., P has full column rank and the policy-induced factor has full row rank); as written it does not follow for an arbitrary MDP. The Cayley–Hamilton bound K ≤ |S||A| is sufficient for the proposition, so the diagonalizable claim should be stated with the needed assumptions.","section":"Proposition 1 proof"},{"comment":"The notation 'argmin_H' conflicts with item (a), which says the coefficients are updated via gradient descent. Please specify whether H[n+1] is the exact minimizer or one (or several) gradient steps on the loss.","section":"Section IV, Eq. (8)"},{"comment":"The term h_{K+1}(Pπ)^{K+1}q(0) uses an extra coefficient, but the dimensions and the scalar/vector status of the coefficients are not defined consistently across Eqs. (6)–(8). Please define all coefficient arrays explicitly.","section":"Section IV, Eq. (7)"},{"comment":"The caption does not say what is randomized over the 15 realizations (initial VF, training seed, or both), and the baselines appear deterministic; please clarify whether the interquartile range applies only to BellNet or also to the classical baselines.","section":"Fig. 2 caption"},{"comment":"The figure is not self-contained: it does not define the meaning of the two parallel graph-filter branches, the role of q(0), or the notation Π(l), q(l). Please expand the caption so the schematic can be read without returning to the text.","section":"Fig. 1 caption and legend"}],"recommendation":"major_revision","confidential_remarks":"The paper is a compact, ideas-first contribution whose central claim is broader than its evidence. The missing convergence analysis for Eq. (8) and the isomorphic transfer experiment are the two main risks; both are addressable with additional analysis/experiments rather than being fundamental contradictions. If the authors can reframe the claims as empirical and provide non-isomorphic transfer results plus convergence checks, the contribution could be suitable for a venue that accepts preliminary results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It's a short paper that unrolls policy iteration as a cascade of nonlinear graph filters. The mathematical core is right: truncated policy evaluation is a polynomial of the transition matrix, and the Cayley-Hamilton reparameterization (Prop 1) correctly bounds the filter order. That's a genuinely nice way to frame DP, and it's new relative to VIN/GVIN/GNIV, which stick to value iteration. The code is on GitHub, experiments report medians and IQRs across 15 realizations, and the weight-sharing variant is a sensible touch.\n\nThe soft spot is exactly the one the stress-test note flags. The training objective in Eq. (8) uses the Bellman target of the current softmax policy, not the optimal Bellman target. The paper states as fact that BellNet converges to the optimal VF regardless of q_bar, but there's no contraction, monotonicity, or fixed-point argument. Because the model is a finite truncated cascade, each step is a projected version of policy iteration, and it's entirely possible the projected operator has a suboptimal fixed point. If that's the case, the Fig. 2 speedups are per-instance curve fitting. The transfer experiment doesn't help: the mirrored grid is the same underlying MDP with relabeled start and cliff positions, so it tests equivariance, not generalization to a genuinely different environment. And there's no comparison against VIN/GVIN/GNIV, which are the obvious unrolled-VI baselines.\n\nThat said, the paper is honest about being preliminary, and the identities in Eqs. (5)-(7) are correct. If the authors can either prove convergence of the Bellman-error training or demonstrate the method on a real MDP with a non-trivially different transition graph, it becomes a solid contribution. The current version is a promising idea with a load-bearing gap.\n\nWorth sending to a serious referee? Yes. The architecture is novel, the math is sound, and the limitations are fixable. A referee could ask for the convergence analysis and better baselines without demanding a rewrite of the core insight. I'd bring it to a reading group, but mainly to discuss whether the training objective can be trusted.","headline":"BellNet gives a clean graph-filter unrolling of policy iteration with correct math, but its core speedup claim rests on an unproven self-referential training objective and a transfer test that only checks symmetry.","tokens_in":9549,"tokens_out":2320,"would_cite":false,"duration_ms":25018,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C40","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"BellNet unrolls dynamic programming into a cascade of learnable nonlinear graph filters, approximating optimal value functions and policies in a fraction of the iterations needed by classical policy and value iteration, and transferring…","keywords":["BellNet","algorithm unrolling","graph signal processing","graph filters","dynamic programming","policy iteration","Bellman error","transferability"],"falsifier":"Train BellNet on the cliff-walking grid as described, then evaluate it on a structurally different MDP such as a larger grid with different cliff placements or a random sparse transition graph with long effective horizons; if the normalized value error in Eq. (9) stays high or grows with problem size, the few-iteration and transferability claims fail beyond the demonstrated environment. A second check: run the Eq. (8) optimization from several diverse random value-function initializations and verify that the resulting Bellman residual consistently decreases toward zero; if some initializations fail to reduce the residual, the 'regardless of q-bar' claim is false.","tokens_in":8513,"feed_emoji":"🧠","tokens_out":3348,"duration_ms":44338,"temperature":0.7,"pith_summary":"The paper claims that the iterative steps of dynamic programming, specifically policy iteration, can be unrolled into a trainable deep network it calls BellNet, where each layer is a nonlinear graph filter acting on the rewards of a Markov decision process. By interpreting the transition probability matrix as the adjacency matrix of a weighted directed graph, the authors show that policy evaluation is a matrix polynomial filter and that policy improvement is a softmax nonlinearity. They train the filter coefficients to minimize the Bellman error from random value-function initializations, and report that on a cliff-walking grid the learned model matches or beats truncated policy and value iteration with significantly fewer layers. They also report that the trained model transfers to a mirrored grid environment without retraining.","feed_headline":"BellNet finds optimal DP policies with far fewer iterations","feed_subtitle":"A cascade of learned graph filters unrolls policy iteration and transfers across grid worlds without retraining.","key_machinery":"The key object is the nonlinear graph-filter cascade: each BellNet layer computes a reduced-order polynomial of the current policy's transition matrix applied to the reward, plus a bias term from the previous value estimate, then applies a softmax to obtain the next policy. This relies on the identity that policy evaluation is a graph filter, $q^{(k+1)} = \\sum_{j=0}^{k-1} \\gamma^j (P_\\pi)^j r + \\gamma^k (P_\\pi)^k q^{(0)}$, and on the Cayley–Hamilton argument that the infinite filter has an equivalent finite degree $K \\le |S||A|$ (or $K \\le |S|$ when $P_\\pi$ is diagonalizable). The learnable coefficients $h^{(l)}$ adapt this filter to minimize Bellman error, and weight sharing across layers enables reuse beyond training depth for transfer.","core_discovery":"The central claim is that an unrolled, parametric version of policy iteration — BellNet — can be represented as a cascade of nonlinear graph filters, and that this representation yields an effective DP solver with fewer required iterations than classical methods. Concretely, the paper derives that the value function for a fixed policy is a finite-order polynomial of the transition matrix applied to the reward vector, $q^\\pi = \\sum_{j=0}^{K} \\bar{h}_j (P_\\pi)^j r$, with the order bounded by the Cayley–Hamilton theorem, so each exact policy-evaluation step is a graph filter. BellNet stacks $L+1$ such filters, each followed by a row-wise softmax, and learns the filter coefficients by minimizing the squared Bellman error of its own previous output, $\\| r + \\gamma P_{\\Pi[n]} q[n] - \\Phi(\\bar{q}, H) \\|_2^2$. The paper further claims that this learned model outperforms truncated policy iteration in recovering the optimal value function on a grid-world, and that weight-shared variants transfer across environment modifications without retraining.","pith_inferences":["The self-referential Bellman-error training in Eq. (8) resembles a fixed-point learning problem; if it converges, BellNet could be viewed as a learned amortized DP solver that compresses many classical iterations into one forward pass, an interpretation the paper hints at but does not formalize.","The transfer result to a mirrored grid suggests the learned filters capture structural features of grid navigation (avoid cliffs, head toward goal) rather than memorizing one reward map; a testable extension would be to transfer across different grid sizes or obstacle layouts, which the paper does not run.","Annealing the softmax temperature $\\tau$ during inference could recover exactly the greedy max of policy improvement, potentially making BellNet's policy provably improving while retaining differentiability during training; this is an extension the paper leaves implicit.","The graph-filter formulation may extend beyond finite tabular MDPs to graph-structured state spaces where the transition matrix is sparse, since the polynomial filter only needs matrix-vector products — a scalability path the paper does not explore."],"forward_implications":["If BellNet's few-iteration claim holds beyond the cliff-walking grid, DP problems with large state-action spaces could be solved with substantially fewer repeated sweeps, cutting inference cost.","The graph-filter view gives an explicit handle on complexity: the filter order $K$ and unrolling depth $L$ directly trade off approximation quality versus computation, potentially allowing users to pick a desired accuracy regime.","Weight-shared BellNet blocks can be reused beyond their training depth, suggesting a path to scalable inference on larger or modified MDPs where retraining is prohibitive.","Because the method uses the known transition matrix rather than sampled data, it sits between model-based DP and model-free RL, offering a data-free way to learn acceleration across related MDPs.","Finite-order exactness from the Cayley–Hamilton bound means the representation is not a heuristic truncation: an exact policy evaluation is recoverable with a sufficiently high filter order, unifying value and policy iteration as special cases."],"supporting_citations":[{"why":"Supplies the fixed-point theorem guaranteeing that iterating the Bellman evaluation converges to the true value function, which is the sequence BellNet unrolls.","marker":"[3]"},{"why":"Defines graph filters as matrix polynomials of the adjacency matrix, the core reparameterization used for policy evaluation.","marker":"[10]"},{"why":"Provides the Cayley–Hamilton theorem used in Proposition 1 to bound the filter order by the state-action space size.","marker":"[33]"},{"why":"Introduces Bellman's optimality equations, the fixed-point system that BellNet is trained to satisfy via Bellman error.","marker":"[23]"},{"why":"Gives the algorithm-unrolling methodology that maps iterative DP steps into a layered parametric network.","marker":"[8]"},{"why":"Establishes transferability properties of graph filters and graph neural networks that the paper invokes to explain BellNet's cross-environment generalization.","marker":"[18]"}],"fun_headline_variants":["BellNet unrolls DP as graph filters, cuts iterations","Graph-filter BellNet learns DP policies faster","Unrolled DP becomes learnable graph filters in BellNet","BellNet: DP via graph filters, fewer steps, transferable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training objective in Eq. (8) minimizes the Bellman error of the network's own previous outputs, and the paper assumes this self-referential objective drives the model to the optimal value function and policy from any random initialization, but no convergence proof is supplied.","fun_headline_variants_meta":{"raw":{"variants":["BellNet unrolls DP as graph filters, cuts iterations","Graph-filter BellNet learns DP policies faster","Unrolled DP becomes learnable graph filters in BellNet","BellNet: DP via graph filters, fewer steps, transferable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000508,"raw_usage":{"total_tokens":2485,"prompt_tokens":963,"completion_tokens":1522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1455}},"tokens_in":579,"tokens_out":1522,"duration_ms":14766,"temperature":1.0,"reasoning_tokens":1455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:27:25.645614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train BellNet on the cliff-walking grid as described, then evaluate it on a structurally different MDP such as a larger grid with different cliff placements or a random sparse transition graph with long effective horizons; if the normalized value error in Eq. (9) stays high or grows with problem size, the few-iteration and transferability claims fail beyond the demonstrated environment. A second check: run the Eq. (8) optimization from several diverse random value-function initializations and verify that the resulting Bellman residual consistently decreases toward zero; if some initializations fail to reduce the residual, the 'regardless of q-bar' claim is false.","supporting_citations":[{"cited_title":"Bertsekas, Dynamic Programming and Optimal Control: Volume I , vol","cited_arxiv_id":null,"evidence_quote":"Supplies the fixed-point theorem guaranteeing that iterating the Bellman evaluation converges to the true value function, which is the sequence BellNet unrolls."},{"cited_title":"Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing,","cited_arxiv_id":null,"evidence_quote":"Gives the algorithm-unrolling methodology that maps iterative DP steps into a layered parametric network."}],"review_version":1}