Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes NO-SAC, a soft actor-critic controller whose actor and critic networks read features from a DeepONet pretrained on backstepping controllers, and argues this removes the slow-variation delay assumption while stabilizing…

desk verdict A genuinely new architecture—DeepONet as a feature extractor inside SAC—but the headline claim about eliminating the delay assumption outruns the evidence; needs seeds, training details, and an ablation. read the letter →

arxiv 2501.18201 v1 pith:6PPLGDJR submitted 2025-01-30 cs.AI cs.SYeess.SY

Jiaqi Hu , Jie Qi , Jing Zhang This is my paper · ORCID
classification cs.AIcs.SYeess.SY MSC 93C2068T07
keywords first-orderhyperbolicPDEspatially-varyingdelaybacksteppingcontrolDeepONetneuraloperatorsoftactor-criticreinforcementlearningboundary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a reinforcement-learning controller can stabilize an unstable first-order hyperbolic PDE whose state is delayed by a spatially-varying function $\tau(x)$, without the slow-variation assumption $|\tau'(x)| < 1$ that the analytic backstepping design requires. The proposed method, NO-SAC, trains a DeepONet to approximate the backstepping controller operator and then embeds copies of that network as feature extractors in the actor and critic networks of a soft actor-critic agent. In simulations on the delay $\tau(x) = 0.7 + 0.3\cos(4\arccos x)$, which violates the assumption, NO-SAC converges faster and eliminates steady-state error compared to plain SAC. When the delay satisfies the assumption, both RL controllers give smaller overshoot and shorter settling times than the analytic backstepping controller. The significance would be a general recipe: analytical PDE control knowledge, converted into an operator approximator, can warm-start model-free RL and widen the class of delay functions that can be controlled.

What carries the argument

The mechanism is the composition of a DeepONet and an SAC agent. The DeepONet is a branch-and-trunk neural operator trained to approximate the backstepping boundary-control map $U(\tau, v, u)$; its branch network encodes the three functions sampled on a $21 \times 21$ grid and its trunk network encodes coordinates on the same grid, with the Cartesian product of the two outputs giving a 441-dimensional feature vector. Five copies of this trained operator are inserted into the policy network and the two action-value networks, so the RL agent's decisions are conditioned on features extracted from the analytical backstepping law. To turn the non-Markovian delayed evolution into an MDP, the state is augmented with the transport-delay coordinate $u(x,r,t)$ that carries the delayed information, with the reward split into a running term and a terminal term.

What would settle it

Train the DeepONet only on delays satisfying $|\tau'(x)| < 1$, then run NO-SAC on a delay with $|\tau'(x)|$ significantly greater than 1, for example $\tau(x) = 0.1 + 1.9x$, and record the closed-loop $L^2$ norm; if the state fails to converge or the NO-SAC reward collapses while SAC still learns, the claim that the delay assumption is eliminated would be contradicted.

Watch

Extended reading notes

Core claim

The central claim is that the delay assumption needed for backstepping can be removed by giving the RL agent a neural-operator representation of the backstepping solution rather than requiring the delay to belong to the admissible class $\mathcal{D} = \{\tau \in C^2[0,1] : \tau(x) > 0 \text{ for all } x \text{, and if } \tau(x) < x \text{ then } |\tau'(x)| < 1\}$. The paper constructs a DeepONet that learns the controller operator $U(\tau, v, u)$, mapping the delay function, the current state $v(x,t)$, and the delayed state $u(x,r,t)$ to the boundary input, and then uses copies of the trained DeepONet as feature-extraction layers in the SAC actor and critic. The resulting NO-SAC policy is evaluated on a delay that violates the assumption and is reported to stabilize the closed loop faster than baseline SAC, and on an admissible delay where it compares favorably with the analytic controller in transient performance.

Load-bearing premise

The load-bearing premise is that the DeepONet, pretrained on backstepping controllers generated for delays in the admissible class $\mathcal{D}$, generalizes to delay functions outside that class; the paper does not report the training distribution of $\tau$ or any out-of-distribution test beyond the single violating delay used in simulation.

Editorial extensions

If this is right

  • Controllers for this class of delayed PDEs can be obtained without checking $|\tau'(x)| < 1$, provided the operator network generalizes beyond its training set.
  • Analytic backstepping laws can be packaged as pretrained feature extractors, giving RL a warm start that reduces steady-state error and training time.
  • In the admissible-delay regime, the learned policy can match or beat the analytic controller in transient performance, suggesting RL can refine rather than only replace analytical designs.
  • The augmented-state MDP formulation makes delayed boundary-control problems amenable to standard off-policy RL algorithms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The strongest test of the no-assumption claim is a systematic sweep over delays far outside $\mathcal{D}$ with $|\tau'(x)| > 1$; the paper reports only a single violating delay, so the claim's breadth remains untested.
  • Editorial inference: The same operator-pretraining scheme should transfer to other backstepping designs, such as actuator or sensor delay compensation, because the DeepONet only needs to approximate the controller map rather than the PDE coefficients.
  • Editorial inference: A natural testable extension is to vary the $21 \times 21$ spatial discretization and the zero-order-hold update rate, which would reveal how much of the observed gain comes from the operator prior and how much from RL exploration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes NO-SAC, a soft actor-critic (SAC) controller for a first-order hyperbolic PIDE with spatially varying state delay. A DeepONet is first trained to approximate a backstepping boundary controller designed for delays satisfying the slow-variation assumption, and the trained DeepONet is embedded as a feature extractor in the SAC actor and critic networks. The authors claim that this architecture removes the assumption on the delay function required by backstepping, and report simulations in which NO-SAC converges faster than vanilla SAC and exhibits better transient behavior than the analytical backstepping controller.

Significance. If substantiated, the idea of using a neural operator pretrained on an analytic backstepping controller as a feature extractor inside SAC is a useful and timely contribution to learning-based PDE control. The paper makes the right comparisons in principle: it evaluates against both a no-prior RL baseline and the analytic backstepping controller, and the experimental configuration is described in reasonable detail. However, the central generalization claim is currently supported only by a single out-of-assumption simulation, with no training distribution for the DeepONet, no out-of-distribution validation, no ablations, and no repeated trials. The paper does not provide proofs, code, or error bars, so the strengths lie in the clarity of the problem formulation and the plausibility of the approach rather than in the strength of the evidence.

major comments (4)
  1. [§2, Definition 1, and §4.2] The controller operator U is defined only on D × C^1 × C^1 with D given by Eq. (11), yet the abstract's central claim is that NO-SAC eliminates the delay assumption by working for delays outside D. The manuscript never states the distribution of τ used to pretrain the DeepONet, reports no out-of-distribution prediction error, and evaluates only one delay, τ(x)=0.7+0.3 cos(4 arccos x), that violates the assumption. Because the DeepONet is trained on examples generated by the backstepping controller for delays in D, this single experiment does not establish that the learned operator remains informative outside D; this is a load-bearing evidential gap in the claim that the delay assumption is eliminated.
  2. [§3.2 and Algorithm 1] During SAC training, the DeepONet weights φ_N and ϑ_N are updated by backpropagation together with the fully connected layers (Algorithm 1, lines 13-14). Without an ablation that freezes the DeepONet features or replaces them with random features, the observed improvement over baseline SAC cannot be attributed to the backstepping prior; it could instead result from the RL agent adapting directly to the plant dynamics. The paper should report such an ablation to support the stated role of the DeepONet as the mechanism transferring prior knowledge.
  3. [§4.1 and §4.2] All conclusions are drawn from single training runs. The reward curves in Fig. 4 and the state evolutions in Figs. 5-7 contain no error bars, no multiple random seeds, and no sensitivity analysis with respect to the reward weights Γ, σ, ζ or the SAC hyperparameters. Consequently, the statement in §4.2 that NO-SAC 'consistently outperforms' the baseline is unsupported; the evidence shows one favorable trajectory, not a statistically reliable comparison.
  4. [§4.1, §4.2, and §5] The comparison with the backstepping controller may be confounded by implementation differences. The RL control is updated every 100 steps with zero-order hold and is bounded by [-30,30], but no implementation details are given for the backstepping controller used in Figs. 6-7; if the backstepping controller is evaluated as a continuous-time signal, the claim of smaller overshoot and shorter settling time is not an apples-to-apples comparison. In addition, the conclusion's statement that NO-SAC 'stabilizes' the PDE is stronger than what a 5-second finite-horizon simulation can establish, since no asymptotic or quantitative convergence criterion is reported.
minor comments (5)
  1. [§4.1 and §4.2 captions] The delay function is written inconsistently: Fig. 3 defines τ(x)=0.7+0.3 cos(4 arccos(x)), while the Fig. 5 caption says τ(x)=0.7+0.3 cos(arccos(x)). Please correct this, as the exact delay affects reproducibility.
  2. [§3.1, Eq. (15)] The reward r_mid uses s_{t-1} without defining s_{-1} at the start of an episode; please specify the initial previous state.
  3. [Throughout] There are several typos: 'bacsktepping' in Section 2, 'actot-critic' in Section 3.2, and 'whitout' in the Fig. 5 caption. The manuscript should be proofread.
  4. [§2 and §3.1] The symbol U is overloaded: it denotes the control input in Eq. (2), the controller operator in Definition 1, and the bound of the action space in Section 3.1. Consider using distinct symbols for these quantities.
  5. [§2] The text says the DeepONet inputs consist of τ, x, and u, while Definition 1 defines the operator as U(τ,v,u). The figure and text should clarify whether the state v is also an input to the DeepONet.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DeepONet feature extractor is trained on a prior backstepping controller, but NO-SAC is evaluated against external baselines and no equation reduces the claimed prediction to a fitted input.

full rationale

The paper's derivation chain is self-contained in the relevant sense. The DeepONet approximates the backstepping controller operator U defined in Definition 1 and is then embedded as a feature extractor in the actor and critic networks of SAC; the actual control policy is still trained by the SAC objective (Eqs. 17, 18, 21) and the DeepONet weights are updated along with the actor and critic parameters in Algorithm 1. No equation in the manuscript defines the NO-SAC output as identically equal to the backstepping controller or to the DeepONet training target, so the claimed performance improvement is not forced by construction. The central claim that the delay assumption |tau'(x)| < 1 is eliminated rests on an empirical out-of-assumption simulation with tau(x) = 0.7+0.3 cos(4 arccos(x)) and on comparison against baseline SAC and the analytical backstepping controller; this is an external test, not a fitted quantity renamed as a prediction. The citations to Zhang and Qi (2021, 2024) and Qi et al. (2024a) are from the same research group and provide the training signal for the DeepONet, but they are prior analytical designs with stated assumptions rather than an imported uniqueness theorem, and they do not by themselves establish the current claim. The absence of an out-of-distribution validation or ablation for the pretrained DeepONet is an evidential gap regarding generalization, not a circular reduction; the approach may or may not generalize, but the paper does not define its result in terms of its inputs. No self-definitional step, fitted input presented as prediction, load-bearing self-citation chain, uniqueness import, ansatz smuggling via citation, or renaming of a known result was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central mechanism rests on the authors' own backstepping design, on the unstated generalization of the pretrained DeepONet, and on the fidelity of the numerical environment. No new physical entities are introduced. The main free parameters are reward-shaping constants and RL hyperparameters that are hand-picked and not subjected to sensitivity analysis.

free parameters (3)
  • Reward shaping weights Gamma, sigma, zeta = Gamma=0.008, sigma=300, zeta=10
    Hand-selected in Section 4.1; zeta=10 is comparable to initial L2 norms, making the terminal success condition easy, and the reward design is not justified by a stability argument.
  • SAC hyperparameters = lambda=0.00009, update frequency 2, eta=0.003, replay buffer 1e5, gamma=0.99
    Hand-selected and inherited from SB3 defaults; no sensitivity or ablation is reported, so the dependence of the claimed improvement on these values is unknown.
  • Control execution period and action bound = zero-order hold every 100 steps (0.2 s), action bounded in [-30,30]
    Chosen in Section 4.1; this discretization directly affects the transient comparison with the continuous-time backstepping controller.
assumptions (4)
  • domain assumption The backstepping controller from Zhang and Qi (2021, 2024) stabilizes the system for any delay tau in D.
    The DeepONet is trained to approximate this controller, so all prior knowledge in NO-SAC inherits the correctness of the backstepping design. This is cited work by the same research group.
  • domain assumption The augmented state (v,u) is a sufficient and Markovian state representation for the delayed PDE.
    Section 3.1 asserts Markovianity after augmentation, but no proof is given that the discretized u fully captures the delay memory or that the stochastic transition is well-defined.
  • domain assumption The Gym discretization of equations (1)-(4) faithfully represents the continuous PIDE.
    Section 4.1 references Bhan et al. (2024) without convergence checks, grid-refinement studies, or comparison against a high-resolution numerical solver.
  • domain assumption The DeepONet can approximate the controller operator well enough to serve as a useful feature extractor.
    Section 2 invokes the universal approximation property of operators, but no training loss, validation error, or approximation error analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay." pith.science (2026). https://pith.science/paper/6PPLGDJR

@misc{pith2026250118201,
  author       = {Pith},
  title        = {Pith review of: Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PPLGDJR}},
  note         = {Machine review of arXiv:2501.18201}
}
read the original abstract

Control of distributed parameter systems affected by delays is a challenging task, particularly when the delays depend on spatial variables. The idea of integrating analytical control theory with learning-based control within a unified control scheme is becoming increasingly promising and advantageous. In this paper, we address the problem of controlling an unstable first-order hyperbolic PDE with spatially-varying delays by combining PDE backstepping control strategies and deep reinforcement learning (RL). To eliminate the assumption on the delay function required for the backstepping design, we propose a soft actor-critic (SAC) architecture incorporating a DeepONet to approximate the backstepping controller. The DeepONet extracts features from the backstepping controller and feeds them into the policy network. In simulations, our algorithm outperforms the baseline SAC without prior backstepping knowledge and the analytical controller.

Figures

Figures reproduced from arXiv: 2501.18201 by the authors.

Figure 1
Figure 1. The structure of DeepONet. Define the backstepping controller designed in (Zhang and Qi (2021, 2024)) as an operator Definition 1. The controller operator U : D × C 1 [0, 1] × C 1 ([0, 1]2 ) 7→ R with U = U(τ, v, u), (12) where τ (x) is the delay function dependent on x, v(x, ·) and u(x, r, ·) are the system state and the delayed state, respectively. U is the control input. We apply a DeepONet to learn the controlle… view at source ↗
Figure 2
Figure 2. SAC architecture incorporating a DeepONet for [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Delay function τ (x) = 0.7 + 0.3 cos(4 arccos(x)). Each episode lasts 5 seconds and consists of 2500 steps under a temporal step size setting of ∆t = 0.002. The training process comprises 100 episodes in total taking approximately 40 minutes. For plant coefficient, the atten￾uation factor c(x) = 20(1−x) and heat transfer coefficient f(x, q) = 5 cos(2πq)+5 sin(2πx) are used. We use the delay function τ (x) = 0.7 + 0.… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The reward evolution [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Close-loop state v(x, t) with initial condition v0 = 6 and delay τ (x) = 0.7 + 0.3 cos(arccos(x)) using the NO-SAC (left) and the SAC (right) [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Close-loop state v(x, t) with initial condition v0 = 6 and delay function τ (x) = e −0.7x using the NO-SAC controller (Left), the SAC controller (Middle) and the backsteping controller (Right) [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Control input U(t) (Left), and L2 norm of state v(x, ·) (Right). The top and bottom panels correspond to τ (x) = 0.7 + 0.3 cos(arccos(x)) and τ (x) = e −0.7x , respectively. 105050. Zhang, J. and Qi, J. (2024). Corrigendum to “Compen￾sation of spatially-varying state d…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 19 canonical work pages

  1. [1]

    Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. (2017). Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30

    work page 2017

  2. [2]

    Bhan, L., Bian, Y., Krstic, M., and Shi, Y. (2024). PDE control gym: A benchmark for data-driven boundary control of partial differential equations. arXiv preprint arXiv:2405.11401

    work page Pith review arXiv 2024

  3. [3]

    Bhan, L., Shi, Y., and Krstic, M. (2023). Neural operators for bypassing gain and control computations in PDE backstepping. IEEE Transactions on Automatic Control

    work page 2023

  4. [4]

    and Ichise, R

    Bougie, N. and Ichise, R. (2020). Towards interpretable reinforcement learning with state abstraction driven by external knowledge. IEICE TRANSACTIONS on Information and Systems, 103(10), 2143--2153

    work page 2020

  5. [5]

    Bouteiller, Y., Ramstedt, S., Beltrame, G., Pal, C., and Binas, J. (2020). Reinforcement learning with random delays. In International conference on learning representations

    work page 2020

  6. [6]

    Chow, Y., Nachum, O., Duenez-Guzman, E., and Ghavamzadeh, M. (2018). A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31

    work page 2018

  7. [7]

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 1861--1870

    work page 2018

  8. [8]

    and Smyshlyaev, A

    Krstic, M. and Smyshlyaev, A. (2008). Backstepping boundary control for first-order hyperbolic PDEs and application to systems with actuator and sensor delays. Systems & Control Letters, 57(9), 750--758

    work page 2008

Show all 22 references
  1. [9]

    Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G.E. (2021). Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3), 218--229

    2021

  2. [10]

    Mo, S., Wu, N., Qi, J., Pan, A., Feng, Z., Yan, H., and Wang, Y. (2024). Proximal policy optimization learning based control of congested freeway traffic. Optimal Control Applications and Methods, 45(2), 719--736

    2024

  3. [11]

    Nu \ n o, E., Basa \ n ez, L., and Ortega, R. (2011). Passivity-based control for bilateral teleoperation: A tutorial. Automatica, 47(3), 485--495

    2011

  4. [12]

    Parisi, S., Ramstedt, S., and Peters, J. (2017). Goal-driven dimensionality reduction for reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4634--4639. IEEE

    2017

  5. [13]

    (2024 a )

    Qi, J., Hu, J., Zhang, J., and Krstic, M. (2024 a ). Neural operator feedback for a first-order PIDE with spatially-varying state delay. arXiv preprint arxiv:2412.08219

    2024

  6. [14]

    (2024 b )

    Qi, J., Zhang, J., and Krstic, M. (2024 b ). Neural operators for pde backstepping control of first-order hyperbolic PIDE with recycle and delay. Systems & Control Letters, 185, 105714

    2024

  7. [15]

    Quartz, T., Zhou, R., De Sterck, H., and Liu, J. (2024). Stochastic reinforcement learning with stability guarantees for control of unknown nonlinear systems. arXiv preprint arXiv:2409.08382

    2024 arXiv

  8. [16]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  9. [17]

    Song, Y., Romero, A., M \"u ller, M., Koltun, V., and Scaramuzza, D. (2023). Reaching the limit in autonomous racing: Optimal control versus reinforcement learning. Science Robotics, 8(82), eadg1462

    2023

  10. [18]

    and Zhao, X

    Yu, H. and Zhao, X. (2022). Deep reinforcement learning with reward design for quantum control. IEEE Transactions on Artificial Intelligence, 5(3), 1087--1101

    2022

  11. [19]

    and Qi, J

    Zhang, J. and Qi, J. (2021). Compensation of spatially-varying state delay for a first-order hyperbolic PIDE using boundary control. Systems & Control Letters, 157, 105050

    2021

  12. [20]

    C ompensation of spatially-varying state delay for a first-order hyperbolic PIDE using boundary control

    Zhang, J. and Qi, J. (2024). Corrigendum to “ C ompensation of spatially-varying state delay for a first-order hyperbolic PIDE using boundary control”[syst. control lett. 157 (2021) 105050]. Systems & Control Letters, 105964

    2024

  13. [21]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence afte...

  14. [22]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.