Online Policy Evaluation for MDPs with Dynamic UBSR Measures

Erick Delage; Weikai Wang

REVIEW 2 major objections 3 minor 115 references

Online Policy Evaluation for MDPs with Dynamic UBSR Measures

T0 review · 2 major / 3 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents UBSR-TD, the first fully online, linear function-approximation policy evaluation algorithm for MDPs with dynamic utility-based shortfall risk (UBSR), and proves its almost-sure convergence under a discount-factor conditi

desk verdict A genuinely useful TD-style algorithm for dynamic UBSR policy evaluation, but the main a.s. convergence theorem is not fully proven as stated—the proof smuggles in a spectral condition on the features via a rescaling claim that doesn't hold for the given update. read the letter →

arxiv 2607.23030 v1 pith:UJMJGQDM submitted 2026-07-25 cs.LG math.OC

Weikai Wang , Erick Delage This is my paper · ORCID

classification cs.LGmath.OC MSC 90C4062L20

keywords risk-awarereinforcementlearningutility-basedshortfallrisktemporal-differencepolicyevaluationlinearfunctionapproximationalmost-sureconvergencedynamicmeasures

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes UBSR-TD, a fully online temporal-difference algorithm for evaluating policies in Markov decision processes where risk is measured by a dynamic utility-based shortfall risk (UBSR), a flexible family that includes expectation, expectile, and entropic risk. The central claim is that by passing the TD error through the UBSR loss function, the algorithm solves a projected risk-aware Bellman equation, and under a discount-factor bound relative to the loss's slopes, the iterates converge almost surely to a unique stationary point. If correct, this provides the first almost-sure convergence guarantee for fully online, function-approximated policy evaluation under dynamic UBSR, and it shows that classical risk-neutral TD machinery can be reused by simply inserting a loss. The paper also supplies accelerated variants and demonstrates the method on a perishable inventory control problem with shelf-life uncertainty.

What carries the argument

The engine is the semi-gradient temporal-difference update with the loss function inserted into the TD error: θ_{n+1} = θ_n + η_n φ(X_n) ℓ((γφ(X_{n+1}) − φ(X_n))ᵀθ_n + c(X_n)). The analysis relies on the mean-field operator Ĥ(θ) = α E[φ(X)ℓ(·)] + θ and proving it is a contraction in Euclidean norm under the γ-condition, so the ODE has a unique equilibrium that coincides with the fixed point of the projected risk-aware Bellman operator. The loss slopes [ε₁, L₁] and the discount bound γ < ε₁/L₁ are what make the contraction possible.

What would settle it

Check the unstated spectral bound on diag(q)^{1/2} Φ used in the proof of Lemma 4: if there exists a full-rank feature matrix Φ satisfying the normalization assumptions under which the operator Ĥ is not a contraction in Euclidean norm (e.g., by numerically computing its Lipschitz constant), the convergence argument collapses and the theorem would need revision.

Watch

Extended reading notes

Core claim

The paper establishes that the UBSR-TD update, θ_{n+1} = θ_n + η_n φ(X_n) ℓ((γφ(X_{n+1}) − φ(X_n))ᵀθ_n + c(X_n)), converges almost surely to the unique θ* whose Φθ* is the fixed point of the projected risk-aware Bellman operator ΠH, provided the Markov chain is ergodic, the loss has slopes in [ε₁, L₁], and γ < ε₁/L₁. This makes UBSR-TD the first fully online, linear-function-approximation policy evaluation algorithm for dynamic UBSR measures with an almost-sure convergence guarantee. The proof works by showing the associated mean-field ODE has a unique, globally asymptotically stable equilibrium and by applying a recent stochastic-approximation result for Markovian noise.

Load-bearing premise

The convergence proof hinges on the discount factor being smaller than the ratio of the loss function's smallest slope to its largest slope — a bound that common risk-aversion settings (e.g., expectile at τ=0.9) violate, and when it is violated the limiting ODE can have many or unstable equilibria.

Editorial extensions

If this is right

Risk-neutral TD(0), TD(λ), and Newton-style accelerations can be adapted to dynamic UBSR by replacing the linear TD error with the loss-applied version.
Policy evaluation under expectile or soft-quantile risk can run fully online on streaming transitions without a simulator.
The method enables risk-aware approximate policy iteration for large state spaces, as shown on a perishable inventory problem where it beats static, myopic, and risk-neutral baselines.
The convergence guarantee pins down the fixed point as the solution to a root-finding problem, giving a well-defined optimization objective for risk-aware evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The discount-factor restriction γ < ε₁/L₁ is likely the main practical obstacle: for expectile risk at τ=0.9, it forces γ below about 0.11, while many applications use 0.6–0.9; the paper's own Example 5 shows the condition is not merely technical, so relaxing it would require a different contraction argument or a sharper bound using transition and feature structure.
Because TD(λ) with λ>0 generally solves a different root-finding problem in the risk-aware setting, practitioners should use λ=0 (or the Newton variant) if they want the projected value function.
The paper's evidence that UBSR-TD sometimes converges even when the γ-condition fails suggests the bound is sufficient but not necessary; a sharper characterization of the exact threshold would be a natural next step.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: this is a worthwhile paper that does what it claims conceptually—online linear-function-approximation policy evaluation for dynamic UBSR—but the proof of the main theorem has a real hole, and the theorem as stated overclaims. I'd send it to a referee, and I expect the fix is manageable, but it's not done yet.

What's actually new: UBSR-TD is TD(0) with the TD error passed through the loss function. That's a natural idea, and the paper's derivation from the UBSR Bellman equation via the semi-gradient principle is clean. The genuinely new piece is the a.s. convergence analysis, which has to handle a nonlinear mean-field operator and therefore can't use the classical TD proof. The authors correctly identify that the discount factor must be bounded by epsilon1/L1 for the projected Bellman operator to be a contraction in the q-norm, and I appreciate that they show a concrete counterexample (Example 5) where the condition fails and the ODE develops a continuum of equilibria or an unstable one. The experiments, including the perishable-inventory application, are extensive and back up the claims—in particular, they honestly run cases where gamma violates the bound and the method still converges, which is the right kind of robustness check.

Where it's soft: two places in the proof of Theorem 1. The first is Lemma 4's spectral assumption on sqrt(diag(q))Phi. The theorem never states this assumption, and the proof's 'feature rescaling is WLOG' claim doesn't hold for the stated algorithm: rescaling features by a diagonal matrix D changes the update by D^2 unless the step size is also rescaled, which doesn't happen. So the contraction argument for the mean-field operator currently covers a different algorithm than the one in Theorem 1. This is load-bearing, not a cosmetic gap. The second is condition (iv) of the Borkar et al. theorem, which is dismissed with 'a similar step' when the asymptotic ODE involves the infinity loss; this is probably fine if the spectral condition is added, but the details aren't there. The gamma restriction is severe for expectile risk at tau=0.9 (gamma<0.11), but that's an explicit limitation, not a flaw, and the paper says the bound is conservative.

Who should read it: anyone working on risk-aware RL, especially those extending TD-type methods to dynamic risk measures. The algorithm and the experimental study are useful even before the proof is fixed; just don't rely on the theorem as stated.

Recommendation: engage, invite a revision. Ask the authors to either add the spectral condition to Assumption 5/Theorem 1 or fix the rescaling argument, and to spell out the infinity-ODE verification. Once that's done, the paper is a solid contribution.

Referee Report

2 major / 3 minor

Summary. The paper studies online policy evaluation for MDPs with dynamic utility-based shortfall risk (UBSR) under linear function approximation. It proposes UBSR-TD, a semi-gradient TD update that applies the loss function to the TD error, and claims almost-sure convergence to the parameter θ* whose feature vector Φθ* is the fixed point of the projected risk-aware Bellman operator ΠH (Theorem 1), under Assumptions 2–5 and the discount restriction γ < ε1/L1. The proof proceeds via contraction of the mean-field operator H (Lemma 1), uniqueness of the projected fixed point (Lemmas 2–3), contraction of the transformed operator Ĥ (Lemma 4), and an application of a stochastic-approximation theorem for Markovian noise (Borkar et al., 2025). Variants UBSR-TD(λ) and UBSR-Newton are introduced, and numerical experiments, including a perishable inventory ADP application, are presented.

Significance. If Theorem 1 holds, it is the first almost-sure convergence result for fully online policy evaluation with function approximation under a dynamic UBSR measure in a discounted MDP, extending TD(0) to general loss functions. The formulation is elegant and the numerical evidence, including the inventory application, suggests practical value. The proof uses a modern ODE-based SA framework; the contraction analysis is nontrivial. However, the proof of Theorem 1 currently contains an unstated spectral condition and an incomplete verification of the scaled-ODE stability, so the claimed result is not yet fully established.

major comments (2)

[Appendix B, Lemma 4 (proof of Theorem 1)] The proof of Lemma 4 introduces a spectral condition on sqrt(diag(q))Φ (largest singular value bounded by sqrt((2ε1-γ(ε1+L1))/(α^2 L1^2 (γ+1)^2))) that is absent from Theorem 1 and Assumptions 1–5. The claim that feature rescaling is WLOG is not valid for update (3.4): a scalar rescaling φ→cφ changes the update by a factor c and is equivalent only to a step-size change η_n→c^2 η_n. Moreover, the contraction factor is only shown to be <1, not >−1; the stated bound with α^2 in the denominator does not ensure the coefficient is negative when α<1. Since Lemma 4 is the only proof that the mean-field ODE is globally asymptotically stable, Theorem 1 is not fully established for arbitrary full-rank features. This is repairable by adding the spectral condition as an explicit assumption or by choosing α sufficiently small and deriving the contraction accordingly.
[Section 4, proof of Theorem 1, condition (iv)] The stability of the scaled ODE (B.8) is asserted by 'a similar step' to (v). But (v) uses Lemma 4, whose hypotheses are for the original loss ℓ; for ℓ∞ (Lemma 5) the slope bounds ε1∞, L1∞ may differ, and γ<ε1/L1 does not imply γ<ε1∞/L1∞. A separate contraction analysis for h∞ is required; otherwise the invocation of Theorem 2 of Borkar et al. is incomplete.

minor comments (3)

[Appendix A, Algorithm boxes] Typos: 'Finial estimator' should be 'Final estimator'; 'invertable' should be 'invertible'; Table C.1 heading 'Compuational' should be 'Computational'.
[Section 6.1.1] The experiments with γ=0.9 (and τ=0.9, γ=0.6) violate the theorem's γ<ε1/L1 condition. The text notes convergence nonetheless, but it would help to state explicitly which panels are outside the theoretical guarantee, so readers do not confuse numerical evidence with verification of Theorem 1.
[Figure 6.1] Labels 'TD(0.0)', 'TD(λ)', 'NT' are informal; define them as UBSR-TD with the corresponding λ value and UBSR-Newton in the caption or text.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: UBSR-TD targets a fixed point defined independently of the algorithm, and the sole self-citation is a non-load-bearing technical lemma.

full rationale

The convergence claim is not circular. UBSR-TD (3.4) is derived from the first-order condition (3.3), and θ* is independently characterized in Lemma 2 as the parameter whose span-projection is the fixed point of the projected risk-aware Bellman operator ΠH. Theorem 1 then proves convergence of the stochastic approximation to that root; no fitted data or hyperparameter is renamed as a prediction. The contraction analysis is self-contained given Assumptions 2–5, with the γ < ε1/L1 restriction explicitly exposed via Example 5. The only self-citation is Lemma 5, quoting Lemma B.19 of Wang & Delage (2025) to establish existence of the asymptotic loss ℓ∞; this is a parameter-free regularity lemma with stated assumptions that do not include the target convergence result, so it does not carry the load of the theorem. Two non-circular proof gaps should be weighed for correctness: Lemma 4 assumes a spectral bound on sqrt(diag(q))Φ that is absent from Theorem 1, justified by an imprecise feature-rescaling WLOG; and condition (iv) of Theorem 2 is concluded by 'following a similar step' without re-verifying contraction for ℓ∞. These omissions affect rigor, not circularity.

Assumptions & free parameters 2 free parameters · 8 assumptions · 0 invented entities

The central algorithm introduces no new entities; it relies on the standard UBSR loss ell, features phi, and constants. The main external/unproved inputs are the stochastic-approximation theorem of Borkar et al. (2025) and a self-cited lemma from Wang & Delage (2025). The hand-tuned step size and warm-up length are experimental parameters, and the proof secretly adds a spectral condition on the feature matrix.

free parameters (2)

UBSR-TD step-size schedule = eta_n = 2/(n+100)^(2/3)
Calibrated in preliminary experiments (Section 6.1) to promote convergence; not part of the theorem but used in all main convergence plots.
UBSR-Newton warm-up length n0 = 500 (synthetic experiments) / 1000 (inventory)
Chosen so that the empirical information matrix H_{n0} is invertible; trajectories with singular H are discarded, a heuristic exclusion.

assumptions (8)

domain assumption Assumption 1: the policy-induced Markov chain P is finite, aperiodic, and irreducible (hence geometrically ergodic).
Used to define the stationary distribution q, guarantee a unique invariant measure, and satisfy the Lyapunov-drift condition when invoking Borkar et al. (2025).
domain assumption Assumption 2: the UBSR loss ell is continuous, strictly increasing, and has slopes in [epsilon_1, L_1] with epsilon_1 > 0.
Gives contraction of H and defines the admissible discount bound gamma < epsilon_1/L_1; excludes ERM.
domain assumption Assumption 3: ell is convex or concave on [0, infinity) and on (-infinity, 0).
Needed for Lemma 5 on the existence of the asymptotic loss ell_infinity used in condition (iv) of the SA theorem.
domain assumption Assumption 4: features are uniformly bounded, max_x ||phi(x)||_2 <= M.
Standard bounded-feature condition used in Lipschitz estimates of H.
domain assumption Assumption 5: step sizes eta_n satisfy sum eta_n = infinity, sum eta_n^2 < infinity, and lim(eta^{-1}_{n+1} - eta^{-1}_n) exists.
Stronger than Robbins-Monro; required by Theorem 1 of Borkar et al. (2025).
ad hoc to paper Unstated spectral condition in Lemma 4: lambda_M(diag(q)^{1/2} Phi) < sqrt((2 epsilon_1 - gamma(epsilon_1 + L_1))/(alpha^2 L_1^2 (gamma+1)^2)).
Introduced in the proof of Lemma 4, not in the theorem assumptions; claimed WLOG via feature rescaling.
ad hoc to paper Lemma B.19 of Wang & Delage (2025): ell_s(z) = ell(s z)/s converges uniformly on compact sets to ell_infinity satisfying Assumptions 2-3.
Self-cited black-box result; load-bearing for Lemma 6 and condition (iv).
standard math Theorem 1 of Borkar et al. (2025): almost-sure convergence of SA with Markovian noise under conditions (i)-(v).
External theorem used unproved as the main convergence engine.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Online Policy Evaluation for MDPs with Dynamic UBSR Measures." pith.science (2026). https://pith.science/paper/UJMJGQDM

@misc{pith2026260723030,
  author       = {Pith},
  title        = {Pith review of: Online Policy Evaluation for MDPs with Dynamic UBSR Measures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJMJGQDM}},
  note         = {Machine review of arXiv:2607.23030}
}

read the original abstract

Developing efficient function-approximation methods for policy evaluation is a fundamental challenge in risk-aware reinforcement learning. Existing approaches either focus on restrictive classes of risk measures or rely on access to a simulator, limiting their applicability in fully online settings. In this work, we propose computationally efficient online learning algorithms for policy evaluation in Markov decision processes (MDPs) with dynamic utility-based shortfall risk (UBSR) measures under linear function approximation. Specifically, we introduce the UBSR-TD algorithm, establish conditions under which it converges almost surely, and develop several variants designed to accelerate convergence. Our formulation shows that existing policy evaluation algorithms for risk-neutral MDPs can be readily adapted to dynamic UBSR settings by incorporating a loss function into the temporal-difference error. Numerical experiments support our theoretical findings, and an application to a perishable inventory management problem with shelf-life uncertainty demonstrates the practical effectiveness of the proposed methods.

Figures

Figures reproduced from arXiv: 2607.23030 by the authors.

**Figure 6.1.** Convergence results for expectile risk 13 [PITH_FULL_IMAGE:figures/full_fig_p013_6_1.png] view at source ↗

**Figure 6.2.** (Left) Optimal risk-aware policies for m = 3; (Right) Optimality gaps of policy iteration with UBSR-TD and UBSR-Newton (TD and NT) for m = 3. 0 50 100 150 200 250 300 Simulated Cost 0.0 0.5 1.0 Empirical CDF (50, 1, 20, 5) = 0.1 = 0.5 = 0.9 0 50 100 150 200 250 Simulated Cost 0.0 0.5 1.0 (20, 1, 20, 5) = 0.1 = 0.5 = 0.9 0 50 100 150 200 250 Simulated Cost 0.0 0.5 1.0 (10, 1, 20, 5) = 0.1 = 0.5 = 0.9 0 50 100 150 200… view at source ↗

**Figure 6.3.** Empirical cumulative distributions of simulated costs under policies obtained via policy iteration [PITH_FULL_IMAGE:figures/full_fig_p016_6_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

115 extracted references · 4 linked inside Pith

[1]

Tsitsiklis and B

John N. Tsitsiklis and B. An analysis of temporal-difference learning with function approximation , journal =. 1997 , pages =

1997
[2]

Liu and J

W. Liu and J. Tu and X. Chen and Y. Zhang , title =. The Annals of Statistics , volume =. 2025 , pages =

2025
[3]

Tobia and Tobias Sommer and Klaus Obermayer , title =

Yun Shen and Michael J. Tobia and Tobias Sommer and Klaus Obermayer , title =. Neural Computation , year =
[4]

Quantitative Finance , year =

Fabio Bellini and Valeria Bignozzi , title =. Quantitative Finance , year =
[5]

SIAM Journal on Control and Optimization , year =

Yun Shen and Wilhelm Stannat and Klaus Obermayer , title =. SIAM Journal on Control and Optimization , year =
[6]

Borkar , title =

Vivek S. Borkar , title =
[7]

D. P. Bertsekas and J. N. Tsitsiklis , title =
[8]

R. S. Sutton , title =. Machine Learning , volume =

Show all 115 references

[9]

V. S. Borkar and S. P. Meyn , title =. SIAM Journal on Control and Optimization , year =
[10]

Garcia and F

J. Garcia and F. Fernandez , title =. Journal of Machine Learning Research , year =
[11]

Mathematical Finance , year =

Anthony Coache and Sebastian Jaimungal , title =. Mathematical Finance , year =
[12]

Guidelines for human-

Amershi, Saleema and Weld, Dan and Vorvoreanu, Mihaela and others , booktitle=. Guidelines for human-
[13]

IEEE Transactions on Intelligent Transportation Systems , volume=

Risk assessment methodologies for autonomous driving: A survey , author=. IEEE Transactions on Intelligent Transportation Systems , volume=
[14]

Nature , volume=

Mastering the game of go without human knowledge , author=. Nature , volume=
[15]

Sutton and Andrew G

Richard S. Sutton and Andrew G. Barto , title =. 2018 , address =

2018
[16]

Proceedings of the 26th International Conference on Machine Learning , pages=

Fast gradient-descent methods for temporal-difference learning with linear function approximation , author=. Proceedings of the 26th International Conference on Machine Learning , pages=
[17]

Artzner and F

P. Artzner and F. Delbaen and J. Eber and D. Heath , title =. Mathematical Finance , year =
[18]

Cont and R

R. Cont and R. Deguest and G. Scandolo , title =. Quantitative Finance , year =
[19]

Danielsson , title =

J. Danielsson , title =
[20]

N. S. Lambert and D. M. Pennock and Y. Shoham , title =. Proceedings of the 9th ACM Conference on Electronic Commerce , year =
[21]

Gneiting , title =

T. Gneiting , title =. Journal of the American Statistical Association , year =
[22]

Wang and M

Y. Wang and M. P. Chapman , title =. Artificial Intelligence , year =
[23]

Marzban and E

S. Marzban and E. Delage and J. Y. Li , title =. Quantitative Finance , year =
[24]

Coache and S

A. Coache and S. Jaimungal and A. Cartea , title =. SIAM Journal on Financial Mathematics , year =
[25]

Planning and learning in average risk-aware

Wang, Weikai and Delage, Erick , booktitle =. Planning and learning in average risk-aware
[26]

L. A. Prashanth and Mohammad Ghavamzadeh , title =. Advances in Neural Information Processing Systems , volume =
[27]

Risk-averse learning by temporal difference methods with

K. Risk-averse learning by temporal difference methods with. Journal of Machine Learning Research , volume=
[28]

Naval Research Logistics , year =

Zhaolin Hu and Dali Zhang , title =. Naval Research Logistics , year =
[29]

Giesecke and T

K. Giesecke and T. Schmidt and S. Weber , title =. Journal of Investment Management , year =
[30]

Journal of Risk , year =

Susanne Emmer and Marie Kratz and Dirk Tasche , title =. Journal of Risk , year =
[31]

arXiv preprint arXiv:2005.01643 , year=

Offline reinforcement learning: Tutorial, review, and perspectives on open problems , author=. arXiv preprint arXiv:2005.01643 , year=

2005 arXiv
[32]

Journal of the American Statistical Association , volume=

Online bootstrap inference for policy evaluation in reinforcement learning , author=. Journal of the American Statistical Association , volume=
[33]

The Eleventh International Conference on Learning Representations , year=

Risk-aware reinforcement learning with coherent risk measures and non-linear function approximation , author=. The Eleventh International Conference on Learning Representations , year=
[34]

Risk-averse dynamic programming for

Ruszczy. Risk-averse dynamic programming for. Mathematical Programming, Series B , volume=
[35]

Proceedings of the 41st International Conference on Machine Learning , pages=

Provable risk-sensitive distributional reinforcement learning with general function approximation , author=. Proceedings of the 41st International Conference on Machine Learning , pages=
[36]

Journal of Machine Learning Research , volume=

Bridging distributional and risk-sensitive reinforcement learning with provable regret bounds , author=. Journal of Machine Learning Research , volume=
[37]

Provably Efficient Iterated

Yu Chen and Yihan Du and Pihe Hu and Siwei Wang and Desheng Wu and Longbo Huang , booktitle=. Provably Efficient Iterated
[38]

Liu, Shuze Daniel and Chen, Shuhang and Zhang, Shangtong , journal=. The
[39]

arXiv preprint arXiv:2302.07690 , year =

Xiang Li and Jiadong Liang and Zhihua Zhang , title =. arXiv preprint arXiv:2302.07690 , year =

arXiv
[40]

Journal of Machine Learning Research , volume=

An analysis of quantile temporal-difference learning , author=. Journal of Machine Learning Research , volume=
[41]

Journal of Machine Learning Research , volume=

Policy evaluation with temporal differences: A survey and comparison , author=. Journal of Machine Learning Research , volume=
[42]

Approximate value iteration for risk-aware

Yu, Pengqian and Haskell, William B and Xu, Huan , journal=. Approximate value iteration for risk-aware
[43]

Föllmer and A

H. Föllmer and A. Schied , title =. 2016 , address =

2016
[44]

IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

A review of safe reinforcement learning: Methods, theories and applications , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=
[45]

The Annals of Probability , year =

Berkes, Istvan and Philipp, Walter , title =. The Annals of Probability , year =
[46]

Billingsley , title =

P. Billingsley , title =
[47]

Finance and Stochastics , year=

Convex measures of risk and trading constraints , author=. Finance and Stochastics , year=
[48]

Compressed Sensing, Theory and Applications , publisher =

Roman Vershynin , title =. Compressed Sensing, Theory and Applications , publisher =
[49]

Pascal Massart , title =
[50]

Yurii Nesterov , title =
[51]

The Annals of Probability , year =

Merlevède, Florence and Peligrad, Magda , title =. The Annals of Probability , year =
[52]

Annales de l'Institut Henri Poincaré, Probabilités et Statistiques , volume=

The functional central limit theorem for strongly mixing processes , author=. Annales de l'Institut Henri Poincaré, Probabilités et Statistiques , volume=
[53]

2010 , address =

Haim Brezis , title =. 2010 , address =

2010
[54]

Khalil , title =

Hassan K. Khalil , title =
[55]

Borkar and Shuhang Chen and Adithya Devraj and Ioannis Kontoyiannis and Sean Meyn , title =

Vivek S. Borkar and Shuhang Chen and Adithya Devraj and Ioannis Kontoyiannis and Sean Meyn , title =. The Annals of Applied Probability , year =
[56]

Rick Durrett , title =
[57]

Randal Douc and Pierre Priouret and Eric Moulines and Philippe Soulier , title =
[58]

Benveniste, A and Métivier, M and Prioret, P , title =
[59]

Proceedings of the 31st International Conference on Machine Learning , year =

Aviv Tamar and Shie Mannor and Huan Xu , title =. Proceedings of the 31st International Conference on Machine Learning , year =
[60]

Borkar , title =

Siddharth Chandak and Vivek S. Borkar , title =. Stochastic Systems , year =
[61]

Maguluri and Sanjay Shakkottai and Karthikeyan Shanmugam , title =

Zaiwei Chen and Siva T. Maguluri and Sanjay Shakkottai and Karthikeyan Shanmugam , title =. Operations Research , year =
[62]

Roman Vershynin , title =
[63]

A Generalized

David Choi and Benjamin. A Generalized. Discrete Event Dynamic Systems , year =
[64]

B. T. Polyak and A. B. Juditsky , title =. SIAM Journal on Control and Optimization , year =
[65]

Sean Meyn and Richard Tweedie , title =
[66]

SIAM Journal on Control and Optimization , year =

Huizhen Yu , title =. SIAM Journal on Control and Optimization , year =
[67]

Least Squares Policy Evaluation Algorithms with Linear Function Approximation , journal =

Angelia Nedi\'. Least Squares Policy Evaluation Algorithms with Linear Function Approximation , journal =. 2003 , volume =

2003
[68]

Proceedings of the AAAI Conference on Artificial Intelligence , volume =

Yangchen Pan and Adam White and Martha White , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =
[69]

Asian Conference on Machine Learning , year =

Arash Givchi and Maziar Palhang , title =. Asian Conference on Machine Learning , year =
[70]

Proceedings of the 25th International Conference on Machine Learning , year =

Hengshuai Yao and Zhi-Qiang Liu , title =. Proceedings of the 25th International Conference on Machine Learning , year =
[71]

Ruppert, David , title =
[72]

2021 , publisher=

Lectures on Stochastic Programming: Modeling and Theory, 2nd Edition , author=. 2021 , publisher=

2021
[73]

Proceedings of the 28th International Conference on Artificial Intelligence and Statistics , year =

Hau, Jia Lin and Delage, Erick and Derman, Esther and Ghavamzadeh, Mohammad and Petrik, Marek , title =. Proceedings of the 28th International Conference on Artificial Intelligence and Statistics , year =
[74]

Machine learning , volume=

Technical update: Least-squares temporal difference learning , author=. Machine learning , volume=
[75]

Advances in Neural Information Processing Systems , year =

Hau, JL and Delage, E and Ghavamzadeh, M and Petrik, M , title =. Advances in Neural Information Processing Systems , year =
[76]

European Journal of Operational Research , volume=

Time consistency and risk averse dynamic decision models: Definition, interpretation and practical consequences , author=. European Journal of Operational Research , volume=
[77]

2022 IEEE 61st Conference on Decision and Control (CDC) , pages=

Risk-averse reinforcement learning via dynamic time-consistent risk measures , author=. 2022 IEEE 61st Conference on Decision and Control (CDC) , pages=. 2022 , organization=

2022
[78]

Stochastic approximation for risk-aware

Huang, Wenjie and Haskell, William B , journal=. Stochastic approximation for risk-aware
[79]

and Powell, Warren B

Jiang, Daniel R. and Powell, Warren B. , title =. Mathematics of Operations Research , year =
[80]

Soumen Pachal and Mizhaan Prajit Maniyar and Prashanth, L. A. , booktitle =. Policy. 2026 , volume =

2026
[81]

Semismooth Newton methods for risk-averse

Gargiani, Matilde and Micheli, Francesco and Tsiamis, Anastasios and Lygeros, John , journal=. Semismooth Newton methods for risk-averse. 2026 , volume =

2026
[82]

Asadi, Kavosh and Sabach, Shoham and Liu, Yao and Gottesman, Omer and Fakoor, Rasool , journal=
[83]

Insurance: Mathematics and Economics , volume=

Generalized quantiles as risk measures , author=. Insurance: Mathematics and Economics , volume=
[84]

INFORMS Journal on Computing , year =

Hossein Abouee-Mehrizi and Mahdi Mirjalili and Vahid Sarhangian , title =. INFORMS Journal on Computing , year =
[85]

2011 , address =

Steven Nahmias , title =. 2011 , address =

2011
[86]

Powell , title =

Warren B. Powell , title =
[87]

Natheaniel Korda and L. A. Prashanth , title =. Proceedings of the 32nd International Conference on Machine Learning , year =
[88]

G. W. Stewart , title =
[89]

Devraj and Fan Lu and Ana Bušić and Sean P

Shuhang Chen and Adithya M. Devraj and Fan Lu and Ana Bušić and Sean P. Meyn , title =. Advances in Neural Information Processing Systems , year =
[90]

Devraj and Sean P

Adithya M. Devraj and Sean P. Meyn , title =. Advances in Neural Information Processing Systems , year =
[91]

J. C. Spall , title =. IEEE Transactions on Automatic Control , year =
[92]

The Annals of Statistics , year =

David Ruppert , title =. The Annals of Statistics , year =
[93]

Meyn , title =

Sean P. Meyn , title =. arXiv preprint arXiv:2307.02632 , year =

arXiv
[94]

Nature , volume=

Human-level control through deep reinforcement learning , author=. Nature , volume=
[95]

Proceedings of the 24th International Conference on Artificial Intelligence and Statistics , year =

Joan Bas-Serrano and Sebastian Curi and Andreas Krause and Gergely Neu , title =. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics , year =
[96]

Operations Research , volume=

Approximation algorithms for perishable inventory systems , author=. Operations Research , volume=
[97]

Management Science , volume=

Asymptotic optimality of base-stock policies for perishable inventory systems , author=. Management Science , volume=
[98]

Operations Research , year =

Jalaj Bhandari and Daniel Russo and Raghav Singal , title =. Operations Research , year =
[99]

Finite sample analyses for

Dalal, Gal and Sz. Finite sample analyses for. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
[100]

Operations Research , year =

Youyi Feng and Baichun Xiao , title =. Operations Research , year =
[101]

Computational Optimization and Applications , year =

Soumya Ranjan Pathy and Hamed Rahimian , title =. Computational Optimization and Applications , year =
[102]

Yalçındağ and S

S. Yalçındağ and S. B. Güre and G. Carello and E. Lanzarone , title =. Health Care Management Science , year =
[103]

Journal of Ambient Intelligence and Humanized Computing , year=

Blood supply chain management: robust optimization, disruption risk, and blood group compatibility (a real-life case) , author=. Journal of Ambient Intelligence and Humanized Computing , year=
[104]

L. a. Prashanth and Mohammad Ghavamzadeh , journal=. Actor-critic algorithms for risk-sensitive
[105]

Advances in Neural Information Processing Systems , volume=

Policy gradient for coherent risk measures , author=. Advances in Neural Information Processing Systems , volume=
[106]

Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , pages=

Entropic risk optimization in discounted MDPs , author=. Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , pages=
[107]

Advances in Neural Information Processing Systems , volume=

Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=
[108]

Soft robust

Zhang, Runyu and Hu, Yang and Li, Na , booktitle=. Soft robust
[109]

Systems & Control Letters , volume=

On tight bounds for function approximation error in risk-sensitive reinforcement learning , author=. Systems & Control Letters , volume=
[110]

arXiv preprint arXiv:2605.07857 , year =

Yudong Luo and Erick Delage , title =. arXiv preprint arXiv:2605.07857 , year =

arXiv
[111]

Mathematics of Operations Research , volume=

A learning algorithm for risk-sensitive cost , author=. Mathematics of Operations Research , volume=
[112]

On the convergence and optimality of policy gradient for

Huang, Audrey and Leqi, Liu and Lipton, Zachary C and others , journal=. On the convergence and optimality of policy gradient for
[113]

Risk-sensitive

Howard, Ronald A and Matheson, James E , journal=. Risk-sensitive
[114]

Proceedings of the 38th International Conference on Machine Learning , pages=

Risk-sensitive reinforcement learning with function approximation: A debiasing approach , author=. Proceedings of the 38th International Conference on Machine Learning , pages=
[115]

Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated

Du, Yihan and Wang, Siwei and Huang, Longbo , booktitle=. Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Tsitsiklis and B

John N. Tsitsiklis and B. An analysis of temporal-difference learning with function approximation , journal =. 1997 , pages =

1997

[2] [2]

Liu and J

W. Liu and J. Tu and X. Chen and Y. Zhang , title =. The Annals of Statistics , volume =. 2025 , pages =

2025

[3] [3]

Tobia and Tobias Sommer and Klaus Obermayer , title =

Yun Shen and Michael J. Tobia and Tobias Sommer and Klaus Obermayer , title =. Neural Computation , year =

[4] [4]

Quantitative Finance , year =

Fabio Bellini and Valeria Bignozzi , title =. Quantitative Finance , year =

[5] [5]

SIAM Journal on Control and Optimization , year =

Yun Shen and Wilhelm Stannat and Klaus Obermayer , title =. SIAM Journal on Control and Optimization , year =

[6] [6]

Borkar , title =

Vivek S. Borkar , title =

[7] [7]

D. P. Bertsekas and J. N. Tsitsiklis , title =

[8] [8]

R. S. Sutton , title =. Machine Learning , volume =