Pith. sign in

REVIEW 1 major objections 6 minor 41 references

Jacobi-like relative value iteration algorithms for ergodic risk-sensitive control of Markov chains

T0 review · 1 major / 6 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read New algorithms solve risk-sensitive control by local contraction

desk verdict New RVI algorithms for finite-state risk-sensitive control with a real but fixable gap in the geometric convergence claim read the letter →

arxiv 2607.06863 v1 pith:2UH2LZ46 submitted 2026-07-07 math.OC math.PR

Sumith Reddy Anugu , Guodong Pang , Nicola Sassone This is my paper · ORCID
classification math.OCmath.PR
keywords algorithmscontrolfinitemarkovpropertiesrisk-sensitivebellman-likebi-lipschitz
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses a computational bottleneck in risk-sensitive control of Markov chains: how to numerically find optimal policies when the cost criterion involves an exponential utility, which makes the standard convergence tools from average-cost settings break down. The authors propose two iterative algorithms—a Jacobi-like and a Gauss-Seidel-like relative value iteration—that jointly update a value function and an optimal cost estimate. The key insight is that although the multiplicative (exponential) structure of the risk-sensitive Bellman operator prevents global contraction arguments, one can establish local contraction and local bi-Lipschitz continuity properties that are strong enough to guarantee geometric convergence. The central mechanism is a variational (entropy-based) reformulation of the Bellman operator that converts the multiplicative structure into an additive form with an auxiliary maximization over probability measures, allowing the authors to bound the operator's sensitivity using a carefully constructed weighted supremum norm. This norm, built from the transition graph's reachability structure, ensures that each iteration contracts the error by a fixed factor, provided the iterates stay within a bounded region—a condition the authors verify by tracking the step-size parameter and the state of the cost iterate.

What carries the argument

The entropy variational formula (Donsker-Varadhan), applied to rewrite the logarithm inside the risk-sensitive Bellman operator as a supremum over probability measures; a weighted supremum norm built from the state-space partition by reachability classes and the minimum positive transition probability η; Dynkin's formula applied to the transformed Markov chain with law Q*; and tail-probability bounds on return times under Q* that circumvent the potentially infinite exponential moments in the stochastic representation of the fixed point h_λ.

What would settle it

A counterexample would be a controlled Markov chain satisfying the irreducibility assumption where the iterates of Algorithm 1 or 2 fail to converge geometrically—most plausibly because the local bi-Lipschitz constant N̄*(m, λ) blows up as m increases, meaning the region where the local estimates hold shrinks faster than the iterates can be confined, or because the fixed point h_λ ceases to exist for λ-values the algorithm visits during its line-search updates.

Watch

Extended reading notes

Core claim

The paper's central claim is that the multiplicative structure of risk-sensitive control, which defeats the global contraction arguments used for average-cost problems, can be tamed locally: the risk-sensitive Bellman-like operators F and G (defined via logarithms of the multiplicative operator) satisfy local contraction properties under a weighted supremum norm, and the fixed points of these operators depend on the cost parameter λ in a locally bi-Lipschitz way. These two local properties, combined with a step-size-controlled line-search update for λ, suffice to prove geometric convergence of both algorithms. The bi-Lipschitz proof is the most delicate part: it constructs a transformed Mark

Load-bearing premise

The local bi-Lipschitz continuity of the map from cost parameter to fixed point (Proposition 3.2) is load-bearing. The proof constructs a transformed Markov chain whose transition probabilities depend on the unknown fixed points h_λ and h_λ', then bounds expected return times under this transformed chain using tail-probability estimates. If these tail bounds are not uniform—or if the exponential moments in the stochastic representation of h_λ fail to be finite for the λ-range

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper proposes two new relative value iteration (RVI) algorithms—a Jacobi-like version (Algorithm 1) and a Gauss-Seidel-like version (Algorithm 2)—for the ergodic risk-sensitive control (ERSC) problem of a finite-state controlled Markov chain. Under Assumption 2.1 (irreducibility and recurrence under every stationary Markov policy), the authors prove that the iterates converge geometrically to the solution pair (V*, λ*) of the multiplicative Bellman equation. The key technical innovation is overcoming the multiplicative structure of the risk-sensitive cost criterion, which prevents direct adaptation of the global contraction and bi-Lipschitz continuity arguments from the risk-neutral setting (Bertsekas, 1998). The authors instead establish local contraction properties for the risk-sensitive Bellman-like operators F and G (Propositions 3.1 and 4.3) and a local bi-Lipschitz continuity property for the fixed-point map λ ↦ h_λ (Proposition 3.2), using an entropy variational formula, Dynkin's formula, and a weighted supremum norm construction adapted from Tseng (1990). The paper concludes with two numerical examples: service-effort control for a finite-capacity queue and exit-rate maximization on a graph.

Significance. The paper makes a genuine contribution to the ERSC literature. The existing RVI algorithm for ERSC (Bielecki–Hernandez-Hernández–Pliska; Cavazos-Cadena–Montes-de Oca; Borkar–Meyn) requires either a self-loop positivity condition or an aperiodicity transformation, and its convergence analysis relies on span-semi-norm contraction of the multiplicative Bellman operator. The proposed algorithms avoid these requirements and introduce a fundamentally different update structure: the cost iterate λ_k is updated via a one-dimensional line-search rather than by direct evaluation of the Bellman operator at the reference state. The local contraction property (Proposition 3.1) is cleanly derived via the entropy variational formula and the weighted norm construction, and the local bi-Lipschitz continuity result (Proposition 3.2) is a non-trivial adaptation that addresses the failure of exponential moments to be finite for all λ ∈ ℝ (Remark 3.3). The Gauss-Seidel analysis (Lemma 4.1, Proposition 4.3) handles the additional recursive structure with appropriate uniform bounds. The numerical experiments are well-chosen, contrasting tridiagonal and sparse/dense transition structures, and the stepsize

major comments (1)
  1. Theorem 2.2, 'in particular' clause (Section 4.3): The geometric convergence claim is not justified for all step-size sequences satisfying γ_k ∈ (0, γ̄]. The one-step bounds (2.23) give |λ^{k+1} − λ*| ≤ c_λ(γ_k, m) |λ^k − λ*|, where c_λ(γ, m) = max{c_1(γ, m), c_2(m)} with c_1(γ, m) = max{1 − γA, β_m γ B} (A, B > 0; from Lemmas D.3–D.4) and c_2(m) = (1+β_m)/2. As γ → 0, the term 1 − γA → 1, so c_λ(γ, m) → 1. For a sequence γ_k → 0, the product Π_k c_λ(γ_k, m) may decay only polynomially (e.g., γ_k = 1/k gives ~k^{−A}) or may not converge to zero at all (e.g., γ_k = ξ^k gives Π(1 − Aξ^k) > 0). The proof in Section 4.3 simply asserts ||h^k − h*||_m ≤ C_0 e^{−C_1 k} without addressing this. The fix is straightforward: either require γ_k ≥ γ_ > 0 for all k (in which case c_λ(γ_k, m) ≤ c_λ(γ_, m) < 1 uniformly and geometric convergence follows), or restrict the geometric rate claim to constant
minor comments (6)
  1. Section 5: The numerical experiments use γ_k → 0 (γ_k ∝ ξ^k in Section 5.1, γ_k ∝ k^{−1} in Section 5.2), which are not covered by the geometric convergence guarantee even after the fix above. The authors should clarify that these are practical heuristics and that the theoretical guarantee applies to constant or bounded-away-from-zero step sizes.
  2. Remark 2.7: The comparison with the existing RVI algorithm's convergence rate is useful, but it would help to note explicitly that the existing algorithm's multi-step span-semi-norm contraction (Eq. 2.24) and the proposed algorithms' one-step weighted-norm contraction (Eq. 2.23) are not directly comparable in terms of practical convergence speed, as confirmed by the numerical results in Table 1 where neither algorithm uniformly dominates.
  3. Equation (2.17): The definition of η involves p(i,j) := inf_{u∈U} p(i,j,u), but the notation p(i,j) is also used for the transition probability p(i,j,u) with u suppressed. Consider using a distinct symbol (e.g., p̄(i,j)) for the infimum to avoid ambiguity.
  4. Proposition 3.2, proof: The construction of the transformed DTMC with law Q* (Eq. 3.18) and the tail bound (3.24–3.25) are correct, but the inductive argument in (3.25) could benefit from explicitly stating the base case reference (it references (3.24) implicitly).
  5. Appendix A: The notation table is helpful. Consider adding entries for χ(·) and η̄ (defined in (4.5) and Lemma 4.1), which appear in the Gauss-Seidel analysis.
  6. Section 5.2: The recasting of the exit-rate problem as an ERSC problem is clean. It would be helpful to note that the running cost c(i,u) = log p_0(i,u) may be negative, and confirm that the theory (developed for c: S_U → ℝ_+) extends to this case. The lower bound c̄ in (2.22) would need adjustment.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the derivation chain is self-contained, building on standard external results (entropy variational formula, Tseng's weighted norm) without self-definitional or self-citational loops

full rationale

The paper's main convergence theorem (Theorem 2.2) is derived from three independently established properties: (P1) local contraction of F(·,λ) under a weighted supremum norm (Proposition 3.1), (P2) Lipschitz continuity of F(h,·) (Proposition 4.3), and (P3) local bi-Lipschitz continuity of λ↦h_λ (Proposition 3.2). Each of these is proved from first principles or standard external results. The local contraction (Prop 3.1) uses the entropy variational formula (Proposition B.1, attributed to Dupuis-Ellis [24]) and the weighted norm construction from Tseng [39] (Pg. 293). The bi-Lipschitz property (Prop 3.2) is derived via Dynkin's formula applied to a transformed DTMC, with return-time bounds obtained through tail probability estimates (Eqs. 3.24–3.26) that depend on the original transition kernel and uniform path-probability constants (L, α from Eqs. 3.12–3.13), not on the target result. The convergence proof (Section 4) combines these local properties with Proposition 4.1 (which ensures iterates remain in a bounded set) to establish one-step contraction bounds (2.23). No step in this chain reduces to its inputs by construction. The Bellman equation (Theorem 2.1, cited from [18]) characterizes optimal policies but is not used to define the operators F, G or their fixed points in a circular manner—rather, the operators are defined independently (Eqs. 2.6, 2.8) and their fixed points are shown to coincide with the Bellman solution. The skeptic's concern about step-size sequences γ_k→0 potentially degrading the geometric rate is a correctness/generality issue, not a circularity issue: the one-step bounds (2.23) are genuinely derived, not tautological.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new mathematical entities or postulated objects. The operators F, G, and the weighted norm are constructions built from standard ingredients (entropy variational formula, Tseng's norm). The algorithms are new combinations of known components (line-search + Bellman-like operator + Jacobi/Gauss-Seidel structure). All axioms are either standard mathematical results or domain assumptions standard in the ERSC literature.

free parameters (3)
  • γ_k (step-size sequence)
    User-specified step-sizes; the paper proves convergence for γ_k ∈ (0, γ̄] but the specific values in the numerical examples (γ_k = 0.95k̂, γ_k = 0.1k̂⁻¹, etc.) are chosen by experimentation (Section 5).
  • m (weighted norm parameter)
    Defined in Eq. 4.1/4.13 in terms of initial conditions (h⁰, λ⁰) and problem data; controls the locality radius for contraction. Not fitted to data but depends on initialization.
  • n (reference state)
    Fixed reference state for the RVI normalization; chosen as state n by convention. Not a fitted parameter but a structural choice.
assumptions (4)
  • domain assumption Assumption 2.1: The DTMC is irreducible under all stationary Markov policies, and c(i,u) and p(i,j,u) are continuous in u.
    Invoked throughout; ensures existence of optimal stationary policies (Theorem 2.1) and uniform recurrence bounds (τ ≤ τ < ∞). Without this, the Bellman equation may not have a solution and the return time bounds fail.
  • standard math Theorem 2.1 (Cavazos-Cada and Fernández-Gaucherand 2002): Under Assumption 2.1, there exists V* unique up to multiplicative constant satisfying the multiplicative Bellman equation (Eq. 2.3).
    External theorem providing the target fixed point; the paper's algorithms are designed to find this pair. Not proved here but well-established in the literature.
  • standard math Entropy variational formula (Proposition B.1, from Dupuis-Ellis 1997): log E[e^{f(X)}] = sup_Q [E_Q[f] - R(Q||P)].
    Used in Lemma 3.1 to recast the logarithm in F(h,λ) into a variational form that enables the contraction proof. Standard result in large deviations theory.
  • standard math Weighted supremum norm construction from Tseng (1990), Pg. 293.
    The weights w_i^m (Eq. 2.18) and contraction constant β_m (Eq. 2.19) follow this construction. The partition {S_k} and the bound on the maximizing probability vector (Eq. 3.8–3.10) follow Lemma 3 of Tseng.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jacobi-like relative value iteration algorithms for ergodic risk-sensitive control of Markov chains." pith.science (2026). https://pith.science/paper/2UH2LZ46

@misc{pith2026260706863,
  author       = {Pith},
  title        = {Pith review of: Jacobi-like relative value iteration algorithms for ergodic risk-sensitive control of Markov chains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UH2LZ46}},
  note         = {Machine review of arXiv:2607.06863}
}
read the original abstract

We propose a Jacobi-like relative value iteration (RVI) algorithm and a Gauss-Seidel-like implementation for the ergodic risk-sensitive control (ERSC) problem of a controlled discrete time Markov chain (DTMC) on a finite state space. Under the assumption that the DTMC is irreducible and recurrent under every stationary Markov policy, we prove that the iterates of the proposed RVI algorithms converge at a geometric rate. The main challenge stems from the multiplicative structure of the ERSC cost criterion and the associated Bellman-like operators, which prevents us from adapting the analogous global contraction and bi-Lipschitz continuity properties that underlie the proof of convergence in the average cost setting. We overcome this by establishing local contraction properties for the risk-sensitive Bellman-like operators and a local bi-Lipschitz continuity property for their fixed points, and use these properties to show the iterates converge geometrically. We conclude by implementing our proposed RVI algorithms on two examples: service effort control for a single-server queue of finite capacity, and maximizing the exit rate from a finite domain (on a graph).

Figures

Figures reproduced from arXiv: 2607.06863 by the authors.

Figure 1
Figure 1. Plot of λ k vs the iteration number k for the single server queue with a maximum capacity n = 20, α = 0.4, U = {0.1, 0.25, 0.4, 0.5, 0.75, 0.9}, c(i, s) = 5(i−1)+ +0.25s 2 , γk = 0.95kˆ (with updating thresholds θ = 0.75, 0.85 for Algorithms 1 and 2, respectively), and δ = 1 × 10−2 . The existing RVI algorithm and Algorithms 1 and 2 fall within the tolerance of ε = 1 × 10−4 , after 120, 119, and 105 iterations, resp… view at source ↗
Figure 2
Figure 2. Plot of −λ k vs the iteration number k for graphs with two different connectivities and parameters n = 20, R = 10, U = {1, 1.3, 1.6, 1.9}, and m = 5. On the left, the graph is complete and the iterates of the existing RVI algorithm and Algorithms 1 and 2 fall within the tolerance of ε = 1 × 10−4 , after 18, 104 and 88 iterations, respectively. On the right, the graph is neither complete nor a tree and the iterates o… view at source ↗
Figure 3
Figure 3. Plot of −λ k vs the iteration number k for graphs with two different connectivities and parameters n = 20, R = 10, U = {1, 1.3, 1.6, 1.9}, and m = 5, where both problems have undergone the aperiodicity transformation with aperiodicity coefficient κ = 0.05. On the left, the graph is complete and the iterates of the existing RVI algorithm and Algorithms 1 and 2 fall within the tolerance of ε = 1 × 10−4 , after 12, 71 … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages

  1. [1]

    Arapostathis and V

    A. Arapostathis and V. S. Borkar. On the relative value iteration with a risk-sensitive criterion. Banach Center Publications , 122:9–24, 2020

    work page 2020

  2. [2]

    Arapostathis, V

    A. Arapostathis, V. S. Borkar, and K. S. Kumar. Relative value iteration for stochastic differential games. In V. Křivan and G. Zaccour, editors, Advances in Dynamic Games: Theory, Applications, and Numerical Methods , Annals of the International Society of Dynamic Games 13, pages 3–27. Springer International Publishing, 2013

    work page 2013

  3. [3]

    Barz and K

    C. Barz and K. Waldmann. Risk-sensitive capacity control in revenue management. Mathe- matical Methods of Operations Research , 65:565–579, 2007

    work page 2007

  4. [4]

    A. Basu, T. Bhattacharyya, and V. S. Borkar. A learning algorithm for risk-sensitive cost. Mathematics of Operations Research , 33(4):880–898, 2008

    work page 2008

  5. [5]

    D. P. Bertsekas. A new value iteration method for the average cost dynamic programming problem. SIAM Journal on Control and Optimization , 36(2):742–759, 1998

    work page 1998

  6. [6]

    D. P. Bertsekas. Dynamic programming and optimal control . Athena Scientific, 2005

    work page 2005

  7. [7]

    Bielecki, D

    T. Bielecki, D. Hernandez-Hernandez, and S. Pliska. Value iteration for controlled Markov chains with risk sensitive cost criterion. Proceedings of the 38th IEEE Conference on Decision and Control , 1:126–130, 1999

    work page 1999

  8. [8]

    Bielecki, D

    T. Bielecki, D. Hernández-Hernández, and S. Pliska. Risk sensitive control of finite state Markov chains in discrete time, with applications to portfolio management. Mathematical Methods of Operations Research , 50:167–188, 1999

    work page 1999

Show all 41 references
  1. [9]

    Bielecki and S

    T. Bielecki and S. Pliska. Risk-sensitive dynamic asset management. Applied Mathematics and Optimization, 39:337–360, 1999

    1999

  2. [10]

    Bielecki, S

    T. Bielecki, S. Pliska, and S.-J. Sheu. Risk sensitive portfolio management with Cox–Ingersoll– Ross interest rates: the HJB equation. SIAM Journal on Control and Optimization , 44(5):1811– 1843, 2005

    2005

  3. [11]

    Biswas and V

    A. Biswas and V. S. Borkar. Ergodic risk-sensitive control—a survey. Annual Reviews in Control, 55:118–141, 2023

    2023

  4. [12]

    Biswas and S

    A. Biswas and S. Pradhan. Ergodic risk-sensitive control of Markov processes on countable state space revisited. ESAIM: Control, Optimisation and Calculus of Variations , 28, 2021. R VI ALGORITHMS FOR ERGODIC RISK-SENSITIVE CONTROL OF MARKOV CHAINS 29

    2021

  5. [13]

    V. Borkar. A sensitivity formula for risk-sensitive cost and the actor–critic algorithm. Systems & Control Letters , 44(5):339–346, 2001

    2001

  6. [14]

    V. S. Borkar. Q-learning for risk-sensitive control. Mathematics of Operations Research , 27(2):294–311, 2002

    2002

  7. [15]

    V. S. Borkar and S. P. Meyn. Risk-sensitive optimal control for Markov decision processes with monotone cost. Mathematics of Operations Research , 27(1):192–209, 2002

    2002

  8. [16]

    Bouakiz and M

    M. Bouakiz and M. J. Sobel. Inventory control with an exponential utility criterion. Operations Research, 40(3):603–608, 1992

    1992

  9. [17]

    Bäuerle and A

    N. Bäuerle and A. Jaśkiewicz. Markov decision processes with risk-sensitive criteria: An overview. Mathematical Methods of Operations Research , 99:141–178, 2024

    2024

  10. [18]

    Cavazos-Cadena and E

    R. Cavazos-Cadena and E. Fernández-Gaucherand. Risk-sensitive optimal control in communi- cating average Markov decision chains. In M. Dror, P. L’Ecuyer, and F. Szidarovszky, editors, Modeling Uncertainty: An Examination of Stochastic Theory, Methods, and Applications , In- te...

    2002

  11. [19]

    Cavazos-Cadena and R

    R. Cavazos-Cadena and R. Montes-de Oca. The value iteration algorithm in risk-sensitive average Markov decision chains with finite state space. Mathematics of Operations Research , 28(4):752–776, 2003

    2003

  12. [20]

    Chen and Q

    X. Chen and Q. Wei. Risk-sensitive average optimality for discrete-time Markov decision processes. SIAM Journal on Control and Optimization , 61(1):72–104, 2023

    2023

  13. [21]

    Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: a cvar optimization approach. In Advances in Neural Information Processing Systems 28 , pages 1522–1530, 2015

    2015

  14. [22]

    S. P. Coraluppi and S. I. Marcus. Risk-sensitive and minimax control of discrete-time, finite- state Markov decision processes. Automatica, 35(2):301–309, 1999

    1999

  15. [23]

    M. H. Davis. Mathematical finance: a very short introduction . Oxford University Press, 2019

    2019

  16. [24]

    Dupuis and R

    P. Dupuis and R. S. Ellis. A weak convergence approach to the theory of large deviations . John Wiley & Sons, 1997

    1997

  17. [25]

    Dupuis, M

    P. Dupuis, M. James, and I. Petersen. Robust properties of risk-sensitive control. Mathematics of Control, Signals, and Systems , 13:318–332, 2000

    2000

  18. [26]

    Feng and B

    Y. Feng and B. Xiao. A risk-sensitive model for managing perishable products. Operations Research, 56(5):1305–1311, 2008

    2008

  19. [27]

    W. H. Fleming and D. Hernández-Hernández. Risk-sensitive control of finite state machines on an infinite horizon i. SIAM Journal on Control and Optimization , 35(5):1790–1810, 1997

    1997

  20. [28]

    W. H. Fleming and S. Sheu. Optimal long term growth rate of expected utility of wealth. The Annals of Applied Probability , 9(3):871–903, 1999

    1999

  21. [29]

    S. Guin, V. S. Borkar, and S. Bhatnagar. An actor–critic algorithm with function approx- imation for risk sensitive cost Markov decision processes. IEEE Transactions on Automatic Control, 71(1):474–481, 2026

    2026

  22. [30]

    S. C. Jaquette. A utility criterion for markov decision processes. Management Science , 23(1):43–49, 1976

    1976

  23. [31]

    D. A. Levin, Y. Peres, and E. L. Wilmer. Markov Chains and Mixing Times . American Mathematical Society, 2008

    2008

  24. [32]

    X. Li, H. Zhong, and M. L. Brandeau. Quantile Markov decision processes. Operations Re- search, 70(3):1428–1447, 2022

    2022

  25. [33]

    Moharrami, Y

    M. Moharrami, Y. Murthy, A. Roy, and R. Srikant. A policy gradient algorithm for the risk- sensitive exponential cost mdp. Mathematics of Operations Research , 50(1):431–458, 2024

    2024

  26. [34]

    Murthy, M

    Y. Murthy, M. Moharrami, and R. Srikant. On the convergence of modified policy iteration in risk sensitive exponential cost Markov decision processes. Forthcoming in Operations Research, 2025. 30 SUMITH REDDY, GUODONG PANG, AND NICOLA SASSONE

    2025

  27. [35]

    Nagai and S

    H. Nagai and S. Peng. Risk-sensitive dynamic portfolio optimization with partial information on infinite time horizon. The Annals of Applied Probability , 12(1):173–195, 2002

    2002

  28. [36]

    Noorani, C

    E. Noorani, C. Mavridis, and J. Baras. Risk-sensitive reinforcement learning with exponential criteria. IEEE Transactions on Cybernetics , 55(8):3774–3787, 2025

    2025

  29. [37]

    I. R. Petersen, M. R. James, and P. Dupuis. Minimax optimal control of stochastic uncer- tain systems with relative entropy constraints. IEEE Transactions on Automatic Control , 45(3):398–412, 2002

    2002

  30. [38]

    M. J. Sobel. Mean-variance tradeoffs in an undiscounted MDP. Operations Research, 42(1):175– 183, 1994

    1994

  31. [39]

    P. Tseng. Solving H-horizon, stationary Markov decision problems in time proportional to log(H). Operations Research Letters, 9(5):287–297, 1990

    1990

  32. [40]

    D. White. Dynamic programming, Markov chains, and the method of successive approxima- tions. Journal of Mathematical Analysis and Applications , 6(3):373–376, 1963

    1963

  33. [41]

    Wu and R

    Z. Wu and R. Xu. Risk-sensitive Markov decision process and learning under general utility functions. arXiv:2311.13589, 2023. R VI ALGORITHMS FOR ERGODIC RISK-SENSITIVE CONTROL OF MARKOV CHAINS 31 Appendix A. Summary of key notation and constants Λ Set of parameters λ for whic...

    2023 arXiv

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.