Pith. sign in

REVIEW 2 major objections 4 minor 52 references

When Maximum Entropy Misleads Policy Optimization

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's entropy bifurcation extension can make the maximum-entropy-optimal policy at a chosen state match any arbitrary action distribution — even the worst — while leaving all other states' optimal policies untouched.

desk verdict The main theorem overreaches: the constructed Q-function in Lemma B.1 is not realizable under Definition 5.1, so the arbitrary-policy claim fails; the toy example and experiments are still worth a look. read the letter →

arxiv 2506.05615 v1 pith:NHW6WUC6 submitted 2025-06-05 cs.LG cs.AI

Ruipeng Zhang , Ya-Chien Chang , Sicun Gao This is my paper · ORCID
classification cs.LGcs.AI
keywords maximumentropyreinforcementlearningSoftActor-Criticpolicyoptimizationbifurcationextensionrobustness-optimalitytradeoffcontinuouscontrolQ-valuelandscapetrap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the entropy bonus in maximum-entropy reinforcement learning is not merely an exploration aid that costs a little optimality: at convergence it can flip which policy is optimal, and the flip can be made arbitrarily large. Its main theorem constructs, for any Markov decision process and any chosen state, an entropy bifurcation extension — new branching states with tunable rewards — such that the MaxEnt-optimal policy at that state becomes any prescribed action distribution, even the worst one, while the ordinary optimal policy on every other state is unchanged. The same soft-value mechanism is then used to explain why SAC reliably fails on realistic control tasks such as quadrotor trajectory tracking, high-speed vehicle path following, and quadruped locomotion where PPO succeeds, and why SAC excels on exploration-friendly benchmarks. Understanding the mechanism matters because it separates the genuine exploration benefits of entropy from a convergence-level bias that reward design alone cannot remove.

What carries the argument

The carrying object is the Entropy Bifurcation Extension (Definition 5.1): for each successor state $s'$ of the targeted state $s$, insert a new state $s_\mu = \mu(s')$ whose action space splits into a return interval $A_1^\mu$ that leads back to $s'$ and a dead-end interval $A_2^\mu$ that leads to a new terminal state with a tunable reward. Two lemmas do the work. Backward Compatibility (Lemma 5.3) shows that any target distribution $\pi(\cdot|s)$ can be matched by setting $Q(s,a) = \alpha \log \pi(a|s) + v_s$, with the constant $v_s$ chosen so the desired soft value is realized and $D_{\mathrm{KL}}(\pi \| \pi^*_Q) = 0$. Forward Compatibility (Lemma 5.4) shows that the branch widths $|A_1^\mu|$, $|A_2^\mu|$ and the terminal reward $r(s_T^\mu)$ provide enough freedom to realize any target soft value $V(s_\mu)$ without changing the original successor's value $V(s')$. Together the lemmas let the extension reshape the soft-Q landscape at $s$ — and hence the Boltzmann policy that probability-matches it — independently of all other states.

What would settle it

Take a three-state MDP in which two distinct actions at the targeted state $s$ both transition deterministically to the same successor $s'$, and prescribe a target policy $\pi(\cdot|s)$ that assigns these two actions unequal probabilities. In the extension both actions pass through the same inserted state $s_\mu$, so they are forced to share one soft value; substituting the required equations $Q(s,a) = \alpha \log \pi(a|s) + v_s$ for both actions decides whether the arbitrary-policy clause of Theorem 5.5 holds for such MDPs, because unequal target probabilities make the two equations inconsistent.

Watch

Extended reading notes

Core claim

The paper's central claim is that in the maximum-entropy RL framework the soft Bellman backup makes the optimal policy itself a function of the soft Q-value landscape: the MaxEnt-optimal policy at a state is the Boltzmann distribution over $Q(s,a)$, so whoever can shape that landscape controls the policy. The paper proves this control can be exercised essentially arbitrarily. Given any MDP, any state $s$, and any desired distribution $\pi(\cdot|s)$ over actions, there is an entropy bifurcation extension of the MDP in which $\pi(\cdot|s)$ is the MaxEnt-optimal policy at $s$, the soft value $V(s)$ is unchanged, and no other state's optimal policy is altered (Theorem 5.5). Because the construction can be applied at every state independently, the MaxEnt-optimal policy of the extended MDP can realize the worst policy's value while the ground-truth optimal policy retains the best value (Corollary 5.7). The paper emphasizes that this is an effect at convergence of the exact MaxEnt objective, not an artifact of function approximation, exploration noise, or sample efficiency, and it argues with soft-versus-plain Q-value comparisons on learned control tasks that the same bifurcation structure arises naturally when critical states demand precise low-entropy actions and deviations lead to irrecoverable failure.

Load-bearing premise

The construction assumes that every action at the targeted state can be routed through its own new branching state with an independently tunable reward; when several actions share the same successor state, they share one backup value, so a prescribed policy that treats those actions differently may not be realizable.

Editorial extensions

If this is right

  • At convergence the MaxEnt-optimal policy can be arbitrarily worse than the standard optimal policy: applying the extension at every state makes the MaxEnt optimum realize the worst policy's value $J^-$ while the ground-truth optimum keeps $J^+$ (Corollary 5.7).
  • The misleading effect is intrinsic to the soft Bellman objective, so it survives perfect function approximation and infinite samples; it is a property of the objective, not of training dynamics.
  • Control tasks whose critical states require narrow, low-entropy actions (quadrotors, high-speed vehicles, quadrupeds) will systematically favor high-entropy 'mediocre' branches that drift into irrecoverable failure, which the paper argues explains the SAC failures where PPO succeeds.
  • The same mechanism explains MaxEnt successes: on exploration-friendly tasks such as Hopper, Acrobot, and Obstacle2D, entropy-smoothed Q-landscapes push the policy through risky but promising moves, whereas plain advantage landscapes trap PPO in zero-level sets.
  • Switching the policy update from soft Q to plain Q where the two landscapes diverge (SAC-AdaEnt) recovers performance in the failing environments, confirming the divergence as the cause; the authors present this as a diagnostic rather than a scalable algorithm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Following the paper's closing suggestion about human-feedback scenarios, the extension reads as a recipe for a stealthy policy attack: an adversary who can add states or shape rewards — for example through a learned reward model — can steer the MaxEnt policy to any target distribution while leaving the return-optimal policy untouched.
  • A natural repair for the shared-successor gap is to first split shared successors at the targeted state (one copy per action) before applying the extension; if the arbitrary-distribution theorem then holds verbatim, the shared-backup constraint is confirmed as the sole obstruction, and a refined theorem should be tested in that setting.
  • The soft-versus-plain Q discrepancy that SAC-AdaEnt exploits is measurable online, so the same idea could serve as a monitoring signal in deployed control systems: when the two landscapes diverge sharply at a state, the agent is at risk of an entropy trap and should fall back to plain-value updates or raise the cost of exploration there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that maximum-entropy RL can be systematically misled in control problems that require low-entropy policies. It presents a toy MDP, a general construction called the Entropy Bifurcation Extension, and a theorem (Theorem 5.5) claiming that for any MDP the extension can make an arbitrary policy at a target state the MaxEnt-optimal policy while leaving all other states' optimal policies unchanged. It then reports qualitative experiments comparing SAC and PPO on vehicle, quadrotor, quadruped, and benchmark tasks, plus an adaptive-entropy variant (SAC-AdaEnt). The toy calculation in Appendix A.1 is analytically correct, but the proof of the general theorem contains a load-bearing realizability gap: the Q-function constructed in Lemma B.1 is not shown to be realizable under the transition structure of Definition 5.1, and in the simplest MDPs it cannot be.

Significance. If Theorem 5.5 were true, it would be an important formal caveat: MaxEnt methods could converge to arbitrarily bad policies while standard RL remains unaffected, with direct implications for SAC and related algorithms. The paper also contains a useful pedagogical toy example and plausible qualitative evidence that entropy regularization can hurt in tasks with narrow feasible action regions. However, the central theoretical claim is unsound as stated. The toy example and experiments do not repair the gap, because the experiments are not instances of the formal construction and the failure of the theorem is structural rather than a matter of proof detail.

major comments (2)
  1. [Section 5, Definition 5.1; Appendix B, Lemma B.1] The proof of Lemma B.1 constructs Q(s,a)=α log π(a|s)+v_s (Eq. (5) in the appendix) and declares the target policy MaxEnt-optimal, but it never verifies that this Q is a Bellman backup in the extended MDP. Under Definition 5.1, every action with positive transition to a given successor s' is rerouted to the same new state s_μ=μ(s'), so the actual backup at s is Q(s,a)=r_M(s,a)+γV(s_μ). When r_M is constant over the actions mapping to s', the induced Q is constant on that class and the MaxEnt-optimal policy is uniform within it. The target π can be recovered only if π happens to be uniform on every successor class, or if r_M happens to reproduce it, but r_M is fixed by the original MDP. Concretely, for M with S={s,s_T}, A={a1,a2}, P(s_T|s,a_i)=1, and r≡0, every extension under Definition 5.1 gives Q(s,a1)=Q(s,a2), so the MaxEnt policy is (1/2,1/2); no choice of |A_1^μ|, |A_2^μ|, or r(s_T^μ) can achieve π=(0.9,0.1). This falsifies Theorem 5.5 and the worst-policy corollary (Corollary 5.7).
  2. [Appendix B, Theorem B.3] The composition step V(s_μ)=Q(s,a)/P(s'|s,a) is not a valid definition. For a fixed successor class, Q(s,a) must be constant over a for the construction to be realizable, while the quotient depends on a through P(s'|s,a); if two actions in the class have different transition probabilities, the same state s_μ receives conflicting target values. The expression is also undefined for transitions with probability zero, and the proof does not state how such actions are handled. The forward-compatibility lemma cannot absorb this inconsistency, since the target value imposed on s_μ must be single-valued.
minor comments (4)
  1. [Appendix B, Lemma B.1] The proof refers to 'Definition 4' for the soft value, but no Definition 4 appears in the main text; the definition appears only as Eq. (4) inside the lemma's proof.
  2. [Section 5, Notation 5.2] The set of extensions is written E_s(M), whereas Definition 5.1 writes E(M,s); the notation should be unified.
  3. [Theorem 5.5] The theorem statement says 'Let M be an MDP', but Definition 5.1 assumes a continuous action space; the theorem should state the same assumption, or the definition should be extended to discrete actions.
  4. [Section 6.3] The sentence 'the simple change of SAC-MaxEnt is not intended as a new efficient algorithm' appears to refer to SAC-AdaEnt and should be corrected.

Circularity Check

2 steps flagged · score 8.0 of 10

Theorem 5.5's arbitrary-policy conclusion is obtained by defining Q(s,a)=α log π(a|s)+v_s — the inverse of the softmax definition of the MaxEnt-optimal policy — rather than by constructing an Entropic Bifurcation Extension that realizes that Q.

  1. self definitional [Appendix B, Lemma B.1 (proof of Lemma 5.3), Eq. (5); see also Eq. (3)]
    "To ensure DKL(π(· |s) ∥ π∗Q) = 0, we can directly construct Q(s, a) such that it matches the optimal policy π∗Q(a|s) = exp(α−1Q(s, a))/Z(Q). ... Q(s, a) = α log π(a | s) + α log Z(s)."

    The lemma's conclusion — that an arbitrary policy π(·|s) is MaxEnt-optimal — is obtained by choosing Q(s,a) to be the inverse of Eq. (3), the definition of the Boltzmann policy. Eq. (3) defines π*_Q as softmax(Q/α); therefore any Q of this form makes π = π*_Q by construction. The proof never checks that this Q is realizable by the Bellman equation of Definition 5.1, which forces Q(s,a)=r_M(s,a)+γV(μ(s')) for the new state μ(s') shared by all actions with the same successor s'. Thus the theorem's core claim reduces to the freedom to pick the input π and then declare it Boltzmann-optimal for an arbitrary Q.

  2. other [Appendix B, Theorem B.3 proof]
    "Given such target Q(s, a), which now impose target values on the introduced bifurcation states, i.e., V(sµ) = Q(s, a)/P(s′|s, a), because by construction P(sµ|s, a) = P(s′|s, a) > 0."

    This step assumes that the arbitrary Q(s,a) from Lemma B.1 is automatically realizable by the extension. Under Definition 5.1, all actions with the same successor s′ are rerouted through the same new state sμ=μ(s′), so their Bellman backup is V(sμ); hence Q(s,a)=r_M(s,a)+γV(sμ), which is constant over each successor class when r_M is constant. The target Q(s,a)=α log π(a|s)+v_s is arbitrary and generally varies within the class, so the equation V(sμ)=Q(s,a)/P(s′|s,a) is not forced by the construction and is not even well-defined when multiple actions share the successor. The proof therefore takes the desired realizability as an input rather than deriving it from the extension's parameters, making the arbitrary-policy conclusion circular.

full rationale

The paper is not circular through fitted data or self-citation: the experiments are used post hoc to illustrate the theory, and the cited prior work is external. The circularity is in the main existence proof. Lemma B.1 proves backward compatibility by explicitly constructing Q(s,a)=α log π(a|s)+v_s, which is the exact inverse of the definition of the MaxEnt-optimal Boltzmann policy (Eq. 3). That makes the target policy optimal by definition, not by the structure of the Entropic Bifurcation Extension. The subsequent theorem proof then assumes this Q imposes values on the newly introduced states V(sμ)=Q(s,a)/P(s′|s,a), without verifying the Bellman constraint of Definition 5.1: all actions mapping to the same successor s′ share one state sμ, so Q(s,a) cannot be an arbitrary function of a. The central claim of Theorem 5.5 therefore reduces, as written, to the construction of an arbitrary Q-function whose softmax is the desired π, plus an unverified assertion that the extension parameters can realize it. This is a definitional reduction of the main result rather than a derivation from the extension's tunable parameters. The remaining content — toy example, experiments, adaptive-entropy variant — is independent and non-circular, but it does not repair the central theorem's proof. Score 8 reflects that the main result is forced by construction in the proof, not by external evidence or fitted values.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The formal result relies on a construction that adds new states and tunes rewards. The paper does not fit numerical parameters to data, so the circularity burden is low. However, the theorem implicitly assumes a compatibility between the constructed Q-function and the one-state-per-successor wiring that is not stated and is false for general MDPs.

free parameters (1)
  • Bifurcation interval sizes |A_mu_1| and |A_mu_2|, and new-terminal reward r(s_mu_T)
    Existential degrees of freedom used in Lemma 5.4 to hit arbitrary target soft values at the bifurcating state. They are chosen for the proof, not fitted to data, but the central construction depends on them.
assumptions (4)
  • standard math The MaxEnt optimal policy is exactly the Boltzmann distribution over soft Q-values, as in Eq. 3.
    Used throughout to characterize the policy at convergence; this is the standard MaxEnt RL identity and is not in dispute.
  • ad hoc to paper Definition 5.1 creates exactly one new state per successor state, redirecting all actions that led to that successor to the same new state.
    This is the load-bearing structural constraint that forces Q(s,a) to be constant over actions sharing a successor, which conflicts with the arbitrary policy claim.
  • ad hoc to paper The constructed Q(s,a)=alpha log pi(a|s)+vs can be realized as a Bellman backup through the newly introduced states.
    Lemma B.1 assumes this realizability without proving it. It is not guaranteed by the MDP extension in Definition 5.1 unless actions are injective in their successors.
  • domain assumption One may add arbitrary new states to an MDP and freely set their transition rewards.
    This is what makes entropy bifurcation extensions possible at all; it is reasonable for a mathematical construction but limits direct practical translation.
invented entities (1)
  • Entropy bifurcation states s_mu and terminal states s_mu_T
    purpose: Intermediate states that reroute transitions and shape soft-Q landscapes to make a chosen policy MaxEnt-optimal
    These states exist only inside the mathematical construction of Theorem 5.5. They are not physical entities and the paper does not claim they exist in real systems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Maximum Entropy Misleads Policy Optimization." pith.science (2026). https://pith.science/paper/NHW6WUC6

@misc{pith2026250605615,
  author       = {Pith},
  title        = {Pith review of: When Maximum Entropy Misleads Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHW6WUC6}},
  note         = {Machine review of arXiv:2506.05615}
}
read the original abstract

The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework is a leading approach for achieving efficient learning and robust performance across many RL tasks. However, MaxEnt methods have also been shown to struggle with performance-critical control problems in practice, where non-MaxEnt algorithms can successfully learn. In this work, we analyze how the trade-off between robustness and optimality affects the performance of MaxEnt algorithms in complex control tasks: while entropy maximization enhances exploration and robustness, it can also mislead policy optimization, leading to failure in tasks that require precise, low-entropy policies. Through experiments on a variety of control problems, we concretely demonstrate this misleading effect. Our analysis leads to better understanding of how to balance reward design and entropy maximization in challenging control problems.

Figures

Figures reproduced from arXiv: 2506.05615 by the authors.

Figure 1
Figure 1. (Upper) In the quadrotor control environment, SAC learns well under simplified dynamics, but fails to learn when under realistic dynamics models. PPO can learn well despite the use of the latter. (Lower) Intuitive illustration of hard control prob￾lems, where critical states naturally require low-entropy policies, while MaxEnt RL can favor mediocre states with robust policies of low returns that branch out towards f… view at source ↗
Figure 2
Figure 2. (Left) MDP in the Toy Example: The MDP consists of an initial state s0 and two subsequent states sg (good) and sb (bad). It is clear that an optimal policy for s0 should be centered in the left half of the action interval, since only sg can transit to the terminal state s + T with positive reward. (Right) Learning results of SAC and PPO at s0 at convergence. In the SAC plot, the soft Q-values Q(s0, a) is higher for … view at source ↗
Figure 3
Figure 3. MDP M and its entropic bifurcation extension Mˆ . The extension captures the intuition in the toy example, by using ad￾ditional intermediate states which specifically designed reward to mislead MaxEnt-optimal policies that match the soft-Q landscapes. 1. We write N (s) = {s ′ |P(s ′ |s, a) > 0 for some a ∈ AM} to denote the set of next states with non-zero transition probability from s. We introduce new states as fo… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Reward performance of SAC and PPO across five environments with five random seeds. Note that we choose to show SAC and PPO because they are the best representatives of MaxEnt and non-MaxEnt algorithms. MaxEnt algorithms, both when they fail in complex control tasks and…
Figure 5
Figure 5. Figure 5: Soft and Plain Q-value landscapes in Vehicle. (Left) Q landscapes with Qsoft(s, a) and without Qplain(s, a) entropy. Introduced Entropy in SAC elevates the true Q values to encour￾age exploration, risking missing the only feasible optimal actions. (Right) Rendering of …
Figure 7
Figure 7. Figure 7: (Upper) Comparison of soft-Q (left) and plain-Q (right) value landscapes at the current state shown below. (Lower) The second to fourth snapshots show Hopper’s state after taking actions at the circled position of corresponding colors in the action space shown above. T…
Figure 6
Figure 6. Figure 6: Q landscapes in Quadrotor. (Left) The current state is at the end of the black trajectory. The Red dashed line is the target track. (Right) Qsoft and Qplain at this state. SAC fails to push upward with minimal action at this state, leading to failure against gravity. P…
Figure 8
Figure 8. Figure 8: Q/Advantage landscapes of SAC and PPO in Obstacle2D and Acrobot. (Upper) In the Obstacle2D environment, SAC successfully bypasses the wall while PPO fails, as explained by the Soft-Q/Advantage landscapes. (Lower) In Acrobot, SAC learns a more stable control policy (app…
Figure 9
Figure 9. Figure 9: Performance of SAC-AdaEnt v.s. SAC. (Left) Learning curves. (Middle) Full trajectory rendering. (Right) Behavior of policy on critic states. In Vehicle, SAC-AdaEnt successfully steers and brakes to bring the vehicle back on track. In Quadrotor, it effectively lifts the…
Figure 10
Figure 10. Figure 10: Toy Example Results of SAC and PPO at states sg and sb In the MaxEnt framework, the policy at s0 is iteratively updated towards the Boltzmann distribution π ∗ Q(·|s0). Given the simple transitions in the MDP, we can easily calculate the Q values for any action. We use…
Figure 11
Figure 11. Figure 11: Performance of All Algorithms across six environments area because of the instant approaching reward. The advantage landscape reveals that PPO’s policy moves to the center of the positive advantage region but remains confined by the zero-level set. Notably, although t…
Figure 12
Figure 12. Figure 12: Q landscapes in Hopper. Upper: We set torque #0 (top torso) as the current µ SAC 0 to plot Qsoft and Qplain for torque #1 (middle thigh) and #2 (bottom leg) in the state shown in the bottom-left figure. Lower: Rendered hopper’s gestures result from the corresponding p…
Figure 13
Figure 13. Figure 13: Q/Advantage landscapes of SAC and PPO in Obstacle2D and Acrobot. Upper: In Obstacle2D with start (0, 0), goal (3, 0), and a wall at x = 2 spanning y = [−2, 2], SAC succeeds in bypassing the wall whereas PPO fails. We plot the Q/Advantage landscape of the initial state…
Figure 14
Figure 14. Figure 14: Advantage landscapes in Obstacle2D for PPO. (a) to (f) show the advantage landscapes at different training stages for the initial state, where PPO’s policy center remains trapped in front of the wall while σ gradually shrinks. 20 [PITH_FULL_IMAGE:figures/full_fig_p02…
Figure 15
Figure 15. Figure 15: Performance of SAC-AdaEnt v.s. SAC. Left: Reward Improvement. Middle: Full trajectory rendering. Right: Behavior of policy on critic states. In Vehicle, SAC-AdaEnt successfully steers and brakes to bring the vehicle back on track, while in Quadrotor, it effectively li…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 28 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M

    Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018

    arXiv 2018

  3. [3]

    Spinning Up in Deep Reinforcement Learning

    Achiam, J. Spinning Up in Deep Reinforcement Learning . 2018. URL https://github.com/openai/spinningup

    work page 2018

  4. [4]

    Understanding the impact of entropy on policy optimization

    Ahmed, Z., Le Roux, N., Norouzi, M., and Schuurmans, D. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pp.\ 151--160. PMLR, 2019

    work page 2019

  5. [5]

    Openai gym

    Brockman, G. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  6. [6]

    Maximum entropy reinforcement learning via energy-based normalizing flow

    Chao, C.-H., Feng, C., Sun, W.-F., Lee, C.-K., See, S., and Lee, C.-Y. Maximum entropy reinforcement learning via energy-based normalizing flow. arXiv preprint arXiv:2405.13629, 2024

    arXiv 2024

  7. [7]

    and Levine, S

    Eysenbach, B. and Levine, S. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257, 2021

    arXiv 2021

  8. [8]

    Taming the noise in reinforcement learning via soft updates

    Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562, 2015

    arXiv 2015

Show all 52 references
  1. [9]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  2. [10]

    Gr \"u nwald, P. D. and Dawid, A. P. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. 2004

    2004

  3. [11]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research,...

    2018

  4. [12]

    Soft actor-critic algorithms and applications

    Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018 b

    2018 arXiv

  5. [13]

    and Sung, Y

    Han, S. and Sung, Y. A max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 25732--25745, 2021

    2021

  6. [14]

    Provably efficient maximum entropy exploration

    Hazan, E., Kakade, S., Singh, K., and Van Soest, A. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pp.\ 2681--2691. PMLR, 2019

    2019

  7. [15]

    Huang, S., Gallouédec, Q., Felten, F., Raffin, A., Dossa, R. F. J., Zhao, Y., Sullivan, R., Makoviychuk, V., Makoviichuk, D., Danesh, M. H., Roumégous, C., Weng, J., Chen, C., Rahman, M. M., M. Araújo, J. G., Quan, G., Tan, D., Klein, T., Charakorn, R., Towers, M., Berthelot, ...

    2024 arXiv

  8. [16]

    Champion-level drone racing using deep reinforcement learning

    Kaufmann, E., Bauersfeld, L., Loquercio, A., M \"u ller, M., Koltun, V., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning. Nature, 620 0 (7976): 0 982--987, 2023

    2023

  9. [17]

    and Sung, Y

    Kim, W. and Sung, Y. An adaptive entropy-regularization framework for multi-agent reinforcement learning. In International Conference on Machine Learning, pp.\ 16829--16852. PMLR, 2023

    2023

  10. [18]

    Kinematic and dynamic vehicle models for autonomous driving control design

    Kong, J., Pfeiffer, M., Schildbach, G., and Borrelli, F. Kinematic and dynamic vehicle models for autonomous driving control design. In 2015 IEEE intelligent vehicles symposium (IV), pp.\ 1094--1099. IEEE, 2015

    2015

  11. [19]

    Lee, M. H. and Moon, J. Deep reinforcement learning-based uav navigation and control: A soft actor-critic with hindsight experience replay approach. arXiv preprint arXiv:2106.01016, 2021

    2021 arXiv

  12. [20]

    Reinforcement learning and control as probabilistic inference: Tutorial and review

    Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018

    2018 arXiv

  13. [21]

    Continuous control with deep reinforcement learning

    Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  14. [22]

    Mazoure, B., Doan, T., Durand, A., Pineau, J., and Hjelm, R. D. Leveraging exploration in off-policy algorithms via normalizing flows. In Conference on Robot Learning, pp.\ 430--444. PMLR, 2020

    2020

  15. [23]

    Learning robust perceptive locomotion for quadrupedal robots in the wild

    Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., and Hutter, M. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science robotics, 7 0 (62): 0 eabk2822, 2022

    2022

  16. [24]

    Muzahid, A. J. M., Kamarulzaman, S. F., and Rahman, M. A. Comparison of ppo and sac algorithms towards decision making strategies for collision avoidance among multiple autonomous vehicles. In 2021 International Conference on Software Engineering & Computer Systems and 4th Int...

    2021

  17. [25]

    G., D'Souza, J

    Nair, V. G., D'Souza, J. M., Asha, C., and Rafikh, R. M. A scoping review on unmanned aerial vehicles in disaster management: Challenges and opportunities. Journal of Robotics and Control (JRC), 5 0 (6): 0 1799--1826, 2024

    2024

  18. [26]

    Combining policy gradient and q-learning

    O'Donoghue, B., Munos, R., Kavukcuoglu, K., and Mnih, V. Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626, 2016

    2016 arXiv

  19. [27]

    Opencat: Open-source quadruped robot

    PetoiCamp. Opencat: Open-source quadruped robot. URL https://github.com/PetoiCamp/OpenCat?tab=readme-ov-file

  20. [28]

    O., Sedky, A

    Radwan, M. O., Sedky, A. A. H., and Mahar, K. M. Obstacles avoidance of self-driving vehicle using deep reinforcement learning. In 2021 31st International Conference on Computer Theory and Applications (ICCTA), pp.\ 215--222. IEEE, 2021

    2021

  21. [29]

    Stable-baselines3: Reliable reinforcement learning implementations

    Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 0 (268): 0 1--8, 2021

    2021

  22. [30]

    M., Yang, F., Duchi, J

    Raghunathan, A., Xie, S. M., Yang, F., Duchi, J. C., and Liang, P. Adversarial training can hurt generalization. arXiv preprint arXiv:1906.06032, 2019

    1906 arXiv

  23. [31]

    M., Yang, F., Duchi, J., and Liang, P

    Raghunathan, A., Xie, S. M., Yang, F., Duchi, J., and Liang, P. Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002.10716, 2020

    2002 arXiv

  24. [32]

    On stochastic optimal control and reinforcement learning by approximate inference

    Rawlik, K., Toussaint, M., and Vijayakumar, S. On stochastic optimal control and reinforcement learning by approximate inference. 2013

    2013

  25. [33]

    A survey of path following control strategies for uavs focused on quadrotors

    Rub \' , B., P \'e rez, R., and Morcego, B. A survey of path following control strategies for uavs focused on quadrotors. Journal of Intelligent & Robotic Systems, 98 0 (2): 0 241--265, 2020

    2020

  26. [34]

    Trust region policy optimization

    Schulman, J. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    2015 arXiv

  27. [35]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  28. [36]

    M., Vergara, P

    Shengren, H., Salazar, E. M., Vergara, P. P., and Palensky, P. Performance comparison of deep rl algorithms for energy systems optimal scheduling. In 2022 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), pp.\ 1--6. IEEE, 2022

    2022

  29. [37]

    Spong, M. W. The swing up control problem for the acrobot. IEEE control systems magazine, 15 0 (1): 0 49--55, 1995

    1995

  30. [38]

    Is robustness the cost of accuracy? – a comprehensive study on the robustness of 18 deep image classification models

    Su, D., Zhang, H., Chen, H., Yi, J., Chen, P.-Y., and Gao, Y. Is robustness the cost of accuracy? – a comprehensive study on the robustness of 18 deep image classification models. In Computer Vision – ECCV 2018, pp.\ 644--661. Springer International Publishing, 2018

    2018

  31. [39]

    and Karak \"o se, M

    Tan, Z. and Karak \"o se, M. A new approach for drone tracking with drone using proximal policy optimization based distributed deep reinforcement learning. SoftwareX, 23: 0 101497, 2023

    2023

  32. [40]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.\ 5026--5033. IEEE, 2012

    2012

  33. [41]

    Robot trajectory optimization using approximate inference

    Toussaint, M. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pp.\ 1049--1056, 2009

    2009

  34. [42]

    U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al

    Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  35. [43]

    Robustness may be at odds with accuracy

    Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152, 2018

    2018 arXiv

  36. [44]

    and Ni, T

    Wang, Y. and Ni, T. Meta-sac: Auto-tune the entropy temperature of soft actor-critic via metagradient. arXiv preprint arXiv:2007.01932, 2020

    2007 arXiv

  37. [45]

    Tianshou: A highly modularized deep reinforcement learning library

    Weng, J., Chen, H., Yan, D., You, K., Duburcq, A., Zhang, M., Su, H., and Zhu, J. Tianshou: A highly modularized deep reinforcement learning library. arXiv preprint arXiv:2107.14171, 2021

    2021 arXiv

  38. [46]

    Karting racing: A revisit to ppo and sac algorithm

    Xu, C., Zhu, R., and Yang, D. Karting racing: A revisit to ppo and sac algorithm. In 2021 International Conference on Computer Information Science and Artificial Intelligence (CISAI), pp.\ 310--316. IEEE, 2021

    2021

  39. [47]

    A closer look at accuracy vs

    Yang, Y.-Y., Rashtchian, C., Zhang, H., Salakhutdinov, R., and Chaudhuri, K. A closer look at accuracy vs. robustness. arXiv preprint arXiv:2003.02460, 2020

    2003 arXiv

  40. [48]

    Theoretically principled trade-off between robustness and accuracy

    Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning, pp.\ 7472--7482. PMLR, 2019

    2019

  41. [49]

    Humanoid parkour learning

    Zhuang, Z., Yao, S., and Zhao, H. Humanoid parkour learning. arXiv preprint arXiv:2406.10759, 2024

    2024 arXiv

  42. [50]

    Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010

    2010

  43. [51]

    D., Maas, A

    Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp.\ 1433--1438. Chicago, IL, USA, 2008

    2008

  44. [52]

    D., Bagnell, D., and Dey, A

    Ziebart, B. D., Bagnell, D., and Dey, A. K. Maximum causal entropy correlated equilibria for markov games. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010

    2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.