REVIEW 4 cited by
Reinforcement Learning via Fenchel-Rockafellar Duality
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We review basic concepts of convex duality, focusing on the very general and supremely useful Fenchel-Rockafellar duality. We summarize how this duality may be applied to a variety of reinforcement learning (RL) settings, including policy evaluation or optimization, online or offline learning, and discounted or undiscounted rewards. The derivations yield a number of intriguing results, including the ability to perform policy evaluation and on-policy policy gradient with behavior-agnostic offline data and methods to learn a policy via max-likelihood optimization. Although many of these results have appeared previously in various forms, we provide a unified treatment and perspective on these results, which we hope will enable researchers to better use and apply the tools of convex duality to make further progress in RL.
Forward citations
Cited by 4 Pith papers
-
Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Maps
Q-DOT uses gradient maps of input-convex neural networks to compute Wasserstein regularization in offline RL, achieving D4RL scores comparable to or better than IQL without adversarial training.
-
Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL
VAC is a new actor-critic method with a single optimistic objective and a provably near-optimal regret bound in linear Markov decision processes.
-
The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability
An optimism-based algorithm with nonparametric instrumental variables learns an epsilon-optimal policy under information asymmetry and knowledge transfer with O~(1/epsilon^2) sample complexity.
-
Semi-gradient DICE for Offline Constrained Reinforcement Learning
Semi-gradient DICE outputs a policy correction instead of a stationary distribution correction, and CORSDICE recovers the latter to enable accurate cost estimation and safe offline constrained RL.
Discussion (0). Continue with ORCID to comment.