A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

Karthik Narasimhan; Runzhe Yang; Xingyuan Sun

REVIEW 3 major objections 5 minor 3 cited by

A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Envelope Q-learning converges to a fixed point for all preferences, so one network can serve every user.

desk verdict Envelope Q-learning is a real idea with mostly sound exact-operator theory, but the implemented minibatch target is a biased approximation the proofs don't cover — fixable, and worth refereeing. read the letter →

arxiv 1908.08342 v2 pith:CA6I4MZR submitted 2019-08-21 cs.LG cs.AI

Runzhe Yang , Xingyuan Sun , Karthik Narasimhan This is my paper · ORCID

classification cs.LGcs.AI

keywords multi-objectivereinforcementlearningenvelopeQ-learninglinearpreferencesconvexcoveragesetpolicyadaptationpreferenceinferenceBellmancontractionfew-shot

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that one parametric Q-network can hold optimal policies for every linear preference in a multi-objective MDP, so a trained agent can switch behaviors at test time without retraining. To do that it replaces the scalar Bellman update with an envelope optimality operator that, at each state, takes the convex envelope over both actions and preferences, and it proves this operator is a contraction whose fixed point is the preferred optimal value function. If the proof and experiments hold up, multi-objective RL stops being a search over scalarizations or a collection of separate policies, and becomes a fixed-point problem solvable by a single network that also guesses an unseen preference from a handful of episodes.

What carries the argument

The load-bearing object is the envelope optimality filter $(HQ)(s,\omega)=\arg_Q \sup_{a\in A,\omega'\in\Omega} \omega^\top Q(s,a,\omega')$, which solves the convex envelope of the current Pareto frontier; plugged into $(TQ)(s,a,\omega)=r(s,a)+\gamma\mathbb{E}_{s'}\big[(HQ)(s',\omega)\big]$ it defines the multi-objective Bellman operator. The paper proves $T$ is a $\gamma$-contraction under the pseudo-metric $d$ and uses a generalized Banach fixed-point theorem to conclude convergence; in the implemented algorithm the supremum over $\Omega$ is replaced by a sampled minibatch $W$ of preferences, and the loss is annealed from a mean-squared target loss $L_A$ to a utility-alignment loss $L_B$ by homotopy optimization.

What would settle it

Train the envelope variant on the depth-7 fruit-tree task with $N_\omega=1$ and test every one of the 128 leaf preferences against the known optimum $\max_{r\in CCS}\omega^\top r$; if the sampled minibatch target does not preserve the fixed point of $T$, the coverage ratio and adaptation quality should drop materially below the $N_\omega=128$ runs.

Watch

Extended reading notes

Core claim

The paper establishes that the preferred optimal multi-objective value function $Q^*(s,a,\omega)$ is a fixed point of the envelope Bellman operator $T$, and that $T$ is a $\gamma$-contraction under the preference-weighted pseudo-metric $d(Q,Q')=\sup_{s,a,\omega}|\omega^\top(Q-Q')|$. By the paper's generalized Banach fixed-point theorem in a complete pseudo-metric space, iterating $T$ on any bounded multi-objective Q-value function yields a function with the same utility as $Q^*$ under every preference. The paper argues this justifies a single neural network $Q_\theta(s,\omega)$ trained with envelope targets and hindsight replay, and reports that this envelope MORL algorithm achieves the best coverage ratio, adaptation error, and average utility among the compared methods on deep sea treasure, fruit tree navigation, task-oriented dialog, and Super Mario, and infers hidden one-hot preferences in 15 to 100 episodes.

Load-bearing premise

The convergence proof treats the exact envelope operator over every preference in $\Omega$, but the implemented algorithm samples a small minibatch $W$ of preferences for each update; if sampling breaks the fixed-point property, the guarantee of optimal policies for all preferences does not follow.

Editorial extensions

If this is right

After one training run, the same network can output the optimal action for any specified preference by maximizing $\omega^\top Q(s,a,\omega)$ over actions, with no retraining.
Transitions collected under one sampled preference can be replayed under many other preferences via hindsight replay, so each trajectory contributes to many preference-specific updates.
The contraction result means the convergence rate is governed by the discount factor $\gamma$, independent of how many preferences are represented, as long as the exact envelope operator is used.
Envelope updates can transfer a good solution discovered under one preference to improve the value estimate under a different preference, which scalarized updates cannot do.
After training, an unknown preference can be inferred from scalar rewards alone by searching the parameters of a preference distribution, with the paper reporting 15 episodes in fruit-tree navigation and 100 episodes in Super Mario.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper leaves open is annealing the sampled preference minibatch $W$ from a few points toward a dense or adaptive cover of $\Omega$; if the minibatch approximation converges to the exact envelope operator, the contraction guarantee would transfer to the practical algorithm.
The same envelope-operator idea could be applied to non-linear preference functions by replacing the $\arg_Q \sup$ with a differentiable soft maximum, though the paper proves nothing for that case.
Because the pseudo-metric $d$ collapses vector-valued Q-functions that share the same scalar utility, downstream users of such a network should read only scalarized utilities and not interpret individual components of $Q$ as unique expected returns.
The preference-inference scheme is coupled to policy-gradient search, but nothing in the proof prevents replacing it with off-policy utility estimates, which might reduce the number of adaptation episodes further.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: this is a genuine contribution to MORL, and the main theory is in better shape than the prose suggests. The envelope update — max over both actions and preferences in the bootstrap target — is new relative to scalarized conditional networks (Abels et al.) and MOFQI, and the exact-operator analysis (contraction, fixed point, generalized Banach) mostly holds up. I checked the suspicious cancellation in Theorem 1; it works, because the utility of any preference-optimized policy under ω is dominated by the ω-optimal policy, so optimizing over ω′ in the envelope is harmless. The pseudometric is genuinely a pseudometric. Credit where due: four domains, sensible baselines, an Nω sweep, public code, and the FTN/Mario additions are useful for the community.

The soft spot is exactly the one flagged in the stress test. Algorithm 1 replaces the preference space Ω by a minibatch W in the envelope max, and Theorems 1–3 prove convergence for the exact operator only. Theorem 5 is about minibatching state–action pairs, not truncating the preference set; the supplementary sentence claiming it covers this update is wrong. For finite Nω, max over W is downward biased relative to sup over Ω, so the learned target is not Q* in general. This does not sink the paper: the empirical gains could still come from the envelope idea, and the bias disappears as Nω grows. But the headline claim of a single network covering the whole CCS is only proven for the exact operator, and the implemented algorithm is a biased approximation of it. A reviewer should ask for either a finite-W analysis (even a loose bound) or an experimental check on a small domain comparing minibatch targets against exact-envelope targets as Nω varies. The preference-inference tables also need error bars or a quantified accuracy; "concentrated on the diagonal" is suggestive, not a result. The "better sample efficiency in theory" claim in the main text is also stronger than what is proven.

Net: this paper deserves a serious referee. It is a solid NeurIPS-level contribution with a real idea, useful experiments, and one honest gap between theory and practice. I would recommend conditional acceptance with a request for the Nω/exact-envelope comparison and softened claims in the supplementary, not rejection.

Referee Report

3 major / 5 minor

Summary. The paper introduces envelope MOQ-learning, a multi-objective reinforcement learning algorithm that represents Q(s,a,ω) as vector-valued expected returns indexed by a linear preference ω, and updates it with an envelope optimality operator T that maximizes utility over both actions and the full preference space Ω. The authors prove that Q*, the value of the policy that maximizes ω^T return for each ω, is a fixed point of T, that T is a γ-contraction under a preference-weighted sup-norm, and that iterating T converges to Q* in a complete pseudo-metric space. The practical method, Algorithm 1, approximates the envelope by sampling a minibatch W of Nω preferences, uses a homotopy loss combining a squared-error loss and a utility-pull loss, and applies HER-style replay. Experiments on Deep Sea Treasure, Fruit Tree Navigation, task-oriented dialogue, and Super Mario report that the envelope algorithm improves coverage ratio, adaptation error, and average utility relative to MOFQI, CN+OLS, and Scalarized, and that hidden preferences can be inferred within 15–100 episodes.

Significance. If the theoretical and empirical claims hold, the paper makes a useful contribution: it provides an operator-level framework for value-based MORL, a single-network representation of policies over the entire preference simplex, and a practical demonstration on non-trivial domains including dialogue and video game control. The paper ships code, defines reusable evaluation metrics (coverage ratio and adaptation error), and reports preference inference results that are falsifiable and easy to reproduce. The main caveat is that the implemented algorithm is not the operator for which convergence is proven, so the theoretical guarantee currently applies to an idealized version rather than to the algorithm that generates the empirical results. That gap is fixable but is load-bearing for the paper's central claim.

major comments (3)

[Algorithm 1; Section 3; Theorem 5 in Supplementary A.1] The convergence theorems are proven for the exact operator T defined in Eq. (4) with H over all Ω, but Algorithm 1 replaces Ω by a minibatch W of Nω sampled preferences when computing y_ij = r + γ argQ max_{a∈A, ω′∈W} ω_i^T Q(s_{j+1}, a, ω′; θ). Theorems 1–3 do not cover this finite-preference minibatch operator, and the supplementary statement that 'Theorems 1–3 and 5 guarantee the convergence of this minibatch updating' overstates the results: Theorem 5 is a state-action minibatch convergence theorem, not a preference-space truncation theorem. For any finite W, max_{ω′∈W} ≤ sup_{ω′∈Ω}, so every bootstrapped target is a downward-biased estimate of the exact envelope target, and this bias does not vanish as the amount of data grows while Nω is fixed. Since the headline improvement over Scalarized is reported at Nω = 32, the improvement could in principle come partly from this preference-batch effect rather than from the envelope mechanism. Please either prove a convergence or bias bound for the sampled-preference operator, or run a controlled comparison between the minibatch target and an exact-envelope target across Nω values to quantify the gap.
[Supplementary A.2.1, proof of Theorem 1] The proof of Theorem 1 contains a cancellation step labeled 'use def. of argQ' that is not justified by the surrounding argument. The displayed sandwich inequality asserts sup_{ω′∈Ω} ω^T argQ sup_π (ω′^T Q^π) = ω^T argQ sup_π (ω^T_* Q^π), with ω^T_* defined as the preference maximizing the outer ω-projection. This presupposes that the preference achieving the outer supremum is the same as the preference that should be used inside argQ, and it does not follow from the definition of argQ when different policies are optimal for different preferences. The fixed-point property Q* = T Q* is load-bearing for the convergence claims: without it, Theorem 3's contradiction step d(Q⋄, Q*) ≤ γ d(Q⋄, Q*) has no starting point. Please rewrite this proof with explicit conditions under which the sup over ω′ and the argQ selection commute, or give a direct argument using the convex coverage set property of Q*.
[Supplementary A.2.1, proof of Theorem 2] The contraction proof relies on the same cancellation between ω^T and argQ, and additionally uses a 'without loss of generality' sign assumption to drop absolute values. As written, the inequality |ω^T[argQ sup_{a′,ω′} ω^T Q(s′,a′,ω′) − argQ sup_{a′′,ω′′} ω^T Q′(s′,a′′,ω′′)]| ≤ |sup_{a′,ω′} ω^T Q(s′,a′,ω′) − sup_{a′′,ω′′} ω^T Q′(s′,a′′,ω′′)| requires that the argQ selections have scalar projections equal to the corresponding suprema. That is not guaranteed for arbitrary bounded Q, Q′ when Ω is uncountable and Q is not continuous in ω (the supremum may not be attained, and if multiple maximizers exist the selection matters). Please state explicit regularity or selection assumptions, or modify the definition of H so that this inequality is a consequence of the definitions.

minor comments (5)

[Section 3, Eq. (2)] The paper states that d does not satisfy the identity of indiscernibles and is therefore a pseudo-metric. Since Ω is an (m−1)-simplex, the linear functionals ω^T for ω ∈ Ω separate points in R^m, so if d(Q,Q′)=0 then Q(s,a,ω)=Q′(s,a,ω) for all s,a,ω. The space is in fact a metric space; the statement is incorrect, though the resulting convergence conclusions are unaffected.
[Section 3, definition of H] The optimality filter H is defined using argQ, but no rule is given for choosing among multiple maximizers or for cases where the supremum over ω′ is not attained. Please specify the selection convention or state assumptions (e.g., finite A and continuity of Q in ω) that make H single-valued.
[Table 1] The footnote reports p < 0.05 vs MOFQI on all domains, but MOFQI is not evaluated on SuperMario; the significance statement should specify which comparisons were actually performed and whether multiple-comparison corrections were applied.
[Figure 4 and 'Sample Efficiency' paragraph] The text says Figure 4 shows plots over a varying number of sampled preferences Nω, while the figure caption says the panels compare FTN depths d = 5,6,7; please clarify which panel corresponds to which experiment and add axis labels for Nω where applicable.
[Algorithm 2 in Supplementary C.5] Algorithm 2 uses the notation argV max without defining it; please define this analog of argQ for the value-network variant to make the pseudocode self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the envelope fixed-point theory is a genuine first-principles derivation, and the minibatch approximation gap is a soundness issue, not a circular reduction.

full rationale

The paper's central theoretical claim is that the preferred optimal value function Q*(s,a,ω) = argQ sup_π ω^T E[Σ γ^t r] is a fixed point of the envelope operator T (Eq. 4) and that T is a γ-contraction under the pseudo-metric d (Eq. 2). This is a standard Bellman-style fixed-point argument, not a definitional circle: Q* is defined through a supremum over policies, while T is defined through an optimality filter H that takes a supremum over actions and preferences. The proof of Theorem 1 supplies an analytic sandwich inequality to collapse the preference supremum in H to the policy supremum at the same ω, and Theorem 2 uses the standard sup-of-difference bound. These are independent mathematical reductions, not a restatement of the conclusion. The empirical evaluation uses external benchmarks (Deep Sea Treasure, Fruit Tree Navigation, PyDial-based dialog, Super Mario) and compares against published baselines; the inferred preferences in Tables 2-3 are optimization outputs, not parameters fitted to the theory. There are no load-bearing self-citations: the cited contraction and minibatch results (Bertsekas, Banach fixed-point theory) are external and the paper's own authors do not supply a uniqueness theorem or an ansatz through prior self-citation. The one in-scope concern is that Algorithm 1 computes the envelope target using a minibatch W of N_ω sampled preferences, while Theorems 1-3 prove convergence for the exact operator over all Ω; the supplementary statement that 'Theorems 1-3 and 5 guarantees the convergence of this minibatch updating' overstates Theorem 5, which covers state-action minibatching rather than truncation of the preference space. That is a finite-sample bias/correctness gap, not a circularity: the exact operator is not defined in terms of the sampled empirical targets, and no fitted quantity is renamed as a prediction. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central algorithm depends on a set of standard RL assumptions plus three hand-chosen hyperparameters. The most significant load-bearing axiom is that the minibatch preference sample W preserves the envelope filter's fixed point; this is not proven. The metric d is described as a pseudo-metric, but for Omega as the simplex it is in fact a metric; this does not invalidate the contraction proof. No invented entities are introduced.

free parameters (4)

Homotopy weight lambda schedule = increases exponentially from 0.0 to 1.0
Chosen by hand to trade off L_A and L_B; affects training stability and final Q alignment; no principled selection criterion given.
Number of sampled preferences per update N_omega = 32 for Dialog, varied 1..128 for FTN
Hyperparameter controlling the minibatch approximation of the optimality filter H; larger N improves CR/AE but slows updates; chosen per domain.
Preference sampling distribution D_omega = uniform over the (m-1)-simplex (nearly uniform)
Assumed distribution for training and evaluation; the theoretical results require the full support of Omega but the algorithm only samples from D_omega.
Gaussian standard deviation sigma for preference inference = fixed (not reported numerically)
Used in the truncated Gaussian D_omega^m for inferring hidden preferences; the paper states sigma is fixed but does not give its value.

assumptions (5)

standard math The MOMDP has bounded vector rewards and discount factor gamma in [0,1), so Q-functions are bounded and the value space is complete.
Invoked implicitly throughout Section 3 and the supplementary to apply Banach's fixed-point theorem; standard for RL.
domain assumption Preference functions are linear, f_omega(r) = omega^T r.
Restricts the paper to linear preferences; the CCS is then the convex hull of Pareto-optimal returns, stated in Section 2.
domain assumption The preference space Omega is the full simplex (or at least spans R^m) so that the metric d is separating.
The paper calls d a pseudo-metric because 'identity of indiscernibles does not hold' (Section 3), but if Omega is the simplex this is false; the assumption that Omega spans the objective space is not stated.
ad hoc to paper The minibatch-approximated filter with W sampled from D_omega behaves like the exact filter over Omega.
Algorithm 1 computes y_ij using max over a in A, omega' in W, not Omega; no theorem guarantees this preserves the contraction or the fixed point. This is the key gap between theory and practice.
ad hoc to paper A single neural network with (s, omega) input can represent the multi-objective Q function sufficiently well for the envelope updates to align preferences with policies.
Assumed by the deep learning algorithm; not derived; standard deep RL assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation." pith.science (2026). https://pith.science/paper/CA6I4MZR

@misc{pith2026190808342,
  author       = {Pith},
  title        = {Pith review of: A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CA6I4MZR}},
  note         = {Machine review of arXiv:1908.08342}
}

read the original abstract

We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bellman equation to learn a single parametric representation for optimal policies over the space of all possible preferences. After an initial learning phase, our agent can execute the optimal policy under any given preference, or automatically infer an underlying preference with very few samples. Experiments across four different domains demonstrate the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 1908.08342 by the authors.

**Figure 1.** Task-oriented dialogue policy learning is a real-life example of unknown linear preference scenario. Users may expect either briefer dialogue or more informative dialogue depending on the task. In recent years, there has been increased interest in the paradigm of multi-objective reinforcement learning (MORL), which deals with learning control policies to simultaneously optimize over several criteria. Compared to tr… view at source ↗

**Figure 2.** (a) The Pareto frontier may encapsulate local concave parts (points A-H, plus point K), whereas CCS is a convex subset of Pareto frontier (points A-H). Point L indicates a non-optimal solution. (b) Linear preferences select the optimal solution from CCS with the highest utility, represented by the projection length along preference vector. Arrows are different linear preferences, and points indicate possible returns… view at source ↗

**Figure 3.** Illustration of evaluation metrics for MORL. (a.) Coverage ratio (CR) measures an agent’s ability to find all the potential optimal solutions in the convex coverage set of Pareto frontier. Dots with black boundary are solutions in CCS, dots without black boundary are non-optimal returns, and dots in green are solutions retrieved by an MORL algorithm. CR is the F1 based on the precision and recall calculation. (b.)… view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Coverage Ratio (CR) and Adaptation Error (AE) comparison of the scalarized algorithm [13] and our envelope deep MORL algorithm over 5000 episodes of FTN tasks of depths d = 5, 6, 7. Higher CR indicates better coverage of optimal policies, lower AE indicates better adap…

**Figure 5.** Figure 5: The success-weight curves of taskoriented dialog. Each data point is a moving average of closest around 500 dialogues in the interval of around ± 0.05 weight of success. The light shadow indicates the standard deviations of 5 independent runs under each configuratio…

**Figure 6.** Figure 6: An explanation for homotopy optimization method used in the envelope deep MORL algorithm. The MSE loss L A is hard for optimization since there are many local minima over its landscape. Although the value metric loss L B has fewer local minima, it is also hard for opti…

**Figure 7.** Figure 7: 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

**Figure 8.** Figure 8: Fruit Tree Navigation (FTN): An agent travels from the root node to one of the leaf node to pick a fruit according to a post-assigned preference ω on the components of nutrition, treated as different objectives. The observation of an agent is its current coordinates (r…

**Figure 7.** Figure 7: Deep Sea Treasure (DST): An agent controls a submarine searching for treasures in a 10 × 11-grid world. The state st consists of the agent’s current coordinates (x, y). An agent’s action spaces is navigation in four directions. The reward received by the agent is a 2-d…

**Figure 9.** Figure 9: Quantitative evaluation metrics for multi-objective reinforcement learning. (a.) Coverage ratio measures an agent’s ability to find all the potential optimal solutions in the convex coverage set of Pareto frontier. (b.) Adaptation quality measures an agent’s ability of…

**Figure 10.** Figure 10: presents the real CCS and the retrieved solutions of a MORL algorithm. The scalarized and envelope algorithm can find all the whole CCS [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

**Figure 11.** Figure 11: Coverage Ratio (CR) and Adaptation Quality (AQ) comparison of the scalarized deep MORL algorithm and the envelope deep MORL algorithm tested on fruit tree navigation tasks of depths d = 5, 6, 7. Trained on 5000 episodes and test on 2000 and 5000 episode to estimate CR…

**Figure 12.** Figure 12: Comparison of CCS and control frontiers of deep MORL algorithms. The left figure (a) is visualizing the real CCS and retrieved CCS of scalarized and envelope MORL algorithms using t-SNE. The right figure (b) presents the slices of optimal control frontier and the cont…

**Figure 13.** Figure 13: Utility-weight curves for the MORL and single-objective RL dialog policy learning after 3,000 training dialogues. We evaluate policies on 5,000 dialogues with near-uniformly randomly sampled preference. For each curve, each data point is a moving average of the closes…

**Figure 14.** Figure 14: The training curves of the Envelope Multi-Objective A3C (EMoA3C) algorithm. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 108 citations worldwide. Full citation record

Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization
cs.LG 2026-02 conditional novelty 6.0 of 10

D3PO learns a single preference-conditioned policy via per-objective PPO losses, late preference weighting, and a preference-distance-scaled diversity regularizer, reporting improved Pareto fronts on most tested MORL ...
Reinforcement Learning for Multi-Objective Multi-Echelon Supply Chain Optimisation
cs.AI 2025-07 conditional novelty 5.0 of 10

MORL/D, a decomposition-based multi-objective RL method, yields the most balanced Pareto-front approximations across three supply chain network complexities when compared with weighted-sum PPO and NSGA-II.
Multi-Objective Reinforcement Learning for Automated Resilient Cyber Defence
cs.CR 2024-11 conditional novelty 5.0 of 10

In a two-objective CybORG defence game, MOPPO produced policies that trade off network defence against user access, while Pareto Conditioned Networks did not respond reliably to preference prompts.

Reference graph

Works this paper leans on

52 extracted references · 45 canonical work pages · cited by 3 Pith papers

[1]

Concrete problems in ai safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

arXiv 2016
[2]

Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley

Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. J. Artif. Intell. Res., 48:67–113, 2013

work page 2013
[3]

Adaptive weighted sum method for multiobjective optimization: a new method for pareto front generation

Il Yong Kim and OL De Weck. Adaptive weighted sum method for multiobjective optimization: a new method for pareto front generation. Structural and multidisciplinary optimization, 31(2):105–116, 2006

work page 2006
[4]

Multi-objective optimization using genetic algorithms: A tutorial

Abdullah Konak, David W Coit, and Alice E Smith. Multi-objective optimization using genetic algorithms: A tutorial. Reliability Engineering & System Safety, 91(9):992–1007, 2006

work page 2006
[5]

Sequential approximate multiobjective optimization using computational intelligence

Hirotaka Nakayama, Yeboon Yun, and Min Yoon. Sequential approximate multiobjective optimization using computational intelligence. Springer Science & Business Media, 2009

work page 2009
[6]

On min-norm and min-max methods of multi-objective optimization

JiGuan G Lin. On min-norm and min-max methods of multi-objective optimization. Mathematical programming, 103(1):1–33, 2005

work page 2005
[7]

Dynamic preferences in multi-criteria reinforcement learning

Sriraam Natarajan and Prasad Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In Luc De Raedt and Stefan Wrobel, editors, Machine Learning, Proceedings of the Twenty-Second Interna- tional Conference (ICML 2005), Bonn, Germany, August 7-11, 2005, volume 119 of ACM International Conference Proceeding Series, pages 601–608. ACM, 2005

work page 2005
[8]

Learning all optimal policies with multiple criteria

Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors,Machine Learning, Proceedings of the Twenty- Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 , volume 307 of ACM International Conference Proceeding Series, pages 41–47. ACM, 2008

work page 2008

Show all 52 references

[9]

Assael, Diederik M

Hossam Mossalam, Yannis M. Assael, Diederik M. Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. CoRR, abs/1610.02707, 2016

2016 arXiv
[10]

Dynamic Programming

Richard Ernest Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1957

1957
[11]

Hindsight experience replay

Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information ...

2017
[12]

Modern homotopy methods in optimization

Layne T Watson and Raphael T Haftka. Modern homotopy methods in optimization. Computer Methods in Applied Mechanics and Engineering, 74(3):289–305, 1989

1989
[13]

Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher

Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, page TBA, 2019

2019
[14]

Empirical evaluation methods for multiobjective reinforcement learning algorithms

Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning, 84(1-2):51–80, 2011

2011
[15]

Multiobjective reinforcement learning: A comprehensive overview

Chunming Liu, Xin Xu, and Dewen Hu. Multiobjective reinforcement learning: A comprehensive overview. IEEE Trans. Systems, Man, and Cybernetics: Systems, 45(3):385–398, 2015

2015
[16]

The steering approach for multi-criteria reinforcement learning

Shie Mannor and Nahum Shimkin. The steering approach for multi-criteria reinforcement learning. In Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani, editors, Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Syn...

2001
[17]

Kephart, David Levine, Freeman L

Gerald Tesauro, Rajarshi Das, Hoi Chan, Jeffrey O. Kephart, David Levine, Freeman L. Rawson III, and Charles Lefurgy. Managing power consumption and performance of computing systems using reinforcement learning. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis,...

2007
[18]

Drugan, and Ann Nowé

Kristof Van Moffaert, Madalina M. Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013, IEEE Symposium Series on Comput...

2013
[19]

Multi-objective reinforcement learning with continu- ous pareto frontier approximation

Matteo Pirotta, Simone Parisi, and Marcello Restelli. Multi-objective reinforcement learning with continu- ous pareto frontier approximation. In Blai Bonet and Sven Koenig, editors,Proceedings of the Twenty-Ninth AAAI Conference on Artiﬁcial Intelligence, January 25-30, 2015, ...

2015
[20]

Manifold-based multi-objective policy search with sample reuse

Simone Parisi, Matteo Pirotta, and Jan Peters. Manifold-based multi-objective policy search with sample reuse. Neurocomputing, 263:3–14, 2017

2017
[21]

Parallel reinforcement learning for weighted multi-criteria model with adaptive margin

Kazuyuki Hiraoka, Manabu Yoshida, and Taketoshi Mishima. Parallel reinforcement learning for weighted multi-criteria model with adaptive margin. In Masumi Ishikawa, Kenji Doya, Hiroyuki Miyamoto, and Takeshi Yamakawa, editors,Neural Information Processing, 14th International C...

2007
[22]

Multi-objective reinforcement learning for acquiring all pareto optimal policies simultaneously - method of determining scalarization weights

Hitoshi Iima and Yasuaki Kuroe. Multi-objective reinforcement learning for acquiring all pareto optimal policies simultaneously - method of determining scalarization weights. In 2014 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2014, San Diego, CA, USA, ...

2014
[23]

Multi-objective ﬁtted q-iteration: Pareto frontier approximation in one single run

Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. Multi-objective ﬁtted q-iteration: Pareto frontier approximation in one single run. In Proceedings of the IEEE International Conference on Networking, Sensing and Control, ICNSC 2011, Delft, The Netherlands, 11-13 A...

2011
[24]

Tree-based ﬁtted q-iteration for multi- objective markov decision problems

Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. Tree-based ﬁtted q-iteration for multi- objective markov decision problems. In The 2012 International Joint Conference on Neural Networks (IJCNN), Brisbane, Australia, June 10-15, 2012, pages 1–8. IEEE, 2012

2012
[25]

Preference elicitation in combinatorial auctions

Wolfram Conen and Tuomas Sandholm. Preference elicitation in combinatorial auctions. In Proceedings of the 3rd ACM conference on Electronic Commerce, pages 256–259. ACM, 2001

2001
[26]

A POMDP formulation of preference elicitation problems

Craig Boutilier. A POMDP formulation of preference elicitation problems. InProceedings of the Eighteenth National Conference on Artiﬁcial Intelligence and Fourteenth Conference on Innovative Applications of Artiﬁcial Intelligence, July 28 - August 1, 2002, Edmonton, Alberta, C...

2002
[27]

Survey of preference elicitation methods

Li Chen and Pearl Pu. Survey of preference elicitation methods. Technical report, 2004

2004
[28]

Ng and Stuart J

Andrew Y . Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Pat Langley, editor, Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000, pages 663–670. M...

2000
[29]

Pieter Abbeel and Andrew Y . Ng. Apprenticeship learning via inverse reinforcement learning. In Carla E. Brodley, editor, Machine Learning, Proceedings of the Twenty-ﬁrst International Conference (ICML 2004), Banff, Alberta, Canada, July 4-8, 2004, volume 69 of ACM Internation...

2004
[30]

Generative adversarial imitation learning

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural In- formation Processing Systems 29: Annual Conference on Neural Information Process...

2016
[31]

Learning an agent’s utility function by observing behavior

Urszula Chajewska, Daphne Koller, and Dirk Ormoneit. Learning an agent’s utility function by observing behavior. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 35...

2001
[32]

Christopher J. C. H. Watkins and Peter Dayan. Technical note q-learning. Machine Learning, 8:279–292, 1992

1992
[33]

Linear operator theory in engineering and science

Arch W Naylor and George R Sell. Linear operator theory in engineering and science. Springer Science & Business Media, 2000

2000
[34]

Rusu, Joel Veness, Marc G

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra,...

2015
[35]

Williams

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. 11

1992
[36]

Stefan Ultes, Lina Maria Rojas-Barahona, Pei-Hao Su, David Vandyke, Dongho Kim, Iñigo Casanueva, Pawel Budzianowski, Nikola Mrksic, Tsung-Hsien Wen, Milica Gasic, and Steve J. Young. Pydial: A multi- domain statistical dialogue system toolkit. In Proceedings of the 55th Annual...

2017
[37]

Super Mario Bros for OpenAI Gym

Christian Kauten. Super Mario Bros for OpenAI Gym. https://github.com/Kautenja/ gym-super-mario-bros , 2018

2018
[38]

Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu

V olodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016,...

2016
[39]

An introduction to metric spaces and ﬁxed point theory , volume 53

Mohamed A Khamsi and William A Kirk. An introduction to metric spaces and ﬁxed point theory , volume 53. John Wiley & Sons, 2011

2011
[40]

Bertsekas

Dimitri P. Bertsekas. Regular policies in abstract dynamic programming. SIAM Journal on Optimization, 27(3):1694–1727, 2017

2017
[41]

Abstract dynamic programming

Dimitri P Bertsekas. Abstract dynamic programming. Athena Scientiﬁc Belmont, MA, 2018

2018
[42]

Bellemare, Will Dabney, and Rémi Munos

Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 449–458, 2017

2017
[43]

Jost Schatzmann and Steve J. Young. The hidden agenda user simulation model. IEEE Trans. Audio, Speech & Language Processing, 17(4):733–747, 2009

2009
[44]

Stefan Ultes, Pawel Budzianowski, Iñigo Casanueva, Nikola Mrksic, Lina Maria Rojas-Barahona, Pei-Hao Su, Tsung-Hsien Wen, Milica Gasic, and Steve J. Young. Reward-balancing for statistical spoken dialogue systems using multi-objective reinforcement learning. In Proceedings of ...

2017
[45]

Lillicrap, Jonathan J

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2015

2015 arXiv
[46]

Lillicrap, Ilya Sutskever, and Sergey Levine

Shixiang Gu, Timothy P. Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 2829–2838, 2016

2016
[47]

Discrete sequential prediction of continuous actions for deep RL

Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep RL. CoRR, abs/1705.05035, 2017

2017 arXiv
[48]

Deep reinforcement learning with double q-learning

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Dale Schuurmans and Michael P. Wellman, editors,Proceedings of the Thirtieth AAAI Conference on Artiﬁcial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. , pages ...

2016
[49]

Prioritized experience replay

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. CoRR (Published at ICLR 2016), abs/1511.05952, 2015

2016 arXiv
[50]

L.J.P van der Maaten and G.E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Mahcine Learning Research, Nov 2008. 12 Supplementary Material for Generalized Algorithm for Multi-Objective RL and Policy Adaptation A Theoretical Framework for Value-Based MORL Al...

2008
[51]

Convergence Condition: We have ∀Q∈Q k,TQ∈Q k+1; (15)
[52]

distance

Box Condition: For all k,Qk is a Cartesian product of the form Qk =×s∈S,a∈AQk {s},{a}, (16) whereQk S,A is a set of bounded real-valued functions on statesS and actionsA. Then for every Q0∈Q 0 the sequence{Qk} generated by the minibatch iteration algorithm converges to Q∗ [40]...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Concrete problems in ai safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

arXiv 2016

[2] [2]

Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley

Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. J. Artif. Intell. Res., 48:67–113, 2013

work page 2013

[3] [3]

Adaptive weighted sum method for multiobjective optimization: a new method for pareto front generation

Il Yong Kim and OL De Weck. Adaptive weighted sum method for multiobjective optimization: a new method for pareto front generation. Structural and multidisciplinary optimization, 31(2):105–116, 2006

work page 2006

[4] [4]

Multi-objective optimization using genetic algorithms: A tutorial

Abdullah Konak, David W Coit, and Alice E Smith. Multi-objective optimization using genetic algorithms: A tutorial. Reliability Engineering & System Safety, 91(9):992–1007, 2006

work page 2006

[5] [5]

Sequential approximate multiobjective optimization using computational intelligence

Hirotaka Nakayama, Yeboon Yun, and Min Yoon. Sequential approximate multiobjective optimization using computational intelligence. Springer Science & Business Media, 2009

work page 2009

[6] [6]

On min-norm and min-max methods of multi-objective optimization

JiGuan G Lin. On min-norm and min-max methods of multi-objective optimization. Mathematical programming, 103(1):1–33, 2005

work page 2005

[7] [7]

Dynamic preferences in multi-criteria reinforcement learning

Sriraam Natarajan and Prasad Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In Luc De Raedt and Stefan Wrobel, editors, Machine Learning, Proceedings of the Twenty-Second Interna- tional Conference (ICML 2005), Bonn, Germany, August 7-11, 2005, volume 119 of ACM International Conference Proceeding Series, pages 601–608. ACM, 2005

work page 2005

[8] [8]

Learning all optimal policies with multiple criteria

Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors,Machine Learning, Proceedings of the Twenty- Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 , volume 307 of ACM International Conference Proceeding Series, pages 41–47. ACM, 2008

work page 2008