Pith. sign in

REVIEW 3 major objections 6 minor 78 references

Fairness in Reinforcement Learning with Bisimulation Metrics

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Minimizing a group-conditioned bisimulation metric between state-group pairs yields demographic parity in reinforcement learning.

desk verdict A genuinely new fairness-via-bisimulation algorithm with a careful empirical study, but the main theorem overreaches: the proof gives only an expectation bound, and transfer to the original MDP is asserted rather than proven. read the letter →

arxiv 2412.17123 v2 pith:IFX6NX7U submitted 2024-12-22 cs.LG cs.CY

Sahand Rezaei-Shoshtari , Hanna Yurchyk , Scott Fujimoto , Doina Precup , David Meger This is my paper · ORCID
classification cs.LGcs.CY MSC 68T0590C4068Q85
keywords bisimulationmetricgroupfairnessinreinforcementlearningdemographicparityrewardshapingobservationdynamicslong-termsequentialdecisionmakingunconstrainedpolicyoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that group fairness in sequential decision making can be achieved without adding fairness constraints to the learner. The central claim is that minimizing a group-conditioned bisimulation metric—a distance that measures how differently two groups behave under a given policy in terms of immediate rewards and future state distributions—between state-group pairs makes an unconstrained reinforcement learning agent approximately satisfy demographic parity. The proposed algorithm, Bisimulator, does this by reshaping the reward signal and the observations the agent receives, while leaving the underlying environment dynamics and the choice of RL solver unchanged. If the claim holds, fairness becomes a property of reward and observation design rather than of constrained policy optimization, which is why it would matter for real deployments.

What carries the argument

The load-bearing object is the group-conditioned $\pi$-bisimulation operator $F^\pi_{\mathrm{group}}$ of Equation (5), which measures behavioral dissimilarity between state-group pairs $(s_i,g_i)$ and $(s_j,g_j)$ as the absolute difference in policy-conditioned rewards plus a discounted Wasserstein distance between policy-conditioned next-state distributions. It is a conditional extension of the policy-conditioned bisimulation metric, with group membership treated as a deterministic constant transition, so the fixed point $d^\pi_{\mathrm{group}\sim}$ bounds value differences (Theorem 2) and controls the fairness gap. The method minimizes its expectation over the stationary state-group distribution, using quantile matching to pair comparable states across groups, a learnable reward correction $R_\phi(s,a,g)$ for the reward term, and a learned dynamics model $T_\psi(s'|s,a,g)$ with the 2-Wasserstein distance replacing the 1-Wasserstein distance for the dynamics term.

What would settle it

Build a two-group MDP where the shaped reward and observation changes drive the group-conditioned $\pi$-bisimulation metric below any chosen $\epsilon$, then evaluate the resulting policy on the original environment: if any state $s$ has $|V^\pi(s,g_1)-V^\pi(s,g_2)| > \epsilon$ while the expected bound used in Theorem 3 is below $\epsilon$, the pointwise demographic parity claim fails.

Watch

Extended reading notes

Core claim

The paper establishes a formal bridge between bisimulation theory and group fairness in reinforcement learning. It defines a group-conditioned $\pi$-bisimulation metric on pairs consisting of a state and a sensitive group, proves that the associated operator has a least fixed point (Theorem 1), and proves that this fixed point upper-bounds the difference in expected returns between two state-group pairs under a fixed policy (Theorem 2). Theorem 3 then concludes that minimizing the metric in expectation, through the two-term objective of Equation (7), yields demographic parity fairness in the sense of the adopted RL definition. The practical algorithm minimizes the reward term with a learnable group-conditioned reward correction and the dynamics term by adjusting modifiable observation parameters, using a learned next-state model and the 2-Wasserstein distance between its predicted normal distributions. On lending and college admission benchmarks, the paper reports that Bisimulator, run with standard PPO or DQN, closes the recall gap and reduces the credit gap or social burden compared with constrained baselines.

Load-bearing premise

The method assumes that making groups behave similarly in the modified problem, with reshaped rewards and altered observations, makes the learned policy fair in the original problem; the supporting bound is proved only in the modified MDP and only in expectation, while the fairness definition is pointwise, and the paper supplies no transfer theorem.

Editorial extensions

If this is right

  • Any off-the-shelf RL algorithm can be used without modification, because fairness enters only through rewards and observations.
  • A regulator who controls rewards or observable features, such as credit-score overrides or subsidized test-preparation costs, can implement fairness without controlling the learner's objective.
  • The single unconstrained objective in Equation (7) replaces Lagrangian or constrained-optimization machinery, removing a class of hyperparameters and algorithmic couplings.
  • The objective extends to more than two sensitive groups by summing over group pairs; the paper demonstrates this on a ten-group lending variant.
  • The reward-only variant of Bisimulator remains competitive, so fairness gains do not strictly require the ability to change observations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not explore is applying the same reward-and-observation shaping to other fairness notions, such as equality of opportunity, by weighting the metric over the relevant state or action pairs.
  • The pointwise-versus-expectation gap in Theorem 3 suggests a testable strengthening: a per-state version of the bound would be needed to fully match the pointwise demographic parity definition adopted in the paper.
  • A natural deployment test would be a recommendation platform where displayed scores are operator-controlled; the reward-only variant indicates that even fixed environments could benefit from this approach.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes Bisimulator, a method for long-term group fairness in reinforcement learning that, instead of imposing fairness constraints during policy optimization, modifies the reward function and observation dynamics of a group-conditioned MDP so that an unconstrained RL solver (PPO or DQN) learns a policy approximating demographic parity. The authors introduce a group-conditioned π-bisimulation metric (Theorem 1), prove that it upper-bounds value-function differences between state-group pairs (Theorem 2), and claim that minimizing this metric yields demographic parity as defined in Definition 5 (Theorem 3). The practical algorithm learns a reward correction term and adjusts observation dynamics using a Gaussian dynamics model and the closed-form 2-Wasserstein distance. Experiments on lending and college-admission tasks from the ml-fairness-gym benchmark show that Bisimulator achieves low recall gaps and credit gaps relative to several fairness-constrained baselines, with ablations showing that a reward-only variant remains competitive. The paper includes 10-seed runs with confidence intervals, tuned baselines, and a 10-group lending extension.

Significance. The bisimulation-fairness connection is an interesting and potentially useful idea. If the central theoretical claim held, the method would provide an unconstrained alternative to constrained-MDP approaches and would be agnostic to the choice of RL algorithm. The empirical study is careful: it uses 10 seeds, reports 95% confidence intervals, tunes baselines with grid search, includes ablations (reward-only versus reward-plus-dynamics), and demonstrates scalability to 10 groups; the code is submitted as supplementary material. However, the theoretical guarantee is not established as stated. The proof of Theorem 3 controls only an expectation of the π-bisimulation metric over the stationary distribution, whereas Definition 5 requires a pointwise bound at every state, and no transfer argument connects the modified MDP in which the metric is minimized to the original MDP in which fairness is claimed. The practical algorithm also replaces the exact metric with learned approximations (a Gaussian dynamics model, the W2 distance, and quantile matching). The contribution is therefore conditional: it is a promising empirical method with an incompletely justified theoretical foundation.

major comments (3)
  1. [Appendix A.2 / Theorem 3] The proof of Theorem 3 bounds E_{ρ(s,g)}[|V^π(s_i,g_i) - V^π(s_j,g_j)|] ≤ E_{ρ(s,g)}[d^π_group~((s_i,g_i),(s_j,g_j))] ≤ ε, which is an expectation over the stationary state-group distribution. Definition 5 requires a pointwise bound |J^π(s,g_i) - J^π(s,g_j)| ≤ ε for every state s. A small expectation does not control pointwise values: the metric can be arbitrarily large on states with low stationary mass, and since Eq. (7) minimizes precisely this expectation over quantile-matched pairs, perfect minimization of Eq. (7) is compatible with large violations of Definition 5 at rarely visited states. The theorem should be restated as a demographic-parity-in-expectation guarantee (with a matching amendment of Definition 5), or the proof needs an additional argument establishing a pointwise bound, which is not provided.
  2. [Section 4 / Algorithm 1] Theorems 1 and 2 are stated for a fixed group-conditioned MDP. In Bisimulator, the reward is replaced by R_original + α R_φ (Eq. 9) and the observation dynamics are modified (Section 4.2), so the π-bisimulation metric is computed for a modified MDP M'. The introduction claims that unconstrained policy optimization in the modified MDP satisfies fairness in the original, unmodified MDP, but the paper provides no transfer result showing that a policy that equalizes returns under the modified reward and observations also equalizes returns under the original dynamics and reward. Without such a bound, the fair behavior observed in the modified MDP need not persist in the original environment, which is the environment actually evaluated in Section 5. This gap is load-bearing for the paper's central claim.
  3. [Section 4.2 / Eq. (11)] The objective J in Eq. (7) is defined in terms of the exact π-bisimulation metric d^π_group~, but Algorithm 1 approximates it by training a Gaussian dynamics model T_ψ and using the closed-form 2-Wasserstein distance (Eq. 11) over quartile-matched batches. These substitutions are not covered by Theorems 1-3. To support the statement that the algorithm minimizes the bisimulation metric, the paper should either provide error bounds between the true metric and the estimated objective, or explicitly state that the method is a heuristic approximation whose theoretical guarantee applies only to the exact, non-implemented objective.
minor comments (6)
  1. [Definition 5] Definition 5 uses the notation J^π(s,g_i), but J^π was previously defined as an unconditional expectation over ρ^π(s,g); the conditional quantity should be defined explicitly (for example, as V^π(s,g) or as a state-conditioned return) to avoid ambiguity.
  2. [Appendix B.1] The paragraph titled 'Fairness Metrics' in the lending environment appendix describes social burden and cumulative admissions, which are not used in the lending experiments; this appears to be a copy-paste error from the college admissions section.
  3. [Section 5.2] In the 'Baselines' paragraph of the college admissions section, the text says 'the same RL baselines described in Section 5.2' but should refer to Section 5.1.
  4. [Equation (7)] Equation (7) leaves the joint distribution over the pair ((s_i,g_i),(s_j,g_j)) unspecified; the expectation should indicate that pairs are drawn from the per-group stationary distributions (or via the quantile-matching procedure) before minimization.
  5. [Appendix A.1] The proofs of Theorems 1 and 2 reduce to results in Castro (2020), but the paper should state the regularity conditions under which Castro's fixed-point and value-bound theorems apply (for example, finite or compact state spaces with continuous rewards and transitions) and verify that the extended MDP with state space S×G satisfies them.
  6. [Section 7] The authors acknowledge that convergence of the alternating policy-update and metric-minimization scheme is an open topic; this caveat is important for the practical algorithm and should also be stated where Theorem 3 is invoked, rather than only in the limitations section.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core reductions use external Castro (2020) results, the Eq. (7) objective is a designed sufficient bound, and the fairness evaluation is external; the expectation-vs-pointwise issue is a proof gap, not circularity.

full rationale

No circular step can be exhibited. Theorems 1 and 2 are explicit reductions to Castro (2020)'s pi-bisimulation fixed-point and value-difference bounds, which are external, parameter-free mathematical results whose stated assumptions do not include the fairness target; the Ferns et al. (2004, 2011) metric results are background and one is co-authored by Precup, but they are independent foundational results and the paper's proofs rest on Castro (2020), not on any unverified claim of its own. The objective in Eq. (7) is a designed sufficient condition: by Theorem 2, |V^pi(s_i,g_i) - V^pi(s_j,g_j)| <= d^pi_group~(...), so minimizing an expectation of the metric is a legitimate mechanism rather than a disguised restatement of demographic parity. The empirical fairness metrics (recall gap, credit gap, social burden) are evaluated externally and differ from the fitted terms J_rew. and J_dyn., so no fitted parameter is renamed as a prediction. The genuine weaknesses—Theorem 3 proves only an expectation bound in Appendix A.2 ('exactly equal to our definition of J in Equation (7)') while Definition 5 demands a pointwise per-state guarantee, and Section 7 concedes that convergence proofs for pi-bisimulation RL are open—are correctness and completeness gaps, not circular reductions.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The central claim rests on standard bisimulation theory (Castro 2020; Ferns et al. 2011) and on the stated domain assumptions that sensitive attributes are observable and constant over time. The method introduces learnable reward correction and observation modification parameters that are fitted to minimize the bisimulation discrepancy; these are the main free parameters. The only invented entity is an environment latent variable (conscientiousness) that is not part of the theoretical framework.

free parameters (5)
  • Reward correction weight alpha = 5 for PPO, 1.5 for DQN
    Scales the learned reward correction in Eq (9); chosen by grid search (Table 8) and controls how strongly the agent's reward is reshaped toward equality across groups.
  • Reward correction network parameters phi = Learned (no closed form)
    The network R_phi(s,a,g) is optimized by gradient descent to minimize J_rew (Eq 8); it changes the agent's objective and is the main reward-based fairness lever.
  • Observation dynamics parameters omega = Learned (no closed form)
    Credit score increments (c+, c-) and score-modification costs are optimized by (1+1)-ES to minimize J_dyn (Eq 10); these directly reshape the agent's observations and are central to the empirical results.
  • Dynamics model parameters psi = Trained on collected trajectories
    The group-conditioned transition model T_psi is fit to data to compute the W2 distance in Eq (11); its accuracy affects the J_dyn optimization.
  • Quantile matching granularity (quartiles) = 4 quartiles
    State pairs are matched by quartile of the group state distributions (Section 4, Appendix D.1); this choice determines which state pairs are used to estimate J_rew and J_dyn.
assumptions (6)
  • standard math Castro (2020) pi-bisimulation fixed point and value-difference bound (Theorems 2 and 3 of Castro)
    Theorems 1 and 2 in this paper reduce directly to Castro's results by concatenating S and G (Appendix A.1).
  • standard math Ferns et al. (2011) existence and continuity of bisimulation metric fixed points
    Invoked in the reduction proof of Theorem 1 (Appendix A.1).
  • domain assumption Assumption 1: sensitive attributes are observable to the decision making algorithm
    Made in Section 3 and used to define the group-conditioned MDP and the policy pi(a|s,g).
  • domain assumption Assumption 2: sensitive attributes and group memberships remain constant during training
    Used in the proof of Theorem 1 to treat G as a deterministic, constant component of the state, enabling the reduction to Castro (2020).
  • domain assumption Allowing reward and observation dynamics to be modified by the agent/regulator
    The method assumes it is legitimate to add a group-conditioned reward correction and to override credit scores or subsidize test-score costs (Sections 4.1, 4.2, 5.1, 5.2). This is a modeling choice about the real-world application, not a mathematical given.
  • domain assumption Infinite-horizon extension of demographic parity with discounted returns
    Appendix A.2 replaces the finite-horizon definition of Satija et al. (2023) with the discounted expected return; the theoretical claim is stated for this adapted notion.
invented entities (1)
  • Conscientiousness latent variable (lending environment)
    purpose: Makes the probability of loan repayment depend on an unobservable factor beyond credit score, creating a more sequential and challenging benchmark (Section 5.1, Appendix B.1)
    Introduced as an environment extension in the simulation; it has no falsifiable handle outside the paper and is not part of the theoretical framework.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fairness in Reinforcement Learning with Bisimulation Metrics." pith.science (2026). https://pith.science/paper/IFX6NX7U

@misc{pith2026241217123,
  author       = {Pith},
  title        = {Pith review of: Fairness in Reinforcement Learning with Bisimulation Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IFX6NX7U}},
  note         = {Machine review of arXiv:2412.17123}
}
read the original abstract

Ensuring long-term fairness is crucial when developing automated decision making systems, specifically in dynamic and sequential environments. By maximizing their reward without consideration of fairness, AI agents can introduce disparities in their treatment of groups or individuals. In this paper, we establish the connection between bisimulation metrics and group fairness in reinforcement learning. We propose a novel approach that leverages bisimulation metrics to learn reward functions and observation dynamics, ensuring that learners treat groups fairly while reflecting the original problem. We demonstrate the effectiveness of our method in addressing disparities in sequential decision making problems through empirical evaluation on a standard fairness benchmark consisting of lending and college admission scenarios.

Figures

Figures reproduced from arXiv: 2412.17123 by the authors.

Figure 1
Figure 1. Lending results. The first row (a-d) shows the lending scenario where the repayment probability is only a function of the credit score, while the second row (e-f) presents the case where the repayment probability is a function of the credit score and a latent conscientiousness parameter. (a, e) Average return. (b, f) Recall for group 1. (c, g) Recall for group 2. (d, h) Credit gap measured as the Kantorovich distanc… view at source ↗
Figure 2
Figure 2. Credit gaps of Bisim￾ulator and PPO. Solid lines show the gap between the actual credit scores that govern the MDP dy￾namics, and the dashed line shows the gap between the modified credit scores that are observed by the agent. Fairness Metrics. Similarly to D’Amour et al. (2020), we use three metrics for evaluating the long-term fairness: (a) changes in the credit score distributions measured by the Kantorovich dist… view at source ↗
Figure 3
Figure 3. College admission results. The shaded regions show 95% confidence intervals and plots [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Initial credit score distribution for each group. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Lending results. Cumulative loans given to each group over the course of evaluation [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Lending results. Recall gaps between the two groups over the training steps. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Comparison of Bisimulator and Bisimulator (Reward only). The first row [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: College admission results. Cumulative admissions granted to each group over the course [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: College admission results. Recall values for each group over the training steps. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: College admission results. (a) Average return. (b) Recall gap. (c) Social burden for group 1. (d) Social burden for group 2. Results are obtained on 10 seeds and 5 evaluations episodes per seed. The shaded regions show 95% confidence intervals and plots are smoothed f…
Figure 11
Figure 11. Figure 11: Lending results with 10 groups. (a) Average return. (b) Recall gap, (c) Mean, and (d) Standard deviation of the recall across all groups. The shaded regions show 95% confidence intervals and plots are smoothed for visual clarity [PITH_FULL_IMAGE:figures/full_fig_p024…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 51 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Design and control of soft robots using differentiable simulation

    Moritz B \"a cher, Espen Knoop, and Christian Schumacher. Design and control of soft robots using differentiable simulation. Current Robotics Reports, 2 0 (2): 0 211--221, 2021

    work page 2021

  3. [3]

    Fairness and machine learning: Limitations and opportunities

    Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limitations and opportunities. MIT press, 2023

    work page 2023

  4. [4]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    arXiv 2013

  5. [5]

    My fair bandit: Distributed learning of max-min fairness with multi-player bandits

    Ilai Bistritz, Tavor Baharav, Amir Leshem, and Nicholas Bambos. My fair bandit: Distributed learning of max-min fairness with multi-player bandits. In International Conference on Machine Learning, pp.\ 930--940. PMLR, 2020

    work page 2020

  6. [6]

    Jack Blandin and Ian A. Kash. Group fairness in reinforcement learning via multi-objective rewards. Transactions on Machine Learning Research, 2024. ISSN 2835-8856

    work page 2024

  7. [7]

    Openai gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  8. [8]

    Scalable methods for computing state similarity in deterministic markov decision processes

    Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 10069--10076, 2020

    2020

Show all 78 references
  1. [9]

    Scalable co-optimization of morphology and control in embodied machines

    Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of morphology and control in embodied machines. Journal of The Royal Society Interface, 15 0 (143): 0 20170937, 2018

    2018

  2. [10]

    Heuristic-guided reinforcement learning

    Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 13550--13563, 2021

    2021

  3. [11]

    Intrinsically motivated reinforcement learning

    Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, 17, 2004

    2004

  4. [12]

    On welfare-centric fair reinforcement learning

    Cyrus Cousins, Kavosh Asadi, Elita Lobo, and Michael Littman. On welfare-centric fair reinforcement learning. Reinforcement Learning Journal, 3: 0 1124--1137, 2024

    2024

  5. [13]

    Fairness is not static: deeper understanding of long term fairness via simulation studies

    Alexander D'Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, David Sculley, and Yoni Halpern. Fairness is not static: deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency,...

    2020

  6. [14]

    What hides behind unfairness? exploring dynamics fairness in reinforcement learning

    Zhihong Deng, Jing Jiang, Guodong Long, and Chengqi Zhang. What hides behind unfairness? exploring dynamics fairness in reinforcement learning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 , pp.\ 3908--3916. Internation...

    2024

  7. [15]

    Desharnais, A

    J. Desharnais, A. Edalat, and P. Panangaden. Bisimulation for labeled Markov processes. Information and Computation, 179 0 (2): 0 163--193, Dec 2002

    2002

  8. [16]

    Dynamic potential-based reward shaping

    Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Proceedings of the 11th international conference on autonomous agents and multiagent systems, pp.\ 433--440. IFAAMAS, 2012

    2012

  9. [17]

    Contextual bandits with concave rewards, and an application to fair ranking

    Virginie Do, Elvis Dohmatob, Matteo Pirotta, Alessandro Lazaric, and Nicolas Usunier. Contextual bandits with concave rewards, and an application to fair ranking. In The Eleventh International Conference on Learning Representations, 2022

    2022

  10. [18]

    On the analysis of the (1+ 1) evolutionary algorithm

    Stefan Droste, Thomas Jansen, and Ingo Wegener. On the analysis of the (1+ 1) evolutionary algorithm. Theoretical Computer Science, 276 0 (1-2): 0 51--81, 2002

    2002

  11. [19]

    Fairness through awareness

    Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp.\ 214--226, 2012

    2012

  12. [20]

    Stochastic spatio-temporal optimization for control and co-design of systems in robotics and applied physics

    Ethan N Evans, Andrew P Kendall, and Evangelos A Theodorou. Stochastic spatio-temporal optimization for control and co-design of systems in robotics and applied physics. Autonomous Robots, pp.\ 1--24, 2022

    2022

  13. [21]

    Fair lending implications of credit scoring systems

    The Federal Deposit Insurance Corporation FDIC. Fair lending implications of credit scoring systems. https://www.fdic.gov/regulations/examinations/supervisory/insights/sisum05/sisummer2005-article03.html, 2005. [Last updated 23-07-2023]

    2005

  14. [22]

    Metrics for finite M arkov decision processes

    Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite M arkov decision processes. In UAI, volume 4, pp.\ 162--169, 2004

    2004

  15. [23]

    Bisimulation metrics for continuous M arkov decision processes

    Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous M arkov decision processes. SIAM Journal on Computing, 40 0 (6): 0 1662--1714, 2011

    2011

  16. [24]

    Fair off-policy learning from observational data

    Dennis Frauen, Valentyn Melnychuk, and Stefan Feuerriegel. Fair off-policy learning from observational data. In Forty-first International Conference on Machine Learning, 2024

    2024

  17. [25]

    Potential based reward shaping for hierarchical reinforcement learning

    Yang Gao and Francesca Toni. Potential based reward shaping for hierarchical reinforcement learning. In Proceedings of the 24th International Conference on Artificial Intelligence, pp.\ 3504--3510, 2015

    2015

  18. [26]

    Equivalence notions and model minimization in M arkov decision processes

    Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in M arkov decision processes. Artificial Intelligence, 147 0 (1-2): 0 163--223, 2003

    2003

  19. [27]

    Bisimulation makes analogies in goal-conditioned reinforcement learning

    Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisimulation makes analogies in goal-conditioned reinforcement learning. In International Conference on Machine Learning, pp.\ 8407--8426. PMLR, 2022

    2022

  20. [28]

    Strategic classification

    Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science, pp.\ 111--122, 2016 a

    2016

  21. [29]

    Equality of opportunity in supervised learning

    Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016 b

    2016

  22. [30]

    Fair algorithms for multi-agent multi-armed bandits

    Safwan Hossain, Evi Micha, and Nisarg Shah. Fair algorithms for multi-agent multi-armed bandits. Advances in Neural Information Processing Systems, 34: 0 24005--24017, 2021

    2021

  23. [31]

    Achieving long-term fairness in sequential decision making

    Yaowei Hu and Lu Zhang. Achieving long-term fairness in sequential decision making. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 9549--9557, 2022

    2022

  24. [32]

    Striking a balance in fairness for dynamic systems through reinforcement learning

    Yaowei Hu, Jacob Lear, and Lu Zhang. Striking a balance in fairness for dynamic systems through reinforcement learning. In 2023 IEEE International Conference on Big Data (BigData), pp.\ 662--671. IEEE, 2023

    2023

  25. [33]

    Chainqueen: A real-time differentiable physical simulator for soft robotics

    Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simulator for soft robotics. In 2019 International conference on robotics and automation (ICRA), p...

    2019

  26. [34]

    Learning to utilize shaping rewards: A new approach of reward shaping

    Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33: 0 15931--15941, 2020

    2020

  27. [35]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 0 (274): 0 1--18, 2022

    2022

  28. [36]

    Fairness in reinforcement learning

    Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in reinforcement learning. In International conference on machine learning, pp.\ 1617--1626. PMLR, 2017

    2017

  29. [37]

    Learning fairness in multi-agent systems

    Jiechuan Jiang and Zongqing Lu. Learning fairness in multi-agent systems. Advances in Neural Information Processing Systems, 32, 2019

    2019

  30. [38]

    Fairness in learning: Classic and contextual bandits

    Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. Advances in neural information processing systems, 29, 2016

    2016

  31. [39]

    Achieving fairness in multi-agent markov decision processes using reinforcement learning

    Peizhong Ju, Arnob Ghosh, and Ness B Shroff. Achieving fairness in multi-agent markov decision processes using reinforcement learning. arXiv preprint arXiv:2306.00324, 2023

    2023 arXiv

  32. [40]

    Stochastic hillclimbing as a baseline method for evaluating genetic algorithms

    Ari Juels and Martin Wattenberg. Stochastic hillclimbing as a baseline method for evaluating genetic algorithms. Advances in Neural Information Processing Systems, 8, 1995

    1995

  33. [41]

    Towards robust bisimulation metric learning

    Mete Kemertas and Tristan Aumentado-Armstrong. Towards robust bisimulation metric learning. Advances in Neural Information Processing Systems, 34: 0 4764--4777, 2021

    2021

  34. [43]

    The long road to fairer algorithms

    Matt J Kusner and Joshua R Loftus. The long road to fairer algorithms. Nature, 578 0 (7793): 0 34--36, 2020

    2020

  35. [44]

    Larsen and Arne Skou

    Kim G. Larsen and Arne Skou. Bisimulation through probabilistic testing. Information and Computation, 94 0 (1): 0 1--28, 1991. ISSN 0890-5401. doi:https://doi.org/10.1016/0890-5401(91)90030-6. URL https://www.sciencedirect.com/science/article/pii/0890540191900306

    1991

  36. [45]

    Delayed impact of fair machine learning

    Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pp.\ 3150--3158. PMLR, 2018

    2018

  37. [46]

    Calibrated fairness in bandits

    Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David C Parkes. Calibrated fairness in bandits. arXiv preprint arXiv:1707.01875, 2017

    2017 arXiv

  38. [47]

    Diffaqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation

    Pingchuan Ma, Tao Du, John Z Zhang, Kui Wu, Andrew Spielberg, Robert K Katzschmann, and Wojciech Matusik. Diffaqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation. ACM Transactions on Graphics (TOG), 40 0 (4): 0 1--14, 2021

    2021

  39. [48]

    Socially fair reinforcement learning

    Debmalya Mandal and Jiarui Gan. Socially fair reinforcement learning. arXiv preprint arXiv:2208.12584, 2022

    2022 arXiv

  40. [49]

    Comparison of three methods for selecting values of input variables in the analysis of output from a computer code

    MD McKay, RJ Beckman, and WJ Conover. Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21 0 (2): 0 239--245, 1979

    1979

  41. [50]

    A survey on bias and fairness in machine learning

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54 0 (6): 0 1--35, 2021

    2021

  42. [51]

    Offline contextual bandits with high probability fairness guarantees

    Blossom Metevier, Stephen Giguere, Sarah Brockman, Ari Kobren, Yuriy Brun, Emma Brunskill, and Philip S Thomas. Offline contextual bandits with high probability fairness guarantees. Advances in neural information processing systems, 32, 2019

    2019

  43. [52]

    The social cost of strategic classification

    Smitha Milli, John Miller, Anca D Dragan, and Moritz Hardt. The social cost of strategic classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp.\ 230--239, 2019

    2019

  44. [53]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  45. [54]

    Learning optimal fair policies

    Razieh Nabi, Daniel Malinsky, and Ilya Shpitser. Learning optimal fair policies. In International Conference on Machine Learning, pp.\ 4674--4682. PMLR, 2019

    2019

  46. [55]

    Fairness and sequential decision making: Limits, lessons, and opportunities

    Samer B Nashed, Justin Svegliato, and Su Lin Blodgett. Fairness and sequential decision making: Limits, lessons, and opportunities. arXiv preprint arXiv:2301.05753, 2023

    2023 arXiv

  47. [56]

    Policy invariance under reward transformations: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart J Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, pp.\ 278--287, 1999

    1999

  48. [57]

    Labelled Markov Processes

    Prakash Panangaden. Labelled Markov Processes. IMPERIAL COLLEGE PRESS, 2009. doi:10.1142/p595. URL https://www.worldscientific.com/doi/abs/10.1142/p595

    2009 doi

  49. [58]

    Discrimination-aware data mining

    Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. Discrimination-aware data mining. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.\ 560--568, 2008

    2008

  50. [59]

    Group fairness in reinforcement learning

    Harsh Satija, Alessandro Lazaric, Matteo Pirotta, and Joelle Pineau. Group fairness in reinforcement learning. Trans. Mach. Learn. Res., 2023, 2023

    2023

  51. [60]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  52. [61]

    Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards

    Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In International Conference on Machine Learning, pp.\ 8905--8915. PMLR, 2020

    2020

  53. [62]

    Intrinsically motivated reinforcement learning: An evolutionary perspective

    Satinder Singh, Richard L Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2 0 (2): 0 70--82, 2010

    2010

  54. [63]

    Reward design via online gradient ascent

    Jonathan Sorg, Richard L Lewis, and Satinder Singh. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23, 2010

    2010

  55. [64]

    Learning-in-the-loop optimization: End-to-end control and co-design of soft robots through learned deep latent representations

    Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du, Wojciech Matusik, and Daniela Rus. Learning-in-the-loop optimization: End-to-end control and co-design of soft robots through learned deep latent representations. Advances in Neural Information Processing Systems, 32, 2019

    2019

  56. [65]

    Co-learning of task and sensor placement for soft robotics

    Andrew Spielberg, Alexander Amini, Lillian Chin, Wojciech Matusik, and Daniela Rus. Co-learning of task and sensor placement for soft robotics. IEEE Robotics and Automation Letters, 6 0 (2): 0 1208--1215, 2021

    2021

  57. [66]

    Terry, Ariel Kwiatkowski, John U

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasiu...

    2023

  58. [67]

    Softzoo: A soft robot co-design benchmark for locomotion in diverse environments

    Tsun-Hsuan Wang, Pingchuan Ma, Andrew Everett Spielberg, Zhou Xian, Hao Zhang, Joshua B Tenenbaum, Daniela Rus, and Chuang Gan. Softzoo: A soft robot co-design benchmark for locomotion in diverse environments. arXiv preprint arXiv:2303.09555, 2023

    2023 arXiv

  59. [68]

    Curriculum-based co-design of morphology and control of voxel-based soft robots

    Yuxing Wang, Shuang Wu, Haobo Fu, Qiang Fu, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Curriculum-based co-design of morphology and control of voxel-based soft robots. In The Eleventh International Conference on Learning Representations, 2022

    2022

  60. [69]

    Algorithms for fairness in sequential decision making

    Min Wen, Osbert Bastani, and Ufuk Topcu. Algorithms for fairness in sequential decision making. In International Conference on Artificial Intelligence and Statistics, pp.\ 1144--1152. PMLR, 2021

    2021

  61. [70]

    Adapting static fairness to sequential decision-making: Bias mitigation strategies towards equal long-term benefit rate

    Yuancheng Xu, Chenghao Deng, Yanchao Sun, Ruijie Zheng, Xiyao Wang, Jieyu Zhao, and Furong Huang. Adapting static fairness to sequential decision-making: Bias mitigation strategies towards equal long-term benefit rate. In Forty-first International Conference on Machine Learning, 2024

    2024

  62. [71]

    Long-term fairness with unknown dynamics

    Tongxin Yin, Reilly Raab, Mingyan Liu, and Yang Liu. Long-term fairness with unknown dynamics. arXiv preprint arXiv:2304.09362, 2023

    2023 arXiv

  63. [72]

    Policy optimization with advantage regularization for long-term fairness in decision systems

    Eric Yu, Zhizhen Qin, Min Kyung Lee, and Sicun Gao. Policy optimization with advantage regularization for long-term fairness in decision systems. Advances in Neural Information Processing Systems, 35: 0 8211--8213, 2022

    2022

  64. [73]

    Fair deep reinforcement learning with preferential treatment

    Guanbao Yu, Umer Siddique, and Paul Weng. Fair deep reinforcement learning with preferential treatment. In ECAI, pp.\ 2922--2929, 2023

    2023

  65. [74]

    Learning invariant representations for reinforcement learning without reconstruction

    Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2020

    2020

  66. [75]

    Fairness in multi-agent sequential decision-making

    Chongjie Zhang and Julie A Shah. Fairness in multi-agent sequential decision-making. Advances in Neural Information Processing Systems, 27, 2014

    2014

  67. [76]

    On learning intrinsic rewards for policy gradient methods

    Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018

    2018

  68. [77]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  69. [78]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  70. [79]

    Desharnais and A

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2020 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.