{"id":"b344daef-dabc-4ddb-8d81-5f61f0650f35","arxiv_id":"2412.17123","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Bisimulator uses bisimulation metrics to modify rewards and observations, letting unconstrained RL policies approximately satisfy demographic parity in lending and college admissions benchmarks.","lead":"An RL fairness method, Bisimulator, reshapes the reward and the observations an agent sees, guided by a bisimulation metric, so that an unmodified RL algorithm treats different groups fairly. It matters because it offers a way to enforce long-term fairness in sequential decisions, like lending and admissions, without adding fairness constraints to the RL solver.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3 proves only an expectation bound, while Definition 5 requires a pointwise per-state bound; the proof in Appendix A.2 therefore does not establish demographic parity as stated.","rationale":"I read the paper as trying to establish that unconstrained RL on a reward- and observation-modified MDP produces a policy that is fair in the original problem. Theorem 2 is a clean reduction to Castro's pointwise bound. The load-bearing break is the step from Theorem 2 to Theorem 3: Jensen's inequality gives an expectation bound, but Definition 5 is pointwise, and Eq. (7) minimizes the expectation. This is not a cosmetic issue; because the minimized quantity is a stationary-distribution average over quantile-matched state pairs, it can vanish while some states retain large group gaps. The transfer concern raised by the reader is also real, but the expectation/pointwise gap is more immediate and directly invalidates the theorem as stated. The empirical results may support a weaker claim such as average-case fairness or practical fairness, and the paper is transparent about convergence limitations, so a conditional verdict with a corrected theorem or a softened claim is appropriate. I do not see a reason to reject the paper outright; the algorithmic idea is plausible and the experiments are informative, but the central theoretical guarantee needs repair or restatement.","tokens_in":26202,"tokens_out":6689,"duration_ms":68607,"concrete_test":"Re-derive Theorem 3 with the state in Definition 5 held fixed rather than averaged: instantiate the proof at a single state s and check whether |J^pi(s,g_1) - J^pi(s,g_2)| <= E_{rho}[d^pi_group~] is a valid inequality. Then construct a simple two-group MDP where Eq. (7) is zero for the quantile-matched pairs (e.g., all high-probability states have zero group-conditioned bisimulation distance) but some low-probability or unmatched state s has |V^pi(s,g_1) - V^pi(s,g_2)| > epsilon. If such an MDP exists, Theorem 3 is false as stated and the Appendix A.2 inequality cannot be converted into Definition 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 5 requires that for every state s and every pair of groups, |J^pi(s,g_i) - J^pi(s,g_j)| <= epsilon. The proof of Theorem 3 in Appendix A.2 instead starts from |E_{rho(s,g)}[V^pi(s_i,g_i)] - E_{rho(s,g)}[V^pi(s_j,g_j)]| and bounds this by E_{rho(s,g)}[d^pi_group~((s_i,g_i),(s_j,g_j))] <= epsilon. A bound on the stationary-distribution-weighted mean of the metric does not bound the metric at any fixed state. Moreover, the algorithmic objective in Eq. (7) is exactly this expectation and is estimated only over quantile-matched pairs, so even a perfect minimization of Eq. (7) to zero is compatible with arbitrarily large pointwise violations of Definition 5 at states that are rarely visited or not selected by quantile matching. The proof's statement that the third line is 'exactly equal to our definition of J in Equation (7)' confirms that only the expectation is controlled. What is shown is at most demographic parity in expectation over the on-policy state distribution, not the pointwise guarantee demanded by Definition 5, so Theorem 3 does not follow as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Bisimulator, a method for long-term group fairness in reinforcement learning that, instead of imposing fairness constraints during policy optimization, modifies the reward function and observation dynamics of a group-conditioned MDP so that an unconstrained RL solver (PPO or DQN) learns a policy approximating demographic parity. The authors introduce a group-conditioned π-bisimulation metric (Theorem 1), prove that it upper-bounds value-function differences between state-group pairs (Theorem 2), and claim that minimizing this metric yields demographic parity as defined in Definition 5 (Theorem 3). The practical algorithm learns a reward correction term and adjusts observation dynamics using a Gaussian dynamics model and the closed-form 2-Wasserstein distance. Experiments on lending and college-admission tasks from the ml-fairness-gym benchmark show that Bisimulator achieves low recall gaps and credit gaps relative to several fairness-constrained baselines, with ablations showing that a reward-only variant remains competitive. The paper includes 10-seed runs with confidence intervals, tuned baselines, and a 10-group lending extension.","tokens_in":26388,"tokens_out":6636,"duration_ms":59273,"significance":"The bisimulation-fairness connection is an interesting and potentially useful idea. If the central theoretical claim held, the method would provide an unconstrained alternative to constrained-MDP approaches and would be agnostic to the choice of RL algorithm. The empirical study is careful: it uses 10 seeds, reports 95% confidence intervals, tunes baselines with grid search, includes ablations (reward-only versus reward-plus-dynamics), and demonstrates scalability to 10 groups; the code is submitted as supplementary material. However, the theoretical guarantee is not established as stated. The proof of Theorem 3 controls only an expectation of the π-bisimulation metric over the stationary distribution, whereas Definition 5 requires a pointwise bound at every state, and no transfer argument connects the modified MDP in which the metric is minimized to the original MDP in which fairness is claimed. The practical algorithm also replaces the exact metric with learned approximations (a Gaussian dynamics model, the W2 distance, and quantile matching). The contribution is therefore conditional: it is a promising empirical method with an incompletely justified theoretical foundation.","major_comments":[{"comment":"The proof of Theorem 3 bounds E_{ρ(s,g)}[|V^π(s_i,g_i) - V^π(s_j,g_j)|] ≤ E_{ρ(s,g)}[d^π_group~((s_i,g_i),(s_j,g_j))] ≤ ε, which is an expectation over the stationary state-group distribution. Definition 5 requires a pointwise bound |J^π(s,g_i) - J^π(s,g_j)| ≤ ε for every state s. A small expectation does not control pointwise values: the metric can be arbitrarily large on states with low stationary mass, and since Eq. (7) minimizes precisely this expectation over quantile-matched pairs, perfect minimization of Eq. (7) is compatible with large violations of Definition 5 at rarely visited states. The theorem should be restated as a demographic-parity-in-expectation guarantee (with a matching amendment of Definition 5), or the proof needs an additional argument establishing a pointwise bound, which is not provided.","section":"Appendix A.2 / Theorem 3"},{"comment":"Theorems 1 and 2 are stated for a fixed group-conditioned MDP. In Bisimulator, the reward is replaced by R_original + α R_φ (Eq. 9) and the observation dynamics are modified (Section 4.2), so the π-bisimulation metric is computed for a modified MDP M'. The introduction claims that unconstrained policy optimization in the modified MDP satisfies fairness in the original, unmodified MDP, but the paper provides no transfer result showing that a policy that equalizes returns under the modified reward and observations also equalizes returns under the original dynamics and reward. Without such a bound, the fair behavior observed in the modified MDP need not persist in the original environment, which is the environment actually evaluated in Section 5. This gap is load-bearing for the paper's central claim.","section":"Section 4 / Algorithm 1"},{"comment":"The objective J in Eq. (7) is defined in terms of the exact π-bisimulation metric d^π_group~, but Algorithm 1 approximates it by training a Gaussian dynamics model T_ψ and using the closed-form 2-Wasserstein distance (Eq. 11) over quartile-matched batches. These substitutions are not covered by Theorems 1-3. To support the statement that the algorithm minimizes the bisimulation metric, the paper should either provide error bounds between the true metric and the estimated objective, or explicitly state that the method is a heuristic approximation whose theoretical guarantee applies only to the exact, non-implemented objective.","section":"Section 4.2 / Eq. (11)"}],"minor_comments":[{"comment":"Definition 5 uses the notation J^π(s,g_i), but J^π was previously defined as an unconditional expectation over ρ^π(s,g); the conditional quantity should be defined explicitly (for example, as V^π(s,g) or as a state-conditioned return) to avoid ambiguity.","section":"Definition 5"},{"comment":"The paragraph titled 'Fairness Metrics' in the lending environment appendix describes social burden and cumulative admissions, which are not used in the lending experiments; this appears to be a copy-paste error from the college admissions section.","section":"Appendix B.1"},{"comment":"In the 'Baselines' paragraph of the college admissions section, the text says 'the same RL baselines described in Section 5.2' but should refer to Section 5.1.","section":"Section 5.2"},{"comment":"Equation (7) leaves the joint distribution over the pair ((s_i,g_i),(s_j,g_j)) unspecified; the expectation should indicate that pairs are drawn from the per-group stationary distributions (or via the quantile-matching procedure) before minimization.","section":"Equation (7)"},{"comment":"The proofs of Theorems 1 and 2 reduce to results in Castro (2020), but the paper should state the regularity conditions under which Castro's fixed-point and value-bound theorems apply (for example, finite or compact state spaces with continuous rewards and transitions) and verify that the extended MDP with state space S×G satisfies them.","section":"Appendix A.1"},{"comment":"The authors acknowledge that convergence of the alternating policy-update and metric-minimization scheme is an open topic; this caveat is important for the practical algorithm and should also be stated where Theorem 3 is invoked, rather than only in the limitations section.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is an interesting empirical contribution with a potentially appealing theoretical framing. The main theoretical gaps—the pointwise-versus-expectation mismatch in Theorem 3 and the missing transfer from the shaped MDP to the original environment—are substantial and require new results, not just local edits. If the authors can either strengthen the theory (for example, by proving a uniform bound or an explicit transfer inequality) or reframe the contribution as an empirical method with a heuristic connection to bisimulation, the paper could be publishable. The current abstract and introduction overstate the guarantee, so the revision should align the claims with what is actually proved; the careful empirical evaluation is a definite strength and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The algorithmic idea is genuinely new: instead of constrained optimization, Bisimulator reshapes rewards and observation dynamics using a group-conditioned pi-bisimulation metric, then runs an unmodified RL solver. That is a real contribution, and the empirical study is careful—10 seeds, CIs, grid-searched baselines, plus an honest extension of the lending benchmark with a latent conscientiousness variable. On the college admissions domain the method is cleanly better than the constrained baselines, and it matches or beats ELBERT-PO on most lending metrics while staying solver-agnostic.\n\nThe theory does not support the prose. Theorem 3 promises demographic parity per Definition 5 for every state, but the proof in A.2 bounds the stationary-distribution-weighted expectation of the value difference, not the pointwise value. The stress-test note is right: even zeroing the objective in Eq. (7) is compatible with arbitrarily large pointwise violations at states that are rarely visited or not selected by quantile matching. Since Eq. (7) is literally that expectation, the gap is structural. The fix is straightforward—either restate Theorem 3 as an expectation bound, or add a uniform pointwise argument. A second soft spot is that Theorem 2 bounds value differences in the modified MDP; there is no transfer theorem showing a policy that is fair there is fair in the original environment. The empirical evidence, particularly Figure 2, mitigates this, but the formal claim exceeds the formal support.\n\nMinor: code is promised in the supplemental but not publicly available; that limits reproducibility of the empirical claims. Not fatal.\n\nThis paper is for the RL fairness subfield and for people working on bisimulation metrics. The algorithm and the experimental protocol deserve a serious referee; the theorem statements do not, as written. I would send it to review and require a major revision that honestly restates what is proven. I would cite the algorithm and the empirical results, but not the current Theorem 3.","headline":"A genuinely new fairness-via-bisimulation algorithm with a careful empirical study, but the main theorem overreaches: the proof gives only an expectation bound, and transfer to the original MDP is asserted rather than proven.","tokens_in":27020,"tokens_out":2279,"would_cite":true,"duration_ms":20984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","90C40","68Q85"],"pacs":[],"model":"deepseek-v4-flash","headline":"Minimizing a group-conditioned bisimulation metric between state-group pairs yields demographic parity in reinforcement learning.","keywords":["bisimulation metric","group fairness in reinforcement learning","demographic parity","reward shaping","observation dynamics","long-term fairness","sequential decision making","unconstrained policy optimization"],"falsifier":"Build a two-group MDP where the shaped reward and observation changes drive the group-conditioned $\\pi$-bisimulation metric below any chosen $\\epsilon$, then evaluate the resulting policy on the original environment: if any state $s$ has $|V^\\pi(s,g_1)-V^\\pi(s,g_2)| > \\epsilon$ while the expected bound used in Theorem 3 is below $\\epsilon$, the pointwise demographic parity claim fails.","tokens_in":25924,"feed_emoji":"⚖️","tokens_out":10362,"duration_ms":85602,"temperature":0.7,"pith_summary":"This paper tries to show that group fairness in sequential decision making can be achieved without adding fairness constraints to the learner. The central claim is that minimizing a group-conditioned bisimulation metric—a distance that measures how differently two groups behave under a given policy in terms of immediate rewards and future state distributions—between state-group pairs makes an unconstrained reinforcement learning agent approximately satisfy demographic parity. The proposed algorithm, Bisimulator, does this by reshaping the reward signal and the observations the agent receives, while leaving the underlying environment dynamics and the choice of RL solver unchanged. If the claim holds, fairness becomes a property of reward and observation design rather than of constrained policy optimization, which is why it would matter for real deployments.","feed_headline":"One metric can make unconstrained RL fair across groups","feed_subtitle":"Reshaping rewards and observations guided by behavioral similarity equalizes long-term returns across sensitive groups.","key_machinery":"The load-bearing object is the group-conditioned $\\pi$-bisimulation operator $F^\\pi_{\\mathrm{group}}$ of Equation (5), which measures behavioral dissimilarity between state-group pairs $(s_i,g_i)$ and $(s_j,g_j)$ as the absolute difference in policy-conditioned rewards plus a discounted Wasserstein distance between policy-conditioned next-state distributions. It is a conditional extension of the policy-conditioned bisimulation metric, with group membership treated as a deterministic constant transition, so the fixed point $d^\\pi_{\\mathrm{group}\\sim}$ bounds value differences (Theorem 2) and controls the fairness gap. The method minimizes its expectation over the stationary state-group distribution, using quantile matching to pair comparable states across groups, a learnable reward correction $R_\\phi(s,a,g)$ for the reward term, and a learned dynamics model $T_\\psi(s'|s,a,g)$ with the 2-Wasserstein distance replacing the 1-Wasserstein distance for the dynamics term.","core_discovery":"The paper establishes a formal bridge between bisimulation theory and group fairness in reinforcement learning. It defines a group-conditioned $\\pi$-bisimulation metric on pairs consisting of a state and a sensitive group, proves that the associated operator has a least fixed point (Theorem 1), and proves that this fixed point upper-bounds the difference in expected returns between two state-group pairs under a fixed policy (Theorem 2). Theorem 3 then concludes that minimizing the metric in expectation, through the two-term objective of Equation (7), yields demographic parity fairness in the sense of the adopted RL definition. The practical algorithm minimizes the reward term with a learnable group-conditioned reward correction and the dynamics term by adjusting modifiable observation parameters, using a learned next-state model and the 2-Wasserstein distance between its predicted normal distributions. On lending and college admission benchmarks, the paper reports that Bisimulator, run with standard PPO or DQN, closes the recall gap and reduces the credit gap or social burden compared with constrained baselines.","pith_inferences":["An extension the paper does not explore is applying the same reward-and-observation shaping to other fairness notions, such as equality of opportunity, by weighting the metric over the relevant state or action pairs.","The pointwise-versus-expectation gap in Theorem 3 suggests a testable strengthening: a per-state version of the bound would be needed to fully match the pointwise demographic parity definition adopted in the paper.","A natural deployment test would be a recommendation platform where displayed scores are operator-controlled; the reward-only variant indicates that even fixed environments could benefit from this approach."],"forward_implications":["Any off-the-shelf RL algorithm can be used without modification, because fairness enters only through rewards and observations.","A regulator who controls rewards or observable features, such as credit-score overrides or subsidized test-preparation costs, can implement fairness without controlling the learner's objective.","The single unconstrained objective in Equation (7) replaces Lagrangian or constrained-optimization machinery, removing a class of hyperparameters and algorithmic couplings.","The objective extends to more than two sensitive groups by summing over group pairs; the paper demonstrates this on a ten-group lending variant.","The reward-only variant of Bisimulator remains competitive, so fairness gains do not strictly require the ability to change observations."],"supporting_citations":[{"why":"Establishes the bisimulation metric, its unique fixed point, and the bound relating the metric to differences in optimal value functions.","marker":"Ferns et al. (2004; 2011)"},{"why":"Defines the policy-conditioned (pi-)bisimulation metric and the value-difference bound that the paper extends to state-group pairs.","marker":"Castro (2020)"},{"why":"Supplies the definition of demographic parity fairness in RL that the paper adopts as its target.","marker":"Satija et al. (2023)"},{"why":"Provides the lending and college admission benchmark environments used in the empirical evaluation.","marker":"D'Amour et al. (2020)"},{"why":"Motivates replacing the 1-Wasserstein distance with the analytically computable 2-Wasserstein distance for Gaussian next-state distributions.","marker":"Zhang et al. (2020)"},{"why":"Supplies the quantile-matching technique used to pair comparable state samples across groups.","marker":"McKay et al. (1979)"},{"why":"Introduces the lending loan-application scenario that the benchmark extends.","marker":"Liu et al. (2018)"},{"why":"Introduces demographic parity as a group fairness criterion, which the paper adopts through the RL-specific definition.","marker":"Dwork et al. (2012)"}],"fun_headline_variants":["Bisimulation metrics make RL fair across groups","Reshaping rewards with bisimulation achieves group fairness in RL","Fair RL via bisimulation: reshaping rewards and observations","Bisimulation metric minimizes to ensure demographic parity in RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that making groups behave similarly in the modified problem, with reshaped rewards and altered observations, makes the learned policy fair in the original problem; the supporting bound is proved only in the modified MDP and only in expectation, while the fairness definition is pointwise, and the paper supplies no transfer theorem.","fun_headline_variants_meta":{"raw":{"variants":["Bisimulation metrics make RL fair across groups","Reshaping rewards with bisimulation achieves group fairness in RL","Fair RL via bisimulation: reshaping rewards and observations","Bisimulation metric minimizes to ensure demographic parity in RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2680,"prompt_tokens":833,"completion_tokens":1847,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":1782}},"tokens_in":449,"tokens_out":1847,"duration_ms":12637,"temperature":1.0,"reasoning_tokens":1782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:47:00.672439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a two-group MDP where the shaped reward and observation changes drive the group-conditioned $\\pi$-bisimulation metric below any chosen $\\epsilon$, then evaluate the resulting policy on the original environment: if any state $s$ has $|V^\\pi(s,g_1)-V^\\pi(s,g_2)| > \\epsilon$ while the expected bound used in Theorem 3 is below $\\epsilon$, the pointwise demographic parity claim fails.","supporting_citations":[{"cited_title":"Metrics for finite M arkov decision processes","cited_arxiv_id":null,"evidence_quote":"Establishes the bisimulation metric, its unique fixed point, and the bound relating the metric to differences in optimal value functions."},{"cited_title":"Group fairness in reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of demographic parity fairness in RL that the paper adopts as its target."},{"cited_title":"Fairness is not static: deeper understanding of long term fairness via simulation studies","cited_arxiv_id":null,"evidence_quote":"Provides the lending and college admission benchmark environments used in the empirical evaluation."},{"cited_title":"Learning invariant representations for reinforcement learning without reconstruction","cited_arxiv_id":null,"evidence_quote":"Motivates replacing the 1-Wasserstein distance with the analytically computable 2-Wasserstein distance for Gaussian next-state distributions."},{"cited_title":"Comparison of three methods for selecting values of input variables in the analysis of output from a computer code","cited_arxiv_id":null,"evidence_quote":"Supplies the quantile-matching technique used to pair comparable state samples across groups."},{"cited_title":"Delayed impact of fair machine learning","cited_arxiv_id":null,"evidence_quote":"Introduces the lending loan-application scenario that the benchmark extends."}],"review_version":1}