{"id":"4450f8c1-69dc-4dac-b21e-8fb5f9d5a4fd","arxiv_id":"2608.08743","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A quantile-based data preprocessing method, CFSMDM, makes offline reinforcement learning approximately counterfactually fair under non-additive noise, with bounded suboptimality and unfairness.","lead":"This paper introduces a data preprocessing method that makes reinforcement learning policies counterfactually fair, so a patient's race or gender would not change the care they receive. It generalizes prior work by handling non-additive noise, with mathematical guarantees and tests on a digital health study for opioid misuse.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4's stated unfairness bound is ill-defined on the event δ_{T,n} > λ0/(4dL), and the proof imports its key step from [31] without derivation.","rationale":"The reader's weakest_assumption is the strict monotonicity in Assumptions 1-2. That is a genuine identifiability condition, but it is explicitly stated and its violation is an acknowledged limitation. A sharper, text-verifiable problem is that the main theorems are not actually proved in the manuscript: Theorem 3 and Theorem 4 both import the decisive FQI and quantile-estimation error controls from the self-cited preprint [31], and Remark A1 only asserts the needed high-probability quantile convergence without proof. More seriously, Theorem 4 as stated has an internal inconsistency on the event δ_{T,n} > λ0/(4dL), where the denominator of ξ_n becomes negative and O(ξ_n^α) is undefined for non-integer α. This is not a matter of disagreement with external consensus; it is a correctness issue visible from the theorem statement itself. The simulation results against a known SCM provide empirical support and should be credited, but they do not exercise the bad event that the theorem's own probability term retains, so they do not settle the validity of the bound. The appropriate disposition is conditional acceptance: the central claim can stand only after the proof is made self-contained and the theorem statement is repaired to handle the complement event explicitly.","tokens_in":38099,"tokens_out":7136,"duration_ms":81669,"concrete_test":"Re-derive Eq. (4) in the proof of Theorem 4 from Assumptions A1-A4 without citing [31], and re-state Theorem 4 with ξ_n replaced by a truncated positive version, e.g. ξ_n 1{δ_{T,n} ≤ λ0/(4dL)} plus a constant on the complement. Verify that (i) the re-derived Eq. (4) follows with explicit constants, and (ii) the expectation of the truncated O(ξ_n^α) term plus √2 P(δ_{T,n} > λ0/(4dL)) gives a finite, non-negative upper bound. If Eq. (4) cannot be derived, or the truncated bound does not follow, the theorem requires revision before the claimed guarantee can be accepted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The advertised central guarantee is Theorem 4's counterfactual unfairness bound. Two linked problems make it insecure as stated. First, the proof does not derive its key step: Eq. (4) is justified only by 'the same argument as the one in the proof of Theorem 4 in [31]', and Theorem 3 similarly imports the FQI error control from the authors' unpublished preprint [31]. No conditions or derivation are supplied, so Theorems 3-4 are not self-contained. Second, even granting the imported bound, Theorem 4's RHS is not well-defined outside the event B_n = {δ_{T,n} ≤ λ0/(4dL)}. The term ξ_n contains δ_{T,n}/(λ0(λ0 − 4dL δ_{T,n})); on B_n^c the denominator is negative, making ξ_n negative and ξ_n^α undefined for a general α > 0. The theorem nevertheless retains E[O(ξ_n^α)] on the whole probability space and separately adds √2 P(B_n^c). To be valid, the O(ξ_n^α) term must be restricted to B_n or truncated, and the complement event handled separately. As written, the theorem statement and proof are internally inconsistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CFSMDM, a data-preprocessing method that aims to enforce counterfactual fairness in offline reinforcement learning. The method sequentially estimates counterfactual states and rewards by matching quantile levels under different sensitive-attribute values, relying on strict monotonicity of states and rewards in scalar exogenous noises (Assumptions 1 and 2). The authors prove identifiability results (Theorems 1 and 2), give finite-sample bounds on the suboptimality gap and counterfactual unfairness of the learned policy (Theorems 3 and 4), and support the method with simulations under additive and nonadditive noise plus an application to the PowerED digital-health dataset.","tokens_in":1780,"tokens_out":1893,"duration_ms":107464,"significance":"If the theoretical guarantees are established, the paper makes a useful contribution by extending counterfactually fair RL preprocessing from additive-noise models to a strictly larger class of monotone structural models. The identification argument is natural and the explicit counterexample in Theorem A1 showing that monotonicity is strictly weaker than additivity is a genuine strength. The empirical section is thorough in its baselines, in its use of both additive and nonadditive CMDPs, and in its sensitivity analyses for sample size, effect size, horizon, and quantile-grid size. However, the two main performance bounds are not derived in the manuscript: the key FQI error estimates and the unfairness contraction are imported from the authors' unpublished preprint [31], and Theorem 4 as stated is ill-defined outside a high-probability event. The identification proof in Theorem 1 also contains an apparent typographical error in the t>=1 step. These issues must be resolved before the advertised guarantees can be considered verified.","major_comments":[{"comment":"The term E[O(xi_n^alpha)] is not defined on the event {delta_{T,n} > lambda0/(4dL)}. Since xi_n = C1 d L Rmax delta_{T,n}/(lambda0(lambda0 - 4dL delta_{T,n})(1-gamma)^2) + C2 d Rmax kappa log(n)/((1-gamma)^2 lambda0 sqrt(n)) + gamma B Rmax/(1-gamma), the denominator lambda0(lambda0 - 4dL delta_{T,n}) is negative on that event, so xi_n is negative and xi_n^alpha is not a real number for general alpha > 0. The proof applies the bound (4) only under A_n intersect B_n and then, after marginalizing over Xi_n, retains O(xi_n^alpha) over the whole probability space. The theorem should restrict the O(xi_n^alpha) term to B_n, use a truncated denominator such as (delta_{T,n} and lambda0/(8dL)), or state a conditional bound. As written, the statement and proof are internally inconsistent.","section":"Section 4, Theorem 4"},{"comment":"The main performance bounds are not self-contained. The proof of Theorem 3 obtains the high-probability control of the sup-norm error of the estimated Q function by 'the same argument as the one in the proof of Theorem 3 in [31]', and inequality (4) in the proof of Theorem 4 is justified only by 'the same argument as the one in the proof of Theorem 4 in [31]'. Reference [31] is an unpublished preprint by the same research group, and no statement of its conditions, lemmas, or derivations is reproduced. Since Theorems 3 and 4 are the central theoretical guarantees advertised in the abstract, the proofs must either be given in full or the precise dependencies on [31] must be stated so that a reader can verify them.","section":"Section I.3.4, proofs of Theorems 3 and 4"},{"comment":"The assertion that P(delta_{T,n} <= lambda0/(4dL)) converges to 1 exponentially fast for linear quantile regression and quantile-regression neural networks is unproved. This probability appears as a multiplicative factor in Theorem 3 and as an additive term in Theorem 4, so the bounds are not meaningful unless the claim is established. Remark A1 says it can be shown by concentration inequalities but gives no theorem, no conditions, and no citation. Either provide a rigorous statement with explicit regularity conditions on the quantile estimators, or replace Remark A1 by an explicit assumption in A1-A4.","section":"Remark A1"},{"comment":"In the t>=1 step of the proof of Theorem 1, the third equality is not justified as written. The displayed chain contains P(f^{S(i)}(z^{(j)}, a_{t-1}, s^{z^{(k)}}_{t-1}, U^{S(i)}_t) <= f^{S(i)}(z^{(j)}, a_{t-1}, s^{z^{(j)}}_{t-1}, u^{S(i)}_t)) as an intermediate term, but to equal P(U^{S(i)}_t <= u^{S(i)}_t), both sides of the inequality must use the same state argument s^{z^{(j)}}_{t-1}. With s^{z^{(k)}}_{t-1} on the left and s^{z^{(j)}}_{t-1} on the right, strict monotonicity in the noise alone does not imply the claimed equality. The analogous reward proof in Theorem 2 presumably needs the same correction. I suspect this is a typographical slip, but as it stands the proof of the identification result is incomplete.","section":"Section I.1, proof of Theorem 1"}],"minor_comments":[{"comment":"In the inductive step there is a typo: the maximum is written as max over i in {t,...,M}, but it should be max over i in {1,...,M}.","section":"Section I.3.3, proof of Lemma A1"},{"comment":"The notation in Assumption A4 is garbled: the set subtraction A \\ arg max_a Qopt(...) needs parentheses, and the roles of Q^* and Qopt should be defined consistently. As printed, the margin condition is hard to parse.","section":"Assumption A4"},{"comment":"In Step 4 of Algorithm 1, the conditional quantile function W is defined using P(R_t <= x | Z=z, S_{t-1}=s, A_{t-1}=a), which mixes the index of R with the index of (S,A). The same function is used in Step 15 for R_{t-1} and in Section I.3.1 it is defined as P(R_t | Z=z, S_t=s, A_t=a). The time index should be made consistent in the algorithm.","section":"Algorithm 1"},{"comment":"The text states that CFSMDM noticeably outperforms CFSDP for education and ethnicity, but the reported mean differences are small relative to the standard deviations, and for sex CFSMDM is slightly worse. A more cautious reading of the table is advisable.","section":"Section 6, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central identification result is promising and the experiments are reasonably thorough, but the manuscript cannot be accepted in its current form: Theorem 4 is ill-defined, Theorems 3-4 depend on the authors' unpublished preprint [31] without derivation, and Remark A1 asserts an unproved concentration claim. These are fixable within the scope of a revision, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a serious look. The core idea—using sequential quantile matching to estimate counterfactual states and rewards under strict monotonicity, subsuming additive noise as a special case—is genuinely new and cleanly motivated. Theorems 1 and 2 are self-contained and correct as far as I can tell, and the simulation on a known SCM supports the claim that the method works where CFSDP fails. The authors also openly acknowledge the circularity in the real-data CF metric (Remark 1), which is honest.\n\nThat said, the performance guarantees are not in usable shape. Theorem 4 as stated is internally inconsistent. The term ξ_n contains δ_{T,n}/(λ0(λ0 − 4dL δ_{T,n})). On the event δ_{T,n} > λ0/(4dL), the denominator is negative, so ξ_n is negative and ξ_n^α is undefined for general α > 0. The theorem still writes E[O(ξ_n^α)] over the whole space, then adds a separate term for P(B_n^c). The O term has to be restricted to B_n or truncated; as it stands, the theorem is not well-posed. The proof also imports its central step from the authors' own preprint [31] with 'the same argument as Theorem 4 in [31]'—no conditions, no derivation, and Theorem 3 does the same for the FQI error bound. Remark A1 asserts exponential convergence of the quantile estimators without proof. These are fixable, but they are not cosmetic.\n\nThe identification result and the algorithm are solid enough that I would send this to referees, with instructions to require the authors to make the bounds self-contained, to state clearly which results are borrowed and prove the quantile concentration, and to fix the Theorem 4 issue. If Theorem 4 can be repaired, this will be a useful tool for fairness-aware offline RL, especially in healthcare.","headline":"CFSMDM is a genuinely useful sequential extension of distribution-mapping fairness, but Theorem 4 as stated is internally inconsistent and the main performance bounds lean heavily on an unpublished preprint.","tokens_in":38893,"tokens_out":3183,"would_cite":true,"duration_ms":32975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantile matching makes reinforcement learning counterfactually fair.","keywords":["counterfactual fairness","reinforcement learning","data preprocessing","quantile regression","sequential marginal distribution mapping","offline reinforcement learning","causal inference","fairness in healthcare"],"falsifier":"Run CFSMDM on data generated from a known CMDP that satisfies all assumptions except strict monotonicity—for example, make the state depend on noise through a non-monotone function such as a sine or a quadratic U-shaped map—and check whether the empirical CF metric (the rate at which the learned policy changes its action for the same individual under switched sensitive attributes) remains small. Under the paper’s claim this metric should converge to zero as the sample size grows; a persistent large value would falsify the guarantee.","tokens_in":37842,"feed_emoji":"⚖️","tokens_out":5975,"duration_ms":60626,"temperature":0.7,"pith_summary":"This paper claims that a data-preprocessing step—matching each individual’s observed state and reward to the same conditional quantile under every possible sensitive attribute value—can strip sensitive information out of reinforcement-learning trajectories, so that the policy learned afterward treats each person the same whether they are, for example, one ethnicity or another. The method, called CFSMDM, works with existing offline RL algorithms and, unlike the main prior approach, does not require the noise in the environment dynamics to enter additively: strict monotonicity in a scalar noise variable is enough. The paper proves that under this monotonicity assumption, the learned policy is approximately counterfactually fair—its action probabilities change little if an individual’s sensitive attribute were switched—and that its loss in expected reward relative to the optimal policy is bounded. If the claim is right, fairness-sensitive applications such as clinical decision support can correct for implicit bias without redesigning the RL algorithm or assuming a restrictive model of the environment.","feed_headline":"Quantile matching makes reinforcement learning counterfactually fair","feed_subtitle":"A preprocessing step swaps a patient's sensitive attribute in data, so the learned policy gives each person the same chance of care.","key_machinery":"Sequential marginal distribution mapping (SMDM). The paper fits conditional quantile functions for the initial state, the transition distribution, and the reward distribution, all conditional on the sensitive attribute and history. For a given individual it estimates the quantile level of the observed state or reward under the observed attribute, then uses that same quantile level together with the quantile models under every alternative attribute value to reconstruct counterfactual trajectories. The invariance of quantile levels across counterfactual worlds (Theorems 1 and 2) is the load-bearing mechanism that turns observed data into counterfactual targets; the recursion through time is what makes the counterfactual state sequence well defined.","core_discovery":"On the paper’s own terms, the central claim is that counterfactual fairness in RL is achievable by sequential quantile distribution mapping. For each individual and each time step, one finds the quantile level of the observed state (and reward) within the conditional distribution given the individual’s sensitive attribute and past history; by Theorems 1 and 2 this quantile level is invariant to switching the sensitive attribute when each state dimension and reward is strictly increasing in its own scalar exogenous noise. The counterfactual state under an alternative attribute value is then read off at the same quantile from the conditional distribution under that alternative value, recursively through time. Stacking these counterfactual states into an augmented state, and weighting counterfactual rewards by the population distribution of the sensitive attribute, yields an MDP whose optimal policy is counterfactually fair by construction. Theorems 3 and 4 bound how far a policy learned on estimated augmented states and rewards is from that optimum and how much per-step unfairness remains, with both bounds shrinking as the sample size grows and quantile estimates improve.","pith_inferences":["One can test the load-bearing quantile-invariance assumption directly in any dataset: for individuals sharing the same observed history but different sensitive attributes, check whether the rank of each state or reward within its own conditional distribution is exchangeable; large deviations would cast doubt on strict monotonicity before deployment.","The augmented-state construction removes the sensitive attribute from the information available to the policy, so the same preprocessing should in principle make other objectives—such as fairness-constrained or safety-constrained RL—work on the same counterfactual representation.","Because the augmented reward is a weighted average of counterfactual rewards, the approach implicitly assumes the population mix of sensitive attributes is known; a deployment site with a different mix would need to reweight, a point the paper leaves implicit."],"forward_implications":["If the theorems hold, any existing offline RL algorithm can be made approximately counterfactually fair by first running SMDM on the training trajectories and then learning on the augmented states and rewards.","The fairness guarantee is individual-level: for each person, the probability of receiving a given action under the learned policy is nearly unchanged if the sensitive attribute were switched, not merely equal across groups.","The suboptimality gap bound means fairness is purchased at a controlled price: the policy learned on augmented data is close to the best counterfactually fair policy, and the gap shrinks with more data and better quantile estimation.","Because monotonicity is strictly weaker than additivity, the method covers all environments the additive-noise method CFSDP covers and more, including nonadditive dynamics such as the cube-root transitions used in the simulations."],"supporting_citations":[{"why":"Supplies the definition of counterfactual fairness in RL and the augmented-MDP theorem that any policy on augmented states and rewards is counterfactually fair, which this method builds on and extends.","marker":"[31]"},{"why":"Introduces counterfactual fairness for single-stage prediction, the individual-level criterion that the RL definition generalizes.","marker":"[19]"},{"why":"Provides the FLAP data-preprocessing algorithm for single-stage counterfactual fairness, which CFSMDM generalizes to sequential decision-making via marginal distribution mapping.","marker":"[2]"},{"why":"Establishes quantile regression, the estimation tool used to fit the conditional quantile models in SMDM.","marker":"[17]"},{"why":"Gives the regression quantiles estimation method used in the implementation of CFSMDM.","marker":"[18]"},{"why":"Supplies fitted Q iteration, the offline policy-learning algorithm used with the preprocessed trajectories and whose error terms appear in Theorems 3 and 4.","marker":"[28]"},{"why":"Provides the three-step procedure for counterfactual inference that underlies the formulation of counterfactual states, actions, and rewards in the generative model.","marker":"[25]"}],"fun_headline_variants":["Quantile mapping swaps sensitive attributes for fair RL","Sequential quantile mapping enables counterfactually fair RL","RL fairness by quantile-based counterfactual preprocessing","Quantile method ensures fair policies in RL","Counterfactual fairness in RL via quantile distribution mapping"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every state dimension and the reward are strictly increasing functions of their own scalar noise variable, so that the quantile level of an observed value is preserved when the sensitive attribute is switched; if that monotonicity fails, the counterfactual reconstructions are biased and the fairness and near-optimality guarantees do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Quantile mapping swaps sensitive attributes for fair RL","Sequential quantile mapping enables counterfactually fair RL","RL fairness by quantile-based counterfactual preprocessing","Quantile method ensures fair policies in RL","Counterfactual fairness in RL via quantile distribution mapping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000876,"raw_usage":{"total_tokens":3784,"prompt_tokens":933,"completion_tokens":2851,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2776}},"tokens_in":549,"tokens_out":2851,"duration_ms":20453,"temperature":1.0,"reasoning_tokens":2776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:26:00.275291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CFSMDM on data generated from a known CMDP that satisfies all assumptions except strict monotonicity—for example, make the state depend on noise through a non-monotone function such as a sine or a quadratic U-shaped map—and check whether the empirical CF metric (the rate at which the learned policy changes its action for the same individual under switched sensitive attributes) remains small. Under the paper’s claim this metric should converge to zero as the sample size grows; a persistent large value would falsify the guarantee.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FLAP data-preprocessing algorithm for single-stage counterfactual fairness, which CFSMDM generalizes to sequential decision-making via marginal distribution mapping."},{"cited_title":"(2005).Quantile Regression","cited_arxiv_id":null,"evidence_quote":"Establishes quantile regression, the estimation tool used to fit the conditional quantile models in SMDM."},{"cited_title":"and Bassett, G","cited_arxiv_id":null,"evidence_quote":"Gives the regression quantiles estimation method used in the implementation of CFSMDM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies fitted Q iteration, the offline policy-learning algorithm used with the preprocessed trajectories and whose error terms appear in Theorems 3 and 4."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the three-step procedure for counterfactual inference that underlies the formulation of counterfactual states, actions, and rewards in the generative model."}],"review_version":1}