{"id":"9e22d061-53c5-41ff-aeea-26d30572c45e","arxiv_id":"2608.09899","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper reproduces and stress-tests claims that exposure-based metrics NDKL and AWRF reveal subgroup-pair rank disparities invisible to demographic parity, and that MORAL's post-processing reduces them at minimal utility cost.","lead":"This study independently reruns and extends Mattos et al. (2025) on fair link prediction. It confirms that demographic parity can miss rank-position exposure disparities, that NDKL detects them, and that MORAL removes most of them with little utility loss.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All fairness metrics and MORAL's objective reference the same observed π; no alternative target is tested, so 'reduces these biases' is only established for distribution-preserving fairness, which the paper concedes in §3.3 and §5.","rationale":"The reader's weakest assumption — that observed π is the correct reference distribution — is the same concern that our independent analysis identifies as most load-bearing. The paper is transparent about this limitation, and claims C1/C2 (∆DP hides within-group exposure shifts; NDKL detects them) are convincingly supported and do not depend on the normative target. The vulnerability is specifically in the C3/abstract wording 'reduces these biases,' which is only established relative to the observed π. The AWRF check strengthens the result by changing the aggregation mechanism, but it does not address the shared-reference issue. Because the paper explicitly frames distribution-preserving fairness as a design choice, a conditional verdict is appropriate; no change is needed. The proposed concrete test with alternative π targets would sharpen the boundary of the claim and is worth running regardless.","tokens_in":27995,"tokens_out":6606,"duration_ms":64497,"concrete_test":"Re-run the six main experiments (Tables 2–4) with two alternative targets: (i) uniform π_alt = (1/3,1/3,1/3), and (ii) a corrective π_alt proportional to inverse observed pair-type frequency, over-weighting the least-exposed type. Retrain/re-rank MORAL against π_alt and evaluate NDKL_alt and AWRF_alt (computed with the same π_alt) for MORAL and the FairWalk/DetConstSort baselines, plus Precision/NDCG. If MORAL's relative advantage collapses or its NDKL_alt is large, the 'bias reduction' result is specific to distribution-preserving fairness; if MORAL still dominates under both alternative references, the concern is substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 sets MORAL's target to the observed subgroup-pair proportions π and explicitly calls distribution-preserving fairness 'a design choice... not a universally correct definition of fairness'; Section 5 repeats that low NDKL/AWRF should not support 'a more general claim about fairness.' Despite this, the abstract's headline claim is that MORAL 'reduces these biases.' The load-bearing problem is that every fairness quantity in the evaluation — MORAL's greedy KL objective (Algorithm 1), NDKL (Eq. 3), and the adapted AWRF (Eq. 7) — is computed against the same observed π. The AWRF robustness check therefore varies only the aggregation mechanism (prefix-KL vs. single ℓ1 exposure distance), not the reference distribution. If π itself encodes historical bias (e.g., homophily arising from past discrimination), matching π reproduces that bias while reporting near-zero NDKL/AWRF. The empirical results thus cannot distinguish 'reducing bias' from 'reproducing the observed graph composition.' This is not a hidden flaw: the authors state it. But it is load-bearing because the central claim's practical recommendation — 'use MORAL when top-K exposure is a fairness concern' — extends beyond the distribution-preserving objective, and no experiment tests an alternative target such as uniform exposure or a corrective reweighting. The paper's own caveats are in tension with the abstract's unqualified 'reduces these biases.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a reproducibility study of Mattos et al. (2025), which argues that dyadic demographic parity (ΔDP) can hide within-group exposure bias in ranked link prediction and proposes the rank-aware NDKL metric and the MORAL post-processing re-ranker. The authors formalize the original claims as C1–C3, reimplement the framework on six real-world datasets after identifying and correcting inconsistencies in the released code, and extend the original work with an asymmetric homophily stress-test on synthetic DPAH graphs, a metric robustness analysis using AWRF and NDCG, and categorical sensitive attributes up to 55 subgroup-pair types. The reproduced results show that ΔDP is largely invariant to within-group exposure shifts while NDKL changes substantially, and that MORAL achieves near-zero NDKL/AWRF with Precision/NDCG comparable to baselines. The paper is transparent about its limitations, explicitly stating in Section 3.3 and Section 5 that the fairness target π is the observed subgroup-pair distribution and that this is a design choice rather than a universally correct definition of fairness.","tokens_in":28261,"tokens_out":13186,"duration_ms":104075,"significance":"The paper's main strengths are its concrete reproducibility artifacts and the breadth of its robustness checks. It documents the discrepancies in the original codebase (Section 3.5), releases a corrected implementation, and reports hyperparameters, seeds, compute time, and CO2 estimates. The central empirical demonstrations—ΔDP invariance under intra-group exposure reallocation (Figure 2), and MORAL consistently attaining near-zero NDKL/AWRF with small standard deviations (Tables 2 and 4)—are clean and support the reproduced claims C1–C3 for the distribution-preserving fairness target. The homophily stress-test is a valuable addition, showing that exposure-based metrics are most sensitive at intermediate absolute homophily, and the categorical-attribute extension identifies a genuine fairness-scalability trade-off. The main limitation is acknowledged in the paper itself: every fairness quantity is measured against the same observed π, so the results establish that MORAL reproduces the observed graph composition rather than achieving a normatively independent notion of fairness.","major_comments":[{"comment":"The abstract states that MORAL 'reduces these biases' and Section 5 recommends 'use MORAL when outputs are consumed as rankings and top-K exposure is a fairness concern.' These statements are in tension with the paper's own caveats in Section 3.3 and the final paragraph of Section 5, which state that π is a design choice and that low NDKL/AWRF values 'should be interpreted as evidence of strong performance under distribution-preserving fairness, rather than for support of a more general claim about fairness.' Since Algorithm 1, Eq. (3), and Eq. (7) all reference the same observed π, no experiment distinguishes 'reducing bias' from 'reproducing the observed graph composition.' I recommend qualifying the abstract and the practical recommendations, or adding an experiment with an alternative reference distribution (e.g., uniform exposure) to substantiate the stronger phrasing.","section":"Abstract and Section 5 (practical recommendations)"}],"minor_comments":[{"comment":"The ΔNDKL values for MORAL on Credit and Facebook are inconsistent with the original values in Table 14: the original NDKL for Credit is 0.01 and for Facebook 0.04, while the reproduced values are 0.00 and 0.01, so Δ should be −0.01 and −0.03, respectively, not −0.04 and 0.00. Please correct these entries.","section":"Table 2, MORAL row"},{"comment":"Because the reproduction changes the data split, negative sampling, the direction of the KL divergence, and the normalization (Section 3.5 and Appendix A.2), the Δ columns compare different evaluation pipelines. The text should explicitly state that 'reproduced or improved' is relative to a corrected implementation rather than a like-for-like replication of the original results.","section":"Section 4.1, Tables 2 and 3"},{"comment":"The text refers to 'Appendix A for the full algorithm' and later to 'Appendix A.1'; please unify the cross-reference to a single location.","section":"Section 3.3"},{"comment":"The preprocessing statement 'all graphs are treated as undirected by removing self-loops, merging duplicate edges, and treating (u,v) as equivalent to (v,u)' appears before the definition of the candidate set C; please clarify whether this preprocessing also applies to the DPAH synthetic graphs and to the categorical-attribute Credit graph.","section":"Section 3.5"},{"comment":"The right panel's y-axis is NDKL with values around 0.000–0.010; please state in the caption that the plots are computed at K=1000 and on the same data splits as the main experiments.","section":"Figure 2 caption"}],"recommendation":"minor_revision","confidential_remarks":"The manuscript header states 'Published in Transactions on Machine Learning Research (05/2026)' and includes an OpenReview link; if this submission is intended for a different venue, the authors should remove those identifiers. I have not factored the publication status into my verdict. The paper's internal caveats largely neutralize the circularity concern, but the abstract and practical recommendations need to be aligned with the distribution-preserving framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a careful, honest reproducibility study. The central reproductions (C1–C3) are credible, and the new synthetic homophily stress-test and categorical-attribute scaling are genuinely informative additions. The main caveat—that MORAL's fairness gains are measured against the same observed subgroup-pair distribution π that MORAL is explicitly optimizing—is correctly identified by the authors in the body, but the abstract oversells it as \"reduces these biases.\"\n\nWhat's actually new: a DPAH-based asymmetric homophily sweep showing NDKL and AWRF peak around absolute homophily hedge≈0.6 rather than at relative homophily Δh; a categorical-attribute extension up to 55 subgroup-pair types; and an AWRF robustness check that separates MORAL from baselines more strongly than NDKL does. The pre/post rerank ablation in Appendix G is a good addition: it shows most of the fairness gain comes from the greedy KL aggregation, not from the decoupled predictors. The code is public, seeds are fixed, standard deviations are small, and the documentation of inconsistencies in the original repo is transparent.\n\nSoft spots, in order of size. First, circularity. MORAL's objective is D_KL(q'||π) at each rank; NDKL is exactly the accumulated prefix D_KL(·||π). AWRF uses a different aggregation but still compares to the same π. So the AWRF result shows MORAL isn't gaming the particular KL summation, but it does not show MORAL is reducing bias relative to an independent fairness target. The paper says this in §3.3 and §5, so it is a qualified rather than hidden flaw. Second, the nominal reproduction changes the original code: NDKL was reimplemented, data splits were added, baselines were rerun. That means the ΔNDKL columns in Tables 2–3 are not a strict apples-to-apples reproduction, though the reported anomalies in the original results (e.g., below-chance DetConstSort) make the corrections look justified. Minor. Third, the absolute-vs-relative homophily claim is based on synthetic graphs with i.i.d. features; that is the right first test, but real graphs entangle features and mixing, and the authors acknowledge the peak location may shift.\n\nWho it's for: people doing empirical work on fair link prediction or ranking fairness. It does not break new theory, but it makes the case for rank-aware, distribution-preserving evaluation much more concrete. It deserves a serious referee—I would send it out, with the request that the abstract be toned down to match the body's own caveats.","headline":"Careful reproduction with worthwhile new stress-tests; the 'reduces biases' headline outruns the distribution-preserving evidence.","tokens_in":28823,"tokens_out":3209,"would_cite":true,"duration_ms":31710,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dyadic demographic parity can mask within-group exposure disparities in ranked link prediction, and the MORAL re-ranker removes them with negligible utility loss.","keywords":["fairness","link prediction","demographic parity","exposure","NDKL","MORAL","reproducibility","ranked lists"],"falsifier":"Take a graph whose true edge mix is known to be biased against one subgroup pair, run MORAL on it, and check whether the output still matches that biased mix; if it does, the method is preserving the status quo rather than achieving fairness.","tokens_in":27778,"feed_emoji":"⚖️","tokens_out":10952,"duration_ms":87493,"temperature":0.7,"pith_summary":"Fairness in ranked link prediction is usually audited with dyadic demographic parity, which compares only whether same-group and cross-group links are predicted at similar rates. This paper reproduces and extends a prior claim that such a coarse check can hide real bias: it can show aggregate parity while, say, female–female links are systematically ranked below male–female links. The authors confirm that the rank-aware Normalized Discounted KL-divergence (NDKL) detects such disparities, and that MORAL, a post-processing re-ranker, removes them on six social-graph datasets while leaving top-K precision essentially unchanged. They stress-test this on synthetic homophily graphs, add the exposure metric AWRF and the utility metric NDCG, and extend the framework to categorical sensitive attributes with up to ten groups. If the results hold, the practical consequence is that exposure per subgroup-pair type should replace aggregate parity in audits of ranked predictions.","feed_headline":"NDKL catches the bias demographic parity misses; MORAL fixes it","feed_subtitle":"Rank-aware metrics expose the hidden bias; the MORAL re-ranker removes it with minimal utility loss.","key_machinery":"The machinery is the pair of rank-aware exposure objects. NDKL (Normalized Discounted KL-divergence) is the normalized sum, over ranking prefixes, of the position-discounted KL divergence between the empirical subgroup-pair distribution $\\hat{\\pi}_k$ and the target distribution $\\pi$; it is the detector that $\\Delta_\\mathrm{DP}$ cannot be. MORAL is a greedy aggregation procedure: it trains one GCN-based predictor per subgroup-pair type, and at each rank chooses the top remaining candidate whose insertion would keep the running distribution closest to $\\pi$ in KL divergence. Because both the metric and the optimiser use the same $\\pi$, they are aligned; the paper adds AWRF (a single $\\ell^1$ distance on the position-weighted exposure distribution) as a check that the fairness gains are not merely an artifact of optimizing the same divergence that the metric evaluates.","core_discovery":"The central discovery is that exposure, not predicted-link parity, is the right currency for fairness in ranked link prediction. The authors show that $\\Delta_\\mathrm{DP}$ is permutation-invariant over the ranking and aggregates intra-group links together, so it cannot detect that one subgroup-pair type is systematically ranked lower than another; their reproduced experiments display exactly this pattern, with $\\Delta_\\mathrm{DP}$ flat while NDKL varies substantially. NDKL, defined as the normalized rank-discounted KL divergence between the subgroup-pair distribution of each prefix and the observed distribution $\\pi$ of the original graph, does detect the shift. MORAL then fixes it: using one decoupled predictor per subgroup-pair type, it greedily builds a ranking whose running type distribution stays close to $\\pi$, and the reproduction finds near-zero NDKL and AWRF across all six datasets with Precision@1000 at or near the best baseline. The paper further argues that the framework generalizes beyond binary attributes and that its behavior responds predictably to graph homophily, with exposure disparity peaking at intermediate absolute homophily.","pith_inferences":["Beyond the paper, the same two metric definitions could audit any ranked recommender system, since NDKL and AWRF need only a ranking and subgroup labels; the results therefore generalize to candidate-generation rankings in search and recommendation.","A testable extension is to replace the distribution-preserving target $\\pi$ with a corrective target, such as equalized exposure across subgroup-pair types, and measure how the greedy re-ranker trades utility against that alternative objective.","Because MORAL trains one predictor per subgroup-pair type, its cost grows as $O(m^2)$ with attribute cardinality; parameter sharing across types is a natural next step that the paper's own future-work section gestures at.","The peak of exposure disparity at intermediate homophily suggests that audits looking only at highly homophilic networks could miss the worst case; sampling medium-homophily settings is the more informative check."],"forward_implications":["Auditors of ranked link prediction should report exposure per subgroup-pair type (NDKL or AWRF) alongside or instead of $\\Delta_\\mathrm{DP}$, because aggregated parity can report a clean bill of health while one pair type is buried at the bottom of the ranking.","MORAL is a practical post-processing fix: across six datasets it drives NDKL to 0.00–0.01 and AWRF to 0.001–0.003 while keeping Precision@1000 at or near the best baseline.","The fairness gains come from the re-ranking step itself: the ablation shows a 66.9% NDKL reduction and a 92.6% AWRF reduction relative to the raw decoupled predictions.","At higher attribute cardinality, exposure alignment becomes harder: NDKL and AWRF increase monotonically with the number of subgroup-pair types $T = m(m+1)/2$, even though utility stays high, revealing a fairness-scalability trade-off.","Exposure-based audits are most informative at intermediate absolute homophily (edge homophily around 0.6), because that is where NDKL and AWRF peaks occur; very high homophily constrains the candidate pools and reduces divergence."],"supporting_citations":[{"why":"Supplies the three claims under test, the NDKL and MORAL definitions, the datasets, and the baseline results this study reproduces.","marker":"Mattos et al. 2025"},{"why":"Defines the dyadic demographic parity baseline that the paper shows can hide subgroup-pair exposure disparities.","marker":"Li et al. 2021"},{"why":"Source of the NDKL metric and the DetConstSort re-ranking baseline, giving the normalizer and rank discounting.","marker":"Geyik et al. 2019"},{"why":"Provides the DPAH generator used to create the synthetic homophily graphs for the stress-test extension.","marker":"Espín-Noboa et al. 2022"},{"why":"Original attention-weighted rank fairness definition that is adapted into the AWRF metric.","marker":"Sapiezynski et al. 2019"},{"why":"Provides the exposure-distribution formulation used to adapt AWRF to subgroup-pair types.","marker":"Raj & Ekstrand 2022"},{"why":"Defines NDCG, the rank-aware utility metric used to check that MORAL preserves ranking quality.","marker":"Järvelin & Kekäläinen 2002"},{"why":"Supports the paper's caveat that distribution-preserving fairness is a design choice, not a universal target.","marker":"Mitchell et al. 2021"}],"fun_headline_variants":["Exposure, not parity, is the real fairness metric in link ranking","NDKL detects the exposure bias that demographic parity misses","MORAL re-ranks links to fix exposure bias with little utility loss","Why demographic parity is not enough for fair link prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fairness argument rests on treating the graph's current subgroup-pair link proportions as the target distribution; the paper itself calls this a design choice, so if those proportions encode historical bias, matching them preserves the bias rather than removing it.","fun_headline_variants_meta":{"raw":{"variants":["Exposure, not parity, is the real fairness metric in link ranking","NDKL detects the exposure bias that demographic parity misses","MORAL re-ranks links to fix exposure bias with little utility loss","Why demographic parity is not enough for fair link prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1761,"prompt_tokens":975,"completion_tokens":786,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":713}},"tokens_in":591,"tokens_out":786,"duration_ms":7414,"temperature":1.0,"reasoning_tokens":713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:22:15.320406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a graph whose true edge mix is known to be biased against one subgroup pair, run MORAL on it, and check whether the output still matches that biased mix; if it does, the method is preserving the status quo rather than achieving fairness.","supporting_citations":[],"review_version":1}