{"id":"9a716256-5431-44b5-b953-aa59f3106ede","arxiv_id":"2506.04140","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A quantification-based protocol using query-matched correction sets estimates group prevalence in rankings, enabling multiclass query fairness evaluation under label unawareness.","lead":"The paper proposes using quantification methods, trained on query-matched correction sets, to estimate the prevalence of demographic groups in ranked search results when true group labels are missing. This matters because it gives search platforms a way to audit query-level fairness under 'unawareness', extending to multiclass attributes and multiple queries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.3's 'sampling bias vanishes' step assumes marginal relevance matching between L_q and U_q implies full class-conditional matching; this is not derived and is the load-bearing assumption for the protocol's reliability claim.","rationale":"The reader's weakest-assumption analysis already targeted the L_corr representativeness assumption; I agree and sharpen it. The paper's own derivation attempts to justify the match by equating marginal relevance rates (P_Lq(Q)≈P_Uq(Q)), but the required object is the class-conditional joint distribution P(X,Q|Y). The two conditions are logically independent: a pool can have the same overall fraction of query-relevant items while having different per-class relevance rates and different content within relevance strata, which is precisely the setting that breaks ACC/PACC/KDEy. This is load-bearing because the entire query-time correction procedure is only as good as the L_q to U_q transfer; if the assumption fails, the method reduces to an unadjusted classifier count on a biased sample. The empirical results on a single same-collection split do not resolve the concern, although the public code and the KDEy dominance over baselines are genuine strengths. Given that the paper acknowledges the limitation, the right verdict remains conditional: accept with a required diagnostic or broader evaluation and a softened novelty claim. No change to the reader's verdict is needed.","tokens_in":17194,"tokens_out":5342,"duration_ms":47267,"concrete_test":"On TREC 2022, with held-out true labels, reconstruct L_q and U_q for all 97 queries. For each class y and relevance stratum Q, estimate P_Lq(Q|Y) vs P_Uq(Q|Y) and the distributions of classifier posteriors P_Lq(phi(X)|Y,Q) vs P_Uq(phi(X)|Y,Q) (e.g., via two-sample KS or MMD). Then reweight L_q to match U_q's per-class relevance rates and recompute AE(rKL) for KDEy and PACC. If error changes materially, the Section 3.3 assumption is violated; if not, the assumption holds on this benchmark and the remaining risk is external validity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The protocol's central step is the claim in Section 3.3 that if P_Lq(Q)≈P_Uq(Q) then P_Lq(X,Q|Y)≈P_Uq(X,Q|Y), 'and the sampling bias shift vanishes'. This implication is not proven and is stronger than it appears. By the chain rule, P(X,Q|Y)=P(X|Y,Q)P(Q|Y). Equality of the marginal relevance rate P(Q) says nothing about the class-conditional relevance rates P(Q|Y) or about the feature distributions within relevance strata P(X|Y,Q). In fact, the classifier's posteriors used by PACC and KDEy are functions of X, so the correction matrices are only valid if P(phi(X)|Y,Q) and P(Q|Y) coincide for the two pools. The experimental setup (random 50/50 split of one collection, same BM25) makes the stronger condition plausible by construction, but no evidence is given for realistic correction pools built from voluntary disclosure, temporal drift, or a different retrieval model. The paper's own Limitations state that correction pools may exhibit self-selection effects that are not modeled. The claim to have established a 'reliable protocol' therefore rests on an unverified structural assumption, not just on the empirical comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a quantification-based protocol for query fairness estimation (QFE) under unawareness of sensitive attributes. The authors split a labelled demographic pool into a classifier-training set and a correction pool; for each query, they retrieve the top items from the correction pool with the same retrieval model (BM25) used on the unlabelled production pool, yielding a query-matched labelled set L_q. They then use L_q to estimate the correction matrix (PACC) or class-conditional densities (KDEy) and predict group prevalence in top-k test rankings, from which rKL or rND fairness scores are computed. Experiments on the TREC 2022 Fair Ranking Track across three multi-valued attributes and 97 queries show that KDEy with the query-matched correction achieves the lowest absolute error in binary and multiclass settings, and the paper claims this is the first reliable protocol for QFE across multiple queries and groups.","tokens_in":96,"tokens_out":8674,"duration_ms":186299,"significance":"If the method holds, it addresses a practically important problem: fairness evaluation when sensitive labels are unavailable. The multiclass extension, the use of standard quantification machinery, the public code, and the consistent empirical gains of KDEy over CC and PMC variants are clear strengths. However, the central theoretical justification rests on an unverified assumption about class-conditional distribution matching between L_q and U_q, and the experiments, as designed, do not actually induce sample selection bias. The significance is therefore real but narrower than claimed; the paper is better read as an empirical demonstration on a single favourable benchmark than as establishing a generally reliable protocol.","major_comments":[{"comment":"The step 'since we can now assume P_Lq(Q)≈P_Uq(Q) then we can also assume P_Lq(X,Q|Y)≈P_Uq(X,Q|Y)' is not justified. Equality of the marginal relevance rate P(Q) does not imply equality of the class-conditional joint distribution: by the chain rule, P(X,Q|Y)=P(X|Y,Q)P(Q|Y), and matching P(Q) says nothing about P(Q|Y) or about P(X|Y,Q) within relevance strata. Because PACC's correction matrix and KDEy's densities are functions of X (through classifier posteriors) and Y, the correction is valid only if P(phi(X)|Y,Q) and P(Q|Y) coincide between L_q and U_q. This is a load-bearing assumption for the claim that 'the sampling bias shift vanishes', and the paper provides neither a derivation nor evidence from realistic correction pools. The experimental split makes the assumption plausible by construction, and the Limitations acknowledge self-selection effects that are not modeled, but the main text presents the implication as established.","section":"Section 3.3, paragraph 'Main idea'"},{"comment":"The experimental protocol does not actually induce sample selection bias. The labelled and unlabelled pools are a random 50/50 split of the same collection, so the marginal relevance distributions P_L(Q) and P_U(Q) are nearly identical; the only shift introduced is the per-group caps in lines 2 and 11 of Algorithm 1, which changes class prevalences rather than the selection variable Q. Consequently, the reported results do not demonstrate robustness to SSB; they demonstrate behaviour under a mild prior shift in a setting where the query-matched correction is likely to be similar to the test pool by construction. A direct SSB test would require, for example, a correction pool with a systematically different relevance distribution (different domain, different retrieval model, or a deliberately relevance-biased subset), and the paper should either add such tests or soften the claim that the method is robust to SSB.","section":"Section 4.3 and Algorithm 1"},{"comment":"The evidence for the 'reliable protocol' claim rests on a single collection (TREC 2022 Fair Ranking Track), a single retrieval model (BM25), and one fairness-metric family (rKL/rND). The random split of one collection means the query-matched correction pool is drawn from the same distribution as the test pool, which is a favourable condition not representative of deployment settings such as voluntary disclosure, temporal drift, or a different retrieval backend. The paper's own Limitations mention self-selection effects, but these are not tested. The abstract and contributions claim the 'first reliable protocol for measuring fairness under unawareness across multiple queries and groups'; this overstates what the current evidence supports. I recommend either adding experiments that vary the collection, retrieval model, or correction-pool sampling mechanism, or revising the generality claims to the demonstrated setting.","section":"Section 4.1, Tables 1–2, and Abstract"}],"minor_comments":[{"comment":"The sentence 'The displayed values are averaged scores of the absolute error on the prediction of rKL' should refer to rND, since the table reports rND prediction errors.","section":"Section 5.1, Table 1"},{"comment":"The text states statistical significance 'in the majority of cases', but Table 2 does not show the significance markers used in Table 1; the markers should be added or the significance claims should be revised.","section":"Section 5.2, Table 2"},{"comment":"In the 'A caveat on terminology' paragraph, 'Thorough this paper' should be 'Throughout this paper'.","section":"Section 3.2"},{"comment":"In the definition of y, the term `(1−b)` should presumably be `(1−β)` for consistency with the preceding notation.","section":"Section 4.4, Equation (15)"},{"comment":"RAE is undefined when a true prevalence p_i is zero; since minority groups can be absent from top-k rankings, the paper should state how zero prevalences are handled in the evaluation.","section":"Section 4.2, Equation (13)"},{"comment":"The y-axis label '10 1' appears to be a formatting error for '10^1'.","section":"Figure 4"},{"comment":"The cap of 200 documents per group in L_q is not justified; since it is one of the two shift-inducing mechanisms in the protocol, a sentence explaining this choice would help.","section":"Section 4.3, Algorithm 1, line 11"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: the Section 3.3 implication from marginal relevance matching to full class-conditional matching is not derived, and the experimental setup, which uses a random split of one collection with only per-group caps as the induced shift, cannot validate it. The paper is a solid empirical study with a practical idea, but the 'reliable protocol' claim needs either a formal condition or evidence from realistic SSB scenarios before it is supportable. I see no ethical concerns; the contribution is within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper has a real idea and it mostly works. The idea is simple: instead of training the quantifier's correction on the labeled pool as a whole, retrieve a query-specific subset L_q from an auxiliary labeled pool using the same query and retrieval model, and use that to estimate the correction matrix. That is a genuinely new twist on quantification for ranking fairness, and it is the first to handle multiclass sensitive attributes in this setting, which matters because most previous work (Ghazimatin et al.) is binary-only. The empirical work is honest and reproducible: public code, external TREC 2022 benchmark, multiple attributes, rank cutoffs, correction pool sizes, and statistical tests. KDEy consistently beats the baselines on both binary and multiclass, and the improvements are usually significant. That is a credible result.\n\nThe soft spot is exactly where the reader's report puts it: Section 3.3 claims that if P_Lq(Q) ≈ P_Uq(Q) then P_Lq(X,Q|Y) ≈ P_Uq(X,Q|Y), and 'the sampling bias shift vanishes.' That implication does not follow from the marginal equality alone; nothing in the chain rule forces the class-conditional relevance rates or feature distributions to match. What saves it in the paper is that L and U are a random 50/50 split of one collection, so by construction the pools are exchangeable and the assumption is likely to hold approximately. But that is not evidence for the realistic cases the paper motivates: voluntary disclosure, temporal drift, a different retrieval model, or a correction pool that is not a random sample. The limitations paragraph does mention self-selection, which is good, but it does not rescue the 'reliable protocol' claim for production settings. So the paper overstates itself in the abstract and in Section 3.3. It should either soften the claim or add experiments with a genuinely different correction pool.\n\nI would not call that a load-bearing flaw for the empirical comparison — within the TREC collection, KDEy's advantage over CC and PMC is real and the ablation with pool sizes is informative. But the generality claim needs tempering.\n\nWho is this for? Anyone working on fairness auditing in search, especially under label unavailability. It deserves a serious referee and, with revisions, publication. I'd bring it to a reading group and I'd cite it, mostly for the query-matched correction idea.\n\nRecommend send to peer review; ask for a softened novelty claim and, ideally, a stress test of the assumption on a second collection or with a mismatched correction pool.","headline":"A genuinely new and useful trick — training the quantifier correction on a query-matched labeled set — with solid empirical support on TREC 2022, but the headline claim of a reliable protocol rests on a distributional assumption that is plausible but not demonstrated beyond the same-collection split.","tokens_in":18006,"tokens_out":2087,"would_cite":true,"duration_ms":19274,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that query fairness under unawareness can be estimated reliably by learning each query's correction from a query-matched labelled retrieval, and that this is the first protocol to make quantification robust to sample…","keywords":["quantification","query fairness estimation","fairness under unawareness","sample selection bias","ranking fairness","multiclass sensitive attributes","prior probability shift","top-k exposure"],"falsifier":"Use a correction pool built from a visibly different population, such as documents from an earlier time period or a different corpus, while keeping the same query and retrieval model, then compute the protocol's estimated rKL scores and compare them with ground-truth rKL computed from true labels; if errors grow systematically with the pool-to-production divergence, the central assumption fails.","tokens_in":16919,"feed_emoji":"⚖️","tokens_out":8709,"duration_ms":79384,"temperature":0.7,"pith_summary":"Search systems are increasingly audited for fairness, but audits usually require knowing a sensitive attribute such as race, gender, or age for every ranked item, and that information is often unavailable. This paper argues that group proportions in a query's ranking can be estimated reliably by treating the problem as one of quantification, that is, estimating class prevalences rather than individual labels. The key move is to retrieve, for the same query and with the same retrieval engine, a query-biased labelled subset from a separate correction pool, so that the classifier's error rates are measured on data that resemble the actual test ranking. The paper shows that with this correction, standard quantifiers such as PACC and KDEy estimate group prevalence in top-$k$ rankings accurately, including for multi-valued sensitive attributes, and that KDEy outperforms prior correction methods and naive label counting. If accepted, this gives practitioners a protocol for fairness audits under unawareness across many queries and groups without per-query human annotation.","feed_headline":"Group fairness in rankings is measurable without sensitive labels","feed_subtitle":"A query-matched correction keeps per-query prevalence estimates accurate under sample selection bias.","key_machinery":"The machinery is the query-matched correction bag. A quantifier is a function $\\lambda: \\mathcal{N}_\\mathcal{X} \\to \\Delta^{n-1}$ that maps a bag, here the ranked list treated as unordered, to a vector of class prevalence estimates. The paper's protocol splits the labelled data into $L_\\phi$ for training the classifier and $L_{\\mathrm{corr}}$ for correction; for each query $q$, the same retrieval model issues $q$ against $L_{\\mathrm{corr}}$ to form $L_q$, which is then used to compute the correction matrix $M$ in the linear system $t = Mp$ for PACC, or the density mixture in the KL-minimization variant for KDEy. The point of $L_q$ is to make the labelled query-biased distribution match the unlabelled query-biased distribution, so the standard prior-probability-shift assumption $P_{L_q}(X, Q|Y) \\approx P_{U_q}(X, Q|Y)$ holds and the quantifier's correction is learned on data with the same selection bias as the test data.","core_discovery":"The central claim is that sample selection bias, not just prior probability shift, is the main distortion in query-level fairness estimation, and that it can be neutralized by constructing query-matched labelled bags. The protocol separates classifier training from correction learning: it trains a classifier or representation $\\phi$ on a labelled set $L_\\phi$, then for each query $q$ retrieves a query-biased labelled bag $L_q$ from a disjoint correction pool $L_{\\mathrm{corr}}$ using the same retrieval model and the same query that produced the unlabelled test bag $U_q$. With the assumption $P_{L_q}(X,Q|Y) \\approx P_{U_q}(X,Q|Y)$, the usual quantification assumption of prior-probability shift is restored, and quantifiers such as PACC and KDEy can estimate the prevalence vector $\\hat{p}_k$ for each prefix of depth $k$. The paper reports that KDEy with this protocol yields the lowest absolute error in estimating the normalized discounted KL-divergence fairness score across geographic location, gender, and age-of-topic attributes on a 6.5 million-document benchmark with 97 queries, including in binary comparisons against post-metric correction baselines.","pith_inferences":["A natural extension is that the same query-matched correction recipe should transfer to any quantifier whose representation is a mean embedding and whose correction is learned from labelled data, making the contribution a protocol rather than a single estimator.","The paper leaves implicit that production deployments will need to monitor drift between the correction pool and the production corpus, since self-selection of who discloses sensitive attributes is an acknowledged unmodelled risk.","A normalized-discounted variant of relative absolute error, which the paper mentions as future work, would likely change the relative ranking of PACC versus Classify and Count, since RAE favours methods that protect minority-class estimates.","Because the correction is tied to the exact retrieval model used at query time, updating the retrieval engine would require re-issuing queries against the correction pool, an operational cost the paper does not discuss."],"forward_implications":["Per-query fairness scores become computable for any top-$k$ prefix without per-query human annotation, as long as a labelled correction pool exists.","Multiclass sensitive attributes, not just binary protected groups, can be tracked, which the paper says is a first for query fairness estimation.","The approach gives the first quantification method claimed to be robust to sample selection bias, not only to prior probability shift.","The two quantifier variants, PACC and KDEy, are fast enough at query time, with single-digit millisecond predictions, to be used in a live search pipeline.","Naive label counting, the Classify and Count approach, remains unreliable under this shift, and the query-matched correction is what closes the gap."],"supporting_citations":[{"why":"Defines Classify and Count and Adjusted Classify and Count, the base quantifiers whose correction the paper extends.","marker":"(Forman, 2005)"},{"why":"Supplies PACC, one of the two quantifiers tested with the query-matched correction.","marker":"(Bella et al., 2010)"},{"why":"Supplies KDEy, the kernel-density-based multiclass quantifier that achieves the best results in the paper.","marker":"(Moreo et al., 2024)"},{"why":"Defines the post-metric correction baselines and the rND metric that the paper compares against in the binary setting.","marker":"(Ghazimatin et al., 2022)"},{"why":"Establishes that counting classifier predictions is unreliable for prevalence estimation, motivating the quantification approach.","marker":"(Esuli et al., 2023)"},{"why":"Defines the normalized discounted KL divergence (rKL) fairness metric used to evaluate query fairness.","marker":"(Zehlike et al., 2022)"},{"why":"Provides the dataset shift taxonomy and the sample selection bias concept that the paper targets.","marker":"(Storkey, 2009)"},{"why":"Provides the public benchmark collection with multi-valued sensitive attributes and 97 queries used in the experiments.","marker":"(Ekstrand et al., 2022c)"}],"fun_headline_variants":["Fairness metrics now work when sensitive labels are missing","Quantifying unfair rankings without group labels","Query-matched correction enables fair ranking measurement","Measure ranking fairness even without protected attributes","Multi-attribute fairness estimation without labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The protocol assumes that the labelled correction pool is drawn from the same population as production, so that for any query the group-by-group mix of retrieved documents matches; if the pool's population differs, the correction is tuned to the wrong bias.","fun_headline_variants_meta":{"raw":{"variants":["Fairness metrics now work when sensitive labels are missing","Quantifying unfair rankings without group labels","Query-matched correction enables fair ranking measurement","Measure ranking fairness even without protected attributes","Multi-attribute fairness estimation without labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1534,"prompt_tokens":973,"completion_tokens":561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":497}},"tokens_in":589,"tokens_out":561,"duration_ms":6527,"temperature":1.0,"reasoning_tokens":497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:47:07.026464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a correction pool built from a visibly different population, such as documents from an earlier time period or a different corpus, while keeping the same query and retrieval model, then compute the protocol's estimated rKL scores and compare them with ground-truth rKL computed from true labels; if errors grow systematically with the pool-to-production divergence, the central assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Classify and Count and Adjusted Classify and Count, the base quantifiers whose correction the paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies PACC, one of the two quantifiers tested with the query-matched correction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies KDEy, the kernel-density-based multiclass quantifier that achieves the best results in the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the post-metric correction baselines and the rND metric that the paper compares against in the binary setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that counting classifier predictions is unreliable for prevalence estimation, motivating the quantification approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the normalized discounted KL divergence (rKL) fairness metric used to evaluate query fairness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the dataset shift taxonomy and the sample selection bias concept that the paper targets."}],"review_version":1}