Pith. sign in

REVIEW 3 major objections 7 minor 59 references

Quantifying Query Fairness Under Unawareness

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that query fairness under unawareness can be estimated reliably by learning each query's correction from a query-matched labelled retrieval, and that this is the first protocol to make quantification robust to sample…

desk verdict A genuinely new and useful trick — training the quantifier correction on a query-matched labeled set — with solid empirical support on TREC 2022, but the headline claim of a reliable protocol rests on a distributional assumption that is plausible but not demonstrated beyond the same-collection split. read the letter →

arxiv 2506.04140 v1 pith:MCK5FLS2 submitted 2025-06-04 cs.IR

Thomas Jaenich , Alejandro Moreo , Alessandro Fabris , Graham McDonald , Andrea Esuli , Iadh Ounis , Fabrizio Sebastiani This is my paper · ORCID
classification cs.IR
keywords quantificationqueryfairnessestimationunderunawarenesssampleselectionbiasrankingmulticlasssensitiveattributespriorprobabilityshifttop-kexposure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Search systems are increasingly audited for fairness, but audits usually require knowing a sensitive attribute such as race, gender, or age for every ranked item, and that information is often unavailable. This paper argues that group proportions in a query's ranking can be estimated reliably by treating the problem as one of quantification, that is, estimating class prevalences rather than individual labels. The key move is to retrieve, for the same query and with the same retrieval engine, a query-biased labelled subset from a separate correction pool, so that the classifier's error rates are measured on data that resemble the actual test ranking. The paper shows that with this correction, standard quantifiers such as PACC and KDEy estimate group prevalence in top-$k$ rankings accurately, including for multi-valued sensitive attributes, and that KDEy outperforms prior correction methods and naive label counting. If accepted, this gives practitioners a protocol for fairness audits under unawareness across many queries and groups without per-query human annotation.

What carries the argument

The machinery is the query-matched correction bag. A quantifier is a function $\lambda: \mathcal{N}_\mathcal{X} \to \Delta^{n-1}$ that maps a bag, here the ranked list treated as unordered, to a vector of class prevalence estimates. The paper's protocol splits the labelled data into $L_\phi$ for training the classifier and $L_{\mathrm{corr}}$ for correction; for each query $q$, the same retrieval model issues $q$ against $L_{\mathrm{corr}}$ to form $L_q$, which is then used to compute the correction matrix $M$ in the linear system $t = Mp$ for PACC, or the density mixture in the KL-minimization variant for KDEy. The point of $L_q$ is to make the labelled query-biased distribution match the unlabelled query-biased distribution, so the standard prior-probability-shift assumption $P_{L_q}(X, Q|Y) \approx P_{U_q}(X, Q|Y)$ holds and the quantifier's correction is learned on data with the same selection bias as the test data.

What would settle it

Use a correction pool built from a visibly different population, such as documents from an earlier time period or a different corpus, while keeping the same query and retrieval model, then compute the protocol's estimated rKL scores and compare them with ground-truth rKL computed from true labels; if errors grow systematically with the pool-to-production divergence, the central assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that sample selection bias, not just prior probability shift, is the main distortion in query-level fairness estimation, and that it can be neutralized by constructing query-matched labelled bags. The protocol separates classifier training from correction learning: it trains a classifier or representation $\phi$ on a labelled set $L_\phi$, then for each query $q$ retrieves a query-biased labelled bag $L_q$ from a disjoint correction pool $L_{\mathrm{corr}}$ using the same retrieval model and the same query that produced the unlabelled test bag $U_q$. With the assumption $P_{L_q}(X,Q|Y) \approx P_{U_q}(X,Q|Y)$, the usual quantification assumption of prior-probability shift is restored, and quantifiers such as PACC and KDEy can estimate the prevalence vector $\hat{p}_k$ for each prefix of depth $k$. The paper reports that KDEy with this protocol yields the lowest absolute error in estimating the normalized discounted KL-divergence fairness score across geographic location, gender, and age-of-topic attributes on a 6.5 million-document benchmark with 97 queries, including in binary comparisons against post-metric correction baselines.

Load-bearing premise

The protocol assumes that the labelled correction pool is drawn from the same population as production, so that for any query the group-by-group mix of retrieved documents matches; if the pool's population differs, the correction is tuned to the wrong bias.

Editorial extensions

If this is right

  • Per-query fairness scores become computable for any top-$k$ prefix without per-query human annotation, as long as a labelled correction pool exists.
  • Multiclass sensitive attributes, not just binary protected groups, can be tracked, which the paper says is a first for query fairness estimation.
  • The approach gives the first quantification method claimed to be robust to sample selection bias, not only to prior probability shift.
  • The two quantifier variants, PACC and KDEy, are fast enough at query time, with single-digit millisecond predictions, to be used in a live search pipeline.
  • Naive label counting, the Classify and Count approach, remains unreliable under this shift, and the query-matched correction is what closes the gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is that the same query-matched correction recipe should transfer to any quantifier whose representation is a mean embedding and whose correction is learned from labelled data, making the contribution a protocol rather than a single estimator.
  • The paper leaves implicit that production deployments will need to monitor drift between the correction pool and the production corpus, since self-selection of who discloses sensitive attributes is an acknowledged unmodelled risk.
  • A normalized-discounted variant of relative absolute error, which the paper mentions as future work, would likely change the relative ranking of PACC versus Classify and Count, since RAE favours methods that protect minority-class estimates.
  • Because the correction is tied to the exact retrieval model used at query time, updating the retrieval engine would require re-issuing queries against the correction pool, an operational cost the paper does not discuss.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a quantification-based protocol for query fairness estimation (QFE) under unawareness of sensitive attributes. The authors split a labelled demographic pool into a classifier-training set and a correction pool; for each query, they retrieve the top items from the correction pool with the same retrieval model (BM25) used on the unlabelled production pool, yielding a query-matched labelled set L_q. They then use L_q to estimate the correction matrix (PACC) or class-conditional densities (KDEy) and predict group prevalence in top-k test rankings, from which rKL or rND fairness scores are computed. Experiments on the TREC 2022 Fair Ranking Track across three multi-valued attributes and 97 queries show that KDEy with the query-matched correction achieves the lowest absolute error in binary and multiclass settings, and the paper claims this is the first reliable protocol for QFE across multiple queries and groups.

Significance. If the method holds, it addresses a practically important problem: fairness evaluation when sensitive labels are unavailable. The multiclass extension, the use of standard quantification machinery, the public code, and the consistent empirical gains of KDEy over CC and PMC variants are clear strengths. However, the central theoretical justification rests on an unverified assumption about class-conditional distribution matching between L_q and U_q, and the experiments, as designed, do not actually induce sample selection bias. The significance is therefore real but narrower than claimed; the paper is better read as an empirical demonstration on a single favourable benchmark than as establishing a generally reliable protocol.

major comments (3)
  1. [Section 3.3, paragraph 'Main idea'] The step 'since we can now assume P_Lq(Q)≈P_Uq(Q) then we can also assume P_Lq(X,Q|Y)≈P_Uq(X,Q|Y)' is not justified. Equality of the marginal relevance rate P(Q) does not imply equality of the class-conditional joint distribution: by the chain rule, P(X,Q|Y)=P(X|Y,Q)P(Q|Y), and matching P(Q) says nothing about P(Q|Y) or about P(X|Y,Q) within relevance strata. Because PACC's correction matrix and KDEy's densities are functions of X (through classifier posteriors) and Y, the correction is valid only if P(phi(X)|Y,Q) and P(Q|Y) coincide between L_q and U_q. This is a load-bearing assumption for the claim that 'the sampling bias shift vanishes', and the paper provides neither a derivation nor evidence from realistic correction pools. The experimental split makes the assumption plausible by construction, and the Limitations acknowledge self-selection effects that are not modeled, but the main text presents the implication as established.
  2. [Section 4.3 and Algorithm 1] The experimental protocol does not actually induce sample selection bias. The labelled and unlabelled pools are a random 50/50 split of the same collection, so the marginal relevance distributions P_L(Q) and P_U(Q) are nearly identical; the only shift introduced is the per-group caps in lines 2 and 11 of Algorithm 1, which changes class prevalences rather than the selection variable Q. Consequently, the reported results do not demonstrate robustness to SSB; they demonstrate behaviour under a mild prior shift in a setting where the query-matched correction is likely to be similar to the test pool by construction. A direct SSB test would require, for example, a correction pool with a systematically different relevance distribution (different domain, different retrieval model, or a deliberately relevance-biased subset), and the paper should either add such tests or soften the claim that the method is robust to SSB.
  3. [Section 4.1, Tables 1–2, and Abstract] The evidence for the 'reliable protocol' claim rests on a single collection (TREC 2022 Fair Ranking Track), a single retrieval model (BM25), and one fairness-metric family (rKL/rND). The random split of one collection means the query-matched correction pool is drawn from the same distribution as the test pool, which is a favourable condition not representative of deployment settings such as voluntary disclosure, temporal drift, or a different retrieval backend. The paper's own Limitations mention self-selection effects, but these are not tested. The abstract and contributions claim the 'first reliable protocol for measuring fairness under unawareness across multiple queries and groups'; this overstates what the current evidence supports. I recommend either adding experiments that vary the collection, retrieval model, or correction-pool sampling mechanism, or revising the generality claims to the demonstrated setting.
minor comments (7)
  1. [Section 5.1, Table 1] The sentence 'The displayed values are averaged scores of the absolute error on the prediction of rKL' should refer to rND, since the table reports rND prediction errors.
  2. [Section 5.2, Table 2] The text states statistical significance 'in the majority of cases', but Table 2 does not show the significance markers used in Table 1; the markers should be added or the significance claims should be revised.
  3. [Section 3.2] In the 'A caveat on terminology' paragraph, 'Thorough this paper' should be 'Throughout this paper'.
  4. [Section 4.4, Equation (15)] In the definition of y, the term `(1−b)` should presumably be `(1−β)` for consistency with the preceding notation.
  5. [Section 4.2, Equation (13)] RAE is undefined when a true prevalence p_i is zero; since minority groups can be absent from top-k rankings, the paper should state how zero prevalences are handled in the evaluation.
  6. [Figure 4] The y-axis label '10 1' appears to be a formatting error for '10^1'.
  7. [Section 4.3, Algorithm 1, line 11] The cap of 200 documents per group in L_q is not justified; since it is one of the two shift-inducing mechanisms in the protocol, a sentence explaining this choice would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fairness estimates are computed via standard quantification and validated on an external TREC benchmark.

full rationale

The paper's derivation is self-contained and empirically grounded. The central estimation equations (ACC, PACC, KDEy) are standard quantification machinery, and the proposed protocol uses them to estimate prevalence from retrieved test bags; no estimated quantity is reused as an input to the same calculation. The per-query correction is learned from L_q, a query-matched labeled subset of a separate correction pool, and then applied to U_q; the target rKL/rND values are computed from the ground-truth labels of U_q and compared with estimates, so the evaluation is not fitted. The only hyperparameter tuned (KDEy bandwidth) is selected on TREC 2021, an external collection, and all comparisons use TREC 2022. The Section 3.3 'sampling bias vanishes' step relies on the assumption P_Lq(Q)≈P_Uq(Q) implying P_Lq(X,Q|Y)≈P_Uq(X,Q|Y); this is an unproven distributional assumption, not a circular reduction, and the paper explicitly acknowledges in Limitations that correction pools may exhibit self-selection effects. Self-citations to prior work by the authors are background or method references (e.g., KDEy, QuaPy, Learning to Quantify) and are not used to justify the paper's central claim. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on one new ingredient: the query-matched correction set L_q. All other components (PACC, KDEy, rKL, rND, LR, BM25) are standard and cited. There are no invented theoretical entities. Three free parameters are tuned or chosen, but none is fitted to the reported fairness errors.

free parameters (3)
  • Logistic Regression hyperparameters (C, class weight) = C in {10^-4..10^4}, class weight in {Balanced, None}, selected by 5-fold CV on L_phi
    Used by all methods; tuned offline on the labelled classifier-training split, not per query. It is a standard model-selection parameter, not an ad hoc constant.
  • KDEy kernel bandwidth = selected from {0.01,...,0.10} using 100 TREC 2021 queries
    The only KDEy-specific hyperparameter. Tuned on a related external collection, not on the TREC 2022 test queries, so it is not fitted to the reported results, but it is still a chosen value that affects KDEy's accuracy.
  • Maximum documents per group in L_q (200) = 200
    A hand-chosen cap in Algorithm 1, Line 11, used to simulate prior-probability shift between L_q and U_q. It affects the experimental setup, though not the deployed model itself.
assumptions (4)
  • domain assumption Query-matched correction set has the same class-conditional distribution as the test set, P_Lq(X|Y) = P_Uq(X|Y), once P_Lq(Q) ≈ P_Uq(Q) is assumed.
    Section 3.3 'Main idea' states this directly; it is the premise on which the whole correction rests. It is not empirically verified in the paper beyond the TREC split.
  • standard math The classifier is a measurable function, so equal class-conditional distributions of X imply equal confusion rates P(Ŷ|Y).
    Invoked in Section 3.3 to justify using correction-set confusion rates for test data; follows from standard measure theory (Lipton et al., 2018).
  • domain assumption The bag representation Φ(X) = mean of per-item embeddings is sufficient for prevalence estimation.
    Equation (7) assumes the mean embedding captures the class-mixing information needed to solve t = Mp.
  • standard math The multiclass system t = Mp is solvable (M invertible or the optimization problem has a unique minimizer).
    PACC and KDEy rely on solving Equations (8) and (9); the paper does not discuss identifiability or degenerate cases for small correction pools.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantifying Query Fairness Under Unawareness." pith.science (2026). https://pith.science/paper/MCK5FLS2

@misc{pith2026250604140,
  author       = {Pith},
  title        = {Pith review of: Quantifying Query Fairness Under Unawareness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCK5FLS2}},
  note         = {Machine review of arXiv:2506.04140}
}
read the original abstract

Traditional ranking algorithms are designed to retrieve the most relevant items for a user's query, but they often inherit biases from data that can unfairly disadvantage vulnerable groups. Fairness in information access systems (IAS) is typically assessed by comparing the distribution of groups in a ranking to a target distribution, such as the overall group distribution in the dataset. These fairness metrics depend on knowing the true group labels for each item. However, when groups are defined by demographic or sensitive attributes, these labels are often unknown, leading to a setting known as "fairness under unawareness". To address this, group membership can be inferred using machine-learned classifiers, and group prevalence is estimated by counting the predicted labels. Unfortunately, such an estimation is known to be unreliable under dataset shift, compromising the accuracy of fairness evaluations. In this paper, we introduce a robust fairness estimator based on quantification that effectively handles multiple sensitive attributes beyond binary classifications. Our method outperforms existing baselines across various sensitive attributes and, to the best of our knowledge, is the first to establish a reliable protocol for measuring fairness under unawareness across multiple queries and groups.

Figures

Figures reproduced from arXiv: 2506.04140 by the authors.

Figure 1
Figure 1. Schematic of our proposed approach for Query Fairness Evaluation. Demographic [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Distribution of predicted relevance score per document rank across all queries. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Variations in quantification performance (measured in terms of RAE – lower is [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Variations in quantification performance (measured in terms of RAE – lower is [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 58 canonical work pages

  1. [1]

    A., Davis, R

    Adjaye - Gbewonyo, D., Bednarczyk, R. A., Davis, R. L., \ Omer, S. B. 2014 . Using the bayesian improved surname geocoding method (bisg) to create a working classification of race and ethnicity in a diverse managed care population: A validation study \ Health Services Research , 49\/ (1), 268--283

    work page 2014

  2. [2]

    Bella, A., Ferri, C., Hernández-Orallo, J., \ Ramírez-Quintana, M. J. 2010 . Quantification via probability estimators \ In Proceedings of the 11th IEEE International Conference on Data Mining (ICDM 2010) , \ 737--742, Sydney, AU

    work page 2010

  3. [3]

    J., Gummadi, K

    Biega, A. J., Gummadi, K. P., \ Weikum, G. 2018 . Equity of attention: Amortizing individual fairness in rankings \ In The 41st international acm sigir conference on research & development in information retrieval , \ 405--414

    work page 2018

  4. [4]

    Bogen, M., Rieke, A., \ Ahmed, S. 2020a . Awareness in practice: T ensions in access to sensitive attribute data for antidiscrimination \ In Proc, of FAccT

    work page

  5. [5]

    Bogen, M., Rieke, A., \ Ahmed, S. 2020b . Awareness in practice: Tensions in access to sensitive attribute data for antidiscrimination \ In Proc. of the 3rd ACM Conference on Fairness, Accountability, and Transparency (FAT* 2020) , \ 492--500, Barcelona, ES

    work page 2020

  6. [6]

    Bunse, M. 2022a . On multi-class extensions of adjusted classify and count \ In Proceedings of the 2nd International Workshop on Learning to Quantify (LQ 2022) , \ 43--50, Grenoble, IT

    work page 2022

  7. [7]

    Bunse, M. 2022b . Unification of algorithms for quantification and unfolding \ In Proceedings of the Workshop on Machine Learning for Astroparticle Physics and Astronomy , \ 459--468

    work page

  8. [8]

    E., Huang, L., Keswani, V., \ Vishnoi, N

    Celis, L. E., Huang, L., Keswani, V., \ Vishnoi, N. K. 2021 . Fair classification with noisy protected attributes: A framework with provable guarantees \ In International Conference on Machine Learning , \ 1349--1361. PMLR

    work page 2021

Show all 59 references
  1. [9]

    \ \ Fang, H

    Chen, F. \ \ Fang, H. 2023 . Learn to be fair without labels: A distribution-based learning framework for fair ranking \ In Proc. of SIGIR , \ 23--32

    2023

  2. [10]

    Chen, J., Kallus, N., Mao, X., Svacha, G., \ Udell, M. 2019 . Fairness under unawareness: Assessing disparity when protected class is unobserved \ In Proceedings of the Conference on Fairness, Accountability, and Transparency , \ 339--348

    2019

  3. [11]

    Chen, L., Ma, R., Hann \'a k, A., \ Wilson, C. 2018 . Investigating the impact of gender on rank in resume search engines \ In Proceedings of the 2018 chi conference on human factors in computing systems , \ 1--14

    2018

  4. [12]

    Craswell, N., Zoeter, O., Taylor, M., \ Ramsey, B. 2008 . An experimental comparison of click position-bias models \ In Proc. of WSDM

    2008

  5. [13]

    D., Biega, A

    Diaz, F., Mitra, B., Ekstrand, M. D., Biega, A. J., \ Carterette, B. 2020 . Evaluating stochastic rankings with expected exposure \ In Proc. of CIKM

    2020

  6. [14]

    D., Das, A., Burke, R., Diaz, F., et al

    Ekstrand, M. D., Das, A., Burke, R., Diaz, F., et al. 2022a . Fairness in information access systems \ Foundations and Trends in Information Retrieval , 16\/ (1-2), 1--177

  7. [15]

    D., McDonald, G., Raj, A., \ Johnson, I

    Ekstrand, M. D., McDonald, G., Raj, A., \ Johnson, I. 2022b . Overview of the trec 2021 fair ranking track \ In Proc. of TREC

    2021

  8. [16]

    D., McDonald, G., Raj, A., \ Johnson, I

    Ekstrand, M. D., McDonald, G., Raj, A., \ Johnson, I. 2022c . Overview of the TREC 2022 Fair Ranking Track \ In Proc. of TREC

    2022

  9. [17]

    Esuli, A., Fabris, A., Moreo, A., \ Sebastiani, F. 2023 . Learning to quantify . Springer Nature, Cham, CH

    2023

  10. [18]

    Regulation on harmonized rules on artificial intelligence (artificial intelligence act) \ Official Journal of the European Union

    European Commission 2024 . Regulation on harmonized rules on artificial intelligence (artificial intelligence act) \ Official Journal of the European Union. Accessed: 2024-08-06

    2024

  11. [19]

    Fabris, A., Esuli, A., Moreo, A., \ Sebastiani, F. 2023 . Measuring fairness under unawareness of sensitive attributes: A quantification-based approach \ Journal of Artificial Intelligence Research , 76 , 1117--1180

    2023

  12. [20]

    Forman, G. 2005 . Counting positives accurately despite inaccurate classification \ In Proceedings of the 16th European Conference on Machine Learning (ECML 2005) , \ 564--575, Porto, PT

    2005

  13. [21]

    A., Scheidegger, C., \ Venkatasubramanian, S

    Friedler, S. A., Scheidegger, C., \ Venkatasubramanian, S. 2021 . The (im)possibility of fairness: D ifferent value systems require different mechanisms for fair decision making \ Communications of the ACM , 64\/ (4), 136--143

    2021

  14. [22]

    C., Ambler, S., \ Kenthapadi, K

    Geyik, S. C., Ambler, S., \ Kenthapadi, K. 2019 . Fairness-aware ranking in search and recommendation systems with application to Linkedin talent search \ In Proc. of KDD

    2019

  15. [23]

    Ghazimatin, A., Kleindessner, M., Russell, C., Abedjan, Z., \ Golebiowski, J. 2022 . Measuring fairness of rankings under noisy sensitive information \ In Proc. of FAccT

    2022

  16. [24]

    Ghosh, A., Dutt, R., \ Wilson, C. 2021 . When fair ranking meets uncertain inference \ In Proc. of SIGIR

    2021

  17. [25]

    Ghosh, A., Kvitca, P., \ Wilson, C. 2023 . When fair classification meets noisy protected attributes \ In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society , \ 679--690

    2023

  18. [26]

    Gonz \'a lez, P., Moreo, A., \ Sebastiani, F. 2024 . Binary quantification and dataset shift: an experimental investigation

    2024

  19. [27]

    Hardt, M., Price, E., \ Srebro, N. 2016 . Equality of opportunity in supervised learning \ Proc. of NeurIPS , 29

    2016

  20. [28]

    Heuss, M., Sarvi, F., \ de Rijke, M. 2022 . Fairness of exposure in light of incomplete exposure estimation \ In Proc. of SIGIR

    2022

  21. [29]

    Holstein, K., Wortman Vaughan, J., Daum \'e III, H., Dudik, M., \ Wallach, H. 2019 . Improving fairness in machine learning systems: What do industry practitioners need? \ In Proc. of CHI

    2019

  22. [30]

    Jaenich, T., McDonald, G., \ Ounis, I. 2023 . ColBERT-FairPRF: Towards fair pseudo-relevance feedback in dense retrieval \ In European Conference on Information Retrieval , \ 457--465. Springer

    2023

  23. [31]

    Jaenich, T., McDonald, G., \ Ounis, I. 2024 . Fairness-aware exposure allocation via adaptive reranking \ In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , \ 1504--1513

    2024

  24. [32]

    K rnap, \"O ., Diaz, F., Biega, A., Ekstrand, M., Carterette, B., \ Yilmaz, E. 2021 . Estimation of fair ranking metrics with incomplete judgments \ In Proc. of The Web Conference

    2021

  25. [33]

    male nurse

    Kopeinik, S., Mara, M., Ratz, L., Krieg, K., Schedl, M., \ Rekabsaz, N. 2023 . Show me a "male nurse"! how gender bias is reflected in the query formulation of search engine users \ In Proc. of (CHI)

    2023

  26. [34]

    Kuhlman, C., Gerych, W., \ Rundensteiner, E. 2021 . Measuring group advantage: A comparative study of fair ranking metrics \ In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , \ 674--682

    2021

  27. [35]

    Linkedin settings \ New York City Council Legislation

    LinkedIn 2024 . Linkedin settings \ New York City Council Legislation. Accessed: 2024-08-06

    2024

  28. [36]

    C., Wang, Y., \ Smola, A

    Lipton, Z. C., Wang, Y., \ Smola, A. J. 2018 . Detecting and correcting for label shift with black box predictors \ In Proceedings of the 35th International Conference on Machine Learning (ICML 2018) , \ 3128--3136, Stockholm, SE

    2018

  29. [37]

    Macdonald, C., Tonellotto, N., MacAvaney, S., \ Ounis, I. 2021 . PyTerrier: Declarative experimentation in Python from BM25 to dense retrieval \ In Proc. of CIKM

    2021

  30. [38]

    Moreo, A., Esuli, A., \ Sebastiani, F. 2021 . QuaPy: A Python -based framework for quantification \ In Proceedings of the 30th ACM International Conference on Knowledge Management (CIKM 2021) , \ 4534--4543, Gold Coast, AU

    2021

  31. [39]

    Moreo, A., González, P., \ del Coz, J. J. 2024 . Kernel density estimation for multiclass quantification

    2024

  32. [40]

    Morik, M., Singh, A., Hong, J., \ Joachims, T. 2020 . Controlling fairness and bias in dynamic learning-to-rank \ In Proc. of SIGIR

    2020

  33. [41]

    Mozannar, H., Ohannessian, M., \ Srebro, N. 2020 . Fair learning with private demographic data \ In International Conference on Machine Learning , \ 7066--7075. PMLR

    2020

  34. [42]

    Local law 144 of 2021 \ New York City Council Legislation

    New York City Council 2021 . Local law 144 of 2021 \ New York City Council Legislation. Accessed: 2024-08-06

    2021

  35. [43]

    Pedreschi, D., Ruggieri, S., \ Turini, F. 2008 . Discrimination-aware data mining \ In Proc. of KDD

    2008

  36. [44]

    \ \ Ekstrand, M

    Raj, A. \ \ Ekstrand, M. D. 2022 . Measuring fairness in ranked results: An analytical and empirical comparison \ In Proc. of SIGIR

    2022

  37. [45]

    E., Walker, S., Jones, S., Hancock-Beaulieu, M

    Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M. M., Gatford, M., et al. 1995 . Okapi at TREC-3 \ NIST Special Publication Sp , 109 , 109

    1995

  38. [46]

    Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., \ Wilson, C. 2019 . Quantifying the impact of user attentionon fair group representation in ranked lists \ In Proc. of Companion Proceedings of WWW

    2019

  39. [47]

    Sch \" o lkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., \ Mooij, J. M. 2012 . On causal and anticausal learning \ In Proceedings of the 29th International Conference on Machine Learning (ICML 2012) , Edinburgh, UK

    2012

  40. [48]

    Schumacher, T., Strohmaier, M., \ Lemmerich, F. 2023 . A comparative evaluation of quantification methods

    2023

  41. [49]

    Sebastiani, F. 2020 . Evaluation measures for quantification: A n axiomatic approach \ Information Retrieval Journal , 23\/ (3), 255--288

    2020

  42. [50]

    Simson, J., Fabris, A., \ Kern, C. 2024 . Lazy data practices harm fairness research \ In The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2024, Rio de Janeiro, Brazil, June 3-6, 2024 , \ 642--659. ACM

    2024

  43. [51]

    \ \ Joachims, T

    Singh, A. \ \ Joachims, T. 2018 . Fairness of exposure in rankings \ In Proc. of KDD

    2018

  44. [52]

    Storkey, A. 2009 . When training and test sets are different: C haracterizing learning transfer \ In Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., \ Lawrence, N. D. , Dataset shift in machine learning , \ 3--28. The MIT Press, Cambridge, US

    2009

  45. [53]

    Wang, S., Guo, W., Narasimhan, H., Cotter, A., Gupta, M., \ Jordan, M. 2020 . Robust optimization for fairness with noisy protected groups \ Advances in Neural Information Processing systems , 33 , 5190--5203

    2020

  46. [54]

    J., Szary, J., Trindel, K., \ Polli, F

    Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L. J., Szary, J., Trindel, K., \ Polli, F. 2021 . Building and auditing fair algorithms: A case study in candidate screening \ In Elish, M. C., Isaac, W., \ Zemel, R. S. , FAccT '21: 2021 ACM Conference on Fairness, Account...

    2021

  47. [55]

    Yang, E., J\" a nich, T., Mayfield, J., \ Lawrie, D. 2024 . Language fairness in multilingual information retrieval \ In Proc. of Companion Proceedings of SIGIR

    2024

  48. [56]

    \ \ Stoyanovich, J

    Yang, K. \ \ Stoyanovich, J. 2017 . Measuring fairness in ranked outputs \ In Proc. of SSDBM

    2017

  49. [57]

    Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., \ Baeza-Yates, R. 2017 . Fa*ir: A fair top-k ranking algorithm \ In Proc. of CIKM

    2017

  50. [58]

    Zehlike, M., Yang, K., \ Stoyanovich, J. 2022 . Fairness in ranking, part I : Score -based ranking \ ACM Computing Surveys , 55\/ (6), 1--36

    2022

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.