Pith. sign in

REVIEW 2 major objections 4 minor 62 references

Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that the reported performance decline of two-tower learning-to-rank models under strong production logging policies is not a confounding effect of the logging policy: if the model exactly matches user click behavior and…

desk verdict A solid, honest disentangling of identifiability from logging-policy confounding in two-tower models, with a genuine feature-overlap identifiability result and a softer misspecification-amplification lemma. read the letter →

arxiv 2506.20501 v1 pith:FT42FTZ5 submitted 2025-06-25 cs.IR cs.LG

Philipp Hager , Onno Zoeter , Maarten de Rijke This is my paper · ORCID
classification cs.IRcs.LG
keywords unbiasedlearningtoranktwo-towermodelsclickidentifiabilityloggingpolicypositionbiasmodelmisspecificationinversepropensityweighting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Two-tower models for unbiased learning to rank train a relevance tower together with a bias tower, then serve only the relevance tower. Industry reports say that training them on clicks from better-ranking production systems makes them perform worse. This paper argues that the decline is not a confounding effect of the logging policy: if the model exactly matches user click behavior and its parameters are identifiable, the policy has no effect. The decline comes from two separate, tractable causes — the model cannot be uniquely identified from the data (no document swaps or feature overlap across ranks), and a misspecified model's errors, when correlated with document placement, get amplified by the policy. It also proposes inverse-display-propensity weighting to reduce the policy's amplification of misspecification bias.

What carries the argument

The engine of the analysis is the additive click-probability model $P(C=1\mid q,d,k)=\sigma(\theta_k + r(x_{q,d}))$, where $\theta_k$ is the position-bias logit and $r(\cdot)$ the relevance tower, together with the model's invariance to additive parameter shifts: because equal click probabilities force $\theta_k + r(x)=\theta'_k + r'(x)$, any alternative parameterization differs by a rank-dependent offset $\Delta_k$. Identifiability therefore reduces to showing that all offsets collapse to one global constant. Document swaps connect the position graph; for feature-based models, connected overlap of feature supports plus continuity (formally, a shared Lipschitz constant $L$ in the proof) collapse the offsets via the bound $|\Delta_k-\Delta_{k'}|\le 2L\|x_1-x_2\|_2$. The policy analysis uses the gradient of the negative log-likelihood: for a well-specified model the policy-weighted residuals vanish at the true parameters, whereas for a misspecified model the policy-weighted averages of the residuals must vanish, which shifts the parameters when residuals correlate with placement.

What would settle it

Run the Section 7 simulation with a strong deterministic policy ($\alpha=1$, temperature $\tau=0$) but a well-specified relevance model and exact document swaps between adjacent ranks; if the estimated position-bias parameters deviate from the simulated ground truth as policy strength changes, Lemma 5.1 is false. For Theorem 4.1, measure the empirical Lipschitz constant of the trained relevance network and check whether, for two positions with overlapping feature support, the estimated offset difference ever exceeds $2L\|x_1-x_2\|_2$; a continuous network with finite but large $L$ that still fails to recover parameters would show that the continuity assumption, not the overlap condition, carries the theorem.

Watch

Extended reading notes

Core claim

The central claim is that the observed performance decline of two-tower models under strong logging policies is caused by lack of identifiability and by misspecification bias amplified by the policy, not by confounding. Theorem 4.1 states that a feature-based additive two-tower model is approximately identifiable up to an additive constant when the feature supports of different ranks form a connected graph and the relevance tower is continuous. Lemma 5.1 states that a well-specified, identifiable model's parameter estimates are unaffected by the logging policy, and Lemma 5.2 states that a misspecified model's bias can be amplified when its click-prediction errors correlate with document placement across positions. The simulations recover true position-bias parameters under a strong deterministic policy when the model is well-specified and identifiable, and exhibit the predicted amplification in three misspecification scenarios.

Load-bearing premise

The proof that connected feature overlap identifies the model assumes the relevance tower and any alternative parameterization are Lipschitz continuous with a shared constant $L$, while the theorem statement only says continuous; if the model class permits arbitrarily steep continuous functions, the bound imposes no constraint and feature overlap no longer guarantees approximate identifiability.

Editorial extensions

If this is right

  • A well-specified, identifiable two-tower model trained on clicks from a strong deterministic policy recovers the same position-bias parameters as one trained on random data; the reported decline is not intrinsic to two-tower models.
  • Identifiability requires either randomized document swaps that connect all positions or continuous relevance models with overlapping feature distributions across ranks; deterministic rankings and high-dimensional features undermine both.
  • The practical danger is model misspecification: when click-prediction errors correlate with document placement, stronger logging policies amplify estimation bias, so monitoring residuals for such correlation is a usable early warning.
  • Weighting each observation inversely to its display probability reduces policy amplification in misspecified models, provided every document has non-zero probability of appearing at other ranks, but it does not remove the misspecification bias itself.
  • Simulations that sort documents by expert labels rather than by a trained policy introduce a subtle omitted-variable bias that overstates logging-policy effects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence that goes beyond the paper: on production logs with natural swap rates and feature overlap, the policy-strength degradation should be far smaller than in expert-sorted simulations; measuring that would separate real effects from simulation artifacts.
  • Because overlap shrinks with feature dimensionality, one could build a practical identifiability diagnostic: estimate the Lipschitz constant of the relevance tower and the distributional overlap of rank-specific feature vectors (say, with a two-sample test), and use the pair to predict when approximate identifiability holds.
  • The display-propensity weighting is a step toward training on a uniform-exposure distribution; a natural extension is clipped or self-normalized weights to control variance, and combining it with standard position-bias inverse propensity scoring since the two corrections target different mechanisms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies additive two-tower models for unbiased learning to rank and asks whether the observed performance drop under strong logging policies is due to confounding or to identifiability and misspecification. It makes three claims: (RQ1) two-tower models are identifiable either through document swaps across positions or through overlapping feature distributions across ranks, up to an additive constant; (RQ2) well-specified identifiable models are unaffected by the logging policy, while misspecified models can have their estimation bias amplified by the policy when residuals correlate with placement; and (RQ3) an inverse-propensity sample-weighting scheme mitigates this amplification. The theoretical results are Theorem 4.1, Lemma 5.1, and Lemma 5.2, supported by simulations on MSLR30K with synthetic relevance labels, plus a code repository.

Significance. If the theoretical results are made rigorous, the paper would be a valuable contribution to the ULTR literature: it sharply separates identifiability from logging-policy confounding, extends prior identifiability results (Chen et al., ICML 2024) from swap-based connectivity to feature overlap, and gives practical guidance about model capacity and propensity weighting. The empirical setup is a genuine strength: rather than relying only on nDCG, the simulations directly compare recovered bias parameters with ground truth, and the code and full results are released. The significance is conditional, however, because the two main theoretical pillars—Theorem 4.1 and Lemma 5.2—contain gaps that currently make the RQ1 and RQ2 conclusions rest on unstated assumptions and simulation evidence.

major comments (2)
  1. [Section 4.4, Theorem 4.1, Eq. (14)] The proof introduces a shared Lipschitz constant L for r and r′ that is not part of the theorem statement, which only assumes continuity. Continuity alone does not yield the bound |Δ_k − Δ_k′| ≤ 2L‖x1 − x2‖, and no bound on L is given for the two-layer ELU relevance towers used in Section 7, so the 'approximately identifiable' conclusion has no quantitative content. Moreover, if the support-overlap condition (11) is taken literally, then choosing x1 = x2 at a point in the intersection of supports would make Δ_k = Δ_k′ exactly, so the theorem would imply exact identifiability up to a constant rather than approximate identifiability; the proof's use of 'x1 ≈ x2' suggests an intended near-overlap condition that is never stated. Please state the precise regularity assumption, define 'approximately identifiable' with an explicit error bound, and prove the theorem under those assumptions.
  2. [Section 5.2, Lemma 5.2] The proof of Lemma 5.2 is a mechanism sketch rather than a proof. It defines the residual ε(q,d,k), rewrites the gradient conditions as policy-weighted averages (Eq. 21), and then asserts that the optimizer 'must shift model parameters' and that stronger policy concentration 'can amplify estimation bias.' No argument establishes that a non-zero parameter shift exists under the stated conditions, that the shift is unique, or that its magnitude grows monotonically with the strength of the logging policy. Because the paper's answer to RQ2 rests on this lemma and on the simulations, please either provide a formal theorem with explicit conditions and a bound on the bias as a function of the policy, or clearly label Lemma 5.2 as a conjecture supported by experiments.
minor comments (4)
  1. [Abstract and Section 4.4] The abstract says the model parameters can be 'recovered' when features overlap, but Theorem 4.1 concludes only approximate identifiability; please align the wording with the actual theorem.
  2. [Section 7, Eq. (25)] For α = 0 the formula gives s_{q,d} = 0 because sign(0) = 0, which does not produce the 'random sorting' claimed in Figure 1 and in Section 7; please clarify the intended definition for α = 0 (for example, by treating α = 0 as s = u, the uniform noise term).
  3. [Section 4.4, Eq. (11)] The term 'support' is defined colloquially; please provide a formal definition and specify whether the supports are closed, open, or compact, since this affects whether a point in the intersection of supports can actually be observed and whether exact or approximate identifiability follows.
  4. [Figure 3 and Figure 4 captions] The captions contain the typo 'Ommitted Variables' for 'Omitted Variables'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the central derivations are self-contained, with an unstated Lipschitz regularity gap in Theorem 4.1 that is a correctness risk rather than a circular reduction.

full rationale

The paper's claim chain does not reduce to its inputs. Theorem 4.1 extends the graph-connectedness identifiability criterion of Chen et al. [9] from exact document swaps to overlap in feature support; the proof (Eqs. 12-14) derives approximate equality of rank offsets from continuity and does not assume the conclusion. The proof does contain a missing-support flag: it states "Assuming that r(·) and r′(·) share a Lipschitz constant L" at Eq. (14), while the theorem states only continuity, so the theorem overstates what is proved; Section 4.5 acknowledges the practical difficulty ("deep networks can produce large jumps"), which is a regularity caveat, not a circularity. Lemmas 5.1 and 5.2 are analytical consequences of the expected loss (Eq. 16) and the residual equations (Eqs. 20-21); they do not fit a parameter and then relabel it as a prediction. The simulations of Section 8 verify recovery under a known data-generating process, an internal consistency check rather than a fitted-input-called-prediction. Self-citations ([16], [22], [21], [41]) are contextual or support standard methodology and are not load-bearing for the main theorem or lemmas. Section 9.2's stated limitations (idealized simulations, practical identifiability) further show the authors do not claim more than the analysis supports. Overall, no step is equivalent to its inputs by construction, so circularity is minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new postulated entities and fits no free parameters; its analysis rests on the additive two-tower click model from prior work, a prior identifiability theorem, a Lipschitz regularity condition used in the proof of Theorem 4.1, and the standard positivity assumption for inverse propensity weighting. The theoretical claims are derived from these assumptions rather than from fitted constants.

assumptions (4)
  • domain assumption Clicks are generated by an additive two-tower model: P(C=1|q,d,k)=σ(θ_k+γ_{q,d})
    This is the user behavior model assumed throughout the paper; it is inherited from prior two-tower ULTR work (Yan et al., Guo et al.) and not derived here. The paper's conclusions about well-specified models depend on the true click distribution being in this family.
  • standard math The connected position graph criterion from Chen et al. [9, Theorem 1] applies to additive parameter shifts
    The paper adapts prior identifiability results for multiplicative click models to additive two-tower models; the reader must accept that theorem to follow the swap-based identification argument in Section 4.3.
  • ad hoc to paper The relevance tower r and any alternative parameterization r' are Lipschitz continuous with a shared constant L
    The proof of Theorem 4.1 uses this assumption to bound |Δ_k-Δ_{k'}| ≤ 2L||x1-x2||, but the theorem statement only says 'continuous'. This is an extra regularity condition introduced in the proof, not stated as a formal hypothesis.
  • domain assumption Propensity scores satisfy positivity (π(d,k|q)>0 for all reachable ranks) for the proposed IPS sample weighting
    The sample weighting method requires a non-zero probability of each document appearing at each position; the paper explicitly notes this fails under deterministic policies (Section 8.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank." pith.science (2026). https://pith.science/paper/FT42FTZ5

@misc{pith2026250620501,
  author       = {Pith},
  title        = {Pith review of: Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FT42FTZ5}},
  note         = {Machine review of arXiv:2506.20501}
}
read the original abstract

Additive two-tower models are popular learning-to-rank methods for handling biased user feedback in industry settings. Recent studies, however, report a concerning phenomenon: training two-tower models on clicks collected by well-performing production systems leads to decreased ranking performance. This paper investigates two recent explanations for this observation: confounding effects from logging policies and model identifiability issues. We theoretically analyze the identifiability conditions of two-tower models, showing that either document swaps across positions or overlapping feature distributions are required to recover model parameters from clicks. We also investigate the effect of logging policies on two-tower models, finding that they introduce no bias when models perfectly capture user behavior. However, logging policies can amplify biases when models imperfectly capture user behavior, particularly when prediction errors correlate with document placement across positions. We propose a sample weighting technique to mitigate these effects and provide actionable insights for researchers and practitioners using two-tower models.

Figures

Figures reproduced from arXiv: 2506.20501 by the authors.

Figure 1
Figure 1. Two-tower models trained on deterministic logging [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Evaluating position bias for three well-specified models matching the simulated user behavior. All models converge [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Evaluating position bias under model misspecification: (left) training a linear relevance model on non-linear user [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Reweighting samples during training inversely to the propensity of a policy placing an item in a given position. IPS [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Two-tower models trained on deterministic logging [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 7
Figure 7. Figure 7: Ranking performance of the relevance tower under [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 16 canonical work pages

  1. [1]

    Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork

  2. [2]

    Aman Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Li, Marc Najork, and Thorsten Joachims. 2019. Estimating Position Bias without Intrusive Interven- tions. In The 12th ACM International Conference on Web Search and Data Mining (WSDM)

    2019

  3. [3]

    Bruce Croft

    Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W. Bruce Croft. 2018. Un- biased Learning to Rank with Unbiased Propensity Estimation. In The 41st In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    2018

  4. [4]

    Jessa Bekker, Pieter Robberechts, and Jesse Davis. 2019. Beyond the Selected Completely at Random Assumption for Learning from Positive and Unlabeled Data. In Machine Learning and Knowledge Discovery in Databases: European Conference (ECML PKDD)

    2019

  5. [5]

    Giuseppe Di Benedetto, Alexander Buchholz, Ben London, Matej Jakimov, Yan- nik Stein, Jan Malte Lichtenberg, Vito Bellini, and Matteo Ruffini. 2023. Con- textual Position Bias Estimation Using a Single Stochastic Logging Policy. (2023). https://www.amazon.science/publications/contextual-position-bias- estimation-using-a-single-stochastic-logging-policy

    2023

  6. [6]

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: Composable Transformations of Python+NumPy Programs. http://github.com/google/jax

    2018

  7. [7]

    Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2019. An Analysis of the Softmax Cross Entropy Loss for Learning-to-Rank with Binary Relevance. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR)

    2019

  8. [8]

    Olivier Chapelle and Yi Chang. 2011. Yahoo! Learning to Rank Challenge Overview. Journal of Machine Learning Research (JMLR) 14 (2011), 1–24

    2011

Show all 62 references
  1. [9]

    Mouxiang Chen, Chenghao Liu, Zemin Liu, Zhuo Li, and Jianling Sun. 2024. Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank. In The 41st International Conference on Machine Learning (ICML)

    2024

  2. [10]

    Xiaoshu Chen, Xiangsheng Li, Kunliang Wei, Bin Hu, Lei Jiang, Zeqian Huang, and Zhanhui Kang. 2023. Multi-Feature Integration for Perception-Dependent Examination-Bias Estimation. In Proceedings of The Sixteen ACM International Conference on Web Search and Data Mining (WSDM)

    2023

  3. [11]

    Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015. Click Models for Web Search . Morgan & Claypool. https://doi.org/10.2200/ S00654ED1V01Y201507ICR043

    2015

  4. [12]

    Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An Exper- imental Comparison of Click Position-bias Models. In Proceedings of the 2008 International Conference on Web Search and Data Mining (WSDM)

    2008

  5. [13]

    Alexander D’Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasjeet Sekhon. 2021. Overlap in Observational Studies with High-dimensional Covariates. Journal of Econometrics 221, 2 (2021), 644–654. https://doi.org/10.1016/j.jeconom.2019.10. 014

    2021 doi

  6. [14]

    Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast Ranking with Additive Ensembles of Oblivious and Non-Oblivious Regression Trees.ACM Transactions on Information Systems (TOIS) 35, 2,...

    2016

  7. [15]

    Domenico Dato, Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, and Nicola Tonellotto. 2022. The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation. In The 45th International ACM SIGIR Conference on Research and Development in Information Retr...

    2022

  8. [16]

    Romain Deffayet, Philipp Hager, Jean-Michel Renders, and Maarten de Rijke

  9. [17]

    Romain Deffayet, Jean-Michel Renders, and Maarten De Rijke. 2023. Evaluating the Robustness of Click Models to Policy Distributional Shift. ACM Transactions on Information Systems (TOIS) 41, 4, Article 84 (2023)

    2023

  10. [18]

    Zhichong Fang, Aman Agarwal, and Thorsten Joachims. 2019. Intervention Harvesting for Context-Dependent Examination-Bias Estimation. In The 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    2019

  11. [19]

    Borgwardt, Malte J

    Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A Kernel Two-sample Test. The Journal of Machine Learning Research (JMLR) 13, null (March 2012), 72–773

    2012

  12. [20]

    Huifeng Guo, Jinkai Yu, Qing Liu, Ruiming Tang, and Yuzhou Zhang. 2019. PAL: A Position-bias Aware Learning Framework for CTR Prediction in Live Recom- mender Systems. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys)

    2019

  13. [21]

    Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, and Harrie Oosterhuis

  14. [22]

    Philipp Hager, Maarten de Rijke, and Onno Zoeter. 2023. Contrasting Neural Click Models and Pointwise IPS Rankers. In The 45th European Conference on Information Retrieval (ECIR)

    2023

  15. [23]

    Malay Haldar, Prashant Ramanathan, Tyler Sax, Mustafa Abdool, Lanbo Zhang, Aamir Mansawala, Shulin Yang, Bradley Turnbull, and Junshuo Liao. 2020. Im- proving Deep Learning for Airbnb Search. InThe 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (WSDM)

    2020

  16. [24]

    Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Ronde- pierre, Andreas Steiner, and Marc van Zee. 2024. Flax: A Neural Network Library and Ecosystem for JAX . http://github.com/google/flax

    2024

  17. [25]

    Leonid Hurwicz. 1950. Generalization of the Concept of Identification. Statistical Inference in Dynamic Economic Models 10 (1950), 245–57

    1950

  18. [26]

    Rolf Jagerman, Harrie Oosterhuis, and Maarten de Rijke. 2019. To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions. In The International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    2019

  19. [27]

    Rolf Jagerman, Xuanhui Wang, Honglei Zhuang, Zhen Qin, Michael Bendersky, and Marc Najork. 2022. Rax: Composable Learning-to-Rank Using JAX. In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

    2022

  20. [28]

    Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay

  21. [29]

    Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. In Proceedings of the Tenth ACM Inter- national Conference on Web Search and Data Mining (WSDM)

    2017

  22. [30]

    Kirill Khrylchenko and Alexander Fritzler. 2023. Personalized Transformer- based Ranking for e-Commerce at Yandex. arXiv:2310.03481 [cs.IR] https: //arxiv.org/abs/2310.03481

    2023 arXiv

  23. [31]

    Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models: Principles and Techniques. MIT press

    2009

  24. [32]

    Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. 2019. Generalized Sliced Wasserstein Distances. In Advances in Neural Information Processing Systems (NeurIPS) , Vol. 32

    2019

  25. [33]

    Tjalling C Koopmans and Olav Reiersol. 1950. The Identification of Structural Characteristics. The Annals of Mathematical Statistics 21, 2 (1950), 165–181

    1950

  26. [34]

    Arthur Lewbel. 2019. The Identification Zoo: Meanings of Identification in Econometrics. Journal of Economic Literature 57, 4 (2019), 835–903

    2019

  27. [35]

    Muthukrishnan, Vishwa Vinay, and Zheng Wen

    Shuai Li, Yasin Abbasi-Yadkori, Branislav Kveton, S. Muthukrishnan, Vishwa Vinay, and Zheng Wen. 2018. Offline Evaluation of Ranking Policies with Click Models. In The 24th ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining (KDD)

    2018

  28. [36]

    David Lopez-Paz and Maxime Oquab. 2017. Revisiting Classifier Two-Sample Tests. In International Conference on Learning Representations (ICLR)

    2017

  29. [37]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. (2019). arXiv:1711.05101 [cs.LG]

    2019 arXiv

  30. [38]

    Dan Luo, Lixin Zou, Qingyao Ai, Zhiyu Chen, Chenliang Li, Dawei Yin, and Brian D. Davison. 2024. Unbiased Learning-to-Rank Needs Unconfounded Propen- sity Estimation. In The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    2024

  31. [39]

    Maddison, Andriy Mnih, and Yee Whye Teh

    Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The Concrete Distribu- tion: A Continuous Relaxation of Discrete Random Variables. InThe International Conference on Learning Representations (ICLR)

    2017

  32. [40]

    Harrie Oosterhuis. 2022. Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank. In Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval (SIGIR)

    2022

  33. [41]

    Harrie Oosterhuis and Maarten de Rijke. 2020. Policy-Aware Unbiased Learning to Rank for Top-k Rankings. In The International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    2020

  34. [42]

    Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. arXiv:1306.2597 [cs.IR]

    2013 arXiv

  35. [43]

    Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Mike Bendersky, and Marc Najork. 2021. Are Neural Rankers still Out- performed by Gradient Boosted Decision Trees?. In International Conference on Learning Representations (ICLR)

    2021

  36. [44]

    Filip Radlinski and Thorsten Joachims. 2006. Minimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs. InThe 21st National Conference on Artificial Intelligence (AAAI)

    2006

  37. [45]

    A. Raue, C. Kreutz, T. Maiwald, J. Bachmann, M. Schilling, U. Klingmüller, and J. Timmer. 2009. Structural and Practical Identifiability Analysis of Partially Observed Dynamical Models by Exploiting the Profile Likelihood. Bioinformatics 25, 15 (06 2009), 1923–1929. https://do...

    2009 doi

  38. [46]

    Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting Clicks: Estimating the Click-through Rate for New Ads. In Proceedings of the 16th International Conference on World Wide Web (WWW) . Unidentified and Confounded? Understanding Two-Tower Models for Unbiased ...

    2007

  39. [47]

    Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata

  40. [48]

    Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020. When Inverse Propensity Scoring Does Not Work: Affine Corrections for Unbiased Learning to Rank. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM)

    2020

  41. [49]

    Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to Rank with Selection Bias in Personal Search. InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    2016

  42. [50]

    1989.Learning from Delayed Rewards

    Christopher John Cornish Hellaby Watkins. 1989.Learning from Delayed Rewards. Ph. D. Dissertation. University of Cambridge

    1989

  43. [51]

    Rafael Wilms, Eveline Mäthner, Lothar Winnen, and Ralf Lanwehr. 2021. Omitted Variable Bias: A Threat to Estimating Causal Relationships.Methods in Psychology 5 (2021), 100075. https://doi.org/10.1016/j.metip.2021.100075

    2021

  44. [52]

    Xinwei Wu, Hechang Chen, Jiashu Zhao, Li He, Dawei Yin, and Yi Chang. 2021. Unbiased Learning to Rank in Feeds Recommendation. In The 14th ACM Interna- tional Conference on Web Search and Data Mining (WSDM)

    2021

  45. [53]

    Le Yan, Zhen Qin, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2022. Revisiting Two-tower Models for Unbiased Learning to Rank. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    2022

  46. [54]

    Yunan Zhang, Le Yan, Zhen Qin, Honglei Zhuang, Jiaming Shen, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2023. Towards Disentangling Relevance and Bias in Unbiased Learning to Rank. In The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

    2023

  47. [55]

    Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. 2019. Recommending What Video to Watch Next: A Multitask Ranking System. In Proceedings of the 13th ACM Conference on Recommender System...

    2019

  48. [56]

    Honglei Zhuang, Zhen Qin, Xuanhui Wang, Michael Bendersky, Xinyu Qian, Po Hu, and Dan Chary Chen. 2021. Cross-Positional Attention for Debiasing Clicks. In Proceedings of the Web Conference 2021 (WebConf)

    2021

  49. [57]

    Lixin Zou, Haitao Mao, Xiaokai Chu, Jiliang Tang, Wenwen Ye, Shuaiqiang Wang, and Dawei Yin. 2022. A Large Scale Search Dataset for Unbiased Learning to Rank. In Advances in Neural Information Processing Systems (NeurIPS) . ICTIR ’25, July 18, 2025, Padua, Italy Philipp Hager,...

    2022

  50. [2005]

    In In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    Accurately Interpreting Clickthrough Data as Implicit Feedback. In In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

  51. [2019]

    In The World Wide Web Conference (WWW)

    Addressing Trust Bias for Unbiased Learning-to-Rank. In The World Wide Web Conference (WWW)

  52. [2020]

    In International Conference on Web Search and Data Mining (WSDM)

    Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback. In International Conference on Web Search and Data Mining (WSDM)

  53. [2023]

    In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

    An Offline Metric for the Debiasedness of Click Models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

  54. [2024]

    In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM)

    Unbiased Learning to Rank: On Recent Advances and Practical Applications. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM)

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.