{"id":"730c9aba-d05c-4b66-9ae8-cf04f2122e6f","arxiv_id":"2508.20798","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A user-aware inverse-propensity estimator corrects personalized bias in learning to rank from click logs, with proofs of unbiasedness and lower variance.","lead":"This paper shows that standard unbiased learning to rank gives biased results when different users issue different queries and browse results differently. It proposes a user-aware inverse propensity estimator that reweights clicks by query-specific user examination distributions, with proven unbiasedness and lower variance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core theory is sound under the explicit shared-relevance assumption; the main risk is that personalized relevance judgments would invalidate the ideal-loss target and the unbiasedness proof. This scope limitation should be stated more prominently.","rationale":"The reader's weakest_assumption identifies the shared-relevance assumption as the foundation of the ideal-loss definition, the do-calculus derivation, and the unbiasedness proof. I agree that this is the single most load-bearing assumption: if it fails, the core theoretical claims do not survive. The paper is transparent about the assumption, so this is not a hidden flaw, but it is a substantive scope limitation. The reader's CONDITIONAL verdict already reflects this and other empirical loose ends (e.g., the Baidu-ULTR ideal-skyline anomaly, GenAI labels, and the variance-condition issue). My stress-test does not find grounds to move the verdict: the in-scope theory is correct, the experiments support the claims under the stated assumptions, and the shared-relevance limitation is explicitly acknowledged. Therefore, I recommend no change to the reader's verdict. The proposed concrete test would substantiate the scope limitation empirically; if it shows large bias under personalized relevance, the authors should revise the title/abstract to avoid implying a fully general solution. Even so, the paper's contribution remains valid for the shared-relevance setting, which covers much of the existing ULTR literature.","tokens_in":18614,"tokens_out":18535,"duration_ms":184467,"concrete_test":"Extend the semi-synthetic simulation in Section 4 to include user-specific relevance: add a u->r edge to Figure 1(b), e.g., sample a per-user relevance perturbation δ_u(q,d) around the shared annotation, so P(r=1|q,d,u) = base + δ_u. Re-run the Table 1 comparison (Yahoo! LETOR or Baidu-ULTR) with the user-aware estimator. Measure the bias of the estimated relevance against (a) the shared-annotation ideal and (b) the user-averaged ideal loss. If the user-aware estimator no longer recovers the shared ideal, this confirms that the central claim holds only under the shared-relevance assumption and should be scoped accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the user-aware estimator (Eq. 9) is unbiased and lower-variance than the straightforward estimator—rests on the assumption stated in Section 3.1: 'users have the same relevance judgments for given query-document pairs, so r is only affected by x.' This assumption is load-bearing in three specific places. First, the ideal loss in Eq. 5 writes r(d) without a user index; if r also depends on u, then there is no single 'ideal' ranking loss to estimate. Second, the do-calculus derivation in Section 3.3 identifies P(r=1|x) from click data via the second rule of do-calculus; the rule requires all backdoor paths from e to c to be blocked by x. If the graph acquires a direct u->r edge, the path e<-u->r->c remains unblocked even after conditioning on x, so the do-calculus step P(c=1|do(e=1),x)=P(c=1|e=1,x) is no longer valid. Third, Theorem 3.1's unbiasedness proof uses E_u[P(e=1|k,u)P(u|q)] as the propensity; if r varies by user, then E[c(d)] = r(d,u)P(e=1|k,u), and the cancellation to l_ideal fails because r(d,u) cannot be factored out of the expectation. The authors explicitly acknowledge this as future work, but the acknowledgment appears only in Section 3.1 and the conclusion; the abstract and title claim a general solution to 'personalized bias' without this caveat. Because the method's guarantees are restricted to a regime where users agree on relevance—a strong assumption, especially in search—this is the most load-bearing concern for the central claim. The issue is not an internal inconsistency; it is a scope condition that should be carried through the paper's claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces user heterogeneity into unbiased learning-to-rank (ULTR). It augments the standard click-generation causal graph with a user node that affects both query issuance and examination, and argues that existing user-oblivious inverse-propensity-scoring (IPS) estimators are biased because of the additional backdoor path e ← u → q → x → r → c. It proposes a user-aware IPS estimator (Eq. 9) whose propensity is Σ_u P(e=1|k(d),u)P(u|q), proves unbiasedness (Theorem 3.1) under a positive-examination assumption and shared relevance, proves lower variance than a per-user estimator (Theorem 3.3) under a condition relating the query-level propensity to the empirical session average, and reports experiments on two semi-synthetic datasets and a real-world dataset.","tokens_in":18996,"tokens_out":12505,"duration_ms":128627,"significance":"Within its stated scope, the theoretical contribution is valuable: it identifies a realistic failure mode of standard ULTR and provides a simple, computationally cheap correction with a clean proof. The unbiasedness proof is direct, the variance-reduction argument is non-trivial, and the experiments cover oracle-propensity, estimated-propensity, and real-world settings, with code released. The main limitation is that 'personalized' is restricted to query-issuing and examination behavior; personalized relevance judgments are explicitly outside the model. If the empirical anomaly noted below is resolved, this would be a useful contribution to the ULTR literature.","major_comments":[{"comment":"The ideal loss, the do-calculus step in Section 3.3, and the unbiasedness proof all assume that relevance r(d) has no user index. If relevance judgments vary by user, Eq. (5) is no longer a well-defined target, and the causal graph acquires a direct u→r edge, leaving the backdoor path e←u→r→c open; the equality P(c=1|do(e=1),x)=P(c=1|e=1,x) and the cancellation in Theorem 3.1 then fail. The paper acknowledges this in Section 3.1 and the conclusion, but the abstract and title claim a more general solution to 'personalized bias.' Please move this scope restriction to the front and adjust the abstract/introduction claims accordingly.","section":"Section 3.1, Eq. (5), Theorem 3.1"},{"comment":"The lower-variance guarantee is conditional on the inequality Σ_u P(e|k,u)P(u|q) ≥ (1/|N_q|) Σ_{(c,u)∈N_q} P(e|k,u). This is not an innocuous condition; it is automatic only when P(u|q) is the empirical user frequency computed from the same sessions, in which case it holds with equality. The theorem statement should present this condition explicitly rather than in a footnote, and the variance derivation should clarify that P(u|q) is treated as known, so estimation error in P(u|q) is not included. As stated, the 'lower variance' result is conditional, not unconditional.","section":"Theorem 3.3 and footnote 6"},{"comment":"The user-aware estimator exceeds the 'ideal' skyline on Baidu-ULTR at nDCG@1 (0.5183 vs 0.5138), nDCG@3 (0.5318 vs 0.5300), and nDCG@5 (0.5464 vs 0.5437). Since the ideal model is trained on 100% relevance annotations and is described as an upper bound, this is inconsistent with standard expectations and suggests a setup or evaluation issue (e.g., test-label noise, model selection, or a mismatch between the training objective and the evaluation metric). Please explain or correct the experimental protocol.","section":"Table 1 (Baidu-ULTR block)"}],"minor_comments":[{"comment":"The user-oblivious IPS-PBM propensity is written as Σ_u P(e|k,u)P(u). If the logged corpus is generated by users with different query distributions, the correct marginal is P(e|k)=Σ_u P(e|k,u)P(u|k), not P(u). Please clarify or correct the display.","section":"Eq. (7)"},{"comment":"The derivation replaces P(u|x=x0) with P(u|q) in the denominator. This step should be justified from the causal graph (e.g., u ⊥ d | q and x=(q,d)); otherwise the notation is ambiguous.","section":"Section 3.3"},{"comment":"The expectation E_{u,e} should be defined explicitly over the session-generation process (u ~ P(u|q), e ~ P(e|k,u)) to avoid ambiguity about which distribution is being averaged.","section":"Theorem 3.1 proof"},{"comment":"The training loss uses graded labels 0.25·y(d), while the theoretical estimator is developed for binary relevance r(d). Please explain how the inverse-propensity weights are applied to the listwise softmax loss and how the binary-relevance theory connects to the graded-label experiments.","section":"Section 4, Eq. (13)"},{"comment":"Minor typographical issues: in the Theorem 3.3 proof, 'P(e(d) = 1|k(d, u′)' is missing a closing parenthesis; Section 4.3 has 'when|U| = 5' with missing spacing. Please proofread the equations.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely sound within its explicitly stated scope, and the shared-relevance limitation is acknowledged in the body. The main issues are the overbroad framing in the abstract and, more importantly, the Baidu ideal-skyline anomaly in Table 1, which needs a careful check before the empirical claim can be trusted. If the authors clarify the condition in Theorem 3.3 and resolve or explain the skyline inconsistency, I would support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a legitimate and useful extension of IPS-based ULTR. The new causal graph with a user node affecting both query and examination is the real contribution, and the user-aware estimator, which averages examination probabilities over the query-specific user distribution, is a sensible way to block the u-induced backdoor path without the variance blowup of per-session propensities. The unbiasedness proof (Thm 3.1) is clean under the stated PBM and shared-relevance assumptions, and the lower-variance theorem (Thm 3.3) is algebraically correct. The condition for that theorem, however, is guaranteed by the empirical P(u|q) estimated from the same logs, not by the true distribution; the footnote says exactly this. So the theoretical variance claim is narrower than the abstract implies. That is a fixable gap, not a fatal one.\n\nThe load-bearing caveat is the Sec 3.1 assumption that all users share the same relevance judgments. If r depends on u, Eq. (5) has no single ideal loss, and the do-calculus step in Sec 3.3 no longer holds because a direct u->r edge leaves a backdoor path open. The authors do acknowledge this in Sec 3.1 and the conclusion, but the abstract and title claim a general solution to \"personalized bias\" without the caveat. That scope condition needs to be carried through.\n\nOn the empirical side, the semi-synthetic experiments are thorough and the MSE results are informative. The real-world evaluation is thin: 360 annotated queries with GenAI-generated labels at Kappa 0.64, and the user-aware estimator beats the \"ideal\" skyline on Baidu-ULTR on several metrics. That is unexplained and suggests the skyline or the evaluation protocol is off. It needs an explanation before the paper is accepted.\n\nOverall: this is a serious, honest paper with a novel causal reformulation of ULTR. It deserves a full peer review, not a desk reject. With the variance claim tightened, the shared-relevance caveat made prominent, and the skyline anomaly resolved, it would be a solid contribution.","headline":"A clean causal refinement of ULTR whose theory holds under the explicit shared-relevance assumption; the main issues are under-advertised scope and a real-world skyline anomaly.","tokens_in":19580,"tokens_out":3838,"would_cite":true,"duration_ms":36376,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-05T14:48:22.438301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}