{"id":"760764d1-12fe-425c-8aaa-be6053ca24af","arxiv_id":"2507.07909","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"IPSsim adds a similarity-to-top-documents term to the IPS weight and achieves small NDCG improvements in large top-n settings on simulated click benchmarks.","lead":"The paper introduces IPSsim, a modification of Inverse Propensity Scoring that blends a document-similarity term into the click-debiasing weight for unbiased learning to rank. In simulated click experiments on two public datasets, IPSsim reports modest NDCG gains over standard estimators, mainly when ranking top-30 or top-50 documents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 3% NDCG gain over DR at n=50 is not reproducible from the paper because the comparison conditions are inequivalent: IPSsim's weight includes a tuned alpha and a similarity term that is not an inverse propensity, and no unbiasedness argument is given.","rationale":"The reader's weakest assumption correctly identifies that the convex combination is not an inverse propensity and lacks theoretical support. I agree, but I want to strengthen the point with the concrete mechanism: the validation-set selection of alpha and |T| makes the reported gain a selected maximum, and the lack of an unbiasedness proof means the estimator could be arbitrarily biased. The paper's own caveat—that IPSsim matches the logging policy in many cells—indicates that the similarity term often has no effect, underscoring that the gains are conditional on tuned hyperparameters. A fixed-alpha baseline would settle whether the method itself, rather than the selection procedure, is responsible for the improvement. No ad hominem: the critique is on the statistical validity of the experimental claim.","tokens_in":10147,"tokens_out":782,"duration_ms":10881,"concrete_test":"Re-run the comparison with a fixed, non-tuned alpha (e.g., alpha=0.5) and a pre-specified |T| (e.g., |T|=5), and report the mean NDCG over 5 runs for all configurations without selecting the best. If the fixed-alpha IPSsim does not beat DR by a statistically significant margin after a Bonferroni correction for the 6 IPSsim variants, the headline empirical claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is empirical: IPSsim outperforms DR at n>=30, e.g., 2.88% NDCG gain at n=50 on ISTELLA-S (Table 1, n=50, clicks=10^6: DR 0.8135 vs IPSsim(cosa5) 0.8344, ~2.6%). However, the comparison is not apples-to-apples. IPSsim's estimator (Eq. 2) replaces the inverse propensity 1/rho_i with a convex combination (1-alpha)/rho_i + alpha*Avgsim(i,T). This is no longer an inverse propensity score, and the paper provides no proof of unbiasedness or consistency for this weighted estimator under the click model. In fact, if the similarity term is not proportional to the true inverse propensity, the estimator is biased for any alpha>0. The authors acknowledge this indirectly by saying they 'hypothesise' (Section 3) rather than prove. Moreover, alpha is selected on the validation set over a grid (0.1 increments) and |T| is chosen from {5,10}, so the reported best IPSsim result is a selected maximum over 6 configurations (3 similarity measures x 2 |T|) and 10 alpha values. No multiple-comparison correction is applied; the t-test compares only the final selected configuration against DR. Given that the selected alpha for IPSsim(cosa5) at n=50, 10^6 clicks is 0.9, the method is essentially 90% similarity-based and only 10% IPS. Thus the '~3% improvement' is the best of 60 tuned configurations, not a property of the proposed estimator family. The claim that IPSsim is 'more effective than existing IPS estimators' is an overstatement; the paper's own tables show IPSsim matches LP exactly in many cells (e.g., MSLR-WEB30K n=10), indicating the similarity term often contributes nothing and the estimator degenerates to a constant weight.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IPSsim, an extension of Inverse Propensity Scoring (IPS) for unbiased learning to rank. Instead of weighting each clicked document by the inverse propensity 1/rho_i alone, Eq. (2) uses a convex combination (1-alpha)/rho_i + alpha*Avgsim(i,T), where Avgsim is the average similarity of a document to the top-ranked documents in the historical ranking. The method is evaluated on the ISTELLA-S and MSLTR-WEB30K datasets under simulated clicks following Oosterhuis's setup, across top-n settings n=10,30,50 and click counts 10^4,10^6,10^8, comparing against Naive, IPS, RPS, DR, MIPSLTR, and DRM. The authors report that IPSsim outperforms existing IPS estimators particularly for n>=30, with a claimed statistically significant ~3% NDCG improvement over DR. The paper contains no theoretical bias or consistency analysis of the modified estimator; alpha and |T| are selected on validation data, and the reported headline result is the best among several tuned configurations.","tokens_in":10504,"tokens_out":4437,"duration_ms":45485,"significance":"If the central empirical claim were robust, the contribution would be a simple and practical way to inject document similarity into existing IPS-style debiasing for learning to rank, with improvements in top-heavy rankings on two standard public benchmarks. The paper has clear strengths: it follows a widely used semi-synthetic evaluation protocol, uses two large datasets, varies top-n and click volume, reports results for three similarity instantiations, and provides significance tests over five runs. The learned alpha values are reported for every configuration, which is transparent. However, the current evidence does not yet support the headline claim. The modified estimator is not an inverse propensity score and no unbiasedness argument is given; the reported gains are selected as the best of many validation-tuned configurations without multiple-comparison correction; and a large fraction of table cells show IPSsim exactly matching the logging policy, indicating the method often has no effect. These issues are load-bearing for the paper's central claim, so the manuscript needs substantial revision before the empirical conclusion can be accepted.","major_comments":[{"comment":"The proposed weight (1-alpha)/rho_i + alpha*Avgsim(i,T) is not an inverse propensity score, and the paper provides no unbiasedness or consistency argument under the affine click model described in Section 4. For any alpha > 0, the second term introduces bias unless Avgsim(i,T) happens to be proportional to the true inverse propensity; the conditions under which this holds are neither stated nor tested. The authors' own Section 3 frames the connection as a 'hypothesis' rather than a derivation. Additionally, Eq. (1) defines T as documents highly ranked in H, not necessarily relevant, although the motivating hypothesis refers to 'highly ranked relevant documents'. Please provide a bias/variance decomposition for Eq. (2) under the click model, state the conditions on Avgsim under which the estimator is unbiased, or at minimum compare the empirical bias of IPSsim and IPS against known ground-truth propensities in a controlled experiment.","section":"Section 3, Eq. (2)"},{"comment":"Alpha is learned on the validation set over a grid in increments of 0.1, |T| is selected from {5,10}, and three similarity measures are tested, giving 60 tuned configurations per dataset. The tables underline the best IPSsim variant for each setting, and the significance tests compare only that underlined configuration against the baselines, with no correction for multiple comparisons. The reported 2.88%/'~3%' improvement is therefore the maximum of many configurations, not a property of the estimator family. Please report the performance of all configurations (or a summary such as mean/median across configurations), describe the model-selection procedure, and account for the selection step in the significance testing, for example by using a separate model-selection holdout or a multiple-comparison correction.","section":"Section 4, Tables 1 and 2"},{"comment":"Many table cells show IPSsim exactly matching the logging policy (LP), e.g., all IPSsim variants at MSLTR-WEB30K n=10 with 10^4 clicks equal LP=0.4734, and numerous other cells in both tables are identical to LP. This indicates that in many settings the learned weights produce no effective debiasing, which directly conflicts with the abstract's claim that 'our IPSsim estimator is more effective than the existing IPS estimators'. The paper should quantify the consistency of the improvement, for example by reporting the number of settings (out of 18 per dataset) in which each IPSsim variant outperforms DR, and by discussing the settings where IPSsim has no effect.","section":"Tables 1 and 2"},{"comment":"The headline example is internally inconsistent. The abstract states a ~3% NDCG improvement over DR when n=50, while Section 6 states a 2.88% improvement when n=30. Table 1 shows that for n=30 with 10^6 clicks, DR=0.8048 and the best IPSsim is 0.8280 (a 2.88% relative gain), whereas for n=50 with 10^6 clicks, DR=0.8135 and the best IPSsim is 0.8344 (about 2.57% relative gain). The authors should correct the abstract and Section 6 so that the claimed improvement matches the table, and should specify the exact comparison (dataset, click count, IPSsim variant) for every numerical claim.","section":"Abstract and Section 6"}],"minor_comments":[{"comment":"There is a typographical fragment 'wheresim' after the equation; the sentence should be completed and the notation sim(i,i_m) defined before first use.","section":"Section 3, Eq. (1)"},{"comment":"The click counts in Tables 1 and 2 are printed as '104', '106', and '108', which is easily misread as one hundred and four rather than 10^4; please use superscripts or explicit notation.","section":"Section 4"},{"comment":"The trust-bias correction term \\hat{\\epsilon}_{k_h(i)} in Eq. (2) is not defined precisely; the text says it is 'a trust bias correction' but does not give the formula or a precise reference to the version used.","section":"Section 4, click model"},{"comment":"The paper reports mean NDCG with significance triangles but no standard deviations or confidence intervals; adding these would help readers assess the stability of the reported gains over the five runs.","section":"Section 5"},{"comment":"The description of MIPSLTR and DRM in Eqs. (3) is somewhat terse; in particular, the role of the sum over s documents and the exact definition of M(k|i_j) should be clarified for reproducibility.","section":"Section 4, MIPSLTR/DRM"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for SIGIR and the underlying idea is reasonable, but the current manuscript overstates the empirical evidence by selecting the best validation-tuned configuration. The absence of any theoretical grounding for the modified estimator is a further risk. I would encourage the authors to resubmit after a rigorous revision that addresses the selection issue and reports the failure cases honestly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a modest, plausible extension of IPS for ULTR, but the headline claim overstates what the tables show. The idea of blending inverse propensity with average similarity to top-ranked documents is not in the prior work they cite, so there is a small novel kernel. The evaluation on ISTELLA-S and MSLTR-WEB30K follows a standard semi-synthetic setup, and the authors are transparent that they are testing a hypothesis rather than proving unbiasedness.\n\nThe main problem is the significance testing. Alpha is tuned per variant on the validation set, and the tables underline the best of six IPSsim configurations; the t-test then compares baselines against that underlined configuration without any correction for the selection. Many cells show IPSsim exactly matching the logging policy (e.g., all IPSsim variants at n=10 on MSLR-WEB30K), so the method frequently does nothing. Claiming that IPSsim 'is more effective than existing IPS estimators' based on a few selected n>=30 cells is an overstatement. The most honest reading is: in some top-n settings, the best tuned variant beats DR by a couple of NDCG points.\n\nThere are also missing experimental details: lambda in exp(-lambda*d) is never specified, and the target ranker architecture is not given. Both should be in the paper.\n\nThat said, this is not a bad paper. The authors don't oversell the theory; they explicitly say 'hypothesise.' The distribution analysis in Figures 1 and 2 is a nice attempt to explain dataset differences. The comparison with MIPS and DRM is reasonable.\n\nFor a short SIGIR paper, this could be worth referee time if the authors provide code, specify the missing hyperparameters, and redo the significance testing with a multiple-comparison correction or a pre-registered configuration. As it stands, the central claim should be read as conditional. I would not cite it as a definitive result, but it is a legitimate increment for ULTR researchers.","headline":"Useful incremental idea, but the headline gain is the best of many tuned variants and needs more careful significance testing before the claim is taken at face value.","tokens_in":11080,"tokens_out":2758,"would_cite":false,"duration_ms":31208,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Similarity-aware propensity weights improve unbiased learning to rank, with significant NDCG gains over the Doubly Robust estimator in top-30 and top-50 settings.","keywords":["unbiased learning to rank","inverse propensity scoring","position bias","document similarity","NDCG","click debiasing","counterfactual learning to rank"],"falsifier":"If the same NDCG improvements persist when Avgsim(i,T) is computed from randomly shuffled document features (so similarity is pure noise) while alpha is re-tuned, then the claimed benefit is not attributable to document similarity and the central hypothesis fails.","tokens_in":9894,"feed_emoji":"🔍","tokens_out":6790,"duration_ms":64828,"temperature":0.7,"pith_summary":"Position-bias correction in learning to rank usually reweights clicks by the inverse of the probability that a document is seen at its rank. The paper argues that this ignores a useful signal: low-ranked documents that resemble the documents at the top of the historical ranking are likely relevant too. It therefore proposes IPSsim, which weights each click by a convex combination of the inverse propensity and the average similarity between the clicked document and the top-ranked documents. On two standard public LTR datasets with simulated clicks, the paper reports that IPSsim outperforms the existing IPS, RPS, Doubly Robust, and MIPS-style estimators, with the gains concentrated in top-30 and top-50 evaluation settings (for example, a statistically significant ~3% NDCG improvement over the Doubly Robust estimator at n=50). If the claim holds, similarity is an effective additional debiasing signal that requires no extra side information beyond the document features already present in LTR datasets.","feed_headline":"Similarity-aware clicks beat standard debiasing at top-50","feed_subtitle":"A convex mix of inverse propensity and document similarity beats Doubly Robust estimators when ranking many documents.","key_machinery":"The load-bearing object is the IPSsim estimator (Eq. 2), which blends the standard inverse-propensity weight with an average-similarity term: each click on document i is weighted by (1-alpha)/rho_i + alpha*Avgsim(i,T), where Avgsim(i,T) is the mean pairwise similarity between i and the documents in T, the set of documents highly ranked in the historical ranking. The hyperparameter alpha is learned on a validation set in increments of 0.1, and three feature-based similarity measures (cosine, Euclidean, Manhattan) are tested, all normalized to the unit interval. The estimator also keeps the trust-bias correction term from the affine IPS formulation. This convex blend is what carries the argument: it gives low-ranked documents that resemble top-ranked documents a larger effective weight, on the hypothesis that such documents are likely relevant despite their low rank.","core_discovery":"The central claim is that incorporating document similarity into inverse-propensity estimation yields a more effective debiasing estimator for unbiased learning to rank. For each document i, IPSsim replaces the standard propensity weight 1/rho_i with (1-alpha)/rho_i + alpha*Avgsim(i,T), where T is the set of documents ranked at the top of the historical ranking and alpha in [0,1] is tuned on validation data. The paper reports that this estimator improves NDCG over Naive, IPS, RPS, DR, MIPSLTR, and DRM baselines on ISTELLA-S and MSLTR-WEB30K, particularly when n>=30 and when more training clicks are available. The intended mechanism is that low-ranked documents similar to top-ranked documents receive a boosted effective weight, counteracting the propensity penalty they incur for being ranked low.","pith_inferences":["Because IPSsim is not an inverse propensity and has no unbiasedness proof, part of its empirical gain may come from variance reduction or regularization rather than from correctly recovering relevance; a variance decomposition across the five runs would separate these effects.","The same convex similarity correction could be applied to the imputation term of the Doubly Robust estimator or to RPS, not just to the propensity weight; the paper does not test these combinations.","Since alpha is tuned per dataset and top-n setting, the method's success depends on the validation set being representative; a sensitivity analysis across validation splits would show how fragile the gains are."],"forward_implications":["The paper reports that IPSsim achieves statistically significant NDCG gains over existing IPS estimators in top-n settings with n>=30, including a 2.88% improvement over the Doubly Robust estimator on ISTELLA-S.","The gains over baselines grow as the number of training clicks increases from 10^4 to 10^8.","The choice of similarity measure has little effect on IPSsim's performance, with the cosine variant often the best.","The method needs only the document feature vectors already present in LTR datasets, not external side information.","The authors propose adapting IPSsim to transformer-based ranking models and datasets with real user clicks as future work."],"supporting_citations":[{"why":"Establishes IPS-based unbiased learning to rank from biased clicks, the estimator that IPSsim extends.","marker":"[15]"},{"why":"Provides the Doubly Robust estimator and the experimental and click simulation setup that the paper follows and compares against.","marker":"[22]"},{"why":"Supplies the affine IPS formulation and trust-bias correction term that IPSsim inherits.","marker":"[29]"},{"why":"Introduces the Ratio-Propensity Scoring estimator used as a variance-reduction baseline.","marker":"[30]"},{"why":"Introduces Marginalised IPS using item similarity, which the paper adapts into the MIPSLTR and DRM baselines.","marker":"[27]"},{"why":"Provides evidence that documents with high similarity tend to share relevance labels, motivating the paper's hypothesis.","marker":"[18]"},{"why":"Shows that item similarity can help mitigate position bias in recommender systems, a key prior supporting the approach.","marker":"[26]"},{"why":"First proposed IPS for mitigating selection bias in personal search, the foundational formulation cited for the method.","marker":"[31]"}],"fun_headline_variants":["Similarity-aware IPS reduces position bias in LTR","Use top-ranked similarity to debias click data","Doc similarity enhances IPS for unbiased ranking","Similarity-boosted propensity beats Doubly Robust at depth","Blend IPS with document similarity to counter click bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a convex blend of the inverse propensity and average similarity to top-ranked documents remains an effective debiasing weight, even though IPSsim is not an inverse propensity and the paper provides no bias or consistency proof for it.","fun_headline_variants_meta":{"raw":{"variants":["Similarity-aware IPS reduces position bias in LTR","Use top-ranked similarity to debias click data","Doc similarity enhances IPS for unbiased ranking","Similarity-boosted propensity beats Doubly Robust at depth","Blend IPS with document similarity to counter click bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000425,"raw_usage":{"total_tokens":2205,"prompt_tokens":999,"completion_tokens":1206,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1132}},"tokens_in":615,"tokens_out":1206,"duration_ms":12704,"temperature":1.0,"reasoning_tokens":1132,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:29:11.522998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If the same NDCG improvements persist when Avgsim(i,T) is computed from randomly shuffled document features (so similarity is pure noise) while alpha is re-tuned, then the claimed benefit is not attributable to document similarity and the central hypothesis fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes IPS-based unbiased learning to rank from biased clicks, the estimator that IPSsim extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Doubly Robust estimator and the experimental and click simulation setup that the paper follows and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the affine IPS formulation and trust-bias correction term that IPSsim inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Ratio-Propensity Scoring estimator used as a variance-reduction baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that documents with high similarity tend to share relevance labels, motivating the paper's hypothesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that item similarity can help mitigate position bias in recommender systems, a key prior supporting the approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"First proposed IPS for mitigating selection bias in personal search, the foundational formulation cited for the method."}],"review_version":1}