{"id":"9056b8a3-4a95-4109-a4af-e94a1c51cf45","arxiv_id":"2507.23208","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"LiDu, a list-wise uncertainty based on pairwise ranking probabilities, is more correlated with Top-N recommendation performance than training loss and QPP baselines.","lead":"This paper proposes LiDu, a list-level uncertainty score that estimates how confident a recommender is about its whole ranking. It shows that LiDu correlates with actual ranking quality across several models and datasets, without needing user feedback labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2) multiplies dependent pairwise events, so LiDu is not the claimed negative log likelihood; the empirical correlation may come from the heuristic product and should be checked against exact list probabilities before the central claim is accepted.","rationale":"The most load-bearing point is exactly the one the reader flagged: LiDu's definition in Eq. (2) is presented as the probability of a ranking list, but it is obtained by multiplying marginal pairwise probabilities. Under Eq. (1)'s Gaussian score model, the pairwise events are correlated, especially because the UQ methods are applied to a shared user embedding; the pairwise variance σ_i^2+σ_j^2 also ignores covariance in the score differences. Thus LiDu is not a negative log likelihood, and the theoretical motivation in Section 3.2 does not directly apply. The empirical results in Table 3 may still show that this particular heuristic correlates with NDCG, but that would be a weaker and different claim than recommending models being self-aware in the sense of calibrated list uncertainty. This is an internal inconsistency, not a disagreement with the field, and it is addressable. A Monte Carlo exact-list-probability check would settle whether the product form or the underlying uncertainty drives the correlation. If the exact variant performs similarly, the central empirical claim survives and only the presentation needs revision; if it does not, the method's status as a principled uncertainty measure is undermined. Since the reader already assigns CONDITIONAL with high confidence, my stress-test does not move the verdict; it sharpens the condition: the authors must either justify Eq. (2) as an approximation or validate the heuristic against an exact joint-probability baseline.","tokens_in":16240,"tokens_out":8397,"duration_ms":99344,"concrete_test":"On the synthetic MF data and on at least one real dataset/model pair (e.g., Grocery + BPRMF), compute LiDu in two ways: (i) the paper's Eq. (11) with closed-form πij; (ii) a Monte Carlo exact list probability: draw 1000 score vectors from the MC-dropout/ensemble posterior used by the paper, retain the observed top-N order for each user, and take the negative log of the Monte Carlo estimate of P(s1>s2>...>sN) (or of the estimated multivariate Gaussian ordering probability). Correlate both variants with NDCG@K using Pearson's r and Win Rate-δ. If variant (ii) matches or beats LiDu, the independence assumption is not the source of the signal and the central empirical claim survives; if variant (ii) is materially worse, the reported advantage is a heuristic artifact and the likelihood interpretation in Eq. (3) should be removed or qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (2) presents P(r1=1,...,rK=K) as a product of pairwise probabilities πij, and Eq. (3) calls −log of this quantity LiDu, i.e., a negative log likelihood. Under the paper's own Gaussian score model, the ranking events are not independent: item scores for a user share the same user representation, and the MC-dropout procedure used to obtain σ adds noise to the user embedding, so score perturbations are correlated across items. Consequently Var(s_i−s_j) is not generally σ_i^2+σ_j^2, and P(s_i>s_j and s_i>s_k) ≠ πijπik. Eq. (2) is therefore not the probability of the ranking list, and LiDu is not the negative log likelihood of any well-defined list distribution. The Top-N variant in Eq. (11) adds position-bias exponents and a 'step' that omits pairs, making the quantity a deliberately weighted heuristic rather than a probability. This matters because the paper's motivation in Section 3.2 and the label 'uncertainty' rest on the likelihood interpretation; without it, the observed correlation with NDCG may be a property of this particular product, not of calibrated list uncertainty. The concern is fixable by treating Eq. (2) explicitly as an approximate score and testing an exact list-probability variant.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LiDu, a list-level uncertainty measure for Top-N recommendation, defined as the negative log probability (Eq. 3) of a ranking list under a Gaussian score model, where the list probability is computed as a product of pairwise comparison probabilities (Eq. 2). The authors validate LiDu as a label-free performance estimator on a synthetic matrix factorization task and on six real-world datasets with five recommenders, comparing against training loss and three QPP baselines. They report that LiDu correlates more strongly with NDCG than the baselines, and further analyze how LiDu varies with user activeness, interest dynamism, and recommendation diversity. The central empirical claim is that list-level uncertainty is a useful label-free performance estimator; the theoretical framing claims that LiDu is a true negative log likelihood of the ranking distribution.","tokens_in":16509,"tokens_out":5105,"duration_ms":53330,"significance":"If the empirical claim holds, the paper makes a useful contribution to a relatively underexplored problem: label-free performance estimation for Top-N recommendation. The study is reasonably broad, covering 30 model-dataset settings, three uncertainty implementations (MC dropout, deep ensemble, variational Bayesian), and comparisons with classical QPP baselines. The use of ReChorus and the public repository are positive reproducibility features. The main weakness is that the theoretical interpretation of LiDu as a probability is not mathematically justified, and the empirical advantages over baselines, while consistent, are modest in absolute terms. The paper is best seen as presenting a heuristic estimator with empirical support rather than a derived likelihood-based uncertainty measure.","major_comments":[{"comment":"Equation (2) treats the probability of the ranking list as the product of pairwise probabilities π_{i,j}. Under the Gaussian score model of Eq. (1), and especially when MC dropout is applied to the shared user embedding, pairwise ranking events are dependent: the score perturbations of different items for the same user are correlated, so Var(s_i - s_j) is not generally σ_i^2 + σ_j^2 and P(s_i > s_j and s_i > s_k) ≠ π_{i,j}π_{i,k}. Consequently Eq. (2) is not the true joint probability of the list, and LiDu in Eq. (3) is not the negative log likelihood of any well-defined ranking distribution. Equation (11) adds position-bias exponents and an undefined \"step\" term, making the quantity a deliberately weighted heuristic rather than a probability. This matters because the paper's motivation and the label 'uncertainty' rely on the likelihood interpretation. The issue is fixable: either derive an exact list probability under the model, or explicitly present LiDu as an approximate/heuristic score and compare it against an exact-list-probability variant to show that the heuristic retains its empirical value.","section":"Section 3.1, Eqs. (1)-(3)"},{"comment":"The Boltzmann argument is an analogy, not a derivation. Equation (4) asserts Q ∝ -E without justification, and Eq. (5) is the generic Boltzmann distribution with no demonstrated connection to LiDu or to the pairwise product in Eq. (2). The text calls this a 'Boltzmann Hypothesis' and treats it as theoretical support, but no quantitative bridge is provided. This does not invalidate the empirical study, but the paper should soften this to a heuristic motivation or provide a formal connection between ranking loss, energy, and list probability.","section":"Section 3.2.1"},{"comment":"The central empirical claim is supported by Table 3, but the average differences over the best baselines are small: win rate 0.654 for LiDu-dp vs. 0.629 for the best baseline, and −Pearson's r 0.198 vs. 0.159. Some individual settings clearly go against the trend, such as the ML-1M rows and SimpleX on Yelp. The significance markers (∗,†,‡) are not accompanied by a description of the statistical test, the number of test samples, or a multiple-comparison correction. Please report the number of ranking samples used in each correlation, provide confidence intervals or paired significance tests across users/seeds, and clarify whether the comparisons are performed on paired samples.","section":"Table 3 and Section 5.1"},{"comment":"The hyperparameter selection for the estimators appears to use the evaluation labels. For QPP baselines, the top-k value is tuned over {10, 100, 1000} to the best estimation; for LiDu, N is set to 10 for XING and 100 for other datasets after inspecting performance, and the sensitivity analysis in Section 5.2 motivates L=1000. If these choices are made using the same test NDCG values used for evaluation, the comparison is not fully label-free and may be optimistic. Please specify clearly whether any tuning is performed on a separate validation split, and if not, describe how the dataset-dependent N values were chosen without using test labels.","section":"Sections 4.3.4 and 5.2"}],"minor_comments":[{"comment":"The term s_n = high_bit(n) + n is undefined; the paper should define high_bit and explain why this step function is used. Also, the position-bias exponent is written as p_n = log2(i+1), but the index i is not defined in that context; it should presumably be p_n = log2(n+1).","section":"Eq. (11)"},{"comment":"The caption says 'The negative Pearson's r is reported', but the table shows positive values. Please state explicitly that the absolute value of Pearson's r is reported, or change the column labels accordingly.","section":"Table 3 caption"},{"comment":"In the definition of Win Rate-δ, the conditioning on e_i > e_j + δ and the choice δ = 5% of the test set size need more explanation; in particular, how ties and near-ties are handled should be stated, and the claim that chance performance is 0.5 should be justified for the conditional event.","section":"Section 4.3.2"},{"comment":"The paper omits point-wise uncertainty baselines after reporting that they show insignificant correlation. For reproducibility and completeness, at least one such baseline (e.g., average prediction variance over top items) should be included in the main table or the supplementary material.","section":"Section 4.3.1"},{"comment":"The text states that sARE results are presented in the supplementary information, but no supplementary file is included with the manuscript. Either include the supplementary results or remove the reference to them.","section":"Section 5.3 / supplementary material"},{"comment":"The caption should clarify that the figure shows Pearson's correlation between LiDu and ranking performance for BPRMF, and should define the 'Ranking Uncertainty' axis; currently the reader must infer the setup from later sections.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper's empirical core is plausible and within the scope of an IR venue. The main risk is the overclaimed likelihood interpretation of Eq. (2); the paper can be published after reframing LiDu as a heuristic and adding an exact-list-probability comparison or at least an explicit acknowledgment of the approximation. I do not see circularity in the main experiments, since LiDu is not fitted to NDCG. The small effect sizes in Table 3 and the under-specified significance tests should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's concrete contribution is a list-wise uncertainty score, LiDu, built as a product of pairwise comparison probabilities, and a broad empirical study showing it tracks NDCG better than training loss and standard QPP baselines across 30 dataset-model settings. That is a real, useful result for people who want to estimate recommender performance without labels, and the win-rate metric is a sensible way to measure estimator quality. The synthetic MF experiment and the active/inactive user analysis are also thoughtful additions.\n\nWhat's new here is the list-level construction: earlier UQ work in RecSys averaged pointwise variances or used score variance directly, and LiDu does something different by combining pairwise probabilities. The MC dropout implementation that perturbs only the user embedding is computationally cheap and clever. I believe the empirical evidence that LiDu beats the baselines on average, even if correlations are modest (Pearson r around 0.2), is credible and worth taking seriously.\n\nThe main soft spot is the math in Eq. (2). The paper calls LiDu the negative log likelihood of the ranking list, but the product of pairwise marginal probabilities is not the joint probability of the ranking under their own Gaussian score model. Score differences for the same user are correlated because the user embedding is shared and MC dropout perturbs it jointly, so P(s_i > s_j) times P(s_i > s_k) is not P(s_i > s_j and s_i > s_k). The paper never flags this as an approximation. That matters for the theoretical motivation: the 'list likelihood' language is doing real work in the narrative. But the method can survive as a heuristic score, and the empirical correlation with NDCG does not depend on the exact probabilistic interpretation. I'd want the authors to either soften the language or test an exact list-probability variant, say by sampling or using a more principled correlated model. This is fixable.\n\nTwo more minor points. First, there are no error bars on the real-data results; the synthetic experiments used five seeds, but the main tables report single numbers. Second, the per-dataset choice of N and L for XING looks like selection on the test set; they should justify it using a validation split or show sensitivity. Neither sinks the paper.\n\nWho is this for? RecSys and IR researchers working on evaluation, model selection, and offline-online gaps. It deserves a serious referee; the revisions are not hard. I'd engage with it, and I'd cite it for the empirical finding that list-wise uncertainty is a practical performance estimator.\n\nRecommendation: send to peer review, with a request to clarify the approximation status of Eq. (2) and report variance across seeds.","headline":"A useful heuristic for label-free ranking performance estimation, but the likelihood framing in Eq. (2) is mathematically shaky and should be fixed before publication.","tokens_in":17083,"tokens_out":1732,"would_cite":true,"duration_ms":22394,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a recommender can estimate its own ranking quality, without labels, from the uncertainty of the list it would generate.","keywords":["recommender systems","uncertainty quantification","list-wise uncertainty","label-free performance estimation","Top-N recommendation","self-awareness","query performance prediction","matrix factorization"],"falsifier":"Synthesize a recommender where item scores share common noise factors, so pairwise comparisons are strongly dependent; compute LiDu and NDCG@K over many train/test splits and check whether the negative correlation persists when the true list likelihood differs sharply from the product approximation. If the correlation becomes positive or vanishes in a regime the paper claims LiDu generalizes to, the central claim would be refuted.","tokens_in":15977,"feed_emoji":"📊","tokens_out":6404,"duration_ms":68427,"temperature":0.7,"pith_summary":"This paper asks whether a recommendation model can estimate its own ranking quality before any user sees the list, and answers yes in a specific sense. It proposes LiDu (List Distribution uncertainty), defined as the negative log-probability of the model generating the observed Top-N ranking, computed from per-item score means and variances. Across a synthetic matrix-factorization task and 30 settings from six real-world datasets and five recommenders, LiDu correlates negatively with NDCG@K: the best variant (LiDu-vb) reaches an average Pearson r of 0.222 and win rate 0.665, beating training loss and three QPP baselines. The paper also reports that LiDu tracks users' interest dynamism during training and list diversity at inference. A sympathetic reader would take the paper as establishing an empirical link between list-level self-uncertainty and actual recommendation performance.","feed_headline":"A recommender's own uncertainty predicts its ranking quality","feed_subtitle":"New list-level measure LiDu beats training loss and QPP baselines across 30 model-dataset settings.","key_machinery":"The central object is LiDu, shorthand for probability-based List Distribution uncertainty. It converts each item's predicted score and estimated variance into a Gaussian comparison probability, then aggregates pairwise comparison probabilities into an approximate likelihood for the generated Top-N ranking list; the estimator is the negative log of that list probability. To keep the tail of the list from dominating, the paper adds a step (only compare items sufficiently far apart) and a position bias inspired by NDCG. The variances feeding LiDu come from MC dropout, deep ensembles, or variational Bayesian last layers. This machinery carries the argument because it is the component that makes uncertainty a property of the rank list rather than of isolated predictions.","core_discovery":"The paper's central claim is that a recommender's uncertainty about the whole ranking list, rather than about individual scores, behaves as a label-free predictor of its ranking performance. The proposed measure LiDu assumes each item score is Gaussian with mean and variance from uncertainty quantification, computes pairwise probability $P(r_i > r_j)$ via Eq. (1), multiplies these pairwise probabilities to approximate the probability of the observed ranking (Eq. 2), and takes the negative log of that product (Eq. 3). The paper argues theoretically that under a matrix-factorization model trained with negative log-likelihood, optimal variance equals squared prediction error (Eq. 8), so high uncertainty aligns with low accuracy; it then demonstrates on six datasets that LiDu is more strongly correlated with NDCG@K than training loss, SMV, NQC, W-Graph, and point-wise variance. It further reports beyond-accuracy properties: higher LiDu for users with more dynamic interests and for more diversified lists.","pith_inferences":["If list-level uncertainty truly tracks performance, then score variance in recommender embeddings encodes information that pointwise scores discard; this suggests a testable extension in which LiDu is computed from learned variance layers on item embeddings, not just user embeddings, to model cold items.","The independence approximation in Eq. (2) understates correlated errors across items; an exact joint Gaussian likelihood over the top-K scores might yield a strictly stronger estimator, providing a concrete next step the paper does not take.","One could combine LiDu with score-magnitude QPP features such as SMV and NQC in a simple regression, since they rely on different signals; the paper evaluates them separately and does not test such a fusion."],"forward_implications":["A deployed recommender could rank its own output quality before showing it to users, flagging low-confidence lists for intervention.","Because LiDu works for inactive users in several datasets, it can serve as a signal for cold-user evaluation and personalized strategy adjustment.","LiDu's correlation with user interest dynamism and list diversity gives a training-time and inference-time diagnostic that standard ranking metrics do not provide.","The paper's success criteria imply downstream uses such as uncertainty-weighted ensembling, data augmentation from uncertain samples, and uncertainty-aware training losses, all named as future work in the paper."],"supporting_citations":[{"why":"Supplies MC dropout, the uncertainty estimator used for LiDu-dp.","marker":"[9]"},{"why":"Supplies deep ensembles, the estimator behind LiDu-en.","marker":"[22]"},{"why":"Supplies variational Bayesian last layers, the estimator behind LiDu-vb.","marker":"[11]"},{"why":"Provides the SMV query performance prediction baseline LiDu must beat.","marker":"[35]"},{"why":"Provides the NQC query-drift baseline used for comparison.","marker":"[34]"},{"why":"Provides the W-Graph retrieval coherency baseline used for comparison.","marker":"[2]"},{"why":"Defines NDCG, the performance ground truth that LiDu is correlated against.","marker":"[18]"},{"why":"Empirical calibration-performance link in image classification that motivates the uncertainty-performance hypothesis.","marker":"[10]"},{"why":"Provides BPRMF, the matrix factorization recommender used in the primary experiments and Figure 1.","marker":"[31]"}],"fun_headline_variants":["Uncertainty predicts recommender ranking quality without labels","Self-aware recommenders: uncertainty as a performance gauge","LiDu links recommender uncertainty to ranking performance","Recommenders can self-assess: LiDu predicts ranking quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on Eq. (2) treating the ranking-list probability as a product of independent pairwise comparisons, even though the paper's own Gaussian score model implies those comparisons are correlated; if that approximation is bad in a dataset, LiDu may still rank lists but is no longer the true list likelihood.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty predicts recommender ranking quality without labels","Self-aware recommenders: uncertainty as a performance gauge","LiDu links recommender uncertainty to ranking performance","Recommenders can self-assess: LiDu predicts ranking quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":2023,"prompt_tokens":925,"completion_tokens":1098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1034}},"tokens_in":541,"tokens_out":1098,"duration_ms":10471,"temperature":1.0,"reasoning_tokens":1034,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:56:35.817986+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Synthesize a recommender where item scores share common noise factors, so pairwise comparisons are strongly dependent; compute LiDu and NDCG@K over many train/test splits and check whether the negative correlation persists when the true list likelihood differs sharply from the product approximation. If the correlation becomes positive or vanishes in a regime the paper claims LiDu generalizes to, the central claim would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SMV query performance prediction baseline LiDu must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NQC query-drift baseline used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the W-Graph retrieval coherency baseline used for comparison."}],"review_version":1}