{"id":"1f4f30a4-9bb1-496d-8378-7b80dfb96c58","arxiv_id":"2608.12778","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DrEM is a dual-side robust ensemble ranking framework that uses a shared logit-space noise model to denoise proxy preferences and stabilize inputs, improving ranking quality under upstream pxtr prediction noise.","lead":"Industrial video recommendation ranks candidate videos by fusing several predicted user behavior probabilities, but those predictions themselves carry noise. This paper proposes a training method that adjusts for that noise on both the supervision and input sides, reporting small but consistent gains in online A/B tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 is false: for overestimated flip probabilities the robust loss Eq. (3) moves the expected risk farther from the clean risk than the basic pairwise loss, so the paper's theoretical guarantee of superior robustness under estimation error is invalid.","rationale":"Even if Assumption 1 held exactly, Theorem 1's proof mistake invalidates the central theoretical claim that the robust loss is always closer to the clean risk. The denominator 1−2ε̂ amplifies overestimation, so the method could degrade in practice when the variance estimator overestimates flip probabilities. The reader's weakest assumption (Assumption 1) is also important, but the theorem flaw is more decisive: it is a logical error independent of data. The empirical gains in Tables 1-2 may still hold, but the theoretical support needs major repair (either a corrected theorem with explicit bounds or a repositioned claim). Since the reader already recommended CONDITIONAL, this concern reinforces that verdict rather than changing it.","tokens_in":16542,"tokens_out":10869,"duration_ms":89386,"concrete_test":"Numerically evaluate the counterexample: set ε=0.1, ε̂=0.4, ℓ(s_i,s_j)=−log σ(s_i−s_j) with s_i−s_j=1, and compute the expected risk of the robust loss using Eqs. (2) and (3) versus the basic loss. If |E[L_rob]−ℓ_clean| ≥ ε|ℓ_rev−ℓ_clean|, Theorem 1's universal claim is refuted. Then sweep ε̂ over (0,0.5) for fixed ε and identify the non-empty region where the robust loss is worse than the basic loss, confirming the missing condition for the theorem.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4, Theorem 1 claims that for any ε_ij, ε̂_ij ∈ (0,1/2), the expected risk of the robust loss (Eq. 3) is closer to the clean risk than the basic pairwise loss, with monotone improvement as |ε̂−ε| decreases. The proof defines c(ε̂) = (1−ε−ε̂)/(1−2ε̂) and shows c(ε̂) > c(0) = 1−ε and c'(ε̂)>0. This only shows the clean-term coefficient is larger, not that it is closer to 1. In fact, for ε̂ > ε, c(ε̂) > 1, so the robust loss overshoots. The expected risk difference from clean risk is |c(ε̂)−1|·|ℓ_clean−ℓ_rev| = |ε̂−ε|/(1−2ε̂) · |ℓ_clean−ℓ_rev|, while for the basic loss it is ε·|ℓ_clean−ℓ_rev|. Taking ε=0.1, ε̂=0.4 gives ratio 0.3/0.2 = 1.5 versus 0.1: the robust risk is 15× farther from the clean risk. Thus the universal superiority guarantee in Section 3.2 and the abstract's 'prove that the robust loss remains superior under flip probability estimation error' are unsupported. Theorem 1 is false as stated; no bound on estimation error or restriction to the safe region (e.g., ε̂ < 2ε/(1+2ε) for BPR-type losses) is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the ensemble ranking stage of an industrial short-video recommender, where upstream multi-task predictions (pxtrs) are used both as input features and as proxy supervision. It models upstream prediction noise as additive zero-mean Gaussian in logit space and proposes DrEM, which (i) corrects the pairwise ranking loss by reweighting forward and reverse terms with estimated preference-flip probabilities and (ii) regularizes the model to be stable under noise-consistent perturbations on the input pxtrs. The authors provide theorems for the flip probability and noise variance estimation, report offline GAUC experiments on an industrial dataset under injected perturbation strengths, and present 7-day online A/B results with statistically significant gains on both EMER and EASQ backbones.","tokens_in":16825,"tokens_out":7894,"duration_ms":75246,"significance":"If the theoretical claims held, DrEM would be a practically valuable and reasonably principled solution to a real industrial problem. The strongest parts are the clear problem decomposition (supervision-side versus feature-side), the pair-level and item-level adaptation through a shared noise model, and the unusually complete empirical evaluation, including online A/B tests with p<0.005 and a flip-probability-stratified analysis (Figure 3) that gives a checkable mechanism for the gains. The paper does not provide code or public data, and the central theoretical guarantee requires correction, but the empirical contribution is solid and the proposed framework is coherent.","major_comments":[{"comment":"The theorem as stated is false. The proof defines c(\\hat{\\varepsilon}_{ij})=(1-\\varepsilon_{ij}-\\hat{\\varepsilon}_{ij})/(1-2\\hat{\\varepsilon}_{ij}) and shows c(\\hat{\\varepsilon})>c(0) and c'(\\hat{\\varepsilon})>0. However, the expected risk of the robust loss differs from the clean risk by |c(\\hat{\\varepsilon})-1| \\cdot |\\ell_clean-\\ell_rev|, and the basic loss corresponds to \\hat{\\varepsilon}=0 with deviation \\varepsilon \\cdot |\\ell_clean-\\ell_rev|. For \\hat{\\varepsilon}>\\varepsilon, we have c(\\hat{\\varepsilon})>1 and the robust loss overshoots; for example, with \\varepsilon=0.1 and \\hat{\\varepsilon}=0.4, the deviation is 1.5 \\cdot |\\ell_clean-\\ell_rev| versus 0.1 \\cdot |\\ell_clean-\\ell_rev| for the basic loss, i.e., 15 times farther from the clean risk. Thus the universal superiority guarantee claimed in the abstract and in Section 3.2 is unsupported. Please either restrict the claim to a safe region (e.g., \\hat{\\varepsilon} below the crossing point where |c(\\hat{\\varepsilon})-1|=\\varepsilon), prove an explicit bound with estimation-error dependence, or replace Theorem 1 with a statement that only asserts monotone improvement as \\hat{\\varepsilon} approaches \\varepsilon from below.","section":"Section 4, Theorem 1 and Section 3.2"},{"comment":"The variance estimator is not guaranteed to be nonnegative. If in a bucket the empirical behavior rate exceeds the predicted pxtr rate, then |\\Phi^{-1}(E_B[p_i]) / \\Phi^{-1}(E_B[y_i])| < 1 and \\hat{\\sigma}_k^2 is negative; no clipping or sign-restricted estimator is specified. Since \\hat{\\sigma}_k^2 feeds both the flip-probability formula (Eq. (10)) and the perturbation sampler (Section 3.3), this is a load-bearing gap. Please add a nonnegative estimator or explicitly define the regime in which the formula applies, and discuss the behavior under systematic pxtr miscalibration.","section":"Section 4, Theorem 3, Eq. (12)"},{"comment":"The paper provides no diagnostic for the assumed additive zero-mean Gaussian, independent noise on the industrial pxtr logits. The justification via asymptotic normality of parametric models does not transfer automatically to the upstream multi-task model, and any bias or cross-item correlation would be absorbed by the Theorem 3 estimator and then mis-specify Eq. (10) and the perturbation distribution. Because the same pxtr-behavior discrepancy is used both to estimate the noise and to correct it, this is not a cosmetic concern. Please add an empirical validation of Assumption 1 on the actual data (e.g., residual analysis or calibration checks) or explicitly state this as a limitation with a sensitivity analysis.","section":"Section 4, Assumption 1"}],"minor_comments":[{"comment":"The notation in the proof is confusing: 'E[r_i]=E[\\sigma(z_i+\\xi_i)]' mixes the latent clean value r_i with the observed noisy logit; the derivation should be written in terms of p_i, z_i^*, and the conditional expectation of y_i.","section":"Section 4, Theorem 3"},{"comment":"The probit approximation \\sigma(x)\\approx\\Phi(\\lambda x) with \\lambda=\\sqrt{\\pi/8} is used before it is explicitly defined; state it before Theorem 3 for readability.","section":"Eq. (12)"},{"comment":"The evaluation perturbation \\tilde{z}_i = z_i + \\alpha \\epsilon_i uses a sampled Gaussian perturbation, but it is not clear whether \\epsilon_i is drawn with the estimated \\hat{\\sigma}_i^2 or with unit variance; please clarify the relation between \\alpha and the estimated noise scale.","section":"Section 5.1"},{"comment":"With many online metrics and two backbones, it would be helpful to state whether any multiple-testing correction was applied, even if all individual p-values are below 0.005.","section":"Table 2"},{"comment":"The explanation that high-flip-probability buckets for dense tasks are dominated by statistical noise would be more checkable if the number of pairs per bucket were reported.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid industrial paper whose main empirical contribution appears sound, but the theoretical centerpiece is currently overstated. The false Theorem 1 can be fixed by restricting the guarantee to a safe estimation-error region or by proving a different statement, and the abstract and Section 3.2 must be revised accordingly. I would not reject on this basis given the strength of the offline and online evaluations, but I would insist on the correction. I also note the absence of code or public data release, which limits independent verification beyond the reported metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2608.12778: it is a serious industrial paper with a genuinely new idea, and its main theory claim is wrong as stated. The empirical core may still survive, but the paper needs careful reading.\n\nWhat is actually new: DrEM is the first ensemble-ranking method I have seen that treats upstream pxtr prediction noise on both the supervision and feature sides under one shared heteroscedastic logit-space noise model. The supervision-side loss reweights each pair by estimated flip probability; the feature side samples item-specific perturbations and enforces ranking consistency only where the perturbation did not flip the pair's preference. That combination is sensible and the experiments back it: offline GAUC gains on two backbones across seven tasks, ablations showing both sides help, and online A/B gains on Follow (+1.197%) and Comment (+1.388%) over EMER at p<0.005. For an industrial ranking system those are meaningful numbers.\n\nNow the soft spots, in proportion. First, Theorem 1 is false as stated. The proof shows c(eps_hat) > c(0) and monotone increasing, but that only means the clean-term coefficient is larger. It does not imply the robust risk is closer to the clean risk. When eps_hat > eps, c(eps_hat) exceeds 1 and the loss over-corrects. With eps=0.1 and eps_hat=0.4, the robust risk is fifteen times farther from the clean risk than the basic loss. The abstract's claim that the robust loss 'remains superior under flip probability estimation error' is unsupported. To salvage the theorem the authors need either a bound on estimation error or a safe-region condition (e.g., eps_hat <= 2eps/(1+2eps) for BPR-style losses). This is not a minor wording issue; it is the paper's central theoretical guarantee.\n\nSecond, the noise variance estimator (Theorem 3) leans hard on Assumption 1—additive zero-mean Gaussian noise in logit space, independent across items—with no diagnostic on real pxtr data. If the actual noise is biased or correlated, the flip probabilities and perturbations are mis-targeted, and the estimator will absorb calibration error. The probit approximation also has no error bound. This is a typical industrial-modeling limitation, but the paper should be upfront that the method's validity is conditional on a fitted estimate of a strong distributional assumption.\n\nThird, no code or data. The offline evaluation is on proprietary data with injected Gaussian perturbations, so external reproducibility is nil. That is not disqualifying for an industry paper, but it raises the bar for the theory to be airtight, and it currently is not.\n\nWho is this for? Anyone building multi-stage ranking with upstream model outputs used as both features and labels. The paper deserves a serious referee even though I would not accept it as is. Send it to review with the instruction to fix or remove Theorem 1 and add robustness diagnostics for the noise model. If the A/B results are representative, this is a useful template; the theory just needs to stop claiming more than it proves.","headline":"Useful industrial recipe for dual-side noise correction, but Theorem 1 overclaims; the empirics may still hold.","tokens_in":17423,"tokens_out":2264,"would_cite":true,"duration_ms":21595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video recommendation ranking can be made robust to noisy user-preference predictions by correcting both supervision and features under one shared noise model.","keywords":["Video Recommendation","Ensemble Ranking","Prediction Noise","Robust Learning","pairwise preference flip","risk-denoising loss","consistency regularizer","pxtr"],"falsifier":"On logged production data where an item's pxtr predictions can be compared with its long-run observed behavior in narrow prediction buckets, compute the empirical logit residuals $z_i - \\logit(\\bar{y}_i)$. If those residuals have nonzero mean, cross-item correlation, or variance that the probit estimator $\\hat{\\sigma}_k^2$ cannot reproduce, then Eq. (10) mis-estimates the flip probabilities; a direct test would inject noise of known non-Gaussian shape into held-out pxtrs and check whether the robust loss still recovers the clean-risk behavior predicted by Theorem 1.","tokens_in":16300,"feed_emoji":"🎬","tokens_out":12762,"duration_ms":103453,"temperature":0.7,"pith_summary":"Industrial video recommendation uses an upstream multi-task model to predict several user preferences (pxtrs), then an ensemble ranking model fuses them into a single score. This paper argues that those pxtrs are not clean signals: their prediction noise corrupts both the pairwise preference labels derived from pxtr comparisons and the feature inputs themselves. The proposed DrEM models the noise as additive zero-mean Gaussian in logit space with per-item variance, reweights each pair's loss with an estimated flip probability on the supervision side, and trains the ranking model to stay consistent under noise-sampled perturbations on the feature side. The paper proves that the corrected loss stays closer to the clean risk than the basic pairwise loss whenever estimated flip probabilities lie in $(0,1/2)$, and reports offline and online gains, with the largest gains on sparse interaction tasks where flip probabilities are highest.","feed_headline":"One noise model fixes both sides of noisy video ranking","feed_subtitle":"Correcting both labels and inputs with one noise model lifts online engagement metrics.","key_machinery":"The shared engine is the logit-space additive Gaussian noise model $z_i = z_i^* + \\xi_i$, $\\xi_i \\sim \\mathcal{N}(0,\\sigma_i^2)$, independent across items. On the supervision side it yields the maximum-likelihood flip probability $\\hat{\\varepsilon}_{ij} = \\Phi(-(z_i-z_j)/\\sqrt{\\sigma_i^2+\\sigma_j^2})$ (Theorem 2), which enters the risk-denoising robust loss $L_{\\text{rob}} = \\sum_{(i,j)\\in P}\\frac{(1-\\hat{\\varepsilon}_{ij})\\ell(s_i,s_j)-\\hat{\\varepsilon}_{ij}\\ell(s_j,s_i)}{1-2\\hat{\\varepsilon}_{ij}}$ and makes the expected loss equal to the clean risk when $\\hat{\\varepsilon}_{ij}=\\varepsilon_{ij}$ (Theorem 1). On the feature side the same variances sample perturbations $\\tilde{z}_i = z_i + \\epsilon_i$, $\\epsilon_i\\sim\\mathcal{N}(0,\\sigma_i^2)$, and the preference-preserving regularizer $L_{\\text{cons}}$ applies a pairwise order-consistency loss only to pairs whose order is unchanged by the perturbation, preventing conflict with the main ranking objective. The two sides are coupled by Theorem 3's probit bucketing estimator $\\hat{\\sigma}_k^2 = \\lambda^{-2}\\left[\\left(\\Phi^{-1}(E_{B_k}[p_i])/\\Phi^{-1}(E_{B_k}[y_i])\\right)^2 - 1\\right]$ with $\\lambda=\\sqrt{\\pi/8}$, which extracts per-bucket variance from the systematic gap between predicted pxtrs and observed feedback.","core_discovery":"The central claim is that the noise in upstream pxtr predictions can be summarized by one quantity per item, a logit-space noise variance $\\sigma_i^2$, and that this single summary drives two aligned corrections. Writing the observed pxtr logit as $z_i = z_i^* + \\xi_i$ with $\\xi_i \\sim \\mathcal{N}(0,\\sigma_i^2)$ independent across items, the probability that a preference pair flips is approximately $\\hat{\\varepsilon}_{ij} = \\Phi(-(z_i-z_j)/\\sqrt{\\sigma_i^2+\\sigma_j^2})$. Substituting this pair-specific estimate into the risk-denoising loss $L_{\\text{rob}}$ cancels the reverse-loss contamination: with exact estimates the risk equals the clean risk, and with any estimates in $(0,1/2)$ the robust loss is strictly closer to the clean risk than the basic pairwise loss, monotonically improving as the estimate sharpens. The same $\\sigma_i^2$ values feed the feature-side regularizer, which adds sampled perturbations to the pxtr logits and enforces ranking consistency only on pairs whose order survives perturbation. The variances themselves are estimated by a bucketing probit method that compares average predicted pxtr with average observed behavior within narrow prediction buckets, so the whole scheme needs no extra labels beyond the posterior feedback industrial systems already log.","pith_inferences":["If correct, the same dual-side correction scheme transfers to any multi-stage system whose upstream outputs are reused as both features and supervision, such as ad scoring or LLM-as-judge pipelines, whenever those outputs are noisy.","The preference-preserving filtering rule, regularize only pairs whose order survives perturbation, is a generic recipe for consistency training under input noise and could be applied to other pairwise ranking objectives.","A direct test of the Gaussian assumption would be to compare DrEM's bucketing variance estimates with empirical residual variances computed from logged pxtrs and observed behavior; the theory predicts they should match closely, so large mismatches would indicate mis-specified noise.","The stratification result in Figure 3 suggests a monitoring diagnostic for production: the GAUC gain over the base model should increase with estimated flip probability; a deviation from that monotone pattern would flag noise-model miscalibration."],"forward_implications":["DrEM is a plug-in module: the backbone ranking model, the upstream pxtr model, and the final ranking formula are all unchanged; the only added cost is one perturbed forward pass during training.","Tasks with sparse user interactions (follow, comment, forward) should gain the most, since their pxtrs carry larger noise variance and hence higher flip probabilities.","Even a rough flip-probability estimate lying anywhere in $(0,1/2)$ makes the robust loss strictly closer to the clean risk than the basic pairwise loss, so the method degrades gracefully when the variance estimator is imperfect.","The two corrections are additive because they address two independent propagation paths of the same noise; the full method outperforms either side alone at every tested perturbation strength.","Because the variance estimator needs only posterior user behavior, the whole pipeline is deployable with the feedback data industrial recommendation systems already log."],"supporting_citations":[{"why":"Supplies the risk-correction principle that Section 3.2 instantiates as the robust pairwise loss.","marker":"[5]"},{"why":"Provides the noisy-label correction framework that the risk-denoising loss generalizes to pairwise flips.","marker":"[17]"},{"why":"Justifies the Gaussian logit-space noise assumption through asymptotic normality of parametric estimators.","marker":"[39]"},{"why":"Defines the EMER backbone whose pxtr-comparison pairwise supervision DrEM corrects.","marker":"[10]"},{"why":"Defines the EASQ backbone, whose questionnaire alignment already mitigates part of the supervision-side noise.","marker":"[15]"},{"why":"Is the feature-side perturbation baseline whose global-scale perturbations DrEM's item-level sampling improves on.","marker":"[19]"},{"why":"Is the adversarial Gaussian augmentation baseline that DrEM's consistency regularizer is compared against.","marker":"[27]"},{"why":"Is the sampled-softmax supervision-side baseline used in the comparison.","marker":"[29]"},{"why":"Is the bounded-softmax supervision-side baseline used in the comparison.","marker":"[31]"}],"fun_headline_variants":["One noise variance fixes both label and feature noise in ranking","Single variance model corrects both sides of noisy video ranking","No extra labels: one noise estimate denoises ranking supervision and features","Unified noise parameter improves video ranking robustness without new labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hinges on the assumption that prediction noise is additive, zero-mean, independent across items, and Gaussian in logit space; if the real upstream noise is biased, correlated, or heavy-tailed, the flip probabilities and perturbation scales point at the wrong target.","fun_headline_variants_meta":{"raw":{"variants":["One noise variance fixes both label and feature noise in ranking","Single variance model corrects both sides of noisy video ranking","No extra labels: one noise estimate denoises ranking supervision and features","Unified noise parameter improves video ranking robustness without new labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000552,"raw_usage":{"total_tokens":2696,"prompt_tokens":1070,"completion_tokens":1626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":1557}},"tokens_in":686,"tokens_out":1626,"duration_ms":12400,"temperature":1.0,"reasoning_tokens":1557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:29:01.886866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On logged production data where an item's pxtr predictions can be compared with its long-run observed behavior in narrow prediction buckets, compute the empirical logit residuals $z_i - \\logit(\\bar{y}_i)$. If those residuals have nonzero mean, cross-item correlation, or variance that the probit estimator $\\hat{\\sigma}_k^2$ cannot reproduce, then Eq. (10) mis-estimates the flip probabilities; a direct test would inject noise of known non-Gaussian shape into held-out pxtrs and check whether the robust loss still recovers the clean-risk behavior predicted by Theorem 1.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the risk-correction principle that Section 3.2 instantiates as the robust pairwise loss."},{"cited_title":"Dhillon, Pradeep K","cited_arxiv_id":null,"evidence_quote":"Provides the noisy-label correction framework that the risk-denoising loss generalizes to pairwise flips."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the Gaussian logit-space noise assumption through asymptotic normality of parametric estimators."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the EASQ backbone, whose questionnaire alignment already mitigates part of the supervision-side noise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the feature-side perturbation baseline whose global-scale perturbations DrEM's item-level sampling improves on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the adversarial Gaussian augmentation baseline that DrEM's consistency regularizer is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the sampled-softmax supervision-side baseline used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the bounded-softmax supervision-side baseline used in the comparison."}],"review_version":1}