{"id":"0d52675e-ba41-4895-b2f3-dda269344329","arxiv_id":"2607.20655","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A monthly closed-loop RL system for ranking sales leads, using position- and speed-weighted conversion rewards and a listwise GRPO-style loss, reports +7.9% NDCG@K offline and +4.7%/+8.7% conversion lift in a 160-day production A/B test.","lead":"SalesLoop uses live conversion outcomes from deployed sales teams to retrain a lead-ranking model each month, weighting rewards by where leads appeared in the ranking and how fast they converted. A 160-day production test at an electric-vehicle company claims a cumulative lift in signed deals of roughly 5–9% over a static baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward in Eq. 2 uses the deployed model's own ranks, so conversion labels are confounded with follow-up effort; without propensity correction or exploration, the offline gains and A/B 'compounding' may reflect self-reinforcement rather than improved ranking.","rationale":"The reader's weakest assumption is the same one I find most load-bearing: the reward's dependence on the model's own ranks creates a feedback bias. This is not an ad hominem or a consensus dispute; it is an internal inconsistency with the paper's own Gap 1 admission that conversion labels are confounded by human effort. The production A/B is a genuine strength and could survive the concern if newly surfaced leads convert well; therefore I do not move the verdict to REJECT. But the paper's offline claims are not currently interpretable, and the A/B's 'compounding' mechanism is confounded. Since the reader already marked CONDITIONAL, no change. Secondary issues (p=0.047 fragility, zero-positive batches) are real but secondary; the reward endogeneity is the structural weakness.","tokens_in":15331,"tokens_out":16735,"duration_ms":165418,"concrete_test":"Re-analyze the treatment A/B logs month by month: for each model update, split the treatment Top-K into (a) leads that were also in the previous iteration's Top-K and (b) newly surfaced leads (in Top-K now, below K before). Compute their 30-day conversion rates and compare with the same split for the static control. If newly surfaced treatment leads do not convert at a rate at least as high as control's newly surfaced leads—or if nearly all treatment lift comes from group (a)—then the reward is reinforcing the prior ranking rather than finding new value. To fully resolve, run a follow-up deployment with a 5% random holdout outside Top-K to obtain unbiased counterfactual labels and re-train with propensity-weighted rewards; the offline gains should replicate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"R_i = y_i·g(r_i)·v(τ_i) (Eq. 2) makes the learning signal a function of the very ranking f_{θ_{t-1}} being updated. Since only the Top-K leads receive specialist follow-up (Stage 1, §3.2), y_i is not an unbiased label: a lead converts partly because the previous model ranked it high. The paper's own Gap 1 (§1) acknowledges exactly this confound in historical labels, but the reward and the offline protocol never correct for it. In the offline benchmark (§4.2), held-out labels were generated under the static f_{θ_0}; leads below K have unobserved counterfactuals, so a re-ranked model's P@K/NDCG cannot be evaluated without propensity weighting. In the A/B test, the treatment model is rewarded for re-ranking leads it had already surfaced, which can amplify its prior decisions. The 'compounding' temporal divergence (§4.3.2) is therefore ambiguous: it may indicate genuine discovery of high-intent leads, or it may indicate a self-fulfilling loop that merely consolidates the old model's biases. No exploration, no inverse-propensity weighting, and no counterfactual analysis is provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SalesLoop is a closed-loop learning framework for sales-lead ranking. It proposes a performance-aware reward (Eq. 2) that weights conversion outcomes by the rank assigned by the deployed model and by conversion latency, and a 'Discriminative GRPO' objective (Eqs. 5--6) that matches a reward-derived softmax target to the model's score softmax. The authors report offline gains over static baselines (NDCG@K +7.9%, P@K +15.8% at K=10,000) and a 160-day production A/B test across two provincial markets with 280 sales specialists, claiming +4.7% (p=0.047) and +8.7% (p=0.002) cumulative lift in lock-in conversions, with the gap widening over time. The paper interprets this as evidence of a compounding feedback loop that adapts to delayed, sparse conversion signals.","tokens_in":15677,"tokens_out":6107,"duration_ms":58016,"significance":"If the central claims hold, the contribution is practically significant: long-horizon production evidence of an online ranking loop with delayed feedback is rare in the learning-to-rank literature, and the proposed reward is simple and interpretable. The paper is also honest in listing limitations and in not over-claiming that the objective is a literal GRPO transplant. However, the paper's core interpretation—that the gains reflect improved ranking quality rather than a self-reinforcing loop—is not yet supported, and the offline protocol and statistical analysis are under-specified. The strengths are the real deployment scale and the clear separation of offline, ablation, and online evidence; those strengths are weakened by unresolved confounding and missing reproducibility details.","major_comments":[{"comment":"The learning signal is endogenous to the model being trained. R_i = y_i·g(r_i)·v(τ_i) uses the rank r_i produced by the deployed f_{θ_{t-1}}, and y_i is only observed after follow-up effort that was allocated according to that same rank. A lead converts partly because the model surfaced it, so the reward rewards the model for repeating its own prior choices. The paper itself states Gap 1 — that historical labels are confounded by human effort — but neither the reward design nor the offline protocol applies inverse-propensity weighting, exploration, or a counterfactual analysis to break this loop. Consequently, the 'compounding' divergence in §4.3.2 and the offline gains in Table 2 are also consistent with a self-reinforcing feedback effect rather than an improved estimate of lead quality. Please provide at minimum an effort/propensity correction or a diagnostic that separates genuine qua","section":"§3.3, Eq. (2); §4.2; §4.3.2"},{"comment":"The offline benchmark protocol is under-specified to the point of non-reproducibility. Table 2 reports SALESLOOP evaluated after '1 iteration using held-out feedback', and Table 5 reports an ablation 'offline evaluation after 4 iterations', but the paper never states how ranks r_i are replayed offline, how many leads have observed conversion labels after the Top-K cutoff, or how multiple iterations are simulated without new deployment data. The reader cannot tell whether P@K/NDCG are computed on all leads or only on those that received follow-up, and whether the 4-iteration result is a genuine multi-round simulation or repeated fitting of the same static labels. Please specify the exact data-generation procedure for each iteration, including how unobserved counterfactual outcomes for non-surfaced leads are handled.","section":"§4.2, Tables 2 and 5"},{"comment":"The statistical support for the headline lift is thin. The evidence is a weekly paired t-test on cumulative conversions (p=0.047 and p=0.002), with no confidence intervals, no adjustment for multiple weekly looks or autocorrelation in cumulative series, and no pre-registered analysis plan. Province A's p is marginal, and its monthly lifts are negative in Dec, Jan, and Feb and near zero in Apr; the 'compounding' claim rests on an eyeballed regime change in Fig. 4 rather than a formal trend or interaction test. Please report effect sizes with confidence intervals, a pre-specified or corrected test, and a formal test for divergence (e.g., treatment×time interaction or segmented regression).","section":"§4.3.1, Table 3; §4.3.2–4.3.4"},{"comment":"The 'Discriminative GRPO' is not a policy-gradient objective in the usual sense: there is no sampling, no importance ratio, and no exploration term; Eq. (6) is a deterministic KL/cross-entropy matching between a reward-derived softmax target and the model's score softmax. This may be a valid listwise loss, but calling it RL and RLPF — and ascribing the online gains to reinforcement learning — requires a clearer statement of what is being optimized and why group-relative normalization is necessary. As written, the loss has no mechanism for exploring new rankings; this weakens the claim that the closed loop 'discovers' high-intent leads (Abstract, §4.5) rather than merely re-weighting the current ranker's outputs.","section":"§3.4, Eqs. (5)–(6)"},{"comment":"The A/B unit is the sales specialist, not the lead. The paper reports that specialists were randomly assigned, but it does not describe how incoming leads are routed to treatment vs. control specialists. If routing depends on territory, shift, or specialist load, the treatment and control groups may receive different lead streams; the shared seasonality in Fig. 5 does not establish balance. Please report lead volumes and lead-feature distributions per group, or describe the randomization/assignment mechanism.","section":"§4.3.1, A/B configuration"}],"minor_comments":[{"comment":"Please clarify the meaning of the rank r_i in a monthly-cycle, daily-deployment setting. Is it the lead's rank within the day it arrived, within the monthly batch, or the rank at the moment of conversion? This affects the reward magnitude and the offline replay.","section":"§3.3, Eq. (2)"},{"comment":"There is an inconsistency in the reported 7-day Top-10% recall: Table 6 says mean 44.1%, while Fig. 7 shows mean 43.7%. Please harmonize.","section":"§4.5, Table 6 vs. Fig. 7"},{"comment":"The column header 'June 2025' appears to be a typo; the test window is December 2025–June 2026.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about reward endogeneity largely lands. In my view this is the central issue to resolve: the A/B result may be a valid system-level comparison, but the paper's mechanism claims ('compounding', 'discovery of high-intent leads') are not yet disentangled from a self-reinforcing loop. The offline protocol also needs to be specified precisely before the paper can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SalesLoop is a serious industrial lead-ranking system with a real 160-day A/B test, and the authors clearly identify three plausible gaps that separate offline accuracy from online value. The central claim, however, rests on a feedback loop that can reinforce the model's own prior decisions instead of discovering genuinely better rankings, and the paper never fully addresses that confound. The p-values and a numerical inconsistency make me cautious about the headline lift.\n\nWhat's new and useful: the framing of offline-online mismatch, pointwise-listwise misalignment, and temporal drift is clear and practical. Adapting GRPO to a listwise discriminative loss (group-relative advantage plus KL to softmax targets) is a reasonable combination; the components are known, but the application to delayed, sparse conversion feedback is not common. The production A/B test across two provinces with 280 specialists is a rare long-horizon deployment, and the ablation cleanly shows the listwise loss and position gain matter. The per-specialist normalization is a good check.\n\nThe soft spots are real and load-bearing. The reward in Eq. 2 uses r_i from the deployed model, and since only the top-K leads receive follow-up, conversion labels are missing for everything below K, with reward set to zero. The model is therefore trained to re-rank leads it already surfaced. There is no exploration, no inverse propensity weighting, no counterfactual correction. The paper's own Gap 1 acknowledges the same confound in historical labels, but the reward reintroduces it. The compounding divergence in Figure 4 could just be a self-fulfilling loop. That is not a minor issue; it changes what the A/B test actually proves. The offline benchmark is also under-specified: how exactly is the one-iteration feedback loop replayed on historical data? And the A/B statistics are thin: p=0.047 after multiple tests, weekly paired t-tests on autocorrelated data, no confidence intervals. Finally, Section 4.5 reports 4,600 incremental high-intent leads per day over 103 days with a 4.66% conversion rate, which implies roughly 22,000 incremental conversions, yet the paper reports 7,220. That does not add up.\n\nThis paper is worth a serious referee: it is a genuine large-scale deployment and a good case study for feedback-loop bias in industrial ML. A revision should either correct for the selection bias or explicitly scope the claim as adaptive re-ranking under deployment bias, fix the statistical analysis, and resolve the numbers discrepancy. I'd bring it to a reading group to spark discussion, but I would not cite it in its current form.","headline":"A plausible industrial closed-loop ranking system, but the A/B lift is confounded by self-reinforcement and the stats are too thin to trust the headline.","tokens_in":16117,"tokens_out":4087,"would_cite":false,"duration_ms":38950,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SalesLoop closes the loop between deployed lead rankings and real conversion outcomes, reporting +4.7% to +8.7% lift in a 160-day A/B test.","keywords":["lead ranking","CRM","reinforcement learning","performance feedback","listwise optimization","GRPO","A/B testing","conversion prediction"],"falsifier":"Run the same feedback loop in an offline replay where the ranks r_i are assigned randomly or held out instead of by the model f_θ; if SalesLoop's advantage over static baselines disappears or reverses under random rank assignment, the position-weighting term is exploiting the model's own decisions rather than learning lead quality. Alternatively, a production A/B test with outcome-only rewards (no g(r)) that matches the full-reward treatment's lift would show position weighting is unnecessary.","tokens_in":1581,"feed_emoji":"📈","tokens_out":1848,"duration_ms":41738,"temperature":0.7,"pith_summary":"The paper claims that high offline accuracy in lead-ranking models often fails to translate to production because of three structural mismatches: offline metrics do not reflect online conversion behavior, pointwise training does not optimize the listwise Top-K ranking that actually drives sales, and the lead distribution drifts over time. SalesLoop addresses all three by closing a feedback loop: deploy the ranker, observe which leads convert and how fast, convert those observations into a position- and velocity-weighted reward, and retrain the model monthly with a listwise objective. Offline, this yields +7.9% NDCG@K and +15.8% P@K over the strongest static baseline; in production, a 160-day A/B test across two provincial markets with 280 specialists and 16.5M leads shows cumulative lock-in lifts of +4.7% and +8.7%, both statistically significant, with the treatment advantage growing over time. A sympathetic reader would care because the result suggests a practical, general recipe for continuously improving ranking systems from deployment feedback rather than relying on static historical labels.","feed_headline":"Live feedback loop lifts sales conversions 4.7–8.7%","feed_subtitle":"160-day A/B test on 16.5M leads shows compounding gains when the ranker is retrained on real conversion outcomes.","key_machinery":"The central mechanism is the pair: (1) a performance-aware reward, R_i = y_i · g(r_i) · v(τ_i), where g(r_i) = 1/log2(r_i + 1) encodes logarithmic attention decay with rank and v(τ_i) = 1 - τ_i/T rewards faster conversions, turning sparse binary labels into a dense, listwise-relevant signal; and (2) Discriminative GRPO, which treats each training batch as a group, standardizes rewards into advantages A_i via Eq. 5, and optimizes L_list = D_KL(softmax(A/γ) || softmax(s)) via Eq. 6, coupling all leads in a batch so that changing one score shifts the relative probability mass. A BCE regularization term L_BCE(s, y) anchors calibration, and monthly warm-start iterations with a reduced learning ra","core_discovery":"SalesLoop establishes that a discriminative lead-ranking model can be continuously improved by optimizing a listwise objective against performance-aware rewards built from real conversion outcomes, instead of retraining on static labels. The reward R_i = y_i · g(r_i) · v(τ_i) gives credit only to converted leads, gives more credit to conversions that appear near the top of the deployed ranking, and gives more credit to faster conversions. Discriminative GRPO standardizes rewards within each training batch to group-relative advantages and minimizes the KL divergence between the advantage distribution and the model's score distribution, while a BCE regularization term preserves calibration. Th","pith_inferences":["The reward depends on the rank the deployed model itself assigned, so the training signal is partly self-referential: a lead under-ranked by the model gets less follow-up, which can suppress its conversion and lower its reward, potentially reinforcing the model's own errors. Removing or randomizing the rank weight would test whether the observed lift reflects true quality learning.","The roughly linear growth of the treatment-control gap could be a genuinely compounding learning effect, but it could also be a slowly converging treatment effect; longer observation or per-iteration learning curves would sharpen the distinction.","Because the position gain g(r_i) is tied to the operational capacity K, the reported quantitative lifts at K = 10,000 may not transfer to deployments with different capacity without re-tuning γ and α.","The authors propose faster feedback signals (test drives, showroom visits) as future work; a natural testable extension is whether weekly iterations using those intermediate signals outperform the monthly loop in both speed and final lift."],"forward_implications":["Offline, SalesLoop improves NDCG@K by +7.9% and P@K by +15.8% over the strongest static baseline at K = 10,000.","In the 160-day production A/B test, cumulative lock-in conversions rose by +4.7% (p = 0.047) in one market and +8.7% (p = 0.002) in the other, with per-specialist gains of +1.6 and +2.7 conversions.","The treatment-control gap grows roughly linearly after crossover, indicating a compounding, non-saturating advantage within the observation window.","The lift is largest when capacity is tight: +21.4% P@K at K = 1,000, diminishing to +1.9% at K = 100,000, showing the listwise objective matters most when only a small top fraction is surfaced.","The underlying ranker already delivers 44.1% Top-10% recall (4.4× random) and identifies incremental high-intent leads that convert at 2.3× the specialist-flagged baseline, with no degradation over 103 days."],"fun_headline_variants":["Closed-loop ranker boosts conversions by up to 8.7%","Ranking model that learns from outcomes lifts sales 4.7–8.7%","Reinforcement learning for lead ranking: +8.7% conversions","Live performance feedback retrains ranker, lifting conversions 4.7–8.7%"],"cache_read_input_tokens":17408,"weakest_assumption_plain":"The reward for each lead uses the rank that the deployed model itself assigned to that lead, so the training signal is partly a product of the model's own prior decisions; if this self-referential feedback is not separated from true lead quality, the claimed learning signal may partly be the model confirming its own ranking.","fun_headline_variants_meta":{"raw":{"variants":["Closed-loop ranker boosts conversions by up to 8.7%","Ranking model that learns from outcomes lifts sales 4.7–8.7%","Reinforcement learning for lead ranking: +8.7% conversions","Live performance feedback retrains ranker, lifting conversions 4.7–8.7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1563,"prompt_tokens":760,"completion_tokens":803,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":715}},"tokens_in":504,"tokens_out":803,"duration_ms":9005,"temperature":1.0,"reasoning_tokens":715,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:42:03.471736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same feedback loop in an offline replay where the ranks r_i are assigned randomly or held out instead of by the model f_θ; if SalesLoop's advantage over static baselines disappears or reverses under random rank assignment, the position-weighting term is exploiting the model's own decisions rather than learning lead quality. Alternatively, a production A/B test with outcome-only rewards (no g(r)) that matches the full-reward treatment's lift would show position weighting is unnecessary.","supporting_citations":[],"review_version":1}