{"id":"88e8a176-bcb1-4520-9cab-a510f90f5869","arxiv_id":"2608.06792","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A three-phase post-training pipeline (LP-FFT-RFT) aligns a recommender foundation model with business metrics and outperforms single-phase fine-tuning and direct reward-model ranking in offline and online experiments.","lead":"This paper proposes a three-phase recipe for adapting a recommender foundation model to business goals: linear probing, full fine-tuning, then reinforcement fine-tuning against a learned reward model. It reports offline gains and production A/B lift over a conventional ranking baseline at NAVER WEBTOON.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RFT stage is formally underspecified: Section 3.2.2 and Algorithm 1 never define π_θ(a|u) as a normalized distribution over a fixed action set, so the GRPO/DPO policy ratios and KL term in Eq.","rationale":"The reader's weakest_assumption already identifies the undefined policy distribution in Section 3.2.2 and Algorithm 1 as the key risk. My reading agrees: the formal identity of the RFT stage depends entirely on π_θ(a|u) being a well-defined conditional distribution, and the paper never supplies that definition. This matters because the offline and online comparisons use the names 'GRPO' and 'DPO' to draw conclusions about continuous versus pairwise reward signals (Section 4.2.4); if the actual implemented objective is a different reward-weighted ranking loss, the scientific claim about alignment methods is not supported. However, the concern is not a demonstrated error: a reference implementation is provided, and a well-defined softmax policy could be present in code even though missing from the text. The online A/B result provides independent empirical support for the overall pipeline, but it does not by itself resolve the question of which objective produced the gains. Since the reader's verdict is already CONDITIONAL with moderate confidence, and the concern is exactly the condition that needs to be verified, I recommend keeping the verdict unchanged rather than escalating to rejection. The concrete test above would settle whether the concern lands: if the code defines a proper policy and the numbers reproduce, the central claim stands; if not, the verdict should move to UNVERDICTED or REJECT because the named algorithm is not what was evaluated.","tokens_in":11748,"tokens_out":6235,"duration_ms":57657,"concrete_test":"Inspect the reference implementation at https://github.com/webtoon/rec-fm-progressive-alignment and identify the exact code path for Phase 3: how π_θ(a|u), π_ref(a|u), the policy ratio, and the KL term are computed. Then rerun the 'GRPO w/ RM' row of Table 1 with a properly defined stochastic policy, e.g., softmax over the top-32 greedy candidates with a fixed temperature, and check whether Rank NDCG 0.463 and Funnel NDCG 0.637 are reproduced within ±0.002. As a control, also run the same GRPO configuration with ρ_k fixed to 1 for all k; if results do not change, the method is not actually using a policy ratio and the name 'GRPO' is misleading.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the progressive LP-FFT-RFT framework outperforms single-phase alternatives and that reward-based alignment beats using the reward model directly. The load-bearing step is the RFT stage. In Section 3.2.2, the policy ratio is ρ_k(θ)=π_θ(a_k)/π_ref(a_k), and Eq. 1 includes D_KL(π_θ(·|s)∥π_ref(·|s)). Algorithm 1 line 20 says 'Greedily select top-K candidate items A_K using π_θ(·|u)' and line 25 computes ρ_k from π_θ(a_k|u) and π_ref(a_k|u). Nowhere does the paper define π_θ(a|u) as a probability distribution over items. The serving policy head is trained with cross-entropy to predict whether a click occurs after impression; that is a binary-event probability, not a distribution over candidate actions. If π_θ is a softmax over the greedily selected top-32 set, then the candidate set is chosen deterministically by the current policy, so π_ref and π_θ are not defined on a common fixed action set as training progresses. If π_θ is defined over the full catalog, the KL term is intractable and no sampling or temperature is described. The same issue affects the DPO variant, which requires log π_θ(a_k). Consequently, Eq. 1 is not a well-defined GRPO expectation, and the offline comparison in Table 1 between 'GRPO w/ RM' and 'DPO w/o RM' cannot be attributed to the stated algorithms; the updates may reduce to a reward-weighted ranking loss. This is an internal-consistency concern, not a disagreement with the general practice of reward-based fine-tuning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-phase post-training pipeline for adapting a pretrained recommender foundation model to a specific serving surface. Phase 1 (LP) freezes the backbone and trains downstream heads; Phase 2 (FFT) unfreezes all parameters with a lower backbone learning rate; Phase 3 (RFT) applies GRPO or DPO using a reward model trained on a six-level engagement funnel with inverse propensity weighting. Offline experiments on a three-week production log report that the full pipeline (LP→FFT→GRPO) improves Rank NDCG and Funnel NDCG over one- and two-phase SFT baselines, and that aligning a click-trained policy with the reward model beats serving directly with the reward model. Online A/B tests report significant positive lifts over a non-foundation production control on in-session metrics.","tokens_in":12116,"tokens_out":7359,"duration_ms":57456,"significance":"If the method performs as reported, the paper's contribution is practically useful: it demonstrates a staged recipe for adapting a sequential recommender foundation model to a production surface, with an explicit separation between dense-signal adaptation and sparse business-metric alignment. The online A/B experiment against a conventional production model is a particularly valuable piece of evidence, and the reference implementation supports reproducibility. However, the central technical novelty—the RFT stage—is formally under-specified in the current manuscript, and the offline evaluation lacks the statistical rigor needed to support the claimed comparisons. The online results are encouraging, but the offline claims about the relative merits of GRPO/DPO and of reward-based alignment rest on undefined quantities and unquantified variance.","major_comments":[{"comment":"The policy ratio ρ_k(θ)=π_θ(a_k)/π_ref(a_k) and the KL term D_KL(π_θ(·|s)∥π_ref(·|s)) require π_θ(·|u) to be a normalized probability distribution over a fixed action set. The manuscript never defines this distribution. In Section 4.1 the serving policy head is described as predicting 'whether a click occurs after impression'—a binary event probability, not a distribution over candidate items. Algorithm 1 line 20 greedily selects top-K items 'using π_θ(·|u)', but if π_θ is a deterministic score, no stochastic policy exists; if π_θ is a softmax over the top-K set, the action set changes as the policy updates, so π_ref and π_θ are not defined on a common set; if π_θ is a softmax over the full catalog, the paper does not describe how the KL term is computed or how actions are sampled. Consequently, Eq. (1) is not a well-defined GRPO objective, and the gains attributed to 'GRPO w/ RM' in Table 1 cannot be unambiguously attributed to the stated algorithm. Please define the action distribution precisely (e.g., a softmax over a fixed candidate set with temperature) and adapt the algorithm and the KL term accordingly.","section":"Section 3.2.2, Eq. (1), Algorithm 1 lines 20–26"},{"comment":"The test set consists of 10% of users sampled in the same three-week window used for training, and the text does not state whether these users' interactions were excluded from the training set. If they were not excluded, the offline metrics are potentially inflated by training/test overlap, and the differences between configurations in Table 1 are confounded. Please clarify the split (e.g., user-level holdout, session-level holdout) and explicitly state that no test user appears in training, or restructure the evaluation to avoid overlap.","section":"Section 4.1, Dataset"},{"comment":"No variance, confidence interval, or significance test is reported for any offline comparison. For instance, the claimed advantage of 'GRPO w/ RM' (Rank NDCG 0.463) over the '2-Phase SFT' baseline (0.437) is presented as a conclusion without an error bar or p-value. Since the paper's central offline claims are pairwise comparisons among methods, please report standard errors or bootstrap confidence intervals (ideally across multiple training runs or user resamples) and, where possible, significance tests.","section":"Table 1"},{"comment":"The graded relevance labels used in Funnel NDCG are the same six-level funnel depths used to train the reward model via ordinal regression. The RFT methods in Table 1 are optimized against this reward model, so their Funnel NDCG improvements are partially self-referential: the metric evaluates ranking by the very labels that the alignment signal was trained to predict. The online A/B results provide external validation, but the offline claim that reward-based alignment is better than serving with the reward model itself should be supported by a metric whose labels are not used to train the reward model, or the circularity should be acknowledged and discussed.","section":"Section 4.1.1 'Funnel NDCG' and Section 3.1.2"}],"minor_comments":[{"comment":"The HSTU backbone is not explicitly cited at the point of use; please add a citation to the appropriate reference (e.g., reference [19]) when introducing the HSTU encoder.","section":"Section 4.1"},{"comment":"The phrase 'the policy head and FM backbone is optimized' should read 'are optimized'.","section":"Section 3.2.2"},{"comment":"The equation for the self-normalized IPS weight is garbled in the manuscript text; please ensure it is typeset correctly (e.g., w̃_uv = s_uv / Σ_{(u',v')∈D_r} s_{u'v'}).","section":"Section 3.1.2"},{"comment":"The statistical test yielding 'p < 0.001' is not described; please state the test procedure and whether multiple-comparison corrections were applied across the four metrics.","section":"Section 4.2"},{"comment":"The notation π_θ is used both for the click-prediction head in Section 4.1 and for the policy in the GRPO objective in Section 3.2.2 without an explicit definitional link; please clarify the relationship (the same issue is raised in Major Comment 1, but a one-sentence clarification would help the reader).","section":"Section 4.1 and Section 3.2.2"},{"comment":"The text refers to 'the daily trajectory of Click/Imp and LastPaid/Imp,' but the figure description does not mention error bars or confidence bands; please add them so that the claimed stability can be assessed visually.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's online A/B results are a strength, and the reference implementation is commendable. The main risk is that the RFT objective is formally undefined as written; the authors should be required to provide the exact policy distribution and implementation details. The offline evaluation needs a user-level held-out split and confidence intervals. If these are addressed, the paper could be a solid industry-practice contribution to RecSys."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for its online A/B evidence and for the clean separation it draws between training a dense-signal policy and using sparse business labels only for reward modeling. But the central RFT section never defines the policy distribution it optimizes, so the GRPO/DPO claims are not actually backed by the stated equations.\n\nThe three-phase recipe is straightforward, the LP-FFT motivation is sensible, and the comparison against directly serving the reward model is a useful ablation that most recommender RFT papers skip. The online A/B with several funnel metrics is stronger evidence than the usual offline-only paper, and the finding that GRPO w/ RM pushes deeper-funnel metrics while DPO w/o RM helps clicks is the kind of result practitioners care about.\n\nThe major problem is that pi_theta is never defined as a probability distribution over candidate actions. Algorithm 1 greedily selects top-K items, then computes policy ratios pi_theta(a_k)/pi_ref(a_k). For a deterministic ranker there is no such ratio; if the paper intends a softmax over the candidate group, that distribution changes as the policy changes, so the reference ratio is not well-defined either. As written, Eq. 1 is not a valid GRPO objective. The stress-test note holds. This is not a pedantic point: the authors claim a specific RLHF-style procedure and Table 1 attributes gains to GRPO, but the actual update may just be a reward-weighted ranking loss. The fix is to specify the policy distribution explicitly, or stop calling it GRPO.\n\nAlso, the offline table has no variance or significance, and the test set is a random 10% of users from the same three-week window used for training, so user-level leakage is possible. Funnel NDCG uses the same ordinal funnel labels that train the reward model, so part of the reported alignment gain may be circular. These are fixable: report bootstrap intervals, hold out a disjoint time window, and report Funnel NDCG with the reward model removed or at least acknowledge the overlap. The GitHub link is a plus, but I haven't verified whether the implementation actually defines a sampling distribution.\n\nWho this is for: practitioners at companies with a sequential backbone and a business funnel, and reviewers who want a cautionary example of what \"GRPO\" does and does not mean in an industrial setting. It deserves a serious referee, and I would send it back for major revision on the policy distribution definition and offline evaluation. The empirical payload is real and the online A/B is valuable; the formal gaps are addressable.","headline":"A useful industrial recipe with real online A/B evidence; the RFT stage is formally underspecified and offline stats are thin.","tokens_in":12673,"tokens_out":2009,"would_cite":true,"duration_ms":18423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a three-phase post-training sequence—linear probing, full fine-tuning, then GRPO-based reinforcement fine-tuning against a learned reward model—beats single-phase SFT on click and funnel-depth ranking, and that the…","keywords":["recommender systems","foundation models","progressive post-training","reinforcement fine-tuning","reward modeling","GRPO","DPO","ordinal regression"],"falsifier":"Compute $\\sum_{a\\in\\mathcal{V}} \\pi_\\theta(a|u)$ for a trained policy and compare greedy top-32 selection with sampling from the stated distribution. If the policy scores are not a normalized distribution over the full catalog, or if replacing the learned reward model with a deterministic oracle or with observed funnel labels does not change the offline gains, then the reported improvements cannot be attributed to GRPO-based alignment.","tokens_in":11514,"feed_emoji":"📈","tokens_out":5501,"duration_ms":42818,"temperature":0.7,"pith_summary":"The paper proposes a three-phase post-training recipe for adapting a pretrained recommender foundation model to a concrete serving surface. Phase one freezes the backbone and trains only the new downstream heads (linear probing, LP); phase two unfreezes everything and fine-tunes jointly, with a smaller learning rate on the backbone (full fine-tuning, FFT); phase three runs GRPO-style reinforcement fine-tuning against a reward model learned from a six-level engagement funnel, with DPO as an alternative. The paper tries to establish that this progression is more stable and more accurate than single-phase SFT, and that using the reward model as a training signal beats using it directly as the serving ranker. If true, the payoff is a practical path from a shared sequential user-behavior model to a deployed policy that ranks for immediate clicks and for deeper business outcomes at the same time.","feed_headline":"Reward-aligned recommender FM beats single-phase fine-tuning","feed_subtitle":"Offline and online tests show progressive LP-FFT-RFT post-training beats single-phase fine-tuning.","key_machinery":"The engine is the three-phase pipeline itself plus the reward model. The reward model maps sparse, delayed business outcomes into dense scalar utility by ordinal regression over six funnel stages, with a clipped inverse-propensity weight and self-normalization to correct exposure bias. Phase 3 wraps that reward in GRPO: for each user state the current policy greedily selects top-32 titles, the reward model scores them, advantages are normalized within the group, and the policy is updated with a clipped surrogate objective plus a KL penalty against the frozen post-FFT reference policy. The same reward scores can also be converted into preference pairs for a DPO variant. The load-bearing separation is that dense implicit feedback trains the policy, sparse business labels train the reward, and the reward steers the policy.","core_discovery":"The central discovery, on the paper's own terms, is that the business-metric reward and the serving policy should play different roles. A policy head trained on dense implicit feedback (clicks) provides full-catalog discrimination: in the offline ablation it reaches Rank NDCG 0.437 after LP to FFT. The reward model, trained with inverse-propensity-weighted ordinal regression on a six-level funnel, captures engagement depth: used directly as a ranker it reaches Funnel NDCG 0.637 but Rank NDCG only 0.394. GRPO fine-tuning of the click-trained policy against this reward model takes the policy to Rank NDCG 0.463 and Funnel NDCG 0.637, matching the reward model's funnel quality while retaining and exceeding the click-ranking discrimination. The paper reads this as evidence that sparse business objectives are best used as alignment signals, not as supervised labels or direct serving scores.","pith_inferences":["The paper does not test whether the gains persist when the policy is sampled stochastically rather than greedily top-32; a natural extension is to compare GRPO with full-catalog sampling against the greedy variant to see how much of the advantage comes from the objective versus the candidate-selection heuristic.","The six-level funnel is platform-specific, but the separation principle—train the serving policy on dense feedback, train the reward on sparse business outcomes, then align—should transfer to other marketplaces with a well-defined engagement ladder, such as video completion or purchase conversion.","The observation that the reward model underperforms on Rank NDCG when used directly suggests that business-utility scores are not calibrated for discrimination; an implicit extension is that calibrating or re-ranking reward scores might narrow the gap between serving the reward model and aligning with it.","The paper's online comparison is against a non-foundation production control, not against a foundation model aligned with other RLHF variants; an open question is how GRPO w/RM would fare against PPO or reward-model-free policy-gradient baselines on the same funnel metrics."],"forward_implications":["A pretrained recommender foundation model can be introduced into an existing production ranking stack through LP to FFT and yield consistent online gains over a conventional CTR/CVR product-rank baseline, even before any reinforcement stage.","Reward-model alignment produces a serving policy whose funnel-depth ranking matches the reward model itself while retaining much stronger full-catalog click ranking, so business objectives can be injected without sacrificing ranking discrimination.","GRPO with a learned reward model emphasizes mid-to-deep engagement (Engaged/Imp and LastFree/Imp), whereas DPO with observed funnel preferences emphasizes immediate clicks, so the choice of alignment method selects which part of the engagement funnel the policy optimizes.","The paper's stabilization choices—LP before FFT and a one-order-of-magnitude smaller learning rate on the backbone during FFT and RFT—are presented as necessary for the gains: single-phase LP or FFT alone underperforms the two-phase sequence on both reported metrics."],"supporting_citations":[{"why":"Supplies the ordinal regression target construction used for the reward model's multi-stage funnel labels.","marker":"[1]"},{"why":"Supplies the result that linear probing followed by full fine-tuning outperforms direct fine-tuning, motivating the LP-to-FFT split.","marker":"[7]"},{"why":"Supplies evidence that reinforcement fine-tuning naturally mitigates catastrophic forgetting, motivating the RFT phase.","marker":"[8]"},{"why":"Supplies the DPO objective used as the alternative alignment method.","marker":"[13]"},{"why":"Supplies the PPO formulation that GRPO adapts by removing the value network.","marker":"[16]"},{"why":"Supplies the GRPO algorithm that is the core of Phase 3.","marker":"[17]"},{"why":"Supplies the CTCVR product formulation used for the non-foundation production control baseline in the online A/B test.","marker":"[18]"}],"fun_headline_variants":["Progressive LP-FFT-RFT: policy on clicks, reward on business","Recommender FM: separate adaptation from alignment","Reward model as guide, not ranker: better serving policy","Dense clicks for policy, sparse goals for reward—progressive wins","Multi-phase post-training improves recommender foundation model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The GRPO stage assumes that $\\pi_\\theta(a|u)$ is a genuine probability distribution over candidate items, so that policy ratios, advantage normalization, and the KL penalty in Eq. (1) describe an actual stochastic policy; the paper never defines that distribution and Algorithm 1 selects the top-32 items greedily, so the reinforcement objective may not be the GRPO procedure it is claimed to be.","fun_headline_variants_meta":{"raw":{"variants":["Progressive LP-FFT-RFT: policy on clicks, reward on business","Recommender FM: separate adaptation from alignment","Reward model as guide, not ranker: better serving policy","Dense clicks for policy, sparse goals for reward—progressive wins","Multi-phase post-training improves recommender foundation model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1636,"prompt_tokens":996,"completion_tokens":640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":554}},"tokens_in":612,"tokens_out":640,"duration_ms":5839,"temperature":1.0,"reasoning_tokens":554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:29:57.865285+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $\\sum_{a\\in\\mathcal{V}} \\pi_\\theta(a|u)$ for a trained policy and compare greedy top-32 selection with sampling from the stated distribution. If the policy scores are not a normalized distribution over the full catalog, or if replacing the learned reward model with a deterministic oracle or with observed funnel labels does not change the offline gains, then the reported improvements cannot be attributed to GRPO-based alignment.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the result that linear probing followed by full fine-tuning outperforms direct fine-tuning, motivating the LP-to-FFT split."}],"review_version":2}