{"id":"888de9ef-c63d-4b38-87cb-d0c006e258d1","arxiv_id":"2412.18082","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"PROMO uses top positive-feedback users as item prompts with per-item prompt networks, reporting state-of-the-art cold-start recommendation, but the offline evaluation as written may leak the test label through the prompt.","lead":"This paper proposes PROMO, a prompt-tuning method for item cold-start recommendation that uses the identities of a new item's most enthusiastic users, called pinnacle feedback, as the prompt instead of text or content descriptions. It reports large offline gains over baselines on four datasets and a deployment at Kuaishou with higher click and play-time rates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The offline evaluation leaks the test user's ID into the item prompt: Pos_i is built from all users who viewed the item with no temporal split, so the held-out user can be in the pinnacle feedback list for their own test item (Eq. 10).","rationale":"The paper's central empirical assertion is that pinnacle positive feedback is superior to content descriptions as prompt information, and that PROMO achieves SOTA cold-start performance (Table 2). The mechanism for this gain is Eq. (10), which averages the embedding of the pinnacle feedback user IDs into the item representation. For the offline evaluation to be sound, the prompt for a test item must be built from information available before the prediction time, i.e., it must not include the held-out user. The paper does not state this: Section 4.1.1 selects Pos_i from all 'users who have viewed the item' without a temporal split, while Section 5.1 holds out each user's last interaction. Consequently, for the typical test pair (u,i) where i is u's last interaction, u is a viewer of i and will often be in Pos_i. The averaged prompt term in Eq. (10) then contains e_u itself, and the inner product with e_final_u produces a spuriously high score. The variant results in Table 3 support this reading: replacing user-ID prompts with content features removes the leak and the PROMO advantage shrinks from 43.3 to ~35 H@10 on ML-100K. I therefore agree with the reader's weakest_assumption: this is the single most load-bearing concern. It is not a matter of consensus or taste; it is an internal inconsistency between the prompt-construction protocol and the evaluation protocol. The released code and the industrial A/B test are positive signs, but the A/B test is described only as percentage lifts on unspecified metrics and cannot validate the offline SOTA claim, and the code should be checked for the temporal split. If the leak is present, the headline 'positive feedback is more suitable' is not established. If a re-run without the leak preserves the gains, the concern is resolved and the verdict would need upward revision. My recommendation is to keep REJECT until the evaluation is fixed.","tokens_in":17063,"tokens_out":3560,"duration_ms":33440,"concrete_test":"Re-run the MovieLens 100K and KuaiRand evaluations with a leakage-free prompt construction: for each test pair (u,i), build Pos_i and Neg_i using only feedback from users whose interaction with i is strictly earlier than u's held-out interaction (or simply remove u from Pos_i/Neg_i at test time). If PROMO's H@5/H@10 in Table 2 drops toward the level of PROMO-I/PROMO-F variants (e.g., ML-100K H@10 from 43.3 toward ~35) or toward the SASRec/PPR baselines, the headline improvement is an artifact of label leakage. A minimal code-level check: verify in the released repository whether the test user ID appears in the prompt lists used to compute Eq. (10) for that user's test item.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that positive feedback ('pinnacle feedback') is the best prompt information rests on the offline evaluation in Table 2. Section 4.1.1 defines Pos_i as the top-k users among 'users who have viewed the item i' with no temporal restriction. Section 5.1 uses leave-one-out: each user's last interaction is the test item. Therefore, for every test pair (u,i), user u is a viewer of item i and will in general be included in Pos_i. Eq. (10) then forms e_final_i as MLP(h_i, mean(S_pos_i), e_p_i), so the item embedding contains e_u, and the final score <e_final_u, e_final_i> is inflated by the inner product of e_u with a term containing e_u. This is direct label leakage: the model 'predicts' the held-out interaction partly from the test user's own ID. The ablation in Table 3 is consistent with this mechanism: replacing the user-ID prompt with item IDs or item features (PROMO-I/F/IF) yields much smaller gains (34.8-35.7 H@10 vs 43.3 on ML-100K), exactly what one expects if the large PROMO numbers come from self-matching rather than from the semantic value of positive feedback. The paper never states that Pos_i is constructed with a temporal split excluding the test interaction; the released code could clarify but the text as written does not rule out the leak. Unless this is fixed, the headline SOTA and the claim that positive feedback is more suitable than content descriptions are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PROMO is a prompt-tuning method for item cold-start CTR prediction. It constructs a 'pinnacle feedback' prompt as the top-k users who gave positive feedback on an item, encodes these user IDs through item-wise personalized prompt networks, and combines the result with base-model embeddings to obtain the final item representation. The paper claims improvements over state-of-the-art methods on MovieLens, KuaiRand, and TMall, and reports a large-scale online A/B test at Kuaishou. The core idea is parameter-efficient and avoids manual annotation. However, the offline evaluation suffers from label leakage: the pinnacle feedback list for a test item can contain the held-out user, so the test label is used in computing the prediction score. This undermines the central empirical claims.","tokens_in":17411,"tokens_out":5648,"duration_ms":51839,"significance":"The manuscript addresses a practically important problem, and the proposed mechanism is original in using user IDs of positive feedback as prompts. It also provides a deployed system and code. If the evaluation were clean, the SOTA claims would be significant. Because the offline results are confounded by direct label leakage and the online results lack statistical detail, the significance is not established in the current version.","major_comments":[{"comment":"The construction of the pinnacle feedback list Pos_i in Sec. 4.1.1 is defined for all 'users who have viewed the item i' with no temporal or held-out restriction. Sec. 5.1 uses leave-one-out, taking each user's last interaction as the test item. Therefore, for a test pair (u,i), user u is a viewer of i and is in general a candidate for Pos_i; under Eq. (10), the term (1/k) * sum(S_pos_i) then contains the test user's own ID embedding e_u. The final score <e_final_u, e_final_i> is thus inflated by an inner product with e_u itself, i.e., the model's prediction uses the ground-truth label. This is direct label leakage and invalidates the headline results in Tables 2-4 and the claim in the abstract that positive feedback is more suitable as prompt information. The paper must specify and implement a temporal split that excludes the test interaction from Pos_i (and Neg_i), and re-report all results; the current text does not rule out the leak.","section":"Sec. 4.1.1, Sec. 5.1, Eq. (10)"},{"comment":"No confidence intervals, standard errors, or significance tests are reported for any of the offline metrics in Table 2, even though PROMO's gains over SASRec on some datasets (e.g., MovieLens 1M) are small (57.5 vs 56.9 H@5). The online A/B test in Table 5 reports point estimates (+3.2% to +4.8%) without confidence intervals or p-values, and the baseline is described only as 'similar to the SASRec baseline method.' These omissions make it impossible to assess whether the claimed improvements are statistically reliable; the paper should provide bootstrap CIs or significance tests for the offline comparisons and a proper online significance analysis.","section":"Tables 2 and 5"}],"minor_comments":[{"comment":"The abstract contains the typo 'pinnaclce feedback'.","section":"Abstract"},{"comment":"The text states that results are reported for H@1, H@5, H@10, N@1, N@5, and N@10, but Table 2 only contains H@5, H@10, N@5, and N@10; the claimed H@1 and N@1 are missing.","section":"Sec. 5.3"},{"comment":"The baseline descriptions for SASRec and DSSM are identical; the DSSM bullet appears to be a copy of the SASRec bullet and should be corrected.","section":"Sec. 2.2"},{"comment":"Eq. (4) uses CR_{u,i} and IR_{u,i} but these quantities are not formally defined, and it is unclear how they are computed for MovieLens and TMall, which lack staying time and interaction scores.","section":"Sec. 4.1.1"},{"comment":"The online deployment section does not state the exact size of each group beyond 'more than 30 million users' and does not precisely identify the baseline serving model, which limits reproducibility.","section":"Sec. 5.8"}],"recommendation":"reject","confidential_remarks":"I recommend reject. The label leakage is a decisive flaw in the offline evaluation, and the online claims are under-specified. If the authors can re-run the experiments with a proper temporal split and provide statistically rigorous comparisons, a resubmission would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The paper's actual contribution is a clean idea: instead of using content or text as the prompt for cold-start items, use the user IDs of the item's strongest positive feedback, and let each item own a small prompt network. That is a meaningful departure from PPR and PLATE, and the parameter-efficiency numbers are plausible. The reported Kuaishou deployment, with 30M users per arm and consistent lifts across four metrics, is also a real asset; it suggests the mechanism works in practice.\n\nThe problem is the offline evaluation. Section 4.1.1 builds Pos_i from \"users who have viewed the item\" with no temporal restriction. Section 5.1 leaves out each user's last interaction as the test pair. So for that test pair, the user is a viewer of the item and can appear in Pos_i. Eq. (10) then averages the test user's own embedding into the item representation, and the dot product in the score is partly a self-match. That is label leakage, and it is load-bearing: the large gains over PROMO-I/F/IF on ML-100K look exactly like what self-matching would produce. As written, the headline claim that positive feedback is more suitable than content is not established.\n\nOther soft spots are minor by comparison: no confidence intervals or significance tests for Tables 2 and 5; key hyperparameters (alpha, beta, k, lambda_1, lambda_2) not reported; the cold-start threshold is arbitrary; and the online A/B test does not describe the exact baseline or statistical testing procedure. The phrase \"experimentally prove\" oversells.\n\nI'm not accusing the authors of deliberate gaming. The code is linked, and a careful re-run with Pos_i built only from interactions strictly before the test point would settle it. If that exclusion was always there, the paper is a decent practical contribution. If it was not, the offline results collapse and only the A/B test carries the conclusion.\n\nWho gets value: practitioners in industrial recsys and anyone working on prompt-based cold-start. It deserves a serious referee, because the idea is new and the deployment signal is meaningful, but the referee should ask for the temporal split to be stated explicitly and for a corrected evaluation before trusting the numbers.","headline":"Useful industrial prompt-tuning idea, but the offline evaluation as written leaks the test user's ID into the item prompt; the core SOTA claim is not established without a temporal exclusion.","tokens_in":17946,"tokens_out":2978,"would_cite":false,"duration_ms":29481,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Positive feedback beats text prompts for cold-start items","keywords":["item cold-start recommendation","prompt tuning","pinnacle feedback","personalized prompt network","CTR prediction","model bias","prompt-enhanced loss","sequential recommendation"],"falsifier":"Re-run the MovieLens and KuaiRand experiments with a strict temporal split: for each user, build each test item's pinnacle feedback list only from interactions timestamped before the user's last interaction, and check whether PROMO still beats the baselines. If the gains shrink to noise, the claimed advantage of positive feedback as prompt comes from label leakage rather than task-relevant information.","tokens_in":16862,"feed_emoji":"📈","tokens_out":5022,"duration_ms":44689,"temperature":0.7,"pith_summary":"Prompt-tuning for cold-start recommendations usually feeds the model text descriptions or content features, which are costly to annotate and semantically remote from the click-prediction task. PROMO instead builds each new item's prompt from its high-value positive feedback, meaning the users who most strongly engaged with the item, and encodes that feedback through per-item personalized prompt networks while freezing the pretrained backbone. The paper argues this removes both problems at once: feedback is directly task-relevant, and per-item networks stop popular items from dominating the tuning updates. On four public datasets it reports consistent gains over prior cold-start and prompt-based methods, and a live A/B test on a large short-video platform reports higher click rate, play time, likes, and collections for cold-start items. The paper is trying to establish that positive feedback is the right prompt signal for item cold-start recommendation and that it can be deployed cheaply at scale.","feed_headline":"Positive feedback beats text prompts for cold-start items","feed_subtitle":"Each item's top fans become its prompt, lifting live cold-start clicks 3.2% and play time 4.8%.","key_machinery":"Two mechanisms carry the argument. First, pinnacle feedback as prompt data: for each item, the top-k users by a weighted score of dwell time and interaction such as like, follow, or forward form the positive list, and k non-interacting users form the negative list, with their pretrained ID embeddings serving as the prompt. Items without direct feedback borrow the most similar popular item's pinnacle list as pseudo-prompt information. Second, the personalized prompt network: a learnable prompt embedding is reshaped into the weights and biases of a small MLP for that item alone, so the prompt information is encoded by item-specific parameters rather than a shared network. The main losses are a pairwise gap between the encoded pinnacle and negative feedback representations plus a batch-level loss that pushes cold-start positive scores above warm-item negative scores, while the frozen pretrained backbone contributes the final item embedding through a fusion MLP.","core_discovery":"The central claim is that, for item cold-start recommendation, the most informative prompt is not an item's content description but its pinnacle feedback: the small set of users who gave the strongest positive reactions to the item. Treating these users' ID embeddings as prompt input recasts user-item matching as user-user matching and supplies interest signals that content features lack. To prevent warm-up items from dominating the tuned parameters, PROMO generates a separate prompt network for each item from a learnable prompt embedding, so each item only updates its own network. Two auxiliary losses, the pinnacle-feedback prompt-enhanced loss and the intra-batch popularity-aware prompt-enhanced loss, sharpen the separation between cold-start positives and negatives and between cold-start positives and popular-item negatives. The paper reports that this design outperforms prior content-prompt, collaborative-filtering, meta-learning, and prompt baselines on MovieLens, KuaiRand, and TMall, and that the deployed system lifted cold-start click rate by 3.2%, play time by 4.8%, likes by 3.9%, and collections by 4.0% in a 14-day A/B test.","pith_inferences":["Extension: the same pinnacle-feedback prompt idea could be applied to user cold-start by swapping roles, using high-affinity items as the prompt for a new user.","Extension: a strict temporal split of feedback lists would test whether the offline gains survive when the held-out user's own engagement is excluded from the prompt; this is the main threat the current leave-one-out setup leaves open.","Extension: the dwell-time-and-interaction scoring rule for selecting pinnacle feedback could be replaced with a learned scorer, which may help in e-commerce domains where explicit signals differ.","Connection: representing an item as a function of its best users links PROMO to neighbor-based collaborative filtering ideas, suggesting it could be combined with item-item similarity graphs."],"forward_implications":["If positive feedback is the right prompt signal, cold-start recommendation can be improved without manual annotation or text descriptions for new items.","Per-item prompt networks make the tuning phase parameter-efficient: the paper reports tuning only 17.7% to 27.6% of the parameters used by full fine-tuning.","The pinnacle-feedback formulation reframes item cold-start as user-user matching, which could help cold-start items gain traffic earlier and transition to popular status.","The popularity-aware loss provides a concrete recipe for counteracting warm-item bias in two-tower CTR models.","The reported A/B gains suggest the method can move from offline benchmarks to a billion-user production recommender."],"supporting_citations":[{"why":"Supplies the prompt-based cold-start baseline and the two-stage pretrain-then-prompt setup that PROMO extends and compares against.","marker":"[40]"},{"why":"Provides the content-to-collaborative idea used to generate pseudo-pinnacle feedback for items without direct user feedback.","marker":"[3]"},{"why":"SASRec serves as the pretrained sequential backbone in PROMO.","marker":"[19]"},{"why":"DSSM is the dual-tower model used in the pilot experiment that compares positive-feedback user IDs with content features as item input.","marker":"[18]"},{"why":"KuaiRand is the dataset used in the motivational experiments showing that positive feedback carries more task-relevant signal than content features.","marker":"[12]"},{"why":"Exemplifies the content- and annotation-based prompt methods whose data cost and semantic gap PROMO targets.","marker":"[44]"},{"why":"PLATE is a prompt-tuning baseline for multi-scenario recommendations against which PROMO is evaluated.","marker":"[37]"},{"why":"MetaEmb is a meta-learning cold-start baseline that PROMO compares against and outperforms.","marker":"[28]"}],"fun_headline_variants":["Top fans' feedback beats text for cold-start items","Pinnacle feedback: better prompts for new items","Item cold-start? Use your biggest fans as prompts","Why positive feedback outperforms text in cold-start","Top fan feedback beats item descriptions for cold-start"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The offline evaluation assumes that the pinnacle feedback list used to build a test item's embedding is constructed without including the held-out test user's own interaction with that item; otherwise the prediction is partly computed from the label it is supposed to predict.","fun_headline_variants_meta":{"raw":{"variants":["Top fans' feedback beats text for cold-start items","Pinnacle feedback: better prompts for new items","Item cold-start? Use your biggest fans as prompts","Why positive feedback outperforms text in cold-start","Top fan feedback beats item descriptions for cold-start"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000551,"raw_usage":{"total_tokens":2673,"prompt_tokens":1033,"completion_tokens":1640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":1566}},"tokens_in":649,"tokens_out":1640,"duration_ms":10437,"temperature":1.0,"reasoning_tokens":1566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:03:28.624745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the MovieLens and KuaiRand experiments with a strict temporal split: for each user, build each test item's pinnacle feedback list only from interactions timestamped before the user's last interaction, and check whether PROMO still beats the baselines. If the gains shrink to noise, the claimed advantage of positive feedback as prompt comes from label leakage rather than task-relevant information.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prompt-based cold-start baseline and the two-stage pretrain-then-prompt setup that PROMO extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the content-to-collaborative idea used to generate pseudo-pinnacle feedback for items without direct user feedback."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KuaiRand is the dataset used in the motivational experiments showing that positive feedback carries more task-relevant signal than content features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Exemplifies the content- and annotation-based prompt methods whose data cost and semantic gap PROMO targets."}],"review_version":1}