{"id":"641381d7-acd6-4e7e-983e-e402cb28afa0","arxiv_id":"2412.06308","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A mixture-of-experts fusion of trainable LLM semantic embeddings with collaborative item IDs, followed by universal-then-targeted pre-training, improves sequential recommendation on Amazon and WeChat.","lead":"PRECISE combines the text-derived semantic embeddings of a large language model with the item-ID embeddings used by collaborative filtering, then pre-trains a sequence model on all user behaviors before fine-tuning it for one recommendation scenario. In offline tests and live WeChat experiments, this two-stage hybrid improves recall and ranking over strong baselines, with the largest gains on long-tail items.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cold-start claim is untested offline: Section V-B excludes zero-training items from the cold-item evaluation, so the paper's core motivation (handling unseen items) lacks direct evidence; online rows cover cold-start users, not zero-training items.","rationale":"The paper's empirical package is internally consistent and the general recommendation improvements are plausible; I do not suspect fabrication. However, the paper's distinctiveness over prior ID-based and text-based recommenders is its claimed ability to serve items with no collaborative signal (cold-start). The evaluation deliberately excludes zero-training items, so the core novelty is not directly supported. This is exactly the reader's weakest_assumption, and I agree. The erroneous loss equations (Eq. 5 and Eq. 6) and missing variance estimates are additional reproducibility defects, but they do not by themselves invalidate the comparative results; the cold-start gap is more load-bearing because it targets the paper's raison d'être. Since the reader's CONDITIONAL verdict already conditionalizes on this issue, my read does not move the verdict; the authors should add a zero-shot-item evaluation or explicitly reframe the claim as long-tail recommendation. I set verdict_should_be to UNCHANGED.","tokens_in":19959,"tokens_out":6032,"duration_ms":61954,"concrete_test":"Re-run the offline evaluation on both Amazon and WeChat-AllScene with the test set augmented to include items that appear in the next-day behaviors but have zero training instances in all previous days (i.e., true zero-shot items). Compare PRECISE-UT against HLLM and HSTU on this zero-shot subset, reporting R@10 and N@10. If the protocol cannot assign an ID embedding to such items, fall back to an item-side cold-start test on the WeChat data using only text-derived embeddings for both methods, and check whether PRECISE's semantic fusion retains its edge. If PRECISE-UT beats HLLM on true zero-shot items, the cold-start claim survives; otherwise it should be narrowed to long-tail recommendation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-B defines the cold-item experiment as items with fewer than 1,000 interactions and then states: 'for a fair comparison with ID-based models, new items with zero training instances were not included in the evaluation.' This directly removes the regime the paper claims to address: the introduction and abstract motivate the framework by ID-based models' 'incapacity to handle long-tail items and cold-start scenarios,' and the semantic embeddings are supposed to give a representation to items with no collaborative history. Table IV therefore measures long-tail-but-seen items, not zero-shot items; the reported +13–14% gains over HLLM are for a setting where ID embeddings are still trainable. The online evidence (Table VIII) also does not fill the gap: 'cold-start users' are defined by low reading frequency in Recommendation Box, and the metrics are user-side (active users, clicks, shares, reading time), with no item-level cold-start breakdown. The strongest justification for the method's cold-start improvement is thus not tested by any experiment in the paper. The claim could still hold, but the evidence as presented does not support it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PRECISE is a pre-training framework for sequential recommendation that combines collaborative item-ID embeddings with LLM-generated semantic token embeddings through an MoE-based fusion module, pre-trains a decoder-only Transformer on all-scene behavior sequences, and then adapts the model to target scenarios with a BPR-style objective. The paper reports offline gains over SAS-Rec, HSTU, LEARN, and HLLM on Amazon and WeChat-AllScene, ablation and scalability studies, and online A/B improvements in WeChat recall and ranking. The authors additionally describe periodic warm-up and alternate training strategies used in deployment.","tokens_in":20259,"tokens_out":5106,"duration_ms":55215,"significance":"If the empirical results hold, this is a valuable industrial-scale demonstration that combining collaborative and semantic information can improve sequential recommendation and transfer across scenarios. The paper's strengths include experiments on a very large industrial dataset, online deployment with statistical significance tests, ablations that isolate the contribution of each input component, and a scalability study. The main concerns are that the proposed cold-start advantage is not directly tested in the offline evaluation, the online cold-start results concern users rather than unseen items, and the two printed loss equations do not match the described training objectives. These issues need to be corrected before the stated claims can be accepted.","major_comments":[{"comment":"The NIP loss in Eq. (5) is not the sampled softmax cross-entropy described in the text. As printed it is -sum log(pos)/(log(pos)+sum log(neg)), which is a ratio of log-dot-products rather than a cross-entropy over sampled candidates, and it is undefined whenever a dot product is non-positive. Similarly, Eq. (6) prints -log(pos - neg) for the BPR loss, whereas the standard BPR objective is -log sigma(pos - neg). Since these equations define the training objectives of the two main training stages, they are load-bearing and must be corrected or replaced with the actual objectives used in the experiments.","section":"Section IV-C, Eq. (5) and Section IV-D, Eq. (6)"},{"comment":"The cold-item evaluation explicitly excludes the regime the paper claims to address. The text states 'for a fair comparison with ID-based models, new items with zero training instances were not included in the evaluation,' and Table IV evaluates items with fewer than 1,000 interactions. Such items still have training instances and learnable ID embeddings. The abstract and introduction motivate PRECISE by the inability of ID-based models to handle 'cold-start scenarios' and 'newly published items,' so the claimed advantage for truly unseen items is not supported by the offline experiments. Please either include zero-training items in the evaluation or revise the paper's claims to explicitly target long-tail items with limited but nonzero interactions.","section":"Section V-B, Table IV"},{"comment":"The online experiments do not fill the cold-start item gap. Cold-start users are defined by low reading frequency in the Recommendation Box, and the reported metrics are user-side (active users, clicks, shares, reading time). No item-level cold-start breakdown is provided, and the online serving schemes retrieve items by embedding similarity rather than testing items never seen during training. The paper should either provide item-level cold-start evidence or clearly separate the supported claim (better recall and ranking for users and for long-tail items with limited interactions) from the unsupported claim about zero-training items.","section":"Section VI-B, Table VIII and Table IX"}],"minor_comments":[{"comment":"The related-work text attributes the same reference [24] to both 'Li et. al.' and 'Zhang et. al.'; please verify the citation and attribute the work correctly.","section":"Section II-A"},{"comment":"Several hyperparameters that affect the reported results are not specified, including the number of MoE experts K, the top-k expert selection count, the number of negative samples per target item, the periodic warm-up cycle, and the alternate-training convergence criterion; please report these to make the experiments reproducible.","section":"Section V-A"},{"comment":"The paragraph beginning 'We have deployed both U2I and U2I2I recall paradigms in the Article Recommendation task' is repeated almost verbatim immediately before the results discussion; please remove the duplicated passage.","section":"Section VI-B"},{"comment":"The sentence 'This result is consistent with our previous conclusions based on offline experiments in Section IV-A' refers to the wrong section; offline experiments are presented in Section V.","section":"Section VI-C"},{"comment":"There is a typo in the sentence preceding Eq. (3): 'sof tmaxfunction' should read 'softmax function.'","section":"Section IV-B"},{"comment":"The 'Improve' row in Table III does not state the reference baseline; please specify explicitly that the improvement is computed against HLLM, the runner-up.","section":"Table III"}],"recommendation":"major_revision","confidential_remarks":"This is a strong industrial empirical paper, but the mismatch between the claimed cold-start motivation and the evaluation is the main correctness risk; the loss-equation errors also need to be fixed. If the authors cannot run an evaluation with zero-training items, they should scope the claims accordingly. I would also suggest that the authors transparently note that the GNN baseline used in Table VIII is their own prior work, as this is relevant context for interpreting the online comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PRECISE deserves a serious referee. To my knowledge it is the first industrial paper to make the ID-plus-LLM embedding combination work in sequential recommendation — HLLM, its closest baseline, explicitly reports that adding ID embeddings hurt performance, and PRECISE's alternative (MoE-gated fusion of trainable token embeddings concatenated with ID embeddings, plus a universal-to-targeted warm-start schedule) produces consistent gains. That is a genuine engineering contribution, and the paper earns credit for the ablations: every input modality contributes, MoE beats average pooling, and the diagnosis of why ID embeddings under-train (Figure 4b) is a useful observation. The online A/B evidence, with tens to hundreds of millions of users and significance tests, is unusually strong for this line of work.\n\nThe soft spots are real but not fatal. The stress-test's main concern holds up on reading: Section V-B explicitly excludes items with zero training instances from the cold-item evaluation, so Table IV measures long-tail-but-seen items, not the truly unseen items the intro motivates. The online 'cold-start users' are low-frequency readers, not zero-training items. The strong cold-start claim is therefore not tested anywhere in the paper. The authors should either add a zero-shot evaluation (Amazon would allow holding out new items) or honestly reframe the claim to long-tail items.\n\nSecond, the two printed loss equations are wrong as written. Equation (5) is a ratio of log-scores rather than a proper sampled softmax, and Eq. (6) sums over all other users and takes the log of a difference that can be negative. Likely notation typos, but they undermine auditability until fixed. Third, offline tables have no variance estimates or significance tests, and no code or data are released; the WeChat data obviously cannot be public, but Amazon-side configs and code should be.\n\nThe paper is not deceptive: the zero-training exclusion is stated plainly, and the online seesaw on shares is acknowledged. My verdict matches the reader's CONDITIONAL. Who is this for? Engineers building multi-scenario industrial recommenders and researchers working on ID-semantic fusion. It deserves peer review, not desk rejection, and I would send it out with a request for corrected equations, a zero-shot item experiment, and Amazon-side artifacts.","headline":"A real industrial contribution to ID+LLM fusion for sequential recommenders, with honest reporting — but the cold-start claim is untested on truly unseen items and the printed losses need fixing.","tokens_in":20736,"tokens_out":4813,"would_cite":true,"duration_ms":47954,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PRECISE claims that fusing item IDs with LLM semantic embeddings—gated by a Mixture-of-Experts network—and pre-training on all-scene behavior before target-scene fine-tuning improves sequential recommendation, including long-tail and…","keywords":["sequential recommendation","pre-training","large language models","semantic embedding","collaborative signals","Mixture of Experts","cold-start recommendation","industrial recommendation system"],"falsifier":"Rerun the Amazon or WeChat evaluation on a held-out set of items with zero training instances and compare Recall@K against the strongest pure-ID baseline; if PRECISE does not win there, the cold-start claim is not supported. A second check: retrain with average pooling instead of the MoE gate; if offline metrics do not drop, the gating mechanism is not what carries the improvement.","tokens_in":19755,"feed_emoji":"🎯","tokens_out":7803,"duration_ms":71322,"temperature":0.7,"pith_summary":"PRECISE is a pre-training framework for sequential recommendation that combines two item signals usually kept apart: collaborative patterns carried by item IDs, and semantic meaning carried by LLM-generated text-token embeddings. The paper argues that ID-only models under-serve long-tail and cold-start items, text-only models miss collaborative similarity, and fusing both through a Mixture-of-Experts gate yields a user representation that transfers across recommendation scenarios. The model is pre-trained to predict the next item in users' all-scene behavior sequences, then warm-started and fine-tuned on target-scene behavior with a ranking loss. If correct, platforms can reuse one pre-trained representation across many recommendation surfaces while keeping collaborative signal, and the reported online A/B tests show lifts in active users, clicks, shares, and reading time.","feed_headline":"Fusing item IDs with LLM text beats pure recommender baselines","feed_subtitle":"A two-stage pretraining scheme transfers cross-scene user interests and lifts clicks, shares, and reading time in A/B tests.","key_machinery":"The load-bearing mechanism is the Embedding Fusion module: for each item, an ID embedding is concatenated with a semantic embedding produced by K attention-based expert networks, with a gating network that selects the top-k experts and weights their outputs. The MoE gate lets collaborative training decide how much each text token contributes, which is what distinguishes this from simple pooling. Around that, Universal Training uses a masked decoder-only Transformer with next-item prediction and sampled negatives over all-scene sequences, while Targeted Training warm-starts from those parameters, removes the attention mask, pools the final layer's item embeddings through an MLP, and optimizes BPR loss. Two training practices carry the industrial claim: periodically re-warming the targeted model from the universal model keeps newly published items benefiting from all-scene pre-training, and alternately freezing token embeddings while updating ID embeddings prevents the pre-trained text from starving collaborative ID learning.","core_discovery":"The paper's central claim is that semantic and collaborative information are complementary at the item-representation level, and that a universal-to-targeted training scheme lets one model serve many scenarios. Concretely, PRECISE concatenates a learnable item-ID embedding with a semantic embedding produced by a Mixture-of-Experts network that distills the item's LLM token embeddings into one vector. A decoder-only Transformer is pre-trained with a next-item prediction loss over all-scene behavior sequences, and its parameters are then transferred to a target task where bidirectional attention and a Bayesian Personalized Ranking loss adapt holistic user interests to the specific scenario. In offline experiments, PRECISE-UT, the universal-trained encoder, outperforms ID-only and text-only baselines on both Amazon and WeChat-AllScene, with the largest relative gains on the cold-item subset; the full PRECISE pipeline also beats universal-only and targeted-only training on click and share tasks.","pith_inferences":["The offline cold-item evaluation excludes items with zero training instances, so the evidence as presented supports infrequent items rather than truly unseen items; testing on zero-training items is a direct way to extend the claim.","Because the MoE gate selects experts per item, an unstated implication is that the model learns which text parts matter for collaborative behavior; inspecting gate weights could test this.","Since the framework accepts any token encoder, it should extend to video, live, and music items whose titles, hashtags, or transcribed speech are available as text—a natural deployment next step.","The online cold-start results show a seesaw effect where click embeddings can reduce shares, suggesting multi-objective or task-specific fine-tuning when multiple business metrics matter."],"forward_implications":["The same pre-trained user and item embeddings can be plugged into recall, both user-to-item and user-to-item-to-item, and into ranking as input features, with reported online gains in active users, clicks, shares, and reading time.","Semantic-plus-ID representation should beat pure ID models specifically on long-tail items, because the text channel carries signal before many interactions accumulate.","Universal Training alone beats Targeted-only training on sparse target tasks, so cross-scene pre-training can substitute for some target-scene data.","Scaling behavior sequence length, LLM size, and transformer depth each improve offline performance, implying the framework absorbs more data and parameters.","Periodic warm-up keeps the fraction of pre-trained items above 70 percent, so continuous item production does not erode the benefit of universal training."],"supporting_citations":[{"why":"Defines the self-attentive sequential recommendation baseline that PRECISE must beat and provides the Transformer building blocks used in the model.","marker":"[30]"},{"why":"Provides the HSTU generative recommender baseline, the strongest ID-only comparison in the paper.","marker":"[3]"},{"why":"Provides the LEARN baseline that adapts LLM knowledge to recommendation using semantic embeddings only, the main text-only comparison.","marker":"[2]"},{"why":"Provides the HLLM baseline, a hierarchical LLM item-and-user model that is the strongest semantic-only rival.","marker":"[15]"},{"why":"Supplies the ChatGPT-style next-item prediction training paradigm that PRECISE adapts for Universal Training.","marker":"[12]"},{"why":"Supplies the Bayesian Personalized Ranking loss used in Targeted Training to optimize ranking on the target task.","marker":"[36]"},{"why":"Supplies the bidirectional attention design that Targeted Training adopts to let each item see the full target-scene sequence.","marker":"[35]"},{"why":"Provides a standard recall baseline that the online experiments augment with PRECISE embeddings in user-to-item recall.","marker":"[40]"},{"why":"Provides the ranking architecture into which pre-trained user and item embeddings are concatenated as features online.","marker":"[42]"}],"fun_headline_variants":["Merging item IDs with LLM text lifts cold-start recommendations","Two-stage pretraining unifies collaborative and semantic signals","PRECISE: Hybrid ID+text representations beat pure baselines","LLM text plus item IDs: Better recs for long-tail and cold-start","PRECISE fuses collaborative and semantic vectors for next-item prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a long-tail item with a few interactions is a fair stand-in for a cold-start item: the paper's offline cold-item evaluation deliberately excludes items with zero training instances, and some online cold-start share metrics moved negatively.","fun_headline_variants_meta":{"raw":{"variants":["Merging item IDs with LLM text lifts cold-start recommendations","Two-stage pretraining unifies collaborative and semantic signals","PRECISE: Hybrid ID+text representations beat pure baselines","LLM text plus item IDs: Better recs for long-tail and cold-start","PRECISE fuses collaborative and semantic vectors for next-item prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000807,"raw_usage":{"total_tokens":3567,"prompt_tokens":994,"completion_tokens":2573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2482}},"tokens_in":610,"tokens_out":2573,"duration_ms":17832,"temperature":1.0,"reasoning_tokens":2482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:48:01.976543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Amazon or WeChat evaluation on a held-out set of items with zero training instances and compare Recall@K against the strongest pure-ID baseline; if PRECISE does not win there, the cold-start claim is not supported. A second check: retrain with average pooling instead of the MoE gate; if offline metrics do not drop, the gating mechanism is not what carries the improvement.","supporting_citations":[{"cited_title":"Self-attentive sequential recommenda- tion,","cited_arxiv_id":null,"evidence_quote":"Defines the self-attentive sequential recommendation baseline that PRECISE must beat and provides the Transformer building blocks used in the model."},{"cited_title":"Actions speak louder than words: Trillion- parameter sequential transducers for generative recommendations,","cited_arxiv_id":null,"evidence_quote":"Provides the HSTU generative recommender baseline, the strongest ID-only comparison in the paper."},{"cited_title":"Bpr: Bayesian personalized ranking from implicit feedback,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian Personalized Ranking loss used in Targeted Training to optimize ranking on the target task."}],"review_version":1}