{"id":"5aac55cb-6124-43c2-8169-bdf24f2404d1","arxiv_id":"2412.16933","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RSLLM mixes item ID embeddings from classical recommenders with text titles inside an LLM prompt and uses two-stage contrastive fine-tuning to improve sequential recommendation.","lead":"This paper proposes RSLLM, a way to make large language models better at next-item recommendation by feeding them both the text titles of items and the ID-style vectors learned by classic recommender systems. It reports higher accuracy than existing LLM-based recommenders on three public datasets, though without releasing code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison may not be controlled: LLaRA numbers appear borrowed from the original paper, so the few-point HR@1 gains over LLaRA could be artifacts of different candidate sets or prompts.","rationale":"The reader's weakest assumption was that the candidate-based evaluation protocol is a faithful and unbiased measure, with the missing variance and MetaST baseline noted as supporting concerns. I agree that the evaluation setup is the load-bearing element, but I identify a more specific, concrete flaw: at least the principal baseline (LLaRA) appears to be cited from the prior LLaRA paper rather than reproduced under controlled conditions, as stated in Table 4's 'results taken from (Liao et al., 2024)'. Because the performance margins between RSLLM and LLaRA are only a few tenths of a percent, small differences in candidate-set draws, prompt formatting, or decoding settings could entirely explain the reported gains. This does not change the verdict from CONDITIONAL; it sharpens the condition. The paper should be accepted only if the authors either release code and reproducible evaluation scripts or rerun all baselines in a unified harness and show that the cited numbers are stable. The absence of standard deviations further weakens the significance claim, but the borrowed baseline issue is the most directly load-bearing: without it, the central claim has no controlled evidence. No internal technical error in the model design was identified; the novelty and ablations are plausible. Therefore, the reader's conditional accept remains appropriate, but the condition should explicitly include a controlled re-benchmarking of baselines.","tokens_in":15516,"tokens_out":14942,"duration_ms":129107,"concrete_test":"Run all baselines (LLaRA with GRU4Rec/Caser/SASRec backbones, plus MoRec, TALLRec, and MetaST) in the same codebase as RSLLM, using identical candidate-set generation (e.g., fixed shared random seeds), identical prompt templates, and identical decoding hyperparameters. Recompute Table 2 with 5 runs and report mean ± std. If LLaRA's reproduced numbers shift by more than the RSLLM-minus-LLaRA gap (e.g., >0.0044 HR@1 on any dataset), the central claim is not robust. Also verify whether the LLaRA checkpoint used is the same as in Liao et al. and whether any prompt differs in candidate ordering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 5.2) is that RSLLM 'significantly outperforms all baseline models'. The comparison with the strongest baseline, LLaRA, may not be controlled: Table 4's baseline row (♣) is explicitly labelled 'results taken from (Liao et al., 2024)', indicating that at least one set of baseline numbers was not reproduced in the authors' evaluation harness. Section 5.1 describes the candidate-set protocol (20 randomly sampled non-interacted items per sequence; HitRatio@1; 5 runs averaged). Since the absolute margins over LLaRA are as small as 0.0004 on Steam (RSLLM-Caser 0.4953 vs LLaRA-SASRec 0.4949 in Table 2), any difference in candidate-set sampling, prompt template, or decoding settings between the two papers could change HitRatio@1 by more than the reported gap. Without re-running LLaRA and the other baselines under identical conditions, the claimed significant improvement cannot be established. Additionally, no standard deviations are reported, so the paired t-test statement in the Table 2 caption is unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RSLLM, a framework for sequential recommendation that integrates traditional ID-based recommender embeddings with LLM textual prompts. Items are represented in a hybrid prompt containing both textual metadata and behavioral tokens derived from collaborative filtering embeddings via a trainable projector (Section 4.1). Training is a two-stage LoRA fine-tuning procedure that combines a next-item language modeling loss with two contrastive losses (Eqs. 2-5). The method is evaluated on MovieLens, Steam, and LastFM, using HitRatio@1 and ValidRatio under a 20-item candidate-set protocol following LLaRA (Section 5.1). The paper claims significant improvement over all baselines, including traditional recommenders and LLM-based methods, with ablations supporting the contribution of textual features, ID information, contrastive alignment, and two-stage training.","tokens_in":15772,"tokens_out":3370,"duration_ms":32199,"significance":"If the empirical claims hold, RSLLM is a useful contribution to LLM-based sequential recommendation: it explicitly aligns traditional recommender embeddings with LLM token space at multiple granularities, and the two-stage contrastive fine-tuning scheme is clearly described and reproducible in principle. The consistent gains over strong baselines across three datasets and three recommender backbones are a positive signal. However, the paper's central claim is empirical and depends entirely on the controlled comparison with LLaRA and other baselines. The manuscript does not ship code or release per-run results, and the baseline numbers for LLaRA are borrowed rather than re-run in the same evaluation harness. These issues currently prevent verifying the claimed significance, although they are fixable within the scope of a revision.","major_comments":[{"comment":"The central claim that RSLLM 'significantly outperforms all baseline models' is not established by the reported evidence. In Table 4, the baseline row (♣) is explicitly labeled 'results taken from (Liao et al., 2024)', meaning the strongest LLM baseline, LLaRA, was not evaluated in the authors' own harness. The margins over LLaRA can be very small: on Steam, RSLLM-Caser achieves HitRatio@1 of 0.4953 versus LLaRA-SASRec's 0.4949 (Table 2), a difference of 0.0004. Given that HitRatio@1 is computed over randomly sampled 20-item candidate sets (Section 5.1), any difference in candidate sampling, prompt template, decoding settings, or sequence filtering between the two papers could shift the metric by more than this gap. The authors must re-run LLaRA and the other baselines under the identical candidate-set protocol and report the resulting numbers, or clearly demonstrate that borrowed numbers are directly comparable.","section":"Section 5.2, Tables 2 and 4"},{"comment":"The paired t-test claim in the Table 2 caption is unverifiable. The paper reports only averages over five runs, with no standard deviations, no test statistics, no degrees of freedom, and no statement of whether the pairing is across runs or across sequences. A claim of 'significant best and second-best results' with p<0.05 requires at minimum per-run values or error bars and a clear description of the paired test setup. Without this, the reader cannot assess whether the observed improvements over LLaRA are statistically meaningful, especially for margins as small as those on Steam.","section":"Section 5.1 and Table 2 caption"},{"comment":"The ablation study has the same controlled-comparison problem as the main results: the baseline row is borrowed from Liao et al. (2024), while the RSLLM rows are produced in this paper's harness. Moreover, the ablation text draws conclusions such as 'without pre-loading item embeddings, the model achieves similar performance as when the embeddings are included' without reporting standard deviations or significance tests. Given that the differences between adjacent rows are often small (e.g., 0.4890 vs. 0.4905 on Steam for w/o IID Tokens versus w/o Pre-loading), the ablation conclusions require the same statistical support as the main results. Please report error bars or per-run results for all ablation configurations.","section":"Section 5.3 and Table 4"},{"comment":"The baseline enumeration in Section 5.1 lists 'MetaST' as a baseline, but MetaST does not appear in Table 2 or anywhere in the results or appendices. Either provide its results under the same protocol or remove it from the baseline list. In addition, the setup does not state which hyperparameters or prompt templates were used for MoRec and TALLRec, nor whether these baselines were re-run by the authors or taken from prior work. The reader needs this information to judge the fairness of the comparison.","section":"Section 5.1, Baseline list"}],"minor_comments":[{"comment":"The sentence preceding Eq. (2) says 'the first m tokens correspond to all items except the last one, with the remaining n−m tokens dedicated to the target item,' but the notation is ambiguous because the sequence is written as (u1, u2, ..., un) and m is not defined in terms of n or the number of items. Please clarify the token-level indexing.","section":"Section 4.2, Eq. (2)"},{"comment":"The phrase 'removing item ID representation or IID tokens also delegate the performance' should read 'degrade the performance'. Also, the sentence 'It is recommended to directly train the item representation parameters' is confusing after the row 'w/o Pre-loading item Embeddings' shows similar performance; please clarify whether the recommendation is to train from scratch without preloading.","section":"Section 5.3, text near Table 4"},{"comment":"The caption of Figure 3 lists 'LLaRA' among the compared representations, but the text describes only numerical indexing, behavior tokens, text feature, and RSLLM representation. Please make the figure caption consistent with the text and explain how LLaRA's representation is defined in this comparison.","section":"Section 5.4 and Figure 3 caption"},{"comment":"The hyperparameters γ=0.3, β=0.4, and τc=0.5 are said to be determined by grid search, but no grid ranges or selection criterion are given. State the search ranges and whether the same tuning budget was applied to baselines, and report the selected hyperparameters per dataset if they vary.","section":"Section 5.1, Implementation Details"},{"comment":"The paper does not mention code or data release. Given that the empirical protocol (candidate sampling, prompt templates, sequence filtering) is central to the claims, releasing code and processed data, or at least detailed pseudocode for the evaluation, would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical claims are plausible but currently rest on a comparison whose strongest baseline (LLaRA) was not re-run in the same harness, and the statistical support is missing. The authors should be asked to re-run all baselines under the identical candidate-set protocol, report standard deviations and test statistics, and clarify the hyperparameter tuning budget. If these requirements are met, the contribution could be a solid systems paper. I also note that the novelty claim of being 'the first work to investigate multi-granularity alignment' is somewhat broad given prior work combining text and IDs; the revision should position the two-stage contrastive alignment more precisely as the main differentiator."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zheng et al. propose RSLLM, which fuses ID-based item embeddings from a traditional sequential recommender with item text in the LLM prompt, plus a projector and two contrastive losses. The real addition over prior work like LLaRA is the item/user-level contrastive alignment; LLaRA already used ID-plus-text prompts and two-stage training, as the paper's own Table 1 shows. That is a legitimate extension, not a new paradigm.\n\nThe strongest part is the ablation study. Removing the contrastive alignment, the item-ID tokens, or the two-stage schedule consistently hurts performance across three datasets and three backbones. That gives me some confidence the design choices matter, and the gains are directionally plausible. The paper is clearly written and follows the LLaRA setup.\n\nThe soft spots are the ones you would expect. The LLaRA baselines are not re-run in this paper's harness: Table 4 labels the baseline row as 'results taken from (Liao et al., 2024)', and the main table appears to use those same numbers. Since the candidate set is randomly sampled (20 non-interacted items per sequence), a different random draw or prompt template can shift HitRatio@1 by more than the reported 0.0004 gap on Steam. No standard deviations are reported anywhere, so the paired t-test statement in the Table 2 caption is not checkable. MetaST is listed as a baseline in Section 5.1 but never appears in the results. There is no code.\n\nNone of this is fatal. The method is coherent and the paper itself flags the borrowed baseline with a footnote. But the headline claim that RSLLM 'significantly outperforms' LLaRA is not established at the reported margins. It is established as a recipe worth trying: the design is sensible, and the ablations support the components.\n\nI would send this to reviewers. The right revision is to re-run LLaRA and the other baselines under identical conditions, report standard deviations and test statistics, add MetaST, and release code. For a reader working on LLM-based sequential recommendation, this is a useful data point despite the weak comparison.","headline":"A useful incremental recipe for fusing ID embeddings and text in LLM recommenders, but the claimed edge over LLaRA rests on borrowed baseline numbers and unverifiable statistics.","tokens_in":16286,"tokens_out":2372,"would_cite":true,"duration_ms":21487,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The RSLLM framework, which merges item ID embeddings with text inside LLM prompts, reports top HitRatio@1 on MovieLens, Steam, and LastFM.","keywords":["sequential recommendation","large language models","recommendation as a new language","contrastive alignment","item ID embeddings","hybrid prompting","LoRA fine-tuning","HitRatio@1"],"falsifier":"Re-run the MovieLens, Steam, and LastFM comparisons with the same candidate protocol but report per-run HitRatio@1 and standard deviations, and additionally evaluate under full-corpus ranking where the true next item must be ranked among all items instead of a sampled 20-item set. If RSLLM's margins over LLaRA shrink to within noise or invert under full-corpus ranking, the central claim would be weakened.","tokens_in":15346,"feed_emoji":"🎯","tokens_out":8186,"duration_ms":57951,"temperature":0.7,"pith_summary":"This paper tries to establish that a large language model can become a better sequential recommender if it is taught a new 'language' that mixes textual item descriptions with the learned ID embeddings produced by a traditional recommender. The proposed framework, RSLLM, builds prompts in which each item appears both as its title and as a behavioral token, aligns the ID embeddings to the LLM's input space with a small projector, and fine-tunes the LLM in two stages using two contrastive losses plus a next-item prediction loss. The authors report that RSLLM outperforms all compared baselines on MovieLens, Steam, and LastFM, with HitRatio@1 gains of 4.3%, 3.0%, and 4.8% over the best baseline on each dataset. If the result is correct, it means the gap between traditional collaborative recommenders and LLM-based recommenders can be narrowed by giving the LLM behavioral signals in addition to text.","feed_headline":"Fusing recommender IDs with text in LLMs lifts next-item hits","feed_subtitle":"RSLLM unifies text and collaborative embeddings in one prompt, beating all baselines on MovieLens, Steam, and LastFM.","key_machinery":"The central mechanism is the hybrid token representation, which concatenates three components for every item: the LLM's own text tokens for the item title, a special '[DSR]' token marking a recommendation ID, and a behavioral token computed by an Adapter built from a traditional sequential recommender (GRU4Rec, Caser, or SASRec). A trainable projector, a two-layer perceptron, maps the recommender's ID embedding into the LLM's input space so the model can read the two kinds of information together. The training objective combines the next-item prediction loss with two InfoNCE-style contrastive losses, one aligning the user-history tower with the target-item tower and one aligning the target item conditioned on history with the target item alone. Two-stage fine-tuning first runs text-only prompts, then unified prompts, with LoRA used for parameter-efficient adaptation. This is what carries the argument: it lets the model use world knowledge from text and collaborative knowledge from ID embeddings at multiple granularities.","core_discovery":"The paper's central claim is that its RSLLM framework significantly outperforms all baseline models across the three datasets in HitRatio@1 and ValidRatio. On MovieLens, RSLLM surpasses the best baseline by 4.3% in HitRatio@1; on Steam the gain is 3.0%; on LastFM it is 4.8%. The authors attribute the improvement to the unified prompting method that combines ID-based item embeddings from a conventional sequential recommender with textual item features, and to a two-stage fine-tuning procedure that first adapts the LLM with text-only prompts and then with unified prompts, using two contrastive losses and a language modeling loss. Ablation experiments support the contribution of each component: removing textual features, item ID representation, IID tokens, or contrastive alignment each reduces performance.","pith_inferences":["A natural stress test is to move from the 20-candidate protocol to full-corpus ranking; if RSLLM's HitRatio@1 margins persist when the true item must be ranked among all items, the collaborative-signal hypothesis would be much stronger. (Editorial inference, not reported in the paper.)","The 'new language' idea is not specific to item IDs: the same projector-plus-two-stage-contrastive recipe could be applied to timestamps, user attributes, or multimodal signals, and the paper's framework suggests a general recipe for injecting structured knowledge into LLMs. (Editorial inference.)","Because no standard deviations are reported, an independent re-run with the same settings and multiple seeds would clarify whether the few-percent gains are consistent or within noise. (Editorial inference.)"],"forward_implications":["LLM-based recommenders can incorporate collaborative signals by treating item IDs as a new language rather than as plain numbers or text alone.","Two-stage training matters: the text-only stage is a necessary foundation for the behavioral stage, and removing either stage reduces HitRatio@1.","Item-item contrastive alignment contributes more than user-item alignment, showing that the model benefits from aligning the predicted target with the actual target at the item level.","The framework works with RNN-, CNN-, and self-attention-based backbones, with SASRec giving the best results, so the gains come from the prompting and training scheme rather than a particular recommender.","Instruction tuning on the recommendation task lifts ValidRatio from 0.17-0.44 for the base Llama2 to above 0.96 for RSLLM, addressing the invalid-answer problem of generative recommenders."],"supporting_citations":[{"why":"Provides the LLaRA baseline and the experimental protocol (20-candidate sampling and HitRatio@1) that RSLLM follows and must beat.","marker":"(Liao et al., 2024)"},{"why":"Supplies the SASRec backbone and the Steam dataset used in the experiments.","marker":"(Kang and McAuley, 2018)"},{"why":"Supplies the GRU4Rec backbone and a traditional baseline whose item embeddings RSLLM injects into the LLM.","marker":"(Hidasi et al., 2016)"},{"why":"Supplies the Caser backbone and a traditional baseline.","marker":"(Tang and Wang, 2018)"},{"why":"Provides the MovieLens dataset, one of the three benchmarks.","marker":"(Harper et al., 2016)"},{"why":"Provides the LastFM dataset used for evaluation.","marker":"(Cantador et al., 2011)"},{"why":"Source of the InfoNCE contrastive loss family that the user-item and item-item alignment losses are based on.","marker":"(van den Oord et al., 2018)"},{"why":"LoRA is the parameter-efficient fine-tuning method used to adapt the LLM in both stages.","marker":"(Hu et al., 2021)"},{"why":"Llama2 is the backbone LLM that RSLLM fine-tunes and that the valid-ratio comparisons are based on.","marker":"(Touvron et al., 2023)"},{"why":"TALLRec is a baseline LLM4Rec method that RSLLM compares against and aims to surpass.","marker":"(Bao et al., 2023)"}],"fun_headline_variants":["RSLLM unifies ID embeddings and text to boost next-item hits","Unified prompt ties collaborative IDs and text, beating baselines","RSLLM fuses item IDs with text for better next-item prediction","Teaching LLMs a new language: recommender IDs lift hit ratio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported superiority of RSLLM rests on the evaluation protocol from LLaRA: for each user sequence, 20 non-interacted items are randomly sampled as candidates, the model must pick the true next item from that set, and success is averaged over five runs without reporting standard deviations; if this candidate-based protocol is a biased measure of real next-item recommendation quality, the few-point gains over LLaRA may not transfer to production settings.","fun_headline_variants_meta":{"raw":{"variants":["RSLLM unifies ID embeddings and text to boost next-item hits","Unified prompt ties collaborative IDs and text, beating baselines","RSLLM fuses item IDs with text for better next-item prediction","Teaching LLMs a new language: recommender IDs lift hit ratio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2538,"prompt_tokens":891,"completion_tokens":1647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1571}},"tokens_in":507,"tokens_out":1647,"duration_ms":12303,"temperature":1.0,"reasoning_tokens":1571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:57:57.172839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the MovieLens, Steam, and LastFM comparisons with the same candidate protocol but report per-run HitRatio@1 and standard deviations, and additionally evaluate under full-corpus ranking where the true next item must be ranked among all items instead of a sampled 20-item set. If RSLLM's margins over LLaRA shrink to within noise or invert under full-corpus ranking, the central claim would be weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LastFM dataset used for evaluation."}],"review_version":1}