{"id":"aa568a40-300d-43c5-9bed-1d76e95fb30d","arxiv_id":"2608.08583","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding cosine-similarity and contrastive structure-preserving losses to the projection step makes LLM-based sequential recommenders use collaborative signals and improves Hit Ratio at 1 on LastFM and MovieLens.","lead":"Large language model recommenders often ignore the collaborative user-behavior numbers they are given, relying mostly on item titles. This paper adds two simple training penalties that keep the shape of the behavioral data intact when it is copied into the language model, and the result is a modest, mostly consistent boost in next-item ranking accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HR@1 is computed only over valid generations; counting invalid generations as failures removes Contrastive-LM's status as highest on the MovieLens GRU4Rec block.","rationale":"The reader's verdict is CONDITIONAL and I do not move it, but the load-bearing concern is different from the reader's formal weakest_assumption. The reader listed the valid-only HR@1 in the rationale, and the paper's Table 3 actually does measure baseline projector distortion (Direct/LLaRA have much lower Tau/Rho/SIM@10 than the proposed methods), so the attribution concern is weaker than the reader suggested. The metric issue is more concrete: a product-level recommendation that generates no valid candidate is a failure, and the paper's strongest quantitative claim is stated in valid-only terms. Recomputing with invalid-as-wrong does not overturn the general finding that the proposed objectives improve over baselines on LastFM and on the SASRec MovieLens block, so rejection is not warranted; it does mean the headline 'Contrastive-LM highest on both datasets' must be checked and likely qualified. This supports the existing CONDITIONAL verdict.","tokens_in":9589,"tokens_out":9133,"duration_ms":104871,"concrete_test":"Re-run the MovieLens (and LastFM) evaluation with five seeds and compute HR_all = N_correct/N_test for every row of Table 2 by multiplying the reported HR@1 by VR (or equivalently treating invalid generations as incorrect). Run paired t-tests on HR_all against the strongest baseline in each block, exactly as in Section 3.3. If Contrastive-LM is no longer the highest or is not significant on the MovieLens GRU4Rec block, revise the Section 3.3 and abstract wording to report all-test HR@1 (or both metrics) and qualify the 'highest on both datasets' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest load-bearing point is the evaluation metric. Section 3.2 defines HR@1 over valid generations only (HR@1 = N_correct/N_valid), with VR = N_valid/N_test. Since invalid generations are excluded from the denominator, a method can appear more accurate by generating valid titles only for easy cases. Table 2 shows Contrastive-LM (GRU4Rec block, MovieLens) has VR = 0.8842 versus 0.9895 for Direct and 0.9684 for LLaRA. If invalid outputs are counted as failures, HR_all = N_correct/N_test = HR@1 × VR: Contrastive-LM drops from 0.4881 to about 0.4316, below Cosine-LM's 0.4632 (0.4681 × 0.9895), while LLaRA is 0.4281. Thus the Section 3.3 sentence 'Contrastive-LM achieving the highest HR@1 on both datasets' no longer holds for the MovieLens GRU4Rec block under an all-test metric. The abstract's 'consistently improves' claim is more resilient, but the strongest quantitative statement and its significance depend on the valid-only denominator.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses modality bias in LLM-based sequential recommenders that integrate collaborative embeddings by projecting them into the LLM embedding space. The authors propose two structure-preserving objectives for the projector: cosine similarity preservation (matching pairwise item similarities between the original and projected spaces) and contrastive preservation (preserving sequence-level positive/negative discrimination). They instantiate these as Cosine-LM and Contrastive-LM, training the projector with a structure-aware pretraining stage followed by joint fine-tuning, and evaluate on LastFM and MovieLens with GRU4Rec and SASRec as collaborative backbones, comparing against Direct and LLaRA baselines. The paper reports HR@1 and valid ratio (VR), a shuffle-based diagnostic showing that the proposed models are sensitive to embedding corruption while baselines are not, and structure-preservation metrics (Kendall's Tau, Spearman's Rho, SIM@10). The central claims are that structure-preserving projection mitigates modality bias and consistently improves recommendation performance.","tokens_in":9807,"tokens_out":6949,"duration_ms":65362,"significance":"If the results hold, the paper offers a simple, backbone-agnostic modification to the projection module of LLM-based recommenders, with a plausible diagnostic for modality reliance. The comparisons are fair in an important sense: the same candidate sets, backbones, and training setups are used across methods, and alpha is tuned on the validation set in the standard way. I do not see a circularity problem: recommendation gains are measured on held-out candidates, and Table 3's metrics, although related to L_sim, are not identical to the training objective. The shuffle diagnostic is a useful idea, and the paper is honest enough to report marginal significance on MovieLens. However, the evaluation denominator issue and the absence of error bars and per-seed results materially weaken the strongest quantitative claims, and the causal attribution to 'projection-induced distortion' is not directly measured. The central empirical finding is likely defensible after a revision, but the current presentation overstates the strength of the evidence.","major_comments":[{"comment":"Section 3.2 defines HR@1 = N_correct/N_valid and VR = N_valid/N_test, so HR@1 is a conditional accuracy computed only over generations that uniquely match a candidate. This is load-bearing for the Section 3.3 claim that 'Contrastive-LM achieving the highest HR@1 on both datasets.' In the MovieLens/GRU4Rec block, Contrastive-LM has VR = 0.8842 versus Cosine-LM's 0.9895; when invalid generations are counted as failures (HR@1 × VR), Contrastive-LM drops to 0.4316, below Cosine-LM's 0.4632 and only marginally above LLaRA's 0.4281. Please report N_correct/N_test (or HR@1 × VR) as a primary metric for every LLM-based method, and state explicitly that ID-based methods in Table 2 are scored with VR=1, so the comparison mixes denominators.","section":"Section 3.2, Table 2"},{"comment":"The statistical support for the headline 'consistently improves' claim is thin on MovieLens, where the reported significance is only marginal (p<0.1). No standard deviations, per-seed values, or confidence intervals are reported anywhere in the paper, so the reader cannot judge the stability of the differences. Please provide full seed-level results with error bars, and clarify whether the paired t-tests are corrected for multiple comparisons across two backbones and two proposed methods.","section":"Section 3.3, Table 2"},{"comment":"The paper attributes baselines' modality bias to 'projection-induced distortion of collaborative structure' but never directly measures the distortion introduced by a baseline projector. The shuffle diagnostic in Figure 1 and Figure 2 and the structure-preservation metrics in Table 3 are computed only for the final trained models, not for the projector before the proposed losses are applied. Please add structure-preservation measurements for a projector trained with only the language-modeling objective, both before and after LLM fine-tuning. If that baseline projector already preserves Tau, Rho, and SIM@10 well, then the proposed losses may be improving recommendation through other mechanisms, and the causal story in Section 1 would need to be revised.","section":"Section 1, Figure 1, Table 3"},{"comment":"The synergy claim that Combined 'consistently outperforms both individual objectives' is supported only by Figure 3b, with no numerical HR@1/VR values and no significance tests. The 30%/70% phase boundaries and the alpha schedule also appear to be chosen on the validation set; please state the selection procedure explicitly and report the Combined numbers in a table with error bars, as done for the main results.","section":"Section 3.6, Figure 3"}],"minor_comments":[{"comment":"The distribution used to sample the negative item e_n is not specified; please state whether negatives are drawn in-batch or from the global item pool and whether more than one negative is used per sequence.","section":"Section 2.2, Eq. (4)"},{"comment":"It is unclear whether alpha = 0.4 and alpha = 0.8 are tuned separately per dataset and backbone or fixed across all runs; please clarify the selection protocol for alpha and for the Combined schedule.","section":"Section 2.3, Section 3.1"},{"comment":"Tau, Rho, and SIM@10 are reported as averages without any measure of spread; adding standard deviations or per-item percentiles would show whether Cosine-LM's higher preservation is stable across items or driven by a subset.","section":"Table 3"},{"comment":"The bar plots do not show error bars or significance annotations, making it difficult to judge the magnitude and reliability of the shuffling effects and the Combined gains.","section":"Figures 2 and 3"},{"comment":"The word 'novel' for the cosine-similarity preservation objective is overstated, as pairwise similarity matching is a standard regularizer; the contributions are better framed as the combination of the two objectives and the diagnostic analysis.","section":"Introduction, Contributions"},{"comment":"No code or data release is mentioned; for a results-driven paper of this type, providing code and trained checkpoints would substantially strengthen reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the main comparisons are fair in terms of candidate sets and backbones. The evaluation-denominator issue is real but fixable by reporting HR@1 × VR as an additional metric; my all-test recomputation shows that most of the claimed gains survive, so I do not think rejection is warranted. The absence of error bars and the marginal MovieLens significance are the main reasons I cannot recommend acceptance now. The paper also cites iLoRA, AFL, and S-DPO in the setup but does not compare against them; for a 6-page paper this is acceptable if explicitly scoped, but the authors should say so."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on LLM-based recommendation. The genuinely new piece is the embedding-shuffle diagnostic: shuffling projected collaborative embeddings at inference while keeping titles fixed barely moves HR@1 for Direct and LLaRA, which is a clean way to show those models treat the embeddings as soft prompts. The two regularizers (cosine-similarity preservation and contrastive preservation) are standard ingredients, but applied to the projection step in a sensible way. The paper shows they increase embedding-shuffle sensitivity and improve HR@1 over the relevant baselines.\n\nWhat is done well: comparisons share candidate sets and backbones; significance is tested with paired t-tests across five seeds; Table 3 directly shows the baseline projectors do distort the collaborative geometry, which supports the causal story better than the weakest-assumption note suggests. The combined curriculum and joint fine-tuning ablations are honest attempts to understand the method.\n\nThe soft spot is real and load-bearing. HR@1 is computed only over valid generations, so invalid generations lower VR but do not count against accuracy. In the MovieLens GRU4Rec block, Contrastive-LM has VR 0.8842 versus 0.9895 for Cosine-LM. Recompute HR over all test instances (HR@1 × VR) and Contrastive-LM drops to about 0.432, below Cosine-LM's 0.463 and barely above LLaRA's 0.428. The sentence that Contrastive-LM achieves the highest HR@1 on both datasets no longer holds for that block. The abstract's \"consistently improves\" claim is more resilient, but the strongest quantitative statement is tied to the valid-only denominator.\n\nOther soft spots are minor: MovieLens gains are only p<0.1, Combined is reported without significance, and no code or standard deviations are released. The attribution that modality bias comes from projection distortion is partly supported by Table 3 but not directly manipulated; that is a fair caveat, not a fatal one.\n\nI would send this to review. The diagnostic is worth publishing, the method is simple enough to reproduce, and the metric issue is fixable by reporting an all-test HR or at least discussing it. A serious referee can push on the denominator and the causal claim without needing to reject the core idea.","headline":"Useful diagnostic plus a plausible regularizer, but the strongest quantitative claim depends on a valid-only HR@1 denominator that flips one MovieLens block.","tokens_in":10346,"tokens_out":2336,"would_cite":true,"duration_ms":25494,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-based recommenders ignore projected collaborative embeddings; structure-preserving projection makes them use those embeddings and lifts HR@1 on both LastFM and MovieLens.","keywords":["sequential recommendation","large language models","modality bias","structure preservation","collaborative embeddings","contrastive learning","cosine similarity preservation","projection"],"falsifier":"Measure Kendall's Tau and SIM@10 between the original SASRec embeddings and the projected embeddings produced by Direct and LLaRA after training; if those baseline projectors already preserve collaborative structure about as well as Cosine-LM does while still showing near-zero embedding-shuffle sensitivity, then projection-induced distortion is not the cause of the modality bias.","tokens_in":9366,"feed_emoji":"📈","tokens_out":8820,"duration_ms":82271,"temperature":0.7,"pith_summary":"The paper claims that when collaborative item embeddings are projected into an LLM's embedding space, the projection can distort the relational geometry of those embeddings, and the LLM then largely ignores them, acting as a text-only recommender. To fix this, it adds two structure-preserving losses to the MLP projector: one that keeps pairwise cosine similarities between items unchanged, and one that keeps the projected sequence closer to the true next item than to sampled negatives. On LastFM and MovieLens, with both GRU4Rec and SASRec embeddings, the resulting Cosine-LM and Contrastive-LM beat the Direct and LLaRA baselines on HR@1, and Contrastive-LM achieves the highest HR@1 on both datasets. Shuffle diagnostics show the new models degrade when collaborative embeddings are shuffled but titles are kept, indicating the LLM now uses the collaborative structure instead of treating it as a soft prompt. A reader should care because this offers a mechanism, not just a recipe, for making LLM-based recommenders genuinely combine textual and collaborative signals.","feed_headline":"Cosine and contrastive losses make LLM recommenders use embeddings","feed_subtitle":"With structure-preserving projection, HR@1 rises on LastFM and MovieLens over Direct and LLaRA baselines.","key_machinery":"The load-bearing object is the MLP projector together with two auxiliary losses that constrain it. Cosine Similarity Preservation, $L_{\\text{sim}} = \\frac{1}{|B|^2}\\sum_{i,j \\in B}\\bigl(S^{\\text{proj}}_{ij} - S^{\\text{orig}}_{ij}\\bigr)^2$, forces the pairwise cosine-similarity matrix of the projected item embeddings to match that of the original collaborative embeddings. Contrastive Preservation, $L_{\\text{contrast}} = -\\log\\sigma(\\langle z_s,z_p\\rangle) - \\log(1-\\sigma(\\langle z_s,z_n\\rangle))$, keeps the projected sequence representation closer to the projected true-next-item embedding than to a projected random negative. Training first pretrains the projector with the chosen structure loss alone, then jointly fine-tunes the projector and the LLM with the structure loss added to the next-token objective; a three-phase curriculum that shifts from cosine weighting to contrastive weighting performs best. These losses are what keep the collaborative geometry alive through the projection, which is what lets the LLM attend to it.","core_discovery":"On the paper's own terms, the central discovery is that projection-induced distortion, rather than the LLM's architecture, is why projected collaborative embeddings contribute so little to LLM-based recommenders. An embedding-shuffle diagnostic on MovieLens shows the baseline behavior: shuffling collaborative embeddings while keeping titles fixed leaves HR@1 nearly unchanged for Direct and LLaRA ($\\Delta \\approx 0$), so those models are effectively running on text alone. Adding the two structure-preserving objectives reverses this pattern, and the intact models reach higher accuracy, with Contrastive-LM achieving the best HR@1 on both datasets (for example, $0.5511$ on LastFM and $0.4993$ on MovieLens with SASRec embeddings, versus $0.4721$ and $0.4531$ for Direct). A structure-preservation analysis using Kendall's Tau, Spearman's Rho, and SIM@10 shows that Cosine-LM preserves the original collaborative geometry far more faithfully, yet Contrastive-LM recommends better; the paper reads this as evidence that task-relevant discriminative structure matters more than global geometric fidelity.","pith_inferences":["If the attribution is right, embedding-shuffle sensitivity should be reported as a standard diagnostic for any LLM-based recommender that mixes text with projected embeddings, since it separates genuine multimodal use from text-only shortcuts.","A testable extension: the method's advantage over Direct should shrink as item text becomes more informative (for example, longer descriptions or richer metadata), because text alone would carry more of the signal the contrastive loss currently supplies.","The contrastive result suggests a general design principle for cross-modal projection: optimize the projector for the downstream discriminative task rather than for global isometry with the source space; this may transfer to other modality-alignment settings such as image-text or user-item alignment."],"forward_implications":["LLM-based recommenders can be made to genuinely use collaborative signals: after the fix, models degrade when collaborative embeddings are shuffled, meaning the embeddings carry information the LLM acts on.","Contrastive preservation is worth more than global geometric preservation for next-item accuracy, and combining both in a cosine-to-contrastive curriculum outperforms either alone.","The gains hold across two datasets and two collaborative backbones, and survive joint fine-tuning of the backbone, where structure-preserving methods improve more than the baselines.","Embedding-shuffle sensitivity becomes an available diagnostic: a recommender that changes little under embedding shuffle is evidence of modality bias, regardless of whether it uses text or embeddings."],"supporting_citations":[{"why":"LLaRA is the main projection-based baseline and the source of the datasets and evaluation setup the paper inherits.","marker":"[12]"},{"why":"CoLLM is the representative approach that projects collaborative embeddings into the LLM space, the target the paper diagnoses and improves.","marker":"[17]"},{"why":"SASRec supplies the pretrained collaborative embeddings used as the geometry to preserve and as a backbone in the main experiments.","marker":"[8]"},{"why":"GRU4Rec supplies the second collaborative backbone, showing the structure-preserving gains are not specific to one embedding model.","marker":"[7]"},{"why":"MovieLens is the dataset for the embedding-shuffle diagnostic that establishes the modality-bias phenomenon.","marker":"[6]"},{"why":"P-Tuning grounds the interpretation that projected embeddings in baselines behave like generic soft prompts rather than informative signals.","marker":"[13]"},{"why":"The vision-language modality-bias study supplies the analogy that models stay overconfident when non-textual inputs are corrupted.","marker":"[18]"}],"fun_headline_variants":["Structure-preserving losses stop LLM recommenders from ignoring embeddings","Projection distortion, not LLM architecture, limits collaborative embeddings","Contrastive loss beats cosine for preserving task-relevant structure","Fix modality bias: keep collaborative geometry in LLM projections","New projection method makes LLM recommenders actually use embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the observed modality bias is caused by the projection distorting the geometry of the pretrained collaborative embedding space, and that this original space is the right geometry to preserve; if the LLM ignores projected embeddings for some other reason, such as their position in the prompt, the training setup, or attention patterns, the causal story would be wrong even if the empirical gains remain.","fun_headline_variants_meta":{"raw":{"variants":["Structure-preserving losses stop LLM recommenders from ignoring embeddings","Projection distortion, not LLM architecture, limits collaborative embeddings","Contrastive loss beats cosine for preserving task-relevant structure","Fix modality bias: keep collaborative geometry in LLM projections","New projection method makes LLM recommenders actually use embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1222,"prompt_tokens":851,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":287}},"tokens_in":467,"tokens_out":371,"duration_ms":4154,"temperature":1.0,"reasoning_tokens":287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:30:22.267514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure Kendall's Tau and SIM@10 between the original SASRec embeddings and the projected embeddings produced by Direct and LLaRA after training; if those baseline projectors already preserve collaborative structure about as well as Cosine-LM does while still showing near-zero embedding-shuffle sensitivity, then projection-induced distortion is not the cause of the modality bias.","supporting_citations":[],"review_version":1}