{"id":"5255324e-a3aa-46af-9791-20d4c763b525","arxiv_id":"2502.03307","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"IRLLRec aligns LLM-generated textual intents with interaction-based intents via contrastive alignment and momentum distillation, improving recommendation accuracy on three datasets.","lead":"Recommender systems get a boost when a large language model summarizes why a user likes or dislikes items, and those summaries are aligned with patterns learned from click histories. The proposed IRLLRec framework reports consistent gains over LLM-enhanced baselines on three public datasets, with the largest gains for users who have few interactions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM intent summaries may include test-set reviews, so the reported gains could be leakage rather than true intent alignment.","rationale":"The reader's weakest assumption is exactly the load-bearing concern here. The paper's headline claim is an empirical performance claim, and that claim is only meaningful if the LLM-constructed textual features do not encode test labels. The paper does not document any split-aware preprocessing, and the prompt in Figure 8 explicitly consumes review text, which is the interaction label itself. This is not a stylistic or peripheral issue: IRLLRec's matching module is designed to align textual and interaction intents, so any overlap between the textual summary and the test interaction gives the model a direct shortcut to the test item. The cold-start analysis in Figure 3 is particularly exposed, because sparse users have few interactions and their LLM summaries can name every interacted item, including test items. I agree with the conditional verdict: the method is coherent and the code is public, but the central empirical claim must be re-validated with a clean, split-aware intent-construction protocol. The other issues noted by the reader (missing error bars, modified AlphaRec baseline, overstated significance) are secondary and addressable; the leakage question is the one that could invalidate the central result.","tokens_in":19707,"tokens_out":3188,"duration_ms":33349,"concrete_test":"In the released repository (https://github.com/wangyu0627/IRLLRec), trace whether the LLM intent-generation step (Eq. 4 and the Figure 8 prompt) runs before or after the train/validation/test split, and whether the 'purchased items' list for a user can contain items whose only interaction is in the test set. Then re-run the Amazon-book and Amazon-movie experiments with intent summaries built only from training interactions; if the Table 2 improvements over RLMRec shrink by more than half or lose significance, the leakage concern is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is empirical superiority over KAR, RLMRec, and AlphaRec (Table 2). That claim depends on the evaluation being free of label leakage. Section 3.1 defines textual user intent as I_u = Aggregate over all items I_u in the user's interaction set (Eq. 4), and Figure 8 shows the LLM prompt includes each purchased item's title and review text. Section 4.1.1 describes only k-core filtering and random splits (3:1:1 or 8:1:1); it never states that validation/test interactions are excluded from the LLM prompt construction. If the prompts are generated from the full interaction set before splitting (as is common in this line of work), then for any user the textual intent embedding contains the titles and reviews of items that later appear in the test set. Because IRLLRec aligns text and interaction intents via contrastive loss and makes predictions from the inner product of user/item representations (Eq. 1), the user embedding can directly encode test item identities. This would inflate Recall/NDCG, and the effect should be largest for the low-interaction 'cold-start' users emphasized in Figure 3 and Section 4.2.2. The same protocol may affect the text-enhanced baselines, but IRLLRec's explicit text-interaction matching could exploit the leaked titles more aggressively, so the relative superiority claim itself is at risk. The released code makes this directly checkable; without a statement or code showing split-aware intent construction, the empirical claim is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IRLLRec, a model-agnostic framework that combines LLM-constructed textual intents with interaction-based intents for collaborative filtering. The method uses a dual-tower architecture with a text encoder and a graph encoder, pairwise and translation alignment losses to bring the two intent spaces together, and an interaction-text matching module based on momentum distillation. Experiments on Amazon-book, Yelp, and Amazon-movie report consistent Recall and NDCG improvements over KAR, RLMRec, AlphaRec, and several base models, with claimed statistical significance.","tokens_in":20047,"tokens_out":3276,"duration_ms":29017,"significance":"If the empirical findings are valid, IRLLRec would be a useful contribution to intent-based and LLM-enhanced recommendation, showing that fine-grained textual intents can be aligned with interaction intents in a plug-in manner. The paper provides a public code link and uses standard public datasets. The main value is the proposed combination of existing techniques (contrastive alignment, translation alignment, momentum distillation) in a model-agnostic package; the individual components are not entirely new, but their integration is clearly presented. However, the significance hinges on the evaluation being free of leakage and on the statistical claims being accurate.","major_comments":[{"comment":"Potential test-set leakage: Eq. (4) defines the textual user intent I_u as the aggregate over all items in I_u, and Figure 8 shows that the LLM prompt includes each purchased item's title and review text. The dataset split described in Section 4.1.1 is purely random (3:1:1 or 8:1:1), and the paper never states that validation/test interactions are excluded from the prompt construction. If the prompts are generated from the full interaction set before splitting, then for a typical user the textual intent embedding contains titles and reviews of items that later appear in the test set. Since IRLLRec aligns textual and interaction intents via contrastive loss and scores recommendations by the inner product of user and item representations (Eq. 1), this could directly leak test item identities into the user embedding. The reported gains, especially for the low-interaction groups in Figure 3, could then reflect leakage rather than true intent alignment. The authors must either confirm explicitly (in text or code) that LLM prompts use only training interactions, or re-run the experiments with split-aware intent construction and report the results. This is load-bearing for the central empirical claim.","section":null},{"comment":"The text claims that IRLLRec shows 'significant improvements across all metrics', but Table 2 contradicts this. In the SGL block on Yelp, IRLLRec has no asterisk for Recall@5 (0.0465) and NDCG@20 (0.0805), and the Recall@5 improvement is reported as -0.43% relative to RLMRec-Gen. Moreover, several other cells in the same block lack asterisks. The significance claim should be corrected to match the table, and the authors should explain which comparisons are significant and which are not.","section":"Table 2, Section 4.2.1"},{"comment":"The authors state that AlphaRec's poor performance 'may stem from our adherence to fairness, where we use RLMRec's profile representation as the item embedding instead of the proposed item title.' This means the reported AlphaRec numbers are not produced by the original AlphaRec method, but by a modified variant. Comparing against a modified, possibly suboptimal version of a baseline weakens the claim that IRLLRec outperforms AlphaRec. The authors should either run the original AlphaRec implementation or clearly label this as a variant and justify why the modification is necessary.","section":"Section 4.2.1"},{"comment":"No measure of variance is reported. The paper states that results are averaged over 5 runs and places asterisks for statistical significance, but no standard deviations, confidence intervals, or details of the significance test (e.g., paired t-test or Wilcoxon, what is paired, across which runs) are provided. Without this information, the significance asterisks cannot be independently verified. Please report standard deviations, at least for the main results in Table 2, and describe the test procedure in Section 4.1.3.","section":"Section 4.2.1, Table 2"}],"minor_comments":[{"comment":"In the definition of the translation alignment perturbation, the second noise variable is typeset as 𝝐𝑟 instead of 𝝐𝑧, which is confusing since both modalities are perturbed independently.","section":null},{"comment":"The individual losses L_user_pair, L_pos_pair, and L_neg_pair are introduced but never explicitly defined. Please specify how each is computed from Eq. (13), or clarify that they are obtained by applying the same InfoNCE loss to different sample types in the batch.","section":null},{"comment":"The loss L_ITM uses 'L_tran' without a definition; presumably it is the sum of L_text_tran and L_inter_tran from Eq. (15), but this should be stated explicitly.","section":null},{"comment":"The phrase 'cold-start (not strictly defined)' is informal for a published paper. Since the sparsity analysis is used to support a specific claim about cold-start users, please define the user groups precisely (e.g., by interaction count ranges) rather than relying on an informal label.","section":null},{"comment":"The paper evaluates five base models but does not describe how IRLLRec modifies each base model's training objective beyond adding the auxiliary losses. A short paragraph stating whether the base model's original loss is retained unchanged, and how the intent alignment loss is combined with it, would improve reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the potential leakage through LLM prompt construction; this is the kind of flaw that, if confirmed, would invalidate the main empirical claim. The paper's own description (Eq. 4 and Figure 8) makes the risk concrete, and the current text offers no mitigation. I would encourage the editor to ask for either a clear statement with code inspection showing that prompts are built only from training interactions, or a re-run with a split-aware construction. The other issues (significance claim vs. Table 2, modified AlphaRec baseline, missing variance) are fixable but should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you spend time on this. The architecture is a well-built combination of known blocks: CoT-summarized textual intents, learnable intent prototypes, InfoNCE-style alignment, and momentum distillation for text-interaction matching. That specific composition is new, and the paper does several things well: it releases code, tests across five backbones on three datasets, includes ablations and hyperparameter sensitivity, and the gains (roughly 1-15% over LLM-enhanced baselines) are believable in magnitude. The cold-start analysis is a nice touch. This deserves a serious referee. The soft spots are real but addressable. The biggest is potential leakage. User intents are defined as an aggregate over the user's interaction set (Eq. 4), and the prompt in Fig. 8 explicitly includes each purchased item's review text. Section 4.1.1 only mentions k-core filtering and random splits; it never states that validation/test interactions are excluded from the LLM prompt. If prompts are built before splitting, the user embedding can directly encode test item identities, and the cold-start gains in Fig. 3 would be inflated. This is checkable with the released code, but the paper needs to state it clearly and, if the protocol does leak, rerun. Without that, the central empirical claim is not fully supported. Second, the AlphaRec baseline is altered. The paper says they used RLMRec's profile representation as the item embedding instead of AlphaRec's proposed item title, \"for fairness.\" That is not the published AlphaRec, and it likely handicaps it. The comparison should be redone with the original setup, or the justification strengthened substantially. Third, the significance claims are overstated. Table 2 has several cells without stars (e.g., SGL on Yelp: Recall@5, NDCG@5, NDCG@20), yet the text claims \"significant improvements across all metrics\" and mentions only one exception. Also no standard deviations or test details are given, so the pairwise significance tests are hard to verify. Minor: the time complexity analysis writes O((M+N)^2 d) for the loss, which should depend on batch size, not the full user/item count. Sloppy but not fatal. The math in Eqs. 5-6 is motivational rather than operational. Overall: the paper is interesting and the experiments are extensive, but the leakage question and the altered baseline need to be resolved before the superiority claim is accepted. I would send it to peer review with a request to address these points. If the leakage is confirmed, major revision; if not, it's a solid contribution.","headline":"Competently engineered intent-alignment framework with credible but not fully verified gains; the leakage risk and the altered AlphaRec baseline need to be settled before the empirical claim is accepted.","tokens_in":782,"tokens_out":900,"would_cite":false,"duration_ms":40253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligning LLM-written intent summaries with interaction intents consistently improves graph-based recommenders across three datasets.","keywords":["intent representation learning","large language models for recommendation","multimodal intent alignment","momentum distillation","collaborative filtering","contrastive learning","text-enhanced recommendation","graph neural networks"],"falsifier":"Inspect the released code or rerun the pipeline with a temporal split (train on early reviews, test on later ones) and with user prompts built only from training-set reviews; if the Recall and NDCG advantages over RLMRec disappear or shrink sharply, the central superiority claim was driven by test information leaking into the LLM summaries.","tokens_in":19528,"feed_emoji":"🎯","tokens_out":6150,"duration_ms":54386,"temperature":0.7,"pith_summary":"The paper sets out to show that recommenders get better when the latent reasons behind user-item interactions, called intents, are modelled in two modalities at once. A large language model reads user reviews and item descriptions to write a short intent summary of what the user likes and dislikes, while a graph encoder learns interaction intents from the collaborative signal. IRLLRec is the model-agnostic wrapper that aligns these two representation spaces and matches them, and the paper reports consistent Recall and NDCG gains over strong LLM-enhanced baselines on Amazon-book, Yelp, and Amazon-movie. The appeal is practical: if the claim is right, any graph-based recommender can be improved by plugging in intent alignment without redesigning the base model.","feed_headline":"LLM intent alignment lifts graph recommenders on 3 datasets","feed_subtitle":"Matching LLM-written intent summaries with interaction intents improves Recall and NDCG across five base models.","key_machinery":"The central object is the multimodal intent pair $\\mathbf{z}$ and $\\mathbf{r}$: a textual intent embedding $\\mathbf{z}$ produced by an LLM summary and a linear text encoder, and an interaction intent embedding $\\mathbf{r}$ obtained by weighting $K$ intent prototypes with softmax attention over graph embeddings. Two mechanisms do the work: Intent Alignment, which combines a pairwise contrastive loss with translation alignment that adds Gaussian noise to each modality and treats the diagonal of the similarity matrix as positive pairs, and Interaction-text Matching, which uses momentum distillation with an EMA-updated teacher encoder to generate pseudo-labels for student matching scores. This teacher-student setup is what lets the model pick out the latent key intents among noisy text and interactions.","core_discovery":"On the paper's own terms, the discovery is that intent is not a single hidden variable but a pair of complementary views that must be aligned and matched. Textual intents are produced by chain-of-thought prompts that aggregate the positive and negative attributes of every item a user interacted with, while interaction intents are computed as an expectation over K learned intent prototypes on LightGCN embeddings. The paper's contribution is a dual-tower framework in which pairwise InfoNCE alignment and Gaussian-noise translation alignment bring the two spaces together, and Interaction-text Matching with momentum distillation learns which pieces of text correspond to which interaction intents. Across five base models and three datasets, the framework reports statistically significant improvements over RLMRec, KAR, and AlphaRec, with the largest gains appearing in the sparsest user groups.","pith_inferences":["A temporal split or a prompt-construction audit would settle whether the reported gains come from genuine intent alignment or from test reviews leaking into the LLM summaries through the aggregation in Eq. (4).","The same dual-tower alignment recipe could be applied to other side information, such as images or audio, where a second modality carries preference signals not visible in the interaction graph.","Because the matching module is borrowed from vision-language pretraining, the framework suggests that recommendation can borrow more cross-modal matching machinery from that literature.","Formalizing the expectation in Eqs. (5)-(6) may yield a variational bound that predicts when intent prototypes help, turning the empirical recipe into a testable model-selection rule."],"forward_implications":["Any graph-based collaborative filtering model can be wrapped with IRLLRec's intent alignment and matching modules and should show higher Recall@20 and NDCG@20 on review-rich datasets.","Sparse and cold-start users benefit most, because the LLM's textual intent carries preference signal before enough interactions accumulate.","The framework makes recommendations more interpretable: the aligned textual intent is a readable natural-language statement of why a user or item behaves as it does.","Better text-embedding models directly translate into better recommendations, since the framework is bottlenecked by the quality of the LLM intent representation.","Removing either the alignment or the matching module degrades performance below the base model, so the two losses are jointly load-bearing."],"supporting_citations":[{"why":"Supplies the text-enhanced recommendation baseline, data processing, and the idea of aligning semantic and interaction spaces.","marker":"[29]"},{"why":"Provides the LightGCN graph encoder used to produce interaction embeddings and served as the main base model.","marker":"[13]"},{"why":"Defines the LLM knowledge-augmented baseline KAR that IRLLRec must outperform.","marker":"[47]"},{"why":"Defines the language-embedding baseline AlphaRec that IRLLRec must outperform.","marker":"[36]"},{"why":"Supplies the momentum-distillation teacher-student matching mechanism used in Interaction-text Matching.","marker":"[18]"},{"why":"Provides the InfoNCE contrastive objective used for pairwise cross-modal alignment.","marker":"[6]"},{"why":"Provides the chain-of-thought prompting method used to elicit structured intent summaries from LLMs.","marker":"[44]"},{"why":"Serves as a graph contrastive base model showing IRLLRec generalizes beyond LightGCN.","marker":"[46]"},{"why":"Serves as a noise-resistant graph contrastive base model on which IRLLRec reports consistent gains.","marker":"[49]"},{"why":"Provides a disentangled intent baseline and the intent-prototype formulation that underpins Eqs. (5)-(9).","marker":"[31]"}],"fun_headline_variants":["LLM intent alignment lifts recommenders on sparse data","Aligning LLM text intents with interaction intents boosts recall","Dual-tower alignment of multimodal intents for recommendations","Momentum distillation matches textual and interaction intents","IRLLRec: LLM-driven intent alignment for sparse-user recs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the LLM intent summaries are generated from training interactions only, so that reviews assigned to the test split never influence the prompts; if they do, the reported improvements could reflect leakage rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["LLM intent alignment lifts recommenders on sparse data","Aligning LLM text intents with interaction intents boosts recall","Dual-tower alignment of multimodal intents for recommendations","Momentum distillation matches textual and interaction intents","IRLLRec: LLM-driven intent alignment for sparse-user recs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000868,"raw_usage":{"total_tokens":3758,"prompt_tokens":938,"completion_tokens":2820,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":2746}},"tokens_in":554,"tokens_out":2820,"duration_ms":19520,"temperature":1.0,"reasoning_tokens":2746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:12:23.086740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released code or rerun the pipeline with a temporal split (train on early reviews, test on later ones) and with user prompts built only from training-set reviews; if the Recall and NDCG advantages over RLMRec disappear or shrink sharply, the central superiority claim was driven by test information leaking into the LLM summaries.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the text-enhanced recommendation baseline, data processing, and the idea of aligning semantic and interaction spaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LightGCN graph encoder used to produce interaction embeddings and served as the main base model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LLM knowledge-augmented baseline KAR that IRLLRec must outperform."},{"cited_title":"Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi","cited_arxiv_id":null,"evidence_quote":"Supplies the momentum-distillation teacher-student matching mechanism used in Interaction-text Matching."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the InfoNCE contrastive objective used for pairwise cross-modal alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the chain-of-thought prompting method used to elicit structured intent summaries from LLMs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as a noise-resistant graph contrastive base model on which IRLLRec reports consistent gains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a disentangled intent baseline and the intent-prototype formulation that underpins Eqs. (5)-(9)."}],"review_version":1}