{"id":"233a2003-eedb-4de9-a68c-fd3e9a43b219","arxiv_id":"2502.09058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLaRD uses LLM-generated preference and relation knowledge plus an information-bottleneck objective to denoise implicit feedback and improve recommendation accuracy on Steam, Yelp, and Amazon-Book.","lead":"This paper presents LLaRD, a recommender-system framework that uses large language models to label noisy user-item interactions and to add inferred preference links before training a graph-based recommender. The authors report consistent gains over prior denoising methods on three datasets and two backbone models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation leakage risk: relation knowledge (Steps 3-4, Eqs. 10-13) is derived from graph neighborhoods that may include validation/test interactions, and the InfoNCE alignment Eq. 20 lets those held-out edges steer the trained representations, plausibly inflating the reported gains.","rationale":"The paper is a coherent, well-specified system with consistent empirical gains, ablations, robustness checks, and a promised code release. I read the central claim in good faith: LLaRD uses LLM knowledge and an information-bottleneck objective to improve denoising, and the experiments are designed to show this across datasets and backbones. The reader's weakest assumption was that the LLM's hard denoising labels are reliable ground truth. I agree that label reliability is unvalidated, but the more acute and falsifiable risk is evaluation leakage: the LLM's relation-knowledge generation draws on graph neighborhoods, and nothing in Sections 3.2, 4.1, or Appendix C states that these neighborhoods are restricted to the training split. If test/validation edges enter G_rel, then Eq. 20 provides a direct gradient path from held-out positives into the learned user representations, which would inflate every headline number in Table 1. This concern is concrete, checkable, and specific to the central claim. It does not require assuming author misconduct; it is a standard protocol ambiguity that the paper does not resolve. The recommended verdict remains CONDITIONAL, but the condition should be sharpened: the authors must demonstrate that knowledge generation is blind to the validation/test splits, or quantify the overlap of LLM-generated edges with held-out positives. If the training-only rerun shows no meaningful drop and no significant overlap, the central claim survives and the paper's contribution stands.","tokens_in":22881,"tokens_out":6173,"duration_ms":66779,"concrete_test":"Re-run Table 1 with a training-only knowledge-generation protocol: in Eq. 10 set U_{i_k} to users who interacted with i_k in the training split; in Eq. 12 set I_{u_k} to items interacted by u_k in the training split; delete any edge in ENoise, ECollab, or EInterests whose user/item pair appears in the validation or test splits before building G_rel. Then compare the average relative gain over BOD with the numbers in Section 4.2 and compute the precision@K overlap of EInterests with held-out test positives. If the adjusted gain drops by more than 20% relative, or the overlap is significantly above the random expectation, the reported superiority is substantially driven by test leakage rather than denoising.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that LLaRD's LLM-generated relation knowledge causes the consistent gains in Table 1. The load-bearing assumption is that this knowledge is derived without access to held-out positives. The paper never states that the LLM in Section 3.2 sees only the training split. Step 3 defines N(2)_u = union_{i_k in N_u(high)} U_{i_k} \\ {u} using 'users who have interacted with item i_k', and Step 4 defines N(3)_u = union over N(2)_u(High) I_{u_k} \\ N(1)_u using 'items interacted by users'; both are taken over the original graph, and Appendix C.1 only describes the 3:1:1 split for evaluation, not for knowledge generation. If the original graph includes validation/test edges, then EInterests can contain a test-positive item for u. That edge is inserted into G_rel (Eq. 15) and, after GNN propagation on G_rel, h_hat_u in Eq. 20 aggregates the test item's embedding. The InfoNCE loss L_rel pulls h'_u toward h_hat_u, so at test time y'_ui = h'_u^T h'_i for that held-out item is inflated. The same channel can carry validation positives into the model through ECollab even without direct user-item edges. This is more direct than the reader's label-reliability concern: even if LLM labels are reasonable, the evaluation protocol must guarantee they were not produced from test information; the paper provides no such guarantee.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLaRD, a framework that uses large language models (LLMs) to generate two types of denoising knowledge for collaborative filtering: preference knowledge derived from user/item text and relation knowledge derived from a user-centric chain-of-thought reasoning process over interaction graphs. The relation knowledge consists of identified noise edges, collaborative user links, and interest item links, which are combined into an enriched graph. A knowledge-enhanced denoising module then applies an information-bottleneck objective, combining BPR loss, InfoNCE alignment with preference and relation knowledge, and an HSIC-based compression term. Experiments on Steam, Yelp, and Amazon-Book with GMF and LightGCN backbones report consistent improvements over denoising baselines, with average relative gains over BOD of about 6.92% for GMF and 11.79% for LightGCN, plus ablation, robustness, and cold-start analyses. The code is publicly available.","tokens_in":23230,"tokens_out":3221,"duration_ms":32548,"significance":"If the central claim holds, LLaRD is a timely and practical contribution: it is one of the first systematic attempts to use LLM-generated knowledge specifically for data-level denoising in recommendation, and it combines semantic knowledge with graph-based collaborative reasoning in a way that goes beyond representation-level LLM alignment such as RLMRec. The release of code, the use of the all-rank evaluation protocol, and the inclusion of ablation, noise-robustness, and cold-start experiments are concrete strengths. However, the significance is conditional on two load-bearing points that the manuscript does not currently establish: the LLM-generated relation knowledge must be derived without access to held-out test/validation interactions, and the LLM's hard noise and interest labels must be reliable enough to guide denoising. Both points affect the interpretation of every reported gain in Table 1.","major_comments":[{"comment":"The evaluation may be optimistic because the relation-knowledge generation is never explicitly restricted to the training split. In Step 3, N(2)_u is defined as the union over U_i_k for items rated High, where U_i_k denotes 'users who have interacted with item i_k'; in Step 4, N(3)_u is defined as the union over I_u_k for second-hop users, where I_u_k denotes 'items interacted by users'. These are taken over the interaction graph without specifying whether validation/test edges are excluded. Appendix C.1 only states that the data are divided into training, validation, and testing with a 3:1:1 ratio; it does not state that the LLM in Section 3.2 sees only the training part. If the original graph includes validation or test edges, then E_Interests can contain a test-positive item for user u, and that edge is inserted into G_rel in Eq. (15). After GNN propagation on G_rel, h_hat_u in Eq. (20) aggregates the test item's embedding, and the InfoNCE loss pulls h'_u toward h_hat_u, inflating the test-time score y'_ui for that held-out item. The same channel can carry validation positives through ECollab even without direct user-item edges. The paper must state clearly that all LLM knowledge generation (Steps 1-4) uses only the training subgraph, and ideally verify empirically that no generated interest/collab edge overlaps the validation/test positive set. This is load-bearing for the central claim of consistent gains.","section":"Section 3.2.1, Eqs. (10)-(15), and Eq. (20)"},{"comment":"The empirical support is weakened by the absence of any variance or significance information. Table 1 reports a single run per configuration, with no error bars, no standard deviations, and no significance tests. Several of the claimed gains are numerically small; for example, on Yelp with GMF, LLaRD's R@10 is 0.0708 versus BOD's 0.0706, and N@10 is 0.0578 versus 0.0574. A difference of 0.0002-0.0004 can easily be within run-to-run noise, especially for deep models with Gumbel-Softmax sampling. Because the paper's abstract and Section 4.2 state that LLaRD 'consistently outperforms' all baselines, the authors should report results over multiple random seeds with standard deviations and, where feasible, paired significance tests against the strongest baseline BOD.","section":"Section 4.2, Table 1"},{"comment":"The hard LLM noise labels are treated as ground truth without any validation. The set E_Noise is formed by the LLM's binary decisions and then removed from the enriched graph in Eq. (15); if the LLM mislabels genuine interactions as noise, the model is trained on a corrupted graph, and the InfoNCE alignment in Eq. (20) reinforces the error by aligning the denoised graph with a relation knowledge graph that already reflects that mislabeling. The paper provides no analysis of label quality: no comparison of E_Noise against known noisy interactions, no human evaluation of a sample of noise labels, and no robustness test with perturbed or partially incorrect labels. Since the denoising contribution is the central novelty, the reliability of these labels is a load-bearing assumption. I recommend adding a label-quality study (e.g., precision against a manually annotated sample, or an ablation that flips a fraction of noise labels) to support the claim that the LLM knowledge improves, rather than merely changes, the denoising signal.","section":"Section 3.2.1, Step 2 and Eqs. (8)-(9)"}],"minor_comments":[{"comment":"Equation (18) defines y'_ui but not y'_uj; the reader must infer that y'_uj = h'^T_u h'_j. Please add the definition for the negative item to make the BPR loss self-contained.","section":"Section 3.3.2, Eq. (18)"},{"comment":"The notation T_u = {T^1_u, T^2_u, ..., T^{|U|}_u} is confusing: the superscript k on T^k_u appears to index users rather than a sequence for a fixed user, and the size |U| suggests one entry per user. Please clarify the indexing, since the same symbol is later used for token-embedding matrices T_u.","section":"Section 3.1, Eq. (2)"},{"comment":"The ablation study in Table 2 is reported only for Amazon-Book and Steam, not for Yelp. Since Table 1 includes Yelp and the method is claimed to work across all three datasets, adding the Yelp ablation (or explaining its omission) would strengthen the completeness of the study.","section":"Section 4.3, Table 2"},{"comment":"The hyperparameter description is incomplete: the values of alpha and beta from Eq. (25), the kernel bandwidths sigma_k and sigma_m from Eq. (21), and the contrastive temperature tau' from Eqs. (19)-(20) are not reported. These are free parameters that affect the reported results, so they should be listed for reproducibility.","section":"Appendix C.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of WWW and the central idea is interesting, but the evaluation-leakage question in Section 3.2.1 is the kind of issue that should be resolved before acceptance; if the authors cannot rule out that test/validation edges enter the LLM-generated relation knowledge, the reported gains may be substantially inflated. The lack of statistical significance reporting is also concerning given the small margins over BOD on Yelp. I would encourage the editor to send the revised manuscript back to the same reviewers to verify that the new experiments use a train-only graph for knowledge generation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a referee slot, but the evaluation protocol has a hole that needs patching before I'd trust the gains.\n\nWhat's actually new: LLaRD is a clearly specified framework that uses LLM semantic profiling and a multi-hop chain-of-thought step to generate hard denoising labels (noise, collaborative users, interest items), then aligns that knowledge through an information-bottleneck-style objective. Using LLMs for data-level denoising rather than just representation augmentation is a reasonable step forward. The ablations cover the main components, the running time is reported, and the code is promised. That earns credit.\n\nThe soft spots, in order of seriousness. First, the leakage risk: the relation knowledge in Steps 3-4 is computed over graph neighborhoods that are not explicitly stated to be training-only. Appendix C.1 describes the 3:1:1 split only for evaluation, not for knowledge generation. If the LLM sees validation or test edges, EInterests can contain test positives, and the InfoNCE alignment in Eq. 20 can pull training representations toward held-out items. That would directly inflate Recall and NDCG. This is the first thing I'd ask the authors: state exactly which graph is used for knowledge generation and re-run with a training-only graph.\n\nSecond, there's an internal inconsistency between the masked graph G' and the BPR loss. Eq. 18 defines D over the original positives, while the model is trained on G'. If dropped edges remain in D, the model is still asked to rank them as positives, which undercuts the denoising story. The authors need to clarify whether D is restricted to E'.\n\nThird, Table 1 reports single-run metrics with no error bars or significance tests. That is common in this subfield, but it makes the 7-12% gains harder to trust. Fourth, the LLM's noise and interest labels are never checked against any ground truth. A small human-annotated sample would help.\n\nThe central idea is plausible and the framework is coherent; none of these flaws is fatal if addressed. The leakage concern is the load-bearing one, and it is fixable by either confirming the current protocol is training-only or rerunning with that guarantee.\n\nWho this is for: people working on denoising recommendation, LLM-augmented collaborative filtering, and graph contrastive learning. I'd send it to peer review, but insist that the knowledge-generation split be stated explicitly and the BPR/mask inconsistency be resolved before acceptance.","headline":"LLaRD is a thoughtful LLM-knowledge-plus-information-bottleneck denoising framework with consistent gains, but the paper never shows its LLM-generated relation graph is built from the training split only, which could inflate the headline numbers.","tokens_in":23784,"tokens_out":2959,"would_cite":false,"duration_ms":30174,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Denoising recommendation is best done, the paper claims, by aligning LLM-generated preference and relation knowledge to recommendation targets through an information bottleneck, beating existing denoising methods on three datasets and two…","keywords":["large language models","denoising recommendation","information bottleneck","chain-of-thought reasoning","collaborative filtering","implicit feedback","graph neural networks","knowledge-enhanced recommendation"],"falsifier":"Run LLaRD on a dataset with known injected noise and known held-out positives, then compute precision and recall of the LLM's $E_{Noise}$ against the injected noise and the overlap of $E_{Collab} \\cup E_{Interests}$ with the held-out positives. Low precision on known noise, or overlap with test positives far above chance, would mean the gains come from label errors or leakage rather than denoising.","tokens_in":22679,"feed_emoji":"🎯","tokens_out":11750,"duration_ms":105908,"temperature":0.7,"pith_summary":"The paper aims to show that large language models can perform the hard part of recommendation denoising: deciding which observed clicks are noise and which unobserved items a user would genuinely like. It proposes LLaRD, which first uses an LLM to enrich user and item profiles into preference knowledge, then runs a chain-of-thought procedure over the user-item graph to produce relation knowledge—noise labels, similar-user links, and latent-interest items—and finally applies an information-bottleneck objective so that only LLM knowledge aligned with recommendation targets influences the denoised graph. The claimed payoff is practical: on Steam, Yelp, and Amazon-Book, with both GMF and LightGCN backbone models, LLaRD outperforms the comparison denoising methods, including the second-best method BOD by average relative gains of roughly 6.92% with GMF and 11.79% with LightGCN. A sympathetic reader would care because existing denoising methods lean on costly side information or on heuristics from observed interactions that fail when data is sparse; LLMs bring external world knowledge that can infer preferences beyond what the interaction log shows.","feed_headline":"LLaRD lifts recommendation accuracy over all denoising baselines","feed_subtitle":"Language-model knowledge, bottleneck-filtered, beats the second-best denoiser by ~7% on GMF and ~12% on LightGCN on average.","key_machinery":"The load-bearing object is the enriched interaction graph $G_{rel}$ built from the LLM's chain-of-thought outputs, together with the information-bottleneck training objective that decides how much of that generated knowledge survives. The chain-of-thought procedure is user-centric: it rates each interacted item High/Medium/Low, labels Low items that do not match the user's profile as noise, finds second-hop users with similar High-rated items, and proposes third-hop interest items, so the graph-level reasoning is decomposed into steps the LLM can handle. The objective maximizes mutual information between the denoised graph and both the recommendation targets and the LLM knowledge, while minimizing mutual information with the original noisy graph, using BPR loss, InfoNCE (a contrastive alignment loss), and HSIC (a kernel-based independence measure) as tractable surrogates. This lets the framework keep useful LLM knowledge and discard hallucinated or irrelevant knowledge during training.","core_discovery":"LLaRD's central claim is that denoising should be treated as a knowledge-generation problem rather than only a loss-design problem. The LLM produces preference knowledge from profiles, keywords, and item text, and relation knowledge from a user-centric chain-of-thought over the interaction graph, yielding three edge sets: $E_{Noise}$ to remove, $E_{Collab}$ to add between similar users, and $E_{Interests}$ to add to latent-interest items. These form the enriched graph $G_{rel} = (U, I, (E - E_{Noise}) \\cup E_{Collab} \\cup E_{Interests})$. A learnable mask generator on the original graph produces the denoised graph $G'$, and the information-bottleneck objective $\\max_{G'} I(R; G') + \\alpha I(K_p, K_r; G') - \\beta I(G'; G)$ is implemented with BPR loss, InfoNCE alignment losses, and an HSIC compression term. The paper reports that this consistently beats mainstream denoising baselines across all three datasets and both backbones, with average relative gains over BOD of about 6.92% on GMF and 11.79% on LightGCN.","pith_inferences":["A direct consequence the paper leaves implicit is that if the LLM's denoising labels are accurate, the cleaned graph can be consumed by any future backbone, decoupling denoising from architecture-specific loss design.","The paper does not check whether the LLM's added interest and collaborative edges overlap with held-out test positives; until such a leakage check is reported, the margin over BOD should be read with that caveat.","A natural extension is to feed the LLM's confidence scores into the learnable mask instead of using the labels only through the contrastive enriched view, letting the information bottleneck downweight uncertain judgments rather than treating them as binary facts."],"forward_implications":["LLaRD outperforms the second-best baseline BOD by average relative gains of about 6.92% with GMF and 11.79% with LightGCN across Steam, Yelp, and Amazon-Book.","Removing the information-compression term (w/o MI_min) or the knowledge-alignment term (w/o MI_max) degrades performance, so both the bottleneck filtering and the LLM knowledge contribute to the result.","The LLM knowledge is generated in a one-time offline pass, keeping per-epoch training cost comparable to mainstream denoising methods despite the added graph and contrastive computations.","Cold-start users with sparse histories also improve, with LLaRD reporting consistent gains over baselines across all five sparsity groups on Amazon-Book and Steam."],"supporting_citations":[{"why":"Supplies the LightGCN backbone on which LLaRD is evaluated.","marker":"[20]"},{"why":"Supplies the GMF backbone used as the second evaluation backbone.","marker":"[31]"},{"why":"RLMRec, the nearest LLM-based representation-learning baseline; LLaRD extends its preference-knowledge idea to data-level denoising.","marker":"[38]"},{"why":"BOD, the strongest comparison baseline, against which LLaRD reports its main relative gains.","marker":"[59]"},{"why":"T-CE, a reweighting denoising method representing the interaction-data-driven family LLaRD claims to surpass.","marker":"[55]"},{"why":"SGDL, a self-guided denoising baseline in the comparison set.","marker":"[15]"},{"why":"Supplies the InfoNCE contrastive loss used to align the denoised graph with LLM knowledge.","marker":"[18]"},{"why":"Provides the graph information bottleneck framing that motivates the compression objective.","marker":"[49]"},{"why":"CGI, the prior IB-based denoising method whose objective LLaRD adapts with LLM knowledge.","marker":"[61]"},{"why":"Supplies the chain-of-thought prompting technique that the user-centric relation-knowledge reasoning is built on.","marker":"[62]"}],"fun_headline_variants":["LLM knowledge beats noise: LLaRD tops denoising baselines","LLaRD: LLM-generated knowledge cleans up recommendations","Turn noise into knowledge: LLaRD boosts rec accuracy","LLaRD: LLM-powered graph denoising lifts rec performance","LLaRD: chain-of-thought plus info bottleneck beats denoisers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method trusts the language model's judgments about which interactions are noise and which new links reflect real interest, and the paper does not check those judgments against ground truth.","fun_headline_variants_meta":{"raw":{"variants":["LLM knowledge beats noise: LLaRD tops denoising baselines","LLaRD: LLM-generated knowledge cleans up recommendations","Turn noise into knowledge: LLaRD boosts rec accuracy","LLaRD: LLM-powered graph denoising lifts rec performance","LLaRD: chain-of-thought plus info bottleneck beats denoisers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":2943,"prompt_tokens":1040,"completion_tokens":1903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":1811}},"tokens_in":656,"tokens_out":1903,"duration_ms":13709,"temperature":1.0,"reasoning_tokens":1811,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:45:27.860572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LLaRD on a dataset with known injected noise and known held-out positives, then compute precision and recall of the LLM's $E_{Noise}$ against the injected noise and the overlap of $E_{Collab} \\cup E_{Interests}$ with the held-out positives. Low precision on known noise, or overlap with test positives far above chance, would mean the gains come from label errors or leakage rather than denoising.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RLMRec, the nearest LLM-based representation-learning baseline; LLaRD extends its preference-knowledge idea to data-level denoising."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"T-CE, a reweighting denoising method representing the interaction-data-driven family LLaRD claims to surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the graph information bottleneck framing that motivates the compression objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CGI, the prior IB-based denoising method whose objective LLaRD adapts with LLM knowledge."}],"review_version":1}