{"id":"8a92fc4b-a18c-49b2-b9a9-bb58e52e127c","arxiv_id":"2508.05198","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Sub-ID-level personalized popularity in RecJPQ-based BERT4Rec improves the accuracy-novelty trade-off in music recommendation relative to item-level PPS.","lead":"This paper adds personalized popularity scores computed over sub-identifiers (sub-ID-level PPS) to a Transformer-based music recommender built on RecJPQ. On two music datasets, the sub-ID signal lets the system recommend less-familiar tracks while keeping ranking accuracy, compared to item-level popularity alone.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training with both PPS and sPPS fixed at α=β=0.4 confounds the PPS-vs-sPPS comparison; a fair comparison requires per-signal training.","rationale":"The reader's weakest assumption—semantic coherence of RecJPQ's SVD-derived sub-IDs—is plausible but secondary. The training-protocol confound is directly evidenced by Section 3 and undermines the causal interpretation of Fig. 2 and Table 2. The paper's own description shows both popularity signals were present during training, so the PPS-only curve is not a clean baseline for the claimed granularity effect. This is more load-bearing because even if sub-IDs are semantically coherent, the reported comparison does not isolate whether sPPS's benefit comes from sub-ID granularity or from the jointly trained checkpoint. The paper may still be correct after a fair retraining comparison, so a CONDITIONAL verdict remains appropriate; hence the reader's verdict is unchanged.","tokens_in":8945,"tokens_out":9259,"duration_ms":107745,"concrete_test":"Retrain the RecJPQ/BERT4Rec model under three separate training protocols: (i) PPS-only (α=0.4, β=0 in Eq. 7 during training), (ii) sPPS-only (α=0, β=0.4), and (iii) no popularity (α=β=0). For each checkpoint, run the same inference-time sweeps used for Fig. 2 (vary α for PPS-only curves, vary β for sPPS-only curves) and recompute the NDCG@40 vs novelty@40 frontiers on both datasets. If the sPPS frontier still lies above the PPS frontier at matched NDCG (e.g., Last.fm at NDCG≈0.32), the claim survives; if the curves cross or overlap, the reported advantage is an artifact of jointly training with both signals. Report at least 3 seeds to also address the missing error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 (Experimental Setup) states: 'we fixed α = 0.4, β = 0.4 at training time, while at inference time we systematically vary α (for PPS) and β (for sPPS).' This means the PPS-only (red) and sPPS-only (blue) frontiers in Fig. 2 are obtained by ablating one signal from a single checkpoint trained with both signals in Eq. (7), not by comparing models trained with the corresponding signal. The central claim—that sub-ID-level popularity yields a better accuracy–novelty trade-off than item-level PPS—depends on the assumption that the learned logits_rec are independent of which popularity signal was present during training. That assumption is not tested. A PPS-only-trained model (β=0 during training) could learn different sub-embeddings and yield a PPS curve shifted upward in novelty, possibly eliminating the claimed 20% relative gain. Conversely, the sPPS curve benefits from a model whose sub-embeddings were optimized with sPPS in the training loss. This confound is more direct than the semantic-coherence assumption: even if sub-IDs are meaningful, the empirical comparison in Fig. 2/Table 2 does not isolate the effect of the popularity signal's granularity.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes sPPS, a personalized popularity signal computed at the sub-ID level using RecJPQ's item decomposition, and combines it with item-level PPS and the recommender's logits via a weighted score function (Eq. (7)). Experiments on Yandex and Last.fm-1K with BERT4Rec+RecJPQ report that sPPS improves the accuracy–personalized-novelty trade-off compared with item-level PPS, citing a 20% relative novelty gain at matched NDCG and about 8–10% NDCG gains at novelty threshold 12. The code and data are public.","tokens_in":9248,"tokens_out":4883,"duration_ms":58603,"significance":"If the empirical claim were solid, this would be a modest but useful contribution: repurposing an efficiency-oriented sub-item representation for a beyond-accuracy objective, with a simple inference-time control. The paper is clearly written, the method is easy to implement, and the public code/data support reproducibility. The scientific value, however, depends almost entirely on the reliability of the two-dataset comparison, because the method is an inference-time re-ranking heuristic rather than a new learning framework.","major_comments":[{"comment":"The central PPS-only vs sPPS-only comparison is confounded by the training-time configuration. The paper states that α=0.4 and β=0.4 are fixed at training time and only varied at inference. Therefore the red and blue frontiers in Figure 2 are obtained by ablating one additive signal from a single checkpoint trained with both signals. The sub-embeddings and logits_rec in that checkpoint were optimized under Eq. (7) with both PPS and sPPS present; removing one signal at inference tests a scoring function the model never trained with. The 20% relative novelty gain attributed to sPPS could be an artifact of training with sPPS in the loss, not an intrinsic property of sub-ID granularity. A fair comparison requires separate checkpoints trained with only PPS, only sPPS, and both, and then reporting inference-time frontiers for each.","section":"§3 Experimental Setup and §4 RQ2"},{"comment":"The NDCG values at novelty thresholds (≥0, ≥10, ≥12, ≥14) are not matched-accuracy comparisons if, as the column headings suggest, the metric is computed only over users/lists whose personalized novelty reaches the threshold. PPS and sPPS will generally have different qualifying user subsets at each threshold, so the +9.7% and +8.0% gains at threshold 12 may reflect population differences rather than recommendation quality. The paper should report the number of qualifying users per method and threshold, or use a threshold-free frontier comparison on the same user set.","section":"Table 2"},{"comment":"All claims of 'significantly higher novelty' rest on a single training run per dataset, with no error bars, confidence intervals, or significance tests. The margins are small (e.g., 0.1242 vs 0.1150 on Yandex), and the comparisons in Figure 2 are read off interpolated curves. At least three seeds with standard deviations, or paired tests across users, are needed to support the central accuracy–novelty claim and the word 'significantly' in the abstract.","section":"§4 and Figure 2"}],"minor_comments":[{"comment":"The text says γ=1−α−β 'ensures a convex combination.' This is only true under the constraints α,β≥0 and α+β≤1. The green-curve sweep described in §4 ('fix β=0.9 and gradually increase α') can violate the latter if α>0.1. State the feasible region explicitly or use unconstrained weights with a different normalization.","section":"§2.4, Eq. (7)"},{"comment":"The legend and labels are hard to read, and the caption text ('β=0 (PPS)', 'β=0 (Sub-ID)', 'β=0.9 (varying)') is inconsistent with the body text describing the green curve. Clarify which parameter is on the x-axis of each sweep and what the marker values are.","section":"Figure 2"},{"comment":"The example 'at fixed NDCG@40 approximately 0.32, PPS yields novelty about 10 whereas sPPS achieves roughly 12' is read off a plot. Provide the underlying data points or a table of (α, β, NDCG, novelty) values to make the 20% claim auditable.","section":"§4 RQ2"},{"comment":"The semantic-coherence assumption—that RecJPQ's SVD-based sub-IDs group items by genre, artist, or similar latent attributes—is central to the method's interpretation but is not validated. An analysis of sub-ID–attribute association would strengthen the causal story behind the novelty gains.","section":"§2.3"},{"comment":"The word 'significantly' is used without statistical support. Please soften to 'higher' or provide significance tests.","section":"Abstract and §4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the paper is transparent about its reuse of the authors' prior work (PPS, RecJPQ). The main issue is experimental: the training-time mixing of α and β means the headline comparison does not isolate the effect of sub-ID granularity. This is fixable by retraining per configuration, though it may be costly. If the retrained experiments confirm the frontier, the paper could be suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the sPPS score is a natural, cheap extension of PPS to RecJPQ sub-IDs, and the paper has a genuinely new empirical object—an accuracy–novelty frontier for popularity at two granularities. But the frontier comparison in Figure 2 is not a clean test of PPS vs sPPS. They train one model with both popularity signals active (α=0.4, β=0.4) and then at inference set β=0 for the PPS curve or α=0 for the sPPS curve. The underlying RecJPQ sub-embeddings were optimized in a model that had both signals in the loss, so ablating one signal at inference doesn't tell you what a model trained with only that signal would do. The conclusion itself lists varying α/β at training as future work, which reads as an admission that this setup is a shortcut. A PPS-only-trained model could have different sub-embeddings and a different PPS curve, possibly wiping out the claimed gain.\n\nCredit where it is earned: Eq. (6) is not in the cited prior work—counting sub-ID occurrences in a user's history to expose shared repetition patterns is a sensible new application of RecJPQ's decomposition. The paper is clearly written, reproducible in intent (code repo linked), and uses two music datasets with a standard evaluation setup. That is real work.\n\nSoft spots beyond the main confound: Table 2 compares NDCG at novelty thresholds over different user subsets, so threshold-to-threshold numbers are not directly comparable; there are no error bars or significance tests anywhere; Eq. (7) is called a convex combination but γ=1−α−β goes negative in the green-curve sweep when α+β>1; and the abstract's “without compromising accuracy” is too strong because at novelty≥0 sPPS NDCG is actually lower than PPS. There is also an unverified semantic-coherence assumption about SVD-derived sub-IDs—if sub-IDs don't track genres/artists, the novelty gain would not generalize. That one is more speculative than the training confound, but worth flagging.\n\nWho this is for: anyone working on sub-ID/quantized recommenders or cheap inference-time novelty control. The idea is worth refereeing, and the method is simple enough to re-test, but I would not treat the central result as established until the training confound is resolved.","headline":"Useful incremental idea—sPPS at sub-ID level—but the headline PPS-vs-sPPS comparison is confounded by training on both signals and ablating at inference; the claim needs a cleaner experiment before the 20% novelty gain is taken seriously.","tokens_in":9763,"tokens_out":3692,"would_cite":false,"duration_ms":46536,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By scoring items through the sub-IDs they share with a user's listening history, the paper shows that music recommenders can gain substantial personalized novelty without sacrificing accuracy.","keywords":["music recommendation","sequential recommendation","personalised popularity","sub-ID popularity","RecJPQ","accuracy-novelty trade-off","product quantisation","novelty"],"falsifier":"Run the same experiments with sub-IDs randomly permuted before computing sPPS (keeping everything else fixed); if sPPS still shows the same or larger novelty-accuracy frontier, the semantic-grouping premise is wrong. Alternatively, measure the genre or artist purity of the items sharing each sub-ID: if purity is near chance, the proposed mechanism is not operating.","tokens_in":8850,"feed_emoji":"🎵","tokens_out":5682,"duration_ms":59874,"temperature":0.7,"pith_summary":"The paper tries to establish that music recommendation can get more personalized novelty without losing accuracy by counting how often the shared sub-parts of item codes recur in each user's history, instead of only counting whole-track replays. It builds on the RecJPQ representation, where each track is a tuple of sub-IDs learned from a low-rank decomposition, and repurposes those sub-IDs as a finer-grained popularity signal. The central empirical claim is that the sub-ID popularity score (sPPS), mixed into the final ranking by a simple convex score, moves the accuracy-novelty frontier outward compared with item-level personalised popularity: on Last.fm at matched NDCG around 0.32, personalised novelty rises from about 10 to about 12. A sympathetic reader cares because repetitive music listening makes recommendations default to already-known tracks, and this offers a lever to surface related-but-unseen content while keeping the recommender accurate.","feed_headline":"Music recommendations gain 20% novelty at same accuracy","feed_subtitle":"Counting sub-ID repeats in a listener's history surfaces unfamiliar related tracks better than whole-item popularity.","key_machinery":"The central object is RecJPQ's sub-item decomposition, which represents each item by $m$ sub-IDs drawn from learned codebooks; the new signal sPPS sums, over the $m$ code positions, $\\log(\\text{count of that sub-ID in the user's history} + \\epsilon)$, then z-score normalizes. It is combined with the recommender's logits and item-level PPS in a convex weighted score (Eq.~7), whose weights $\\alpha$ and $\\beta$ let one sweep the accuracy-novelty frontier at inference time without retraining. The sub-IDs are load-bearing: they must carry semantic grouping for the novelty gain to be relevant.","core_discovery":"Modelling personalised popularity at the sub-ID level, not the item-ID level, captures latent structural recurrence in a user's listening history and yields a better accuracy-novelty trade-off. The final score is $\\text{logits}_{\\text{final}} = \\gamma\\,\\text{logits}_{\\text{rec}} + \\alpha\\,\\text{PPS}_{\\text{std}} + \\beta\\,\\text{sPPS}_{\\text{std}}$ with $\\gamma = 1 - \\alpha - \\beta$, where PPS is the standardised log-frequency of whole items and sPPS is the standardised sum of log-frequencies of the item's constituent sub-IDs. When $\\beta$ alone is active, sPPS reaches roughly 20% higher personalised novelty at the same NDCG; when both signals are active, item-level accuracy can be added with","pith_inferences":["If sub-ID semantic grouping transfers, sPPS could apply to any catalogue with compositional items—outfits, news topics, code snippets—where sub-item recurrence encodes shared attributes; that extension goes beyond the paper's music experiments.","A direct test of the mechanism would be to permute the sub-ID assignments before computing sPPS; the novelty-accuracy advantage should vanish under permutation if semantic grouping is the cause.","The novelty metric used here measures deviation from the user's own listening history, so it does not by itself prove long-term engagement; an online or user-study check would be needed to confirm the added novelty is desirable.","Per-user tuning of $\\alpha$ and $\\beta$ (for instance, higher $\\beta$ for users with long, repetitive histories) is a natural extension implied by the convex score, but the paper only sweeps global values."],"forward_implications":["With sPPS, a music recommender can push novelty at a fixed accuracy level: at NDCG@40 near 0.32 on Last.fm, novelty rises from about 10 (PPS) to about 12 (sPPS), a roughly 20% relative gain.","At a personalised-novelty threshold of 12, NDCG@40 improves from 0.2749 (PPS) to 0.3016 (sPPS) on Last.fm and from 0.1150 to 0.1242 on Yandex, so the gain is not limited to one dataset.","Because $\\alpha$ and $\\beta$ are varied only at inference time, the same trained model can serve different accuracy-novelty operating points without retraining.","Adding item-level PPS on top of a strong sub-ID signal ($\\beta=0.9$, varying $\\alpha$) improves NDCG while losing less novelty than PPS alone, indicating the two signals are complementary.","The integration operates directly on the output scoring function, so the authors expect the approach to transfer to other sub-ID-based sequential recommenders."],"supporting_citations":[{"why":"Supplies the item-level PPS formula and the music-domain experimental setup that this work extends from item-level to sub-ID-level popularity.","marker":"[1]"},{"why":"Provides RecJPQ's SVD-based sub-ID decomposition, the representation on which sPPS operates.","marker":"[15]"},{"why":"BERT4Rec is the Transformer-based sequential recommender whose logits are combined with PPS and sPPS in the final score.","marker":"[23]"},{"why":"Defines the rank-and-relevance novelty metric used to measure personalised novelty.","marker":"[26]"},{"why":"Provides the Last.fm-1K music dataset used in the evaluation.","marker":"[3]"},{"why":"Motivates the log-transformed repetition counts used in the sPPS scoring.","marker":"[18]"},{"why":"Supplies the relevance mapping for Yandex interactions used in NDCG calculation.","marker":"[28]"}],"fun_headline_variants":["Sub-ID counting unlocks 20% more music novelty","Fine-grained repeat patterns beat whole-item popularity","Same accuracy, 20% fresher music via sub-item repeats","How sub-IDs in listen history boost recommendation novelty"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole gain rests on RecJPQ's learned sub-IDs actually grouping items by shared stylistic or structural traits relevant to the user; if the SVD-derived sub-IDs are semantically arbitrary, then the 'novel' items sPPS promotes would be unrelated to the user's tastes and the accuracy-novelty gain would disappear.","fun_headline_variants_meta":{"raw":{"variants":["Sub-ID counting unlocks 20% more music novelty","Fine-grained repeat patterns beat whole-item popularity","Same accuracy, 20% fresher music via sub-item repeats","How sub-IDs in listen history boost recommendation novelty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1207,"prompt_tokens":798,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":542,"tokens_out":409,"duration_ms":5192,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:28:52.751022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiments with sub-IDs randomly permuted before computing sPPS (keeping everything else fixed); if sPPS still shows the same or larger novelty-accuracy frontier, the semantic-grouping premise is wrong. Alternatively, measure the genre or artist purity of the items sharing each sub-ID: if purity is near chance, the proposed mechanism is not operating.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the item-level PPS formula and the music-domain experimental setup that this work extends from item-level to sub-ID-level popularity."},{"cited_title":"Petrov and Craig Macdonald","cited_arxiv_id":null,"evidence_quote":"Provides RecJPQ's SVD-based sub-ID decomposition, the representation on which sPPS operates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Last.fm-1K music dataset used in the evaluation."},{"cited_title":"Petrov, Efi Karra Taniskidou, and Sean Murphy","cited_arxiv_id":null,"evidence_quote":"Motivates the log-transformed repetition counts used in the sPPS scoring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the relevance mapping for Yandex interactions used in NDCG calculation."}],"review_version":1}