{"id":"6dc41001-5981-4e08-a858-e574a7ada69b","arxiv_id":"2501.11391","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Larger language models as news encoders do not consistently improve recommendation accuracy, but they do improve performance for cold-start users, at higher fine-tuning and compute cost.","lead":"Using the MIND news dataset, this paper compares eight language models, from GloVe to Llama 3.1, as news encoders in three recommender systems and finds that larger models do not automatically improve accuracy. The result matters because it gives practitioners a realistic view of when bigger models are worth the compute, particularly for new users with little click history.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-family LM comparison is confounded by different embedding extraction schemes (GloVe concat vs PLM [CLS] vs Llama last-10 tokens), so the central claim that larger LMs do not necessarily help is not cleanly tested.","rationale":"The reader's verdict is CONDITIONAL, and the reader identified two weakest assumptions: (1) fairness of the different embedding extraction schemes, and (2) unclear validation split for hyperparameter search. I agree that the first is the most load-bearing, because it directly undermines the paper's central claim. The paper treats LM size as the independent variable, but in the cross-family comparisons the extraction scheme changes together with size. Over three families, no controlled comparison of 'size alone' exists; the within-family BERT results are controlled but show the opposite of the headline finding. The cold-start result, a positive contribution, is also confounded. This does not mean the paper has no value — it provides a reproducible unification and a useful efficiency analysis — but the core empirical assertion is not yet established. The reader's CONDITIONAL verdict is therefore appropriate, especially if the condition includes controlling for pooling. My agreement is partial because the reader also listed the validation-split issue as load-bearing; I see the pooling confound as the single most critical threat to the abstract's RQ1 and RQ3 claims. The paper's transparency about limitations and its released code are creditworthy, but they do not remove the confound. The concrete test of re-analyzing Table 2 with a fixed pooling scheme would settle whether the concern lands.","tokens_in":12464,"tokens_out":4862,"duration_ms":51848,"concrete_test":"Re-run the Table 2 comparison with a single fixed pooling scheme for every LM: e.g., mean-pool all token embeddings for GloVe, BERT, RoBERTa, and Llama and apply the same FC projection; alternatively, use the same last-10-token pooling for all models. If, under fixed pooling, performance becomes monotonic in model size (larger LMs consistently outperform smaller ones), the central claim is not supported. Report the recomputed AUC/MRR/nDCG for NAML, NRMS, and LSTUR, and state whether the RQ1 and cold-start conclusions change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central RQ1 and RQ3 comparisons vary LM size across families, but Section 3.3 couples model size with the news-embedding extraction scheme: SLMs concatenate all word embeddings, PLMs use only the [CLS] token, and LLMs use the last 10 token embeddings. Every cross-family difference in Table 2 and Fig. 7 — e.g., GloVe beating BERT-tiny/mini/small, or Llama not beating BERT-base — could therefore be caused by the pooling/adapter rather than by LM scale. The within-BERT-family comparisons, which keep [CLS] pooling fixed, actually show that larger models generally perform better (Section 5.1, point 4), so the headline negative result rests almost entirely on the confounded cross-family comparisons. The cold-start finding (Fig. 7) is likewise affected because it compares GloVe, BERT, and Llama with three different pooling rules. In addition, no significance testing or error bars are reported, so differences such as BERT vs RoBERTa (67.30 vs 66.73 AUC for NAML) could be noise. The most load-bearing issue is that the paper's main empirical claim — that larger LMs do not necessarily translate to better performance — has no clean comparison that isolates model size from the embedding extraction scheme.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper revisits the effect of language-model (LM) size on neural news recommender systems (RSs). It compares eight LMs as news encoders—GloVe, five BERT variants, RoBERTa, and Llama 3.1-8B—in three widely used news RSs (NAML, NRMS, LSTUR) on the MIND-small dataset, under both frozen and fine-tuned settings. The authors address three research questions: whether larger LMs consistently improve accuracy (RQ1), how fine-tuning affects performance and efficiency (RQ2), and whether larger LMs help cold-start users (RQ3). The main claims are that larger LMs do not consistently translate to better recommendation performance, that they require more careful fine-tuning and more computational resources, and that they improve performance for users with little click history.","tokens_in":12899,"tokens_out":6280,"duration_ms":60230,"significance":"If the conclusions were robust, this would be a useful benchmark-style contribution: it would qualify the common scaling narrative in news recommendation, provide a practical efficiency trade-off analysis, and identify a concrete benefit of larger LMs for cold-start users. The paper is transparent about its limitations and releases code for reproducibility, which is a genuine strength. However, the central negative result is currently not cleanly established because the cross-family comparisons confound LM size with the news-embedding extraction scheme, and the absence of statistical tests or a clearly described validation split leaves several load-bearing conclusions unsupported. The cold-start finding is interesting and deserves a cleaner test.","major_comments":[{"comment":"The three LM families use different news-embedding extraction schemes: SLMs concatenate all word embeddings, PLMs use only the [CLS] token, and Llama uses the last 10 token embeddings. As a result, cross-family comparisons in Table 2 and Fig. 7 do not isolate LM size. For instance, GloVe's superiority over BERT-tiny/mini/small could be due to concatenation of all word embeddings rather than to model size, and Llama's failure to beat BERT-base could be due to the last-10-token pooling rather than to the underlying LM. The within-BERT-family comparisons in Section 5.1, point (4), actually show that larger models generally perform better, so the headline claim that 'larger LMs do not necessarily translate to better performance' rests almost entirely on the confounded cross-family comparisons. The cold-start analysis in Fig. 7 is likewise affected because it compares GloVe, BERT, and Llama with three different pooling rules. Please add a controlled comparison that holds the embedding-extraction scheme fixed across all LM families (for example, using the same pooling strategy for every LM), or explicitly restrict the claim to within-family evidence.","section":"Section 3.3, Table 2, Fig. 7"},{"comment":"The paper states that for PLMs the authors 'fine-tune varying numbers of layers (from none to all) and select the optimal configuration based on recommendation performance,' but it does not state whether this selection is made on a validation split. If the number of fine-tuned layers and the hyperparameters (negative-sample size, dropout, learning rate) were chosen using the test-set metrics reported in Tables 2 and 3, then the reported results are overfit and the RQ2 conclusions about fine-tuning are not supported. Please describe the exact validation split used for model selection and report the selected layer counts and hyperparameters for each model and backbone.","section":"Section 4.2, Fig. 5"},{"comment":"All reported results are averages over three runs, but no standard deviations, confidence intervals, or significance tests are provided. Many of the differences that carry the RQ1 conclusions are small (e.g., Table 2, NAML: BERT 67.30 vs RoBERTa 66.73 AUC; Table 2, NRMS: GloVe 66.62 vs Llama 66.64), so it is possible that the observed ordering is noise. This is especially important for a claim about absence of improvement, which requires evidence that the differences are null rather than simply unmeasured. Please include variance estimates or, ideally, a paired significance test across users or impressions.","section":"Tables 2 and 3"},{"comment":"There is an inconsistency between the two tables for NAML with Llama. Table 3 reports fine-tuned Llama with AUC 67.90 and non-fine-tuned Llama with AUC 68.39, while Table 2 reports NAML-Llama as 68.39. This suggests that Table 2 does not consistently use the same protocol across all rows (e.g., it appears to use the non-fine-tuned Llama for NAML but fine-tuned configurations for other models). Please clarify whether Table 2 reports the best configuration found per LM or a single uniform protocol, and reconcile the two tables. Without this, readers cannot determine whether the RQ1 comparisons are between fine-tuned systems, non-fine-tuned systems, or a mixture.","section":"Table 3 vs. Table 2"}],"minor_comments":[{"comment":"The prompt text contains a typo: 'recently' is written as 'rencently'.","section":"Fig. 3"},{"comment":"The paper would be easier to reproduce if the final selected hyperparameters (negative-sample size, dropout, learning rate, and number of fine-tuned layers) were listed in a table for each LM and recommender, rather than only provided in the linked code.","section":"Section 4.2"},{"comment":"Please clarify how the five user engagement groups are constructed: are they based on click-history lengths in the training set, the test set, or the full dataset, and do users appear in only one group? This affects the interpretation of Fig. 7.","section":"Section 5.3"},{"comment":"The column header 'abs' should be spelled out as 'abstract' for clarity.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a well-scoped empirical study with a useful code release, and the topic fits the journal. My main concern is that the central negative claim is not cleanly tested because LM size and embedding-extraction method are confounded across families; the within-BERT-family results actually point in the opposite direction. I would not reject the paper if the authors add a controlled pooling comparison and within-family cold-start results, clarify the validation split, and provide variance estimates or significance tests. The Table 2/Table 3 inconsistency should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical reference, not a clean scientific result. The authors compare eight LMs—GloVe, multiple BERT/RoBERTa variants, and Llama-3.1-8B—across NAML, NRMS, and LSTUR on MIND-small, covering fine-tuned and non-fine-tuned settings and an efficiency analysis. They release code. The cold-start subgroup analysis is genuinely new: larger LMs help most for users with very short histories, and the benefit fades as history grows. That is the most original part.\n\nThe paper is honest about its limits and careful to re-implement baselines in a unified setting. The efficiency breakdown (trainable vs. total parameters) is a nice practical contribution.\n\nThe main problem is that the headline claim—larger LMs do not necessarily translate to better performance—is not cleanly tested. Section 3.3 couples model family with the embedding extraction method: GloVe concatenates all word embeddings, BERT/RoBERTa use the [CLS] token, Llama uses the last ten tokens. So every cross-family comparison in Table 2 and Fig. 7 carries a different pooling adapter. The authors never discuss this. Within the BERT family, where pooling is fixed, larger models generally do better. The negative result rests entirely on the confounded cross-family comparisons.\n\nTwo smaller issues: no error bars or significance tests are reported, so a gap like BERT vs. RoBERTa on NAML (67.30 vs. 66.73 AUC) could be noise. And hyperparameter selection in Section 4.2 is described as picking the optimal configuration based on recommendation performance, but no validation split is stated. If that search used the test metric, the fine-tuning conclusions are overfit.\n\nDespite the ambitious framing, the study is a solid benchmark and the cold-start finding is actionable. I would send it to review, with expectations of major revision: control for the pooling scheme, report variances, and clarify the validation procedure.\n\nGood reading-group material for spotting a confound that hides in plain sight.","headline":"Useful empirical benchmark with a new cold-start finding, but the headline scale result is confounded by pooling differences and unsupported by variance statistics.","tokens_in":13250,"tokens_out":4846,"would_cite":true,"duration_ms":45787,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Larger LMs do not consistently improve news recommendation accuracy; the one clear benefit is for cold-start users.","keywords":["news recommendation","language models","model scale","fine-tuning","cold-start users","MIND dataset","GloVe","BERT"],"falsifier":"Run the same eight-LM comparison on a second English news dataset with identical pooling and fine-tuning budgets; if, for instance, all BERT variants then beat GloVe even without fine-tuning, the claim that larger LMs do not consistently improve performance would be overturned. Or isolate users with exactly zero prior clicks and check whether larger LMs still show the largest relative improvement; if the advantage vanishes, the cold-start conclusion would not generalize.","tokens_in":12272,"feed_emoji":"📰","tokens_out":5634,"duration_ms":52012,"temperature":0.7,"pith_summary":"This paper tests a widely repeated assumption in news recommender systems: that bigger language models make better news encoders. Using eight LMs from GloVe to Llama-8B across three recommender architectures on the MIND dataset, it finds that larger models do not consistently improve accuracy and often need stricter fine-tuning and more compute to match smaller ones. Without fine-tuning, GloVe can beat several BERT variants and approach Llama. The one clear benefit of scale appears for cold-start users: larger LMs lift accuracy when click history is short, and the advantage fades as users accumulate more clicks. The net claim is that the value of an LM in news recommendation depends on fine-tuning protocol and user engagement level, not on parameter count alone.","feed_headline":"Bigger LMs don't guarantee better news picks","feed_subtitle":"A systematic comparison across GloVe, BERT, and Llama finds the real gains are for cold-start users.","key_machinery":"The central object is the LM-based news encoder that converts article text into a vector for the recommender, with each LM family using its own readout: GloVe concatenates word embeddings, PLMs use the [CLS] token, and Llama pools the last ten token embeddings. The paper's argument is carried by a controlled comparison that holds the recommendation architecture fixed, toggles the LM and its fine-tuning mode, and then slices users by click-history quintiles to expose the cold-start effect. The non-fine-tuned vs fine-tuned contrast and the user-group slicing are what separate scale effects from adapter and user-history effects.","core_discovery":"The central discovery is that the relationship between LM size and news recommendation quality is non-monotonic and conditional, not monotonic. Across the three news RS methods, ROC-AUC scored with an 8-billion-parameter Llama encoder does not significantly surpass a 110-million-parameter BERT, and the 4-million-parameter BERT-tiny is often beaten by GloVe. Fine-tuning generally helps, but the best number of fine-tuned layers varies by model and task, and for Llama fine-tuning can hurt on NAML. The positive result is that Llama provides the largest AUC improvement over GloVe for users in the lowest click-history quintile, with the benefit shrinking in higher-quintile groups and turning negative for LSTUR's most active users. Thus, scale is a resource trade-off except for cold-start users, where it acts as a substitute for interaction history.","pith_inferences":["The paper's results suggest that published comparisons of LMs in news recommendation should always report the fine-tuning protocol and user-history stratification; otherwise contradictory findings may reflect those choices rather than model quality.","A natural testable extension is a hybrid recommender that starts a new user with an LLM-based encoder and switches to a smaller PLM as the user's history grows, which the diminishing-return pattern suggests could save compute at little accuracy cost.","The RoBERTa-underperforms-BERT result hints that pre-training domain (news-heavy web text vs BookCorpus/Wikipedia) may matter more than parameter count; a direct test would be to fine-tune the same-sized models on the same news corpus and repeat the comparison."],"forward_implications":["Practitioners should not equate model size with recommendation quality; fine-tuning budget and the distribution of user history should drive the choice of news encoder.","In low-resource settings, a non-fine-tuned GloVe encoder can be a strong and efficient baseline, beating small PLMs.","For onboarding new users with very short histories, a large LM encoder measurably improves AUC, so it can serve as a content-only substitute for missing interaction data.","The cold-start advantage shrinks as click history grows, implying the benefit of large encoders is largest exactly when user signals are weakest.","Computing Llama embeddings once and storing them makes LLM-scale encoders feasible for static news collections, trading initial cost for fast inference."],"supporting_citations":[{"why":"Supplies the MIND-small dataset on which all comparisons are run.","marker":"[33]"},{"why":"Defines NAML, one of the three recommender architectures used as testbeds.","marker":"[27]"},{"why":"Defines NRMS, the second recommender architecture.","marker":"[28]"},{"why":"Defines LSTUR, the third recommender architecture, with its GRU user encoder.","marker":"[1]"},{"why":"Provides the GloVe SLM embeddings that serve as the smallest-LM baseline.","marker":"[22]"},{"why":"Provides the BERT models whose size variants (tiny to base) are compared within one family.","marker":"[4]"},{"why":"Provides the RoBERTa PLM used as a second, larger PLM baseline.","marker":"[21]"},{"why":"Provides the Llama-8B LLM used as the largest encoder.","marker":"[24]"},{"why":"Reports that swapping GloVe for BERT hurts NAML and LSTUR, the contradictory prior result this paper revisits.","marker":"[7]"},{"why":"Reports that PLM-based embeddings can underperform random initialization in NAML, another contradictory result motivating the study.","marker":"[8]"}],"fun_headline_variants":["Larger LMs don't guarantee better news picks","For news recs, LM scale helps only cold-start users","Bigger isn't always better: LM scale in news recommendations","Cold-start users gain most from large language models in news recs","LM size trade-offs in news recommender systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the three readout schemes—GloVe concatenation, [CLS] pooling for PLMs, and last-10-token pooling for Llama—are equally fair adapters of each model family to the news-encoding task; if any pooling choice is a poor fit for its model class, the measured differences would reflect the readout rather than model size.","fun_headline_variants_meta":{"raw":{"variants":["Larger LMs don't guarantee better news picks","For news recs, LM scale helps only cold-start users","Bigger isn't always better: LM scale in news recommendations","Cold-start users gain most from large language models in news recs","LM size trade-offs in news recommender systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1577,"prompt_tokens":929,"completion_tokens":648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":566}},"tokens_in":545,"tokens_out":648,"duration_ms":6313,"temperature":1.0,"reasoning_tokens":566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:17:42.874816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight-LM comparison on a second English news dataset with identical pooling and fine-tuning budgets; if, for instance, all BERT variants then beat GloVe even without fine-tuning, the claim that larger LMs do not consistently improve performance would be overturned. Or isolate users with exactly zero prior clicks and check whether larger LMs still show the largest relative improvement; if the advantage vanishes, the cold-start conclusion would not generalize.","supporting_citations":[{"cited_title":"In: Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th 16 Y","cited_arxiv_id":null,"evidence_quote":"Defines NRMS, the second recommender architecture."},{"cited_title":"In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp","cited_arxiv_id":null,"evidence_quote":"Defines LSTUR, the third recommender architecture, with its GRU user encoder."},{"cited_title":"In: Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp","cited_arxiv_id":null,"evidence_quote":"Provides the GloVe SLM embeddings that serve as the smallest-LM baseline."},{"cited_title":"In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pp","cited_arxiv_id":null,"evidence_quote":"Reports that swapping GloVe for BERT hurts NAML and LSTUR, the contradictory prior result this paper revisits."},{"cited_title":"In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp","cited_arxiv_id":null,"evidence_quote":"Reports that PLM-based embeddings can underperform random initialization in NAML, another contradictory result motivating the study."},{"cited_title":"In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the MIND-small dataset on which all comparisons are run."},{"cited_title":"In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp","cited_arxiv_id":null,"evidence_quote":"Defines NAML, one of the three recommender architectures used as testbeds."}],"review_version":1}