REVIEW 4 major objections 4 minor 39 references
Revisiting Language Models in Neural News Recommender Systems
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Larger LMs do not consistently improve news recommendation accuracy; the one clear benefit is for cold-start users.
desk verdict Useful empirical benchmark with a new cold-start finding, but the headline scale result is confounded by pooling differences and unsupported by variance statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the LM-based news encoder that converts article text into a vector for the recommender, with each LM family using its own readout: GloVe concatenates word embeddings, PLMs use the [CLS] token, and Llama pools the last ten token embeddings. The paper's argument is carried by a controlled comparison that holds the recommendation architecture fixed, toggles the LM and its fine-tuning mode, and then slices users by click-history quintiles to expose the cold-start effect. The non-fine-tuned vs fine-tuned contrast and the user-group slicing are what separate scale effects from adapter and user-history effects.
What would settle it
Run the same eight-LM comparison on a second English news dataset with identical pooling and fine-tuning budgets; if, for instance, all BERT variants then beat GloVe even without fine-tuning, the claim that larger LMs do not consistently improve performance would be overturned. Or isolate users with exactly zero prior clicks and check whether larger LMs still show the largest relative improvement; if the advantage vanishes, the cold-start conclusion would not generalize.
Extended reading notes
Core claim
The central discovery is that the relationship between LM size and news recommendation quality is non-monotonic and conditional, not monotonic. Across the three news RS methods, ROC-AUC scored with an 8-billion-parameter Llama encoder does not significantly surpass a 110-million-parameter BERT, and the 4-million-parameter BERT-tiny is often beaten by GloVe. Fine-tuning generally helps, but the best number of fine-tuned layers varies by model and task, and for Llama fine-tuning can hurt on NAML. The positive result is that Llama provides the largest AUC improvement over GloVe for users in the lowest click-history quintile, with the benefit shrinking in higher-quintile groups and turning negative for LSTUR's most active users. Thus, scale is a resource trade-off except for cold-start users, where it acts as a substitute for interaction history.
Load-bearing premise
The comparison assumes the three readout schemes—GloVe concatenation, [CLS] pooling for PLMs, and last-10-token pooling for Llama—are equally fair adapters of each model family to the news-encoding task; if any pooling choice is a poor fit for its model class, the measured differences would reflect the readout rather than model size.
Editorial extensions
If this is right
- Practitioners should not equate model size with recommendation quality; fine-tuning budget and the distribution of user history should drive the choice of news encoder.
- In low-resource settings, a non-fine-tuned GloVe encoder can be a strong and efficient baseline, beating small PLMs.
- For onboarding new users with very short histories, a large LM encoder measurably improves AUC, so it can serve as a content-only substitute for missing interaction data.
- The cold-start advantage shrinks as click history grows, implying the benefit of large encoders is largest exactly when user signals are weakest.
- Computing Llama embeddings once and storing them makes LLM-scale encoders feasible for static news collections, trading initial cost for fast inference.
Reading between the lines
- The paper's results suggest that published comparisons of LMs in news recommendation should always report the fine-tuning protocol and user-history stratification; otherwise contradictory findings may reflect those choices rather than model quality.
- A natural testable extension is a hybrid recommender that starts a new user with an LLM-based encoder and switches to a smaller PLM as the user's history grows, which the diminishing-return pattern suggests could save compute at little accuracy cost.
- The RoBERTa-underperforms-BERT result hints that pre-training domain (news-heavy web text vs BookCorpus/Wikipedia) may matter more than parameter count; a direct test would be to fine-tune the same-sized models on the same news corpus and repeat the comparison.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper revisits the effect of language-model (LM) size on neural news recommender systems (RSs). It compares eight LMs as news encoders—GloVe, five BERT variants, RoBERTa, and Llama 3.1-8B—in three widely used news RSs (NAML, NRMS, LSTUR) on the MIND-small dataset, under both frozen and fine-tuned settings. The authors address three research questions: whether larger LMs consistently improve accuracy (RQ1), how fine-tuning affects performance and efficiency (RQ2), and whether larger LMs help cold-start users (RQ3). The main claims are that larger LMs do not consistently translate to better recommendation performance, that they require more careful fine-tuning and more computational resources, and that they improve performance for users with little click history.
Significance. If the conclusions were robust, this would be a useful benchmark-style contribution: it would qualify the common scaling narrative in news recommendation, provide a practical efficiency trade-off analysis, and identify a concrete benefit of larger LMs for cold-start users. The paper is transparent about its limitations and releases code for reproducibility, which is a genuine strength. However, the central negative result is currently not cleanly established because the cross-family comparisons confound LM size with the news-embedding extraction scheme, and the absence of statistical tests or a clearly described validation split leaves several load-bearing conclusions unsupported. The cold-start finding is interesting and deserves a cleaner test.
major comments (4)
- [Section 3.3, Table 2, Fig. 7] The three LM families use different news-embedding extraction schemes: SLMs concatenate all word embeddings, PLMs use only the [CLS] token, and Llama uses the last 10 token embeddings. As a result, cross-family comparisons in Table 2 and Fig. 7 do not isolate LM size. For instance, GloVe's superiority over BERT-tiny/mini/small could be due to concatenation of all word embeddings rather than to model size, and Llama's failure to beat BERT-base could be due to the last-10-token pooling rather than to the underlying LM. The within-BERT-family comparisons in Section 5.1, point (4), actually show that larger models generally perform better, so the headline claim that 'larger LMs do not necessarily translate to better performance' rests almost entirely on the confounded cross-family comparisons. The cold-start analysis in Fig. 7 is likewise affected because it compares GloVe, BERT, and Llama with three different pooling rules. Please add a controlled comparison that holds the embedding-extraction scheme fixed across all LM families (for example, using the same pooling strategy for every LM), or explicitly restrict the claim to within-family evidence.
- [Section 4.2, Fig. 5] The paper states that for PLMs the authors 'fine-tune varying numbers of layers (from none to all) and select the optimal configuration based on recommendation performance,' but it does not state whether this selection is made on a validation split. If the number of fine-tuned layers and the hyperparameters (negative-sample size, dropout, learning rate) were chosen using the test-set metrics reported in Tables 2 and 3, then the reported results are overfit and the RQ2 conclusions about fine-tuning are not supported. Please describe the exact validation split used for model selection and report the selected layer counts and hyperparameters for each model and backbone.
- [Tables 2 and 3] All reported results are averages over three runs, but no standard deviations, confidence intervals, or significance tests are provided. Many of the differences that carry the RQ1 conclusions are small (e.g., Table 2, NAML: BERT 67.30 vs RoBERTa 66.73 AUC; Table 2, NRMS: GloVe 66.62 vs Llama 66.64), so it is possible that the observed ordering is noise. This is especially important for a claim about absence of improvement, which requires evidence that the differences are null rather than simply unmeasured. Please include variance estimates or, ideally, a paired significance test across users or impressions.
- [Table 3 vs. Table 2] There is an inconsistency between the two tables for NAML with Llama. Table 3 reports fine-tuned Llama with AUC 67.90 and non-fine-tuned Llama with AUC 68.39, while Table 2 reports NAML-Llama as 68.39. This suggests that Table 2 does not consistently use the same protocol across all rows (e.g., it appears to use the non-fine-tuned Llama for NAML but fine-tuned configurations for other models). Please clarify whether Table 2 reports the best configuration found per LM or a single uniform protocol, and reconcile the two tables. Without this, readers cannot determine whether the RQ1 comparisons are between fine-tuned systems, non-fine-tuned systems, or a mixture.
minor comments (4)
- [Fig. 3] The prompt text contains a typo: 'recently' is written as 'rencently'.
- [Section 4.2] The paper would be easier to reproduce if the final selected hyperparameters (negative-sample size, dropout, learning rate, and number of fine-tuned layers) were listed in a table for each LM and recommender, rather than only provided in the linked code.
- [Section 5.3] Please clarify how the five user engagement groups are constructed: are they based on click-history lengths in the training set, the test set, or the full dataset, and do users appear in only one group? This affects the interpretation of Fig. 7.
- [Table 1] The column header 'abs' should be spelled out as 'abstract' for clarity.
Circularity Check
No circularity: the paper is an empirical benchmark study on MIND with no derivation that reduces to its own inputs.
full rationale
The paper's central claims (RQ1: larger LMs do not consistently improve recommendation accuracy; RQ2: fine-tuning affects performance and efficiency; RQ3: larger LMs help cold-start users) are empirical outcomes of experiments, not predictions derived from fitted quantities or from definitions. The comparison pipeline in Section 3.3 couples LM family with embedding extraction (GloVe concatenation, PLM [CLS] pooling, Llama last-10-token pooling), and Section 4.2 selects fine-tuned layer counts and hyperparameters based on recommendation performance; these are threats to the internal validity of the cross-family comparison and potential selection effects, but they do not make any result true by construction. No load-bearing self-citation chain appears: the cited pooling choices ([15,30,35]), prompt format ([10]), and two-step fine-tuning inspiration ([2]) are external prior work, and the benchmark (MIND) and recommender backbones (NAML, NRMS, LSTUR) are independent of the authors' own fitted values. Since there is no equation whose output equals its input by definition, and no fitted parameter is renamed as a prediction, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Number of fine-tuned layers for BERT and RoBERTa =
Varies by RS method; values not reported in the paper
- Hyperparameters (negative sample size, dropout, learning rate) =
|V-| in {1,2,3,4}, dropout in {0.1..0.5}, LR in {1e-4, 1e-5}
- l, number of last tokens used for Llama news embeddings =
10
assumptions (3)
- domain assumption MIND-small is a representative benchmark for news recommendation
- domain assumption The reimplementations of NAML, NRMS, and LSTUR faithfully reproduce the original methods
- ad hoc to paper The three pooling strategies (concatenation, [CLS], last-10-tokens) are fair comparators across LM classes
Cite this review
Pith. "Pith review of Revisiting Language Models in Neural News Recommender Systems." pith.science (2026). https://pith.science/paper/6QPPNDR4
@misc{pith2026250111391,
author = {Pith},
title = {Pith review of: Revisiting Language Models in Neural News Recommender Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/6QPPNDR4}},
note = {Machine review of arXiv:2501.11391}
}
read the original abstract
Neural news recommender systems (RSs) have integrated language models (LMs) to encode news articles with rich textual information into representations, thereby improving the recommendation process. Most studies suggest that (i) news RSs achieve better performance with larger pre-trained language models (PLMs) than shallow language models (SLMs), and (ii) that large language models (LLMs) outperform PLMs. However, other studies indicate that PLMs sometimes lead to worse performance than SLMs. Thus, it remains unclear whether using larger LMs consistently improves the performance of news RSs. In this paper, we revisit, unify, and extend these comparisons of the effectiveness of LMs in news RSs using the real-world MIND dataset. We find that (i) larger LMs do not necessarily translate to better performance in news RSs, and (ii) they require stricter fine-tuning hyperparameter selection and greater computational resources to achieve optimal recommendation performance than smaller LMs. On the positive side, our experiments show that larger LMs lead to better recommendation performance for cold-start users: they alleviate dependency on extensive user interaction history and make recommendations more reliant on the news content.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp
An, M., Wu, F., Wu, C., Zhang, K., Liu, Z., Xie, X.: Neural news recommen- dation with long-and short-term user representations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 336–345 (2019)
work page 2019
-
[2]
arXiv preprint arXiv:2308.08434 (2023)
Bao, K., Zhang, J., Wang, W., Zhang, Y., Yang, Z., Luo, Y., Chen, C., Feng, F., Tian, Q.: A bi-step grounding paradigm for large language models in recommendation systems. arXiv preprint arXiv:2308.08434 (2023)
arXiv 2023
-
[3]
In: Findings of the Association for Computational Linguistics: ACL 2022, pp
Bi, Q., Li, J., Shang, L., Jiang, X., Liu, Q., Yang, H.: Mtrec: Multi-task learning over bert for news recommendation. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 2663–2669 (2022)
work page 2022
-
[4]
arXiv preprint arXiv:1810.04805 (2018)
Devlin, J.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
arXiv 2018
-
[5]
In: Proceedings of the International Conference on Web Intelligence, pp
Gulla, J.A., Zhang, L., Liu, P., Özgöbek, Ö., Su, X.: The Adressa dataset for news recommendation. In: Proceedings of the International Conference on Web Intelligence, pp. 1042–1048 (2017)
work page 2017
-
[6]
arXiv preprint arXiv:2106.09685 (2021)
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
arXiv 2021
-
[7]
Iana,A.,Glavaš,G.,Paulheim,H.:NewsRecLib:Apytorch-lightninglibrary for neural news recommendation. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pp. 296–310 (2023)
work page 2023
-
[8]
Iana, A., Glavaš, G., Paulheim, H.: Mind your language: A multilingual dataset for cross-lingual news recommendation. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 553–563 (2024)
work page 2024
Show all 39 references
-
[9]
In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp
Jiang, J.: Tadi: Topic-aware attention and powerful dual-encoder interac- tion for recall in news recommendation. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15647–15658 (2023)
2023
-
[10]
arXiv preprint arXiv:2307.16645 (2023)
Jiang, T., Huang, S., Luan, Z., Wang, D., Zhuang, F.: Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645 (2023)
2023 arXiv
-
[11]
In: 2018 IEEE international conference on data mining (ICDM), pp
Kang, W.C., McAuley, J.: Self-attentive sequential recommendation. In: 2018 IEEE international conference on data mining (ICDM), pp. 197–206, IEEE (2018)
2018
-
[12]
In: Pro- ceedings of the 2013 International News Recommender Systems Workshop and Challenge, pp
Kille, B., Hopfgartner, F., Brodt, T., Heintz, T.: The plista dataset. In: Pro- ceedings of the 2013 International News Recommender Systems Workshop and Challenge, pp. 16–23 (2013)
2013
-
[13]
Kitchenham, B., et al.: Guidelines for performing systematic literature re- views in software engineering. Tech. rep., Keele University (2007), version 2.3 Revisiting Language Models in Neural News Recommender Systems 15
2007
-
[14]
arXiv preprint arXiv:2410.03432 (2024)
Kruse, J., Lindskow, K., Kalloori, S., Polignano, M., Pomo, C., Srivastava, A., Uppal, A., Andersen, M.R., Frellsen, J.: EB-NeRD: A large-scale dataset for news recommendation. arXiv preprint arXiv:2410.03432 (2024)
2024 arXiv
-
[15]
In: Findings of the Association for Computational Linguistics: ACL 2022, pp
Li, J., Zhu, J., Bi, Q., Cai, G., Shang, L., Dong, Z., Jiang, X., Liu, Q.: Miner: Multi-interest matching network for news recommendation. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 343–352 (2022)
2022
-
[16]
IEEE Access7, 145861–145879 (2019)
Li, M., Wang, L.: A survey on personalized news recommendation technol- ogy. IEEE Access7, 145861–145879 (2019)
2019
-
[17]
arXiv preprint arXiv:2311.05850 (2023)
Li, X., Zhang, Y., Malthouse, E.C.: Exploring fine-tuning chatgpt for news recommendation. arXiv preprint arXiv:2311.05850 (2023)
2023 arXiv
-
[18]
In: European Conference on Information Retrieval, pp
Li, X., Zhang, Y., Malthouse, E.C.: Prompt-based generative news recom- mendation (pgnr): Accuracy and controllability. In: European Conference on Information Retrieval, pp. 66–79, Springer (2024)
2024
-
[19]
arXiv preprint arXiv:2306.05817 (2023)
Lin, J., Dai, X., Xi, Y., Liu, W., Chen, B., Zhang, H., Liu, Y., Wu, C., Li, X., Zhu, C., et al.: How can recommender systems benefit from large language models: A survey. arXiv preprint arXiv:2306.05817 (2023)
2023 arXiv
-
[20]
In: Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp
Liu, Q., Chen, N., Sakai, T., Wu, X.M.: ONCE: Boosting content-based recommendation with both open-and closed-source large language models. In: Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 452–461 (2024)
2024
-
[21]
Liu,Y.:RoBERTa:ArobustlyoptimizedBERTpretrainingapproach.arXiv preprint arXiv:1907.11692 (2019)
2019 arXiv
-
[22]
In: Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp
Pennington, J., Socher, R., Manning, C.D.: GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)
2014
-
[23]
In: Proceedings of the 17th ACM Conference on Recommender Systems, pp
Sanner, S., Balog, K., Radlinski, F., Wedin, B., Dixon, L.: Large language models are competitive near cold-start recommenders for language-and item-based preferences. In: Proceedings of the 17th ACM Conference on Recommender Systems, pp. 890–896 (2023)
2023
-
[24]
arXiv preprint arXiv:2302.13971 (2023)
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
2023 arXiv
-
[25]
In: Advances in Neural Information Processing Systems, vol
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
2017
-
[26]
In: Companion Proceedings of the ACM on Web Conference 2024, pp
Wang, J., Lu, H., Caverlee, J., Chi, E.H., Chen, M.: Large language models as data augmenters for cold-start item recommendation. In: Companion Proceedings of the ACM on Web Conference 2024, pp. 726–729 (2024)
2024
-
[27]
In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp
Wu, C., Wu, F., An, M., Huang, J., Huang, Y., Xie, X.: Neural news rec- ommendation with attentive multi-view learning. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3863–3869 (2019)
2019
-
[28]
In: Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th 16 Y
Wu, C., Wu, F., Ge, S., Qi, T., Huang, Y., Xie, X.: Neural news recom- mendation with multi-head self-attention. In: Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th 16 Y. Zhao et al. International Joint Conference on Natural...
2019
-
[29]
ACM Transactions on Information Systems41(1), 1–50 (2023)
Wu, C., Wu, F., Huang, Y., Xie, X.: Personalized news recommendation: Methods and challenges. ACM Transactions on Information Systems41(1), 1–50 (2023)
2023
-
[30]
In: Proceedings of the 44th international ACM SIGIR Conference on Research and Development in Information Retrieval, pp
Wu, C., Wu, F., Qi, T., Huang, Y.: Empowering news recommendation with pre-trained language models. In: Proceedings of the 44th international ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1652–1656 (2021)
2021
-
[31]
Wu, C., Wu, F., Qi, T., Huang, Y.: Are big recommendation models fair to cold users? arXiv preprint arXiv:2202.13607 (2022)
2022 arXiv
-
[32]
arXiv preprint arXiv:2102.04887 (2021)
Wu, C., Wu, F., Yu, Y., Qi, T., Huang, Y., Liu, Q.: NewsBERT: Distilling pre-trained language model for intelligent news application. arXiv preprint arXiv:2102.04887 (2021)
2021 arXiv
-
[33]
In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp
Wu, F., Qiao, Y., Chen, J.H., Wu, C., Qi, T., Lian, J., Liu, D., Xie, X., Gao, J., Wu, W., et al.: MIND: A large-scale dataset for news recommen- dation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3597–3606 (2020)
2020
-
[34]
World Wide Web27(5), 60 (2024)
Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al.: A survey on large language models for recommen- dation. World Wide Web27(5), 60 (2024)
2024
-
[35]
In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp
Xiao, S., Liu, Z., Shao, Y., Di, T., Middha, B., Wu, F., Xie, X.: Train- ing large-scale news recommenders with pretrained language models in the loop. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4215–4225 (2022)
2022
-
[36]
arXiv preprint arXiv:2405.13007 (2024)
Yada, Y., Yamana, H.: News recommendation with category description by a large language model. arXiv preprint arXiv:2405.13007 (2024)
2024 arXiv
-
[37]
arXiv preprint arXiv:2109.05446 (2021)
Yi, J., Wu, F., Wu, C., Liu, R., Sun, G., Xie, X.: Efficient-FedRec: Efficient federated learning framework for privacy-preserving news recommendation. arXiv preprint arXiv:2109.05446 (2021)
2021 arXiv
-
[38]
In: International Joint Conference in Artificial Intelligence, vol
Zhang, Q., Li, J., Jia, Q., Wang, C., Zhu, J., Wang, Z., He, X.: UNBERT: User-news matching BERT for news recommendation. In: International Joint Conference in Artificial Intelligence, vol. 21, pp. 3356–3362 (2021)
2021
-
[39]
In: Pro- ceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp
Zhang, Z., Wang, B.: Prompt learning for news recommendation. In: Pro- ceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 227–237 (2023)
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.