Pith. sign in

REVIEW 2 major objections 7 minor 1 cited by

Can Large Language Models Understand Preferences in Personalized Recommendation?

T0 review · 2 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current LLMs cannot rank user preferences once rating bias and item quality are removed; the best of 19 models scores only 0.18 Kendall tau on the new PerRecBench.

desk verdict Useful new benchmark, but the ground truth ranking has a load-bearing confound that undercuts the headline claim about LLM preference understanding. read the letter →

arxiv 2501.13391 v1 pith:W6TSOX3G submitted 2025-01-23 cs.CL

Zhaoxuan Tan , Zinan Zeng , Qingkai Zeng , Zhenyu Wu , Zheyuan Liu , Fengran Mo , Meng Jiang This is my paper · ORCID
classification cs.CL
keywords largelanguagemodelspersonalizedrecommendationpreferenceunderstandingPerRecBenchKendalltaurankingevaluationuserratingbiasitemquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rating-based evaluation of LLM recommenders, which computes MAE or RMSE against actual ratings, is the paper's target: those errors are inflated by user rating bias (how generously a person scores) and item quality (how good a product is on average), so strong numbers need not reflect any personalization. To isolate preference, the authors introduce PerRecBench, which groups users who bought the same query item within a short window and asks the model to rank those users by their relative rating, the user's rating minus the user's own average rating. Tested on 19 LLMs, the best average Kendall tau is 0.18, achieved by Claude-3.5-Sonnet, meaning current models are close to chance at telling which user in a group actually liked the shared item once the two confounds are removed. The paper's central claim is that rating-prediction skill and genuine preference understanding are different capabilities, and that benchmarks must separate them rather than let regression metrics hide the failure.

What carries the argument

The carrying object is the user group plus the relative rating transform. For each user $u$ and query item $q$, $\tilde y(q_u) = y(q_u) - \bar y_u$ removes the user's overall tendency to rate high or low; requiring all users in a group to have bought $q$ inside $[t_0, t_0+\Delta t]$ holds item quality roughly constant; and the threshold on pairwise differences $\lambda = 0.6$ ensures the ground-truth order is meaningful. The benchmark's metric, Personalization Tau Correlation, is Kendall's tau between the predicted ranking and the argsort of these relative ratings. These pieces turn personalization into a well-defined ranking problem whose only input variation across users is their own history and profile.

What would settle it

Take a synthetic rating dataset generated from a known additive model where users' true preferences for the query item are identical but their purchase histories differ in average item quality. If PerRecBench's relative ratings still order the users, the ground truth is contaminated by history quality rather than preference; alternatively, scoring the same prompts against explicit like or dislike labels would show whether the near-chance 0.18 reflects the benchmark or the models.

Watch

Extended reading notes

Core claim

The discovery is the benchmark itself and the empirical pattern it exposes. PerRecBench constructs user groups from Amazon review data such that every member bought the same query item within a narrow time interval, then defines ground truth as the ranking of users by relative rating $\tilde y(q_u) = y(q_u) - \bar y_u$, with a minimum pairwise difference of 0.6 between relative ratings so preferences are distinguishable. Models are scored by Kendall's tau between their predicted user ranking and this ground-truth ranking, using pointwise, pairwise, or listwise prompts; all inputs within a group are identical except each user's history and profile, so the output must come from personalization rather than item information. Across 19 off-the-shelf LLMs the average scores range from 0.02 to 0.18, and methods that look good on MAE and RMSE can score below chance on PerRecBench. The paper also reports that pairwise and listwise ranking beat pointwise ranking, that textual user profiles improve scores by about 28 percent, that few-shot and chain-of-thought prompting do not reliably help, that larger models do not always win, and that weight-merging separately fine-tuned models is the strongest of three supervised fine-tuning strategies but still leaves personalization unsolved.

Load-bearing premise

PerRecBench's ground truth assumes that a user's rating minus their own average rating measures only preference for the shared item, with nothing left over from the different average quality of the items each user happened to buy.

Editorial extensions

If this is right

  • Leaderboards built on MAE and RMSE cannot be read as evidence of personalization, because non-personal baselines like the user's average rating or the item's average rating already score well on them.
  • Evaluation of LLM recommenders should move to observed, comparative signals, pairwise or listwise preference judgments, rather than pointwise regression on 1 to 5 ratings.
  • Open-source models above 100B parameters can approach proprietary ones on PerRecBench, so preference capability is not exclusively a function of scale.
  • Weight merging of separately fine-tuned pointwise, pairwise, and listwise models outperforms joint multi-task training, suggesting that mixing training formats can hurt and that merging preserves useful task-specific skills.
  • Profile-augmented zero-shot prompting is the best simple default: adding profiles helps, while few-shot, self-consistency, and chain-of-thought prompting do not consistently improve and can degrade performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to run the same grouped-ranking protocol on synthetic data generated from a known additive preference model; that would separate genuine preference understanding from artifacts of the residual-rating ground truth.
  • The strong effect of user profiles and pretraining-domain frequency suggests preference understanding may be more of a retrieval-and-grounding problem than a scale problem, pointing toward personalized memory or parameter-efficient user encodings as next steps.
  • The advantage of pairwise and listwise formats implies that training on comparative objectives over observed histories could transfer better to personalization than pointwise rating regression.
  • Extending the benchmark to longitudinal splits, where the model must predict which users later change their taste for an item, would test whether LLMs track preference dynamics rather than only static profiles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This paper introduces PERRECBENCH, a benchmark that evaluates LLM-based personalized recommendation by asking models to rank users within groups by their relative rating for a shared query item, where the relative rating is the user's rating minus their average rating. The benchmark groups users who purchased the same item in a short time window, filters for active users and significant relative-rating differences, and uses Kendall's tau between predicted and ground-truth rankings as the metric. The authors evaluate 19 LLMs with pointwise, pairwise, and listwise prompting, find low overall performance (best average tau 0.18), and analyze the relationship with MAE/RMSE, prompting variants, domains, and three supervised fine-tuning strategies. The main claim is that LLMs fail to understand user preferences once user rating bias and item quality are controlled.

Significance. If the benchmark were valid, it would be a valuable contribution: it identifies a real weakness in rating-based evaluation, provides a new grouped-ranking task built on observed preference signals, and offers a broad comparison across 19 LLMs and multiple ranking paradigms, with code and data released. The finding that simple statistics achieve state-of-the-art MAE/RMSE (Figure 1) is important and likely correct. However, the central measurement is threatened by a historical-item-quality confound, so the headline conclusion—that LLMs cannot understand preferences—is not yet supported. The paper's secondary findings (pairwise/listwise vs pointwise, domain effects, SFT strategies) are of interest but depend on the same ground-truth validity.

major comments (2)
  1. [Section 2.1] The ground-truth ranking r* = argsort({y~(q_u)}) with y~(q_u) = y(q_u) − mean_u is claimed to eliminate user rating bias, but under the standard additive rating model y_{ui} = α_u + β_i + ε_{ui}, this quantity equals β_q − (1/|H_u|) Σ_{i∈H_u} β_i plus noise, because the user intercept α_u cancels. Since within a group all users rate the same query item q, β_q is constant, and the ranking is determined by the negative average intrinsic quality of each user's historical basket H_u. The group-selection criteria (temporal co-purchase, active-user threshold γ, relative-rating difference threshold λ) do not control for historical basket quality, so the target ranking can encode historical item quality rather than preference for the query item. Consequently, the low PTC scores, including Claude-3.5-Sonnet's 0.18, may reflect the difficulty of predicting this confounded target rather than a failure of preference understanding. The Limitations section acknowledges item-popularity bias but not this confound, and Figure 1 demonstrates exactly the same mechanism (item-quality statistics masquerading as personalization) that the benchmark fails to control.
  2. [Section 4 / Evaluation Protocol] The evaluation lacks a non-LLM baseline that could diagnose the confound. Given that the paper's own Figure 1 shows that simple user-average and item-average statistics achieve state-of-the-art MAE/RMSE, the authors should report how such statistics perform on PERRECBENCH. For example, ranking users by their average historical rating (or by a regression on historical item qualities) would directly test whether PERRECBENCH measures something beyond those statistics. Without such a baseline, the claim that the low PTC reflects poor preference understanding is not established; the benchmark may simply be hard for any method that does not have access to the historical item-quality estimates.
minor comments (7)
  1. [Section 2.1] The relation between the '663 user groups with five users per group' in Section 2.1 and the '200 groups each with 2, 3, and 4 users' in the abstract is unclear, and Appendix Table 4 reports only 986 distinct users, which is inconsistent with 1,800 user instances for 600 groups unless users are reused across groups; please clarify the exact sampling procedure and report the number of unique users and groups per size.
  2. [Section 2.2] The user profiles are generated by an unspecified 'instruction-tuned LLM'; since profile quality can affect all downstream results, the paper should identify the model and prompt used for profile generation.
  3. [Section 2.3] The computation of Kendall's tau does not specify how ties (e.g., equal predicted relative ratings in pointwise evaluation) are handled; given that the user average rating is provided in the prompt, ties are likely and should be addressed.
  4. [Section 4] The statement that 'the average Kendall's tau scores for pointwise, pairwise, and listwise ranking are 0.19, 0.38, and 0.35' is difficult to reconcile with Table 1, where no individual model's average exceeds 0.18; please specify how these aggregates are computed.
  5. [Section 5] The correlation analysis in Figure 3 uses only one base model (the text says prompt variants were generated using GPT-4O, while the scatter plots are unclear) with a small number of configurations; a broader sweep across multiple models would strengthen the claim that MAE/RMSE are uncorrelated with PTC.
  6. [Section 6] The dashes in Table 3 are not explained; please clarify what settings they denote and why some cells are missing.
  7. [Section 2.1] The free parameters γ (20), λ (0.6), and Δt are fixed without sensitivity analysis; because the group-selection procedure is a core part of the benchmark, the paper would benefit from reporting how PTC varies with these thresholds.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PerRecBench scores are empirical measurements against a directly defined relative-rating ground truth, with only construct-validity caveats and non-load-bearing self-citations.

full rationale

PerRecBench is an evaluation artifact rather than a derivation: the ground-truth ranking r* is defined directly from observed ratings via relative rating y~(q_u)=y(q_u)-mean_u, and no parameter is fitted to the benchmark before computing PTC. The central finding that 19 LLMs score between 0.02 and 0.18 Kendall tau is an empirical measurement against this externally specified target, so it does not reduce to its inputs by construction. The paper's self-citations (Tan and Jiang 2023; Tan et al. 2024a; Tan et al. 2024b) appear in related-work, motivation, and future-work passages; they are not load-bearing for the benchmark construction or the main experimental results. The only substantive concern is construct validity: the assumption that y minus the user's average rating isolates preference for the query item can be violated if users' historical baskets differ in average item quality, because under an additive rating model the relative rating still contains the negative mean of historical item-quality effects. That is a correctness or validity caveat, not a circular step, since the paper's equations do not assume the conclusion; they merely operationalize preference as relative rating. The limitations section acknowledges dataset scale and item-popularity bias but not this historical-basket confound; that omission affects how strongly the 0.18 score can be interpreted as evidence about preference understanding, but it does not make the benchmark score equivalent to its inputs by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The benchmark's ground truth relies on an additive bias model for ratings and on the co-purchase grouping controlling item quality. These are domain assumptions, not derived facts. The free parameters are thresholds and retrieval settings chosen by the authors; they shape the dataset and the task difficulty. No new physical or mathematical entities are introduced.

free parameters (4)
  • active user history threshold gamma = 20
    Section 2.1: users must have more than 20 prior ratings to be included. Chosen by hand; affects which users enter the benchmark.
  • relative rating difference threshold lambda = 0.6
    Section 2.1: pairwise relative rating differences must exceed 0.6. Chosen by hand and filters out ambiguous groups, making the ranking task artificially separable.
  • co-purchase time window Delta_t = unspecified
    Section 2.1: query item purchases must fall in [t0, t0+Delta_t]. The paper never states Delta_t, which is required for reproducing group construction.
  • retrieved history count k = 4 (default)
    Section 3: BM25 retrieval top-k set to 4 by default and varied in analysis. The choice changes the information the LLM receives.
assumptions (3)
  • domain assumption User ratings decompose additively into a user-specific bias, item quality, and preference, so subtracting the user's mean rating removes the user bias.
    Section 2.1 relative rating definition. If rating bias is item-dependent or scale-related, the residual is not a pure preference signal.
  • domain assumption Purchasing the same item within a short time window implies all group members experienced identical item quality.
    Section 2.1 Temporal Item Co-Purchase. The unspecified Delta_t and possible item versioning or seller differences weaken this control.
  • domain assumption The provided top-k retrieved history and LLM-generated user profile contain sufficient information to infer a user's preference for the query item.
    Section 2.2 and Appendix E. If the condensed input omits decisive signals, low PTC could reflect missing information rather than lack of preference understanding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Large Language Models Understand Preferences in Personalized Recommendation?." pith.science (2026). https://pith.science/paper/W6TSOX3G

@misc{pith2026250113391,
  author       = {Pith},
  title        = {Pith review of: Can Large Language Models Understand Preferences in Personalized Recommendation?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6TSOX3G}},
  note         = {Machine review of arXiv:2501.13391}
}
read the original abstract

Large Language Models (LLMs) excel in various tasks, including personalized recommendations. Existing evaluation methods often focus on rating prediction, relying on regression errors between actual and predicted ratings. However, user rating bias and item quality, two influential factors behind rating scores, can obscure personal preferences in user-item pair data. To address this, we introduce PerRecBench, disassociating the evaluation from these two factors and assessing recommendation techniques on capturing the personal preferences in a grouped ranking manner. We find that the LLM-based recommendation techniques that are generally good at rating prediction fail to identify users' favored and disfavored items when the user rating bias and item quality are eliminated by grouping users. With PerRecBench and 19 LLMs, we find that while larger models generally outperform smaller ones, they still struggle with personalized recommendation. Our findings reveal the superiority of pairwise and listwise ranking approaches over pointwise ranking, PerRecBench's low correlation with traditional regression metrics, the importance of user profiles, and the role of pretraining data distributions. We further explore three supervised fine-tuning strategies, finding that merging weights from single-format training is promising but improving LLMs' understanding of user preferences remains an open research problem. Code and data are available at https://github.com/TamSiuhin/PerRecBench

Figures

Figures reproduced from arXiv: 2501.13391 by the authors.

Figure 1
Figure 1. MAE and RMSE performance of user rating bias (average user rating history), item quality (average item rating), and existing LLM-based personalization methods. Simple averages of user rating history and item quality, which do not consider individual prefer￾ences, achieve state-of-the-art performance across four shopping domains, questioning the validity of MAE and RMSE for evaluating personalization. (Srivastava et … view at source ↗
Figure 2
Figure 2. Overview of PERRECBENCH, where the LLM ranks user preferences for a query item using pointwise, pairwise, and listwise prompting. The ground-truth ranking is derived from relative ratings, calculated as the user’s actual rating minus their average rating, to mitigate user rating bias. Finally, Kendall’s tau is computed between the predicted ranking and the ground-truth ranking to evaluate performance. u ∈ U, where x… view at source ↗
Figure 3
Figure 3. Correlation between Kendall’s tau in PER￾RECBENCH and traditional metrics (MAE and RMSE). The results show weak correlation, further confirming that MAE and RMSE are not reliable indicators of per￾sonalization capabilities. family and CLAUDE-3.5-SONNET. These results suggest that with proper optimization, open-source models can be viable alternatives for personaliza￾tion tasks, offering performance close to or on pa… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance across different domains and av [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The statistics of PERRECBENCH, including the distribution of actual rating, relative rating, length of user history, and the purchase date [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 7
Figure 7. Figure 7: LLM performance using pointwise, pairwise, and listwise ranking methods across different user group [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Zero-shot average performance across varying [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Instant Personalized Large Language Model Adaptation via Hypernetwork

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A hypernetwork maps a user profile to LoRA adapter weights in a single forward pass, matching or beating per-user fine-tuning at a fraction of deployment cost.

Reference graph

Works this paper leans on

60 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Sujay Kumar Jauhar, et al. 2023. Knowledge-augmented large language models for personalized contextual query suggestion. arXiv preprint arXiv:2311.06318

    arXiv 2023

  2. [2]

    Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. arXiv preprint arXiv:2305.00447

    arXiv 2023

  3. [3]

    Zheng Chen. 2023. Palr: Personalization aware llms for recommendation. arXiv preprint arXiv:2305.07622

    arXiv 2023

  4. [4]

    Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering chatgpt's capabilities in recommender systems. arXiv preprint arXiv:2305.02182

    work page Pith review arXiv 2023

  5. [5]

    Cosmina Andreea Dejescu, Lucia V Bel, Iulia Melega, Stefana Maria Cristina Muresan, and Liviu Ioan Oana. 2023. Approaches to laparoscopic training in veterinary medicine: A review of personalized simulators. Animals, 13(24):3781

    work page 2023

  6. [6]

    Yingpeng Du, Di Luo, Rui Yan, Hongzhi Liu, Yang Song, Hengshu Zhu, and Jie Zhang. 2023. Enhancing job recommendation through llm-based generative adversarial networks. arXiv preprint arXiv:2307.10747

    arXiv 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023. Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:2303.14524

    arXiv 2023

Show all 60 references
  1. [9]

    Dmitri Goldenberg, Kostia Kofman, Javier Albert, Sarai Mizrachi, Adam Horowitz, and Irene Teinemaa. 2021. Personalization in practice: Methods and applications. In Proceedings of the 14th ACM international conference on web search and data mining, pages 1123--1126

    2021

  2. [10]

    Yulong Gu, Zhuoye Ding, Shuaiqiang Wang, and Dawei Yin. 2020. Hierarchical user profiling for e-commerce recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 223--231

    2020

  3. [11]

    James A Hanley and Barbara J McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29--36

    1982

  4. [12]

    Charles R Harris, K Jarrod Millman, St \'e fan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. 2020. Array programming with numpy. Nature, 585(7825):357--362

    2020

  5. [13]

    Jesse Harte, Wouter Zorgdrager, Panos Louridas, Asterios Katsifodimos, Dietmar Jannach, and Marios Fragkoulis. 2023. Leveraging large language models for sequential recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1096--1102

    2023

  6. [14]

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024. Bridging language and items for retrieval and recommendation. arXiv preprint arXiv:2403.03952

    2024 arXiv

  7. [15]

    Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2023. Large language models are zero-shot rankers for recommender systems. arXiv preprint arXiv:2305.08845

    2023 arXiv

  8. [16]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

    2021

  9. [17]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  10. [18]

    a rvelin and Jaana Kek \

    Kalervo J \"a rvelin and Jaana Kek \"a l \"a inen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422--446

    2002

  11. [19]

    Jianchao Ji, Zelong Li, Shuyuan Xu, Wenyue Hua, Yingqiang Ge, Juntao Tan, and Yongfeng Zhang. 2023. Genrec: Large language model for generative recommendation. arXiv e-prints, pages arXiv--2307

    2023

  12. [20]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  13. [21]

    Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. 2023. https://arxiv.org/abs/2305.06474 Do llms understand user preferences? evaluating llms on user rating prediction . Preprint, arXiv:2305.06474

    2023 arXiv

  14. [22]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  15. [23]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  16. [24]

    Xinyu Li, Zachary C Lipton, and Liu Leqi. 2024. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05133

    2024 arXiv

  17. [25]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  18. [26]

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024 b . Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377

    2024 arXiv

  19. [27]

    Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. 2023 a . Is chatgpt a good recommender? a preliminary study. arXiv preprint arXiv:2304.10149

    2023 arXiv

  20. [28]

    Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. 2023 b . A first look at llm-powered generative news recommendation. arXiv preprint arXiv:2305.06566

    2023 arXiv

  21. [29]

    Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. 2023. Are emergent abilities in large language models just in-context learning? arXiv preprint arXiv:2309.01809

    2023 arXiv

  22. [30]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...

    2022 doi

  23. [31]

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156

    2023 arXiv

  24. [32]

    Nico JD Nagelkerke et al. 1991. A note on a general definition of the coefficient of determination. biometrika, 78(3):691--692

    1991

  25. [33]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  26. [34]

    Muh Putra Pratama, Rigel Sampelolo, and Hans Lura. 2023. Revolutionizing education: harnessing the power of artificial intelligence for personalized learning. Klasikal: Journal of Education, Language Teaching and Science, 5(2):350--357

    2023

  27. [35]

    Xueming Qian, He Feng, Guoshuai Zhao, and Tao Mei. 2013. Personalized recommendation combining user interest and social circle. IEEE transactions on knowledge and data engineering, 26(7):1763--1777

    2013

  28. [36]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.97 Large language models are effective text rankers with pairwise...

    2024 doi

  29. [37]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2024 arXiv

  30. [38]

    Dragomir R Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating web-based question answering systems. In LREC. Citeseer

    2002

  31. [39]

    Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy. 2023. Integrating summarization and retrieval for enhanced personalization via large language models. arXiv preprint arXiv:2310.20081

    2023 arXiv

  32. [40]

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. 2024. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159

    2024 arXiv

  33. [41]

    Biplav Srivastava, Francesca Rossi, Sheema Usmani, and Mariana Bernagozzi. 2020. Personalized chatbot trustworthiness ratings. IEEE Transactions on Technology and Society, 1(4):184--192

    2020

  34. [42]

    Aixin Sun. 2023. Take a fresh look at recommender systems from an evaluation standpoint. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2629--2638

    2023

  35. [43]

    Aixin Sun. 2024. Beyond collaborative filtering: A relook at task formulation in recommender systems. ACM SIGWEB Newsletter, 2024(Spring):1--11

    2024

  36. [44]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.923 Is C hat GPT good at search? investigating large language models as re-ranking agents . In Proceedings of the 2023 ...

    2023 doi

  37. [45]

    Zhaoxuan Tan and Meng Jiang. 2023. User modeling in the era of large language models: Current research and future directions. arXiv preprint arXiv:2312.11518

    2023 arXiv

  38. [46]

    Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. 2024 a . Personalized pieces: Efficient personalized large language models through collaborative efforts. arXiv preprint arXiv:2406.10471

    2024 arXiv

  39. [47]

    Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. 2024 b . Democratizing large language models via personalized parameter-efficient fine-tuning. arXiv preprint arXiv:2402.04401

    2024 arXiv

  40. [48]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  41. [49]

    Andrew Trotman, Antti Puurula, and Blake Burgess. 2014. Improvements to bm25 and language models examined. In Proceedings of the 19th Australasian Document Computing Symposium, pages 58--65

    2014

  42. [50]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research

    2022

  43. [51]

    Cort J Willmott and Kenji Matsuura. 2005. Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. Climate research, 30(1):79--82

    2005

  44. [52]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  45. [53]

    Yunjia Xi, Weiwen Liu, Jianghao Lin, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, and Yong Yu. 2023. Towards open-world recommendation with knowledge augmentation from large language models. arXiv preprint arXiv:2306.10933

    2023 arXiv

  46. [54]

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736

    2024 arXiv

  47. [55]

    Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023 a . Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. arXiv preprint arXiv:2305.07609

    2023

  48. [56]

    Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2023 b . Recommendation as instruction following: A large language model empowered recommendation approach. arXiv preprint arXiv:2305.07001

    2023 arXiv

  49. [57]

    Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. 2024. Personalization of large language models: A survey. arXiv preprint arXiv:2411.00027

    2024 arXiv

  50. [58]

    Aakas Zhiyuli, Yanfang Chen, Xuan Zhang, and Xun Liang. 2023. Bookgpt: A general framework for book recommendation empowered by large language model. arXiv preprint arXiv:2305.15673

    2023 arXiv

  51. [59]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  52. [60]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.