Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PERSONACONVBENCH is a new Reddit-based benchmark showing that LLMs predict sentiment, community scores, and next replies better when given a user's multi-turn conversation history, and it releases public data and code.

desk verdict Useful new benchmark, but the headline gains are inflated by temporal leakage and evaluation choices; fixable. read the letter →

arxiv 2505.14106 v2 pith:VOVGTRYI submitted 2025-05-20 cs.CL cs.AI

Li Li , Peilin Cai , Ryan A. Rossi , Franck Dernoncourt , Branislav Kveton , Junda Wu , Tong Yu , Linxin Song
show 16 more authors
Tiankai Yang Yuehan Qin Nesreen K. Ahmed Samyadeep Basu Subhojyoti Mukherjee Ruiyi Zhang Zhengmian Hu Bo Ni Yuxiao Zhou Zichao Wang Yue Huang Yu Wang Xiangliang Zhang Philip S. Yu Xiyang Hu Yue Zhao
This is my paper
classification cs.CLcs.AI
keywords personalizedbenchmarkconversationalllmspersonaconvbenchclassificationcontextconversations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PERSONACONVBENCH is a benchmark built from real Reddit posts in ten communities, from world news to art. For each user, the benchmark collects the threads they participated in and splits each thread into a known part and a hidden part. Three tasks use this split: predict whether a hidden reply is positive or negative, predict its upvote score, or generate its text. The benchmark compares two ways of prompting a language model: with the user's own conversation history (personalized conversational, P-Conv), and without it (personalized non-conversational, P-NonConv, which sees only the user's posts). The main result is that P-Conv beats P-NonConv across five models and all three tasks. The biggest reported effect is a 198% relative gain on the Matthews correlation coefficient for GPT-4.1 in sentiment classification, but that metric starts near zero, so the relative gain is larger than it sounds. The paper also reports an 11.5% gain in regression and a 35.1% gain in generation, where absolute scores are still modest. Several details should be read with care. Generation scores are computed by generating ten candidate replies and keeping the one most similar to the true answer, which favors the reported numbers. The regression prompt template includes a hint that the score is expected to be positive, which leaks or misstates the label. The user history is not formally time-bounded in the paper's definitions, so temporal leakage is possible if the released code does not enforce it. Overall, the benchmark is a useful new resource for studying user-adaptive dialogue, and its data and code are public, but the headline magnitudes should not be taken at face value.
Extended reading notes

Core claim

The paper's load-bearing empirical assertion is that conditioning an LLM on a user's conversational history (the current thread prefix plus the user's other trajectories, the P-Conv setting) yields substantial gains over the best non-conversational personalized baseline (P-NonConv) on all three tasks, headlined as 'a 198% relative gain over the best non-conversational baseline in sentiment classification' (Abstract), corresponding to GPT-4.1 MCC rising from 0.2266 to 0.6770 in Table 3, plus an 11.5% regression gain and a 35.1% generation gain. If correct, it establishes that personalized multi-turn context, not just user text snippets, drives measurable LLM performance differences on this benchmark.

Load-bearing premise

The temporal-consistency guarantee: the user trajectory set Cu minus the current trajectory C, supplied to the model, must contain only messages temporally prior to the target mτ, and must not share the target's post except through the stated context. The formal definition of Cu (Section 2.1) places no time bound on its trajectories, Section 3.1 includes Cu \ C with no explicit t < tτ filter, and the appendix templates (Table A) show few-shot demonstrations drawn from the same post ID (u1p1) as the test target. If the released code does not enforce the time bound, the reported gains could reflect leakage of future user behavior rather than personalization.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PersonaConvBench, a benchmark built from Reddit conversation trees across 10 domains, with three tasks: personalized conversational sentiment classification, impact forecasting, and follow-up text generation. The authors evaluate five LLMs under a Personalized Conversational condition (P-Conv, which receives the current thread prefix plus the user's other trajectories), a Personalized Non-Conversational condition (P-NonConv, which receives only the root post and a fixed demonstration), and for two models a Non-Personalized Conversational condition (NP-Conv). The central empirical claim is that P-Conv substantially outperforms P-NonConv on all three tasks, headlined as a 198% relative MCC gain for GPT-4.1 on sentiment classification (MCC 0.2266 to 0.6770), an 11.5% regression gain, and a 35.1% generation gain. The paper also releases the dataset and code.

Significance. If the headline results survive scrutiny, PersonaConvBench would be a useful community resource: it is one of the first benchmarks to combine user-level personalization with multi-user, graph-structured conversation, it covers three tasks across 10 domains at reasonable scale (19,215 posts, 111,239 conversations, 3,878 users), and it uses a unified prompting protocol across five diverse LLMs. The open release of data and code, and the paired significance testing for the generation task, are positive features. However, the central empirical contribution currently rests on two load-bearing issues: the temporal-consistency guarantee in Section 2.3 is not implemented in the formal definitions or the prompt templates, and the generation scores in Appendix C.3 are produced by selecting the candidate with the highest SBERT similarity to the ground truth, which is an oracle procedure over the reported metric. These issues must be resolved and the experiments rerun before the claimed gains can be attributed to personalized conversational context.

major comments (4)
  1. [Section 2.1, Section 2.3, Section 3.1, Tables A-D] The temporal-consistency guarantee is not enforced by the written definitions. Section 2.1 defines the user trajectory set Cu as 'all trajectories containing messages authored by u' with no timestamp bound relative to the target mτ, and Section 3.1 states that the prompt includes 'the rest of the user's data Cu \ C' without a t < tτ filter. This contradicts Section 2.3's promise that 'only prior context is accessible ... preventing any leakage from future content.' The risk is concrete: in Tables A-D the few-shot demonstration is taken from the same post (u1p1) as the actual task target (also u1p1), and in Table A the demonstration reply u1p1-c3-c4-c5-c6 lies in the same branch as the target u1p1-c3-c4. If the released code implements Cu \ C literally, the P-Conv condition can see the user's later replies, including replies in the same post tree, so the reported P-Conv gains (e.g., GPT-4.1 MCC 0.2266 to 0.6770 in Table 3) may reflect future-content leakage rather than personalization. Please enforce and document a strict temporal cutoff on all conditioning data, verify that no demonstration or history item shares the target post tree, and rerun the full comparison; the revised numbers should be reported for all models and tasks.
  2. [Appendix C.3; Section 2.3.3; Table 3] The generation evaluation is an oracle over the headline metric. Appendix C.3 states: 'The final score for each input was computed by selecting, from those 10 candidates, the one with the highest SBERT score as the “Best Response.”' Because SBERT similarity to the ground truth is also the reported semantic metric, the generation scores in Table 3 are obtained by optimizing the evaluation metric over a candidate set. This makes the reported SBERT numbers an upper-bound estimate rather than a measure of a deployed sampling strategy, and the lexical metrics computed on the selected candidate inherit the same selection effect. The P-Conv vs P-NonConv comparison is same-treatment, but the absolute numbers and the 35.1% generation gain cannot be interpreted as standard single-output quality. Please report greedy decoding or average-of-samples results as the primary numbers, and present best-of-k selection as a separate analysis.
  3. [Abstract; Section 4.2; Table 3] The headline '198% relative gain' is computed as (0.6770 - 0.2266) / 0.2266 on the MCC scale. MCC is a bounded correlation coefficient, not a ratio-scale quantity, so a relative percentage change is not meaningful and overstates the improvement; the actual change is 0.45 MCC points. Please report absolute differences with confidence intervals, and avoid percentage-change language for MCC and other bounded metrics.
  4. [Appendix E.1; Table 3] Paired significance testing is reported only for the generation task. The central classification and regression comparisons in Table 3 are presented without significance tests or confidence intervals, even though instances are paired by construction across the P-Conv and P-NonConv conditions. Please provide paired tests or bootstrap confidence intervals for all three tasks and all five models, and report effect sizes rather than relying on large-sample point estimates alone.
minor comments (5)
  1. [Section 2.2.1; Appendix C.1] The benchmark construction uses several thresholds (θ, Ns = 3, Ns2 = 2, Nu = 4, Nr = 4, Np = 3, and the bottom-55% positive-score filter), but the paper reports no sensitivity analysis. At minimum, please state whether the headline gains persist under reasonable variations of these thresholds, since they define the entire dataset.
  2. [Table Q; Table E] The NP-Conv value for DeepSeek-R1 impact forecasting in Table Q is RMSE 551.67, far outside the range of all other RMSE values (roughly 282–354), and the table notes that NP-Conv data is marked as '-' for these models in the provided source. Please verify this number and correct any inconsistency.
  3. [Section 3.1 vs Section 2.3] Section 3.1 says that for classification and regression 'the text xτ is revealed,' while Section 2.3 says 'the content of the target message mτ is masked during evaluation.' Please clarify exactly which fields are masked for each of the three tasks; the current wording is contradictory.
  4. [Figure 2 caption] The caption states that RMSE and MAE are normalized to [0,1] using RMSEscaled = (360 - RMSE)/70 and MAEscaled = (120 - MAE)/30, but these formulas can produce values outside [0,1] for the reported ranges. Please use a proper min-max normalization or describe the scaling as a heuristic.
  5. [Appendix C.3] Claude was limited to one generated candidate 'to preserve performance,' while all other models generated 10 candidates. This creates a model-dependent evaluation protocol; please justify this choice or treat Claude's generation scores separately in the comparisons.
Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The benchmark's conclusions rest on a chain of hand-chosen construction thresholds (score-based eligibility, the 55% positive filter, per-user minimums) and on domain assumptions about Reddit scores as sentiment proxies and user histories as personalization signals. The most consequential are the score-derived labels and the filtering rules that shape the class ratio; both are disclosed in the appendix but no sensitivity analysis is given. No new theoretical entities are introduced.

free parameters (8)
  • Sentiment threshold theta = 0 (implicit)
    Binary sentiment is defined as score > theta (Section 2.2.1). The threshold is a construction choice that determines all classification labels; no sensitivity analysis is reported.
  • Prediction-target minimum |score| Ns = 3
    Replies whose |score| is at least 3 are eligible as prediction targets (Appendix C.1). This filters the label set and is hand-chosen.
  • Demonstration-target minimum |score| Ns2 = 2
    Replies with |score| at least 2 are eligible as few-shot demonstration targets (Appendix C.1).
  • Bottom-55% positive filter = 55%
    Posts whose positive-score target replies rank in the bottom 55% by absolute score are dropped (Appendix C.1). This filter directly reshapes the class ratio and is hand-chosen.
  • Minimum users per post Nu = 4
    Posts must have more than 4 distinct users to be included (Appendix C.1).
  • Minimum author replies Nr = 4
    The author must have at least 4 replies in a post to qualify (Appendix C.1).
  • Minimum valid posts per user Np = 3
    A user must have at least 3 qualifying posts to enter the dataset (Appendix C.1).
  • Generation candidate count = 10
    All models except Claude generate 10 candidate responses, and the best-SBERT candidate is scored (Appendix C.3). This selection count is a hand-chosen evaluation parameter that inflates reported generation scores.
assumptions (5)
  • domain assumption Reddit score sign is a valid proxy for reply sentiment
    Classification labels are deterministically derived from the community score (Section 2.2.1). If upvote signals capture agreement and visibility rather than sentiment, the classification task measures score-sign prediction rather than sentiment.
  • domain assumption User style and intent are inferable from prior trajectories in the same dataset
    Personalization is operationalized as conditioning on Cu, the user's other trajectories (Section 2.1). The benchmark assumes these histories carry transferable signal for classification, regression, and generation targets.
  • domain assumption The reply graph edges and timestamps are correct as crawled
    Trajectories are time-ordered paths in the crawled reply graph (Section 2.1); malformed or missing parent links would break the context construction.
  • domain assumption Crawling by 'hotness' gives a representative sample of each domain
    Posts were sorted by hotness and iteratively processed (Appendix C.1), biasing the corpus toward popular, high-engagement threads.
  • standard math Paired t-test assumptions hold for the generation metrics
    Table F reports paired t-statistics at alpha = 0.01; this presumes approximate normality of paired metric differences across instances.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations." pith.science (2026). https://pith.science/paper/VOVGTRYI

@misc{pith2026250514106,
  author       = {Pith},
  title        = {Pith review of: A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VOVGTRYI}},
  note         = {Machine review of arXiv:2505.14106}
}
read the original abstract

We present PersonaConvBench, a large-scale benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs). Unlike existing work that focuses on either personalization or conversational structure in isolation, PersonaConvBench integrates both, offering three core tasks: sentence classification, impact regression, and user-centric text generation across ten diverse Reddit-based domains. This design enables systematic analysis of how personalized conversational context shapes LLM outputs in realistic multi-user scenarios. We benchmark several commercial and open-source LLMs under a unified prompting setup and observe that incorporating personalized history yields substantial performance improvements, including a 198 percent relative gain over the best non-conversational baseline in sentiment classification. By releasing PersonaConvBench with evaluations and code, we aim to support research on LLMs that adapt to individual styles, track long-term context, and produce contextually rich, engaging responses.

Figures

Figures reproduced from arXiv: 2505.14106 by the authors.

Figure 1
Figure 1. Illustration of the personalized conversational setting in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance of GPT-4.1 on our personalized conversation benchmark. Incorporating [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Benchmarking the Personalization Capabilities of Large Language Models

    cs.AI 2026-05 conditional novelty 6.0 of 10

    Across a new sales-outreach benchmark (SDR-Bench), frontier LLMs recover at most ~55% of the strategic pitch points from human-authored deal-winning messages, with no model statistically separating successful from uns...

  2. Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization

    cs.AI 2026-02 conditional novelty 5.0 of 10

    BAO, a behavior-enhanced SFT plus regularized RL pipeline, improves proactive agents' task performance while lowering user-involvement rate, beating UserRL baselines on three UserRL gym tasks.

Reference graph

Works this paper leans on

48 extracted references · 32 canonical work pages · cited by 2 Pith papers

  1. [1]

    Conversational health agents: A personalized llm-powered agent framework.arXiv preprint arXiv:2310.02374, 2023

    Mahyar Abbasian, Iman Azimi, Amir M Rahmani, and Ramesh Jain. Conversational health agents: A personalized llm-powered agent framework.arXiv preprint arXiv:2310.02374, 2023

    arXiv 2023

  2. [2]

    Persobench: Benchmarking personalized response generation in large language models, 2024

    Saleh Afzoon, Usman Naseem, Amin Beheshti, and Zahra Jamali. Persobench: Benchmarking personalized response generation in large language models, 2024

    work page 2024

  3. [3]

    Claude 3.5 sonnet

    anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet ,

    work page

  4. [4]

    A Little Human Data Goes A Long Way

    Dhananjay Ashok and Jonathan May. A little human data goes a long way.arXiv preprint arXiv:2410.13098, 2024

    work page Pith review arXiv 2024

  5. [5]

    Personalized graph-based retrieval for large language models.arXiv preprint arXiv:2501.02157, 2025

    Steven Au, Cameron J Dimacali, Ojasmitha Pedirappagari, Namyong Park, Franck Dernoncourt, Yu Wang, Nikos Kanakaris, Hanieh Deilamsalehy, Ryan A Rossi, and Nesreen K Ahmed. Personalized graph-based retrieval for large language models.arXiv preprint arXiv:2501.02157, 2025

    arXiv 2025

  6. [6]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  7. [7]

    Lore: Personalizing llms via low-rank reward modeling.arXiv preprint arXiv:2504.14439, 2025

    Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, and Maryam Fazel. Lore: Personalizing llms via low-rank reward modeling.arXiv preprint arXiv:2504.14439, 2025

    arXiv 2025

  8. [8]

    Optimal classifier for imbalanced data using matthews correlation coefficient metric.PloS one, 12(6):e0177678, 2017

    Sabri Boughorbel, Fethi Jarray, and Mohammed El-Anbari. Optimal classifier for imbalanced data using matthews correlation coefficient metric.PloS one, 12(6):e0177678, 2017

    work page 2017

Show all 48 references
  1. [9]

    Beyond prompts: Dy- namic conversational benchmarking of large language models

    David Castillo-Bolado, Joseph Davidson, Finlay Gray, and Marek Rosa. Beyond prompts: Dy- namic conversational benchmarking of large language models. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  2. [10]

    Root mean square error (rmse) or mean absolute error (mae).Geoscientific model development discussions, 7(1):1525–1534, 2014

    Tianfeng Chai, Roland R Draxler, et al. Root mean square error (rmse) or mean absolute error (mae).Geoscientific model development discussions, 7(1):1525–1534, 2014

    2014

  3. [11]

    When large language models meet personalization: Perspectives of challenges and opportunities.World Wide Web, 27(4):42, 2024

    Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, et al. When large language models meet personalization: Perspectives of challenges and opportunities.World Wide Web, 27(4):42, 2024

    2024

  4. [12]

    Realm: A dataset of real-world llm use cases.arXiv preprint arXiv:2503.18792, 2025

    Jingwen Cheng, Kshitish Ghate, Wenyue Hua, William Yang Wang, Hong Shen, and Fei Fang. Realm: A dataset of real-world llm use cases.arXiv preprint arXiv:2503.18792, 2025

    2025 arXiv

  5. [13]

    The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.BMC genomics, 21:1–13, 2020

    Davide Chicco and Giuseppe Jurman. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.BMC genomics, 21:1–13, 2020

    2020

  6. [14]

    The matthews correlation coefficient (mcc) is more informative than cohen’s kappa and brier score in binary classification assessment

    Davide Chicco, Matthijs J Warrens, and Giuseppe Jurman. The matthews correlation coefficient (mcc) is more informative than cohen’s kappa and brier score in binary classification assessment. Ieee Access, 9:78368–78381, 2021

    2021

  7. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  8. [16]

    Redcaps: Web-curated image-text data created by the people, for the people.arXiv preprint arXiv:2111.11431, 2021

    Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people.arXiv preprint arXiv:2111.11431, 2021

    2021 arXiv

  9. [17]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, et al. The llama 3 herd of models, 2024

    2024

  10. [18]

    Ruddit: Norms of offensiveness for english reddit comments.arXiv preprint arXiv:2106.05664, 2021

    Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M Mohammad, and Ekaterina Shutova. Ruddit: Norms of offensiveness for english reddit comments.arXiv preprint arXiv:2106.05664, 2021. 10

    2021 arXiv

  11. [19]

    Root mean square error (rmse) or mean absolute error (mae): When to use them or not.Geoscientific Model Development Discussions, 2022:1–10, 2022

    Timothy O Hodson. Root mean square error (rmse) or mean absolute error (mae): When to use them or not.Geoscientific Model Development Discussions, 2022:1–10, 2022

    2022

  12. [20]

    Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, Nedim Lipka, Chien Van Nguyen, Thien Huu Nguyen, and Hamed Zamani

    Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A. Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, Nedim Lipka, Chien Van Nguyen, Thien Huu Nguyen, and Hamed Zamani. Longlamp: A benchmark for personalized long-f...

    2024

  13. [21]

    Mt-eval: A multi-turn capabilities evaluation benchmark for large language models, 2024

    Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models, 2024

    2024

  14. [22]

    A framework for building adaptive intelligent virtual assistants

    Luc Lamontagne, François Laviolette, Richard Khoury, and Alexandre Bergeron-Guyard. A framework for building adaptive intelligent virtual assistants. InArtificial intelligence and applications, volume 10, pages 2014–816, 2014

    2014

  15. [23]

    Teach llms to personalize–an approach inspired by writing education

    Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. Teach llms to personalize–an approach inspired by writing education. arXiv preprint arXiv:2308.07968, 2023

    2023 arXiv

  16. [24]

    Panoptic scene graph generation with semantics-prototype learning.Proceedings of the AAAI Conference on Artificial Intelligence, 38(4):3145–3153, Mar

    Li Li, Wei Ji, Yiming Wu, Mengze Li, You Qin, Lina Wei, and Roger Zimmermann. Panoptic scene graph generation with semantics-prototype learning.Proceedings of the AAAI Conference on Artificial Intelligence, 38(4):3145–3153, Mar. 2024

    2024

  17. [25]

    Artificial intelligence in intelligent tutor- ing systems toward sustainable education: a systematic review.Smart Learning Environments, 10(1):41, 2023

    Chien-Chang Lin, Anna YQ Huang, and Owen HT Lu. Artificial intelligence in intelligent tutor- ing systems toward sustainable education: a systematic review.Smart Learning Environments, 10(1):41, 2023

    2023

  18. [26]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  19. [27]

    Persona-sq: A personalized suggested question generation framework for real-world documents, 2024

    Zihao Lin, Zichao Wang, Yuanting Pan, Varun Manjunatha, Ryan Rossi, Angela Lau, Lifu Huang, and Tong Sun. Persona-sq: A personalized suggested question generation framework for real-world documents, 2024

    2024

  20. [28]

    Soda-eval: Open-domain dialogue evaluation in the age of llms, 2024

    John Mendonça, Isabel Trancoso, and Alon Lavie. Soda-eval: Open-domain dialogue evaluation in the age of llms, 2024

    2024

  21. [29]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/ , 2024. Accessed: 2024-07- 18

    2024

  22. [30]

    Introducing gpt-4.1 in the api

    OpenAI. Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/ , 2025. Accessed: 2025-05-13

    2025

  23. [31]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  24. [32]

    Divya Patel, Pathik Patel, Ankush Chander, Sourish Dasgupta, and Tanmoy Chakraborty. Are large language models in-context personalized summarizers? get an icopernicus test done! In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16...

    2024

  25. [33]

    Nikhil Patel and Sandeep Trivedi. Leveraging predictive modeling, machine learning personal- ization, nlp customer support, and ai chatbots to increase customer loyalty.Empirical Quests for Management Essences, 3(3):1–24, 2020

    2020

  26. [34]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019

    2019

  27. [35]

    Lamp: When large language models meet personalization, 2024

    Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization, 2024. 11

    2024

  28. [36]

    Ai models collapse when trained on recursively generated data.Nature, 631(8022):755– 759, 2024

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data.Nature, 631(8022):755– 759, 2024

    2024

  29. [37]

    Democra- tizing large language models via personalized parameter-efficient fine-tuning

    Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democra- tizing large language models via personalized parameter-efficient fine-tuning. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6476–6491, 2024

    2024

  30. [38]

    An ai-based decision support system for predicting mental health disorders.Information Systems Frontiers, 25(3):1261–1276, 2023

    Salih Tutun, Marina E Johnson, Abdulaziz Ahmed, Abdullah Albizri, Sedat Irgil, Ilker Yesilkaya, Esma Nur Ucar, Tanalp Sengun, and Antoine Harfouche. An ai-based decision support system for predicting mental health disorders.Information Systems Frontiers, 25(3):1261–1276, 2023

    2023

  31. [39]

    Position: Will we run out of data? limits of llm scaling based on human-generated data

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of llm scaling based on human-generated data. InForty-first International Conference on Machine Learning, 2024

    2024

  32. [40]

    Personalized multimodal large language models: A survey.arXiv preprint arXiv:2412.02142, 2024

    Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A Rossi, Franck Dernoncourt, et al. Personalized multimodal large language models: A survey.arXiv preprint arXiv:2412.02142, 2024

    2024 arXiv

  33. [41]

    A survey on recent advances in llm-based multi-turn dialogue systems.arXiv preprint arXiv:2402.18013, 2024

    Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. A survey on recent advances in llm-based multi-turn dialogue systems.arXiv preprint arXiv:2402.18013, 2024

    2024 arXiv

  34. [42]

    xdial-eval: A multilingual open-domain dialogue evaluation benchmark

    Chen Zhang, Luis Fernando D’Haro, chengguang tang, Ke Shi, Guohua Tang, and Haizhou Li. xdial-eval: A multilingual open-domain dialogue evaluation benchmark. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  35. [43]

    Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, and Yi R. Fung. Vlm2-bench: A closer look at how well vlms implicitly link explicit matching visual cues, 2025

    2025

  36. [44]

    Personalization of large language models: A survey.arXiv preprint arXiv:2411.00027, 2024

    Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. Personalization of large language models: A survey.arXiv preprint arXiv:2411.00027, 2024

    2024 arXiv

  37. [45]

    DiQAD: A benchmark dataset for open-domain dialogue quality assessment

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. DiQAD: A benchmark dataset for open-domain dialogue quality assessment. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  38. [46]

    Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong

    Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Personalllm: Tailoring llms to individual preferences, 2025. 12 A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations Supplementary Material A Related Work A.1 Pe...

    2025

  39. [48]

    Best Response

    assess aspects like commonsense knowledge and coherence, while [42] extend such evaluations to multilingual scenarios. Further, [ 9] explore long-term memory and continual learning in more dynamic, multi-round conversational contexts. While these benchmarks advance general-pur...

    2021

  40. [2024]

    Accessed: 2024-06-20

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.