Pith. sign in

REVIEW 2 cited by

T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.

T0 review reviewed 2026-08-04 challenge →

arxiv 2509.24696 v2 pith:JUAI5UFI submitted 2025-09-29 cs.LG cs.AI

T-POP: Test-Time Personalization with Online Preference Feedback

Zikun Qu , Min Zhang , Mingze Kong , Xiang Li , Zhiwei Shang , Zhiyong Wang , Yikun Ban , Shuang Qiu
show 2 more authors
Yao Shu Zhongxiang Dai
This is my paper
classification cs.LG cs.AI
keywords personalizationt-popuseronlinefeedbackpreferencepreferencestest-time
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models usually generate the same style of answer for everyone unless they are retrained or given lots of user history. T-POP tries to adapt a frozen model to a new user during the conversation itself. At each round, it generates two answers to the same question, following two different strategies. The first answer is the best the model can currently produce according to its latest guess about the user's taste. The second answer deliberately tries something different, picking words that look uncertain under the current reward model. The user says which answer they prefer. That one bit of feedback is used to update a small neural-network reward model that scores how well a piece of text matches the user's taste. Over several rounds, the reward model improves, and the first answer becomes more personalized.

The algorithm borrows ideas from dueling bandits, a framework for learning from pairwise choices, and applies them one token at a time. At each decoding step, the candidate tokens are the most likely tokens from the base model. The first response takes the token with the highest score under the current reward model. The second response adds an exploration bonus that measures how much the token's gradient with respect to the reward model differs from the token already chosen in the first response, scaled by an inverse covariance matrix built from past comparisons.

The authors test T-POP on several open-source LLMs using a benchmark setup where GPT-4o plays the role of the user, giving preference feedback based on attributes like 'verbose' or 'concise'. They also use GPT-4o as a judge to measure how often T-POP beats the base model. The reported results show faster improvement with fewer interactions than the baselines, but the evaluation is entirely simu

Core claim

The paper asserts that 'without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences', and that 'T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines'. If correct, the load-bearing result is that a handful of pairwise preference comparisons during text generation suffice to personalize a frozen LLM's decoding policy.

Load-bearing premise

The reward model r(·;θ) is trained on complete responses (Eq. 2) but is applied during decoding to score partial sequences plus a candidate token: Eq. (1) and the token-selection rules in Eqs. (3)-(4) use r([y_p, v]; θ) for every candidate v. If the reward model's value on a prefix is not a faithful estimate of the contribution of token v to the final user preference, the token-level greedy and exploration choices will be misaligned with the sequence-level feedback, and the claimed data efficiency would not follow. This assumption is not stated or tested in the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method relies on a small set of hand-tuned hyperparameters, the most consequential being omega and nu, which directly control the balance between the base model and the learned reward and the strength of exploration. The central algorithmic claim also rests on the unstated assumptions that the reward model transfers from full responses to prefixes and that the gradient-based UCB bonus is informative. No new physical or conceptual entities are introduced.

free parameters (5)
  • Reward weight omega = 1.0
    Selected from ablation on the same test sets (Fig. 3). The central decoding score is omega * r(...).
  • Exploration parameter nu = 0.5
    Set in Table 4 without ablation or sensitivity analysis; scales the UCB exploration bonus in Eq. (4).
  • Candidate token set size k = 40
    Restricts the decoding search space; chosen by hand.
  • Regularization lambda = 1.0
    Weight decay in the BTL loss (Eq. 2); fixed without sensitivity analysis.
  • Max new tokens M = 128
    Decoding length cap; chosen by hand.
axioms (4)
  • domain assumption User preference choices follow the Bradley-Terry-Luce model with unknown reward f.
    Assumed in Sec. 2 to relate pairwise preferences to the reward function.
  • ad hoc to paper The reward model trained on complete responses generalizes to partial sequences during token-level decoding.
    Eq. (1) and Eqs. (3)-(4) score prefixes [y, v], but the loss (Eq. 2) trains only on complete responses from D_t.
  • ad hoc to paper The gradient-difference uncertainty bonus in Eq. (5) measures the information value of a candidate token for learning the user preference.
    Stated as 'Theoretical Justifications' in Sec. 3.3, but no theorem or regret bound is given; the argument is an analogy to neural dueling bandits.
  • ad hoc to paper Summing the base model's probability and the reward output in Eq. (1) is a valid personalized decoding objective.
    The two quantities have different scales; no normalization or theoretical justification is provided.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of T-POP: Test-Time Personalization with Online Preference Feedback." pith.science (2026). https://pith.science/paper/JUAI5UFI

@misc{pith2026250924696,
  author       = {Pith},
  title        = {Pith review of: T-POP: Test-Time Personalization with Online Preference Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JUAI5UFI}},
  note         = {Machine review of arXiv:2509.24696}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback}), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions.

Figures

Figures reproduced from arXiv: 2509.24696 by Mingze Kong, Min Zhang, Shuang Qiu, Xiang Li, Yao Shu, Yikun Ban, Zhiwei Shang, Zhiyong Wang, Zhongxiang Dai, Zikun Qu.

Figure 1
Figure 1. Figure 1: An overview of our T-POP for test-time personalization with online preference feedback. these methods, however, is the availability of sufficient user data. This leaves them inapplicable to new users for whom such data has not yet been collected, a critical challenge in the field of personalization known as the cold-start problem (Zhang et al., 2024). To resolve this problem, a natural solution is to colle… view at source ↗
Figure 2
Figure 2. Figure 2: The effect of the number of user interactions on the Reward Score for different models. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The effect of the reward weight (w) on the alignment performance of T-POP across three different backbone models. 5 ABLATION STUDY The Impact of Reward Weight w [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Natural language descriptions for the personalized preference attributes. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The prompt template used to instruct GPT-4o for preference simulation and win rate [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures

    cs.LG 2026-05 unverdicted novelty 7.0

    Test-time scaling for personalized LLMs follows a logarithmic utility curve under oracle selection but standard reward models suffer user-level collapse and query-level hacking; a probabilistic reward model with learn...

  2. Cloud-native and Distributed Systems for Efficient and Scalable Large Language Models -- A Research Agenda

    cs.DC 2026-04 unverdicted novelty 2.0

    This research agenda argues that cloud-native architectures, microservices, autoscaling, and emerging trends like serverless inference and federated learning are required to make large language models efficient and scalable.

Reference graph

Works this paper leans on

27 extracted references · 22 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Pith/arXiv arXiv

  2. [3]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  3. [4]

    Adaptive sample scheduling for direct preference optimization.arXiv preprint arXiv:2506.17252,

    Zixuan Huang, Yikun Ban, Lean Fu, Xiaojie Li, Zhongxiang Dai, Jianxin Li, and Deqing Wang. Adaptive sample scheduling for direct preference optimization.arXiv preprint arXiv:2506.17252,

    arXiv

  4. [7]

    Args: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv preprint arXiv:2402.01694,

    Pith/arXiv arXiv

  5. [8]

    Few-shot personalization of llms with mis-aligned responses

    Jaehyung Kim and Yiming Yang. Few-shot personalization of llms with mis-aligned responses. arXiv preprint arXiv:2406.18678,

    Pith/arXiv arXiv

  6. [9]

    Learning to rewrite prompts for personalized text generation

    Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. Learning to rewrite prompts for personalized text generation. InProceedings of the ACM Web Conference 2024, pp. 3367–3378, 2024a. Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, and Rui Yan. From 1,000,000 users to every user: Scaling up personalized preference for user-level alignment,

    2024

  7. [11]

    Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    Pith/arXiv arXiv

  8. [13]

    Is chatgpt a good recommender? a preliminary study.arXiv preprint arXiv:2304.10149,

    Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, and Yan Zhang. Is chatgpt a good recommender? a preliminary study.arXiv preprint arXiv:2304.10149,

    Pith/arXiv arXiv

  9. [14]

    Decoding-time realignment of language models.arXiv preprint arXiv:2402.02992,

    Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models.arXiv preprint arXiv:2402.02992,

    Pith/arXiv arXiv

  10. [16]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

    URL https://arxiv.org/abs/2410.21276. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35: 27730–27744,

    Pith/arXiv arXiv

  11. [18]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  12. [20]

    Neural dueling bandits: Preference-based optimization with human feedback.arXiv preprint arXiv:2407.17112,

    Arun Verma, Zhongxiang Dai, Xiaoqiang Lin, Patrick Jaillet, and Bryan Kian Hsiang Low. Neural dueling bandits: Preference-based optimization with human feedback.arXiv preprint arXiv:2407.17112,

    Pith/arXiv arXiv

  13. [21]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts.arXiv preprint arXiv:2406.12845,

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts.arXiv preprint arXiv:2406.12845,

    Pith/arXiv arXiv

  14. [22]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al

    URLhttps://arxiv.org/ abs/2311.09528. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  15. [23]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025a

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025a. URLhttps: //arxiv.org/abs/2506.05176. Zhaowei Zhang, Fengshuo Bai, Qizhi Chen, Chengdong Ma, Mingzhi Wang, Haoran...

    Pith/arXiv arXiv

  16. [24]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

    Pith/arXiv arXiv 1909

  17. [26]

    is a comprehensive, high-quality Al feedback dataset. From this, we selected two subsets:Truthful QA(Lin et al., 2021), using its 811 testing problems to assess factuality, andUltraChat, from which we extracted 3,845 problems to evaluate conversational ability (Zhang et al., 2025b). •Personal Preference Eval(Personal) (Gao et al.,

    2021

  18. [50]

    creative

    Decoding Parameters Max new tokens 128 Candidate tokens (k) 40 For the weight decay schedule,Ndenotes the number of training data points. Embedding and Judge Models.The reward model utilizes embeddings from the Qwen/Qwen3-Embedding-0.6Bmodel (Zhang et al., 2025a). For all experiments requiring preference evaluation, including the simulation of user feedba...

    2024

  19. [1959]

    Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers.arXiv preprint arXiv:2311.09180,

    Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers.arXiv preprint arXiv:2311.09180,

    Pith/arXiv arXiv

  20. [2004]

    Personalized soups: Personalized large language model alignment via post-hoc parameter merging.arXiv preprint arXiv:2310.11564,

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging.arXiv preprint arXiv:2310.11564,

    Pith/arXiv arXiv

  21. [2019]

    This network takes the final hidden-state embeddings from the backbone LLM for a given sequence as input

    A MOREDETAILS ON THEEXPERIMENTALSETTING Reward Model Architecture.The lightweight reward model,r(·;θ), is implemented as a simple Multi-Layer Perceptron (MLP) head. This network takes the final hidden-state embeddings from the backbone LLM for a given sequence as input. The MLP consists of one hidden layer with a size of 1024, and all hidden layers utiliz...

    2025

  22. [2020]

    Persona-db: Efficient large language model personalization for response prediction with collaborative data refinement.arXiv preprint arXiv:2402.11060,

    Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi R Fung, Hou Pong Chan, Kevin Small, ChengXiang Zhai, and Heng Ji. Persona-db: Efficient large language model personalization for response prediction with collaborative data refinement.arXiv preprint arXiv:2402.11060,

    Pith/arXiv arXiv

  23. [2021]

    Prompt optimization with human feedback.arXiv preprint arXiv:2405.17346,

    Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Prompt optimization with human feedback.arXiv preprint arXiv:2405.17346,

    Pith/arXiv arXiv

  24. [2022]

    Principled rlhf from heterogeneous feedback via personalization and preference aggregation.arXiv preprint arXiv:2405.00254, 2,

    Chanwoo Park, Mingyang Liu, Kaiqing Zhang, and Asuman Ozdaglar. Principled rlhf from heterogeneous feedback via personalization and preference aggregation.arXiv preprint arXiv:2405.00254, 2,

    Pith/arXiv arXiv

  25. [2023]

    Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng

    URLhttps: //arxiv.org/abs/2310.06825. Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. Do llms understand user preferences? evaluating llms on user rating prediction.arXiv preprint arXiv:2305.06474,

    Pith/arXiv arXiv

  26. [2024]

    Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, et al

    URLhttps://arxiv.org/abs/2310.01377. Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, et al. Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback.arXiv preprint arXiv:2401.11458,

    Pith/arXiv arXiv

  27. [2025]

    Personalized language modeling from personalized human feedback.arXiv preprint arXiv:2402.05133, 2024b

    Xinyu Li, Ruiyang Zhou, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback.arXiv preprint arXiv:2402.05133, 2024b. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.