Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Supervised fine-tuning makes LLM reward models reliable enough to drive whole-dialogue impression, and DPO turns that feedback into better responses.

desk verdict Solid applied RLAIF recipe with convincing SFT-reward-model results, but the naturalness claim is overgeneralized and the human evaluation is too thin to carry the full claim. read the letter →

arxiv 2501.12698 v2 pith:5NIAEC45 submitted 2025-01-22 cs.CL

Kai Yoshida , Masahiro Mizukami , Seiya Kawano , Canasai Kruengkrai , Hiroaki Sugiyama , Koichiro Yoshino This is my paper · ORCID
classification cs.CL
keywords dialoguesystemsreinforcementlearningfromAIfeedbackrewardmodelimpressiondirectpreferenceoptimizationPPOsupervisedfine-tuningJapanese
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether AI feedback can improve a dialogue system's overall impression across a whole conversation, rather than just the quality of a single reply. It argues that prompting an LLM to grade an entire dialogue is unreliable, and that a reward model supervised fine-tuned on human scores for 12 impression metrics is a far better signal. Using that reward model to tune a dialogue model, the paper reports that direct preference optimization (DPO) raises the rewarded impression scores and lowers perplexity, and human raters judge the responses both more natural and more reflective of the target impression. The practical stake is that whole-dialogue qualities such as consistency, empathy, and personality can be optimized automatically once a reliable reward model exists.

What carries the argument

The load-bearing component is a reward model built by supervised fine-tuning: an LLM with a linear regression head trained, by mean squared error, to output eleven-point scores for 12 dialogue-impression metrics from a dialogue context plus a response. A single model handles all 12 metrics, and the 7-billion-parameter version is then used to score 4-turn dialogue contexts. In DPO, the model generates two responses to each context, the reward model labels one preferred and one rejected, and the dialogue model is trained to prefer the higher-scoring response; in PPO, the reward score is used as the reinforcement learning signal during generation. The argument runs through this reward model: the better it tracks human impression, the more the feedback loop can shape dialogue quality.

What would settle it

Collect fresh human impression scores for 4-turn dialogues, compare them with the reward model's predictions, and run a blind human test of the DPO-tuned model against the base model; if the rank correlation is near zero or the tuned model does not outrank the base model, the claimed feedback loop does not actually improve dialogue impression.

Watch

Extended reading notes

Core claim

The central claim is that supervised fine-tuning of an LLM on human-annotated whole-dialogue impressions creates a reward model that can stand in for human feedback during dialogue-model training. On a test set of 32-turn dialogues, the 7-billion-parameter SFT reward model reaches rank correlations with human scores of roughly 0.8 across the 12 metrics, far above the near-zero-to-0.4 range of prompted models. Applying this reward model to 4-turn JEmpathetic Dialogue contexts, DPO improves the rewarded metric by about 0.5 on an 11-point scale while also reducing perplexity, and human evaluation ranks DPO responses best on most metrics and rates them as more natural. The paper also reports that the reward model can give high scores to natural but dull replies such as "Yes," and identifies that as a remaining issue for RLAIF training.

Load-bearing premise

Everything rests on the reward model trained on 32-turn dialogues scoring 4-turn dialogues the way a human would, even though the paper notes it can reward dull replies such as "Yes."

Editorial extensions

If this is right

  • A dialogue model can be aligned to a chosen impression dimension such as empathy, trust, or personality without collecting new human preferences for every update, once the reward model has been trained.
  • DPO is the stronger of the two training strategies tested: it improves both the rewarded impression score and fluency, whereas PPO improves impression only modestly at the cost of fluency.
  • Prompting-only LLM judges are not enough to drive whole-dialogue impression tuning; the gap between prompted and SFT reward models directly motivates supervised reward-model training.
  • Reward models trained on long 32-turn dialogues transfer, at least in practice, to scoring short 4-turn training contexts, which makes the RLAIF loop feasible on standard dialogue data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the authors do not test is multi-objective DPO, mixing several impression rewards into one preference signal; because their reward model already scores 12 metrics at once, the data to try this exists.
  • The dull-response failure suggests a testable fix: add a diversity or fluency penalty, or train the reward model on labels that explicitly separate naturalness from impression, rather than letting naturalness bleed into impression scores.
  • If the reward model's transfer to short dialogues is real, the same recipe should work on longer or multi-speaker dialogues; if it fails there, the transfer assumption in this paper would be the reason.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using supervised fine-tuning to create regression-based reward models for 12 overall dialogue-impression metrics, then using those reward scores as AI feedback to train dialogue models with PPO and DPO. The reward model is trained on 32-turn JTransformer-Eval dialogues and applied to 4-turn JEmpathetic Dialogue contexts. The authors compare two 7B chat models (calm2 and youri) across 12 metrics, reporting automatic AIF scores and perplexity in Table III, human ranking in Table IV, and human naturalness ratings in Table V. They conclude that DPO tuning with the reward model improves both individual dialogue-impression metrics and response naturalness.

Significance. The problem addressed is important: moving from per-response evaluation to whole-dialogue impression is a key step for engaging dialogue systems. The paper's main strength is the SFT reward model, which reaches Spearman correlations of roughly 0.8 on the JTransformer-Eval test set (Table II), showing that supervised fine-tuning is much better than prompt-only evaluation for this task. The comparison of PPO and DPO across two base models and 12 metrics is also a useful empirical contribution. The DPO-over-baseline signal in human ranking for the calm model (Table IV) is genuinely suggestive. However, the evidence as presented is not yet sufficient for the paper's broad conclusion: the automatic AIF metric is computed by the same reward model used for training, the reward model is unvalidated on the short-dialogue optimization domain, and the human evaluation uses one rater per metric with no significance testing. These gaps are load-bearing because the central claim is that AI feedback improves the targeted impressions.

major comments (4)
  1. [§IV.B.1, Table III] The AIF metric in Table III is computed by the same SFT reward model that labels accepted and rejected responses for DPO in Section III.B and provides rewards for PPO. Reporting that DPO improved these scores after optimizing against the same model is partly tautological and cannot independently verify that the targeted dialogue-impression metrics improved. Please add an evaluation using a reward model not involved in training, or human labels on held-out JEmpathetic dialogues, and report the correlation of the reward model with human judgments on the short-dialogue domain.
  2. [§II.A, §II.C, §IV.A] The reward model is trained and evaluated only on 32-turn JTransformer-Eval dialogues, but it is applied without adaptation or validation to 4-turn JEmpathetic Dialogue contexts in PPO and DPO. Table II reports correlations only on the long-dialogue test set, so the validity of the reward signal on short dialogues is unknown. The paper's own Section VI observation that the reward model gives high scores to dull responses such as 'Yes' and 'I think it's good' further suggests that the model may be capturing a general naturalness prior rather than the intended metric-specific impressions, making the AIF improvements in Table III ambiguous evidence for the broad claim.
  3. [§IV.B.2, Tables IV and V] The human evaluation is based on one rater per metric with no significance tests or inter-annotator agreement, so the reported differences cannot be shown to be reliable. In addition, the naturalness results for youri in Table V are flat or negative relative to the baseline for several metrics (e.g., Agency 1.96 vs. 2.12, Ease 1.68 vs. 1.97), which contradicts the blanket statement in Section V.B that 'the improved naturalness of both PPO and DPO compared to pre-training' was observed. Please report significance tests and give per-model, per-metric conclusions rather than pooling across base models.
  4. [§III.B, Table IV] The 'Win' metric in Table IV is not defined precisely enough to interpret. The caption says it is the percentage 'at or above the same order compared to before the study,' but it is not clear whether ties count in favor of the trained model, what the denominator is, or how the ranking scale (1 best, 3 worst) maps onto win rates. Without a precise definition, the claim that 'DPO has the best results for calm and youri' on Rank and Win cannot be fully assessed.
minor comments (5)
  1. [Introduction, reference [10]] The text cites 'Cheng et al.' for the work on model training from LLM feedback, but reference [10] is by Pang et al.; please correct the author attribution.
  2. [Table III] The PPL value 913.52 for youri Ease under PPO appears to be a typo or an outlier; please verify this entry, as it is more than an order of magnitude larger than all other PPL values.
  3. [Section IV.B.2] The human-evaluation instruction says 'Please rank the questions for each Table I value,' but the task is to rank responses; please clarify the wording and state explicitly whether ties are allowed in the naturalness rating as well as in the ranking task.
  4. [Equation (1), Section IV.B.1] The sentence describing PPL calculation says that 'the base model of the post-enhanced training model is used for PPL evaluation,' which is ambiguous. Please specify whether the PPL is computed with the model being evaluated, with a fixed reference model, or with the SFT base model, since this affects the comparability of PPL values across conditions.
  5. [Section II.A] The dataset name 'JTransformer-Eval' and the model name 'japanese-dialog-transformer' should be spelled consistently, and the relationship between the 1,600 dialogue sessions and the 32-turn dialogues should be stated clearly.

Circularity Check

1 steps flagged · score 6.0 of 10

The automatic AIF result is circular: the same SFT reward model both creates the DPO/PPO training signal and defines the AIF evaluation metric.

  1. self definitional [Section III-B (DPO/PPO training); AIF metric in Section IV-B1]
    "In DPO pre-processing, the model generates two types of responses to update the dialogue history, and the reward model evaluates these two sentences for the dialogue history. The generated sentence with the higher evaluation value is then used for training as accepted and the lower as rejected."

    The automatic AIF metric in Section IV-B1 is the same reward function used to make DPO labels: 'a reward model confirms that the evaluated values reflect the actual AI Feedback (AIF) values.' DPO accepts/rejects by comparing this reward-model score, and PPO maximizes it; AIF is then measured by that same score. An AIF increase after training therefore shows the model moved toward the fitted reward function, not that the 12 impression metrics improved. Section VI admits the reward model rates dull 'Yes' responses highly, confirming AIF is not a faithful proxy. The human evaluation is independent and prevents full circularity, but the automatic result is tautological.

full rationale

The paper's chain is mostly not circular: the reward model is trained on JTransformer-Eval annotations, and the final claim is also checked by human raters, which provides independent evidence. The circularity is confined to the 'AIF' automatic evaluation. Because accepted/rejected responses in DPO are chosen by the reward model and PPO maximizes that same reward model's output, reporting that the AIF score improved is partly definitional: the evaluation metric and the training objective are the same fitted function. The human evaluation stops this from making the entire paper circular, but it is based on one rater per metric and no significance testing, so it does not fully rescue the automatic headline. No load-bearing self-citation or imported uniqueness theorem was found; the JTransformer-Eval dataset citation is a resource, not a circular premise. Score 6 reflects one prediction (AIF improvement) that reduces by construction while the overall paper retains an independent, though weak, human-evaluation component.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new theoretical entities, but its central claim rests on a chain of empirical assumptions: the validity of the 12 metrics, transfer of the reward model from 32-turn to 4-turn dialogues, and reliability of single-rater human evaluation. No scalar free parameters are fit beyond the reward model itself and the RL hyperparameters, which are not reported.

assumptions (4)
  • domain assumption The 12 metrics in JTransformer-Eval are valid and comprehensive measures of overall dialogue impression.
    Section II.A introduces them as the evaluation metrics; the reward model is trained to predict them and the paper treats improvement on them as improvement in dialogue impression.
  • domain assumption The SFT reward model trained on 32-turn JTransformer-Eval dialogues generalizes to 4-turn JEmpathetic Dialogue contexts without adaptation.
    Section III applies the Section II reward model to JEmpathetic (Section IV.A, 4-turn dialogues); no validation of the domain shift is reported, so the reward may be miscalibrated on shorter dialogues.
  • domain assumption Optimizing the reward model's score improves the true perceived impression of the dialogue.
    The paper's own discussion notes the reward model gives high scores to dull, natural responses such as 'Yes' and 'I think it's good', so reward maximization may not correspond to better impression.
  • domain assumption A single human rater's 100 evaluations per metric are reliable enough to support the human evaluation conclusions.
    Section IV.B.2 reports one rater per metric with no inter-annotator agreement or significance testing; ranking with ties is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression." pith.science (2026). https://pith.science/paper/5NIAEC45

@misc{pith2026250112698,
  author       = {Pith},
  title        = {Pith review of: Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NIAEC45}},
  note         = {Machine review of arXiv:2501.12698}
}
read the original abstract

To improve user engagement during conversations with dialogue systems, we must improve individual dialogue responses and dialogue impressions such as consistency, personality, and empathy throughout the entire dialogue. While such dialogue systems have been developing rapidly with the help of large language models (LLMs), reinforcement learning from AI feedback (RLAIF) has attracted attention to align LLM-based dialogue models for such dialogue impressions. In RLAIF, a reward model based on another LLM is used to create a training signal for an LLM-based dialogue model using zero-shot/few-shot prompting techniques. However, evaluating an entire dialogue only by prompting LLMs is challenging. In this study, the supervised fine-tuning (SFT) of LLMs prepared reward models corresponding to 12 metrics related to the impression of the entire dialogue for evaluating dialogue responses. We tuned our dialogue models using the reward model signals as feedback to improve the impression of the system. The results of automatic and human evaluations showed that tuning the dialogue model using our reward model corresponding to dialogue impression improved the evaluation of individual metrics and the naturalness of the dialogue response.

Figures

Figures reproduced from arXiv: 2501.12698 by the authors.

Figure 1
Figure 1. Training of reward model and implementation of PPO and DPO, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 14 canonical work pages

  1. [1]

    Prompted LLMs as chatbot modules for long open-domain conversation

    Gibbeum Lee, V olker Hartmann, Jongho Park, Dimitris Papailiopoulos, and Kangwook Lee. Prompted LLMs as chatbot modules for long open-domain conversation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023 , pages 4536–4554, Toronto, Canada, July 2023. Association for Comput...

    work page 2023

  2. [2]

    Introducing chatgpt

    OpenAI. Introducing chatgpt. https://openai.com/index/chatgpt

    work page

  3. [3]

    Gpt-4 technical report, 2023

    OpenAI, Josh Achiam, and Steven Adler et al. Gpt-4 technical report, 2023

    work page 2023

  4. [4]

    An overview of bard: an early experiment with gener- ative ai, 2023

    James Manyika. An overview of bard: an early experiment with gener- ative ai, 2023. https://ai.google/static/documents/google-about-bard.pdf

    work page 2023

  5. [5]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedbac...

    arXiv 2022

  6. [6]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irv- ing. Fine-tuning language models from human preferences, 2019. arXiv:1909.08593

    arXiv 2019

  7. [7]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  8. [8]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017. arXiv:1706.03741

    arXiv 2017

Show all 25 references
  1. [9]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: Scaling re- inforcement learning from human feedback with ai feedback, 2023. arXiv:2309.00267

    2023 arXiv

  2. [10]

    Language model self-improvement by reinforcement learning contemplation, 2023

    Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-improvement by reinforcement learning contemplation, 2023. arXiv:2305.14483

    2023 arXiv

  3. [11]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan ...

    2022 arXiv

  4. [12]

    Reward design with language models

    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations , 2023

    2023

  5. [13]

    Beyond human data: Scaling self-training for problem- solving with language models

    Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie ...

    2024

  6. [14]

    Language instructed reinforcement learning for human-AI coordination

    Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-AI coordination. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine...

    2023

  7. [15]

    Unsupervised evaluation of inter- active dialog with DialoGPT

    Shikib Mehri and Maxine Eskenazi. Unsupervised evaluation of inter- active dialog with DialoGPT. In Olivier Pietquin, Smaranda Muresan, Vivian Chen, Casey Kennington, David Vandyke, Nina Dethlefs, Koji Inoue, Erik Ekstedt, and Stefan Ultes, editors, Proceedings of the 21th Ann...

    2020

  8. [16]

    MEEP: Is this engaging? prompting large language models for dialogue eval- uation in multilingual settings

    Amila Ferron, Amber Shore, Ekata Mitra, and Ameeta Agrawal. MEEP: Is this engaging? prompting large language models for dialogue eval- uation in multilingual settings. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguist...

    2023

  9. [17]

    InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews

    Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Lun- Wei Ku, An...

  10. [18]

    LLM-eval: Unified multi- dimensional automatic evaluation for open-domain conversations with large language models

    Yen-Ting Lin and Yun-Nung Chen. LLM-eval: Unified multi- dimensional automatic evaluation for open-domain conversations with large language models. In Yun-Nung Chen and Abhinav Rastogi, editors, Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023) , p...

    2023

  11. [19]

    G-eval: NLG evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proc...

    2023

  12. [20]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as- a-judge with mt-bench and chatbot arena, 2023. arXiv:2306.05685

    2023 arXiv

  13. [21]

    Empirical analysis of training strategies of transformer-based japanese chit-chat systems

    Hiroaki Sugiyama, Masahiro Mizukami, Tsunehiro Arimoto, Hiromi Narimatsu, Yuya Chiba, Hideharu Nakajima, and Toyomi Meguro. Empirical analysis of training strategies of transformer-based japanese chit-chat systems. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages ...

    2022

  14. [22]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. arXiv:1707.06347

    2017 arXiv

  15. [23]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christo- pher D. Manning, and Chelsea Finn. Direct preference opti- mization: Your language model is secretly a reward model, 2023. arXiv:2305.18290

    2023 arXiv

  16. [24]

    Towards empathetic open-domain conversation models: A new bench- mark and dataset

    Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open-domain conversation models: A new bench- mark and dataset. In Anna Korhonen, David Traum, and Llu ´ıs M`arquez, editors, Proceedings of the 57th Annual Meeting of the Association for Co...

    2019

  17. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.