{"id":"bce63a85-d5e3-45de-a224-94b2b9b7025a","arxiv_id":"2501.12698","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SFT-trained reward models over 12 dialogue-impression metrics, used as AI feedback in DPO, improve Japanese chit-chat systems' naturalness and impression scores over no tuning.","lead":"This paper trains reward models to score whole conversations on 12 qualities such as empathy and consistency, then uses those scores as AI feedback to fine-tune Japanese dialogue systems. The authors report that DPO training improves perceived naturalness and impression scores compared with untuned systems, though verification is limited.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward model is not validated on 4-turn dialogues and is shown to reward dull responses, so the AI feedback signal may optimize naturalness rather than the intended impression metrics; the independent human evaluation is too weak to rule this out.","rationale":"The reader identified the key weakness: the reward model is applied across a domain gap (32-turn to 4-turn) and has a documented failure mode for dull responses. My stress-test confirms this as the single most load-bearing concern because the entire training signal depends on the reward model's validity. The paper's own discussion admits the reward model rewards undesirable dull outputs, which directly threatens the construct validity of the AI feedback. The automatic evaluation is circular, and the human evaluation, while independent, is too small and lacks statistical testing to establish the broad claim. I therefore agree with the reader's CONDITIONAL verdict: the method is a plausible recipe, but the evidence does not yet support the claim that it improves overall dialogue impression rather than merely naturalness. My recommended verdict stays UNCHANGED because the reader already reached CONDITIONAL; the concern reinforces that condition rather than moving the verdict. If the proposed validation were to fail, a stronger adjustment to REJECT would be warranted, but without that test, the appropriate evaluation remains conditional acceptance pending evidence of reward-model validity on the target dialogues.","tokens_in":9419,"tokens_out":6183,"duration_ms":62274,"concrete_test":"Collect human impression scores from three independent raters on 50 held-out 4-turn JEmpathetic dialogues, with responses from the base, PPO, and DPO models, plus deliberately dull probes (e.g., 'Yes'). For each of the 12 metrics, compute the SFT reward model's Spearman correlation with the mean human score on these short dialogues. If the average correlation is not significantly positive or falls far below the ~0.8 value reported on 32-turn dialogues, the reward model is not a valid feedback signal for the optimization domain. Separately, if the reward model ranks dull probes above more engaged responses on metrics like Empathetic or Personality, the training objective is misspecified, and the reported gains are likely driven by naturalness rather than by the intended dialogue impressions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the reward model being a faithful proxy for the 12 dialogue-impression metrics on the optimization domain. Two facts undermine this. First, the reward model is trained on 32-turn JTransformer-Eval dialogues (Section II) but applied to 4-turn JEmpathetic dialogues without any validation; correlations reported in Table II are only for the long-dialogue test set, so the signal on short dialogues is unknown. Second, Section VI explicitly states that the reward model gives high scores to dull responses such as 'Yes' and 'I think it is good,' indicating that it is capturing an implicit naturalness prior rather than the specific impression metrics. If the training signal rewards dullness, then DPO/PPO may improve naturalness while failing to improve the targeted impressions. The automatic evaluation (Table III) is circular because the same reward model scores the training and the outcome. The only independent evidence is the human evaluation, but it is based on one rater per metric, 100 items per metric, with no significance testing; moreover, the naturalness results for youri are often flat or negative, and several metrics (e.g., Topic) do not improve. Thus, the broad claim that tuning with this reward model improves individual metrics and naturalness is not robustly supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using supervised fine-tuning to create regression-based reward models for 12 overall dialogue-impression metrics, then using those reward scores as AI feedback to train dialogue models with PPO and DPO. The reward model is trained on 32-turn JTransformer-Eval dialogues and applied to 4-turn JEmpathetic Dialogue contexts. The authors compare two 7B chat models (calm2 and youri) across 12 metrics, reporting automatic AIF scores and perplexity in Table III, human ranking in Table IV, and human naturalness ratings in Table V. They conclude that DPO tuning with the reward model improves both individual dialogue-impression metrics and response naturalness.","tokens_in":9725,"tokens_out":4596,"duration_ms":46135,"significance":"The problem addressed is important: moving from per-response evaluation to whole-dialogue impression is a key step for engaging dialogue systems. The paper's main strength is the SFT reward model, which reaches Spearman correlations of roughly 0.8 on the JTransformer-Eval test set (Table II), showing that supervised fine-tuning is much better than prompt-only evaluation for this task. The comparison of PPO and DPO across two base models and 12 metrics is also a useful empirical contribution. The DPO-over-baseline signal in human ranking for the calm model (Table IV) is genuinely suggestive. However, the evidence as presented is not yet sufficient for the paper's broad conclusion: the automatic AIF metric is computed by the same reward model used for training, the reward model is unvalidated on the short-dialogue optimization domain, and the human evaluation uses one rater per metric with no significance testing. These gaps are load-bearing because the central claim is that AI feedback improves the targeted impressions.","major_comments":[{"comment":"The AIF metric in Table III is computed by the same SFT reward model that labels accepted and rejected responses for DPO in Section III.B and provides rewards for PPO. Reporting that DPO improved these scores after optimizing against the same model is partly tautological and cannot independently verify that the targeted dialogue-impression metrics improved. Please add an evaluation using a reward model not involved in training, or human labels on held-out JEmpathetic dialogues, and report the correlation of the reward model with human judgments on the short-dialogue domain.","section":"§IV.B.1, Table III"},{"comment":"The reward model is trained and evaluated only on 32-turn JTransformer-Eval dialogues, but it is applied without adaptation or validation to 4-turn JEmpathetic Dialogue contexts in PPO and DPO. Table II reports correlations only on the long-dialogue test set, so the validity of the reward signal on short dialogues is unknown. The paper's own Section VI observation that the reward model gives high scores to dull responses such as 'Yes' and 'I think it's good' further suggests that the model may be capturing a general naturalness prior rather than the intended metric-specific impressions, making the AIF improvements in Table III ambiguous evidence for the broad claim.","section":"§II.A, §II.C, §IV.A"},{"comment":"The human evaluation is based on one rater per metric with no significance tests or inter-annotator agreement, so the reported differences cannot be shown to be reliable. In addition, the naturalness results for youri in Table V are flat or negative relative to the baseline for several metrics (e.g., Agency 1.96 vs. 2.12, Ease 1.68 vs. 1.97), which contradicts the blanket statement in Section V.B that 'the improved naturalness of both PPO and DPO compared to pre-training' was observed. Please report significance tests and give per-model, per-metric conclusions rather than pooling across base models.","section":"§IV.B.2, Tables IV and V"},{"comment":"The 'Win' metric in Table IV is not defined precisely enough to interpret. The caption says it is the percentage 'at or above the same order compared to before the study,' but it is not clear whether ties count in favor of the trained model, what the denominator is, or how the ranking scale (1 best, 3 worst) maps onto win rates. Without a precise definition, the claim that 'DPO has the best results for calm and youri' on Rank and Win cannot be fully assessed.","section":"§III.B, Table IV"}],"minor_comments":[{"comment":"The text cites 'Cheng et al.' for the work on model training from LLM feedback, but reference [10] is by Pang et al.; please correct the author attribution.","section":"Introduction, reference [10]"},{"comment":"The PPL value 913.52 for youri Ease under PPO appears to be a typo or an outlier; please verify this entry, as it is more than an order of magnitude larger than all other PPL values.","section":"Table III"},{"comment":"The human-evaluation instruction says 'Please rank the questions for each Table I value,' but the task is to rank responses; please clarify the wording and state explicitly whether ties are allowed in the naturalness rating as well as in the ranking task.","section":"Section IV.B.2"},{"comment":"The sentence describing PPL calculation says that 'the base model of the post-enhanced training model is used for PPL evaluation,' which is ambiguous. Please specify whether the PPL is computed with the model being evaluated, with a fixed reference model, or with the SFT base model, since this affects the comparability of PPL values across conditions.","section":"Equation (1), Section IV.B.1"},{"comment":"The dataset name 'JTransformer-Eval' and the model name 'japanese-dialog-transformer' should be spelled consistently, and the relationship between the 1,600 dialogue sessions and the 32-turn dialogues should be stated clearly.","section":"Section II.A"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a dialogue-systems or LLM-alignment venue, and the core idea is worth pursuing. The DPO result for calm in the human ranking evaluation is the most credible piece of evidence, but the circular automatic metric, the reward-model domain mismatch, and the single-rater human evaluation are substantial evidential gaps that require additional experiments rather than simple rewriting. I would not reject the manuscript, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a competent, incremental RLAIF paper with one finding worth taking away: for whole-dialogue impression scoring, an SFT-trained 7b reward model on JTransformer-Eval correlates far better with human ratings (0.77–0.89) than prompting GPT-3.5 (0.2–0.4). That result is robust and immediately useful for anyone building dialogue-evaluation models.\n\nThe DPO experiments are plausible but softer than the abstract claims. The automatic AIF metric in Table III is circular—the same reward model labels the DPO pairs and scores the outcome—so the ~0.5 point AIF gains are expected. The independent human evaluation breaks that circularity and shows DPO improves rank/win for calm on most metrics and for youri on several, but the naturalness claim mostly holds only for calm. For youri, Table V shows flat or worse naturalness on most metrics (e.g., Ease 1.68 vs 1.97, Humanness 1.98 vs 2.02, Topic 1.77 vs 2.13). The abstract's blanket statement about improved naturalness is an overreach.\n\nThe bigger soft spot is the reward model transfer. It is trained on 32-turn JTransformer-Eval dialogues and applied to 4-turn JEmpathetic dialogues with no validation; Table II's correlations are only for the long-dialogue test set. Given Section VI's own admission that the reward model rewards dull responses like 'Yes,' the optimization signal may partly be a naturalness prior rather than the intended impression metric. The human evaluation does ask raters to ignore naturalness, but it uses one rater per metric, 100 items, with no significance tests or agreement statistics. That is thin support for a 12-metric claim.\n\nWhat the paper does well: it compares prompting vs SFT reward models head-to-head, reports full tables, and is honest about the dull-response failure mode. The 7b reward model numbers alone justify a citation.\n\nBottom line: this deserves a serious referee, not a desk reject. It's a useful recipe, not a breakthrough. I'd ask for multiple raters with significance testing, reward-model validation on the target dialogue length, and a tempered conclusion. If those are done, it's an accept.\n\nSend it to review.","headline":"Solid applied RLAIF recipe with convincing SFT-reward-model results, but the naturalness claim is overgeneralized and the human evaluation is too thin to carry the full claim.","tokens_in":10199,"tokens_out":3556,"would_cite":true,"duration_ms":32186,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Supervised fine-tuning makes LLM reward models reliable enough to drive whole-dialogue impression, and DPO turns that feedback into better responses.","keywords":["dialogue systems","reinforcement learning from AI feedback","reward model","dialogue impression","direct preference optimization","PPO","supervised fine-tuning","Japanese dialogue"],"falsifier":"Collect fresh human impression scores for 4-turn dialogues, compare them with the reward model's predictions, and run a blind human test of the DPO-tuned model against the base model; if the rank correlation is near zero or the tuned model does not outrank the base model, the claimed feedback loop does not actually improve dialogue impression.","tokens_in":9185,"feed_emoji":"💬","tokens_out":8185,"duration_ms":73138,"temperature":0.7,"pith_summary":"The paper asks whether AI feedback can improve a dialogue system's overall impression across a whole conversation, rather than just the quality of a single reply. It argues that prompting an LLM to grade an entire dialogue is unreliable, and that a reward model supervised fine-tuned on human scores for 12 impression metrics is a far better signal. Using that reward model to tune a dialogue model, the paper reports that direct preference optimization (DPO) raises the rewarded impression scores and lowers perplexity, and human raters judge the responses both more natural and more reflective of the target impression. The practical stake is that whole-dialogue qualities such as consistency, empathy, and personality can be optimized automatically once a reliable reward model exists.","feed_headline":"Fine-tuned reward models let AI feedback improve dialogue impression","feed_subtitle":"Direct preference optimization beats PPO on impression scores and human-rated naturalness in Japanese chat.","key_machinery":"The load-bearing component is a reward model built by supervised fine-tuning: an LLM with a linear regression head trained, by mean squared error, to output eleven-point scores for 12 dialogue-impression metrics from a dialogue context plus a response. A single model handles all 12 metrics, and the 7-billion-parameter version is then used to score 4-turn dialogue contexts. In DPO, the model generates two responses to each context, the reward model labels one preferred and one rejected, and the dialogue model is trained to prefer the higher-scoring response; in PPO, the reward score is used as the reinforcement learning signal during generation. The argument runs through this reward model: the better it tracks human impression, the more the feedback loop can shape dialogue quality.","core_discovery":"The central claim is that supervised fine-tuning of an LLM on human-annotated whole-dialogue impressions creates a reward model that can stand in for human feedback during dialogue-model training. On a test set of 32-turn dialogues, the 7-billion-parameter SFT reward model reaches rank correlations with human scores of roughly 0.8 across the 12 metrics, far above the near-zero-to-0.4 range of prompted models. Applying this reward model to 4-turn JEmpathetic Dialogue contexts, DPO improves the rewarded metric by about 0.5 on an 11-point scale while also reducing perplexity, and human evaluation ranks DPO responses best on most metrics and rates them as more natural. The paper also reports that the reward model can give high scores to natural but dull replies such as \"Yes,\" and identifies that as a remaining issue for RLAIF training.","pith_inferences":["A natural next step the authors do not test is multi-objective DPO, mixing several impression rewards into one preference signal; because their reward model already scores 12 metrics at once, the data to try this exists.","The dull-response failure suggests a testable fix: add a diversity or fluency penalty, or train the reward model on labels that explicitly separate naturalness from impression, rather than letting naturalness bleed into impression scores.","If the reward model's transfer to short dialogues is real, the same recipe should work on longer or multi-speaker dialogues; if it fails there, the transfer assumption in this paper would be the reason."],"forward_implications":["A dialogue model can be aligned to a chosen impression dimension such as empathy, trust, or personality without collecting new human preferences for every update, once the reward model has been trained.","DPO is the stronger of the two training strategies tested: it improves both the rewarded impression score and fluency, whereas PPO improves impression only modestly at the cost of fluency.","Prompting-only LLM judges are not enough to drive whole-dialogue impression tuning; the gap between prompted and SFT reward models directly motivates supervised reward-model training.","Reward models trained on long 32-turn dialogues transfer, at least in practice, to scoring short 4-turn training contexts, which makes the RLAIF loop feasible on standard dialogue data."],"supporting_citations":[{"why":"Supplies the 32-turn JTransformer-Eval dialogues with 12 human impression scores used to train the reward model, and the JEmpathetic Dialogue corpus used to tune the dialogue model.","marker":"[21]"},{"why":"Provides the direct preference optimization objective that turns reward-model scores into preference pairs and yields the best results in the paper.","marker":"[23]"},{"why":"Provides the proximal policy optimization algorithm used as the comparison training method.","marker":"[22]"},{"why":"Establishes the RLAIF setting in which AI feedback replaces human feedback, the paradigm the paper applies to dialogue impression.","marker":"[9]"},{"why":"Supplies the original RLHF/PPO training recipe whose epoch setting is reused for the PPO runs.","marker":"[5]"},{"why":"Represents the prompting-based LLM-as-judge approach that the paper contrasts with supervised fine-tuning for whole-dialogue evaluation.","marker":"[20]"},{"why":"Provides the original empathetic dialogue data from which the emotion words and situations in JEmpathetic Dialogue were translated.","marker":"[24]"}],"fun_headline_variants":["Fine-tuned reward model improves dialogue impression via DPO","AI feedback from tuned reward model boosts dialogue naturalness","SFT reward model beats prompted LLM for dialogue impression scoring","DPO with reward model enhances dialogue impression and naturalness","Tuned reward model enables AI to improve whole-dialogue impression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the reward model trained on 32-turn dialogues scoring 4-turn dialogues the way a human would, even though the paper notes it can reward dull replies such as \"Yes.\"","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned reward model improves dialogue impression via DPO","AI feedback from tuned reward model boosts dialogue naturalness","SFT reward model beats prompted LLM for dialogue impression scoring","DPO with reward model enhances dialogue impression and naturalness","Tuned reward model enables AI to improve whole-dialogue impression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3032,"prompt_tokens":906,"completion_tokens":2126,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":2043}},"tokens_in":522,"tokens_out":2126,"duration_ms":15355,"temperature":1.0,"reasoning_tokens":2043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:52:42.549822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect fresh human impression scores for 4-turn dialogues, compare them with the reward model's predictions, and run a blind human test of the DPO-tuned model against the base model; if the rank correlation is near zero or the tuned model does not outrank the base model, the claimed feedback loop does not actually improve dialogue impression.","supporting_citations":[{"cited_title":"Empirical analysis of training strategies of transformer-based japanese chit-chat systems","cited_arxiv_id":null,"evidence_quote":"Supplies the 32-turn JTransformer-Eval dialogues with 12 human impression scores used to train the reward model, and the JEmpathetic Dialogue corpus used to tune the dialogue model."},{"cited_title":"Towards empathetic open-domain conversation models: A new bench- mark and dataset","cited_arxiv_id":null,"evidence_quote":"Provides the original empathetic dialogue data from which the emotion words and situations in JEmpathetic Dialogue were translated."}],"review_version":1}