{"id":"8044198e-4bfc-49ba-803c-8cd027dcf2f0","arxiv_id":"2505.10218","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Training a 14B model with GRPO and a keyword-matching reward raises Script-Based Knowledge and Conversation Memory scores on RAIDEN, while lowering four other role-related metrics.","lead":"RAIDEN-R1 trains a 14B role-playing chatbot with reinforcement learning, using role-specific keyword checks as the reward. The authors report gains on two RAIDEN benchmark metrics, but the same metrics also define the reward, and other role-related scores fall.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Keyword-substring reward is a surface-form proxy; without a keyword-free evaluation, GRPO gains on SBK/CM may reflect keyword emission, and Table 1's SCK/RCB drops already undercut the robustness claim.","rationale":"The reader's weakest assumption is the same one I would flag as load-bearing: the accuracy reward reduces role-awareness to containment of an LLM-extracted keyword or a generated Python containment check, and GRPO will optimize exactly that proxy. The paper's own Table 1 shows the trained model improving on SBK and CM while dropping on SCK and RCB, which is the expected signature of proxy overoptimization rather than genuine role-awareness improvement. Because rewards and evaluation share the RAIDEN benchmark, the result may also reflect benchmark-specific keyword distribution rather than a general capability. The proposed keyword-free split would settle whether the gains are driven by keyword emission; without that evidence, the broad 'bridges the non-quantifiability gap' claim and the robustness claim are not currently supported. I keep the reader's CONDITIONAL verdict rather than moving to REJECT, because the narrow empirical result (keyword-based GRPO improves SBK and CM in this benchmark) could survive such a test, and the paper could be repaired with additional validation and honest reporting. The absence of error bars, code, and data further supports keeping the verdict conditional rather than accepting the claims as stated.","tokens_in":9108,"tokens_out":5772,"duration_ms":59189,"concrete_test":"Construct a keyword-free held-out split from the RAIDEN SBK/CM test partition: for each test item, paraphrase the reference answer so that the annotated target keyword and all expanded variants from Section 2.1.1.2 are absent while keeping the answer semantically equivalent; score 14B-Instruct and 14B-GRPO on this split with the same Claude-3.5 judge used in Section 3.2. If the 14B-GRPO SBK/CM advantage over the baseline shrinks substantially (e.g., by more than half) on the keyword-free split, the GRPO gain is explained by keyword emission rather than role-awareness, and the central claim fails. If the advantage persists, the reward tracks something more robust than surface keywords.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires VRAR to be a genuine verifiable role-awareness signal. In Algorithm 1 (Section 2.1.2), STV rewards a response iff the target keyword appears as a substring, and MTDP awards 1 iff a QwQ-generated Python checker returns True, which in the Figure 2 example is another containment test over expanded keyword variants. This is a surface-form criterion: a response can contain the keyword or one of its variants while violating role constraints, and a semantically correct paraphrase that avoids the keyword is scored 0. GRPO will exploit this margin, so training can increase reward by keyword insertion rather than by improving role-awareness. The paper's Table 1 is consistent with this failure mode: 14B-GRPO gains +1.45 on SBK and +8.40 on CM but drops -3.39 on SCK and -11.33 on RCB relative to 14B-Instruct, a selective decline on role-cognition metrics that is difficult to reconcile with the 'maintaining robustness' claim. Because the rewards are constructed from the same RAIDEN benchmark used for evaluation, a benchmark-specific keyword distribution could also be memorized. No independent validation data (e.g., a different role-playing benchmark or human judgment) is provided, so the abstract's claim to 'bridge the non-quantifiability gap' rests entirely on the unverified assumption that keyword presence tracks role-awareness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAIDEN-R1, a reinforcement-learning framework for role-playing conversational agents. The method defines a Verifiable Role-Awareness Reward (VRAR) built from keyword-based checks: Single-Term Validation (STV), which rewards responses containing a target keyword, and Multi-Term Dynamic Parsing (MTDP), which uses generated Python checker functions that test for expanded keyword variants; a format reward enforces <think> tags and a Chinese-character ratio. A cold-start SFT corpus is constructed with DeepSeek-R1 and Claude 3.5, and the system is trained with GRPO on Qwen2.5-14B-Instruct. On the RAIDEN benchmark test set, the 14B-GRPO model reaches 88.04% on Script-Based Knowledge and 88.65% on Conversation Memory, exceeding the 14B-Instruct baseline, while showing lower scores on SCK, RCB, TA, and TS. The paper claims this demonstrates effective and robust role-aware RL training and 'bridges the non-quantifiability gap' in RPCA training; case studies illustrate improved contextual memory and self-correction in misleading-query scenarios.","tokens_in":9401,"tokens_out":5359,"duration_ms":52209,"significance":"If the central claim were fully supported, the paper would be a useful contribution: it applies GRPO with a rule-based reward to a domain usually considered non-quantifiable, and it reports a transparent training recipe, including an honest negative result for cold-start SFT. The multi-LLM data-generation pipeline and the case analyses are informative. However, the significance is currently limited by the reward-validity question: because the reward is a surface-form keyword-containment test built from the same RAIDEN evaluation categories used as the primary outcome, the reported SBK/CM gains may reflect keyword-emission optimization rather than improved role-awareness. The robustness claim is also contradicted by the paper's own Table 1, and no independent validation, error bars, or significance tests are provided.","major_comments":[{"comment":"The claim that 14B-GRPO maintains 'comparable performance on other metrics' is not supported by the table it refers to: relative to 14B-Instruct, SCK drops from 84.75% to 81.36% and RCB from 62.77% to 51.44%, with TA and TS also lower. These are role-cognition metrics, not conversational extras, so the robustness conclusion in the Abstract and Section 3.3 is undercut. Additionally, the table reports a single run with no confidence intervals, error bars, or significance tests, so the primary-metric gains (+1.45 on SBK, +8.40 on CM) cannot be distinguished from noise. Please provide multiple seeds with dispersion measures, or revise the robustness claim to match the observed declines.","section":"§3.3, Table 1"},{"comment":"The accuracy reward is a surface-form keyword-containment test: STV assigns 1 iff the target keyword is a substring of the response, and the MTDP Python checker in the Figure 2 example is itself a containment check over expanded keyword variants. Under this reward, a response that contains the keyword in a negated or role-violating context is scored 1, while a semantically correct paraphrase that omits the keyword is scored 0. GRPO can therefore improve the reward by keyword insertion rather than by improving role-awareness. The paper offers no control evaluation (e.g., keyword-free human judgments, removal of reward keywords at test time, or measurements of keyword-stuffing behavior) to rule out this failure mode. Since the central claim depends on VRAR being a genuine verifiable role-awareness signal, this issue is load-bearing and must be addressed.","section":"§2.1.2, Algorithm 1 and Figure 2"},{"comment":"The reward is constructed from the same RAIDEN evaluation categories (SBK and CM keyword checks) used as the primary outcome, and the GRPO training data are drawn from the RAIDEN training split. Training on a benchmark and testing on its held-out split is standard, but here the reward literally encodes the evaluation criterion, so the SBK/CM improvements may reflect overfitting to the benchmark's keyword distribution rather than transferable role-awareness. No independent validation is provided: no second role-playing benchmark, no human evaluation, and no held-out keyword-free assessment. Section 4 also concedes that the rewards cover only memory and profile-consistency dimensions. Please add an external validation set or human study to support the general claim that the framework 'bridges the non-quantifiability gap'.","section":"§2.1.1, §3.2, §3.3"}],"minor_comments":[{"comment":"The phrase 'irrelevant metrics' is misleading: SCK and RCB are directly relevant to role-awareness, and TA/TS are conversational quality metrics. Please use 'remaining' or 'supplementary' metrics instead.","section":"§3.2"},{"comment":"The Figure 4 caption contains the typo 'Camparisions' for 'comparisons', and Figure 1 contains 'Referneces' for 'References'; both should be corrected.","section":"Figure 4 caption and Figure 1"},{"comment":"Please state the exact composition of the 1,000 RAIDEN samples (how many SBK versus CM) and the breakdown for the 1,000 general role-playing samples; the current wording is ambiguous about whether 1,000 is the total or per category.","section":"§3.1"},{"comment":"No code, data, or trained checkpoints are provided, and some GRPO hyperparameters (e.g., KL coefficient, clipping range, sampling temperature, random seed) are missing; specifying these would materially improve reproducibility.","section":"§3.1, reproducibility"},{"comment":"The footnote marker after 'VRAR' is typeset as an inline number rather than a superscript, and 'Raiden' / 'RAIDEN' are used inconsistently; also, the Section 4 admission that the rewards are limited to memory and profile-consistency dimensions should be reflected in the abstract's broad 'bridges the non-quantifiability gap' phrasing.","section":"Section 1 and 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate new combination—GRPO with a rule-based verifiable reward for role-playing—and the two data-generation workflows (STV and MTDP) plus the cold-start CoT pipeline are concrete contributions. The headline numbers on SBK and CM are real in the narrow sense: 14B-GRPO gets 88.04 and 88.65, above the instruct baseline. But the paper says \"maintaining robustness\" while Table 1 shows SCK dropping from 84.75 to 81.36 and RCB from 62.77 to 51.44. Those are not comparable; they are meaningful drops. Without error bars or multiple runs, I would treat the gains as tentative.\n\nThe bigger issue is the reward. STV gives 1 iff the response contains a pre-extracted keyword; MTDP uses a Python function that in the example is again substring containment over expanded keyword variants. That is surface form. GRPO can raise reward by keyword insertion. Since the reward and the evaluation are built from the same RAIDEN SBK/CM dimensions, the result is closer to \"optimizing a proxy for the metric\" than to \"bridging the non-quantifiability gap.\" A semantically correct paraphrase that avoids the keyword gets 0. There is no independent validation—no other benchmark, no human eval, no ablation removing keyword checking. No code or data are released, so reproducibility is limited. This is the load-bearing weakness.\n\nWhat the paper does well: the cold-start SFT experiment failed and they report it plainly. The case analysis is useful and shows the GRPO model doing first-person reasoning and catching a misleading number (374 vs 385), though it is anecdotal. The Next Steps section honestly limits the reward to memory and profile consistency.\n\nMy take: the narrow empirical claim—keyword-based GRPO improves SBK and CM on this benchmark—may survive. The broad claim about general role-awareness does not yet have support. This is a conditional accept at best; it needs a keyword-free evaluation or an independent role-playing benchmark, plus honest reporting of the supplementary metric drops.\n\nWho is this for? People building role-playing agents or RL pipelines for persona consistency. It is worth a serious referee, but I would not cite it as evidence for general role-awareness. If I were the editor, I would send it to review with a request to reframe the claims and add robustness checks.","headline":"A plausible GRPO-plus-keyword-reward recipe for role-playing training, but the paper's own table undercuts the robustness claim and the reward is too entangled with the metric to support broad role-awareness conclusions.","tokens_in":9917,"tokens_out":2094,"would_cite":false,"duration_ms":20418,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAIDEN-R1 claims GRPO with verifiable keyword rewards lifts role-play memory metrics to about 88 percent.","keywords":["role-playing conversational agents","GRPO","verifiable reward","keyword matching","chain-of-thought","RAIDEN benchmark","role-awareness","reinforcement learning"],"falsifier":"Take the trained 14B-GRPO model and evaluate it on a held-out set where each response contains the target keyword but contradicts a fact in the character profile or an earlier dialogue turn; if human raters judge these responses as role-unaware at roughly the same rate as the base model, the reward measured keyword emission rather than role-awareness.","tokens_in":8906,"feed_emoji":"🎭","tokens_out":4474,"duration_ms":40689,"temperature":0.7,"pith_summary":"The paper claims that role-awareness in role-playing conversational agents can be treated as a verifiable reward signal rather than an open-ended quality judgment. It trains a 14B instruction-tuned model with GRPO, scoring each response by whether it contains a mined role-specific keyword (single-term validation) or passes an auto-generated Python parsing function (multi-term dynamic parsing). On the RAIDEN benchmark, the resulting model reaches 88.04 percent on Script-Based Knowledge and 88.65 percent on Conversation Memory, beating the untrained base model and avoiding the degradation seen with plain SFT. The authors also build a cold-start chain-of-thought dataset and show that SFT on it alone does not improve role-play, while the GRPO stage does. If correct, this shows that a rule-based, quantifiable reward can bridge the gap that made reinforcement learning hard to apply to open-ended role-play.","feed_headline":"Verifiable keyword rewards lift role-play accuracy to 88%","feed_subtitle":"A 14B GRPO-trained model beats its instructor baseline on script knowledge and conversation memory without losing conversational quality.","key_machinery":"The carrying mechanism is VRAR, the Verifiable Role-Awareness Reward, which converts role-consistency into a binary score. Single-Term Validation extracts exactly one validated keyword per WH-question and gives a reward of 1 iff the response contains that keyword. Multi-Term Dynamic Parsing expands keywords into semantically equivalent variants, filters them for legitimacy, and generates a Python function that parses the response, giving a reward of 1 iff the function returns True. A format reward wraps the chain-of-thought in think tags with Chinese-character and repetition constraints, and a cold-start SFT stage supplies first-person chain-of-thought data intended to teach persona-consistent internal monologue before the GRPO stage.","core_discovery":"The central discovery is that a verifiable keyword-based reward, paired with GRPO, improves role-awareness metrics on the RAIDEN benchmark while leaving conversational quality largely intact. The reward design has two tiers: single-term validation for questions with one unambiguous keyword, and multi-term dynamic parsing for cases with multiple acceptable phrasings, where semantic expansion plus generated Python parsers decide correctness. Trained on 1,000 Script-Based Knowledge and Conversation Memory samples plus 1,000 filtered general role-play samples, the 14B-GRPO model scores 88.04 percent SBK and 88.65 percent CM, surpassing the 14B-Instruct baseline while maintaining secondary metrics such as Script-Contradictory Knowledge and Role-Cognition Boundary. The paper also finds that SFT on the GRPO training data or on a multi-LLM cold-start chain-of-thought corpus degrades most role-play metrics, while subsequent GRPO training recovers and improves them; case analysis shows first-person chain-of-thought reasoning emerging, including detection of contradictory profile information.","pith_inferences":["Editorial inference: the same reward template could be applied to emotional-consistency or action-consistency checks, converting any aspect of a character sheet into pass/fail verifiers.","A risk the paper does not fully settle: because correctness is defined by keyword presence, GRPO can optimize for keyword insertion. The degraded secondary metrics in some conditions hint that role-boundary behaviors may not be captured by the reward.","A testable extension: feed the GRPO model adversarial queries where the mined keyword is contextually irrelevant and measure whether it still emits the keyword; if it does, the reward is being gamed.","Another extension: anneal or reweight the accuracy reward over training so the model first learns keyword coverage, then is pushed to justify it in chain-of-thought; the current paper does not explore reward shaping."],"forward_implications":["Role-play improvement can be optimized with a fixed, deterministic reward, making RL viable for role-playing agents without relying on noisy LLM-as-judge feedback.","GRPO on a focused reward preserves base-model conversational skills better than SFT on the same training data, since RL nudges only the scored behavior.","Cold-start chain-of-thought SFT alone is not a sufficient bridge; the verifiable RL stage is what moves the primary metrics.","The method extends to other role dimensions with definitive answers: any metric reducible to keyword or parsing checks can be added as another reward term.","The observed short chain-of-thought outputs suggest role-aware reasoning emerges in compact, first-person form rather than long deliberation."],"supporting_citations":[{"why":"Defines the RAIDEN benchmark metrics (SBK, CM, SCK, RCB) and supplies the training and test dialogues used for reward mining and evaluation.","marker":"[16]"},{"why":"Supplies the multi-stage R1 training recipe (cold-start SFT followed by GRPO) that the paper replicates.","marker":"[6]"},{"why":"Introduces GRPO, the policy-optimization algorithm the paper trains with.","marker":"[9]"},{"why":"Shows that generic chain-of-thought can hurt role-playing ability, motivating the need for verifiable role-aware rewards.","marker":"[5]"},{"why":"Provides the 14B instruction-tuned base model used as the untrained baseline and the starting point for GRPO and SFT.","marker":"[18]"},{"why":"Used to expand keywords into semantically equivalent variants for the multi-term reward parsing functions.","marker":"[11]"},{"why":"Open-source implementation used to run the GRPO training experiments.","marker":"[4]"}],"fun_headline_variants":["Verifiable rewards boost role-play memory to 88%","GRPO with verifiable rewards lifts role consistency","RAIDEN-R1: keyword rewards improve role-play metrics","Verifiable keyword reward drives role-aware LLM gains","Role-play accuracy hits 88% via verifiable GRPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a response containing the mined keyword, or passing the generated Python check, is genuinely role-aware; if keyword occurrence can be satisfied without true role consistency, GRPO can exploit that shortcut.","fun_headline_variants_meta":{"raw":{"variants":["Verifiable rewards boost role-play memory to 88%","GRPO with verifiable rewards lifts role consistency","RAIDEN-R1: keyword rewards improve role-play metrics","Verifiable keyword reward drives role-aware LLM gains","Role-play accuracy hits 88% via verifiable GRPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1382,"prompt_tokens":935,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":551,"tokens_out":447,"duration_ms":4150,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:13:41.652395+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained 14B-GRPO model and evaluate it on a held-out set where each response contains the target keyword but contradicts a fact in the character profile or an earlier dialogue turn; if human raters judge these responses as role-unaware at roughly the same rate as the base model, the reward measured keyword emission rather than role-awareness.","supporting_citations":[{"cited_title":"RAIDEN benchmark: Evaluating role-playing conversational agents with measurement-driven custom dialogues","cited_arxiv_id":null,"evidence_quote":"Defines the RAIDEN benchmark metrics (SBK, CM, SCK, RCB) and supplies the training and test dialogues used for reward mining and evaluation."}],"review_version":1}