{"id":"bb0c99cd-59a4-4aca-8866-80a3543f159a","arxiv_id":"2502.02802","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A state-aware client simulator for motivational interviewing counseling achieves higher profile and receptivity consistency than prior prompting-based methods.","lead":"This paper introduces a framework that keeps simulated counseling clients aligned with their personality, beliefs, and willingness to change. It could make counselor training more scalable and affordable by providing consistent practice partners.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The automatic consistency scores in Table 1 are produced entirely by GPT-4, which also generates the sessions; the entailment step is never validated against human judgments, so the claimed advantage over baselines may be an artifact of this evaluation loop.","rationale":"The central claim is that the proposed framework achieves higher consistency with client profiles and receptivity than previous LLM-based simulation methods. This rests on the automatic consistency scores in Table 1, which use GPT-4 to extract profiles from generated sessions and GPT-4 to judge whether the extracted components are entailed by the ground-truth profiles. The reader's weakest assumption identifies the same concern: the reliability of the GPT-4-based annotation and evaluation pipeline. I agree with that assessment. The paper validates GPT-4 annotations on the AnnoMI data (states, actions, receptivity, profiles) against manual labels, but it does not validate the entailment judgments on generated sessions against human judgment, nor does it validate the extraction of profiles from generated sessions. Because the same model family (GPT-4) is used in generation and evaluation, the consistency advantage may partly reflect an evaluator bias toward the framework's practice of selecting and restating profile text verbatim (Section 3.4). The expert evaluation in Section 4.4 is too small (6 clients) and uses co-author annotators, so it does not provide independent confirmation of the automatic ranking. Despite these issues, the paper does show robustness across multiple generation backbones (Table 14) and includes an expert evaluation with moderate inter-rater agreement, so the central claim is plausible but not fully established. The proposed concrete test would settle whether the automatic metric aligns with human judgment on the same sessions, thereby determining whether the claimed consistency advantage is real or an artifact. Since this concern is the same as the reader's, the existing CONDITIONAL verdict remains appropriate.","tokens_in":30542,"tokens_out":7128,"duration_ms":64758,"concrete_test":"Re-use the sessions rated by experts in Section 4.4 (6 clients × 4 session types) and compute, for each of the four profile components, the GPT-4 entailment outcome used in Table 1. Then measure the agreement between GPT-4 entailment and the expert consistency ratings (Table 3) using Cohen's kappa or a rank correlation. If the agreement is poor (e.g., kappa < 0.4) or the GPT-4 ranking of methods disagrees with the expert ranking, the automatic metric is not a valid basis for the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 measures the central claim with GPT-4 entailment between profiles extracted from generated sessions and GPT-4-annotated ground-truth profiles from AnnoMI. Section 3.6 validates GPT-4's extraction and annotation on AnnoMI against manual co-author labels, but the entailment step and the extraction from generated sessions are not validated against human judgment. The framework itself uses GPT-4 in its modules (state transition, information selection, response generation), so the same model family both produces and evaluates the sessions. The information selection module (Section 3.4) explicitly selects and restates profile text, which may make GPT-4 entailment trivially true for the proposed method while penalizing baselines that convey the same information in other words. The expert evaluation (Section 4.4) is limited to 6 clients and co-author annotators, so it does not independently validate the automatic ranking. If GPT-4's entailment is systematically lenient toward verbatim profile repetition, the consistency advantage in Table 1 is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for simulating clients in Motivational Interviewing (MI) counseling, with the goal of generating client utterances that are consistent with a given client profile (motivation, beliefs, preferred change plans, persona) and a specified receptivity level. The framework consists of four modules: state transition, action selection, information selection, and response generation. It derives state/action knowledge from the AnnoMI dataset using GPT-4 annotations. The evaluation compares the proposed method against four LLM-based baselines using (i) GPT-4 entailment to measure profile consistency, (ii) Spearman correlation for receptivity consistency, (iii) session-level statistics such as action KL divergence, and (iv) a small expert evaluation by co-authors. The paper reports consistent improvements over baselines across all metrics, with the proposed method approaching real-session statistics more closely than baselines.","tokens_in":30757,"tokens_out":5761,"duration_ms":54025,"significance":"If the consistency claim holds, the framework is a meaningful contribution to scalable, profile-aware client simulation for counselor training, an area where existing methods are known to produce overly compliant or inconsistent simulated clients. The paper's strengths include an explicit decomposition of consistency into four profile dimensions plus receptivity, the use of a real counseling dataset to inform state/action distributions, machine-checkable prompt design, and a two-pronged evaluation (automatic and expert) that includes a positive control (real sessions rated highest). The main correctness risk is that the primary automatic metric (GPT-4 entailment) is evaluated with the same model family that also implements the framework, and the entailment step is not validated against human judgments. The expert evaluation provides partial independent support, but it is small in scale and uses co-author annotators. Overall, the contribution is potentially valuable, but the evaluation pipeline needs strengthening before the central claim is fully established.","major_comments":[{"comment":"The central claim of higher profile consistency is measured by GPT-4 entailment between profiles extracted from generated sessions and GPT-4-annotated ground-truth profiles from AnnoMI (Section 3.6). The entailment step itself is never validated against human judgments, and profile extraction from generated sessions is also not separately validated. Because GPT-4 is used in the framework's own modules (state transition, information selection, response generation), the claimed advantage over baselines may be inflated by same-model evaluation. I request a validation study: sample a set of (generated profile component, ground-truth profile component) pairs, obtain human entailment labels, and report agreement (e.g., Cohen's kappa) plus the consistency scores computed from human labels. Alternatively, use an independent LLM from a different family as a second judge and show that the ranking in Table 1 is preserved.","section":"Section 4.2, Table 1"},{"comment":"The information selection module instructs the LLM to 'Restate this reason using the original text.' This means the proposed method can insert verbatim profile snippets into generated utterances, which may make GPT-4 entailment trivially positive while penalizing baselines that express the same information with different wording. The expert evaluation partially mitigates this concern, but Table 1 is the main quantitative evidence of superiority. Please quantify the degree of verbatim reuse (e.g., longest common substring, ROUGE overlap between generated utterances and profile components) for the proposed method and baselines, and show that the consistency advantage persists for paraphrased generations. If verbatim reuse is substantial, consider reporting an additional evaluation using a paraphrase-based consistency metric.","section":"Section 3.4, Table 30"},{"comment":"The expert evaluation used four co-authors as annotators and only six clients (four clients for receptivity), with a moderate inter-rater agreement of 0.61 for the consistency ratings. This limits the strength of the independent confirmation of Table 1. I recommend either recruiting independent annotators with MI expertise (with a clear statement of their non-affiliation), or substantially increasing the number of clients/sessions to provide a more robust external check. At minimum, the report should specify the exact Krippendorff's alpha or Fleiss' kappa variant used and discuss the implications of the moderate agreement for the reliability of the expert-based ranking.","section":"Section 4.4, Tables 3-4"},{"comment":"The action KL divergence is computed between simulated sessions and real AnnoMI sessions, but the action distribution used in the proposed method's action selection module (Section 3.3) is derived from the same AnnoMI subset. The low KL divergence for the proposed method is therefore partly a tautology and not strong evidence of generalizable realism. To support the claim of action realism, the comparison should be performed on a held-out portion of AnnoMI or against an independent reference corpus. At minimum, the paper should explicitly acknowledge this circularity and interpret the KL result as a faithfulness diagnostic rather than an independent evaluation.","section":"Section 4.3, Table 2"}],"minor_comments":[{"comment":"The phrase 'consistent with with its stipulated profiles' contains a duplicated 'with'; please correct.","section":"Abstract"},{"comment":"The sentence 'We finally select 86 clients and their sessions from the dataset4 then utilized GPT-45 to annotate utterances' lacks a comma or period; please rephrase for clarity.","section":"Section 3.6"},{"comment":"The significance markers (* and **) are not explained in the text; please state the statistical test used and report the corresponding p-values or confidence intervals.","section":"Table 1"},{"comment":"The rule 'for profile components not found in the original session (usually the change plan component), negative entailment outcome is always assigned' may bias results against methods that introduce reasonable plans not present in the profile; please report the consistency scores with and without this rule.","section":"Section 4.2"},{"comment":"The y-axis scales differ across the histogram panels, making visual comparison of session length distributions difficult; please use a common scale or clearly annotate the different counts.","section":"Figure 3"},{"comment":"Please specify the exact type of kappa reported (e.g., Fleiss' kappa for multiple raters) and clarify whether the experts independently annotated sessions or discussed before rating.","section":"Section 4.4"},{"comment":"The definition of 'Act KL' is missing; please specify the exact action sets and distributions over which the KL divergence is computed (e.g., marginal action distribution or state-conditioned).","section":"Table 2"},{"comment":"The text states 'We set the top-p and temperature parameters to 0.7 and 0.8 respectively for ChatGPT' but later mentions specific model names (gpt-3.5-turbo-0125). Please clarify which model corresponds to which hyperparameter setting.","section":"Appendix D"},{"comment":"There are several typos in the case study examples, including 'policy saw them' (should be 'police'), 'a policeofficer' (missing space), and a duplicated client utterance in Table 39. Please proofread the appendix.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major revision rather than reject because the expert evaluation, though limited, provides a positive control (real sessions rated highest) and directionally supports the automatic results. The main unresolved issue is the lack of validation of the GPT-4 entailment metric, which is the backbone of Table 1. The authors should be encouraged to provide a human-validated entailment subsample or an independent-model check. The paper's own limitation section is candid, but the evaluation section overstates the conclusiveness of the automatic metrics. The action KL circularity should also be acknowledged in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds a sensible, well-engineered framework for consistent client simulation in motivational interviewing, and the central insight is worth taking seriously: integrate state tracking, action selection tied to real data, information selection, and receptivity control into a single pipeline. The multi-backbone experiments and the session-length/action-distribution analyses are a step beyond what most persona-prompting baselines do. The authors are also candid about limitations—prompt sensitivity, single dataset, MI-only scope—which I appreciate.\n\nWhat is genuinely new is the combination, not any single component. State-aware patient simulators already exist, and AnnoMI has been used before. But grounding the action distribution and state transitions in real counseling data, then explicitly controlling receptivity, is a useful integration for counselor training. The case studies in the appendix show the framework doing what it claims: keeping the client's reasons for change aligned with the profile and preventing premature agreement to plans.\n\nThe soft spots are real, though not fatal. The main evaluation loop uses GPT-4 to extract the ground-truth profile from AnnoMI, then uses GPT-4 entailment again to check whether the same profile appears in generated sessions. The information-selection module explicitly restates profile text, which likely makes entailment easier for the proposed method than for baselines that paraphrase. The entailment step is never validated against human judgment, even though the profile-extraction step is. That gap weakens the headline numbers in Table 1. The expert evaluation is small (six clients) and uses co-authors as annotators, so it helps but is not a clean independent check. The action-KL result is also partly by construction, since the method samples from the same distribution it is evaluated against. Some of these concerns the authors acknowledge, but not the circularity of the entailment loop.\n\nI would not desk-reject this. The framework is coherent, the limitation section is honest, and the problem is practically important. A serious referee should push for an independent human-validated entailment set and ideally a small external expert study, and the authors should be encouraged to release code and data. As is, the paper is a solid conditional accept: the architecture is credible, but the central consistency claim is not yet fully established.","headline":"Plausible framework for consistent MI client simulation, but the GPT-4-based consistency evaluation is not independent enough to fully establish the claimed advantage.","tokens_in":31265,"tokens_out":1859,"would_cite":false,"duration_ms":20964,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a state-tracking client simulator for Motivational Interviewing counseling keeps simulated clients consistent with their assigned motivation, beliefs, preferred change plans, and receptivity, and that this framework…","keywords":["Motivational Interviewing","client simulation","LLM agents","profile consistency","receptivity","state tracking","action selection","counseling"],"falsifier":"Take the 86 AnnoMI sessions, have human experts (not GPT-4) write the ground-truth client profiles, regenerate the sessions with the proposed framework and the baselines, and run the entailment check with a different judge model or with experts. If the framework's consistency scores no longer exceed the Profile-based and Pro+Act-based baselines, or if the rank order changes, the claimed advantage is an artifact of the annotation-and-evaluation loop.","tokens_in":30359,"feed_emoji":"🧑‍⚕️","tokens_out":5290,"duration_ms":47211,"temperature":0.7,"pith_summary":"This paper takes on a gap in simulated mental-health counseling: existing LLM-based clients are told a persona and then asked to role-play, but they drift—agreeing to change for reasons not in their profile, accepting plans the real client would refuse, or sounding far more cooperative than the assigned receptivity level. The authors propose a client simulation framework that explicitly tracks the client's stage of change, selects dialogue actions from distributions learned from real counseling sessions, and only exposes profile information when the selected action calls for it. The central finding is that sessions generated this way score higher on profile consistency—persona, motivation, beliefs, and preferred change plans—and on receptivity consistency than four baseline methods, in both GPT-4-based entailment checks and expert ratings. If true, this gives counselor training a way to generate practice clients who behave like the specific, sometimes resistant, humans a counselor will actually meet.","feed_headline":"Simulated therapy clients stay consistent with their profiles","feed_subtitle":"A state-tracking client simulator keeps motivation, beliefs, plans and receptivity aligned, beating persona-only baselines.","key_machinery":"The central object is a four-module client simulator with an explicit mental-state tracker and a profile gate. State transition controls when a simulated client moves from Precontemplation to Contemplation to Preparation by checking whether the counselor's utterance actually touches the client's stated motivation or addresses the stated belief; information selection restricts what profile content can enter the utterance; action selection samples from a distribution that blends the immediate context with empirical (state, receptivity)-conditioned action frequencies. Together these modules prevent the two failure modes the paper identifies in baseline simulators: agreeing to change for unstated reasons and disclosing plans the profile never authorized.","core_discovery":"The paper's claim is that consistency in client simulation is not a prompt-engineering problem but a control problem: the client's next utterance should be sampled from a state, an action, and selected pieces of profile information rather than generated freely from a persona. The framework models four profile components—motivation, beliefs, preferred change plans, and receptivity—and uses three states from the transtheoretical model (Precontemplation, Contemplation, Preparation) plus Termination. A state-transition module decides when the client moves states based on whether the counselor has mentioned the client's motivation or resolved the client's beliefs; an action-selection module merges a context-aware action distribution with a (state, receptivity)-aware distribution derived from the AnnoMI dataset; an information-selection module chooses which profile detail may be revealed; and a response-generation module writes the utterance. The evaluation shows the method outperforms baselines on entailment-based consistency scores (e.g., motivation 73.37 vs 55.33 for the best baseline), on Spearman correlation of receptivity (0.58 vs 0.33), and on expert-rated consistency and realism, while producing session statistics (receptivity, motivation rate, action distribution) closer to real sessions.","pith_inferences":["We infer that the state-transition and information-selection controls could be adapted to other structured therapeutic protocols, such as cognitive-behavioral therapy, if sessions annotated with state and action labels were available.","We infer that the same architecture offers a testbed for studying counselor errors: because the client's profile is fixed and state transitions are gated on counselor behavior, a session that fails to progress is traceable to which motivation or belief the counselor missed.","The entailment-based evaluation could be made even stronger by using human-written profiles instead of GPT-4-extracted ones; we infer the reported margins would be more convincing if the consistency pipeline did not depend on the same model that generates the ground truth.","We infer that the relapse extension shown in the appendix points toward multi-session simulations where a client can backslide between sessions—a realistic training scenario the single-session evaluation does not cover."],"forward_implications":["Counselor trainees can practice against clients who remain in Precontemplation until the counselor raises the specific motivation in the client's profile, making session progress contingent on counselor skill.","The same framework can produce sessions with low, medium, or high receptivity on demand, and experts' ratings track the assigned level (correlation 0.86), so training can deliberately include resistant clients.","Automatic entailment-based consistency metrics make it feasible to evaluate large numbers of simulated sessions without expert labor.","Because action distributions are learned from data, the framework can be re-targeted to other counseling approaches or datasets when annotated sessions exist.","The simulated clients are less overly compliant than baselines, with average receptivity and motivation rates close to real sessions."],"supporting_citations":[{"why":"Supplies the AnnoMI real counseling sessions and client profiles from which state/action distributions and evaluation ground truth are derived.","marker":"(Wu et al., 2022)"},{"why":"Defines Motivational Interviewing and the client-centered change process the simulation is built to obey.","marker":"(Miller and Rollnick, 2012)"},{"why":"Supplies the transtheoretical stages used as the client states in the framework.","marker":"(Prochaska and Velicer, 1997)"},{"why":"Supports the transtheoretical model and the stage definitions the framework relies on.","marker":"(Hashemzadeh et al., 2019)"},{"why":"Provides the action-sampling idea that the action-selection module adapts to MI counseling.","marker":"(Dutt et al., 2021)"},{"why":"The state-aware patient simulator whose state tracking and action selection are compared and extended with real-data distributions and receptivity control.","marker":"(Liao et al., 2024)"},{"why":"A baseline client simulation method evaluated against; represents the profile-based persona-prompting approach.","marker":"(Yosef et al., 2024)"},{"why":"A baseline example-based method evaluated against; prompts the LLM with a real counseling session exemplar.","marker":"(Chiu et al., 2024)"},{"why":"The Pro+Act baseline evaluated against, which adds action descriptions to the profile prompt.","marker":"(Zhang et al., 2024)"},{"why":"PATIENT-Ψ, a related LLM patient simulator whose approach is contrasted with the proposed real-data-driven state and action control.","marker":"(Wang et al., 2024b)"}],"fun_headline_variants":["Client sims stay in character via state tracking","Motivational interviewing client sims get consistency controls","Consistency in therapy client sims is a control problem","State transitions keep simulated clients on profile","Mental state tracking makes client simulation consistent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliability of the GPT-4 annotation and evaluation pipeline is the load-bearing premise: the same model extracts the ground-truth client profiles from real sessions and later judges whether generated sessions entail those profiles, so any systematic bias in that model could produce the reported consistency scores.","fun_headline_variants_meta":{"raw":{"variants":["Client sims stay in character via state tracking","Motivational interviewing client sims get consistency controls","Consistency in therapy client sims is a control problem","State transitions keep simulated clients on profile","Mental state tracking makes client simulation consistent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1564,"prompt_tokens":952,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":542}},"tokens_in":568,"tokens_out":612,"duration_ms":6573,"temperature":1.0,"reasoning_tokens":542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:01:50.625494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 86 AnnoMI sessions, have human experts (not GPT-4) write the ground-truth client profiles, regenerate the sessions with the proposed framework and the baselines, and run the entailment check with a different judge model or with experts. If the framework's consistency scores no longer exceed the Profile-based and Pro+Act-based baselines, or if the rank order changes, the claimed advantage is an artifact of the annotation-and-evaluation loop.","supporting_citations":[],"review_version":1}