StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation

Dingming Liu; Jiahuan Pei; Koen Hindriks; Min Chen; Qingyu Meng; Xin Sun; Yifan Mo; Yue Su

REVIEW 3 major objections 6 minor 3 references

StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation

T0 review · 3 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multi-agent setup that grounds therapy dialogues in client stories and steers them with MI strategy codes can raise adherence to motivational interviewing practice.

desk verdict Solid multi-agent MI generation pipeline with real ablations and released data; the adherence lift is partly circular by design, so treat clinical-plausibility claims carefully. read the letter →

arxiv 2605.27393 v1 pith:7YUQXOEP submitted 2026-04-18 cs.CL cs.AI

Qingyu Meng , Min Chen , Dingming Liu , Yifan Mo , Yue Su , Xin Sun , Koen Hindriks , Jiahuan Pei This is my paper · ORCID

classification cs.CLcs.AI

keywords motivationalinterviewingmulti-agentdialogueLLMagentssituationalgroundingMIbehavioralcodingtherapysimulationstrategycontrolevaluationmetrics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StoryMI is a multi-LLM agent workflow for generating multi-turn motivational interviewing (MI) dialogues that are both situationally grounded and strategy-controllable. Client profiles start as DSM-5 questionnaire scores and rationales, which an agent expands into short first-person situational stories; a therapist agent and client agent then take turns producing utterances tagged with MI behavioral codes, while an interaction agent chooses the next therapist strategy and keeps shared state so that the conversation follows macro-level MI patterns (reflections over questions, complex reflections, open questions). The authors release a 6K-dialogue synthetic dataset spanning 12 MI codes and 13 symptom domains, plus a two-level evaluation suite of ordinary lexical metrics, MI-specific strategy metrics, LLM-as-judge rubrics, and expert ratings. Ablations and cross-model benchmarks show that removing the stories or the dynamic MI codes measurably hurts strategy adherence and reflection quality, while the full pipeline produces dialogues that better match clinical MI indicators. A sympathetic reader cares because prior LLM therapy simulations either lacked concrete client context or treated strategy labels as static prompts; StoryMI claims that the combination of narrative grounding and turn-by-turn strategy selection is what makes the dialogues clinically more plausible.

What carries the argument

The StoryMI multi-agent loop: questionnaire-to-profile-to-~200-word story, then client/therapist agents that emit (utterance, MI code) pairs while an interaction agent selects the next therapist strategy from recent client codes, updates shared dialogue state, and decides termination—thereby enforcing macro-level MI trajectories rather than turn-level prompting alone.

What would settle it

Run the same full vs. ablation conditions on real client–therapist MI transcripts (or with live trainees/clients) and check whether strategy-adherence gains and human expert preference for the full pipeline disappear or reverse.

Watch

Extended reading notes

Core claim

Situational story grounding plus dynamic macro-level MI code control, coordinated by a dedicated interaction agent, improves MI strategy adherence and clinical plausibility of generated therapist–client dialogues relative to ablations that remove stories and/or MI codes, across six open- and closed-source LLMs.

Load-bearing premise

That synthetic questionnaire profiles, LLM-written short stories, and multi-agent role-play judged mainly by automatic metrics, LLM judges, and two experts on 120 dialogues are a good enough stand-in for real clinical MI dynamics.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

StoryMI is useful systems work for people who need controllable synthetic MI dialogues. The actual new piece is the full pipeline: DSM-5 questionnaire profiles expanded into ~200-word situational stories, then a three-agent loop (client, therapist, interaction manager) that selects MI codes turn-by-turn, plus a 6K dialogue set over 1K story pairs and six operationalized macro metrics. They ship code and data, run six models, ablate story and MI coding on GPT, and check self-vs-cross LLM judges (GPT vs GLM) with a paired t-test. That is more engineering completeness than most therapy-dialogue papers.

What works: the ablations show MI coding is load-bearing for Strategy Adherence (80.9% full vs 64.2% without codes; Table 3), stories help Code Entropy, and lexical metrics alone do not track the strategy metrics—exactly the point they want to make. Human ratings on 120 dialogues (two experts, κ=0.51 overall, 0.79 on Adherence) and the open-question/empathy correlation are honest extras. The metric definitions in Appendix A are explicit enough to reimplement.

Soft spots, in proportion. The stress-test is right on the circularity: Strategy Adherence is exp(-KL to a fixed MISC ideal), and the interaction agent is explicitly told to pick strategies toward that practice (SelectStrategy, phase rules). Removing the selector mostly re-measures that the selector was on. Reflection Depth and Complex Reflection Ratio still improve under the full condition, so the control is real, but the abstract’s jump to “clinical plausibility” is stronger than the evidence. Human–LLM correlations are near zero on Naturalness/Empathy/Adherence; n=120 with two raters is thin; there is no head-to-head against ScriptAlign/RethinkMI/BiMISC on the same metrics. Free parameters (ideal mix, depth weights, δ thresholds, k, turn bounds) are acknowledged but not sensitivity-tested. Limitations section is appropriately cautious about synthetic-only validation.

Who it is for: NLP and computational MI people who want a reproducible generation recipe and strategy metrics, not clinicians looking for outcome evidence. I would bring it to reading group as a clean systems paper, cite the dataset and metric suite if I am building synthetic MI data, and send it to peer review. Expect referees to demand a non-circular external check and tempered language on clinical transfer. Worth engaging.

Referee Report

3 major / 6 minor

Summary. StoryMI proposes a multi-LLM agent workflow for generating motivational interviewing (MI) dialogues: DSM-5 questionnaire profiles are expanded into ~200-word situational stories; client and therapist agents produce MI-coded utterances; an interaction agent selects therapist strategies, updates shared state, and controls termination (Algorithm 1). The authors release ~6K dialogues over 1K questionnaire–story pairs (12 MI codes, 13 symptom domains), define lexical plus six MI strategy metrics (including Strategy Adherence via exp(−DKL) to a fixed MISC ideal mix), and evaluate six LLMs with ablations (on GPT-5-Nano), dual LLM judges (GPT and GLM) with bias checks, and two human experts on 120 dialogues. They claim that situational grounding and macro-level MI control improve MI adherence and clinical plausibility relative to ablations that remove stories and/or MI codes.

Significance. If the claims hold, StoryMI would be a useful, reproducible pipeline for steerable synthetic MI dialogue at scale—valuable for training data, counselor education, and controlled study of strategy–outcome links—where prior work either lacks narrative client grounding, multi-turn strategy control, or MI-aligned evaluation (Table 1). Strengths include a clear multi-agent design with explicit SelectStrategy/SynchronizeState/CheckTermination, formal metric definitions (Appendix A), multi-model benchmarks, ablations, dual-judge bias checks, human ratings, and promised code/data. The contribution is primarily methodological and resource-oriented rather than a clinical efficacy result; its lasting value depends on whether automatic “adherence” is cleanly separated from the controller that produces the codes, and on how far synthetic dialogues transfer beyond simulation.

major comments (3)

Central claim vs. circular Strategy Adherence (§3.3.2–3.3.3, Algorithm 1 lines 16–17; §4.2; Appendix A.2; Table 3). Strategy Adherence is exp(−DKL(P_obs ∥ P_ideal)) with a fixed MISC ideal (50% reflection / 25% question / 20% input / 5% other). In the Full condition the Interaction Agent explicitly selects ℓ*_T before therapist generation, so P_obs largely reflects the selector’s policy. The large drop when MI coding is removed (80.9% → 64.2% / 61.4%) therefore largely re-measures that the selector was active, not independent clinical quality. Reflection Depth and Complex Reflection Ratio also improve under Full but still ride the same code-conditioned path. The abstract and §6.1–6.2 over-read this metric as evidence of improved “MI adherence and clinical plausibility.” Please (i) reframe Strategy Adherence as a control-success / policy-fidelity metric, (ii) report primary quality claims
“Clinical plausibility” is under-supported relative to the claim strength (§4.3, §6.1.2, §6.3, Limitations). Human evaluation covers 120 dialogues (20 per model), two experts, overall κ=0.51 (moderate; Adherence κ=0.79). LLM–human correlations are significant for Depth/Progress but near zero for Coherence, Naturalness, Empathy, and Adherence—the dimensions most relevant to therapeutic fidelity. No client-facing or trainee study is reported. The abstract’s leap from synthetic multi-agent role-play to “clinical plausibility” should be toned to “expert-rated therapeutic quality in simulation,” with explicit limits on transfer. If space allows, enlarge the human sample or report per-dimension reliability and power; at minimum, align claim language with the evidence.
Ablation scope and mixed strategy profile (§6.2 Table 3; Table 2). Ablations (w/o story, w/o MI, both) are reported only for GPT-5-Nano. The headline adherence lift is therefore not shown to generalize across the six models that form the main benchmark. Additionally, GPT meets Complex Reflection and R/Q thresholds but fails Open Question Ratio badly (6.9% vs δ>70%), while LLaMA/Phi invert that pattern; human Empathy tracks open questions more than Strategy Adherence. Please run the same ablations on at least one open-source model, and discuss strategy trade-offs (reflection-heavy vs question-heavy paths) rather than treating a single aggregate adherence score as the primary success criterion.

minor comments (6)

Table 2: mark which models meet each δ threshold more consistently (✓/✗ already present but overall column averages can obscure per-model failures, e.g., Open Question Ratio).
§5.2 lists “seven LLMs” then six generators plus GLM as judge—clarify the inventory once to avoid counting confusion.
Figure 2: radar/bar scales are readable but absolute human vs LLM score ranges differ sharply; a short note on scale use (same rubric, different calibration) would help.
Appendix A.3–A.4: Reflection Depth weights (0.4 Sim / 0.6 Info) and classification thresholds (Sim/Info cutoffs) are free parameters; a one-sentence sensitivity check or fixed seed for embedding model would strengthen reproducibility.
Related work: VirturalMI / VirtualMI spelling inconsistency (Steenstra et al.); fix throughout.
Ethical statement is appropriate; consider one sentence on dual-use (synthetic client data for training vs. unsupervised deployment) already partly covered in §H.

Circularity Check

2 steps flagged · score 6.0 of 10

Strategy Adherence gains largely re-measure the interaction agent's own MI-code selection loop rather than independent clinical quality.

self definitional [§3.3.2–3.3.3 / Algorithm 1 lines 16–17 / §4.2 Strategy Adherence / Appendix A.2 / Table 3]
"ℓ ∗_T ← N_I .SelectStrategy(S.C[−k:], ℓ_C ,M) ▷ Macro-level control ... Strategy Adherence quantifies alignment with the golden MI strategy distribution, computed as the negative exponential of KL divergence from MISC best practices (50% reflections, 25% questions, 20% input, 5% other). ... Full ... Strategy Adherence (%)↑ 80.9 ... w/o MI 64.2 ... w/o Both 61.4"

The Interaction Agent is defined to select therapist MI codes according to MISC strategies and phase rules; Strategy Adherence is defined as closeness of the resulting code distribution to the identical MISC ideal. The Full-vs-ablation lift therefore measures that the selector was present, not an independent therapeutic property. The claim that 'macro-level control can improve MI adherence' reduces by construction to the control loop itself.
self definitional [§6.1.1 / Table 2 / §3.3.1 MI Coding Scheme]
"StoryMI consistently enables macro-level MI strategy control. Strategy Adherence is high (>80%) for most models ... These results directly validate StoryMI’s interaction agent as an effective mechanism for enforcing MI-adherent macro-strategies beyond turn-level prompting. ... Complex Reflection Ratio (δ>50%) ... all models exceeding the recommended threshold of 50%."

High Strategy Adherence and Complex Reflection Ratio are presented as validation of the interaction agent, yet both quantities are produced by the same agent that was instructed to emit the target codes (reflection/question/input) and then scored against those same targets. The 'validation' is definitional success of the controller rather than an external clinical test.

full rationale

The paper's central claim that situational grounding plus macro-level MI control improves MI adherence rests on Strategy Adherence (and related code-distribution metrics) rising under the Full condition and falling in ablations that disable the Interaction Agent's SelectStrategy. By construction, SelectStrategy chooses therapist codes according to MISC/MITI phase rules and client motivational orientation; Strategy Adherence is then defined as exp(-DKL(P_obs || P_ideal)) with P_ideal fixed from the same MISC guidelines (50% reflections, 25% questions, 20% input, 5% other). Removing the selector therefore necessarily lowers the metric; the ablation drop (80.9% → 64.2%/61.4%) largely re-confirms that the control loop was active, not that the resulting dialogues are more clinically plausible. Reflection Depth and Complex Reflection Ratio improve under Full but still travel the same code-conditioned generation path. Human ratings (n=120, two experts, overall κ=0.51) and weak LLM–human correlations on Adherence/Naturalness/Empathy supply partial independent signal, so the circularity is partial rather than total; lexical metrics and story ablation effects remain non-circular. No load-bearing self-citation uniqueness theorems or fitted-parameter-as-prediction appear. Score 6 reflects one clear by-construction reduction on the strongest automatic claim while the broader multi-agent simulation and human protocol retain independent content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central claim rests on clinical coding manuals and questionnaire instruments taken as given, on LLM role-play fidelity, on hand-chosen control thresholds and metric weights, and on the invented multi-agent StoryMI machinery. No free parameters are fitted to real clinical outcomes; several decoding and metric constants are chosen by authors or manuals and then used to declare success.

free parameters (5)

Ideal MI code distribution for Strategy Adherence = 0.50/0.25/0.20/0.05
P_ideal = {reflection:0.50, question:0.25, input:0.20, other:0.05} is taken from MISC guidance and used as the target that adherence scores against; small changes would rescale the main strategy metric.
Reflection Depth weights (Sim vs Info) = 0.4 / 0.6, θ=0.8
Rd uses 0.4·Sim + 0.6·Info with token novelty threshold θ=0.8 and all-MiniLM-L6-v2 embeddings; these choices define ‘depth’ scores reported in tables.
Macro thresholds δ (complex reflection, open question, R/Q) = >0.5, >0.7, >2.0
Success checkmarks use δ>0.5 complex reflections, δ>0.7 open questions, δ>2.0 reflection/question from MISC manual; binary pass/fail depends on these cutoffs.
Dialogue context window k and turn bounds = k≈5; T_min=10 in implementation notes
Recent-code window k (stated as five most recent turns), T_min and T_max control strategy selection and termination; they shape trajectories without being learned from clinical data.
Decoding hyperparameters = temp 0.7, top-p 0.9
temperature=0.7, top-p=0.9 for open-source models; fixed LLaMA generation of all profiles/stories reused across systems.

assumptions (5)

domain assumption MISC/MITI behavioral codes and recommended macro ratios are valid operational standards for MI quality in generated text.
Invoked throughout §3.3.1, §4.2, and Appendix A as the gold structure for codes and thresholds.
domain assumption DSM-5 cross-cutting symptom questionnaire scores plus short LLM rationales adequately represent client heterogeneity for MI simulation.
§3.1 profiling module; Limitations later notes Western diagnostic norms and constrained persona space.
domain assumption LLM agents can alternately play client and therapist with dual (utterance, MI-code) outputs that preserve persona and strategy under shared state.
Core of Algorithm 1 and §3.3; required for any claim that generated dialogues are MI-consistent.
ad hoc to paper Expanding Likert profiles into ~200-word first-person situational stories supplies the narrative context MI needs.
§3.2 design choice bridging questionnaires to dialogue; ablation treats story presence as a causal factor.
standard math Standard probability and embedding operations (Shannon entropy, KL to ideal, MiniLM cosine, token novelty) correctly quantify therapeutic strategy quality.
Appendix A metric definitions; math is standard, mapping to clinical quality is the domain leap.

invented entities (3)

StoryMI multi-agent workflow (client, therapist, interaction manager with SelectStrategy/SynchronizeState/CheckTermination)
purpose: Coordinate steerable MI-coded multi-turn generation from questionnaire–story inputs.
Primary system contribution formalized in Algorithm 1 and Figure 1; no independent clinical existence outside this pipeline.
Six MI strategy metrics suite (Code Entropy, Strategy Adherence, Reflection Depth, Complex Reflection Ratio, Open Question Ratio, R/Q)
purpose: Score macro counseling strategy beyond lexical fluency.
§4.2 and Appendix A operationalize MISC ideas into automatic scores used as main evidence.
6K StoryMI dialogue dataset over 1K questionnaire–story pairs
purpose: Benchmark resource covering 12 MI codes and 13 symptom domains.
Constructed entirely by the pipeline; value is as a synthetic resource, not an external clinical corpus.

how reviews work

0 comments

Cite this review

Pith. "Pith review of StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation." pith.science (2026). https://pith.science/paper/7YUQXOEP

@misc{pith2026260527393,
  author       = {Pith},
  title        = {Pith review of: StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7YUQXOEP}},
  note         = {Machine review of arXiv:2605.27393}
}

read the original abstract

Large language models (LLMs) can generate fluent dialogue, but prior works lack situational grounding, dynamic strategy control, and evaluation aligned with clinical standards in motivational interviewing (MI). We introduce StoryMI, a multi-LLM agent framework for controllable MI dialogue generation, where questionnaire-based client profiles are expanded into situational stories that provide narrative context for the dialogue. Therapist and client agents generate MI-coded utterances guided by MI codes selected by the interaction agent, while an interaction agent dynamically coordinates exchanges to control MI strategies during a multi-turn conversation. We propose a two-level evaluation protocol: lexical metrics and MI-specific measures of macro-level counseling strategies, alongside LLM-as-judge and human expert assessments. We construct a dataset of 6K simulated MI dialogues grounded in 1K questionnaire-story pairs, covering 12 MI codes and 13 symptom domains, and benchmark six open- and closed-source LLMs. Our results show that situational grounding and macro-level control can improve MI adherence and clinical plausibility, demonstrating the effectiveness of a structured multi-agent workflow for psychotherapy dialogue generation. We provide code and data for reproducibility.

Figures

Figures reproduced from arXiv: 2605.27393 by the authors.

**Figure 2.** Therapeutic dialogue quality evaluation in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** presents correlation analysis between LLM-based and human evaluations. 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 Coherence Depth Progress Naturalness Empathy Adherence 0.04 0.01 0.01 0.33*** 0.34*** 0.30*** 0.36*** 0.34*** 0.28*** -0.09 -0.07 -0.06 -0.05 -0.02 -0.01 0.03 0.03 0.03 Pearson Spearman Kendall (a) Dimension-level Correlation 0.5 0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 Gemma-7B LLaMA 3.1-8B OpenChat-7B Phi-4-14B Qwen 2.5-7… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 linked inside Pith

[1]

Tao Tu, Mike Schaekermann, Anil Palepu, and 1 others

Association for Computational Linguistics. Tao Tu, Mike Schaekermann, Anil Palepu, and 1 others
[2]

Jiashuo Wang, Yang Xiao, Yanran Li, Changhe Song, Chunpu Xu, Chenhao Tan, and Wenjie Li

Towards conversational diagnostic artificial intelligence.Nature, 642(8067):442–450. Jiashuo Wang, Yang Xiao, Yanran Li, Changhe Song, Chunpu Xu, Chenhao Tan, and Wenjie Li. 2024a. To- wards a client-centered assessment of LLM therapists by client simulation.Preprint, arXiv:2406.12266. Junda Wang, Zonghai Yao, Zhichao Yang, Huixue Zhou, Rumeng Li, Xun Wan...

arXiv 2024
[3]

I feel so tired

Judging LLM-as-a-judge with MT-bench and Chatbot Arena. InProceedings of the 37th Interna- tional Conference on Neural Information Processing Systems, pages 46595–46623. Curran Associates Inc. A MI Strategy Metric Definition This appendix provides complete mathematical def- initions, computational details, and illustrative ex- amples for the proposed MI-s...

2016

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

Tao Tu, Mike Schaekermann, Anil Palepu, and 1 others

Association for Computational Linguistics. Tao Tu, Mike Schaekermann, Anil Palepu, and 1 others

[2] [2]

Jiashuo Wang, Yang Xiao, Yanran Li, Changhe Song, Chunpu Xu, Chenhao Tan, and Wenjie Li

Towards conversational diagnostic artificial intelligence.Nature, 642(8067):442–450. Jiashuo Wang, Yang Xiao, Yanran Li, Changhe Song, Chunpu Xu, Chenhao Tan, and Wenjie Li. 2024a. To- wards a client-centered assessment of LLM therapists by client simulation.Preprint, arXiv:2406.12266. Junda Wang, Zonghai Yao, Zhichao Yang, Huixue Zhou, Rumeng Li, Xun Wan...

arXiv 2024

[3] [3]

I feel so tired

Judging LLM-as-a-judge with MT-bench and Chatbot Arena. InProceedings of the 37th Interna- tional Conference on Neural Information Processing Systems, pages 46595–46623. Curran Associates Inc. A MI Strategy Metric Definition This appendix provides complete mathematical def- initions, computational details, and illustrative ex- amples for the proposed MI-s...

2016