REVIEW 4 major objections 4 minor 18 references
CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration
T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read CAMI, an LLM counselor agent using client-state inference and topic-tree navigation, outperforms four baselines in motivational interviewing, evoking change talk in 53.1% of simulated clients versus 38.4% for the best baseline.
desk verdict A competent framework paper whose success-rate gains are largely an artifact of a client simulator that shares the system's topic tree; the novel topic-exploration idea deserves a reworked evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the STAR framework's topic tree and its navigation controller. State inference uses zero-shot LLM classification into transtheoretical stages; topic exploration assigns probabilities to superclass topics, then chooses among Step Into, Switch, and Step Out based on client feedback; strategy selection picks up to two strategies from the Motivational Interviewing Skill Code (MISC); response generation creates one candidate per strategy plus one for the pair and prompts the LLM to choose the most contextually coherent response. The topic tree supplies the structured vocabulary that lets the agent descend into a fine-grained subtopic, move to a sibling topic, or return to a broader topic when feedback is weak, and the same tree defines the simulated client's engagement levels, which is how topic proximity produces the measured success signal.
What would settle it
Run the same five counselor agents against (i) a simulator whose engagement and state transition do not depend on matching the pre-specified motivation topic, or (ii) a corpus of real human-client sessions coded for change talk. If CAMI's success-rate advantage over CAMI-TE disappears or reverses, the central claim fails. A direct observational check is whether human clients' engagement correlates with the paper's four topic-distance levels (same fine-grained topic, same coarse topic, same superclass, different superclass), since the simulator assumes it does.
Extended reading notes
Core claim
CAMI's central claim is that a counselor agent which (a) infers the client's transtheoretical stage of change, (b) systematically explores a three-level topic tree of motivation topics (5 superclass, 14 coarse-grained, 59 fine-grained topics), (c) selects at most two MI strategies, and (d) ranks candidate responses by contextual coherence will outperform agents that only do strategy selection. The paper reports that CAMI reaches an overall success rate of 53.1% with GPT-4o and 51.1% with Llama-3.1-70B, ahead of the best baselines (38.4% and 42.6% respectively), and is the only method with non-zero success rates for clients whose motivation topics fall under Law and Education. It also reports higher MI behavior and global scores, better exploration focus, 93.32% state-inference accuracy with GPT-4o, and expert ratings closer to high-quality human counselors. The ablation without topic exploration, CAMI-TE, drops GPT-4o success to baseline level, showing that topic navigation, not strategy prompting, carries the success-rate gain.
Load-bearing premise
The simulated client's transition to Contemplation is triggered by the counselor mentioning the client's pre-specified motivation, and its engagement rises when the counselor's topic is close to that pre-specified topic on the topic tree; if real clients do not respond to topic proximity and motivation cues this way, the reported success-rate advantage would not transfer to real counseling.
Editorial extensions
If this is right
- With GPT-4o, CAMI's success rate reaches 53.1% overall versus 38.4% for the best baseline, and it is the only system to evoke change talk for clients with Law, Education, or Economy motivation topics in non-trivial proportions.
- Removing topic exploration (CAMI-TE) drops GPT-4o success to 38.4%, the same as the baselines, so the topic tree is the main source of the success-rate gain.
- State inference is accurate (93.32% with GPT-4o, 88.56% with Llama-3.1-70B) and is a necessary module: CAMI-TE retains it and still loses most of the gain, while the full CAMI achieves higher MITI global scores.
- Expert evaluators rate CAMI above the CoS baseline on MITI global ratings, change-talk exploration, success in eliciting change talk, and realism, though below high-quality human counselors.
- All LLM counselors, including CAMI, post low reflection-to-question ratios and overuse questions, so MI competence remains incomplete even with the framework.
Reading between the lines
- If the simulator's engagement rules are what make topic navigation pay off, then the topic tree's value may be limited to domains with a comparable motivation hierarchy; a direct test is to run STAR on a CBT or addiction dataset with a fresh topic tree.
- The same generate-then-rank response selection used to curtail strategy overuse could generalize to any multi-strategy dialogue agent as a way to reduce LLM preference bias.
- The success-rate metric here measures the simulator's transition to Contemplation; a field test using human-coded change talk would show whether the advantage survives contact with real clients.
- Because the topic descriptions explicitly link behaviors to health, financial, and legal harms, a deployer should treat the topic tree as a steering mechanism that needs MI safeguards to avoid becoming a pressure or shaming tool.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents CAMI, an LLM-based counselor agent organized around the STAR framework: transtheoretical-model state inference, topic-tree exploration over a three-level tree derived from AnnoMI and Wikipedia, MI strategy selection/ranking, and response generation. The authors also design a simulated client with state transition, dynamic engagement, action selection, and response generation modules, and compare CAMI against Base, DIIR, CoS, and CAMI-TE on MITI behavior counts and global scores, success rate, exploration focus, state-inference accuracy, expert ratings, and client-experience questionnaires. The main reported result is that CAMI outperforms all baselines, e.g., 53.1% success rate vs. 38.4% for the best GPT-4o baseline, and that the topic-exploration and state-inference modules are each important.
Significance. Assuming the evaluation is valid, the paper makes a useful contribution: it is among the first to propose explicit topic-tree exploration for MI counseling, it includes detailed prompts and a modular architecture that is easy to reproduce, and it moves beyond strict ground-truth matching by using simulated client interactions and expert ratings. The ablation design (CAMI vs. CAMI-TE) is a strength. However, the central empirical claim is currently supported only within the authors' custom simulation: the client simulator's state transitions and engagement levels are keyed to the counselor mentioning the client's annotated motivation and to proximity in the paper's own topic tree. Because only CAMI can exploit that structure, the headline success-rate advantage is not yet evidence of better counseling with real clients. The paper's own limitations mention prompt sensitivity and over-questioning but do not address this validation gap.
major comments (4)
- [Appendix A.2; Table 3; Tables 31-33] The success-rate evaluation is confounded by the client simulator's design. The state transition module is instructed that 'The client is expected to enter the Contemplation state only when motivated by specific reason(s) introduced by the counselor... analyzing the counselor's utterances to identify mentions of the client's motivations,' and the dynamic engagement module (Table 33) instructs the simulated client to become more engaged as the counselor's topic approaches the ground-truth motivation topic in the topic tree constructed in §3.2. Because CAMI is the only method with access to this topic tree and a navigation planner, the 53.1% vs. 38.4% success-rate advantage in Table 3 is consistent with the simulator rewarding topic-tree retrieval rather than MI counseling skill. The Limitations section does not address this issue. I would need a validation with an independent client model (one built without the STAR topic tree and state definitions), a human-client study, or at least a per-session analysis showing that success is not driven by proximity to the ground-truth topic.
- [§4.5; Table 4] The state-inference accuracy (93.32% for GPT-4o, 88.56% for Llama-3.1 70B) and the exploration-focus metric are computed against ground-truth labels produced by the authors' own simulator. The simulator's state is defined by the same TTM descriptions used in the counselor's inference prompt (Table 6 vs. Table 31), and the exploration-focus score is measured in the authors' own topic-tree hierarchy. High accuracy may therefore reflect self-consistency between two GPT-4o-prompted modules rather than an ability to infer human client states or explore human-relevant topics. The paper should report evaluation on expert-annotated real counseling sessions or explicitly re-scope the claims to the simulator.
- [§4.2] The CoS baseline is not the original Chain-of-Strategy method: the text states 'Since the original CoS prompt is not available, we implement it as an ablated version of CAMI without state inference and topic exploration.' This makes CoS an ablation rather than a faithful state-of-the-art comparison. The paper's abstract and title claim superiority over state-of-the-art methods; the comparison with CoS cannot support that claim. Please either obtain and use the original CoS prompt or rephrase the claim as an ablation comparison for CoS.
- [§4.6; Table 5] The expert evaluation rates sessions generated with the same simulator, not interactions with real clients. While it provides some evidence that CAMI's language is rated more MI-consistent than CoS's, it cannot independently validate the success-rate or exploration claims, because the client's reactions, including whether change talk is elicited, are produced by the simulator. Please clarify this limitation and consider an expert evaluation on real or human-role-played sessions.
minor comments (4)
- [Throughout] There are typographical errors throughout the paper, including 'Y our' in several prompts and 'he client' in Table 16; these should be corrected in a thorough copyedit.
- [Tables 1-4, 36] The automated evaluation tables report point estimates without confidence intervals or significance tests; given the 38-client/190-session design, paired bootstrap or mixed-effect tests would let the reader judge the stability of the reported differences.
- [§4.1] The claim that 50 turns is 'similar what we observed in real world sessions' lacks a supporting citation or a summary distribution from AnnoMI.
- [Table 30] The text of Table 30 says the instruction includes the matching-outcome description 'from Table 35,' but the relevant topic-matching descriptions are in Table 33; the cross-reference should be corrected.
Circularity Check
The client simulator is partially built from CAMI's own STAR framework and topic tree, and it defines a successful session as the counselor mentioning the client's preprogrammed motivation, so the headline success-rate and exploration-focus gains are partly artifacts of the test harness.
-
self definitional
[Appendix A.2 (Client Agent Implementation); Section 4.4 (Success Rate)]
"we partially adapt the STAR framework to simulate a client... The client is expected to enter the Contemplation state only when motivated by specific reason(s) introduced by the counselor. The state transition module thus ensures consistency by analyzing the counselor's utterances to identify mentions of the client's motivations. If such a mention is found, the next state is Contemplation; otherwise, the state remains unchanged."
The headline success rate is defined as the proportion of clients with whom the counselor 'successfully evokes change talk' (Section 4.4). In the simulator, that success is triggered by a hard-coded rule: the client enters Contemplation only when the counselor's utterance mentions the client's predefined motivations. CAMI is the only method equipped with a topic tree and a navigation planner designed to find those motivations; Base, DIIR, CoS, and CAMI-TE have no access to the tree. The 53.1% versus 38.4% margin therefore largely measures how often CAMI's topic search surfaces the motivation strings the simulator was programmed to reward, rather than an independent counseling outcome.
-
self definitional
[Appendix A.2 (Dynamic Engagement Module); Section 4.4 (Exploration Focus)]
"In the client's profile, we designate the fine grained topic ralated to motivation as the ground truth topic. The distance between the current topic and the ground truth topic is used to instruct the client in providing various types of feedback."
The client's engagement level, which drives CAMI's Step Into/Switch/Step Out topic-navigation decisions, is generated by measuring distance in the paper's own topic tree between the counselor's current topic and the hidden motivation topic. Exploration focus (FOC) is then scored by asking GPT-4o how close the counselor's response is to the client's fine-grained motivation topic in that same tree. Since CAMI is the only system that receives the topic tree and navigation operations, its superior FOC score and the engagement-driven success rate are, by construction, rewards for operating in the system's own representational space. The comparison would be fair only if the simulator's engagement rule were an externally validated model of client behavior, which the paper does not provide.
full rationale
The paper's MI-competency metrics (MITI/MISC behavior counts, global scores) and the response-ranking comparisons are not themselves derived from the circular target and provide some independent content. However, the headline claims of counseling success and exploration focus rest on a simulator that partially adapts the STAR framework and defines positive client reactions in terms of the same topic tree CAMI navigates. Because the baselines have no access to this tree, the reported superiority on success rate and exploration focus is partly an artifact of the evaluation harness. The expert evaluation is conducted by the paper's own co-authors on the same simulator, so it cannot fully break the circularity. There is no load-bearing self-citation or uniqueness-theorem argument here; the circularity is in the evaluation definition rather than in a citation chain. Overall score 6: one central evaluation outcome reduces by construction, while substantial non-circular components remain.
Assumptions & free parameters
free parameters (4)
- topic_probability_threshold =
40%
- initial_exploration_turns =
6
- max_strategies_per_response =
2
- session_max_turns =
50
assumptions (5)
- domain assumption The transtheoretical model's five stages and the advice that counselors should match strategies to the client's current stage are valid for MI counseling.
- ad hoc to paper A client's motivation can be represented by a topic in a three-level Wikipedia-anchored topic tree, and navigating this tree evokes change talk.
- ad hoc to paper The client simulator's behavior (state transitions, dynamic engagement, action distribution) approximates real client responses well enough for evaluating counselor agents.
- domain assumption The BERT classifier plus GPT-4o annotation yields MITI behavior codes accurate enough for comparing systems, at 72.8% accuracy.
- domain assumption AnnoMI high-quality sessions selected are representative of successful counseling and their clients are a suitable test population.
Cite this review
Pith. "Pith review of CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration." pith.science (2026). https://pith.science/paper/IUA3PA3L
@misc{pith2026250202807,
author = {Pith},
title = {Pith review of: CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration},
year = {2026},
howpublished = {\url{https://pith.science/paper/IUA3PA3L}},
note = {Machine review of arXiv:2502.02807}
}
read the original abstract
Conversational counselor agents have become essential tools for addressing the rising demand for scalable and accessible mental health support. This paper introduces CAMI, a novel automated counselor agent grounded in Motivational Interviewing (MI) -- a client-centered counseling approach designed to address ambivalence and facilitate behavior change. CAMI employs a novel STAR framework, consisting of client's state inference, motivation topic exploration, and response generation modules, leveraging large language models (LLMs). These components work together to evoke change talk, aligning with MI principles and improving counseling outcomes for clients from diverse backgrounds. We evaluate CAMI's performance through both automated and manual evaluations, utilizing simulated clients to assess MI skill competency, client's state inference accuracy, topic exploration proficiency, and overall counseling success. Results show that CAMI not only outperforms several state-of-the-art methods but also shows more realistic counselor-like behavior. Additionally, our ablation study underscores the critical roles of state inference and topic exploration in achieving this performance.
Figures
Reference graph
Works this paper leans on
-
[2]
Motivational interview- ing transcripts annotated with global scores. In Pro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11642– 11657. Steven A Cole, Deepa Sannidhi, Yuri Tertilus Jadotte, and Alan Rozanski
work page 2024
-
[3]
Consider starting with small, manageable changes like taking a short walk daily
Y ou are acting as a counselor agent, interacting with the client to help them achieve the goal of [goal] related to their behavior, [behavior]. Each client has their own unique motivations, and the counselor help the client discover their inherent motivation for change. The counselor need to explore various topics and uncover what concerns the client the...
work page 2012
-
[4]
Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation. arXiv preprint arXiv:2402.13211. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova
-
[5]
arXiv preprint arXiv:2309.15025
Large language model align- ment: A survey. arXiv preprint arXiv:2309.15025. Ian Steenstra, Farnaz Nouraei, Mehdi Arjmand, and Timothy W Bickmore
-
[6]
Virtual Agents for Alcohol Use Counseling: Exploring LLM-Powered Motivational Interviewing
Virtual agents for alcohol use counseling: exploring llm- powered motivational interviewing. arXiv preprint arXiv:2407.08095. Xin Sun, Xiao Tang, Abdallah El Ali, Zhuying Li, Xi- aoyu Shen, Pengjie Ren, Jan de Wit, Jiahuan Pei, and Jos A Bosch
-
[7]
arXiv preprint arXiv:2408.06527
Chain-of-strategy planning with llms: Aligning the generation of psychotherapy dialogue with strategy in motivational interviewing. arXiv preprint arXiv:2408.06527. Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al
-
[8]
arXiv preprint arXiv:2401.05654
Towards conversational diagnostic ai. arXiv preprint arXiv:2401.05654. Jiashuo Wang, Yang Xiao, Yanran Li, Changhe Song, Chunpu Xu, Chenhao Tan, and Wenjie Li
-
[9]
arXiv preprint arXiv:2406.12266
Towards a client-centered assessment of llm therapists by client simulation. arXiv preprint arXiv:2406.12266. Anuradha Welivita and Pearl Pu
Show all 18 references
-
[12]
arXiv preprint arXiv:2403.15737
Few-shot dialogue strategy learning for motivational inter- viewing via inductive reasoning. arXiv preprint arXiv:2403.15737. Stav Yosef, Moreah Zisquit, Ben Cohen, Anat Klomek Brunstein, Kfir Bar, and Doron Friedman
-
[13]
In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 1–11
As- sessing motivational interviewing sessions with ai- generated patient simulations. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 1–11. A Detailed Implementation In this section, we provide the detailed impleme...
2024
-
[14]
Table 16 to Table 26 demonstrate the descriptions used in counselor agent
Table 15 shows the prompt used to expand the topics and Figure 2 shows all the collected topics. Table 16 to Table 26 demonstrate the descriptions used in counselor agent. A.2 Client Agent Implementation Unlike previous works that predominantly simu- late simple client persona...
2024
-
[15]
Therefore, the client’s current state in the above context is
in LLM prompts, we partially adapt the STAR frame- work to simulate a client with a few modules that model state transition, dynamic engagement, ac- tion selection, and response generation separately. An input client’s profile includes the client’s behav- ioral problem, state ...
2005
-
[16]
However, the accuracy of the fine-tuned BERT classifier is still not optimal (only 56.2% accuracy)
for a maximum of 10 epochs, with early stopping option and a learning rate of 5e-5. However, the accuracy of the fine-tuned BERT classifier is still not optimal (only 56.2% accuracy). Nevertheless, we found that the recall@5 across all classes was high (89.34%). Therefore, we ...
2024
-
[2000]
Health": 0.3,
and the Working Alliance Inventory - Short Revised (WAI-SR;(Hatcher and Gillaspy, 2006)). We employ GPT-4o with the prompt fromWang et al. (2024) to provide scores for each generated session. As shown in Table 36, CAMI achieves higher client assessment compared to other baseli...
2024
-
[2020]
In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38–45
Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38–45. Zixiu Wu, Simone Balloccu, Vivek Kumar, Rim Helaoui, Diego Reforgiato Recupero, and Dani...
2020
-
[2022]
In ICASSP 2022-2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 6177–6181
Anno-mi: A dataset of expert-annotated counselling dialogues. In ICASSP 2022-2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 6177–6181. IEEE. Zixiu Wu, Rim Helaoui, Diego Reforgiato Recupero, and Daniele Riboni. 2023b. Towards ef...
2022
-
[2023]
arXiv preprint arXiv:2305.08982
Helping the helper: Supporting peer counselors via ai-empowered practice and feed- back. arXiv preprint arXiv:2305.08982. Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Se- ungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, and Jinyoung Yeo
-
[2024]
arXiv preprint arXiv:2401.00820
A computational framework for behavioral assessment of llm therapists. arXiv preprint arXiv:2401.00820. Ben Cohen, Moreah Zisquit, Stav Yosef, Doron Fried- man, and Kfir Bar
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.