ChatMuse: Supporting In-Person Small-Group Conversation Experience with a Proactive Assistive AI Agent in Mixed Reality

Chen Chen; Christine Lisetti; Diana Nelly Rivera Rodriguez; Janet G. Johnson; Joaquin Frangi; Lingyao Li; Rawan Alghofaili; Renkai Ma; Shaoze Zhou

ChatMuse: Supporting In-Person Small-Group Conversation Experience with a Proactive Assistive AI Agent in Mixed Reality

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A mixed-reality AI agent that watches a live group conversation and privately coaches one participant on what to say and how to behave can improve that person's engagement and contribution, and can learn from its own suggestions.

desk verdict ChatMuse is a competent and candid design exploration of MR support for small-group conversation, but its self-improvement claim rests on an unvalidated gaze proxy; RQ1 mostly holds, RQ2 does not. read the letter →

arxiv 2607.18556 v3 pith:5XV2IZA5 submitted 2026-07-20 cs.HC

Shaoze Zhou , Joaquin Frangi , Diana Nelly Rivera Rodriguez , Rawan Alghofaili , Janet G. Johnson , Lingyao Li , Renkai Ma , Christine Lisetti

show 1 more author

Chen Chen

This is my paper

classification cs.HC

keywords MixedrealityproactiveAIagentsmall-groupconversationconversationalsupportnonverbalcuesgaze-basedfeedbackLLMparticipationinequality

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a proactive AI agent in mixed reality can help a participant hold their own in an in-person small-group conversation. ChatMuse listens to what everyone says and watches where they look, then privately suggests example phrases, non-verbal behaviors, and background facts at moments the agent chooses. The system also scores each suggestion by how long the user's gaze rests on it, and uses that score to reshape future support. In six group sessions with 18 participants, the authors report seven benefits and measurable decreases in speaking-time inequality with the feedback loop enabled. The claim matters because it moves conversational AI support from dyadic settings into the harder "many minds" territory of live group interaction.

What carries the argument

The central mechanism is a two-pipeline agentic loop. The feedforward inference pipeline maintains a "group conversation context" — speaker identity, transcribed speech, inferred speech emotion, and a directed attention graph of who looks at whom — serialized into a prompt for a pre-trained LLM, which outputs a proactive-decision flag and support text (max 50 words). The feedback enhancement pipeline then computes a usefulness score u = t / T, where t is the MR user's gaze dwell time on the overlay and T is the expected reading time of the text (60*N/238 seconds), and appends that score to the episode so the LLM can later reference which kinds of support were useful.

What would settle it

Run the system with deliberately unhelpful suggestions (e.g., off-topic or random tips) and measure gaze dwell time; if dwell time remains comparable to helpful suggestions, the usefulness score is not measuring usefulness.

Watch

Extended reading notes

Core claim

ChatMuse is a mixed-reality system for in-person small-group conversation support. It continuously builds a "group conversation context" containing each participant's speech, tone, speaker turns, and a directed graph of where people are looking. A pre-trained LLM reads this context and decides whether and what to suggest — example phrases, non-verbal behavior cues, background facts — rendered privately as an MR overlay. After a suggestion is shown, the system estimates how useful it was by dividing the user's gaze dwell time on the overlay by the expected reading time of the text, and feeds that score back into the context to improve later suggestions. In a within-subject study with six grou

Load-bearing premise

ChatMuse's self-improvement relies on how long the user looks at the suggested text being a true measure of how useful that suggestion was; if people stare out of curiosity or confusion instead, the learning loop is chasing the wrong signal.

Editorial extensions

If this is right

With the feedback loop active, participation inequality fell in all six groups compared to the no-feedback condition.
Support adoption — how much the user's later speech resembles the suggested example — rose significantly in four of six groups (Tau-U, p<.05).
Most conversation partners could not tell the supported participant was receiving help, although some described the MR user as leading or less genuine.
Users reported seven benefits: less friction, summaries, background info, topic-starting, context emphasis, topic extension, and inspiration.
No reliable workload or usability differences were found across conditions, so the support came at no measured cognitive cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The gaze-based usefulness score is a proxy that could be misled by curiosity or confusion; future work could validate it against self-reported usefulness or behavioral uptake of the suggested content.
The proactive rate of roughly 88–90% suggests the agent nearly always chooses to intervene; testing an explicit cost for unwanted interruptions could clarify how to preserve conversation ownership.
The same pipeline could generalize beyond 3–4 person, 15-minute sessions if attention graphs scale, but longer conversations would likely require memory mechanisms to keep context coherent.
Placing overlays above the person being addressed reduces occlusion but may lag fast gaze shifts; an optimization-based placement algorithm could better handle multi-speaker moments.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this for the system design and the honest qualitative findings, not for the feedback-pipeline results. The paper extends proactive LLM-based MR conversation support from dyadic settings (SocialMind) to small groups, which is a genuinely new interaction space. The formative study is a real strength: the three design considerations are sensible and clearly shaped the system. The evaluation is appropriately mixed-methods—seven qualitative benefits, rationales analysis, plus objective metrics—and the limitations section is unusually candid about sample size, headset obtrusiveness, headpose as a gaze proxy, and the short conversation duration.

The soft spots are real but not fatal. First, the abstract says “demonstrate effectiveness,” but SUS and NASA-TLX showed no significant differences; the objective support is partial (participation inequality improved, adoption scores improved in some groups). That overstatement is easy to fix by softening the language. Second, and more important, the feedback enhancement pipeline in Section 4.4 uses u = t/T, where t is gaze duration on the overlay and T is expected reading time. The stress-test note is right: when the overlay is anchored above the partner’s head (the common case), the user’s natural gaze for eye contact can intersect the overlay, inflating t. Confusion or surprise can also produce long dwell times. Since u is the only learning signal that updates the episode context, the RQ2 results—significant Tau-U adoption-score increases and the qualitative rationales—cannot be attributed to genuine self-improvement. The paper offers no validation of u against any independent usefulness rating. This is an internal-validity gap, not a fabrication, but it means RQ2 is not established. Finally, no data, code, or prompt templates are shipped, so the empirical core is not independently checkable.

What holds up: the central feasibility claim for RQ1 is plausible. The qualitative findings are triangulated, the participants’ voices are present, and the authors are clear that this is an exploratory design study. The self-citation of the authors’ prior needs study is expected and appropriate.

Who this is for: HCI researchers working on MR/AR social support, agent-mediated communication, and assistive systems for conversation. The paper deserves a serious referee; it is coherent and honest, but the authors should either validate the usefulness metric or substantially narrow the claims about the feedback pipeline. A major revision focused on RQ2 would make this a solid contribution.

Referee Report

3 major / 5 minor

Summary. ChatMuse is an MR system that supports one participant in a three-to-four-person in-person conversation by analyzing all participants' verbal and non-verbal cues and proactively rendering private suggestions on speech, non-verbal behavior, and background knowledge. The system is designed from a formative provotype/focus-group study that yields three design considerations, and is evaluated in a counterbalanced within-subject study (six groups, N=18; one MR user per group) with post-session questionnaires, focus groups, and logged agent rationales. The paper reports seven key benefits for engagement/contribution, reduced participation inequality, and significant Tau-U support-adoption effects for four of six groups when the feedback enhancement pipeline is enabled. The contribution is framed as a design exploration of proactive MR conversational support, with the novelty concentrated in the feedback pipeline that uses gaze-derived usefulness to update the LLM's context.

Significance. If the results hold, ChatMuse is a credible demonstration that a proactive LLM-based MR overlay can help a less-engaged participant enter and contribute to a small-group conversation without disrupting partners, and the design considerations from the formative study are practically useful. The paper is commendably transparent: counterbalanced ordering, null SUS/TLX results reported, rationales logged and analyzed (209 vs 271), calibration benchmarking in Appendix C, and a frank limitations section. The qualitative material is rich and triangulated. However, the system's most distinctive claim — closed-loop improvement through the feedback pipeline — is not yet supported, because the sole learning signal (usefulness from gaze duration) is unvalidated and confounded by overlay placement, and the outcome measure for adoption is itself an unvalidated proxy. The effectiveness claim is further limited by a treatment sample of six MR users and null objective usability results. Overall, this is a promising design exploration whose central qualitative findings are plausible, but whose headline mechanism needs reframing or further validation.

major comments (3)

[§4.4–4.5, Fig. 5, Table 1] The usefulness score u = t/T (§4.4) is the only learning signal in the feedback pipeline, but it is unvalidated and structurally confounded by the rendering logic in §4.5: in the common placement the overlay is anchored above the head of the partner the MR user is looking at, so gaze dwell on the overlay can reflect eye-contact/attention to the partner rather than reading of the suggestion; dwell time is also inflated by confusion or readability problems. Because u updates the episode context (Fig. 5) and conditions all subsequent LLM generations, the RQ2 findings (Table 1 Tau-U for G3–G6; rationales such as 'Prior support with facts was useful') do not establish genuine self-improvement; the agent's rationales merely restate the unvalidated signal. No calibration against independent usefulness ratings is presented. Fix: validate u on a sample of episodes with human ratings, or re-analyz
[§5.1, §5.3, Table 1, Appendix F] The central effectiveness claim rests on a much smaller treatment sample than the abstract suggests: one MR user per group (orange highlight in Table 4), i.e., six supported participants; the seven benefit themes (RQ1) are drawn largely from these six participants' focus-group remarks, while the other twelve contributed peripheral observations. The quantitative backbone is also weak: SUS and NASA-TLX showed no significant differences (Appendix F), the participation-inequality reduction in Table 1 is purely descriptive and not monotonic (Baseline already has lower inequality than C_NoFeedback for G1, G4, G5), and Q4/Q5 had a majority reporting no perceived difference. The N=18 framing in the abstract should be corrected and the qualitative N=6 basis acknowledged; the descriptive inequality claims should be supported by an appropriate test or discarded.
[§5.2] The support adoption score (max cosine similarity between suggested example speech and the MR user's subsequent utterances, all-MiniLM-L6-v2) is itself an unvalidated proxy for 'adoption.' Since the feedback pipeline's generation is conditioned on the same conversational context — and on the possibly invalid u — an increase in Tau-U (Table 1) could reflect the suggestions increasingly echoing the user's own phrasing or the metric's properties rather than improved support quality. The paper should validate the measure (e.g., against manual coding of whether the user used the suggestion) or discuss this confound explicitly. Also specify how episodes with no user utterance before the next support rendering are scored.

minor comments (5)

[§6.2] The limitations list omits the main validity gap: the gaze-based usefulness proxy of §4.4 is never calibrated against any independent usefulness rating. Given its load-bearing role in RQ2, it should be acknowledged here.
[§5.1] State explicitly that the supported condition was experienced by one participant per group (six total, orange in Table 4); the N=18 framing in the abstract and conclusion obscures the treatment sample size.
[§5.3] Proactive rates of 87.98%/89.91% are reported without a test or discussion. With the agent providing support in ~90% of invocations, the 'proactive decision' rarely withholds, which sits uneasily with DC1 and participants' reports of both unnecessary and missing support; a brief analysis would help.
[Table 1] Clarify the definition of participation inequality (%) and report effect sizes or confidence intervals for the Tau-U indices; also specify how the adoption score is computed when the MR user produced no utterance before the next support rendering.
[§5.3, RQ1] Partner perceptions ('presenter,' 'less genuine,' 'artificial while transitioning topics') are reported but not analyzed; a short paragraph on the risk of over-reliance/inauthenticity would balance the claimed benefits.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the feedback-pipeline claims are measured with independent speech-based metrics, and the lone self-citation is background, not load-bearing.

full rationale

The paper's derivation chain is not circular. The feedback loop defines usefulness as u = t/T (Sec. 4.4) and feeds u back into the episode context, but the RQ2 claims are not evaluated with u. RQ2 uses (i) participation inequality, the standard deviation of speaking-time proportions, and (ii) support adoption score, the maximum cosine similarity between the generated example speech and the MR user's subsequent utterances, tested with Tau-U (Sec. 5.2, Table 1). These metrics are computed from speech/transcripts, not from gaze duration, so improvement in them is not equivalent to u by construction. The only self-citation ([133], Related Work: "Zhou et al. [133] demonstrated the need for and value of providing private information support to facilitate in-person group conversations") is a background statement about prior needs research and is not load-bearing; the present paper independently runs a formative study (Sec. 3) that produces the design considerations that drive the system. No uniqueness theorem, no fitted-parameter-renamed-as-prediction, and no renaming pattern is present. The unvalidated nature of the gaze proxy is an internal-validity threat, and the paper itself acknowledges related sensing limitations in Sec. 6.2, but that is a correctness/validity concern, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The invented 'entity' is a software agent and a usefulness metric, both of which are configurations of existing components. The ledger instead captures the modeling assumptions that the system's effectiveness depends on.

free parameters (3)

Gaze/attention detection threshold = 30 degrees
Section 4.3 sets a 30-degree horizontal angle as the condition for one participant looking at another; this hand-set threshold defines the attention graph that feeds the LLM context.
Reading-speed constant in usefulness score = wpm = 238
Section 4.4 uses 238 words per minute from Brysbaert [20] to compute expected reading time T; not fitted to study data, but a model choice that scales the usefulness score.
Maximum support output length = 50 words
Section 4.2 limits generated support to 50 words as an empirical design constraint; it affects readability and cognitive load but is not a core statistical parameter.

assumptions (5)

domain assumption Headpose direction approximates gaze direction for non-MR participants.
Section 4.1 uses forward-facing headpose as a proxy for gaze, citing Jha and Busso [61] for horizontal-accuracy; the attention graph is built on this proxy.
domain assumption Gaze duration on the support overlay reflects how useful the support was.
Section 4.4 defines usefulness u = t/T and justifies it with the general claim that longer viewing reflects sustained attention; no task-specific validation is provided.
domain assumption Embedding cosine similarity between suggested example speech and subsequent user speech measures actual adoption of the support.
Section 5.2 defines support adoption score as the maximum cosine similarity using all-MiniLM-L6-v2; this is a proxy, not a verified measure of behavioral adoption.
domain assumption Speech-emotion labels from a pretrained classifier are accurate enough for the agent's context.
Section 4.3 relies on a pretrained speech emotion classification model with reported benchmark accuracy 83.79%, and on Ekman's six basic emotions; no in-situ accuracy check is reported.
domain assumption LLM-generated proactive decisions and textual rationales are meaningful evidence of conversational grounding.
Section 5.3 treats the agent's rationales as supporting evidence for RQ2; this assumes the LLM's stated reasons reflect the actual inputs rather than confabulation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ChatMuse: Supporting In-Person Small-Group Conversation Experience with a Proactive Assistive AI Agent in Mixed Reality." pith.science (2026). https://pith.science/paper/5XV2IZA5

@misc{pith2026260718556,
  author       = {Pith},
  title        = {Pith review of: ChatMuse: Supporting In-Person Small-Group Conversation Experience with a Proactive Assistive AI Agent in Mixed Reality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5XV2IZA5}},
  note         = {Machine review of arXiv:2607.18556}
}

read the original abstract

In-person small-group conversations occur across nearly every aspect of daily life and play a crucial role in social interaction. However, achieving effective in-person group conversations can be challenging and cognitively demanding. While recent Mixed Reality (MR) headsets show promise as a conversational support system by presenting relevant information through overlays, it remains unclear how such supporting information should be designed and generated for in-person group conversations. We propose ChatMuse, a novel MR-based proactive assistive system for in-person small-group conversation experience. ChatMuse analyzes verbal and non-verbal cues from all conversation participants and proactively provides real-time guidance on the user's verbal and non-verbal behaviors. The behavioral responses of the supported users are then used to improve ChatMuse's support capabilities in subsequent interactions. We conducted a within-subject study to evaluate and demonstrate the feasibility and effectiveness of ChatMuse in assisting users to engage in and contribute to in-person small-group conversations. Our research around ChatMuse represents a design exploration of a new interaction space that investigates the feasibility of supporting in-person small-group conversations through a proactive assistive AI agent in MR.