ChatNekoHacker: Real-Time Fan Engagement with Conversational Agents

Takuya Sera; Yusuke Hamano

REVIEW 3 major objections 5 minor 5 references

ChatNekoHacker: Real-Time Fan Engagement with Conversational Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In a one-hour livestream, two AI personas representing a music duo raised viewers' self-reported interest, with perceived fun as the only significant driver.

desk verdict A small, honest system-and-study paper whose central 'elevated fan interest' claim is not supported by its post-only, no-control design, but the system description and persona-construction method deserve a look. read the letter →

arxiv 2504.13793 v1 pith:TEEL7ERL submitted 2025-04-18 cs.HC

Takuya Sera , Yusuke Hamano This is my paper · ORCID

classification cs.HC

keywords conversationalagentsfanengagementlargelanguagemodelsretrieval-augmentedgenerationlivestreamingvirtualpersonastext-to-speechmusicfandom

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports on ChatNekoHacker, a system in which two LLM-driven personas represent the music duo Neko Hacker during a live YouTube broadcast, taking viewer comments and replying by text-to-speech in Japanese. The authors aim to show that such real-time conversational agents can increase fan engagement with a musician — specifically self-reported interest in the artist and willingness to listen to more music and attend live events. After a one-hour broadcast to 30 viewers, 83% reported increased interest, and a least-squares regression found that only perceived fun (coefficient $0.59$, $p=0.01$) significantly predicted that interest, with useful, realistic, and unity-related impressions contributing little. The paper concludes that entertaining interaction is key to cultivating fandom and that the approach is a practical, low-barrier option for artists.

What carries the argument

The carrying object is the ChatNekoHacker system itself: Amazon Bedrock Agents with a RAG knowledge base (summaries of the duo's social media posts, classified into 15 categories, plus Wikipedia activity overviews) and an action group for web searches, all wrapped in a Unity 3D live-broadcast environment and voiced by VOICEVOX Japanese text-to-speech. Prompt engineering also forces the two personas, Neko-Chan and Hacker-Chan, to speak in the members' Kansai dialect. The machinery does two jobs: it produces the autonomous, real-time conversational interaction that viewers experienced, and it records the YouTube comments that trigger responses — the interaction being the independent variable whose effect the survey then measures.

What would settle it

Run a between-subjects experiment with the same one-hour broadcast and the same survey, but give the control group a no-agent version of the stream (e.g., captions or a pre-recorded presenter) while keeping music and visual content identical; if the control group shows an equally large rise in self-reported interest, the claim that agent interaction elevates fan interest is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a one-hour live broadcast in which two conversational agents, built from a retrieval-augmented knowledge base of the duo's social media and Wikipedia summaries and voiced by Japanese text-to-speech, engaged with viewer comments significantly elevated fan interest. Among 30 survey respondents, 83% agreed or slightly agreed that their interest in the artist increased. A least-squares regression over four experience items gave adjusted $R^2 = 0.56$ ($F = 10.27$, $p = 0.00005$), with "Fun" the only statistically significant predictor (coefficient $0.59$, $p = 0.01$); "Useful", "Reality", and "Unity" were not significant. The authors further report that respondents expressed stronger intentions to listen to more music and attend more live events, with no significant difference between frequent and infrequent listeners or between those with and without concert experience.

Load-bearing premise

The causal interpretation rests on a single post-broadcast survey with no control group and no pre-measure, so the 83% reported interest increase could reflect pre-existing fandom, the novelty of an AI-hosted event, or the effect of watching any live broadcast rather than the agent interaction specifically.

Editorial extensions

If this is right

Interactivity itself can convert a routine livestream into a fan-engagement channel: 83% of surveyed viewers said their interest in the artist grew after the agent conversation.
Perceived fun, not perceived usefulness or realism, is the lever that moves fan interest, so broadcast design should prioritize entertaining dialogue.
Conversational agents can shift intentions to act — listening to more music and attending live events — even among fans with different prior listening or concert experience.
Real-time comment-driven agents may also nudge purchase behavior, as illustrated by the viewer who bought an item after it was restocked during the broadcast.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If fun is the dominant driver, then future systems should be optimized for response variety, humor, and pacing rather than factual fidelity; the paper's own free-text feedback ("conversation lacks variety") points in this direction.
The same RAG-plus-TTS architecture could transfer to other artists or brands with modest effort, since it relies on public social media and Wikipedia content rather than proprietary fine-tuning; a multi-artist deployment would test this generality.
Because the study has no control group and only a post-broadcast survey, the measured interest gain is likely a mix of agent effect, novelty, and fandom; a within-subject or control-broadcast design would isolate the causal contribution.
The observed purchase anecdote, if replicated in a larger study, suggests that conversational agents could be a measurable sales channel, not just an engagement channel.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a clearly written, honest system description plus a small survey study. What's new is the application: using RAG-based conversational agents to represent two musicians in a live YouTube broadcast, with personas built from LLM-summarized categories of social media posts. That persona-construction method is sensible and much cheaper than fine-tuning, and the integration with Unity and VOICEVOX is a credible feasibility demonstration. The authors list their limitations honestly: small sample, potential misinformation, latency, and limited response variety.

The soft spot is the causal claim. The abstract says agent interaction 'significantly elevated fan interest,' but the design is post-only with no baseline and no control condition. The 83% positive response to a single retrospective Likert item cannot distinguish the agent's effect from pre-existing fandom, novelty, or simply watching any live broadcast. The regression is also only within the exposed group: a significant coefficient for 'Fun' means that viewers who enjoyed the conversation reported more interest; it does not show that the agent caused either variable. The line about the model being 'valid' because the F-statistic is significant is a minor statistical overreach, and there is no correction for multiple inference. No code or data are provided, which limits reproducibility, though that is common for a workshop paper.

None of this is fatal for what the paper actually is: a case study showing that such a system can run stably for an hour, that two personas can converse from a RAG knowledge base, and that viewers can find it fun. The qualitative transcript excerpt gives a concrete feel for the interaction. The authors should soften the abstract to say something like 'viewers who perceived the conversation as fun also reported increased interest' and present the results as a feasibility demonstration with suggestive findings.

I would send this to peer review rather than desk reject, because it contributes a working system and a lightweight persona-building pipeline that other researchers and practitioners might build on. The evidence is too weak for the current claim, but the paper is worth referee time and could be revised into a solid workshop or short-paper contribution. My own verdict is skeptical only on the causal interpretation; the system work is sound.

Referee Report

3 major / 5 minor

Summary. The paper presents ChatNekoHacker, a real-time conversational agent system for music fan engagement. The system integrates Amazon Bedrock Agents for autonomous dialogue, Unity for a 3D livestream environment, and VOICEVOX for Japanese text-to-speech, producing two virtual personas for the duo Neko Hacker. A one-hour YouTube Live broadcast with 30 self-selected viewers was conducted, and a post-broadcast survey measured self-reported interest, enjoyment, usefulness, perceived realism, unity, listening intentions, and live-event intentions. The authors report that 83% of respondents agreed that their interest in the artist increased, and an OLS regression of the Interest item on Fun, Useful, Reality, and Unity yielded a significant positive coefficient only for Fun (coefficient 0.59, p=0.01, adjusted R²=0.56). The paper concludes that agent interaction significantly elevated fan interest, that perceived fun is the dominant predictor, and that the system enhanced willingness to listen to and attend live events, with an anecdote about a viewer purchasing a restocked item.

Significance. If the causal and evaluative claims were supported, this paper would provide valuable evidence that LLM-driven persona agents can engage music fans in live broadcasts, with design implications for entertainment-oriented conversational agents. The system description is concrete and reproducible: it names the cloud services, the TTS voices, the knowledge-base construction (15 categories of social media posts), and the RAG architecture. The authors also list limitations honestly, including small sample size, response diversity, latency, and fact-checking concerns. However, the empirical evaluation design is the main weakness: it is a post-only, no-control, self-selected survey. The data can support correlational statements about relationships among self-reported perceptions, but they cannot support the abstract's claim that agent interaction 'significantly elevated fan interest' in a causal sense. The paper's contribution is therefore best understood as a technical case study with exploratory survey findings, rather than a demonstration of causal effectiveness.

major comments (3)

[Abstract and §3 (Table 2, Interest item)] The central claim that 'agent interaction significantly elevated fan interest' is not supported by the study design. The survey is post-only, with no baseline measure of interest before the broadcast, no control condition (e.g., a non-interactive or human-hosted stream), and no randomization. The 83% positive response to the retrospective item 'Interest: Increased interest in the artist' could reflect pre-existing fandom, the novelty of an LLM-driven persona, social desirability, or simply the effect of watching any live broadcast. The regression reported in Table 2 only shows that, within this exposed group, higher Fun scores are associated with higher Interest scores; it does not establish that interest increased over time or that the agent caused any increase. The abstract, introduction, and discussion should be revised to state that the findings are exploratory and correlational, or the authors must provide a baseline/control comparison.
[§3 (Table 2) and Abstract] The phrase 'perceived fun as the dominant predictor' is an overstatement of the reported statistics. Table 2 reports unstandardized coefficients and p-values, but no standardized coefficients, effect sizes, or model comparisons are provided, so 'dominant' is not quantified. Moreover, all predictors (Fun, Useful, Reality, Unity) are measured simultaneously by the same self-report instrument, raising concerns about common-method bias and multicollinearity; an examination of variance inflation factors or a comparison of nested models would be needed to support the dominance claim. At minimum, the wording should be softened to 'perceived fun was the only statistically significant correlate in this sample.'
[§4 (Discussion and Conclusion)] The anecdote about a viewer commenting on an out-of-stock item and later purchasing it after a restock is presented as evidence that real-time conversational agents 'may foster purchasing intent and support consumer behavior.' This is a single unmeasured anecdote: no data are reported about whether the viewer's purchase was influenced by the agent, whether the restock was caused by the agent's comment, or how many other viewers also purchased. It should be explicitly labeled as an anecdotal observation and removed from the evidence base supporting the system's effectiveness.

minor comments (5)

[§3 (Frequent/Infrequent grouping)] The grouping criteria for the Frequent and Infrequent listener groups are unclear: respondents who listened 'every day' are compared with those who listened 'less than several times per week,' but the survey response options are not reported, and the classification of 'several times per week' is ambiguous. The exact survey scale and the cut-off used should be stated, and the chi-square test statistics, degrees of freedom, and p-values should be reported.
[§2.2 (Knowledge base construction)] The paper states that social media posts were classified into 15 categories and summarized, but no details are given about the categories, the classification prompt, or the accuracy of the classification. For reproducibility, the authors should provide at least a summary of the categories or the prompt template.
[§3 (Survey administration)] No information is provided about participant recruitment, informed consent, or ethical approval. Since the survey was administered during a public live broadcast, the authors should clarify how participants were informed and consent was obtained, as is expected for chi-style user studies.
[Table 3 (not present; Figure 3)] Figure 3 should include explicit group sizes (n for Frequent/Infrequent and Experienced/Inexperienced) and clearly labeled axes, and the caption should identify the survey scale. Currently the figure is referenced but its construction is not described in the text.
[Throughout] Minor typographical and style issues: 'chi-square test conducted for the items ListenMore and JoinMore revealed no significant difference' is missing 'respectively' (there appear to be two separate tests); 'based on the members' past posts' should be 'on the members' past posts'; and reference [2] lacks an accessed date format consistent with the other references.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the regression summarizes within-sample survey associations; the causal claim is a design limitation, not a circularity.

full rationale

This paper does not attempt a mathematical derivation, so the standard circular-derivation failure modes do not apply. The central statistical result is a least-squares regression of the post-only survey item Interest on four simultaneous self-report items (Fun, Useful, Reality, Unity). No item is defined in terms of another by construction: the regression equation is not an identity, and the coefficient for Fun (0.59, p=0.01) is an estimated association, not a quantity fitted to reproduce the outcome. The abstract's phrase 'perceived fun as the dominant predictor' uses 'predictor' in the standard regression sense and is accurately supported by Table 2; it is not a disguised out-of-sample prediction. Similarly, the 83% positive response to 'Interest: Increased interest in the artist' is reported directly from the survey, and the claim that agent interaction 'elevated' interest is an interpretation of that retrospective item, not a reduction of the conclusion to its inputs. The paper contains no load-bearing self-citation: its references are background (music fandom, VOICEVOX, Character-LLM, SimsChat) and are not used to justify the empirical conclusion. The genuine weakness is causal inference: with no pre-broadcast baseline and no control condition, the positive responses could reflect selection, novelty, or social desirability. That is a correctness and validity limitation, explicitly acknowledged in part by the small-sample caveat, but it is not circularity under the stated rules, which require a specific reduction such as an equation equating output to input or a fitted parameter renamed as a prediction. Accordingly, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central quantitative result is a regression of self-reported interest on four self-reported experience items using 30 observations. The four regression coefficients are fitted values, not independent predictions. The load-bearing assumptions are the survey-validity assumption, the sample-representativeness assumption, standard OLS assumptions, and especially the post-only attribution assumption. There are no invented entities.

free parameters (5)

OLS coefficient for Fun = 0.59
Estimated from the survey data in the regression model for Interest; it is the basis for the claim that fun is the dominant predictor.
OLS coefficient for Useful = 0.24
Estimated from the survey data; not statistically significant in the reported model.
OLS coefficient for Reality = 0.00
Estimated from the survey data; not statistically significant in the reported model.
OLS coefficient for Unity = 0.25
Estimated from the survey data; not statistically significant in the reported model.
Number of social media post categories = 15
Chosen by hand for building the knowledge base; influences response quality but is not part of the regression outcome.

assumptions (4)

domain assumption Self-reported Likert ratings of interest, fun, usefulness, reality, and unity correspond to genuine psychological states and predict real engagement behavior.
The entire analysis uses post-broadcast survey items as measures of fan engagement without validation against behavioral outcomes.
domain assumption The 30 viewers who joined the one-hour live stream are an acceptable sample for estimating fan engagement effects.
No recruitment information, response rate, or representativeness analysis is provided; participants are likely pre-existing fans.
standard math OLS assumptions (linearity, independence, homoscedasticity, no multicollinearity) hold with 30 observations and four correlated Likert predictors.
The regression requires these assumptions, but the paper reports only coefficients, p-values, and R-squared, with no diagnostics.
ad hoc to paper Post-only measurements can be attributed to the agent interaction without a baseline or control condition.
This is the load-bearing interpretive assumption behind the phrase 'elevated fan interest'; it is not tested anywhere in the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ChatNekoHacker: Real-Time Fan Engagement with Conversational Agents." pith.science (2026). https://pith.science/paper/TEEL7ERL

@misc{pith2026250413793,
  author       = {Pith},
  title        = {Pith review of: ChatNekoHacker: Real-Time Fan Engagement with Conversational Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TEEL7ERL}},
  note         = {Machine review of arXiv:2504.13793}
}

read the original abstract

ChatNekoHacker is a real-time conversational agent system that strengthens fan engagement for musicians. It integrates Amazon Bedrock Agents for autonomous dialogue, Unity for immersive 3D livestream sets, and VOICEVOX for high quality Japanese text-to-speech, enabling two virtual personas to represent the music duo Neko Hacker. In a one-hour YouTube Live with 30 participants, we evaluated the impact of the system. Regression analysis showed that agent interaction significantly elevated fan interest, with perceived fun as the dominant predictor. The participants also expressed a stronger intention to listen to the duo's music and attend future concerts. These findings highlight entertaining, interactive broadcasts as pivotal to cultivating fandom. Our work offers actionable insights for the deployment of conversational agents in entertainment while pointing to next steps: broader response diversity, lower latency, and tighter fact-checking to curb potential misinformation.

Figures

Figures reproduced from arXiv: 2504.13793 by the authors.

**Figure 1.** ChatNekoHacker User Interface ChatNekoHacker is a real-time conversational agent system that strengthens fan engagement for musicians. It integrates Amazon Bedrock Agents for autonomous dialogue, Unity for immersive 3D livestream sets, and VOICEVOX for high quality Japanese text-tospeech, enabling two virtual personas to represent the music duo Neko Hacker. In a one-hour YouTube Live with 30 participants, we evalua… view at source ↗

**Figure 2.** System Architecture During a one-hour live broadcast, surveys were collected from 30 viewers. The results were analyzed from the following two perspectives: • The contribution to enhancing fan interest and motivation to act. • The advantages and challenges faced by artists when implementing fan engagement via a conversational agent. 2 SYSTEM [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Results for “ListenMore: Felt like Listening to More Songs”, “JoinMore: Felt like Participating in More Live Events” [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 3 canonical work pages

[1]

Jessica Edlom and Jenni Karlsson. 2021. Hang with Me—Exploring Fandom, Brandom, and the Experiences and Motivations for Value Co-Creation in a Music Fan Community. International Journal of Music Business Research 10 (2021), 17–31

work page 2021
[2]

hiroshiba. [n. d.]. VOICEVOX. https://voicevox.hiroshiba.jp/ Accessed: 2025/2/3

work page 2025
[3]

Sihan Li. 2024. Behavior and Identity Construction of Online Fan Community—Taking Pocket 48 as an Example. Interdisciplinary Humanities and Communication Studies (2024)

work page 2024
[4]

Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A Trainable Agent for Role-Playing. ArXiv abs/2310.10158 (2023)

arXiv 2023
[5]

Bohao Yang, Dong Liu, Chen Tang, Chenghao Xiao, Kun Zhao, Chao Li, Lin Yuan, Guang Yang, Lanxiao Huang, and Chenghua Lin. 2024. SimsChat: A Customisable Persona-Driven Role-Playing Agent. ArXiv abs/2406.17962 (2024). GenAICHI 2025: Generative AI and HCI at CHI 2025 5

arXiv 2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Jessica Edlom and Jenni Karlsson. 2021. Hang with Me—Exploring Fandom, Brandom, and the Experiences and Motivations for Value Co-Creation in a Music Fan Community. International Journal of Music Business Research 10 (2021), 17–31

work page 2021

[2] [2]

hiroshiba. [n. d.]. VOICEVOX. https://voicevox.hiroshiba.jp/ Accessed: 2025/2/3

work page 2025

[3] [3]

Sihan Li. 2024. Behavior and Identity Construction of Online Fan Community—Taking Pocket 48 as an Example. Interdisciplinary Humanities and Communication Studies (2024)

work page 2024

[4] [4]

Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A Trainable Agent for Role-Playing. ArXiv abs/2310.10158 (2023)

arXiv 2023

[5] [5]

Bohao Yang, Dong Liu, Chen Tang, Chenghao Xiao, Kun Zhao, Chao Li, Lin Yuan, Guang Yang, Lanxiao Huang, and Chenghua Lin. 2024. SimsChat: A Customisable Persona-Driven Role-Playing Agent. ArXiv abs/2406.17962 (2024). GenAICHI 2025: Generative AI and HCI at CHI 2025 5

arXiv 2024