{"id":"3a9408f9-b233-459f-ba5b-3f91fb2485e3","arxiv_id":"2504.18189","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AI-written danmaku, combining content and emotion types, can match human comment quality and significantly boost learner engagement and quiz gains in short educational videos.","lead":"This paper presents ClassComet, a video platform that uses a large multimodal model to write danmaku (live scrolling comments) for educational videos, and tests whether these AI comments help people learn. In a small user study, videos with both content-focused and emotion-focused AI comments improved engagement and quiz performance compared to plain videos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The significant learning/engagement effect may be an artifact of video-condition confounding: with 12 participants and random assignment, no video factor or balance check is reported, and the reported ANOVA df (F(4,12)) are inconsistent with a one-way repeated-measures design.","rationale":"I focused on the quantitative claim rather than the qualitative feedback. The system contribution (LMM pipeline, persona generation) is real and reproducible, and the quality comparison is suggestive, but the headline empirical claim of significant improvement is the load-bearing part. The within-subjects design with random condition-video pairing makes video a potential confound, and the reported df suggest a statistical mis-specification. Both point to one check: re-analysis with video as a factor. If the effect survives, the concern is resolved; if not, the paper's central claim is unsupported. This is not a disagreement with the research direction, and the paper's own stated limitations (small sample, lab study) do not excuse the missing analysis.","tokens_in":33008,"tokens_out":3978,"duration_ms":37967,"concrete_test":"Obtain the Stage 1 per-participant data and re-fit the learning-outcome and engagement models as repeated-measures ANOVA or linear mixed models with both condition and video as within-subject factors, including the video-by-condition interaction and a balance table of video-condition pairings. If the condition effect on learning outcome is no longer significant after controlling for video, or if the reported F(4,12) values cannot be reproduced, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2.2 states that the four videos (supervised learning, brain structure, music theory, Latin alphabets) were assigned to the four danmaku conditions by random assignment per participant. In this within-subjects design, each participant watches all four videos, one per condition, so condition and video are confounded within each participant. Section 7.1 reports 'one-way repeated measures ANOVAs' on condition, with F(4,12) values (e.g., learning outcome F(4,12)=6.47). For a one-way repeated-measures ANOVA with 4 conditions and 12 participants, the df should be (3,33) (or corrected), so the reported F(4,12) is not consistent with the stated design and the significance tests cannot be taken at face value. No video main effect, video-by-condition interaction, or balance table is reported. With n=12, random assignment does not guarantee that video difficulty or topic interest is balanced across conditions; if, e.g., the music-theory or Latin video was paired more often with No-Danmaku for low-scoring participants, the observed 'significant' improvement could be a video effect rather than a danmaku effect. The abstract's claim that Content+Emotion danmaku 'showed significant improvement' therefore rests on an unverified and possibly mis-specified statistical model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents ClassComet, a platform that uses large multimodal models (LMMs) to automatically generate danmaku (live synchronized video comments) for educational videos. The authors first run a formative study with seven participants to identify valued characteristics, resulting in seven danmaku types split into content-related (Q&A, discussion, highlights, summary) and emotion-related (personal emotion expression, brief compliment, encouragement). They then describe a generation pipeline built on GPT-4o that combines clip-level and text-level video descriptions with six virtual personas and a structured prompt template. The evaluation has two stages: a within-subjects study with 12 participants comparing four conditions (No-Danmaku, Emotion-only, Content-only, Content+Emotion) on learning gain and three engagement dimensions, and a quality comparison of generated danmaku versus human danmaku from Bilibili on relevancy, consistency, fluency, and coherence. The paper concludes that generated danmaku is comparable to human-created danmaku and that Content+Emotion danmaku significantly improves engagement and learning outcomes.","tokens_in":33308,"tokens_out":5672,"duration_ms":58197,"significance":"If the central claims were fully supported, this would be a useful contribution to HCI and educational technology: it provides a concrete design space for AI-generated social annotations and demonstrates a feasible pipeline for supplying educational videos with danmaku when human-generated comments are scarce. The work has notable strengths: the formative study grounds the seven-type taxonomy in user preferences; the pipeline is described in sufficient detail to be replicated, including the prompt templates in appendix C; and the quality comparison uses an external human benchmark from Bilibili, which is the right kind of reference point for the claim. The qualitative data enrich the quantitative results. However, the statistical evidence for the headline claims is currently incomplete and internally inconsistent, so the significance cannot be fully assessed until the analyses are corrected and reported in full.","major_comments":[{"comment":"The reported ANOVA degrees of freedom are inconsistent with the stated design. For a one-way repeated-measures ANOVA with four conditions and N=12, the expected F-statistic has degrees of freedom (3,33) before sphericity correction, not F(4,12). The same F(4,12) pattern appears for learning outcome, behavioral engagement, emotional engagement, and cognitive engagement. Since the abstract's claims of significant improvement rest on these tests, the authors must provide the full ANOVA results, including corrected degrees of freedom if sphericity was adjusted, or a precise description of the actual model fitted. As written, the reported statistics cannot be evaluated.","section":"7.1.1–7.1.4"},{"comment":"The design pairs danmaku condition with video content within each participant, and the reported analyses do not control for this pairing. Each participant watched four videos (supervised learning, brain structure, music theory, and Latin alphabets) with one of the four danmaku conditions randomly paired per participant. With only 12 participants, random assignment does not ensure that video difficulty, topic interest, or prior knowledge is balanced across conditions, and the paper reports no video main effect, no condition-by-video interaction, and no balance table. The significant learning-outcome and engagement differences could therefore be driven by the specific videos rather than by the danmaku condition. The authors should add video as a factor in the model, or provide a convincing balance analysis and supplementary within-video comparisons.","section":"6.2.2 and 7.1"},{"comment":"The quality comparison reports no inferential statistics. The text states that generated danmaku was 'superior' in relevancy, 'similar' in consistency and fluency, and 'lower' but 'almost comparable' in coherence, but no tests, effect sizes, or confidence intervals are reported for these comparisons. The abstract's claim that generated danmaku is 'comparable to human-created ones' is a headline result and needs formal support, for example paired comparisons that account for the six clips and the 12 raters, with appropriate multiple-comparison corrections.","section":"7.1.5 and Figure 9"}],"minor_comments":[{"comment":"The prompt template specifies '15-25 content-related danmaku and 5-10 emotion-related danmaku per minute' and 'more than 10 highlight per minute'; these numbers seem high relative to the earlier statement that danmaku frequency in educational videos ranges from 2 to 30 per minute, and the paper does not report how the generated output was validated against these constraints. Please clarify the intended counts and describe any post-generation filtering or verification.","section":"5.3.3 and Appendix C"},{"comment":"The usage scenario refers to 'Figure 5-A' and 'Figure 5-C' when describing the video player controls and danmaku input box, but the relevant interface figure is Figure 2. Please correct these cross-references.","section":"4.2"},{"comment":"Several appendix passages contain garbled characters and artifacts, such as 'texcl' and 'âĂŹ', which should be cleaned before publication.","section":"Appendices A–C"},{"comment":"The learning-outcome measure is the post-quiz minus pre-quiz score, but the paper does not report the pre-quiz scores or check for ceiling or floor effects. Please include descriptive statistics for both pre and post scores for each condition.","section":"6.2.3 and 7.1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising system and an appropriate external benchmark, but the current statistical reporting is not reliable enough for publication. The repeated F(4,12) values across all four outcomes suggest a systematic reporting error rather than a one-off typo, and the missing video factor and absent inferential statistics for the quality comparison are load-bearing gaps. I would not rule out acceptance after a careful revision that reanalyzes and fully reports the data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on ClassComet. The core idea is genuinely new in combination: LMM-generated educational danmaku, a learning-oriented taxonomy from a formative study, and virtual personas to simulate peer interaction. Prior work did generation for entertainment (LiveBot, VideoIC) and personas for comments (SimTube), but not the educational pipeline with quiz-gain evaluation. That alone makes this worth talking about.\n\nWhat's solid: the formative study is reasonable, and the seven types of danmaku with the content/emotion split are sensible. The qualitative feedback lines up with the design rationale. The pipeline is described in detail with prompts in the appendix, and the external benchmark against human Bilibili danmaku is the right idea to ground the quality claim.\n\nThe problems are mostly quantitative, and they're not minor. The ANOVAs report F(4,12) for a four-condition within-subjects design with 12 participants; those degrees of freedom don't match a one-way repeated-measures model (should be F(3,33) or a corrected variant). This appears across all four outcome measures, so the reported p-values can't be trusted. On top of that, the design randomly assigns four different videos to four conditions, and there's no video main effect, no condition-by-video interaction, and no balance check. With n=12, video topic and difficulty could easily be confounded with condition. The abstract's \"significant improvement in learning outcome\" rests on this unverified footing.\n\nStage 2 has a different problem: the quality comparison reports no inferential statistics at all, just means and error bars. \"Comparable to human danmaku\" is plausible but unverified. Also, the Bilibili clips were selected from the moments with the most danmaku, which should bias in favor of the human comments; the AI still looked competitive on relevancy, which is encouraging but not a statistical result.\n\nI'd push back on one part of the reader's circularity concern: the taxonomy is self-defined and then generated to spec, but the external human-danmaku comparison gives the central quality claim independent grounding. The load-bearing weakness is the statistics, not the grounding.\n\nWho is this for: HCI and learning-tech readers interested in AI-generated social presence in video learning. It deserves serious peer review, but only with revision: fix the df, run or report the missing video analyses, and either temper the significance claims or produce a correctly specified model. I'd read it again after that.","headline":"Fresh systems contribution to AI-generated danmaku for learning, but the headline learning-outcome claim is built on misreported statistics and a possible video-condition confound.","tokens_in":33822,"tokens_out":2272,"would_cite":true,"duration_ms":24134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large multimodal models can generate danmaku comments for educational videos comparable to human-created ones, and videos combining content- and emotion-related danmaku significantly raise engagement and learning gains.","keywords":["danmaku","video-based learning","large multimodal models","AI-generated comments","learner engagement","learning outcomes","virtual personas","online education"],"falsifier":"A reanalysis of the stage-one data that breaks learning gains down by video would settle it: if the Content+Emotion advantage appears only in one video and disappears in the other three, the effect is video-driven rather than danmaku-driven. A preregistered replication with more participants and all four videos fully crossed with all four conditions would give the same answer more cleanly.","tokens_in":32804,"feed_emoji":"💬","tokens_out":9627,"duration_ms":95617,"temperature":0.7,"pith_summary":"Danmaku—live, timestamped viewer comments that scroll over a video—can make online lectures feel social, but most educational videos have too few viewers to produce any. The paper argues that large multimodal models can fill that gap: ClassComet reads a video's frames and transcript, creates six virtual learner personas, and generates danmaku of seven types, split into content-related (Q&A, discussion, highlights, summaries) and emotion-related (personal expressions, compliments, encouragement). In a 12-participant within-subjects study, videos carrying both content- and emotion-related generated danmaku produced significantly larger pre-to-post quiz gains and higher behavioral, emotional, and cognitive engagement than the same videos with no danmaku. A second comparison found the generated comments at least comparable to human-created danmaku on relevance, consistency, and fluency, with coherence still lower. The authors conclude that every lecture video, not just popular ones, could receive a helpful, encouraging live comment stream.","feed_headline":"AI-generated danmaku lifts learning in online lecture videos","feed_subtitle":"Niche courses lack scrolling viewer comments; ClassComet's multimodal pipeline supplies them and boosts engagement.","key_machinery":"The machine that carries the argument is a four-step LMM pipeline. Scene detection and frame sampling turn a long video into clip-level visual descriptions; an automatic transcription service produces timestamped text-level descriptions; a persona-creation prompt produces six distinct virtual viewers with backgrounds, personalities, and comment styles; and a structured prompt with in-context examples and explicit constraints (comments at most 12 characters, no gaps longer than 30 seconds, 15–25 content and 5–10 emotion comments per minute, more than 10 highlights per minute) generates the final danmaku stream. The seven-type taxonomy separates content-related danmaku (Q&A, discussion, highlights, summary) from emotion-related danmaku (personal expression, compliment, encouragement), and the persona layer converts isolated comments into simulated peer exchanges such as answers, praise, and encouragement.","core_discovery":"The central discovery is that a multimodal model with access to both visual and textual information can generate the social layer of an educational video, and that this layer has measurable pedagogical value. The paper establishes a taxonomy of seven valued danmaku types in two classes and shows the classes have distinct effects: content-related danmaku alone improved quiz gains, behavioral engagement, and cognitive engagement; emotion-related danmaku alone improved emotional engagement; and only the combined condition improved all three engagement dimensions at once. Against human danmaku from the same video segments, the generated comments were rated higher on relevance and comparable on factual consistency and linguistic fluency, while human comments retained an edge in conversational coherence. The conclusion is that AI-generated danmaku can stand in for scarce user-generated danmaku without sacrificing quality on the dimensions that matter most for learning.","pith_inferences":["I infer that the active ingredient may be simulated social presence: if the learning gains are real, generated danmaku functions as a scalable substitute for watching with a real peer group, and would be worth comparing directly with other social-presence video augmentations.","The persona layer is a variable the paper does not isolate; I infer that a follow-up comparing persona-driven danmaku with the same text presented without named identities would reveal how much of the effect comes from believable virtual commenters.","Given the small sample and random video-condition pairing, I would not treat the effect sizes as portable until a replication crosses all four videos with all four conditions and checks whether any single topic drives the result.","I infer that adapting generation in real time—thinning content comments when a learner appears to be struggling, or thickening encouragement near difficult sections—is a natural extension the current pre-generated pipeline does not yet support."],"forward_implications":["Educational video platforms can offer a guaranteed danmaku layer on every video, including newly uploaded or low-view content where human comments are scarce.","Content- and emotion-related danmaku are complementary: content-only improves quiz gains and cognitive and behavioral engagement, emotion-only improves emotional engagement, and their combination is the only condition that moves all dimensions at once.","AI-generated danmaku can coexist with user-generated danmaku in the same stream, so platforms do not have to choose between automated and human comments.","The coherence gap in generated danmaku marks a concrete target: multi-turn interactions still read as less natural than human conversations, and improving that dimension is the clearest next step for the generator.","Future versions can vary the knowledge level of virtual personas, including less-expert teachable personas, to reduce the pressure some learners feel when generated answers arrive too quickly."],"supporting_citations":[{"why":"Supplies the frame-sampling and video-to-text method that ClassComet adapts so a multimodal model can understand long educational videos.","marker":"[52]"},{"why":"Provides the baseline statistics on danmaku length, posting frequency, and knowledge-sharing behavior that ClassComet's generation constraints are calibrated to.","marker":"[90]"},{"why":"Establishes the content-related versus social-oriented classification of danmaku and the preference for content-focused discussion that the taxonomy extends.","marker":"[12]"},{"why":"Classifies danmaku interactions into content-related and social-oriented categories and informs the taxonomy used in the prompt template.","marker":"[11]"},{"why":"Introduces automated live video comment generation and the four quality metrics used to evaluate ClassComet's output.","marker":"[55]"},{"why":"Provides the related approach of augmenting educational videos with scripted dialogue, which frames the engagement and quiz-gain measurement strategy.","marker":"[78]"},{"why":"Supplies evidence that persona-based comments are rated higher on relevance, believability, and helpfulness, motivating the virtual-viewer design.","marker":"[37]"}],"fun_headline_variants":["AI danmaku boosts learning in online lecture videos","Generated danmaku matches human quality, lifts engagement","AI fills in scarce danmaku to improve online learning","Multimodal AI creates helpful danmaku for educational videos","ClassComet's AI comments enhance learning and engagement in videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the four videos were equivalent enough, after random pairing with the four danmaku conditions, for differences in quiz gains and engagement to be attributed to the danmaku rather than to video topic, difficulty, or participants' prior knowledge.","fun_headline_variants_meta":{"raw":{"variants":["AI danmaku boosts learning in online lecture videos","Generated danmaku matches human quality, lifts engagement","AI fills in scarce danmaku to improve online learning","Multimodal AI creates helpful danmaku for educational videos","ClassComet's AI comments enhance learning and engagement in videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2570,"prompt_tokens":883,"completion_tokens":1687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":1606}},"tokens_in":499,"tokens_out":1687,"duration_ms":12358,"temperature":1.0,"reasoning_tokens":1606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:22:16.282078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reanalysis of the stage-one data that breaks learning gains down by video would settle it: if the Content+Emotion advantage appears only in one video and disappears in the other three, the effect is video-driven rather than danmaku-driven. A preregistered replication with more participants and all four videos fully crossed with all four conditions would give the same answer more cleanly.","supporting_citations":[{"cited_title":"SimTube: Generating Simulated Video Comments through Multimodal AI and User Personas","cited_arxiv_id":"2411.09577","evidence_quote":"Supplies evidence that persona-based comments are rated higher on relevance, believability, and helpfulness, motivating the virtual-viewer design."}],"review_version":1}