{"id":"f275b40b-9b93-400a-84af-8a8458f4cc7b","arxiv_id":"2509.03212","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AIVA combines a multimodal sentiment network with LLM prompt engineering, text-to-speech, and an animated avatar to produce emotion-aware companion responses, with MSPN reported to outperform prior multimodal sentiment baselines.","lead":"This paper describes AIVA, a virtual companion that reads emotions from images and text and feeds them into a large language model to generate empathetic responses. It reports a multimodal sentiment model called MSPN and claims it beats prior baselines on standard datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's 'w/o CAF' MVSA-Multi row (18.70% accuracy, 67.41% F1) is internally implausible and undermines the reliability of all reported empirical claims.","rationale":"The reader's weakest_assumption and my concern align: the reported numbers are the only support for the central claim, and the w/o CAF row is anomalous. I agree with the CONDITIONAL verdict because the issue could be a typo or a protocol mismatch; it does not by itself prove that the architecture is wrong. However, this is the single most load-bearing issue because every downstream claim—component importance, λ sensitivity, and ultimately AIVA's emotional alignment—is built on the same unverifiable tables. AIVA's end-to-end empathetic interaction is only shown qualitatively (Figure 8), so the strongest quantitative evidence is MSPN's classification performance; that evidence is suspect until the anomaly is resolved. There is no independent support: no code, no data, no formal verification, and no multiple-run variance estimates. The proposed concrete test would settle whether the row is a harmless typo or a sign of protocol inconsistency. Therefore the verdict remains CONDITIONAL—not ACCEPT or REJECT—because the concern is concrete but potentially fixable.","tokens_in":10115,"tokens_out":6572,"duration_ms":63848,"concrete_test":"Run the exact 'w/o CAF' configuration on MVSA-Multi and output the full confusion matrix and per-class precision/recall/F1. Then compute the maximum macro-F1 consistent with the reported 18.70% accuracy under the dataset's label distribution; if the maximum is below 67.41%, the row is internally inconsistent. Independently recompute Table 1 and Table 3 from released code/checkpoints with fixed seeds and report mean±std over at least 3 runs; any shift greater than 1 point would change the conclusion that MSPN's gains are real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All of the paper's empirical conclusions—MSPN beating all baselines in Table 1, every component mattering in Table 3, and λ=1.0 being optimal in Table 4—rest on numbers that cannot be independently checked because no code, data splits, seeds, or confusion matrices are provided. The most concrete red flag is Table 3's 'w/o CAF' row on MVSA-Multi: Accuracy=18.70% but F1=67.41%. For a three-class sentiment problem, macro-F1 is tightly bounded by the per-class true-positive counts; with accuracy 18.7%, the maximum possible macro-F1 is far below 0.67 (under balanced classes an upper bound is roughly 0.24–0.3, and label imbalance does not rescue the weighted version because the majority class would have to be almost entirely mispredicted). Thus at least one of the two numbers was not produced by the described protocol. If this baseline row is misreported, the paper's claim that CAF is essential (accuracy collapses from 73.48% to 18.70%) is unsupported, and it becomes unclear whether the full-model rows in Tables 1 and 4 were computed under the same protocol. This is the load-bearing weakness: the headline 'MSPN outperforms all baselines' cannot be separated from the integrity of the tables. The authors should correct or explain the anomaly and release artifacts before the empirical contribution can be evaluated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AIVA, an LLM-driven virtual companion that extracts sentiment cues from paired text and images with a Multimodal Sentiment Perception Network (MSPN), feeds the predicted emotion into an LLM through an emotion-aware prompt template (EPE), and renders responses via TTS and a Live2D avatar. MSPN combines a Vision Transformer and BERT with cross-attention fusion, a transformer with learnable sentiment prototypes, and bidirectional supervised contrastive learning. The empirical section reports MSPN accuracy/F1 on MVSA-Single, MVSA-Multi, and TumEmo, compares MSPN with six baselines on MVSA, and presents ablations of CAF, CMFT, SCL, and the contrastive-loss weight lambda, plus qualitative examples of the full system.","tokens_in":10406,"tokens_out":6988,"duration_ms":67989,"significance":"If the reported numbers are trustworthy, the contribution is a competent integration of known components (cross-attention, sentiment prototypes, supervised contrastive learning, and prompt-based emotion injection) into a practical LLM-based companion. The MSPN design does not introduce a fundamentally new mechanism, but the systems-level combination and the benchmark comparisons could be useful to the multimodal affective computing and HCI communities. The paper does not provide code, data splits, seeds, or a detailed evaluation protocol, so the empirical claims are not independently reproducible from the manuscript; this is the main limitation of the contribution as presented.","major_comments":[{"comment":"The row 'w/o CAF' on MVSA-Multi reports Accuracy 18.70% and F1 67.41%. For a three-class sentiment problem these two numbers are internally inconsistent under the standard macro-F1 definition: with balanced classes, a total true-positive count of 18.7% of the test set caps macro-F1 at roughly 0.3, and class imbalance does not rescue the pair because a high-F1 majority class would necessarily produce many correct predictions, raising accuracy well above 18.7%. This anomaly makes the ablation claim that removing CAF collapses MVSA-Multi accuracy unsupported and casts doubt on whether all rows in Tables 1, 3, and 4 were produced under the same protocol. Please correct the row or provide the confusion matrix, class distribution, and metric definition that make the reported pair possible.","section":"Ablation Study, Table 3"},{"comment":"The paper reports no train/validation/test split, no number of random seeds or runs, no standard deviations or significance tests, and no code. The baseline numbers in Table 1 may come from prior papers under different protocols, and the text does not state whether the baselines were retrained with the same preprocessing, pretraining, and split as MSPN. The central claim that 'MSPN outperforms all baselines' cannot be evaluated without these details. The authors should release code, data splits, seeds, and metric definitions, or at minimum provide the exact protocol and per-run results.","section":"Experiment Setup; Tables 1, 3, 4"},{"comment":"The TumEmo result (81.81% accuracy) is presented without any baselines or comparison, and the relationship between TumEmo 'pretraining' and the MVSA results is unclear: it is not stated whether the MSPN rows in Tables 1, 3, and 4 were initialized with TumEmo pretraining, nor whether the comparison methods had access to the same pretraining. If only MSPN is pretrained on TumEmo, the reported gains may be due to additional data rather than the proposed architecture. The transfer protocol must be described explicitly.","section":"Table 2 and Experiment Setup"},{"comment":"The empathetic interaction capability of AIVA is supported only by two screenshots in Figure 8; there is no human evaluation, automatic empathy/emotion-consistency metric, or comparison with a text-only LLM baseline. The abstract and conclusion claim that AIVA enables 'emotionally aligned and animated HCI,' but this claim is not substantiated by any measurement. At minimum, the authors should report a small user study or an automatic metric on response emotion alignment, and should temper the claims to match the evidence provided.","section":"Qualitative Results of AIVA; Conclusion"}],"minor_comments":[{"comment":"The sentence 'we evaluate the proposed MSPN on on sev...' contains a duplicated preposition and an incomplete word; the paragraph should be rewritten.","section":"Experiment Setup"},{"comment":"Equation (1) has an extra closing parenthesis in the first attention formula and does not define the projection matrices or d_k; please clarify the notation and the resulting token dimensions in Z_i^0.","section":"Eq. (1)"},{"comment":"The positive set P(i) and the anchor set A(i) in the supervised contrastive losses are not defined; the text should state that positives are samples/prototypes of the same sentiment class and whether the anchor itself is excluded.","section":"Eqs. (7)-(9)"},{"comment":"Figure 3 is not referenced in the main text; either cite it where the MSPN architecture is introduced or remove it.","section":"Figure 3"},{"comment":"The conclusion refers to 'human-robot interactions' while the rest of the paper discusses human-computer interaction; the terminology should be made consistent.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The Table 3 anomaly is the most serious issue and should be investigated before any further consideration. The manuscript is close to a systems/position paper, but the empirical validation is currently too thin for a full archival paper: no artifacts, no human evaluation of the claimed empathetic interaction, and no protocol details. If the authors can correct the anomalous row, release code and splits, and add a small response-quality evaluation, the contribution could become acceptable; as it stands, the central empirical claims are not independently verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere is the quick take: the paper has a coherent, mildly novel architecture (MSPN) wrapped inside a broader AIVA companion framework, and the writing is clear. But the empirical core is not trustworthy as presented. The Table 3 ablation row 'w/o CAF' on MVSA-Multi reports 18.70% accuracy with 67.41% F1; that pair cannot happen for a three-class problem. The reader's stress-test is correct: with accuracy that low, F1 is far below 0.67 under any reasonable averaging. One of the two numbers is wrong, and since the same protocol produced all tables, the headline 'MSPN outperforms all baselines' is now in question.\n\nWhat's actually new: combining cross-attention fusion, a fusion transformer with learnable sentiment prototypes, and bidirectional supervised contrastive loss into one sentiment classifier. That specific combination is not in the cited baselines. The EPE prompt template and GPT-SoVITS/Live2D integration is a reasonable practical framework, though it's not evaluated quantitatively; we only get two qualitative examples. The writing is clear and the related work is adequate.\n\nSoft spots, in order of severity:\n- The impossible ablation row. This is load-bearing. The authors need to correct it or explain it, and release code/data/splits/seeds so the rest of the numbers can be checked.\n- No error bars or repeated runs anywhere; the single-run table is thin evidence for a 0.5-1 point win over MGNNS.\n- The AIVA interaction claims are only illustrated, not measured. No user study, no automated empathy metric.\n- The TumEmo table has no baselines, only MSPN, so it's a sanity check rather than a comparison.\n\nMy verdict: the architecture deserves a look, but the paper in its current form is not ready. The central empirical claim depends entirely on tables that include at least one impossible number. This is exactly the kind of paper that should go to peer review with a request for artifacts and a corrected ablation table; a serious referee could get it into shape. I would not cite it until the numbers are fixed.\n\nRecommendation: send it to review, but make the authors reproduce the ablation and release code before acceptance.","headline":"The architecture is a reasonable incremental combination, but the impossible ablation row (18.70% accuracy / 67.41% F1) makes the empirical claims unreliable until artifacts and corrections appear.","tokens_in":10911,"tokens_out":7239,"would_cite":false,"duration_ms":61116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AIVA argues that giving an LLM a multimodal sentiment-perception front end lets a virtual companion answer with emotionally aligned language, speech, and animated expressions.","keywords":["multimodal sentiment analysis","virtual companion","large language models","cross-modal fusion transformer","supervised contrastive learning","emotion-aware prompt engineering","text-to-speech","animated avatar"],"falsifier":"Re-running MSPN and the reported baselines on MVSA-Single and MVSA-Multi with the same data splits would settle the comparison; one concrete check is the ablation row 'w/o CAF' on MVSA-Multi, which lists 18.70% accuracy with 67.41% F1, a combination that is implausible for a three-way classification and suggests a data or protocol error that either the paper or a replication would have to explain.","tokens_in":9898,"feed_emoji":"🤖","tokens_out":3948,"duration_ms":33584,"temperature":0.7,"pith_summary":"The paper argues that large language models, left to text alone, miss the emotional content carried by faces and voices, and that a companion agent can recover that content by adding a dedicated multimodal sentiment-perception front end. It proposes AIVA, built around the Multimodal Sentiment Perception Network (MSPN), which fuses image and text with cross-attention, a fusion transformer, and prototype-level supervised contrastive learning. The detected sentiment is inserted into the LLM prompt, and the reply is spoken by a TTS system and acted out by an animated avatar. If the reported results hold, the framework offers a working template for emotion-aware agents in companion robotics, social care, and mental-health support.","feed_headline":"Emotion-aware AI companion beats prior multimodal baselines","feed_subtitle":"MSPN fuses text and faces with contrastive prototypes, feeds sentiment into an LLM prompt, and replies via voice and avatar.","key_machinery":"The load-bearing mechanism is the Multimodal Sentiment Perception Network (MSPN), specifically its cross-modal fusion transformer with learnable sentiment prototypes. The prototypes act as category queries that repeatedly attend to the fused image-text token sequence; classification reads from the final prototype layer, and a bidirectional supervised contrastive loss forces prototypes and multimodal representations to share a cluster structure. This is what converts raw pixels and words into the sentiment label that the Emotion-aware Prompt Engineering step hands to the LLM.","core_discovery":"The central claim is that multimodal sentiment classification and LLM-driven dialogue can be joined in one agent without retraining the LLM. MSPN encodes images and text with a Vision Transformer and BERT, lets each modality attend to the other through cross-attention, then passes the combined tokens through a cross-modal fusion transformer whose learnable sentiment prototypes extract category-level emotion signals. A supervised contrastive loss aligns prototypes and multimodal representations in both directions, and the model is pretrained on TumEmo and fine-tuned on the MVSA datasets. On those benchmarks the paper reports MSPN surpassing all compared baselines, with 74.25% accuracy and 72.84% F1 on MVSA-Single and 73.48% accuracy and 70.01% F1 on MVSA-Multi, plus 81.81% accuracy on TumEmo. The paper's contribution is therefore a complete pipeline from raw multimodal input to empathetic response, with the sentiment module doing the perception work that the LLM alone cannot.","pith_inferences":["A natural next test would be to ablate the sentiment prefix itself, comparing the full AIVA against an LLM prompted without any emotional label, and to measure human-rated empathy rather than classification accuracy.","The prototype-contrastive idea is not restricted to still images and text; the same machinery could ingest audio prosody or gesture features and feed continuous valence-arousal values instead of discrete labels.","Because the sentiment cue is a discrete category, the agent's empathy is bounded by the emotion taxonomy; subtle or mixed emotional states may require finer-grained or continuous representations.","The system's perception module and generation module are loosely coupled, so failures in emotion recognition would degrade the prompt cue but not the LLM itself, making the framework easy to upgrade as better sentiment models appear."],"forward_implications":["Emotion-aware agents can be built on top of existing LLMs by prepending a sentiment cue to the prompt rather than retraining the dialogue model.","Multimodal sentiment classification improves when sentiment prototypes are contrastively aligned with fused representations, as the reported gains over MGNNS and other baselines indicate.","Removing any component of MSPN, whether cross-attention fusion, the fusion transformer, or supervised contrastive learning, lowers accuracy and F1, suggesting each part contributes to the result.","Expressive output matters to the claimed effect: TTS and the animated avatar give the generated empathy a visible and audible channel, not just a textual one.","Pretraining on a large image-text emotion corpus and then fine-tuning on smaller labeled sets is the protocol that yields the best reported numbers."],"supporting_citations":[{"why":"Supplies the MVSA-Single and MVSA-Multi datasets used for fine-tuning and evaluation.","marker":"Niu et al. 2016"},{"why":"Supplies the TumEmo pretraining corpus and the OSDA baseline.","marker":"Yang et al. 2020"},{"why":"Supplies MGNNS, the strongest prior multimodal baseline that MSPN is compared against.","marker":"Yang et al. 2021"},{"why":"Supplies the Co-MN-Hop6 baseline for MVSA comparison.","marker":"Xu, Mao, and Chen 2018"},{"why":"Supplies the MultiSentNet baseline for MVSA comparison.","marker":"Xu and Mao 2017"},{"why":"Supplies the HSAN baseline for MVSA comparison.","marker":"Xu 2017"},{"why":"Provides LLaMA2-Chat, the pretrained LLM used inside AIVA for response generation.","marker":"Touvron et al. 2023"},{"why":"Provides GPT-SoVITS, the TTS system used to convert LLM output into expressive speech.","marker":"RVC-Boss 2025"}],"fun_headline_variants":["AI companion reads faces and text for empathetic replies","Multimodal sentiment boosts LLM empathy in virtual companion","Emotion-sensing AI avatar responds with voice and empathy","Cross-modal transformer gives LLMs emotional context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported experimental numbers are accurate and were produced under a consistent evaluation protocol, since the paper's empirical claims about MSPN rest entirely on the tables.","fun_headline_variants_meta":{"raw":{"variants":["AI companion reads faces and text for empathetic replies","Multimodal sentiment boosts LLM empathy in virtual companion","Emotion-sensing AI avatar responds with voice and empathy","Cross-modal transformer gives LLMs emotional context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1633,"prompt_tokens":913,"completion_tokens":720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":658}},"tokens_in":529,"tokens_out":720,"duration_ms":7225,"temperature":1.0,"reasoning_tokens":658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:32:30.226524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running MSPN and the reported baselines on MVSA-Single and MVSA-Multi with the same data splits would settle the comparison; one concrete check is the ablation row 'w/o CAF' on MVSA-Multi, which lists 18.70% accuracy with 67.41% F1, a combination that is implausible for a three-way classification and suggests a data or protocol error that either the paper or a replication would have to explain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TumEmo pretraining corpus and the OSDA baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MGNNS, the strongest prior multimodal baseline that MSPN is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides GPT-SoVITS, the TTS system used to convert LLM output into expressive speech."}],"review_version":1}