{"id":"4a5a38dc-29fe-458d-9129-df610082f295","arxiv_id":"2412.04908","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MERCI is a 30-participant multimodal human-robot conversation dataset that pairs personal profiles and emotion labels with video, audio, and text records.","lead":"The authors built a 30-participant dataset of real conversations between people and a robot, with video, audio, text, personal profiles, and emotion labels. It is one of the few public resources designed to help AI assistants hold personalized and emotionally aware conversations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dataset's emotion labels are produced by unvalidated automatic modules (MobileNetV2, VADER) with no reported accuracy or human verification; the claimed emotionally annotated multimodal dataset is not yet substantiated.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the facial emotion recognition and sentiment analysis modules are assumed accurate without any validation, and this assumption enters at Sections III.C and III.D and underlies the dataset's emotional labels in Section V.B. My read confirms this is the most consequential risk to the paper's central claim. The paper's other weaknesses, such as GPT-4 serving as both generator and evaluator and the absence of accessible data, are real but secondary: even if GPT-4 evaluation is biased, the user evaluation provides independent evidence of positive interaction quality, and the raw multimodal recordings would remain useful even if the automatic emotion labels were imperfect. The internal count inconsistency (911 vs. 900 emotions) strengthens the case that the annotation pipeline needs external verification. Because the underlying data may still be genuine and valuable, and because the missing validation is addressable through a targeted annotation study, the appropriate verdict remains conditional rather than accept or reject. No ad hominem is intended; the concern is about the evidence provided for the emotion-labeling claim, not about the authors' conduct.","tokens_in":11605,"tokens_out":3157,"duration_ms":36343,"concrete_test":"Randomly sample 100 recorded segments from the MERCI dataset, have at least three independent annotators label the participant's emotion using the paper's category set, and compute Cohen's kappa and accuracy against the MobileNetV2/VADER labels. If kappa is below 0.6 or accuracy below 70%, the automatic emotion annotation is not reliable enough to support the dataset's emotional annotation claim, and the affected labels should be re-annotated or explicitly marked as unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claimed contribution is a dataset annotated with emotional states from real human-robot interactions. Those annotations are generated in real time by the MobileNetV2 facial expression module (Section III.D) and VADER sentiment analysis (Section III.C), but the paper reports no training data, no validation accuracy, and no human verification for either module. VADER outputs valence (positive/negative/neutral) rather than discrete emotion categories, so its relationship to the emotion labels in Figure 5 is unspecified. The reported emotion distribution also fails an internal consistency check: the counts given (489 + 245 + 110 + 44 + 21 + 2 = 911) do not match the stated total of 900 emotions. Since GPT-4's responses are conditioned on these unverified emotional states, and since the dataset's novelty rests on emotional annotation, the emotion labels must be validated before the central claim is fully supported. The raw video, audio, transcripts, and profile questionnaires likely remain valuable, which is why this concern supports a conditional disposition rather than rejection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MERCI, a multimodal dataset of human-robot conversations collected from 30 participants interacting with the ARI robot through the PERCY system. PERCY combines speech-to-text (Whisper), GPT-4-based response generation conditioned on participant profiles and real-time emotion estimates, and text-to-speech. The dataset includes video, audio, transcripts, questionnaire profiles, sentiment labels, and emotion labels. The authors report automatic and user evaluations claiming high naturalness, engagement, fluency, consistency, relevance, and empathy. The paper argues this addresses a gap in open-domain HRI dialogue datasets with genuine interactions and emotional/personal annotations.","tokens_in":11789,"tokens_out":3742,"duration_ms":37421,"significance":"If validated, MERCI would provide a useful resource: real human-robot interaction data with personal profiles, multimodal recordings, and emotional annotations. The user study with 30 participants and standard HRI instruments (NASA TLX, NARS) is a concrete strength, and the promise of public code/data enables reproducibility. However, the central novelty—the emotional annotation—currently rests on unvalidated automatic modules, and the quality claims are supported by evaluations that lack baselines or a fully described protocol. The raw recordings, transcripts, and profiles likely retain value even if the emotional labels need re-annotation or validation.","major_comments":[{"comment":"The emotion labels—a key claimed contribution—are produced by a MobileNetV2-based facial expression recognizer and VADER sentiment analysis with no reported validation accuracy, training data, or human verification. VADER outputs valence (positive/negative/neutral) rather than the discrete emotion categories shown in Figure 5, and the mapping between the two is unspecified. Since GPT-4 responses are conditioned on these labels, the reliability of the emotion annotations directly affects the quality of the entire dataset. Please provide validation evidence, such as agreement with human annotators on a subset, benchmark accuracy of the emotion recognition model, or a description of the model's training data and source.","section":"III.D and III.C"},{"comment":"The emotion distribution is internally inconsistent. The counts listed (489 + 245 + 110 + 44 + 21 + 2 = 911) do not sum to the stated total of 900 emotions, and the percentages in Figure 5 (e.g., 53.7% of 900 = 483.3, not 489) do not match the text counts. Please correct these numbers and present exact counts in a table, as this is a direct factual issue in the dataset description.","section":"V.B"},{"comment":"The automatic evaluation metrics (perplexity 24.53, cosine similarity 0.32, MAUVE 0.98) are reported without baselines or comparisons to other dialogue systems, making them uninterpretable as evidence of 'high quality.' In particular, MAUVE requires an explicit reference distribution of human-like responses, which the paper does not specify. Please provide baseline comparisons (e.g., against other open-domain dialogue models) or clarify the reference distributions used.","section":"V.D"},{"comment":"Using GPT-4 to evaluate responses generated by the same GPT-4-based system raises a self-referential concern. The user study offers partial independent grounding, but the GPT-4 evaluation protocol—prompt text, number of samples evaluated, and how scores were aggregated—is not described, and no quantitative agreement with user ratings is reported. Please provide the evaluation details and an inter-rater or correlation analysis between GPT-4 and user evaluations.","section":"V.D.2"}],"minor_comments":[{"comment":"'To add a systemic gap' should read 'To address a systemic gap.'","section":"Abstract and I"},{"comment":"In the example dialogue, 'netural' is a typo for 'neutral.'","section":"IV.B.4"},{"comment":"The captions read 'Video Chanel' and should be 'Video Channel.'","section":"Figure 3"},{"comment":"'Multi-model information' should be 'multimodal information' in the description of MELD.","section":"II.B"},{"comment":"The MobileNetV2 backbone is not cited; please add a proper reference for the architecture.","section":"III.D"},{"comment":"Calling a 30-participant, 1860-utterance dataset 'large-scale' is an overstatement relative to existing dialogue corpora (e.g., DailyDialog, MELD); consider qualifying the claim or comparing with HRI-specific datasets.","section":"V.B"}],"recommendation":"major_revision","confidential_remarks":"The dataset and system have merit, but the evaluation and validation gaps are substantial. The most pressing issue is the unvalidated emotion annotation pipeline, which is load-bearing for the central claim. The self-referential GPT-4 evaluation is a secondary concern; the user study mitigates it, but adding protocol details would strengthen the paper. The internal inconsistency in the emotion counts also needs correction. I would be willing to look at a revised version that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MERCI is a genuine attempt at something the field lacks — a multimodal HRI dataset with personal profiles and emotional states from real robot conversations — but the emotion annotations, which are the selling point, are not yet substantiated. The paper is worth engaging with, not rejecting, and the main fixes are straightforward.\n\nWhat's new: they recorded 30 people talking with an ARI robot, capturing three video channels, one audio channel, transcripts, questionnaire profiles, and ran the whole pipeline through GPT-4 conditioned on sentiment and facial-expression outputs. The combination of real embodied interaction, personalization, and multimodal capture is not present in DailyDialog or MELD, so it fills a gap. They also ran a user study with NASA-TLX and NARS, which gives the interaction-quality claims some independent grounding beyond the model's self-assessment. Credit where due: the limitations section is honest about the small participant pool and volunteer recruitment.\n\nSoft spots, in order of real weight. First, the emotion labels come from a MobileNetV2 facial expression model and VADER, with no training details, no accuracy, and no human verification. Since the dataset's novelty is precisely those emotional annotations, this is load-bearing. Second, the counts in Section V.B don't add up: 489+245+110+44+21+2=911, not the stated 900. The percentages match 911, so the total is likely a typo, but it needs fixing. Third, GPT-4 both generates the responses and evaluates them. The user study partially mitigates this, but the automatic evaluation would be stronger with baselines — perplexity of 24.5 and cosine 0.32 mean little without comparison. MAUVE 0.98 is also suspiciously high; report how it was computed.\n\nThese are all fixable. The raw video, audio, and transcripts are probably the most durable part of the dataset; the emotional labels should be re-annotated or clearly marked as unverified.\n\nBottom line: the paper deserves a serious referee. It is a plausible new resource for personalized empathetic HRI, and the methodological gaps are correctable, not fatal. I would not cite the emotion labels as ground truth until they are validated, but I would bring the paper to a reading group to discuss evaluation practice in dataset papers.","headline":"MERCI is a useful new HRI dataset that needs its emotion-label validation sorted out before the central claim can be trusted.","tokens_in":12288,"tokens_out":3066,"would_cite":false,"duration_ms":29872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MERCI, a multimodal dataset of genuine human-robot conversations from 30 participants, together with PERCY, a GPT-4-driven robot that uses facial expressions and sentiment to personalize empathetic responses.","keywords":["human-robot interaction","multimodal dataset","personalized dialogue","empathetic conversational agents","facial emotion recognition","sentiment analysis","GPT-4","social robotics"],"falsifier":"Take a random sample of roughly 100 utterances from MERCI, have several independent annotators watch the video and audio and choose the same emotion categories the pipeline uses, and measure agreement with the dataset's automated labels; if agreement is near chance levels, the claim that MERCI contains genuine, accurate emotional annotation is falsified.","tokens_in":11437,"feed_emoji":"🤖","tokens_out":9009,"duration_ms":86594,"temperature":0.7,"pith_summary":"This paper is trying to establish that real, open-ended human-robot conversations can be captured together with both personal profiles and emotional signals, and that a GPT-4-driven robot can use those signals to hold natural, empathetic dialogue. The contribution is MERCI, a multimodal dataset recorded from 30 participants talking with the ARI robot run by a system called PERCY. PERCY hears the user with Whisper, reads the face with MobileNetV2, scores sentiment with VADER, and feeds all of that plus the participant's questionnaire profile into GPT-4 to generate the next spoken response. Automatic checks and participant questionnaires rate the resulting dialogue high on naturalness, engagement, fluency, relevance, consistency, and empathy. If those ratings hold, MERCI gives the community a resource for studying and building personalized, emotionally aware human-robot interaction.","feed_headline":"MERCI dataset records 30 real robot chats with emotion labels","feed_subtitle":"User ratings call the exchanges natural, fluent, and empathetic; dataset and system are public.","key_machinery":"The machinery is the PERCY pipeline plus the MERCI corpus it produces. PERCY is a five-module loop on the ARI robot: Whisper converts speech to text; the text goes to both a VADER sentiment analyzer and the GPT-4 response generator; a MobileNetV2-based model classifies the participant's facial expression from the robot's head camera; GPT-4 receives the transcript, sentiment label, emotion label, and the participant's questionnaire profile and generates the next response; the robot speaks it through text-to-speech. That loop creates the dataset's two distinctive ingredients—personal information and emotion state—aligned in time with the conversation. MERCI packages the result as CSV and JSON files containing profiles, chat history, sentiment and emotion labels, and timestamps, together with raw video from the robot head camera and two external cameras, plus a single audio channel.","core_discovery":"The paper's central claim is that a robot can sustain personalized, emotionally aware conversation using only real-time signals and a pre-collected profile, and that the recordings of those conversations form a reusable multimodal dataset. 'Genuine interactions' matter to the authors: participants volunteered personal information and displayed real emotions, unlike scripted dialogues or TV-derived corpora. The dataset contains 30 participants, an average of 20.27 minutes and 62 utterances per conversation (1,860 utterances total), and 900 emotion labels, with neutral at 53.7 percent and happy at 26.9 percent, plus three video channels and one audio channel per session. Evaluations reported in the paper put automatic readability at perplexity 24.53, cosine similarity 0.32, and MAUVE 0.98, and both GPT-4 and user ratings near 4.5 to 4.7 on a five-point scale across naturalness, engagement, fluency, relevance, consistency, and empathy.","pith_inferences":["Beyond the paper: a control condition without the emotion and profile modules would tell whether the high ratings come from GPT-4's general conversational skill or from the personalization itself.","Beyond the paper: the participant sample is young, mostly students, and mostly Asian, so the comfort and empathy results may not transfer to older or more diverse users without further data collection.","Beyond the paper: because the emotion labels come from automated tools with no human verification, the dataset's value as ground truth for emotion research depends on a future validation study; until then, the labels are best read as system outputs rather than ground truth."],"forward_implications":["If these results hold, MERCI gives conversational AI researchers a training and evaluation resource built on real human-robot dialogue rather than scripted or TV-derived exchanges.","The synchronized video, audio, text, profiles, and emotion labels enable multimodal models that connect what people say with how they look and sound while saying it.","PERCY's modular design—Whisper for speech, VADER for sentiment, MobileNetV2 for facial emotion, GPT-4 for generation, and the robot's text-to-speech—offers a reproducible architecture for emotionally aware social robots.","The reported numbers (perplexity 24.53, cosine similarity 0.32, MAUVE 0.98, and ratings near 4.5–4.7) give future personalized dialogue systems a concrete baseline to beat."],"supporting_citations":[{"why":"This citation supplies the closest existing work, a multimodal emotional dialogue dataset taken from a TV series, which MERCI extends by using real human-robot interactions.","marker":"[19]"},{"why":"This citation provides the large open-domain dialogue dataset that, unlike MERCI, lacks video, audio, and profile information.","marker":"[18]"},{"why":"This citation documents the ARI robot platform whose cameras and microphone define the multimodal sensors used to collect the dataset.","marker":"[5]"},{"why":"This citation is the speech-to-text service that generates the transcripts feeding both the sentiment analysis and the response generator.","marker":"[22]"},{"why":"This citation is the NLTK/VADER sentiment analysis tool that produces one of the two emotion signals used to condition responses and label utterances.","marker":"[23]"},{"why":"This citation supplies the evidence that personal factors such as interests and preferences improve human-robot interaction, motivating the questionnaire-based profiles.","marker":"[25]"},{"why":"This citation establishes the GPT-4-as-evaluator method used to score naturalness, engagement, fluency, relevance, consistency, and empathy.","marker":"[39]"}],"fun_headline_variants":["MERCI: 30 real robot chats with emotional and personal depth","Emotion-aware robot dialogue dataset from genuine interactions","New MERCI dataset captures robot empathy in 30 real talks","Personalized robot conversations with emotion labels go public"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the automatic face and sentiment classifiers correctly label how participants felt, but reports no accuracy check or human verification of those labels, and if the labels are wrong, the dataset's main novelty as an emotionally annotated resource is not established.","fun_headline_variants_meta":{"raw":{"variants":["MERCI: 30 real robot chats with emotional and personal depth","Emotion-aware robot dialogue dataset from genuine interactions","New MERCI dataset captures robot empathy in 30 real talks","Personalized robot conversations with emotion labels go public"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1538,"prompt_tokens":931,"completion_tokens":607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":541}},"tokens_in":547,"tokens_out":607,"duration_ms":7123,"temperature":1.0,"reasoning_tokens":541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:08:19.054883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of roughly 100 utterances from MERCI, have several independent annotators watch the video and audio and choose the same emotion categories the pipeline uses, and measure agreement with the dataset's automated labels; if agreement is near chance levels, the claim that MERCI contains genuine, accurate emotional annotation is falsified.","supporting_citations":[{"cited_title":"Using ARI: Unboxing and getting started,","cited_arxiv_id":null,"evidence_quote":"This citation documents the ARI robot platform whose cameras and microphone define the multimodal sensors used to collect the dataset."},{"cited_title":"Whisper: Openai’s speech recognition model,","cited_arxiv_id":null,"evidence_quote":"This citation is the speech-to-text service that generates the transcripts feeding both the sentiment analysis and the response generator."},{"cited_title":"Natural language toolkit,","cited_arxiv_id":null,"evidence_quote":"This citation is the NLTK/VADER sentiment analysis tool that produces one of the two emotion signals used to condition responses and label utterances."},{"cited_title":"Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation,","cited_arxiv_id":null,"evidence_quote":"This citation establishes the GPT-4-as-evaluator method used to score naturalness, engagement, fluency, relevance, consistency, and empathy."}],"review_version":1}