{"id":"526d3032-57a0-4130-9772-41f3b377cb3d","arxiv_id":"2501.01384","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ShareChatX and OmniChat show that large-scale synthetic spoken dialogue data improves multi-turn response quality and emotion prediction, setting a new state of the art on DailyTalk.","lead":"This paper introduces a large synthetic spoken dialogue dataset spanning emotional, audio-event, and music scenarios, and a dialogue model trained on it. The authors report state-of-the-art results on a real-world dialogue benchmark, suggesting synthetic data can help spoken dialogue systems handle complex scenarios.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim on DailyTalk is confounded: OmniChat+Real Data is fine-tuned on the target set while baselines are used zero-shot; fair baseline fine-tuning is required before synthetic-data gains can be assessed.","rationale":"The reader's weakest_assumption focuses on whether synthetic ShareChatX distributions transfer to real dialogue; my concern is a more immediate confound in the evaluation protocol. A fair baseline check is a precondition for any transfer claim: if strong baselines are also fine-tuned on DailyTalk, the reported SOTA margin may shrink or disappear, and the specific contribution of synthetic data would remain unknown. The reader did flag evaluation fairness in the rationale (baseline fine-tuning, statistical significance), but did not name it as the weakest assumption, hence 'partial'. My concern does not overturn the paper's plausibility or its useful empirical findings about synthetic-data sampling ratios; instead, it strengthens the existing CONDITIONAL verdict by adding a specific, testable condition. The paper also has independent support worth acknowledging: a large released dataset is planned, WER-based verification was applied to synthetic audio, and the architecture ablations (Mix-Former, data-scale curves, sampling-ratio experiments) are internally informative. Those strengths do not resolve the fairness issue, but they justify a conditional rather than a rejection stance.","tokens_in":16650,"tokens_out":3742,"duration_ms":36260,"concrete_test":"Fine-tune the strongest baselines (e.g., Qwen2-Audio and FunAudioLLM) on the same DailyTalk training split used for OmniChat+Real Data, with the same number of steps, LoRA configuration, and checkpoint selection, then evaluate on the same 220-sample test set. For isolation, also train an OmniChat variant on DailyTalk real data only (alpha=0.0 in the Section 5.4 mixing scheme) with the same budget. Run each configuration with 3 seeds and report mean +/- std plus bootstrap confidence intervals for BLEU, METEOR, BERTScore, and F1e. If fine-tuned baselines match or exceed OmniChat+Real Data, or if the DailyTalk-only OmniChat variant matches the +Real Data result, the synthetic-data advantage is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2's headline comparison is not controlled for training distribution. OmniChat + Real Data is explicitly fine-tuned on DailyTalk: Appendix C.1 describes sampling real DailyTalk data at rate (1-alpha) in the mixed training, and evaluation uses a 220-sample DailyTalk test set. In contrast, the listed baselines (Qwen2-Audio, FunAudioLLM, StyleTalk, etc.) are used as released pre-trained models, with no reported fine-tuning on DailyTalk. The claimed SOTA therefore conflates two effects: access to target-domain real data and the value of synthetic data. The fairness concern is visible even within Table 2: OmniChat without real data trails Qwen2-Audio on METEOR (12.57 vs 13.89) and GPT-eval (3.96 vs 4.01), so the 'surpassing' result is not uniform and depends on the real-data fine-tuning step. No baseline is given the same fine-tuning treatment, and no ablation trains OmniChat on DailyTalk alone without synthetic data, so the marginal contribution of ShareChatX cannot be isolated. This is load-bearing because the paper's central claim is precisely that synthetic data enables state-of-the-art performance on real-world dialogue; if the margin comes from the real-data fine-tuning term, the synthetic-data contribution is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces ShareChatX, a large-scale synthetically generated multi-turn spoken dialogue dataset with three subsets (emotion, audio, and music), and OmniChat, an LLM-based spoken dialogue system that fuses Whisper, emotion2vec, and BEATs features through a windowed Q-Former-based Mix-Former module. The authors report experiments on DailyTalk and on ShareChatX itself, claim state-of-the-art results on DailyTalk, and propose an optimal 20% synthetic-data sampling ratio (alpha=0.2) in mixed real/synthetic training.","tokens_in":17074,"tokens_out":7067,"duration_ms":65883,"significance":"The resource and the model are potentially valuable: a large, scenario-diverse synthetic spoken dialogue dataset could mitigate the scarcity and privacy issues of real spoken dialogue corpora, and the controlled alpha sweep is a sensible way to study synthetic/real data mixing. The paper also makes concrete architectural choices (frozen audio encoders and LLM, LoRA, windowed Q-Former) and promises release of data and code. However, the headline state-of-the-art claim currently rests on an uncontrolled comparison, and the complex-scenario results are evaluated in-distribution. These issues need to be addressed before the contribution can be accepted at its current strength.","major_comments":[{"comment":"The claim of state-of-the-art performance on DailyTalk is not yet supported because OmniChat+Real Data is trained on DailyTalk (real samples are drawn with probability 1-alpha in the mixed training described in Appendix C.1), while all comparison systems in Table 2 are evaluated as released pre-trained models without fine-tuning. This comparison conflates target-domain fine-tuning with the value of synthetic data. Please fine-tune at least one strong baseline (e.g., Qwen2-Audio or FunAudioLLM) on the same DailyTalk data with the same training recipe, and also include an OmniChat model trained on real DailyTalk only (alpha=1.0) in Table 2. The sentence 'OmniChat demonstrated superior performance across all content-related metrics' is also contradicted by the table for OmniChat without real data: METEOR is 12.57 vs 13.89 for Qwen2-Audio, ROUGE-L is 12.63 vs 12.81, and GPT-eval is 3.96 vs 4.01.","section":"Section 5.2, Table 2; Appendix C.1"},{"comment":"The ShareChatX evaluations are in-distribution: the test dialogues are generated by the same GPT-4/CosyVoice pipeline used to construct the training data, and the baselines are not fine-tuned on ShareChatX. The large margins in Table 3 therefore partly reflect distribution matching rather than generalizable complex-scenario ability. To support the claim that synthetic data improves dialogue in diverse complex scenarios, please either fine-tune the baselines on the ShareChatX training splits, evaluate on an external real-world audio/music dialogue benchmark, or add human evaluation with objective audio-grounding checks.","section":"Section 5.2, Table 3; Appendix C.2"},{"comment":"No error bars, multiple runs, or significance tests are reported. The DailyTalk test set contains only 220 samples, and several headline gaps are small (e.g., BERTScore 86.99 vs 86.14; GPT-eval 4.15 vs 4.01). The alpha=0.2 optimum in Table 4 and the scaling conclusions in Figure 3 are likewise stated without variance estimates, so it is unclear whether the observed differences are meaningful. Please add at least three random seeds with confidence intervals and appropriate statistical tests, such as bootstrap or paired tests on the test set.","section":"Tables 2-5 and Figure 3"}],"minor_comments":[{"comment":"The dimension specified for Hs_i, He_i, Hb_i is unclear: the notation R^{[N x K/L] x D} appears to mix input expert dimension D with the Q-Former hidden dimension, and the bracketed exponent notation is ambiguous. Please clarify whether the output has N*K/L vectors or (N/L)*K vectors and specify the hidden dimension explicitly.","section":"Section 4.1, Eq. (1)"},{"comment":"The sentence 'the BLEU score of the direct speech model improves from 4.65 at 5K to 4.94 at 10K, while the text-based model lags behind with a BLEU score of 4.86' is confusing because 4.65 is below 4.86; please state which data scale and which comparison support the conclusion that speech models surpass text models at 5K-10K.","section":"Section 5.3"},{"comment":"The model is referred to as 'Beat', but the cited work is BEATs (Chen et al., 2023); please use the correct model name and ensure the reference entry is complete.","section":"Section 4.1 and references"},{"comment":"The limitation section spells the model name as 'Omnichat' while the rest of the paper uses 'OmniChat'; please make the spelling consistent.","section":"Appendix A"},{"comment":"The GPT-eval description does not report the exact model version, decoding temperature, or number of evaluation passes, which limits reproducibility; please provide these details.","section":"Appendix C.3"},{"comment":"The paper does not state whether the ASR-based baselines use reference transcriptions or predicted transcriptions on the DailyTalk test set; this should be specified because it affects the comparison between ASR-based and direct systems.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a useful resource and a reasonable training recipe, but the SOTA claim needs a controlled fine-tuning comparison and statistical grounding. I see no ethical concerns with the synthetic-data approach itself, though the 'state-of-the-art' wording should be moderated until baselines are given the same fine-tuning treatment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe ShareChatX dataset is the most useful thing here: 128K synthetic dialogues spanning emotion, audio events, and music, generated with GPT-4 and CosyVoice, with diarization and WER checks. That is a concrete contribution that goes beyond StyleTalk and AF-Dialogue. The OmniChat architecture is a modest extension of Q-Former with per-expert weighting, but the scaling curves (5.3) and the synthetic/real mixing-ratio study (5.4) are genuinely informative empirical results.\n\nThe paper also deserves credit for testing on an external real-world dataset, DailyTalk, rather than only on its own synthetic splits. The F1e improvement on emotion prediction is consistent with the idea that synthetic data diversifies emotional expression.\n\nThe soft spots are in the experimental protocol, and the main one is load-bearing. In Table 2, OmniChat + Real Data is produced by mixed training that includes real DailyTalk data (Appendix C.1), while the baselines—Qwen2-Audio, FunAudioLLM, StyleTalk—are used as released, without fine-tuning on DailyTalk. So the comparison conflates two things: access to target-domain real data and the value of synthetic data. The no-real-data OmniChat row tells the story: it beats Qwen2-Audio on BLEU (3.54 vs 3.03) and BERTScore (86.24 vs 86.14), but trails on METEOR (12.57 vs 13.89) and GPT-eval (3.96 vs 4.01). The claim of state-of-the-art depends on the real-data mixing, and the paper does not isolate the synthetic contribution—there is no ablation that trains OmniChat on DailyTalk alone. That is fixable but needs to be done: fine-tune the baselines on the same data mix, and report a no-synthetic baseline.\n\nThe ShareChatX test results are in-distribution with the synthetic generation pipeline, so they mostly show distribution matching. I would weight the DailyTalk result heavily, which is why the fairness issue matters so much. Also missing: error bars or significance tests on the content metrics; the alpha=0.2 optimum is a single run. And the synthetic data itself is only validated by WER and manual inspection, not by objective emotion or naturalness measures—minor in comparison to the protocol issue.\n\nNet: the dataset is worth having, the DailyTalk benchmark provides a real external checkpoint, and the empirical findings are useful. The central claim is currently over-claimed because of the uncontrolled comparison. This should go to peer review, but with the expectation of a major revision and a mandatory controlled baseline comparison. If the dataset and code ship, I would cite it.","headline":"The dataset is a real resource and the DailyTalk checkpoint is a genuine external result, but the SOTA claim is built on an unfair comparison and needs to be re-run before the synthetic-data story holds.","tokens_in":17463,"tokens_out":3729,"would_cite":false,"duration_ms":35307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large-scale synthetic spoken dialogue data, mixed sparingly with real recordings, lets a direct speech-to-speech model beat ASR-based systems on real-world conversations.","keywords":["spoken dialogue system","synthetic data","multi-turn dialogue","audio-language model","speech emotion recognition","audio event understanding","music understanding","feature fusion"],"falsifier":"Train OmniChat on ShareChatX alone and on the ShareChatX-plus-DailyTalk mix, then have human listeners score response naturalness and emotional appropriateness on held-out real conversations with spontaneous emotion and overlapping background audio; if listeners do not prefer the synthetic-trained model over a model trained only on recordings of equal size, or rate the synthetic corpus's emotional prosody as stereotyped, the transfer claim fails. Concretely, comparing emotion F1 on a spontaneous emotional speech test set against the reported 75.46 on DailyTalk would test whether synthetic emotion coverage transfers.","tokens_in":16503,"feed_emoji":"🎙️","tokens_out":5955,"duration_ms":52907,"temperature":0.7,"pith_summary":"Spoken dialogue systems lag behind text chatbots largely because transcribed, multi-turn, emotionally varied speech data is scarce, costly, and privacy-sensitive. This paper argues that synthetic data can fill that gap at scale: it introduces ShareChatX, a roughly 947,000-dialogue synthetic corpus spanning emotion-heavy talk, audio-event-centred talk, and music-centred talk, generated by an LLM writing scripts and a controllable TTS engine voicing them. On top of the corpus it builds OmniChat, a multi-turn spoken dialogue model whose Mix-Former module weighs separate speech, emotion, and non-speech audio features before they enter a frozen LLM. The paper reports that mixing about one synthetic dialogue in five with real DailyTalk recordings yields the best scores on real DailyTalk, including the best emotion-prediction accuracy among the compared systems, and that speech-only input eventually beats ASR-transcription input as training data grows.","feed_headline":"Synthetic voice data lifts dialogue AI past real-data records","feed_subtitle":"A one-in-five mix of synthetic and real conversations gives the best real-world scores, including emotion accuracy.","key_machinery":"The load-bearing mechanism is the heterogeneous feature fusion module Mix-Former, combined with a synthetic-data training recipe. For each dialogue turn, three frozen expert encoders produce temporally aligned features: Whisper for speech content, emotion2vec for emotional prosody, and BEATs for non-speech audio and music. Mix-Former applies a window-level Q-Former with a single learned query to each expert's features, computes a sigmoid weight per window per expert with linear layers, concatenates the weighted features, and projects them into a frozen Llama-3.1-8B-Instruct LLM trained only through LoRA adapters and the Q-Formers. The trained model predicts both response content and response style, namely the emotion label, which can be fed to a controllable TTS model to synthesize the reply. The dataset-generation pipeline is the other half of the mechanism: GPT-4 writes scenario-specific scripts seeded by AudioCaps and MusicCaps captions, CosyVoice-Instruct synthesizes speech with specified gender, pitch, speed, and emotion, and a diarization-plus-ASR check keeps word error rate under 5 percent.","core_discovery":"OmniChat's central claim is that a spoken dialogue system can be made state-of-the-art on real conversations by pretraining on synthetic speech at scale and then lightly mixing in real data. Trained with synthetic ShareChatX and real DailyTalk at a 1:4 sampling ratio, the model reaches BLEU 4.95, METEOR 14.24, BERTScore 86.99, and emotion F1 75.46 on DailyTalk, surpassing direct audio models such as Qwen2-Audio and ASR-based systems such as FunAudioLLM on content and emotion. On the synthetic ShareChatX test sets, OmniChat outperforms the same baselines in all three scenarios, and ablation studies show that simply concatenating expert features can hurt while the Mix-Former weighted fusion recovers and improves performance. The paper further claims that as dataset size grows from 5K to 80K, direct speech-only input surpasses both text-only and ASR-augmented input, and that pre-training on large general emotion dialogues is needed before fine-tuning on audio or music scenarios.","pith_inferences":["If synthetic scaling transfers as reported, the remaining bottleneck for spoken dialogue is not recording hours but prompt and topic diversity plus TTS controllability, shifting data cost from collection to generation and verification.","The optimal mixing ratio likely depends on how far the synthetic distribution sits from the target real distribution, so for high-stakes or spontaneous domains the paper's alpha of 0.2 would need to be re-estimated rather than assumed.","A testable extension is to apply the same pipeline to lower-resource languages, where real spoken dialogue data is even scarcer and synthetic coverage may matter more.","The paper's quality checks of word error rate, speaker diarization, and manual inspection do not measure whether synthetic emotional prosody sounds genuinely human, so an objective emotion-fidelity metric on synthesized speech would sharpen the transfer claim."],"forward_implications":["A 20 percent synthetic-to-real sampling ratio is the reported sweet spot: lower ratios under-use synthetic diversity, while higher ratios drift from real conversational patterns.","With enough training data, direct speech-input dialogue models overtake pipelines that feed ASR transcriptions, so the recommended design shifts toward end-to-end speech conditioning.","Large-scale synthetic pre-training on general emotional dialogue is a necessary precondition for later fine-tuning on audio-event and music scenarios; small fine-tuning sets alone do not surpass it.","Weighted, query-based fusion of content, emotion, and non-speech features beats naive feature concatenation, especially when music or audio events are present.","ShareChatX provides benchmark splits for emotion, audio, and music dialogue, letting future systems be compared on scenario coverage rather than only on generic text metrics."],"supporting_citations":[{"why":"Supplies the real-world benchmark DailyTalk with its emotion labels, the target dataset OmniChat must beat.","marker":"(Lee et al., 2023)"},{"why":"Establishes the content-plus-style response formulation and the StyleTalk baseline that OmniChat extends.","marker":"(Lin et al., 2024)"},{"why":"Provides CosyVoice-Instruct, the controllable TTS model that turns generated scripts into spoken dialogue.","marker":"(Du et al., 2024)"},{"why":"Supplies GPT-4, used to generate dialogue scripts and to classify audio events as temporary or continuous.","marker":"(OpenAI, 2024a)"},{"why":"Provides AudioCaps captions that seed the audio-event topics in ShareChat-Audio.","marker":"(Kim et al., 2019)"},{"why":"Provides MusicCaps aspect lists that seed the music topics in ShareChat-Music.","marker":"(Agostinelli et al.)"},{"why":"Qwen2-Audio is the strongest direct-audio baseline that OmniChat outperforms on content metrics.","marker":"(Chu et al., 2024)"},{"why":"Whisper supplies the speech content feature extractor and the ASR model used for WER verification.","marker":"(Radford et al., 2023)"},{"why":"emotion2vec supplies the emotional prosody feature extractor used by OmniChat.","marker":"(Ma et al., 2023)"},{"why":"BEATs supplies the non-speech audio and music feature extractor used by OmniChat.","marker":"(Chen et al., 2023)"}],"fun_headline_variants":["Synthetic speech data powers dialogue AI to new highs","Mix 1-in-5 synthetic talks for top dialogue scores","OmniChat: Synthetic data makes spoken AI smarter","Synthetic dialogue data beats real-only for state-of-the-art","Chat AI gets a synthetic boost for real conversations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that synthetic dialogues produced by GPT-4 and CosyVoice faithfully represent real spoken conversation, including emotional fidelity and audio grounding, so that gains measured on ShareChatX transfer to real interactions.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic speech data powers dialogue AI to new highs","Mix 1-in-5 synthetic talks for top dialogue scores","OmniChat: Synthetic data makes spoken AI smarter","Synthetic dialogue data beats real-only for state-of-the-art","Chat AI gets a synthetic boost for real conversations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1709,"prompt_tokens":969,"completion_tokens":740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":661}},"tokens_in":585,"tokens_out":740,"duration_ms":7347,"temperature":1.0,"reasoning_tokens":661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:27:38.421084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train OmniChat on ShareChatX alone and on the ShareChatX-plus-DailyTalk mix, then have human listeners score response naturalness and emotional appropriateness on held-out real conversations with spontaneous emotion and overlapping background audio; if listeners do not prefer the synthetic-trained model over a model trained only on recordings of equal size, or rate the synthetic corpus's emotional prosody as stereotyped, the transfer claim fails. Concretely, comparing emotion F1 on a spontaneous emotional speech test set against the reported 75.46 on DailyTalk would test whether synthetic emotion coverage transfers.","supporting_citations":[{"cited_title":"Dailytalk: Spoken dialogue dataset for conversational text-to-speech","cited_arxiv_id":null,"evidence_quote":"Supplies the real-world benchmark DailyTalk with its emotion labels, the target dataset OmniChat must beat."},{"cited_title":"URL https://aclanthology.org/2024.acl-long.109","cited_arxiv_id":null,"evidence_quote":"Establishes the content-plus-style response formulation and the StyleTalk baseline that OmniChat extends."},{"cited_title":"Audiocaps: Generating captions for audios in the wild","cited_arxiv_id":null,"evidence_quote":"Provides AudioCaps captions that seed the audio-event topics in ShareChat-Audio."},{"cited_title":"End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions","cited_arxiv_id":null,"evidence_quote":"BEATs supplies the non-speech audio and music feature extractor used by OmniChat."}],"review_version":1}