The reviewed record of science sign in
Pith

arxiv: 2203.16502 · v2 · pith:G4LZFRWI · submitted 2022-03-30 · cs.CL · cs.LG· cs.SD· eess.AS

Generative Spoken Dialogue Language Modeling

Tu Anh Nguyen , Eugene Kharitonov , Jade Copet , Yossi Adi , Wei-Ning Hsu , Ali Elkahky , Paden Tomasello , Robin Algayres
show 3 more authors
Benoit Sagot Abdelrahman Mohamed Emmanuel Dupoux
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:G4LZFRWIrecord.jsonopen to challenge →

classification cs.CL cs.LGcs.SDeess.AS
keywords modelspokenableaudiogeneratenaturalisticarchitecturecascaded
0
0 comments X
read the original abstract

We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn-taking compared to a text-based cascaded model.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. High Fidelity Neural Audio Compression

    eess.AS 2022-10 accept novelty 7.0

    EnCodec is an end-to-end trained streaming neural audio codec that uses a single multiscale spectrogram discriminator and a gradient-normalizing loss balancer to achieve higher fidelity than prior methods at the same ...

  2. DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interaction

    cs.HC 2026-06 unverdicted novelty 5.0

    DuplexOmni achieves real-time full-duplex multimodal interaction by separating an interaction layer from a pluggable thinking layer, supported by a Writer-Director pipeline for continuous-interaction training data.