Pith. sign in

REVIEW 5 major objections 5 minor 165 references

Breaking the Barriers of Text-Hungry and Audio-Deficient AI

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that speech can be translated directly from audio, with no text at any step, using a multiscale audio-semantic representation and a fractional diffusion model.

desk verdict A well-motivated but unvalidated proposal: the fractional-diffusion math is mostly sound, but the textless translation claim rests on an untrained linear map and no translation evaluation. read the letter →

arxiv 2506.02443 v1 pith:TKAD7GCW submitted 2025-06-03 cs.SD eess.AS

Hamidou Tembine , Issa Bamia , Massa NDong , Bakary Coulibaly , Oumar Issiaka Traore , Moussa Traore , Moussa Sanogo , Mamadou Eric Sangare
show 26 more authors
Salif Kante Daryl Noupa Yongueng Hafiz Tiomoko Ali Malik Tiomoko Frejus Laleye Boualem Djehiche Wesmanegda Elisee Dipama Idris Baba Saje Hammid Mohammed Ibrahim Moumini Sanogo Marie Coursel Nininahazwe Abdul-Latif Siita Haine Mhlongo Teddy Nelvy Dieu Merci Kouka Mariam Serine Jeridi Mutiyamuogo Parfait Mupenge Lekoueiry Dehah Abdoul Aziz Bio Sidi Bouko Wilfried Franceslas Zokoue Odette Richette Sambila Alina RS Mbango Mady Diagouraga Oumarou Moussa Sanoussi Gizachew Dessalegn Mohamed Lamine Samoura Bintou Laetitia Audrey Coulibaly
This is my paper
classification cs.SDeess.AS MSC 60G2260H1042C4094A1268T50
keywords audio-to-audiotranslationtextlessspeechMultiscaleAudio-SemanticTransformfractionalBrownianmotiondiffusionwavelettonallanguageslow-resourcemean-field-type
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a fully textless, audio-to-audio machine intelligence framework intended for the over 700 million audio-literate people whose languages are unwritten or rarely digitized. Its central claim is that a new representation, the Multiscale Audio-Semantic Transform (MAST), captures the tonal, prosodic, speaker, and expressive content of speech well enough that a learned linear map between wavelet-coefficient spaces can translate meaning from one language to another without any text. To generate the target speech, the paper conditions a fractional diffusion process, driven by fractional Brownian motion, on MAST features and uses an explicit Gaussian score function rather than a trained score network. If correct, the approach would remove the ASR–MT–TTS cascade and make machine translation available directly from recorded speech, even for languages with no writing system. The authors report improved pitch localization and speaker consistency for MAST over Mel spectrograms and other baselines, while acknowledging that textless evaluation metrics are not yet standardized.

What carries the argument

The machinery is the MAST representation plus the fractional diffusion pair. MAST combines a continuous wavelet transform (Morlet or Coiflet) with pitch extraction via dominant wavelet scale, a learned prosody encoder, a pretrained speaker encoder, and an intonation-gesture descriptor, all aligned in time; this tuple is what conditions the generative model. The second piece is a fractional Ornstein–Uhlenbeck-type SDE driven by fractional Brownian motion with Hurst parameter $H \in (0.5, 1)$, whose explicit Gaussian mean and variance make the score function analytic and allow exact endpoint conditioning to a target Gaussian. The wavelet-coefficient mapping in Eq. (8) carries the actual cross-lingual transfer, while the diffusion model turns conditioned coefficients back into audio. The transformer blocks use a radial projective norm (HoloNorm), $x \mapsto x/(1+\|x\|)$, which preserves vector directions and orthogonality, chosen over tanh for audio reconstruction.

What would settle it

Run the proposed pipeline on a tonal language pair (e.g., Tommo-So Dogon to Senufo) using content words that differ only by tone, such as /na/ 'mother' versus 'cow'. If human listeners or an audio-only similarity measure cannot reliably distinguish correct from incorrect tone choices in the reconstructed target audio, then MAST's pitch component is not preserving lexical tone and the central translation claim fails. A second, more direct check is to train the Eq. (8) mapping on paired audio and test whether the reconstructed waveform outperforms a simple spectrogram-based baseline on a meaning-preservation task.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a time-aligned tuple $M(A)(\tau) = (W(s,\tau), P(\tau), E_k(\tau), S(\tau), I(\tau))$—wavelet time–frequency coefficients, instantaneous pitch, learned prosodic embedding, speaker embedding, and intonation gesture descriptor—can serve as a complete conditioning signal for textless speech generation. Translation is reduced to a linear operation on wavelet coefficients: $C_{\text{Senufo}} = W \cdot C_{\text{Tommo}}$, with $W \in \mathbb{R}^{n \times n}$ learned or simulated from content-labeled audio, followed by wavelet reconstruction. The generative side uses a forward fractional SDE $dx(t) = \theta(t)(\bar{m}(t) - x(t))\,dt + \sigma(t)\,dB^H_t$ that ends in a known Gaussian mask, and a reverse-time fractional SDE whose score is explicit, $\nabla_x \log p(x(t)) = -(x(t) - m(t))/v^2(t)$, because the marginal law remains Gaussian. Together these components are claimed to yield high-fidelity, semantically consistent speech-to-speech translation without textual supervision.

Load-bearing premise

The load-bearing premise is that the MAST tuple carries enough of the linguistic and semantic content of an utterance that a learned linear map between source and target wavelet-coefficient spaces transfers meaning across languages without any text or parallel supervision.

Editorial extensions

If this is right

  • Audio-to-audio translation can be offered for unwritten languages because the pipeline never generates or consumes text.
  • Tonal languages should benefit disproportionately, since pitch contours and intonation gestures are explicitly encoded in MAST rather than being incidental byproducts of spectrogram encoding.
  • The explicit Gaussian score removes the need for a trained score network in the diffusion component, simplifying training and reducing a source of approximation error.
  • Long-range dependencies in speech can be modeled through the Hurst parameter, so subdiffusive ($H < 0.5$) and superdiffusive ($H > 0.5$) regimes become tunable for different audio styles.
  • Because the forward process ends in a known Gaussian mask, conditional generation can be steered to exact target distributions, enabling controllable voice and style transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If MAST truly captures lexical tone, the same representation could be applied to tone-sensitive tasks beyond translation, such as tonal speech recognition, emotion recognition, and language identification, without any text.
  • The linear map $W$ is a strong assumption; a natural extension is to test a learned nonlinear map in MAST space and compare translation quality to see whether meaning transfer actually requires nonlinearity.
  • The authors' use of an audio database for reconstruction suggests a retrieval-augmented design: comparing the MAST of generated speech to stored native MAST vectors could turn the decoder into a nearest-neighbor search over high-quality audio, which is a testable variant of the proposal.
  • The paper's own stated limitation—no standardized textless benchmark and reliance on subjective human judgment—implies that a concrete next step is to build a human-listener meaning-preservation benchmark for tonal languages, which would also be the clearest falsifier of the translation claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a fully textless audio-to-audio translation framework for low-text-resource, audio-rich languages. It introduces the Multiscale Audio-Semantic Transform (MAST), a tuple of wavelet coefficients, pitch, prosodic embeddings, speaker embeddings, and intonation descriptors; a fractional diffusion generative model driven by fractional Brownian motion; and a 'Fractional Transfusion' transformer architecture. The stated contributions are a family of audio-to-audio architectures, the MAST representation, a fractional diffusion conditioning framework, and a validation of wavelet representations over Mel spectrograms for tonal speech. The manuscript contains mathematical propositions for fractional Ornstein-Uhlenbeck processes, terminal Gaussian conditioning, and explicit Gaussian score functions, followed by implementation and application sections.

Significance. If the central claims were substantiated, the paper would address a genuine and important gap: speech-to-speech translation for unwritten or predominantly oral languages without relying on text. The authorial emphasis on the closed-form Gaussian score is correct in itself, and the fractional OU solution and terminal-condition construction are presented in a clear proposition-proof format. The literature review is broad and useful as an entry point to textless speech-to-speech work. However, the load-bearing claims are not demonstrated: there is no evidence that MAST carries lexical meaning, no training or evaluation of the linear translation map W, no semantic evaluation of generated target speech, and no benchmark for the claimed fractional diffusion sampler. The only quantitative result is a tone-classification table without dataset size, splits, or error bars. As it stands, the paper is a research proposal rather than a validated system.

major comments (5)
  1. [Section 5.2.2, Eqs. (8)-(10)] The central claim that the system performs 'high-fidelity, semantically consistent speech without reliance on textual supervision' is not tested anywhere in the manuscript. Section 5.2.2 describes translation as 'changing a column while staying on the same row' in a table of content-wise labeled audio files, which is supervised parallel-data alignment rather than textless learning from raw audio. The same subsection states that during reconstruction the output is compared with the original signal and, if close enough, the original signal is chosen as output. This means the pipeline can return the source recording instead of generating a target-language utterance, and no experiment measures semantic equivalence between source and output. The conclusion's own admission that 'the evaluation remains a persistent limitation' and that assessments rely on subjective human judgment is in direct tension with the abstract's claims.
  2. [Section 3, tone classification table] The only quantitative benchmark in the paper is the tone-classification accuracy table (Mel 81.2%, Morlet 91.7%, Coiflet 95.7%). No dataset size, number of speakers, train/test split, repetitions, or error bars are reported. Section 1.2(d) promises 'improved pitch localization and speaker consistency using MAST,' but no pitch-localization metric and no speaker-consistency metric appear in the paper. A table with three numbers, without experimental protocol, cannot support the claimed superiority of MAST-based representations for tonal speech.
  3. [Section 3, Eq. (8)] The map CSenufo = W · CTommo is introduced as 'learned or simulated,' but the paper provides no training data, no loss function, no optimization procedure, and no evaluation for W. Since W is the only mechanism by which source wavelet coefficients are transformed into target-language coefficients, Eq. (10) without a learned or otherwise specified W reduces to wavelet analysis followed by resynthesis. The claim that this constitutes cross-lingual translation is unsupported.
  4. [Section 4.4, Eq. (16)] The reverse-time fractional SDE in Eq. (16) is asserted rather than derived. Fractional Brownian motion with H ≠ 1/2 is not a semimartingale, so the standard score-based reverse-time diffusion theory does not apply without additional arguments. The paper does not provide a replacement proof that the proposed reverse equation samples from the intended conditional distribution. Because Eq. (16) is the generative sampling procedure of the proposed system, this is a load-bearing technical gap, not a presentation issue.
  5. [Section 2, Eqs. (2)-(4)] The claim that the architecture is 'fully textless' and operates 'without reliance on textual supervision' is compromised by the use of Wav2Vec2, Qwen-audio2-7B, and HiFi-GAN as fixed components. These are pretrained models whose training pipelines include text supervision (Qwen-audio is an audio-language model trained with text-associated tasks). The paper does not argue why treating such components as textless is legitimate, so the textless claim is asserted rather than established.
minor comments (5)
  1. [Section 5.2.1] The text states that 13 audio words from Table 14 are selected for a test, but no experiment with these words is reported; the only numeric result remains the tone-classification table in Section 3.
  2. [Section 4.4, Eq. (16)] The reverse-time SDE mentions a coefficient tilde{sigma}(t), but the displayed formula uses sigma*_T T^{-H} e^{Phi(T)-Phi(t)}, and tilde{sigma}(t) is never defined.
  3. [Section 3, Eqs. (11)-(12)] Equation (12) is used twice with different content, first for the admissibility constant and then for the inverse continuous wavelet transform; this duplication will confuse readers.
  4. [Section 1.2(d)] The contribution statement claims validation 'using examples from 24 different languages,' but the reported analyses concern Tommo-So Dogon only; the connection to the 24 languages listed in Section 5.2.1 is not demonstrated.
  5. [Throughout] There are numerous typographical errors, including 'meqn-field' (Section 4.4), 'likinf' (Section 5.2.2), and inconsistent table numbering; a careful proofreading pass is needed.

Circularity Check

2 steps flagged · score 8.0 of 10

Central textless-translation result is defined as row-aligned dictionary lookup and outputs the stored original audio; the MAST/fractional-diffusion pipeline is not the source of the claimed translation.

  1. self definitional [Section 3, Eq. (8), and Section 5.2.2, 'Wavelet analysis for labeled audio data']
    "CSenufo = W · CTommo (8) where W ∈ Rn×n is a transformation matrix learned or simulated. ... With this dictionary visualization approach, audio translation means changing a column while staying on the same row. For each of the 13 words we can plot the original audio signal and save it as yy language name."

    Eq. (8) presents the Tommo-So-to-Senufo map as an unknown transform to be learned or simulated, and Eq. (10) promises a full wavelet pipeline. Section 5.2.2 instead defines translation as keeping the row (a manually content-labeled word) and taking the audio file in the target-language column. Semantic equivalence is therefore imposed by the row label before any computation; MAST and W neither discover nor carry the meaning. The claimed 'semantically consistent speech' is an input label, not a model output, so the central result reduces to a dictionary lookup by construction.

  2. fitted input called prediction [Section 5.2.2, final lines of 'Wavelet analysis for labeled audio data']
    "During the reconstructed step, the reconstructed signal will be compared to the original signal, if they are closed enough, the original signal will be chosen as output making no non-native audio. By doing so the accent and dialects are preserved as they are from the sources."

    The generative pipeline described by Eqs. (9)-(10) and the diffusion decoder is short-circuited: whenever the reconstruction is 'closed enough' to the original audio, the original waveform is chosen as the output. The 'generated' target speech is thus forced to be a stored native recording, guaranteeing fidelity and speaker consistency only because no new audio is produced. The claimed high-fidelity, semantics-preserving translation is the input database replayed, not a consequence of MAST or fractional diffusion.

full rationale

The paper's central claim is that MAST plus fractional diffusion enables textless audio-to-audio translation that generates high-fidelity, semantically consistent speech. The implementation section shows the opposite: translation is 'changing a column while staying on the same row' in a manually content-labeled dictionary, and the output is the original stored audio whenever the reconstruction is close enough. That makes the headline translation result circular by construction: the semantic consistency is an input label, and the fidelity is the original recording. The linear-map equation CSenufo = W·CTommo is never equipped with a loss, training corpus, or evaluation, so it cannot independently produce the claimed result. The Gaussian score derivation in Proposition 4 is mathematically true and self-contained (though trivial); it is not circular and is therefore not counted against the paper. The lone quantitative table (Mel 81.2%, Morlet 91.7%, Coiflet 95.7%) lacks dataset size, split, and error bars and does not evaluate translation; this is a completeness/validation gap rather than circularity. Self-citations appear (e.g., forthcoming mean-field-type game theory volumes by the authors), but they are not load-bearing for the translation claim. Because the central contribution reduces to a labeled lookup and replay of source audio, the circularity score is high.

Assumptions & free parameters 6 free parameters · 5 assumptions · 3 invented entities

The framework rests on the assumption that audio semantics are fully captured by the MAST feature tuple, that a linear map can transfer wavelet coefficients across languages, and that a fractional-Brownian reverse SDE is a valid generative denoiser. It also depends on pretrained models trained with textual or labeled data despite claiming textlessness. These are domain assumptions not backed by experiments.

free parameters (6)
  • Hurst exponent H = 0.8 in experiments
    Chosen in Figures 11-12 to model fractional Brownian motion; no data-driven selection.
  • Terminal mask mean m*_T and variance (σ*_T)^2 = N(10,0.5)
    Chosen by hand as the diffusion endpoint in Section 5.1.
  • Linear translation map W = unspecified
    In Eq. (8) CSenufo = W·CTommo, W is 'learned or simulated' but no training method, data, or loss is given.
  • Drift coefficient θ = 1
    Set to 1 in the implementation examples (Section 5.1).
  • MAST embedding dimensions d, p, q = not specified
    Dimensions of prosody, speaker, and intonation embeddings are free choices; no ablation or tuning procedure is reported.
  • Wavelet central frequency ω0 and Coiflet order N = ω0=5 typical; N unspecified
    Mother wavelet parameters are chosen without justification or sensitivity analysis.
assumptions (5)
  • domain assumption Audio semantic content is fully captured by the MAST tuple (wavelet, pitch, prosody, speaker, intonation)
    The entire translation pipeline conditions on M(A); if meaning lives in features not in this tuple, translation fails. Introduced in Section 3.
  • domain assumption A linear transform can map source wavelet coefficients to target wavelet coefficients preserving meaning
    Eq. (10) assumes x_Senufo = IDWT(W·DWT(x_Tommo)) works; no nonlinear or contextual mapping is specified.
  • domain assumption The reverse-time fractional SDE is a valid generative denoising process
    Section 4.4 writes a reverse SDE driven by time-reversed fBm, but fBm is not a semimartingale for H≠1/2; no theorem supports score-based reverse generation in this setting.
  • ad hoc to paper Pretrained text-supervised models can be treated as textless components
    Section 2 uses Wav2Vec2 and Qwen-audio2-7B, both trained with text/labeled data, yet the paper claims no textual supervision.
  • ad hoc to paper Content-wise labels (hash codes) are not textual supervision
    Section 5.2.2 labels audio 'content-wise' with codes; this is a form of supervision that the paper does not count as text.
invented entities (3)
  • MAST (Multiscale Audio-Semantic Transform)
    purpose: Time-aligned tuple of wavelet, pitch, prosody, speaker, and intonation features used to condition generation
    No external validation; the only evidence is an undocumented accuracy table and illustrative plots.
  • Fractional Transfusion architecture
    purpose: Diffusion transformer driven by fractional Brownian motion for audio generation
    Presented as a model class but never trained or evaluated.
  • Holonorm (projective norm) activation
    purpose: Normalization to replace tanh and preserve orthogonality in audio reconstruction
    Claimed to preserve signal geometry, but no experiment supports the claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the Barriers of Text-Hungry and Audio-Deficient AI." pith.science (2026). https://pith.science/paper/TKAD7GCW

@misc{pith2026250602443,
  author       = {Pith},
  title        = {Pith review of: Breaking the Barriers of Text-Hungry and Audio-Deficient AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKAD7GCW}},
  note         = {Machine review of arXiv:2506.02443}
}
read the original abstract

While global linguistic diversity spans more than 7164 recognized languages, the current dominant architecture of machine intelligence remains fundamentally biased toward written text. This bias excludes over 700 million people particularly in rural and remote regions who are audio-literate. In this work, we introduce a fully textless, audio-to-audio machine intelligence framework designed to serve this underserved population, and all the people who prefer audio-efficiency. Our contributions include novel Audio-to-Audio translation architectures that bypass text entirely, including spectrogram-, scalogram-, wavelet-, and unit-based models. Central to our approach is the Multiscale Audio-Semantic Transform (MAST), a representation that encodes tonal, prosodic, speaker, and expressive features. We further integrate MAST into a fractional diffusion of mean-field-type framework powered by fractional Brownian motion. It enables the generation of high-fidelity, semantically consistent speech without reliance on textual supervision. The result is a robust and scalable system capable of learning directly from raw audio, even in languages that are unwritten or rarely digitized. This work represents a fundamental shift toward audio-native machine intelligence systems, expanding access to language technologies for communities historically left out of the current machine intelligence ecosystem.

Figures

Figures reproduced from arXiv: 2506.02443 by the authors.

Figure 1
Figure 1. Three architectures for voice-to-voice translation. (a) is text-hungry and audio-deficient. (b) and [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 3
Figure 3. Comparison of Mel spectrogram (left) and Morlet wavelet scalogram (right) for a Tommo-So Dogon [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗
Figure 4
Figure 4. Comparison of Mel spectrogram (left), Morlet wavelet scalogram (middle) and Coiflet wavelet [PITH_FULL_IMAGE:figures/full_fig_p024_4.png] view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Wavelet-based direct S2ST architectures. (b) and (d) use Morlet and Coiflet wavelets as input [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: A Unit-based Textless Direct Audio-to-Audio Translation. [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Holonorm map hn(x) = x 1+|x| in 1D and its primitive, derivative, inverse map 36 [PITH_FULL_IMAGE:figures/full_fig_p036_7.png]
Figure 8
Figure 8. Figure 8: Holonorm map hn(kx) = kx 1+|kx| in 1D 37 [PITH_FULL_IMAGE:figures/full_fig_p037_8.png]
Figure 9
Figure 9. Figure 9: Holonorm map hn(x, y) = (x,y) 1+∥(x,y)∥p in 2D. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_9.png]
Figure 10
Figure 10. Figure 10: hn(15x) vs tanh(5x) Choose f(s) = C ∈ R d×d constant. Then: K = λT · CC⊤, λT := H(2H − 1) Z T 0 Z T 0 |s − r| 2H−2 ds dr. Thus: Cov[x(T)] = λT · U(T)CC⊤U(T) ⊤. To match Σ∗ T , choose: C = [PITH_FULL_IMAGE:figures/full_fig_p041_10.png]
Figure 11
Figure 11. Figure 11: Mean-Field-Type Fractional Diffusion 43 [PITH_FULL_IMAGE:figures/full_fig_p043_11.png]
Figure 12
Figure 12. Figure 12: Mean-Field-Type Fractional Diffusion: forward and backward [PITH_FULL_IMAGE:figures/full_fig_p044_12.png]
Figure 13
Figure 13. Figure 13: An audio database matrix for Africa: part 1 [PITH_FULL_IMAGE:figures/full_fig_p046_13.png]
Figure 14
Figure 14. Figure 14: An audio database matrix for Africa: part 2 [PITH_FULL_IMAGE:figures/full_fig_p047_14.png]
Figure 15
Figure 15. Figure 15: An example of blockaudio matrix [PITH_FULL_IMAGE:figures/full_fig_p048_15.png]
Figure 16
Figure 16. Figure 16: MAST wavelet analysis of an audio in Tommo-So Dogon. [PITH_FULL_IMAGE:figures/full_fig_p048_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

165 extracted references · 34 canonical work pages

  1. [1]

    Simultron: On-device simultaneous speech to speech translation

    Alex Agranovich, Eliya Nachmani, Oleg Rybakov, Yifan Ding, Ye Jia, Nadav Bar, Heiga Zen, and Michelle Tadmor Ramanovich. Simultron: On-device simultaneous speech to speech translation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. 50

    2025

  2. [2]

    Qwen 2.5: A comprehensive review of the leading resource-efficient llm with potentioal to surpass all competitors

    Imtiaz Ahmed, Sadman Islam, Partha Protim Datta, Imran Kabir, Naseef Ur Rahman Chowdhury, and Ahshanul Haque. Qwen 2.5: A comprehensive review of the leading resource-efficient llm with potentioal to surpass all competitors

  3. [3]

    Analysis of layer- wise training in direct speech to speech translation using bi-lstm

    Lalaram Arya, Ayush Agarwal, Jagabandhu Mishra, and SR Mahadeva Prasanna. Analysis of layer- wise training in direct speech to speech translation using bi-lstm. In 2022 25th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA) , pages 1–6. IEEE, 2022

    2022

  4. [4]

    Precipitation nowcasting with generative diffusion models

    Andrea Asperti, Fabio Merizzi, Alberto Paparella, Giorgio Pedrazzi, Matteo Angelinelli, and Stefano Colamonaco. Precipitation nowcasting with generative diffusion models. Applied Intelligence, 55(2):1– 21, 2025

    2025

  5. [5]

    Mean-Field-Type Game Theory: Applica- tions, volume 2

    Tamer Ba¸ sar, Boualem Djehiche, and Hamidou Tembine. Mean-Field-Type Game Theory: Applica- tions, volume 2. 2025. Forthcoming

    2025

  6. [6]

    Tamer Ba¸ sar, Boualem Djehiche, and Hamidou Tembine.Mean-Field-Type Game Theory: Foundations and New Directions , volume 1. 2025. Forthcoming

    2025

  7. [7]

    Listen and translate: A proof of concept for end-to-end speech-to-text translation

    Alexandre B´ erard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. Listen and translate: A proof of concept for end-to-end speech-to-text translation. arXiv preprint arXiv:1612.01744 , 2016

    arXiv 2016

  8. [8]

    Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z. Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering , 2024

    2024

Show all 165 references
  1. [9]

    Large language models are strong audio-visual speech recognition learners

    Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, and Maja Pantic. Large language models are strong audio-visual speech recognition learners. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech a...

    2025

  2. [10]

    Low frame-rate speech codec: a codec designed for fast high-quality speech llm training and inference

    Edresson Casanova, Ryan Langman, Paarth Neekhara, Shehzeen Hussain, Jason Li, Subhankar Ghosh, Ante Juki´ c, and Sang-Gil Lee. Low frame-rate speech codec: a codec designed for fast high-quality speech llm training and inference. In ICASSP 2025-2025 IEEE International Conferen...

    2025

  3. [11]

    A speech-to-speech translation based interface for tourism

    Mauro Cettolo, Anna Corazza, Giannino Lazzari, Fabio Pianesi, Emanuele Pianta, and LM Tovena. A speech-to-speech translation based interface for tourism. In Information and Communication Tech- nologies in Tourism 1999: Proceedings of the International Conference in Innsbruck, ...

    1999

  4. [12]

    Exploring in-context learning of textless speech language model for speech classification tasks

    Kai-Wei Chang, Ming-Hao Hsu, Shan-Wen Li, and Hung-yi Lee. Exploring in-context learning of textless speech language model for speech classification tasks. In Proc. Interspeech 2024, pages 4139– 4143, 2024

    2024

  5. [13]

    Audio large language models can be descriptive speech quality evaluators

    Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang, Chao-Han Huck Yang, and Eng Siong Chng. Audio large language models can be descriptive speech quality evaluators. arXiv preprint arXiv:2501.17202, 2025

    2025 arXiv

  6. [14]

    Multi-modal generative ai: Multi-modal llm, diffusion and beyond

    Hong Chen, Xin Wang, Yuwei Zhou, Bin Huang, Yipeng Zhang, Wei Feng, Houlun Chen, Zeyang Zhang, Siao Tang, and Wenwu Zhu. Multi-modal generative ai: Multi-modal llm, diffusion and beyond. arXiv preprint arXiv:2409.14993 , 2024

    2024

  7. [15]

    Blaser: A text-free speech-to-speech translation evaluation metric

    Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk, and Marta R Costa-juss` a. Blaser: A text-free speech-to-speech translation evaluation metric. arXiv preprint arXiv:2212.08486 , 2022

    2022 arXiv

  8. [16]

    Opportunities and challenges of diffusion models for generative ai

    Minshuo Chen, Song Mei, Jianqing Fan, and Mengdi Wang. Opportunities and challenges of diffusion models for generative ai. National Science Review , 11(12):nwae348, 2024. 51

    2024

  9. [17]

    Speech-to-speech translation for a real-world unwritten language

    Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, et al. Speech-to-speech translation for a real-world unwritten language. arXiv preprint arXiv:2211.06474 , 2022

    2022 arXiv

  10. [18]

    Beyond single-audio: Advancing multi-audio processing in audio large language models

    Yiming Chen, Xianghu Yue, Xiaoxue Gao, Chen Zhang, Luis Fernando D’Haro, Robby T Tan, and Haizhou Li. Beyond single-audio: Advancing multi-audio processing in audio large language models. arXiv preprint arXiv:2409.18680 , 2024

    2024 arXiv

  11. [19]

    Mavflow: Preserving paralinguistic elements with conditional flow matching for zero-shot av2av multilingual translation

    Sungwoo Cho, Jeongsoo Choi, Sungnyun Kim, and Se-Young Yun. Mavflow: Preserving paralinguistic elements with conditional flow matching for zero-shot av2av multilingual translation. arXiv preprint arXiv:2503.11026, 2025

    2025 arXiv

  12. [20]

    V2sflow: Video-to-speech generation with speech decomposition and rectified flow

    Jeongsoo Choi, Ji-Hoon Kim, Jinyu Li, Joon Son Chung, and Shujie Liu. V2sflow: Video-to-speech generation with speech decomposition and rectified flow. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2025

    2025

  13. [21]

    Qwen2-audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 , 2024

    2024 arXiv

  14. [22]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 , 2023

    2023 arXiv

  15. [23]

    Diffusion models in vision: A survey

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023

    2023

  16. [24]

    Exploring the benefits of tokenization of discrete acoustic units

    Avihu Dekel and Raul Fernandez. Exploring the benefits of tokenization of discrete acoustic units. arXiv preprint arXiv:2406.05547 , 2024

    2024 arXiv

  17. [25]

    Adiff: Explaining audio difference using natural language

    Soham Deshmukh, Shuo Han, Rita Singh, and Bhiksha Raj. Adiff: Explaining audio difference using natural language. arXiv preprint arXiv:2502.04476 , 2025

    2025 arXiv

  18. [26]

    French- fulfulde textless and cascading speech translation: Towards a dual architecture

    Tala Metalom Diane Carole, Yenke Blaise Omer, and Fendji Kedieng Ebongue Jean Louis. French- fulfulde textless and cascading speech translation: Towards a dual architecture. In International Conference on Safe, Secure, Ethical, Responsible Technologies and Emerging Application...

  19. [27]

    Textless speech-to-speech trans- lation with limited parallel data

    Anuj Diwan, Anirudh Srinivasan, David Harwath, and Eunsol Choi. Textless speech-to-speech trans- lation with limited parallel data. arXiv preprint arXiv:2305.15405 , 2023

    2023 arXiv

  20. [28]

    Polyvoice: Language models for speech to speech translation

    Qianqian Dong, Zhiying Huang, Qiao Tian, Chen Xu, Tom Ko, Yunlong Zhao, Siyuan Feng, Tang Li, Kexin Wang, Xuxin Cheng, et al. Polyvoice: Language models for speech to speech translation. arXiv preprint arXiv:2306.02982, 2023

    2023 arXiv

  21. [29]

    Lauragpt: Listen, attend, understand, and regenerate audio with gpt

    Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023

    2023 arXiv

  22. [30]

    Analyzing speech unit selection for textless speech-to-speech translation

    Jarod Duret, Yannick Est` eve, and Titouan Parcollet. Analyzing speech unit selection for textless speech-to-speech translation. arXiv preprint arXiv:2407.18332 , 2024

    2024 arXiv

  23. [31]

    Enhancing expressivity transfer in textless speech-to-speech translation

    Jarod Duret, Benjamin O’Brien, Yannick Est` eve, and Titouan Parcollet. Enhancing expressivity transfer in textless speech-to-speech translation. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 1–8. IEEE, 2023

    2023

  24. [32]

    Towards massive parallel corpus creation for hausa-to-english machine translation

    Moses E Ekpenyong, Aminu A Suleiman, and Murtala Salihu. Towards massive parallel corpus creation for hausa-to-english machine translation. In Current Issues in Descriptive Linguistics and Digital Humanities: A Festschrift in Honor of Professor Eno-Abasi Essien Urua , pages 50...

    2022

  25. [33]

    Auditory-visual perception of speech

    Norman P Erber. Auditory-visual perception of speech. Journal of speech and hearing disorders , 40(4):481–492, 1975

    1975

  26. [34]

    Cascade or direct speech translation? a case study

    Thierry Etchegoyhen, Haritz Arzelus, Harritxu Gete, Aitor Alvarez, Iv´ an G Torre, Juan Manuel Mart ´ ın-Do˜ nas, Ander Gonz´ alez-Docasal, and Edson Benites Fernandez. Cascade or direct speech translation? a case study. Applied Sciences, 12(3):1097, 2022

    2022

  27. [35]

    Ctc-based non-autoregressive textless speech-to-speech translation

    Qingkai Fang, Zhengrui Ma, Yan Zhou, Min Zhang, and Yang Feng. Ctc-based non-autoregressive textless speech-to-speech translation. arXiv preprint arXiv:2406.07330 , 2024

    2024 arXiv

  28. [36]

    Can we achieve high-quality direct speech-to-speech translation without parallel speech data? arXiv preprint arXiv:2406.07289 , 2024

    Qingkai Fang, Shaolei Zhang, Zhengrui Ma, Min Zhang, and Yang Feng. Can we achieve high-quality direct speech-to-speech translation without parallel speech data? arXiv preprint arXiv:2406.07289 , 2024

    2024 arXiv

  29. [37]

    Generative learning of the solution of parametric partial differential equations using guided diffusion models and virtual observations

    Han Gao, Sebastian Kaltenbach, and Petros Koumoutsakos. Generative learning of the solution of parametric partial differential equations using guided diffusion models and virtual observations. Com- puter Methods in Applied Mechanics and Engineering , 435:117654, 2025

    2025

  30. [38]

    Unsupervised speech technology for low-resource languages

    Heting Gao. Unsupervised speech technology for low-resource languages . PhD thesis, University of Illinois at Urbana-Champaign, 2024

    2024

  31. [39]

    Speech-to-speech translation

    Yuqing Gao, Liang Gu, and Bowen Zhou. Speech-to-speech translation. Advances in Chinese Spoken Language Processing, page 271, 2007

    2007

  32. [40]

    Audio dialogues: Dialogues dataset for audio and music understanding

    Arushi Goel, Zhifeng Kong, Rafael Valle, and Bryan Catanzaro. Audio dialogues: Dialogues dataset for audio and music understanding. arXiv preprint arXiv:2404.07616 , 2024

    2024 arXiv

  33. [41]

    Multilingual speech-to-speech translation into multiple target languages

    Hongyu Gong, Ning Dong, Sravya Popuri, Vedanuj Goswami, Ann Lee, and Juan Pino. Multilingual speech-to-speech translation into multiple target languages. arXiv preprint arXiv:2307.08655 , 2023

    2023 arXiv

  34. [42]

    Joint audio and speech understanding

    Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE, 2023

    2023

  35. [43]

    Tibetan–chinese speech-to-speech translation based on dis- crete units

    Zairan Gong, Xiaona Xu, and Yue Zhao. Tibetan–chinese speech-to-speech translation based on dis- crete units. Scientific Reports, 15(1):2592, 2025

    2025

  36. [44]

    Recent advances in discrete speech tokens: A review

    Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review. arXiv preprint arXiv:2502.06490, 2025

    2025

  37. [45]

    Direct speech-to-speech neural machine translation: A survey

    Mahendra Gupta, Maitreyee Dutta, and Chandresh Kumar Maurya. Direct speech-to-speech neural machine translation: A survey. arXiv preprint arXiv:2411.14453 , 2024

    2024 arXiv

  38. [46]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26584–2...

    2024

  39. [47]

    Physics-inspired approaches in generative diffusion models

    Yuji Hirono. Physics-inspired approaches in generative diffusion models. Journal of the Physical Society of Japan, 94(3):031008, 2025

    2025

  40. [48]

    Exploring in-context learning of textless speech language model for speech classification tasks

    Ming-Hao Hsu, Kai-Wei Chang, Shang-Wen Li, and Hung-yi Lee. Exploring in-context learning of textless speech language model for speech classification tasks. arXiv preprint arXiv:2310.12477 , 2023

    2023 arXiv

  41. [49]

    Chain-of-thought prompting for speech translation

    Ke Hu, Zhehuai Chen, Chao-Han Huck Yang, Piotr ˙Zelasko, Oleksii Hrinchuk, Vitaly Lavrukhin, Jagadeesh Balam, and Boris Ginsburg. Chain-of-thought prompting for speech translation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (IC...

    2025

  42. [50]

    Transpeech: Speech-to-speech translation with bilateral perturbation

    Rongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He, and Zhou Zhao. Transpeech: Speech-to-speech translation with bilateral perturbation. arXiv preprint arXiv:2205.12523, 2022

    2022 arXiv

  43. [51]

    Text- less acoustic model with self-supervised distillation for noise-robust expressive speech-to-speech trans- lation

    Min-Jae Hwang, Ilia Kulikov, Benjamin Peloquin, Hongyu Gong, Peng-Jen Chen, and Ann Lee. Text- less acoustic model with self-supervised distillation for noise-robust expressive speech-to-speech trans- lation. arXiv preprint arXiv:2406.02733 , 2024

    2024 arXiv

  44. [52]

    Massively multi- lingual forced aligner leveraging self-supervised discrete units

    Hirofumi Inaguma, Ilia Kulikov, Zhaoheng Ni, Sravya Popuri, and Paden Tomasello. Massively multi- lingual forced aligner leveraging self-supervised discrete units. In 2024 IEEE Spoken Language Tech- nology Workshop (SLT), pages 899–905. IEEE, 2024

    2024

  45. [53]

    Libris2s: A german-english speech-to-speech translation corpus

    Pedro Jeuris and Jan Niehues. Libris2s: A german-english speech-to-speech translation corpus. arXiv preprint arXiv:2204.10593, 2022

    2022 arXiv

  46. [54]

    Wavchat: A survey of spoken dialogue models

    Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, et al. Wavchat: A survey of spoken dialogue models. arXiv preprint arXiv:2411.13577, 2024

    2024 arXiv

  47. [55]

    Can generative geospatial diffusion models excel as discriminative geospatial foundation models? arXiv preprint arXiv:2503.07890, 2025

    Yuru Jia, Valerio Marsocci, Ziyang Gong, Xue Yang, Maarten Vergauwen, and Andrea Nascetti. Can generative geospatial diffusion models excel as discriminative geospatial foundation models? arXiv preprint arXiv:2503.07890, 2025

    2025 arXiv

  48. [56]

    Listra automatic speech translation: English to lingala case study

    Salomon Kabongo Kabenamualu, Vukosi Marivate, and Herman Kamper. Listra automatic speech translation: English to lingala case study. In Proceedings of the Workshop on Dataset Creation for Lower-Resourced Languages within the 13th Language Resources and Evaluation Conference , ...

    2022

  49. [57]

    Gdplan: Generative network planning via graph diffusion model

    Nuowen Kan, Sa Yan, Junni Zou, Wenrui Dai, Xing Gao, Chenglin Li, and Hongkai Xiong. Gdplan: Generative network planning via graph diffusion model. IEEE Transactions on Networking , 2025

    2025

  50. [58]

    Direct punjabi to english speech translation using discrete units

    Prabhjot Kaur, L Andrew M Bush, and Weisong Shi. Direct punjabi to english speech translation using discrete units. arXiv preprint arXiv:2402.15967 , 2024

    2024 arXiv

  51. [59]

    Textless unit-to-unit training for many- to-many multilingual speech-to-speech translation

    Minsu Kim, Jeongsoo Choi, Dahun Kim, and Yong Man Ro. Textless unit-to-unit training for many- to-many multilingual speech-to-speech translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  52. [60]

    Phi dm-dialog: an experimental speech-to-speech dialog translation system

    Hirotani Kitano. Phi dm-dialog: an experimental speech-to-speech dialog translation system. Com- puter, 24(6):36–50, 1991

    1991

  53. [61]

    High-fidelity simultaneous speech-to-speech translation

    Tom Labiausse, Laurent Mazar´ e, Edouard Grave, Patrick P´ erez, Alexandre D´ efossez, and Neil Zeghi- dour. High-fidelity simultaneous speech-to-speech translation. arXiv preprint arXiv:2502.03382 , 2025

    2025 arXiv

  54. [62]

    Janus-iii: Speech-to-speech translation in multiple languages

    Alon Lavie, Alex Waibel, Lori Levin, Michael Finke, Donna Gates, Marsal Gavalda, Torsten Zeppen- feld, and Puming Zhan. Janus-iii: Speech-to-speech translation in multiple languages. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing , volume 1, ...

    1997

  55. [63]

    Textless speech-to-speech translation on real data

    Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, et al. Textless speech-to-speech translation on real data. arXiv preprint arXiv:2112.08352 , 2021

    2021 arXiv

  56. [64]

    Video diffusion models are strong video inpainter

    Minhyeok Lee, Suhwan Cho, Chajin Shin, Jungho Lee, Sunghun Yang, and Sangyoun Lee. Video diffusion models are strong video inpainter. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 4526–4533, 2025

    2025

  57. [65]

    Speech proportion and accuracy in simultaneous interpretation from english into korean

    Tae-Hyung Lee. Speech proportion and accuracy in simultaneous interpretation from english into korean. Meta, 44(2):260–267, 1999. 54

    1999

  58. [66]

    Diffusion models for audio restoration: A review [special issue on model-based and data-driven audio signal processing]

    Jean-Marie Lemercier, Julius Richter, Simon Welker, Eloi Moliner, Vesa V¨ alim¨ aki, and Timo Gerk- mann. Diffusion models for audio restoration: A review [special issue on model-based and data-driven audio signal processing]. IEEE Signal Processing Magazine , 41(6):72–84, 2025

    2025

  59. [67]

    Conditional diffusion model for missing value imputation

    Binyi Li, Long Long, Xi Zuo, and Long Chen. Conditional diffusion model for missing value imputation. The European Journal on Artificial Intelligence , page 30504554241311182, 2025

    2025

  60. [68]

    Brainecho: Semantic brain signal decoding through vector-quantized spectrogram reconstruction for whisper-enhanced text generation

    Jilong Li, Zhenxi Song, Jiaqi Wang, Min Zhang, and Zhiguo Zhang. Brainecho: Semantic brain signal decoding through vector-quantized spectrogram reconstruction for whisper-enhanced text generation. arXiv preprint arXiv:2410.14971 , 2024

    arXiv

  61. [69]

    Baichuan-audio: A unified framework for end-to-end speech interaction

    Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, et al. Baichuan-audio: A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.17239 , 2025

    2025 arXiv

  62. [70]

    Textless direct speech-to-speech translation with discrete speech representation

    Xinjian Li, Ye Jia, and Chung-Cheng Chiu. Textless direct speech-to-speech translation with discrete speech representation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  63. [71]

    Beyond words: Auralllm and signmst-c for precise sign language production and bidirectional accessibility

    Yulong Li, Yuxuan Zhang, Feilong Tang, Mian Zhou, Zhixiang Lu, Haochen Xue, Yifang Wang, Kang Dang, and Jionglong Su. Beyond words: Auralllm and signmst-c for precise sign language production and bidirectional accessibility. arXiv preprint arXiv:2501.00765 , 2025

    2025 arXiv

  64. [72]

    Align-slm: Textless spoken language models with reinforcement learning from ai feedback

    Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi Lee, and Ivan Bulyko. Align-slm: Textless spoken language models with reinforcement learning from ai feedback. arXiv preprint arXiv:2411.01834 , 2024

    2024 arXiv

  65. [73]

    Haitao Lin, Yufei Huang, Odin Zhang, Siqi Ma, Meng Liu, Xuanjing Li, Lirong Wu, Jishui Wang, Tingjun Hou, and Stan Z. Li. Diffbp: Generative diffusion of 3d molecules for target protein binding. Chemical Science, 16(3):1417–1431, 2025

    2025

  66. [74]

    Handdiffuse: generative controllers for two-hand interactions via diffusion models

    Pei Lin. Handdiffuse: generative controllers for two-hand interactions via diffusion models. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 5280–5288, 2025

    2025

  67. [75]

    A preliminary exploration with gpt-4o voice mode

    Yu-Xiang Lin, Chih-Kai Yang, Wei-Chih Chen, Chen-An Li, Chien-yu Huang, Xuanjun Chen, and Hung-yi Lee. A preliminary exploration with gpt-4o voice mode. arXiv preprint arXiv:2502.09940 , 2025

    2025 arXiv

  68. [76]

    Recent highlights in multilingual and multimodal speech translation

    Danni Liu and Jan Niehues. Recent highlights in multilingual and multimodal speech translation. In Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024) , pages 235–253, 2024

    2024

  69. [77]

    Speech-to-speech low-resource translation

    Hsiao-Chuan Liu, Min-Yuh Day, and Chih-Chien Wang. Speech-to-speech low-resource translation. In 2023 IEEE 24th International Conference on Information Reuse and Integration for Data Science (IRI), pages 91–95. IEEE, 2023

    2023

  70. [78]

    Listening and seeing again: Generative error correction for audio-visual speech recognition

    Rui Liu, Hongyu Yuan, Guanglai Gao, and Haizhou Li. Listening and seeing again: Generative error correction for audio-visual speech recognition. Information Fusion, page 103077, 2025

    2025

  71. [79]

    Slide: Integrating speech language model with llm for spontaneous spoken dialogue generation.arXiv preprint arXiv:2501.00805, 2025

    Haitian Lu, Gaofeng Cheng, Liuping Luo, Leying Zhang, Yanmin Qian, and Pengyuan Zhang. Slide: Integrating speech language model with llm for spontaneous spoken dialogue generation.arXiv preprint arXiv:2501.00805, 2025

    2025 arXiv

  72. [80]

    Llamapartialspoof: An llm-driven fake speech dataset simulating disinformation generation

    Hieu-Thi Luong, Haoyang Li, Lin Zhang, Kong Aik Lee, and Eng Siong Chng. Llamapartialspoof: An llm-driven fake speech dataset simulating disinformation generation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5...

    2025

  73. [81]

    Build llm-based zero-shot streaming tts system with cosyvoice

    Xiang Lyu, Yuxuan Wang, Tianyu Zhao, Hao Wang, Huadai Liu, and Zhihao Du. Build llm-based zero-shot streaming tts system with cosyvoice. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–2. IEEE, 2025

    2025

  74. [82]

    Auto-avsr: Audio-visual speech recognition with automatic labels

    Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis, and Maja Pantic. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP...

    2023

  75. [83]

    Real-time textless dialogue generation

    Long Mai and Julie Carson-Berndsen. Real-time textless dialogue generation. arXiv preprint arXiv:2501.04877, 2025

    2025 arXiv

  76. [84]

    Slamming: Training a speech language model on one gpu in a day

    Gallil Maimon, Avishai Elmakies, and Yossi Adi. Slamming: Training a speech language model on one gpu in a day. arXiv preprint arXiv:2502.15814 , 2025

    2025 arXiv

  77. [85]

    Enhancing low-resource language and instruction following capabilities of audio language models

    Potsawee Manakul, Guangzhi Sun, Warit Sirichotedumrong, Kasima Tharnpipitchai, and Kunat Pi- patanakul. Enhancing low-resource language and instruction following capabilities of audio language models. arXiv preprint arXiv:2409.10999 , 2024

    2024 arXiv

  78. [86]

    Make some noise: Towards llm audio reasoning and generation using sound tokens

    Shivam Mehta, Nebojsa Jojic, and Hannes Gamper. Make some noise: Towards llm audio reasoning and generation using sound tokens. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2025

    2025

  79. [87]

    Deep networks as denoising algorithms: Sample-efficient learning of diffusion models in high-dimensional graphical models

    Song Mei and Yuchen Wu. Deep networks as denoising algorithms: Sample-efficient learning of diffusion models in high-dimensional graphical models. IEEE Transactions on Information Theory , 2025

    2025

  80. [88]

    Amharic speech recognition for speech translation

    Michael Melese, Laurent Besacier, and Million Meshesha. Amharic speech recognition for speech translation. In Atelier Traitement Automatique des Langues Africaines (TALAF). JEP-TALN 2016 , 2016

    2016

  81. [89]

    Parrot: Autoregressive spoken dialogue language modeling with decoder-only transform- ers

    Ziqiao Meng, Qichao Wang, Wenqian Cui, Yifei Zhang, Bingzhe Wu, Irwin King, Liang Chen, and Peilin Zhao. Parrot: Autoregressive spoken dialogue language modeling with decoder-only transform- ers. In Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation

    2024

  82. [90]

    Towards to a direct speech to speech for endangered languages in africa

    Diane Carole Tala Metalom, Jean Louis Fendji Kedieng Ebongue, and Blaise Omer Yenke. Towards to a direct speech to speech for endangered languages in africa. In CARI 2022 , 2022

    2022

  83. [91]

    A unit-based system and dataset for expressive direct speech-to-speech translation

    Anna Min, Chenxu Hu, Yi Ren, and Hang Zhao. A unit-based system and dataset for expressive direct speech-to-speech translation. arXiv preprint arXiv:2502.00374 , 2025

    2025 arXiv

  84. [92]

    Spoken question an- swering and speech continuation using spectrogram-powered llm

    Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question an- swering and speech continuation using spectrogram-powered llm. arXiv preprint arXiv:2305....

    2023 arXiv

  85. [93]

    Towards real-time multilingual multimodal speech-to-speech translation

    Satoshi Nakamura. Towards real-time multilingual multimodal speech-to-speech translation. In SLTU, pages 13–15, 2014

    2014

  86. [94]

    One model, many languages: Meta-learning for multilingual text-to-speech

    Tom´ aˇ s Nekvinda and Ondˇ rej Duˇ sek. One model, many languages: Meta-learning for multilingual text-to-speech. arXiv preprint arXiv:2008.00768 , 2020

    2008 arXiv

  87. [95]

    Spoken Language Modeling from Raw Audio

    Tu Anh Nguyen. Spoken Language Modeling from Raw Audio. PhD thesis, Sorbonne Universit´ e, 2024

    2024

  88. [96]

    Visually grounded speech models for low-resource languages and cognitive modelling

    Leanne Nortje. Visually grounded speech models for low-resource languages and cognitive modelling. arXiv preprint arXiv:2409.02865 , 2024

    2024 arXiv

  89. [97]

    Verbmobil: The use of prosody in the linguistic components of a speech understanding system

    Elmar Noth, Anton Batliner, Andreas Kießling, Ralf Kompe, and Heinrich Niemann. Verbmobil: The use of prosody in the linguistic components of a speech understanding system. IEEE Transactions on Speech and Audio processing, 8(5):519–532, 2000. 56

    2000

  90. [98]

    Exploiting phonological similarities between african languages to achieve speech to speech translation

    Peter Ochieng and Dennis Kaburu. Exploiting phonological similarities between african languages to achieve speech to speech translation. arXiv preprint arXiv:2410.23323 , 2024

    2024 arXiv

  91. [99]

    Let’s go real talk: Spoken dialogue model for face-to-face conversation

    Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. Let’s go real talk: Spoken dialogue model for face-to-face conversation. arXiv preprint arXiv:2406.07867, 2024

    2024 arXiv

  92. [100]

    Long-form speech generation with spoken language models

    Se Jin Park, Julian Salazar, Aren Jansen, Keisuke Kinoshita, Yong Man Ro, and RJ Skerry-Ryan. Long-form speech generation with spoken language models. arXiv preprint arXiv:2412.18603 , 2024

    2024 arXiv

  93. [101]

    Mslm-s2st: A multitask speech language model for textless speech-to-speech translation with speaker style preservation

    Yifan Peng, Ilia Kulikov, Yilin Yang, Sravya Popuri, Hui Lu, Changhan Wang, and Hongyu Gong. Mslm-s2st: A multitask speech language model for textless speech-to-speech translation with speaker style preservation. arXiv preprint arXiv:2403.12408 , 2024

    2024 arXiv

  94. [102]

    Bridging language barriers: Exploring hindi-to-english speech-to-speech translation for multilingual communication

    Divith Phogat, Karnati Sai Prashanth, Mangamuru Sai Rishith, Rachure Charith Sai, Sajja Bala Karthikeya, G Jyothish Lal, and B Premjith. Bridging language barriers: Exploring hindi-to-english speech-to-speech translation for multilingual communication. In Congress on Intellige...

    2023

  95. [103]

    Voice cloning for low-resource languages: Investigating the prospects for tamil

    Vishnu Radhakrishnan, A Aadharsh Aadhithya, Jayanth Mohan, M Visweswaran, G Jyothish Lal, and B Premjith. Voice cloning for low-resource languages: Investigating the prospects for tamil. Automatic Speech Recognition and Translation for Low Resource Languages, pages 243–257, 2024

    2024

  96. [104]

    Sonics: Synthetic or not–identifying counterfeit songs

    Md Awsafur Rahman, Zaber Ibn Abdul Hakim, Najibul Haque Sarker, Bishmoy Paul, and Shaikh Anowarul Fattah. Sonics: Synthetic or not–identifying counterfeit songs. arXiv preprint arXiv:2408.14080, 2024

    2024 arXiv

  97. [105]

    Optimizing direct speech-to-text translation for un-orthographic low-resource tribal lan- guages using source transliterations

    Tonmoy Rajkhowa, Amartya Roy Chowdhury, Prashant Bannulmath, Deepak KT, and SR Mahadeva Prasanna. Optimizing direct speech-to-text translation for un-orthographic low-resource tribal lan- guages using source transliterations. In 2023 26th Conference of the Oriental COCOSDA Int...

    2023

  98. [106]

    Arce, and Mark Stephen

    Andres Ramirez-Jaime, Nestor Porras-Diaz, Gonzalo R. Arce, and Mark Stephen. Super-resolved 3d satellite lidar imaging of earth via generative diffusion models. IEEE Transactions on Geoscience and Remote Sensing, 2025

    2025

  99. [107]

    Predicting dialogue acts for a speech-to-speech translation system

    Norbert Reithinger, Ralf Engel, Michael Kipp, and Martin Klesen. Predicting dialogue acts for a speech-to-speech translation system. In Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP’96, volume 2, pages 654–657. IEEE, 1996

    1996

  100. [108]

    Treatment of incomplete dialogues in a speech-to-speech translation system

    Norbert Reithinger, Elisabeth Maier, and Jan Alexandersson. Treatment of incomplete dialogues in a speech-to-speech translation system. 1995

    1995

  101. [109]

    Text-to-speech systems for filipino using unit selection and deep learning

    Edsel Jedd Renovalles, Crisron Rudolf Lucas, Franz de Leon, Angelina Aquino, and Izza Jalandoni. Text-to-speech systems for filipino using unit selection and deep learning. In 2021 24th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standa...

    2021

  102. [110]

    Audiopalm: A large language model that can speak and listen

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zal´ an Borsos, F´ elix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925 , 2023

    2023 arXiv

  103. [111]

    Fluent translations from disfluent speech in end-to-end speech translation

    Elizabeth Salesky, Matthias Sperber, and Alex Waibel. Fluent translations from disfluent speech in end-to-end speech translation. arXiv preprint arXiv:1906.00556 , 2019

    1906 arXiv

  104. [112]

    Image, text, and speech data augmentation using multimodal llms for deep learning: A survey

    Ranjan Sapkota, Shaina Raza, Maged Shoman, Achyut Paudel, and Manoj Karkee. Image, text, and speech data augmentation using multimodal llms for deep learning: A survey. arXiv preprint arXiv:2501.18648, 2025. 57

    2025 arXiv

  105. [113]

    Direct speech to speech translation: A review

    Mohammad Sarim, Saim Shakeel, Laeeba Javed, Mohammad Nadeem, et al. Direct speech to speech translation: A review. arXiv preprint arXiv:2503.04799 , 2025

    2025 arXiv

  106. [114]

    Acoustic and language modeling of human and nonhuman noises for human-to-human spontaneous speech recognition

    Tanja Schultz and Ivica Rogina. Acoustic and language modeling of human and nonhuman noises for human-to-human spontaneous speech recognition. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 293–296. IEEE, 1995

    1995

  107. [115]

    A faster approach for direct speech to speech translation

    Rashmi T Shankarappa and Sourabh Tiwari. A faster approach for direct speech to speech translation. In 2022 IEEE Women in Technology Conference (WINTECHCON) , pages 1–6. IEEE, 2022

    2022

  108. [116]

    Generative adversarial defense via conditional diffusion model

    Xiaowen Shi, Chao Zhou, and Yuan-Gen Wang. Generative adversarial defense via conditional diffusion model. Multimedia Systems, 31(1):1–14, 2025

    2025

  109. [117]

    Gsqa: An end-to-end model for generative spoken question answering

    Min-Han Shih, Ho-Lam Chung, Yu-Chi Pai, Ming-Hao Hsu, Guan-Ting Lin, Shang-Wen Li, and Hung-yi Lee. Gsqa: An end-to-end model for generative spoken question answering. arXiv preprint arXiv:2312.09781, 2023

    2023 arXiv

  110. [118]

    Spoken language translation in low-resource language

    S Shoba, A Sasithradevi, and S Deepa. Spoken language translation in low-resource language. Auto- matic Speech Recognition and Translation for Low Resource Languages, pages 445–459, 2024

    2024

  111. [119]

    Of speech and time: Temporal speech patterns in interpersonal contexts

    Aron W Sieqman and Stanley Feldstein. Of speech and time: Temporal speech patterns in interpersonal contexts. 1979

    1979

  112. [120]

    A comparative study of llm-based asr and whisper in low resource and code switching scenario

    Zheshu Song, Ziyang Ma, Yifan Yang, Jianheng Zhuo, and Xie Chen. A comparative study of llm-based asr and whisper in low resource and code switching scenario. arXiv preprint arXiv:2412.00721 , 2024

    2024 arXiv

  113. [121]

    Ssr: Alignment-aware modality connector for speech language models

    Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden Tomasello, and Xutai Ma. Ssr: Alignment-aware modality connector for speech language models. arXiv preprint arXiv:2410.00168 , 2024

    2024 arXiv

  114. [122]

    Generative diffusion models: Principles and applications

    Akinori Tanaka. Generative diffusion models: Principles and applications. Journal of the Physical Society of Japan , 94(3):031009, 2025

    2025

  115. [123]

    Machine intelligence in africa: a survey

    Allahsera Auguste Tapo, Ali Traore, Sidy Danioko, and Hamidou Tembine. Machine intelligence in africa: a survey. DSAI, 2024

    2024

  116. [124]

    Sawa Editions, Bamako, Mali, 2023

    Hamidou Tembine, Modibo Bouare, Mariam Dembele, Abdoulaye Diallo, Boubacar Diallo, Abdoulaye Diarra, Bourama Doumbia, Ndeye Molinier, Astou Sidibe, Allahsera Tapo, and Sidy Danioko.Machine Intelligence in Africa in 20 Questions . Sawa Editions, Bamako, Mali, 2023. Published in...

    2023

  117. [125]

    Mean-field-type transformers

    Hamidou Tembine, Manzoor Ahmed Khan, and Issa Bamia. Mean-field-type transformers. Mathemat- ics, 12(22):3506, 2024

    2024

  118. [126]

    Speech-to-speech translation without text

    Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Speech-to-speech translation without text. 2020

    2020

  119. [127]

    Towards a speech-to-speech translation system

    Masaru Tomita, Marion Kee, Hiroaki Saito, Teruko Mitamura, and Hideto Tomabechi. Towards a speech-to-speech translation system. Interface: Journal of Applied Linguistics , 3(1):57–77, 1988

    1988

  120. [128]

    Attacks and defenses for generative diffusion models: A comprehensive survey

    Vu Tuan Truong, Luan Ba Dang, and Long Bao Le. Attacks and defenses for generative diffusion models: A comprehensive survey. ACM Computing Surveys , 57(8):1–44, 2025

    2025

  121. [129]

    Last: Language model aware speech tokenization

    Arnon Turetzky and Yossi Adi. Last: Language model aware speech tokenization. arXiv preprint arXiv:2409.03701, 2024

    2024 arXiv

  122. [130]

    Whisper-gpt: A hybrid representation audio large language model

    Prateek Verma. Whisper-gpt: A hybrid representation audio large language model. arXiv preprint arXiv:2412.11449, 2024

    2024 arXiv

  123. [131]

    Whisper-gpt: A hybrid generative llm for speech and music

    Prateek Verma. Whisper-gpt: A hybrid generative llm for speech and music. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2025. 58

    2025

  124. [132]

    Finite-state speech-to-speech translation

    Enrique Vidal. Finite-state speech-to-speech translation. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 111–114. IEEE, 1997

    1997

  125. [133]

    Quality over quantity? llm-based cura- tion for a data-efficient audio-video foundation model

    Ali Vosoughi, Dimitra Emmanouilidou, and Hannes Gamper. Quality over quantity? llm-based cura- tion for a data-efficient audio-video foundation model. arXiv preprint arXiv:2503.09205 , 2025

    2025

  126. [134]

    Mobile speech-to-speech translation of spontaneous dialogs: An overview of the final verbmobil system

    Wolfgang Wahlster. Mobile speech-to-speech translation of spontaneous dialogs: An overview of the final verbmobil system. Verbmobil: Foundations of speech-to-speech translation, pages 3–21, 2000

    2000

  127. [135]

    Speechalator: Two-way speech-to- speech translation in your hand

    Alex Waibel, Ahmed Badran, Alan W Black, Robert Frederking, Donna Gates, Alon Lavie, Lori Levin, Kevin Lenzo, Laura Mayfield Tomokiyo, Juergen Reichert, et al. Speechalator: Two-way speech-to- speech translation in your hand. In Companion Volume of the Proceedings of HLT-NAACL...

    2003

  128. [136]

    Janus: Speech-to-speech translation using connectionist and non-connectionist techniques

    Alex Waibel, Ajay Jain, Arthur McNair, Joe Tebelskis, Louise Osterholtz, Hiroaki Saito, Otto Schmid- bauer, Tilo Sloboda, and Monika Woszczyna. Janus: Speech-to-speech translation using connectionist and non-connectionist techniques. Advances in neural information processing s...

    1991

  129. [137]

    Janus: a speech-to-speech translation system using connectionist and symbolic processing strate- gies

    Alex Waibel, Ajay N Jain, Arthur E McNair, Hiroaki Saito, Alexander G Hauptmann, and Joe Tebel- skis. Janus: a speech-to-speech translation system using connectionist and symbolic processing strate- gies. In Acoustics, speech, and signal processing, IEEE international conferen...

    1991

  130. [138]

    Audiobench: A universal benchmark for audio large language models

    Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen. Audiobench: A universal benchmark for audio large language models. arXiv preprint arXiv:2406.16020, 2024

    2024 arXiv

  131. [139]

    Why do speech language models fail to generate semantically coherent outputs? a modality evolving perspec- tive

    Hankun Wang, Haoran Wang, Yiwei Guo, Zhihan Li, Chenpeng Du, Xie Chen, and Kai Yu. Why do speech language models fail to generate semantically coherent outputs? a modality evolving perspec- tive. arXiv preprint arXiv:2412.17048 , 2024

    2024

  132. [140]

    Parrot: Seamless spoken dialogue interaction with double- channel large language models

    Qichao Wang, Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Zibin Zheng, Irwin King, Liang Chen, and Peilin Zhao. Parrot: Seamless spoken dialogue interaction with double- channel large language models. 2025

    2025

  133. [141]

    Generative ui design with diffusion models: Exploring automated interface creation and human-computer interaction

    Shixiao Wang, Runsheng Zhang, and Xue Shi. Generative ui design with diffusion models: Exploring automated interface creation and human-computer interaction. Transactions on Computational and Scientific Methods, 5(3), 2025

    2025

  134. [142]

    Enabling auditory large language models for automatic speech quality evaluation

    Siyin Wang, Wenyi Yu, Yudong Yang, Changli Tang, Yixuan Li, Jimin Zhuang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, et al. Enabling auditory large language models for automatic speech quality evaluation. In ICASSP 2025-2025 IEEE International Conference on Acoustic...

    2025

  135. [143]

    What are they doing? joint audio-speech co-reasoning

    Yingzhi Wang, Pooneh Mousavi, Artem Ploujnikov, and Mirco Ravanelli. What are they doing? joint audio-speech co-reasoning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2025

    2025

  136. [144]

    Rnadifffold: gener- ative rna secondary structure prediction using discrete diffusion models

    Zhen Wang, Yizhen Feng, Qingwen Tian, Ziqi Liu, Pengju Yan, and Xiaolin Li. Rnadifffold: gener- ative rna secondary structure prediction using discrete diffusion models. Briefings in Bioinformatics , 26(1):bbae618, 2025

    2025

  137. [145]

    Muchomusic: Evaluating music understanding in multimodal audio-language models

    Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, and Dmitry Bogdanov. Muchomusic: Evaluating music understanding in multimodal audio-language models. arXiv preprint arXiv:2408.01337, 2024

    2024 arXiv

  138. [146]

    Improving textless spoken language understanding with discrete units as intermediate target

    Guan-Wei Wu, Guan-Ting Lin, Shang-Wen Li, and Hung-yi Lee. Improving textless spoken language understanding with discrete units as intermediate target. arXiv preprint arXiv:2305.18096 , 2023. 59

    2023 arXiv

  139. [147]

    Speechgen: Unlocking the generative power of speech language models with prompts

    Haibin Wu, Kai-Wei Chang, Yuan-Kuei Wu, and Hung-yi Lee. Speechgen: Unlocking the generative power of speech language models with prompts. arXiv preprint arXiv:2306.02207 , 2023

    2023 arXiv

  140. [148]

    Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration

    Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu. Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration. arXiv preprint arXiv:2501.14350 , 2025

    2025 arXiv

  141. [149]

    Towards diverse and efficient audio captioning via diffusion models

    Manjie Xu, Chenxing Li, Xinyi Tu, Yong Ren, Ruibo Fu, Wei Liang, and Dong Yu. Towards diverse and efficient audio captioning via diffusion models. arXiv preprint arXiv:2409.09401 , 2024

    2024 arXiv

  142. [150]

    Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner

    Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjie Huang, Xiang Li, Xu Tan, Xixin Wu, and Helen Meng. Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner. arXiv preprint arXiv:2406.10056 , 2024

    2024 arXiv

  143. [151]

    Audio is the achilles’ heel: Red teaming audio large multimodal models

    Hao Yang, Lizhen Qu, Ehsan Shareghi, and Gholamreza Haffari. Audio is the achilles’ heel: Red teaming audio large multimodal models. arXiv preprint arXiv:2410.23861 , 2024

    2024 arXiv

  144. [152]

    Torchaudio: Building blocks for audio and speech processing

    Yao-Yuan Yang, Moto Hira, Zhaoheng Ni, Artyom Astafurov, Caroline Chen, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z Yang, et al. Torchaudio: Building blocks for audio and speech processing. In ICASSP 2022-2022 IEEE International Conference on Ac...

    2022

  145. [153]

    When large language models meet speech: A survey on integration approaches

    Zhengdong Yang, Shuichiro Shimizu, Yahan Yu, and Chenhui Chu. When large language models meet speech: A survey on integration approaches. arXiv preprint arXiv:2502.19548 , 2025

    2025 arXiv

  146. [154]

    Where visual speech meets lan- guage: Vsp-llm framework for efficient and context-aware visual speech processing

    Jeong Hun Yeo, Seunghee Han, Minsu Kim, and Yong Man Ro. Where visual speech meets lan- guage: Vsp-llm framework for efficient and context-aware visual speech processing. arXiv preprint arXiv:2402.15151, 2024

    2024 arXiv

  147. [155]

    Zero-avsr: Zero-shot audio-visual speech recognition with llms by learning language-agnostic speech representations

    Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, and Yong Man Ro. Zero-avsr: Zero-shot audio-visual speech recognition with llms by learning language-agnostic speech representations. arXiv preprint arXiv:2503.06273, 2025

    2025 arXiv

  148. [156]

    L. C. Young. An inequality of the h¨ older type, connected with stieltjes integration.Acta Mathematica, 67:251–282, 1936

    1936

  149. [157]

    Nonequilbrium physics of generative diffusion models

    Zhendong Yu and Haiping Huang. Nonequilbrium physics of generative diffusion models. Physical Review E, 111(1):014111, 2025

    2025

  150. [158]

    Uwspeech: Speech to speech translation for unwritten languages

    Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie-Yan Liu. Uwspeech: Speech to speech translation for unwritten languages. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 14319–14327, 2021

    2021

  151. [159]

    Text-to-image diffusion models in generative ai: A survey

    Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in generative ai: A survey. arXiv preprint arXiv:2303.07909 , 2023

    2023 arXiv

  152. [160]

    Llm-enhanced dialogue management for full-duplex spoken dialogue systems

    Hao Zhang, Weiwei Li, Rilin Chen, Vinay Kothapally, Meng Yu, and Dong Yu. Llm-enhanced dialogue management for full-duplex spoken dialogue systems. arXiv preprint arXiv:2502.14145 , 2025

    2025 arXiv

  153. [161]

    Direct speech-to-speech translation without textual annotation using bottleneck features

    Junhui Zhang, Junjie Pan, Xiang Yin, and Zejun Ma. Direct speech-to-speech translation without textual annotation using bottleneck features. arXiv preprint arXiv:2212.05805 , 2022

    2022 arXiv

  154. [162]

    Intrinsicvoice: Empowering llms with intrinsic real-time voice interaction abilities

    Xin Zhang, Xiang Lyu, Zhihao Du, Qian Chen, Dong Zhang, Hangrui Hu, Chaohong Tan, Tianyu Zhao, Yuxuan Wang, Bin Zhang, et al. Intrinsicvoice: Empowering llms with intrinsic real-time voice interaction abilities. arXiv preprint arXiv:2410.08035 , 2024

    arXiv

  155. [163]

    Textless streaming speech-to-speech translation using semantic speech tokens

    Jinzheng Zhao, Niko Moritz, Egor Lakomkin, Ruiming Xie, Zhiping Xiu, Katerina Zmolikova, Zeeshan Ahmed, Yashesh Gaur, Duc Le, and Christian Fuegen. Textless streaming speech-to-speech translation using semantic speech tokens. In ICASSP 2025-2025 IEEE International Conference o...

    2025

  156. [164]

    Diffs2ut: A semantic preserving diffusion model for textless direct speech-to-speech translation

    Yongxin Zhu, Zhujin Gao, Xinyuan Zhou, Zhongyi Ye, and Linli Xu. Diffs2ut: A semantic preserving diffusion model for textless direct speech-to-speech translation. arXiv preprint arXiv:2310.17570, 2023

    2023 arXiv

  157. [165]

    Automatic speech recognition for low-resource languages using wav2vec2: Modern standard arabic (msa) as an example of a low-resource language, 2021

    Taha Zouhair. Automatic speech recognition for low-resource languages using wav2vec2: Modern standard arabic (msa) as an example of a low-resource language, 2021. 61

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.