Direct Speech-to-Speech Neural Machine Translation: A Survey

Chandresh Kumar Maurya; Mahendra Gupta; Maitreyee Dutta

REVIEW 4 major objections 5 minor 4 cited by

Direct Speech-to-Speech Neural Machine Translation: A Survey

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Direct speech-to-speech translation is close to matching cascade quality on benchmark pairs, this survey argues.

desk verdict A useful survey of direct S2ST with a sensible taxonomy and a small controlled benchmark, but the 'gap almost closed' claim outruns the evidence, and table inconsistencies need fixing. read the letter →

arxiv 2411.14453 v1 pith:DCVPF4PP submitted 2024-11-13 cs.CL cs.SDeess.AS

Mahendra Gupta , Maitreyee Dutta , Chandresh Kumar Maurya This is my paper · ORCID

classification cs.CLcs.SDeess.AS

keywords directspeech-to-speechtranslationend-to-endspeechcascadediscreteunitstextlesssimultaneousLLM-basedevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct speech-to-speech translation (S2ST) aims to convert audio in one language into audio in another without first writing down an intermediate transcription. This paper argues that the field has reached a turning point: on some benchmark language pairs the quality gap between direct and cascade models—the traditional pipeline of speech recognition, machine translation, and speech synthesis—has almost closed, while direct models additionally preserve voice, emotion, and prosody and can in principle serve languages with no writing system. The survey organizes the literature into offline, simultaneous, and LLM-based model families, reviews strategies for coping with scarce parallel speech data, and catalogs evaluation metrics and remaining challenges. A reader should care because the claim, if correct, means end-to-end speech translation is becoming a practical alternative for real-time dubbing, conference interpretation, and unwritten-language communication.

What carries the argument

The load-bearing machinery is the survey's taxonomy and its benchmark comparison. Direct S2ST models are split into offline models, which decode after the whole utterance; simultaneous models, which begin decoding on partial input; and LLM-based models, which prompt a speech language model with discrete speech units. Underneath that taxonomy, the performance story is carried by a few architectural choices: discrete-unit speech representations (typically a self-supervised speech encoder such as HuBERT clustered with k-means) used as decoder targets, two-pass decoding that separates linguistic and acoustic generation, multitask learning from text transcripts, pre-training and data augmentation to compensate for scarce parallel speech, and a vocoder for final waveform synthesis. The tables assemble reported quality scores and the authors' own re-implementations to argue that the direct/cascade gap has narrowed.

What would settle it

Take any language pair where the survey reports near-cascade direct performance, train or fine-tune the same direct and cascade systems on identical parallel speech data, then score both with the same automatic scoring pipeline and the same human listening test; if direct systems fail to match cascade quality across several such pairs, the paper's central comparison collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that direct S2ST has matured from a proof-of-concept into a family of architectures that, on certain datasets, produce translation-quality and human-listening naturalness scores close to or better than cascade baselines, especially when the direct model uses discrete-unit targets, a two-pass decoder, multitask learning with text, or pre-trained encoders and decoders. The same evidence shows that textless direct models trained without any external text still lag behind cascades, and that simultaneous and LLM-based direct models are too few and too inconsistently evaluated to establish general conclusions. The survey also asserts that no earlier comprehensive survey of direct S2ST existed, so it supplies a taxonomy—offline, simultaneous, and LLM-based—plus benchmark tables and a list of open problems as a roadmap for future work.

Load-bearing premise

The load-bearing premise is that the translation-quality and naturalness scores collected from different papers, produced on different training sets and evaluated with different automatic or human-rating setups, can be compared as if they came from one experiment, and that the authors' own small-scale re-runs fairly stand in for the original models.

Editorial extensions

If this is right

If the performance-gap claim holds, direct S2ST becomes a competitive default for high-resource spoken translation, weakening the argument for maintaining a separate ASR, MT, and TTS pipeline.
Unwritten languages become a realistic target: textless direct models can be trained and used where no transcripts exist, provided enough parallel or mined speech data can be obtained.
Evaluation practice would need to shift from scoring generated speech by first transcribing it toward text-free metrics, because transcription both biases quality scores and is unavailable for many languages.
Simultaneous and LLM-based direct S2ST, though promising, should be judged on standardized datasets and latency measures before they can be fairly compared with offline or cascade systems.
The near-closed gap is currently established only on a handful of language pairs; large-scale multi-pair comparisons are the direct next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that, if the gap is truly closed on several pairs, the strongest remaining reason to prefer direct models is not raw translation quality but the preservation of voice, emotion, timing, and code-switching behavior, so evaluations should include speaker-similarity and prosody measures.
The authors' Table 3 re-implementations were trained from scratch on a single GPU; a plausible engineering inference is that direct models are more data-hungry than cascades, meaning advances in speech mining and synthetic target speech may push direct models past cascade performance faster than architectural tweaks alone.
A testable extension is to apply the same comparison protocol to distant language pairs and low-resource or unwritten languages, where the cascade pipeline often lacks an ASR or TTS component entirely and direct models may show their largest advantage.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a survey of direct speech-to-speech translation with a three-way taxonomy (offline, simultaneous, LLM-based) and a small original benchmark run on CVSS-C Es->En. The survey part is competently organized and covers the data-scarcity tricks, metrics, and architectural variations well. For a beginner, it's a reasonable entry point. The taxonomy is current and the LLM-based category is a useful addition. The authors also deserve credit for attempting a controlled comparison in Table 3, even though it only reimplements four older models.

The soft spots are real but mostly fixable. The headline claim that the direct-vs-cascade quality gap is 'now almost closed' is not backed by the authors' own controlled setup. Their Table 3 shows a 9-point BLEU gap between the best direct model (23.47) and an ASR+MT+TTS cascade (32.6) on the same data. The models that supposedly close the gap—UnitY and the LLM-based systems—are not in Table 3, so the claim rests entirely on cross-paper numbers that are not directly comparable (different training data, different ASR for BLEU computation, different cascade baselines). Tempering that claim or adding UnitY to the controlled run would be the main fix.

There are also several internal inconsistencies: Table 2 and Table 3 reuse IDs 43-44 for different papers; section 10 references 'ID: 42-48' while the table lists 43-52; the discussion of 'ID: 46' appears to mix up the S2UT and Translatotron rows. These are minor, but they make the comparison hard to audit. The 'first comprehensive survey' claim is also not verified against earlier surveys, and there are duplicated references (Chen et al. 2023a/b, Hsu et al. 2021a/b).

None of this undercuts the organizational value of the survey. The general conclusion that direct models still lag behind cascades on quality is actually supported by their data; it's the 'almost closed' framing that goes beyond the evidence. With a revision that fixes the table IDs and softens the gap claim, this would be a usable reference.

I'd send it to review rather than desk-reject—it deserves referee time. The errors are fixable, and the field could use an up-to-date survey entry point.

Referee Report

4 major / 5 minor

Summary. This paper is a survey of direct speech-to-speech translation (S2ST). It organizes the literature into offline, simultaneous, and LLM-based models; discusses data-scarcity strategies, representation learning, segmentation, evaluation metrics, training strategies, and application issues; and presents a new benchmark experiment on the CVSS-C Es→En pair in Table 3. The paper's central comparative claim, stated in §3 and revisited in §10, is that the quality gap between direct and cascade S2ST models has become 'almost closed' on some benchmark language pairs, while acknowledging that direct models still face data scarcity, evaluation, and latency challenges.

Significance. If its claims hold, the survey would be a useful entry point for researchers and practitioners, with a reasonable taxonomy and broad coverage. The authors make a commendable effort to include a controlled experiment (Table 3) rather than only re-reporting published numbers, and the discussion of textless metrics and unwritten languages is valuable. However, the central empirical claim is not fully substantiated: the controlled experiment omits the very models that support the 'gap almost closed' conclusion, and the cross-paper BLEU comparison in Tables 1–2 is not controlled for evaluation pipeline differences. The paper is nevertheless a defensible survey contribution once the empirical claims are appropriately qualified and the internal inconsistencies are fixed.

major comments (4)

[§3 and §10, Tables 1 and 3] The central claim that the direct-vs-cascade performance gap 'is now almost closed' is not validated by the paper's own controlled experiment. Table 3 re-implements only older direct models (S2UT variants and Translatotron 1) and omits UnitY (Inaguma et al., 2023) and the LLM-based models of Table 2, which are exactly the systems that produce the competitive BLEU scores in Table 1 (e.g., IDs 14, 21, 26, 33). On the matched CVSS-C Es→En setup, the best direct model reaches BLEU 23.47 (ID 52) versus 32.6 for the implemented ASR+MT+TTS cascade (ID 43), a gap of about 9 BLEU points. The caveat in §3 that the surveyed studies are on limited language pairs partially mitigates the issue, but the section still presents the gap as almost closed; the controlled evidence in Table 3 undercuts that conclusion for the models that are actually compared under matched conditions.
[§10, Tables 2 and 3] The row IDs are internally inconsistent. The text says 'The models (ID: 42-48)' are trained, but Table 3 lists IDs 43–52. More seriously, IDs 43 and 44 in Table 2 denote Peng et al. (2024) and Gong and Veluri (2024), while in Table 3 the same IDs denote the ASR+MT+TTS and ST+TTS cascade baselines. This reuse of IDs across tables makes it impossible to trace a row to a cited work without guesswork; the tables need a single globally unique numbering scheme.
[§5 and Table 1] Table 1 pools BLEU and MOS scores from papers with different evaluation pipelines. BLEU for speech output is typically computed after ASR transcription, and §5 correctly states that the metric depends on ASR quality; the table, however, does not report the ASR system, the BLEU variant (SacreBLEU vs raw BLEU), or the MOS elicitation protocol for each row. Differences of a few BLEU points between direct and cascade rows (e.g., ID 14: 36.4 vs 32.9; ID 21: 33.1 vs 34.2) are within the range that pipeline differences could produce. The 'almost closed' claim should either be restricted to matched evaluations or accompanied by a robustness argument; as presented, the evidence cannot distinguish true gap closure from evaluation-pipeline variation.
[Table 3] The new experiment in Table 3 is reported without code, data splits, number of runs, or variance and significance measures. Single-run BLEU differences, especially for models trained from scratch on a single GPU, should not be read as a reliable ordering; for example, the difference between ID 49 and ID 50 (21.19 vs 20.09) is smaller than typical run-to-run variation in speech translation. The authors should either release the implementation or reduce the strength of the comparative statements they draw from these rows.

minor comments (5)

[§1 and §12] The introduction says the paper discusses 'challenges in §11, and finally concluded in §11', but the Conclusion is a separate §12; the section cross-reference should be corrected.
[§2, Eq. (1)] Equation (1) has notation problems: the product index is written as 'kY T=1' instead of a product over a lowercase index, and the left-hand side uses y while the factors use f^t_T; the definition of y as a full utterance should be reconciled with the per-frame factorization.
[§5] The metric name 'ScareBLEU' should be 'SacreBLEU', and the spacing in 'V oice Activity Detection (V AD)' should be fixed.
[References] Several references are duplicated: Chen et al. 2023a and 2023b are the same paper, Hsu et al. 2021a and 2021b are the same paper, and Kano et al. 2020a and 2020b are the same paper; these should be merged.
[Table 3 and Table 4] Table 3 contains the misspelling 'TF-Translatoron' for 'Translatotron', and Table 4 contains 'Geerman' for 'German' and 'FLEURES' for 'FLEURS'; the #Speakers entry for CVSS-C ('1 Canon. F') is also unclear.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is a survey with external citations and a separate benchmark table; no claim reduces to its own input.

full rationale

This paper is a literature survey of direct speech-to-speech translation, not a derivation or a fitted modeling claim. Its taxonomy, model descriptions, and performance comparisons in Tables 1 and 2 are supported by citations to external published systems and datasets, so the survey's statements are not restatements of its own inputs. The only original empirical content is Table 3, which reports the authors' single-GPU re-implementations of existing models on CVSS-C Es-to-En; this is a benchmark measurement produced by training runs, not a parameter fitted to the survey's conclusion. No equation in the paper defines a claimed result in terms of itself, no fitted parameter is renamed as a prediction, and no load-bearing argument depends on a self-citation from the present authors. The skeptical concern that cross-paper BLEU and MOS scores may not be directly comparable is a validity or auditability issue, not circularity, and the Table 3 row-ID conflict similarly affects verifiability without making the argument circular. Under the stated rules, a finding of no significant circularity with score 0 is the appropriate outcome.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey's conclusions assume that prior papers' reported numbers are accurate and comparable, that the selected papers represent the field, and that the authors' own re-implementations match the original methods. These are not derived in the paper. No free parameters or invented entities are introduced.

assumptions (3)

domain assumption Reported BLEU and MOS scores from cited papers are accurate and directly comparable.
Tables 1 and 2 aggregate numbers across papers with different datasets, ASR systems, and human evaluation setups; comparability is assumed, not established.
domain assumption The selected set of papers and datasets is representative of the direct S2ST field.
The taxonomy and conclusions depend on coverage; the authors include only models with results on standard datasets and state that some models could not be implemented.
domain assumption The authors' single-GPU re-implementations in Table 3 faithfully reproduce the original models.
The comparison uses 'provided hyperparameter settings' without listing them, and the authors acknowledge code reproducibility issues.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Direct Speech-to-Speech Neural Machine Translation: A Survey." pith.science (2026). https://pith.science/paper/DCVPF4PP

@misc{pith2026241114453,
  author       = {Pith},
  title        = {Pith review of: Direct Speech-to-Speech Neural Machine Translation: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCVPF4PP}},
  note         = {Machine review of arXiv:2411.14453}
}

read the original abstract

Speech-to-Speech Translation (S2ST) models transform speech from one language to another target language with the same linguistic information. S2ST is important for bridging the communication gap among communities and has diverse applications. In recent years, researchers have introduced direct S2ST models, which have the potential to translate speech without relying on intermediate text generation, have better decoding latency, and the ability to preserve paralinguistic and non-linguistic features. However, direct S2ST has yet to achieve quality performance for seamless communication and still lags behind the cascade models in terms of performance, especially in real-world translation. To the best of our knowledge, no comprehensive survey is available on the direct S2ST system, which beginners and advanced researchers can look upon for a quick survey. The present work provides a comprehensive review of direct S2ST models, data and application issues, and performance metrics. We critically analyze the models' performance over the benchmark datasets and provide research challenges and future directions.

Figures

Figures reproduced from arXiv: 2411.14453 by the authors.

**Figure 2.** Strategies for addressing data scarcity unlabeled datasets. Several pre-trained DL models, including BERT (Devlin et al., 2019), BART (Lewis et al., 2020) (using unit mBART version), Wav2Vec 2.0 (Baevski et al., 2020), HuBERT (Hsu et al., 2021a), w2v-BERT (Chung et al., 2021), and VQ-VAE (Oord et al., 2017), which are trained on extensive unlabeled data, serve as the foundation for various NLP and speech tasks. The… view at source ↗

**Figure 3.** Typical architecture of offline direct S2ST [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Locate-and-Focus: Enhancing Terminology Translation in Speech Language Models
cs.CL 2025-07 conditional novelty 6.0 of 10

Locate-and-Focus localizes the audio span of a terminology in an utterance and uses the located clip, a matched audio replacement, and a special <Term> cue to make speech LLMs translate the terminology correctly.
Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey
cs.CV 2026-04 unverdicted novelty 5.0 of 10

The paper offers the first focused review of MLLM-based video translation organized by a three-role taxonomy of Semantic Reasoner, Expressive Performer, and Visual Synthesizer, plus open challenges.
Breaking the Barriers of Text-Hungry and Audio-Deficient AI
cs.SD 2025-06 reject novelty 4.0 of 10

A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.
Language translation, and change of accent for speech-to-speech task using diffusion model
cs.CL 2025-05 reject novelty 3.0 of 10

A diffusion-based pipeline is proposed for simultaneous language translation and accent change, but only text-to-speech subtasks are evaluated and the combined S2ST result is not demonstrated.

Reference graph

Works this paper leans on

14 extracted references · 5 canonical work pages · cited by 4 Pith papers

[5]

In The Twelfth Interna- tional Conference on Learning Representations

Polyvoice: Language models for speech to speech translation. In The Twelfth Interna- tional Conference on Learning Representations. OpenReview.net. Qianqian Dong, Fengpeng Yue, Tom Ko, Mingx- uan Wang, Qibing Bai, and Yu Zhang. 2022. Leveraging pseudo-labeled data to improve di- rect speech-to-speech translation. In Pro- ceedings of InterSpeech 2022 , Inc...

arXiv 2022
[6]

arXiv preprint arXiv:2307.08655

Multilingual speech-to-speech trans- lation into multiple target languages. arXiv preprint arXiv:2307.08655. Hongyu Gong and Bandhav Veluri. 2024. Seam- lessexpressivelm: Speech language model for expressive speech-to-speech transla- tion with chain-of-thought. arXiv preprint arXiv:2405.20410. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Sc...

arXiv 2024
[11]

In Proceed- ings of the 58th Annual Meeting of the As- sociation for Computational Linguistics , pages 7871–7880, Online

BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceed- ings of the 58th Annual Meeting of the As- sociation for Computational Linguistics , pages 7871–7880, Online. Association for Computa- tional Linguistics. Xiang Lisa Li and Percy Liang. 2021. Prefix- tuning: Optimizing continuous ...

work page 2021
[12]

In 2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , pages 1–5, Rhodes Island, Greece

Textless direct speech-to-speech trans- lation with discrete speech representation. In 2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , pages 1–5, Rhodes Island, Greece. IEEE. Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-End Speech Translation with Knowle...

arXiv 2023
[13]

Bitext Mining for Low-Resource Languages via Contrastive Learning

A Survey of End-to-End Driving: Ar- chitectures and Training Methods. IEEE Trans- actions on Neural Networks and Learning Sys- tems, 33(4):1364–1384. Weiting Tan and Philipp Koehn. 2022. Bitext min- ing for low-resource languages via contrastive learning. arXiv preprint arXiv:2208.11194. Yun Tang, Hongyu Gong, Ning Dong, Chang- han Wang, Wei-Ning Hsu, Jia...

work page Pith review arXiv 2022
[14]

Direct" provide details about the direct models, while the columns under

Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03926. Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, Jiahong Yuan, Kenneth Church, and Liang Huang. 2020. Fluent and low-latency simul- taneous speech-to-speech translation with self- adaptive training. In Findings of the Associ- ation...

arXiv 2019
[1881]

Hirofumi Inaguma, Sravya Popuri, Ilia Ku- likov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino

Association for Computational Linguis- tics. Hirofumi Inaguma, Sravya Popuri, Ilia Ku- likov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023. UnitY: Two-pass direct speech-to-speech translation with discrete units. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

arXiv 2023
[1901]

Alexandre Bérard, Laurent Besacier, Ali Can Ko- cabiyikoglu, and Olivier Pietquin

Curran Associates, Inc. Alexandre Bérard, Laurent Besacier, Ali Can Ko- cabiyikoglu, and Olivier Pietquin. 2018. End- to-end automatic speech translation of audio- books. In 2018 IEEE International Confer- ence on Acoustics, Speech and Signal Process- ing (ICASSP), pages 6224–6228, Calgary, AB, Canada. IEEE. Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bans...

arXiv 2018

Show all 14 references

[1997]

In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 99–102, Munich, Germany

Janus-iii: speech-to-speech translation in multiple languages. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 99–102, Munich, Germany. IEEE. Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yous...

1997 arXiv
[2015]

arXiv preprint arXiv:1503.02531

Distilling the knowledge in a neural net- work. arXiv preprint arXiv:1503.02531. Wei Ning Hsu, Benjamin Bolte, Yao Hung Hu- bert Tsai, Kushal Lakhotia, Ruslan Salakhut- dinov, and Abdelrahman Mohamed. 2021a. Hubert: Self-supervised speech representation learning by masked pred...

2022 arXiv
[2020]

34th Conference on Neural Information Processing Systems (NeurIPS 2020)

Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech syn- thesis. 34th Conference on Neural Information Processing Systems (NeurIPS 2020). Patricia K. Kuhl, Barbara T. Conboy, Sharon Coffey-Corina, Denise Padden, Maritza Rivera- Gaxiola, and Tobey...

2020
[2022]

In 2022 IEEE Spoken Language Technology Workshop, SLT 2022 - Proceedings, pages 798–805

Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop, SLT 2022 - Proceedings, pages 798–805. IEEE. Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidire...

2022 arXiv
[2023]

arXiv preprint arXiv:2312.04691, abs/2312.04691

Simul-llm: A framework for ex- ploring high-quality simultaneous translation with large language models. arXiv preprint arXiv:2312.04691, abs/2312.04691. Alex Agranovich, Eliya Nachmani, Oleg Ry- bakov, Yifan Ding, Ye Jia, Nadav Bar, Heiga Zen, and Michelle Tadmor Ramanovich

arXiv
[2024]

arXiv preprint arXiv:2406.02133

Simultron: On-device simultaneous speech to speech translation. arXiv preprint arXiv:2406.02133. Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz Google, Ruoming Pang, Wei Li, and Colin Raffel. 2019. Monotonic infinite lookback at- tention for...

2019 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [5]

In The Twelfth Interna- tional Conference on Learning Representations

Polyvoice: Language models for speech to speech translation. In The Twelfth Interna- tional Conference on Learning Representations. OpenReview.net. Qianqian Dong, Fengpeng Yue, Tom Ko, Mingx- uan Wang, Qibing Bai, and Yu Zhang. 2022. Leveraging pseudo-labeled data to improve di- rect speech-to-speech translation. In Pro- ceedings of InterSpeech 2022 , Inc...

arXiv 2022

[2] [6]

arXiv preprint arXiv:2307.08655

Multilingual speech-to-speech trans- lation into multiple target languages. arXiv preprint arXiv:2307.08655. Hongyu Gong and Bandhav Veluri. 2024. Seam- lessexpressivelm: Speech language model for expressive speech-to-speech transla- tion with chain-of-thought. arXiv preprint arXiv:2405.20410. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Sc...

arXiv 2024

[3] [11]

In Proceed- ings of the 58th Annual Meeting of the As- sociation for Computational Linguistics , pages 7871–7880, Online

BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceed- ings of the 58th Annual Meeting of the As- sociation for Computational Linguistics , pages 7871–7880, Online. Association for Computa- tional Linguistics. Xiang Lisa Li and Percy Liang. 2021. Prefix- tuning: Optimizing continuous ...

work page 2021

[4] [12]

In 2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , pages 1–5, Rhodes Island, Greece

Textless direct speech-to-speech trans- lation with discrete speech representation. In 2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , pages 1–5, Rhodes Island, Greece. IEEE. Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-End Speech Translation with Knowle...

arXiv 2023

[5] [13]

Bitext Mining for Low-Resource Languages via Contrastive Learning

A Survey of End-to-End Driving: Ar- chitectures and Training Methods. IEEE Trans- actions on Neural Networks and Learning Sys- tems, 33(4):1364–1384. Weiting Tan and Philipp Koehn. 2022. Bitext min- ing for low-resource languages via contrastive learning. arXiv preprint arXiv:2208.11194. Yun Tang, Hongyu Gong, Ning Dong, Chang- han Wang, Wei-Ning Hsu, Jia...

work page Pith review arXiv 2022

[6] [14]

Direct" provide details about the direct models, while the columns under

Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03926. Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, Jiahong Yuan, Kenneth Church, and Liang Huang. 2020. Fluent and low-latency simul- taneous speech-to-speech translation with self- adaptive training. In Findings of the Associ- ation...

arXiv 2019

[7] [1881]

Hirofumi Inaguma, Sravya Popuri, Ilia Ku- likov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino

Association for Computational Linguis- tics. Hirofumi Inaguma, Sravya Popuri, Ilia Ku- likov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023. UnitY: Two-pass direct speech-to-speech translation with discrete units. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

arXiv 2023

[8] [1901]

Alexandre Bérard, Laurent Besacier, Ali Can Ko- cabiyikoglu, and Olivier Pietquin

Curran Associates, Inc. Alexandre Bérard, Laurent Besacier, Ali Can Ko- cabiyikoglu, and Olivier Pietquin. 2018. End- to-end automatic speech translation of audio- books. In 2018 IEEE International Confer- ence on Acoustics, Speech and Signal Process- ing (ICASSP), pages 6224–6228, Calgary, AB, Canada. IEEE. Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bans...

arXiv 2018