Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation

Benyou Wang; Jingbo Zhu; Kaiqi Kou; Peizhuo Liu; Tong Xiao; Weiqiao Shan; Xiangnan Ma; Yuhao Zhang; Yuxin Huang; Zhengtao Yu

REVIEW 4 major objections 5 minor 34 references

Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an unsupervised pseudo-text mined from discrete speech units — the 'unit language' — lets a textless speech-to-speech translation model match the BLEU of a model trained with real text on VoxPopuli.

desk verdict A genuinely new pseudo-text construction for textless S2ST with modest, single-run gains; worth a serious referee if the authors address variance and the norm-unit dependency. read the letter →

arxiv 2505.15333 v1 pith:PN3FT5ZE submitted 2025-05-21 cs.CL cs.AIcs.SDeess.AS

Yuhao Zhang , Xiangnan Ma , Kaiqi Kou , Peizhuo Liu , Weiqiao Shan , Benyou Wang , Tong Xiao , Yuxin Huang

show 2 more authors

Zhengtao Yu Jingbo Zhu

This is my paper

classification cs.CLcs.AIcs.SDeess.AS

keywords speech-to-speechtranslationtextlessS2STunitlanguagen-grammodelingmulti-tasklearningtaskpromptdiscretespeechunitsVoxPopuli

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Textless speech-to-speech translation (S2ST) usually has to learn both what speech means and how languages align without any written text. This paper claims that a text-like tokenization of discrete speech units, which it calls a unit language, can provide that missing guidance. The unit language is mined unsupervised with 2-gram language modeling: contiguous speech units are grouped into pseudo-words by maximizing their probability, giving a compressed, text-like sequence. The authors train their textless S2ST model with two auxiliary losses — one predicting the source unit language to sharpen acoustic-to-meaning (cross-modal) learning, one predicting the target unit language to sharpen language alignment (cross-lingual learning) — and add task prompts to stop these two losses from fighting. On four VoxPopuli translation directions, the method reaches 21.5 average BLEU, matching a model trained with real text and beating the 20.3 strong baseline.

What carries the argument

The load-bearing object is the unit language, a text-like transcription built by n-gram language modeling over discrete speech units. Given a unit sequence, the algorithm merges at most K contiguous units into pseudo-words, choosing the segmentation that maximizes the sequence probability under a 2-gram model whose conditional probabilities are estimated by counting in the unit corpus; equations (4)-(8) give the dynamic-programming recursion. The pseudo-words are then fed to two extra decoder heads: a source text decoder for the cross-modal loss and a target text decoder for the cross-lingual loss, with a learnable task prompt (bCM or bCL) inserted at the r-th encoding layer to keep the two tasks from interfering. The unit language's compression (sequence length between characters and text) is what makes alignment tractable, and the norm-unit preprocessing makes the units clean enough for the n-gram statistics to be meaningful.

What would settle it

Run the full recipe on a language pair where no speaker-normalization model exists, using raw pretrained discrete units only; if the average BLEU gain over the baseline stays near the +0.2 reported in Table 11 rather than +0.9, the unit-language gains are driven by the unit normalizer, not by the n-gram pseudo-text. A second check is to vary K and r as in Tables 4 and 5 and see whether any setting on a new pair recovers the +0.9 gain.

Watch

Extended reading notes

Core claim

The central claim is that unsupervised pseudo-text can substitute for real transcription in S2ST training. The unit language groups every sequence of discrete speech units into pseudo-words via a maximum-likelihood n-gram segmentation (2-gram, at most K=3 units per word), using dynamic programming over the recursion in equations (4) and (7). When these pseudo-words are used as auxiliary decoder targets — the source unit language for cross-modal modeling and the target unit language for cross-lingual modeling — the textless model improves by 1.2 average BLEU over the reproduced strong textless baseline and reaches parity with the same model trained on recognized text (21.5 vs. 21.5). The paper further claims that the two auxiliary tasks operate differently — CM filters noise, CL captures semantics — and that their simultaneous use conflicts; task-prompt modeling, two learnable vectors inserted at the r-th encoder layer with a negative mean-square-error diversity term, resolves the conflict and yields the best results. The authors also report that the gain depends on speaker-normalized units: without normalization the average improvement falls from +0.9 to +0.2 BLEU.

Load-bearing premise

The entire improvement rests on having speaker-normalized discrete units; when the unit language is built from unnormalized units, the average BLEU gain collapses from +0.9 to +0.2, so if normalization is unavailable or fails to transfer to a new language the method's benefit mostly disappears even though the algorithm is unchanged.

Editorial extensions

If this is right

If the claim holds, textless S2ST no longer needs real transcriptions to reach text-level BLEU: the unit language provides the same guidance that recognized text provides.
The conflict between cross-modal and cross-lingual auxiliary losses is real and general, and task prompts offer a mechanism to reconcile depth-split supervision in a single encoder.
Unit language is a drop-in auxiliary target: source unit language improves noise filtering, target unit language improves semantic alignment, and both contribute to the full gain.
Because the unit language compresses unit sequences toward text length, it can serve as pseudo-text for any speech sequence, not only translation pairs.
Gains depend on speaker-normalized units, so the method inherits the portability of unit normalization; languages without a norm-unit model may need an unsupervised normalizer first.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the same n-gram mining could produce pseudo-text for other speech tasks — speech summarization, speech-to-text translation without transcripts, and TTS from unwritten languages — since the segmentation is unsupervised and language-agnostic.
Editorial extension: the observed CM/CL conflict suggests a design rule for multi-task speech models: inject text-like supervision at different encoder depths, with task prompts and a diversity penalty as one generalizable instance.
Editorial extension: a testable next step is higher-order n-grams with pruning; the paper keeps to 2-gram for computational cost (K=4 already needs about two days), and a pruned 3-gram unit language could compress further and improve cross-lingual alignment.
Editorial extension: the claimed parity with text-trained models should be checked beyond BLEU; the paper itself notes the lack of human evaluation of tone and fluency.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper introduces a 'unit language'—an unsupervised n-gram segmentation of discrete speech units into pseudo-words, used as auxiliary multi-task supervision in textless speech-to-speech translation. The construction, maximum-likelihood segmentation with dynamic programming, is genuinely different from the BPE-based pseudo-languages in prior work, and the authors include a reproduced BPE baseline as a control. That is the real contribution, and it is a reasonable one.

The experimental core is honest in scope: four VoxPopuli directions, consistent gains of 0.4–1.2 BLEU over a reproduced S2UT baseline, and performance matching text-supervised multi-task training. The unit language edges out BPE by about 0.5 BLEU on average. The sparseness and localness analyses are a bit hand-wavy but give a plausible mechanism for why cross-modal and cross-lingual losses help and why they conflict.

The soft spots are real but not fatal. The biggest is statistical: every number in the main tables is a single run with no error bars or significance tests. A 0.4–1.2 BLEU difference is often within seed-to-seed variance in S2ST, so the abstract's 'significant' overstates what is shown. Second, the method depends heavily on speaker-normalized units; Appendix G shows the average gain collapses to +0.2 BLEU without them. That makes the headline result conditional on a preprocessing step that may not transfer to other languages or datasets. Third, the task prompt uses a negative-weight MSE loss (-3.0) that is never ablated, so the final +1.2 average is partly driven by an unexamined hyperparameter. Fourth, the 'state-of-the-art' claim should be read narrowly: Seamless scores far higher on the same pairs, so this is SOTA only among textless baselines, and even that is caveated by the norm-unit dependency.

Bottom line: the underlying idea is new, clearly described, and the experiments include a useful control. The paper deserves a serious referee, but the authors should be asked for multi-seed runs, confidence intervals, an ablation of the negative prompt weight, and a qualified SOTA claim. People working on unit-based speech models or unwritten languages will get value from this.

Referee Report

4 major / 5 minor

Summary. The paper proposes a "unit language" for textless speech-to-speech translation (S2ST): discrete speech units are merged into pseudo-words via n-gram language modeling, yielding a text-like representation that requires no transcriptions. This unit language is used as auxiliary supervision through two additional decoders and losses, LCM (cross-modal) and LCL (cross-lingual), which guide the encoder of a S2UT-style model. Since applying source and target unit languages together initially degrades performance, the authors introduce learnable task prompts to reconcile the two tasks. Experiments on four VoxPopuli language pairs report average BLEU improvements of +0.4 to +1.2 over a reproduced strong baseline, reaching an average of 21.5 BLEU that is comparable to multi-task training with recognized text.

Significance. If the reported improvements are robust, the paper makes a useful contribution: it shows that an unsupervised, text-like tokenization of speech units can provide auxiliary supervision for textless S2ST, reducing the need for parallel text or ASR transcripts. The construction is self-supervised (the unit language is derived from the same unlabeled unit corpus used for training), and the main BLEU evaluations are on held-out test sets, so the reported gains are not circular. The paper also includes a controlled comparison against a BPE-based pseudo-language (Table 6), an analysis of hyperparameter sensitivity (Tables 4 and 5), and diagnostic sparseness/localness analyses intended to explain why CM and CL help. The main reservations are that all results come from single runs with no significance testing, the benefit is strongly dependent on external norm-unit preprocessing (Appendix G), and the final task-prompt loss includes an unablated negative-weight term.

major comments (4)

[Table 2 / Abstract] The abstract and Section 3.2 describe the gains as "significant" and "consistent", but Table 2 reports only single-run BLEU scores with no error bars, multiple seeds, or significance tests. The claimed average improvements of +0.4 to +1.2 BLEU are comparable in magnitude to the hyperparameter sensitivity shown in Tables 4 and 5 (e.g., En-Es +CM ranges from 22.3 to 23.5 as r varies from 4 to 2; K=2 versus K=3 changes En-Es +CM from 23.3 to 23.5). Without variance estimates or paired significance tests, the central quantitative claim is not supported. Please provide at least three seeds per configuration or bootstrap confidence intervals for the main comparisons.
[Appendix G, Table 11] The value of the proposed unit language is almost entirely contingent on norm-unit preprocessing: with non-normalized units, the average LCM&LCL gain drops from +0.9 to +0.2 BLEU (Table 11). Since norm units require an additional speaker-normalization model (Lee et al., 2022b) that may not exist or transfer to new or unwritten languages, the paper should explicitly scope the central claims to "unit language built on norm units". As written, the abstract and conclusion state a general claim about unit-language supervision that the non-norm condition does not support.
[Section 2.4, Figure 3] The task-prompt mechanism is underspecified, which harms reproducibility. The text says b_CM is concatenated at the first position and later "replaced" by b_CL after the r-th layer, but it does not state whether the prompt is an extra token in the sequence (changing length), whether it is added to all layer inputs or only the r-th layer, or how the replacement is implemented. In addition, the negative MSE loss between b_CM and b_CL with weight −3.0 is not ablated: without removing this term or varying its weight, we cannot tell whether the +1.2 BLEU of "+Task prompt" comes from the prompt mechanism itself or from this auxiliary loss. Please provide a precise formal definition and an ablation over the MSE weight (including 0).
[Section 3.2 / Introduction] The paper claims "new state-of-the-art performance on the textless S2ST task using the VoxPopuli dataset", but Table 2 only compares against the reproduced S2UT baseline and the text-based Seamless model. No published textless S2ST systems (e.g., UnitY or other direct S2ST methods evaluated on VoxPopuli) are included. To support the state-of-the-art claim, the authors should compare with published textless S2ST results on the same data or temper the claim to "comparable to S2UT with text supervision".

minor comments (5)

[Section 2.1, Eq. (8)] The conditional-probability notation in Eq. (8) is ambiguous: the index ranges in the numerator and denominator do not make the conditioning event explicit, and l is defined rather indirectly. Please rewrite with a clear definition of l and the segment lengths involved.
[Abstract and throughout] There are repeated typos such as "V oxpupil" (abstract) and "langauge" (Section 4.6). Please proofread the manuscript.
[Figure 4 caption] The caption label "FrameUnitCharUnit_lText" is difficult to parse; separate the entries as "Frame, Unit, Char, Unit_l, Text" for clarity.
[Table 6] The BPE comparison is performed for the LCM&LCL setting without the task prompt, while the final system includes the task prompt. Reporting BPE with the same task-prompt augmentation would make the comparison against the proposed method fairer.
[Sections 4.2–4.3] The sparseness and localness analyses use only 200 training samples and no error bars; they are suggestive diagnostics but should be described as such rather than as definitive explanations of the underlying mechanisms.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the unit language is a self-supervised auxiliary representation, and the reported BLEU gains are held-out evaluations that do not reduce by construction to fitted inputs.

full rationale

The paper's central claim is that an unsupervised unit language, obtained by n-gram language modeling over discrete speech units (Eqs. 2 and 6), improves textless S2ST when used as an auxiliary multi-task target (Eqs. 11--13). This is not circular: the unit language is constructed from the unlabeled unit corpus, not from the evaluation labels, and the reported BLEU scores in Tables 2, 4, 5, 6, 10, and 11 are held-out ASR-BLEU measurements rather than quantities fitted by the method. The auxiliary losses are trained on the same unit data used for the main S2ST objective, but that is self-supervision, not definitional circularity: no equation in the paper defines the unit language in terms of the downstream metric, and no fitted parameter is renamed as a prediction. Hyperparameters such as K, r, loss weights, and the negative prompt-difference weight are selected via ablations, which is standard model selection rather than a circular derivation. Appendix G shows sensitivity to the externally provided norm-unit preprocessing (Lee et al., 2022b), but this is a dependence on an external tool, not a circular import of the paper's own conclusion. The only self-citation is Zhang et al. (2023), used to motivate choosing an intermediate textual-encoder layer and the benefit of length reduction; this is a non-load-bearing design motivation, and the layer choice is independently ablated in Table 4. The central empirical claim therefore stands on independent held-out evaluation, and no load-bearing step reduces to its own inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on several premises the reader does not verify: that n-gram pseudo-words capture linguistic structure aligned with words, that the norm-unit preprocessing from prior work is available, that the 2-gram approximation with K=3 is adequate, and that the sparseness/localness metrics license the mechanistic story. There are five hand-set hyperparameters documented above; none has a formal derivation. No new physical entities are postulated; the 'unit language' is an algorithmic construct whose external validity is established only by this paper's experiments.

free parameters (5)

K (max units per pseudo-word) = 3
K=3 is used for all pairs in the main results, but Table 5 shows K=4 yields further small gains on En-Es (23.7 vs 23.5 for CM), so the value is tuned per language pair.
r (encoder layer used for source unit language decoder) = 2
r=2 is chosen; Table 4 shows r=0 harms performance (e.g., 17.9 vs 19.5 on En-Fr with CM), so the choice is load-bearing.
n-gram order = 2
2-gram modeling is chosen for computational tractability; higher-order n-grams are not evaluated (Appendix B), so the order is a free choice affecting unit language quality.
loss weights alpha, beta, gamma = 8, 8, 8
alpha is fixed at 8 and beta/gamma are set to 8 when activated (Eq. 13); no ablation is reported, so the balance is an unexamined free choice.
negative MSE weight for task prompt diversity = -3.0
The extra loss that keeps b_CM and b_CL diverse uses a weight of -3.0 (Section 2.4); this value is ad hoc and not ablated.

assumptions (4)

domain assumption The unit language is implicitly aligned with real text (Section 2.1, Figure 1).
The auxiliary supervision assumes n-gram pseudo-words map to meaningful linguistic units; the paper supports this with qualitative case studies (Table 3) and length statistics (Figure 4), not with direct alignment evidence.
domain assumption Norm unit preprocessing (Lee et al., 2022b) is required for the method to work.
Appendix G (Table 11) shows gains drop from +0.9 to +0.2 average BLEU without normalization; the method inherits this dependency on prior work.
ad hoc to paper 2-gram conditional independence over pseudo-words is a sufficient model of unit language (Section 2.1, Eqs. 7-8).
Chosen for tractability; higher-order n-grams are not evaluated, so the approximation is an untested modeling choice.
ad hoc to paper Sparseness and localness metrics reflect the intended CM and CL mechanisms (Sections 4.2-4.3).
The mechanistic interpretation (CM as noise filtering, CL as semantics) rests on these metrics computed on 200 training samples without error bars.

invented entities (1)

Unit language
purpose: A text-like pseudo-tokenization of discrete speech units used as auxiliary training targets (LCM, LCL) to guide cross-modal and cross-lingual modeling.
The construct is defined and evaluated only within this paper; no external benchmark independently validates that unit language behaves like text, beyond the authors' own BLEU and case-study results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation." pith.science (2026). https://pith.science/paper/PN3FT5ZE

@misc{pith2026250515333,
  author       = {Pith},
  title        = {Pith review of: Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PN3FT5ZE}},
  note         = {Machine review of arXiv:2505.15333}
}

abstract

The success of building textless speech-to-speech translation (S2ST) models has attracted much attention. However, S2ST still faces two main challenges: 1) extracting linguistic features for various speech signals, called cross-modal (CM), and 2) learning alignment of difference languages in long sequences, called cross-lingual (CL). We propose the unit language to overcome the two modeling challenges. The unit language can be considered a text-like representation format, constructed using $n$-gram language modeling. We implement multi-task learning to utilize the unit language in guiding the speech modeling process. Our initial results reveal a conflict when applying source and target unit languages simultaneously. We propose task prompt modeling to mitigate this conflict. We conduct experiments on four languages of the Voxpupil dataset. Our method demonstrates significant improvements over a strong baseline and achieves performance comparable to models trained with text.

Figures

Figures reproduced from arXiv: 2505.15333 by the authors.

**Figure 2.** Architecture of textless S2ST. The dashed line [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The guiding process of the task prompt. the auxiliary data. The losses LCM′ and LCL′ represent the scenarios where we replace the unit language with the source and target text, respectively [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Average lengths of different types of tokens [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: The influences of CM and CL on Fr-En (left) and En-Fr (right) tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: The influences of CM′ and CL′ on Fr-En (left) and En-Fr (right) tasks. most comparable performance with the text-based method in cross-lingual learning. This demonstrates that our unit language, mined through language modeling, can effectively function like text. Whe…

**Figure 7.** Figure 7: Localness of attention weight on different tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Localness of attention weight with/without task prompt on different language tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 9.** Figure 9: The influence of CM and CL on Es-En (left) and En-Es (right) tasks. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 10.** Figure 10: The influence of CM′ and CL′ on Es-En (left) and En-Es (right) tasks. 2 4 6 8 10 12 0.20 0.30 0.40 0.50 Localness Baseline CM ′ CL ′ CM ′&CL ′ (a) Es-En 2 4 6 8 10 12 0.20 0.30 0.40 0.50 0.60 Localness (b) Fr-En 2 4 6 8 10 12 0.20 0.30 0.40 0.50 0.60 Localness (c) En-…

**Figure 11.** Figure 11: Localness of attention weight on different tasks. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 18 canonical work pages

[1]

PD Aguero, Jordi Adell, and Antonio Bonafonte. 2006. Prosody generation for speech-to-speech translation. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages I--I. IEEE

work page 2006
[2]

Lo \" c Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. Seamlessm4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596

arXiv 2023
[3]

Francisco Casacuberta, Hermann Ney, Franz Josef Och, Enrique Vidal, Juan Miguel Vilar, Sergio Barrachina, Ismael Garc a-Varea, David Llorens, C \'e sar Mart nez, Sirko Molau, et al. 2004. Some approaches to statistical and finite-state speech-to-speech translation. Computer Speech & Language, 18(1):25--47

work page 2004
[4]

William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211

arXiv 2015
[5]

Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, and Ann Lee. 2023. https://doi.org/10.18653/v1/2023.findings-acl.307 Speech-to-speech translation for a real-world unwritten language . I...

work page doi:10.18653/v1/2023.findings-acl.307 2023
[6]

Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Qibing Bai, and Yu Zhang. 2022. https://doi.org/10.21437/Interspeech.2022-10011 Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation . In Proc. Interspeech 2022, pages 1781--1785

work page doi:10.21437/interspeech.2022-10011 2022
[7]

Qingkai Fang, Yan Zhou, and Yangzhou Feng. 2023. https://api.semanticscholar.org/CorpusID:263834729 Daspeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation . ArXiv, abs/2310.07403

work page Pith review arXiv 2023
[8]

Alex Graves, Santiago Fern\' a ndez, Faustino Gomez, and J\" u rgen Schmidhuber. 2006. https://doi.org/10.1145/1143844.1143891 Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks . In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 369–376, New York, NY, USA. Associ...

arXiv 2006

Show all 34 references

[9]

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processin...

2021
[10]

Rongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He, and Zhou Zhao. 2022. Transpeech: Speech-to-speech translation with bilateral perturbation. In The Eleventh International Conference on Learning Representations

2022
[11]

Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023. https://doi.org/10.18653/v1/2023.acl-long.872 U nit Y : Two-pass direct speech-to-speech translation with discrete units . In Proc...

2023 doi
[12]

Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, and Nobu Morioka. 2022. https://doi.org/10.21437/Interspeech.2022-10938 Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation . In Proc. Interspeech 2022, pages...

2022 doi
[13]

Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2021. https://api.semanticscholar.org/CorpusID:248863434 Translatotron 2: High-quality direct speech-to-speech translation with voice preservation . In International Conference on Machine Learning

2021
[14]

Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu

Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. https://doi.org/10.21437/Interspeech.2019-1951 Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model . In Proc. Interspeech 2019, pages 1123--1127

2019 doi
[15]

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022--17033

2020
[16]

Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66--71

2018
[17]

Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022 a . https://doi.org/10.18653/v1/2022.acl-long.235 Direct speech-to-speech translation with discrete units . In Proceedings of...

2022 doi
[18]

Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2022 b . https://doi.org/10.18653/v1/2022.naacl-main.63 Textless speech-to-speech translation on real data . In Proceed...

2022 doi
[19]

Xinjian Li, Ye Jia, and Chung-Cheng Chiu. 2023. https://doi.org/10.1109/ICASSP49357.2023.10096797 Textless direct speech-to-speech translation with discrete speech representation . In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (...

2023
[20]

Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355

2021 arXiv
[21]

Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. 2022. https://doi.org/10.21437/Interspeech.2022-11032 Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation . In Proc. Inter...

2022 doi
[22]

Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

2018 doi
[23]

Feiyu Shen, Yiwei Guo, Chenpeng Du, Xie Chen, and Kai Yu. 2024. Acoustic bpe for speech generation with discrete tokens. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11746--11750. IEEE

2024
[24]

Kun Song, Yi Ren, Yi Lei, Chunfeng Wang, Kun Wei, Lei Xie, Xiang Yin, and Zejun Ma. 2023. https://doi.org/10.21437/Interspeech.2023-648 StyleS2ST: Zero-shot Style Transfer for Direct Speech-to-speech Translation . In Proc. INTERSPEECH 2023, pages 42--46

2023 doi
[25]

Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2019. https://doi.org/10.1109/ASRU46091.2019.9003853 Speech-to-speech translation between untranscribed unknown languages . In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593--600

2019
[26]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

2017
[27]

Enrique Vidal. 1997. Finite-state speech-to-speech translation. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 111--114. IEEE

1997
[28]

Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. https://doi.org/10.18653/v1/2021.acl-long.80 V ox P opuli: A large-scale multilingual speech corpus for representation learning, semi-sup...

2021 doi
[29]

Felix Wu, Kwangyoun Kim, Shinji Watanabe, Kyu J Han, Ryan McDonald, Kilian Q Weinberger, and Yoav Artzi. 2023. Wav2seq: Pre-training speech-to-text encoder-decoder models using pseudo languages. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal ...

2023
[30]

Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and Jingbo Zhu. 2021. https://doi.org/10.18653/v1/2021.acl-long.204 Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders . In Proceedings of the 59th...

2021 doi
[31]

Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie-Yan Liu. 2020. https://www.microsoft.com/en-us/research/publication/uwspeech-speech-to-speech-translation-for-unwritten-languages/ Uwspeech: Speech to speech translation for unwritten languages . In AAAI 2021

2020
[32]

Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.663 Rethinking and improving multi-task learning for end-to-end speech translation . In Proceedings of the 2023 Conference on Empirical Methods...

2023 doi
[33]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
[34]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

PD Aguero, Jordi Adell, and Antonio Bonafonte. 2006. Prosody generation for speech-to-speech translation. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages I--I. IEEE

work page 2006

[2] [2]

Lo \" c Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. Seamlessm4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596

arXiv 2023

[3] [3]

Francisco Casacuberta, Hermann Ney, Franz Josef Och, Enrique Vidal, Juan Miguel Vilar, Sergio Barrachina, Ismael Garc a-Varea, David Llorens, C \'e sar Mart nez, Sirko Molau, et al. 2004. Some approaches to statistical and finite-state speech-to-speech translation. Computer Speech & Language, 18(1):25--47

work page 2004

[4] [4]

William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211

arXiv 2015

[5] [5]

Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, and Ann Lee. 2023. https://doi.org/10.18653/v1/2023.findings-acl.307 Speech-to-speech translation for a real-world unwritten language . I...

work page doi:10.18653/v1/2023.findings-acl.307 2023

[6] [6]

Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Qibing Bai, and Yu Zhang. 2022. https://doi.org/10.21437/Interspeech.2022-10011 Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation . In Proc. Interspeech 2022, pages 1781--1785

work page doi:10.21437/interspeech.2022-10011 2022

[7] [7]

Qingkai Fang, Yan Zhou, and Yangzhou Feng. 2023. https://api.semanticscholar.org/CorpusID:263834729 Daspeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation . ArXiv, abs/2310.07403

work page Pith review arXiv 2023

[8] [8]

Alex Graves, Santiago Fern\' a ndez, Faustino Gomez, and J\" u rgen Schmidhuber. 2006. https://doi.org/10.1145/1143844.1143891 Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks . In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 369–376, New York, NY, USA. Associ...

arXiv 2006