Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Contrastive Learning for Task-Independent SpeechLLM-Pretraining

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper seeks to establish that layer-wise contrastive pretraining on speech-text pairs gives a SpeechLLM a task-agnostic foundation, so that finetuning only its projector on 10% of task-specific data can match or beat specialized…

desk verdict Useful pretraining recipe, but the 'surpasses specialized models' claim is an artifact of a normalized average that hides losses on WER and BLEU. read the letter →

arxiv 2412.15712 v2 pith:LOE4K5MH submitted 2024-12-20 cs.CL cs.HC

Maike Züfle , Jan Niehues This is my paper · ORCID
classification cs.CLcs.HC
keywords contrastivelearningspeechLLMspeech-textalignmentInfoNCElosstask-independentpretraininglow-resourcetranslationspokenquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a two-stage recipe for adapting an LLM to speech: first pretrain a projector to align speech and text representations with a contrastive loss applied after every layer, then finetune only the projector on task-specific data. It argues that this task-independent pretraining outperforms traditional ASR pretraining and, when scaled to 1,400 hours of pretraining data, lets a SpeechLLM trained on only 10% of downstream data match or beat systems specialized on speech translation and spoken question answering. The reason to care is that it offers a cheap, task-agnostic alignment stage as an alternative to large labelled downstream datasets. The claim is strongest on the paper's normalized average of word error rate, COMET, and F1, and should be read together with the per-metric tables.

What carries the argument

The engine is the InfoNCE contrastive loss applied to paired speech and text embeddings, summed over the LLM's layers (in practice every fifth layer approximates all layers). Similarity is measured either by cosine similarity after mean-pooling the sequence or by the negative Wasserstein distance computed with a Sinkhorn approximation to handle unequal sequence lengths. The only trainable component is a Q-Former projector; the HuBERT speech encoder and the Llama-3.1-8B-Instruct LLM stay frozen, so the alignment must be learned in the projector's output space.

What would settle it

Recompute the comparison without the normalized average: count how many of the raw metrics the best contrastive model wins against Whisper (ASR), Seamless (speech translation), and the dedicated SQA baseline. The paper's Table 4 already shows word error rate 9.31 for the best model versus 6.54 for Whisper, so a per-metric majority test would fail on ASR and would settle whether the 'surpasses specialized models' claim survives without the averaging assumption.

Watch

Extended reading notes

Core claim

The central claim is that contrastive pretraining, specifically the InfoNCE loss with cosine similarity or Wasserstein distance applied at every LLM layer, produces a task-agnostic speech-to-text alignment that is a better foundation for SpeechLLMs than pretraining on the ASR task itself. The authors show that with only a frozen HuBERT speech encoder, a frozen Llama-3.1-8B-Instruct LLM, and a trainable Q-Former projector, their best model, cosine contrastive pretraining on all layers combined with an ASR loss and 1,400 hours of pretraining data, reaches a normalized average of 102.15 versus 100 for the specialized baselines after finetuning on just 10% of the task-specific data. They take this as evidence that a projector-only SpeechLLM can surpass specialized systems on speech translation and spoken question answering while remaining competitive on ASR.

Load-bearing premise

The load-bearing premise is that the normalized average of word error rate, COMET, and F1 defined in Eq. (3) is a meaningful single score; if that averaging is not accepted, the paper's central claim of surpassing specialized models loses its footing, because the best model still has a higher word error rate than Whisper and lower BLEU than Seamless on some language pairs.

Editorial extensions

If this is right

  • Contrastive pretraining with the loss applied at every layer outperforms ASR pretraining as a task-agnostic foundation for a frozen-encoder, frozen-LLM SpeechLLM.
  • Finetuning on 10% of task-specific data after scaled contrastive pretraining can match or exceed the normalized performance of specialized ASR, speech translation, and spoken QA models.
  • Adding an ASR loss on top of the contrastive loss during pretraining improves low-resource ASR and spoken QA beyond either loss alone.
  • Contrastive alignment does not discard paralinguistic information; contrastively pretrained models achieve higher gender accuracy than ASR-pretrained and cascaded baselines on the MuST-SHE translation benchmark.
  • Because only the projector is trained, the LLM's text abilities stay intact and the speech encoder or LLM can be swapped without retraining the whole model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same layer-wise InfoNCE recipe could transfer to other modality-pair bridges, such as image-text or video-text projectors, because it needs only paired sequences and a frozen backbone.
  • Editorial inference: the 'surpasses specialized models' headline is metric-dependent; the paper's own raw numbers show the best model still trails Whisper on word error rate, so the result is better stated as 'ahead on the combined normalized average' rather than 'ahead on every task'.
  • Editorial inference: a stronger low-resource test would finetune on a language or domain absent from the pretraining corpus, since the paper's 10% setting is a random subset of the same in-domain data.
  • Editorial inference: the paper lists hard-negative mining as future work; selecting in-batch negatives that are close to the positive pair would probably sharpen the contrastive signal and is a direct next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes contrastive pretraining for SpeechLLMs, where a frozen speech encoder and LLM are connected through a trainable projector. The authors compare cosine and Wasserstein-based InfoNCE losses applied to embeddings at multiple layers, using ASR data (Must-C, optionally GigaSpeech) as the pretraining corpus. After pretraining, they fine-tune only the projector on ASR, speech translation (ST), and spoken question answering (SQA), often with only 10% of the task-specific data. Their central claims are that contrastive pretraining outperforms traditional ASR pretraining and that their models 'surpass specialized models' on ST and SQA with 10% of the data. The paper also studies combinations with ASR loss and mixed speech-text next-word prediction, and analyzes the preservation of paralinguistic information.

Significance. If the claims hold, the paper demonstrates that a lightweight, parameter-efficient pretraining signal can provide a task-agnostic foundation for SpeechLLMs, substantially reducing downstream data needs. The controlled comparisons across pretraining objectives are valuable, and the paper is thorough in its ablations (layer selection, loss types, data scaling, combinations with ASR loss, paralinguistic probes). The authors ship code and provide detailed hyperparameters, which aids reproducibility. The main weakness is that the strongest headline result--'surpassing specialized models' with 10% of the data--rests on a normalized aggregate that hides per-metric deficits on standard metrics such as WER and BLEU.

major comments (4)
  1. [Abstract; Section 5.4, Table 4] The claim that the 10%-data model 'surpasses specialized models' is not supported by the per-metric results. The best 10% model (contr-cos-all+asr+giga) has WER 11.12 versus Whisper's 6.54 and BLEU 29.90 versus Seamless's 30.99; it wins only on COMET (81.29 vs. 80.02) and SQA F1 (82.52 vs. 77.10). Since the normalized average (Eq. 3) masks these deficits, the headline claim should be reworded or the analysis should demonstrate superiority on the specific metrics claimed.
  2. [Section 4.5, Eq. (3) and Section 5.4] The normalized average used to support the 'surpassing' result depends on an arbitrary choice of metrics and bounds. For ST, the paper reports both BLEU and COMET but excludes BLEU from the average; substituting BLEU for COMET in Eq. (3) would substantially lower the normalized score of the 10% model and potentially flip the conclusion. The paper should justify the metric selection, report the average under alternative metric sets, or present the normalized average only as a secondary analysis rather than the basis for the main contribution claim.
  3. [Tables 1-4; Section 5.4] No confidence intervals, standard deviations, or significance tests are reported for the central comparisons. The COMET margin between the best 10% model and Seamless is only 1.27 points, while WER and BLEU deficits are large; without estimates of run-to-run variance or statistical testing, the claim of 'surpassing' specialized models is not empirically grounded. This is load-bearing because the abstract and conclusions invoke this comparison.
  4. [Section 4.3, Section 5.4] The pretraining corpus (Must-C ASR data) overlaps with the finetuning data for ASR and ST, as the paper acknowledges. Since the 10% finetuning subset is drawn from the same TED-talk corpus, the low-resource advantage could be partly due to the model having seen the transcripts during pretraining. Reporting results with non-overlapping splits or explicitly filtering the pretraining data would make the task-independence and low-resource claims more convincing.
minor comments (4)
  1. [Section 5.5] There is a typo: 'paralinugistic' should be 'paralinguistic' in the sentence introducing the paralinguistic feature analysis.
  2. [Equation (3)] The notation T is used for the set of tasks without a formal definition, and the terms 'lower bound' and 'upper bound' are confusing for WER, where a lower value is better; the normalization direction should be made explicit.
  3. [Table 2] Table 2 omits the BLEU and EM columns that appear in Tables 1 and 4, which makes it difficult to compare mixed-input variants with the main results; the caption should state why these metrics are omitted or include them.
  4. [Appendix G] There is a typo: 'signifiant' should be 'significant' in the sentence about combined losses.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical comparisons against external baselines, and its normalized-average reporting choice is not a derivation that reduces to its own inputs.

full rationale

The paper's central claims are empirical comparisons between trained models and external baselines (Whisper, Seamless, You et al., BLSP, Qwen2-Audio). The contrastive pretraining objective (Eq. 1) is a standard InfoNCE loss, and the downstream results are measured with standard metrics (WER, BLEU, COMET, EM, F1) against those external systems. No parameter is fitted to the reported evaluation metrics and then renamed as a prediction. The normalized average (Eq. 3) uses lower and upper bounds set by the cascaded HuBERT+Llama model and by specialized models; this is an arbitrary normalization choice that affects how the aggregate 'surpass' claim is read, and a metric-substitution test could plausibly weaken the headline, but it is not circular because the per-metric scores are independently measured and are not constructed from the normalization bounds. The paper does report test-set contrastive loss, but it uses this only as a sanity check of alignment, not as the evidence for the headline downstream claim. There are no load-bearing self-citations, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via self-citation; hyperparameters follow Verdini et al. (2024), but that citation does not force the conclusion. The overlap between pretraining and finetuning data for ASR/ST is acknowledged and is a data-reuse concern, not a circularity concern. Overall, the derivation chain is self-contained and empirically grounded, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on standard contrastive learning and optimal transport machinery plus standard SpeechLLM components. The main hand-set numbers are hyperparameters (temperature, blur) and the evaluation normalization, which are not fitted to the test set but do shape the headline comparisons.

free parameters (5)
  • contrastive temperature tau = 0.1
    Set for both cosine and Wasserstein contrastive losses (Table 7); not fitted to test data but chosen by hand and affects the sharpness of the contrastive objective.
  • Sinkhorn blur = 0.5
    Used in the Wasserstein distance computation (Table 7); a regularisation hyperparameter.
  • layer selection (every 5th layer) = multiples of 5
    Chosen as an efficient approximation of applying the loss to all layers after initial ablations (Section D, Table 11); the choice is based on observed performance.
  • loss combination weight = equal weights
    When combining ASR and contrastive losses, individual losses are summed with equal weight (Section 5.3); no tuning of the relative weights is reported.
  • normalized average bounds = specialized and cascaded baselines
    The lower and upper bounds in Eq. (3) are set by the cascaded model and by specialized systems; the headline 'surpasses specialized models' depends on this normalization.
assumptions (4)
  • standard math InfoNCE loss is a valid objective for aligning paired speech and text representations while separating unpaired ones
    Introduced by van den Oord et al. (2019) and used in many contrastive learning papers; the paper relies on this to make pretraining work.
  • standard math Sinkhorn algorithm provides a differentiable approximation of the Wasserstein distance
    Used in Eq. (2) and the following paragraph; standard optimal transport approximation.
  • domain assumption A frozen HuBERT encoder and frozen Llama-3.1-8B-Instruct, with only a Q-Former projector trained, can learn to align speech and text through contrastive pretraining
    The architecture choice is borrowed from prior SpeechLLMs and is not derived; the paper assumes this setup supports the alignment and downstream transfer.
  • domain assumption Contrastive alignment does not destroy paralinguistic information captured by the speech encoder
    The paper tests this hypothesis in Section 5.5 and finds supporting evidence, but it is an assumption about the information retention of the frozen encoder.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contrastive Learning for Task-Independent SpeechLLM-Pretraining." pith.science (2026). https://pith.science/paper/LOE4K5MH

@misc{pith2026241215712,
  author       = {Pith},
  title        = {Pith review of: Contrastive Learning for Task-Independent SpeechLLM-Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOE4K5MH}},
  note         = {Machine review of arXiv:2412.15712}
}
read the original abstract

Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.

Figures

Figures reproduced from arXiv: 2412.15712 by the authors.

Figure 1
Figure 1. An overview of our two-stage training process. First we perform task-agnostic pretraining, by leveraging [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Analysis of contrastive pretraining performed on different layers. We report the normalized average over [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Analysis of a contrastive pretraining performed on multiple layers for different tasks. The performance [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Contrastive loss during pretraining performed on multiple layers for models trained with contrastive [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Across controlled ASR and speech translation experiments, dense feature prepending does not outperform cross-attention in quality and is slightly slower and more memory hungry.

Reference graph

Works this paper leans on

58 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf wav2vec 2.0: A framework for self-supervised learning of speech representations . In Advances in Neural Information Processing Systems, volume 33, pages 12449--12460. Curran Associates, Inc

    2020

  2. [2]

    Di Gangi, Roldano Cattoni, and Marco Turchi

    Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mattia A. Di Gangi, Roldano Cattoni, and Marco Turchi. 2020. https://doi.org/10.18653/v1/2020.acl-main.619 Gender in danger? evaluating speech translation technology on the M u ST - SHE corpus . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923--6933, On...

    doi:10.18653/v1/2020.acl-main.619 2020

  3. [3]

    Umberto Cappellazzo, Enrico Fini, Muqiao Yang, Daniele Falavigna, Alessio Brutti, and Bhiksha Raj. 2024. https://doi.org/10.18653/v1/2024.findings-acl.223 Continual contrastive spoken language understanding . In Findings of the Association for Computational Linguistics: ACL 2024, pages 3727--3741, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.findings-acl.223 2024

  4. [4]

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang, and 2 others. 2021. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audi...

    work page 2021

  5. [5]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. https://arxiv.org/abs/2407.10759 Qwen2-audio technical report . Preprint, arXiv:2407.10759

    arXiv 2024

  6. [6]

    Liu, and Hung-yi Lee

    Shun-Po Chuang, Tzu-Wei Sung, Alexander H. Liu, and Hung-yi Lee. 2020. https://doi.org/10.18653/v1/2020.acl-main.533 Worse WER , but better BLEU ? leveraging word embedding as intermediate in multitask end-to-end speech translation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5998--6003, Online. Assoc...

    work page doi:10.18653/v1/2020.acl-main.533 2020

  7. [7]

    Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, and 49 others. 2023. https://arxiv.org/abs/2...

    arXiv 2023

  8. [8]

    Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi

    Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. https://doi.org/10.18653/v1/N19-1202 M u ST - C : a M ultilingual S peech T ranslation C orpus . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and ...

    doi:10.18653/v1/n19-1202 2019

Show all 58 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  2. [10]

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666

    2024 arXiv

  3. [11]

    Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.486 STEMM : Self-learning with speech-text manifold mixup for speech translation . In Proceedings of the 60th Annual Meeting of the Association for Computational Lingui...

    2022 doi

  4. [12]

    Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2023. https://arxiv.org/abs/2307.11795 Prompting large language models with speech recognition abilitie...

    2023 arXiv

  5. [13]

    Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2024. https://doi.org/10.18653/v1/2024.naacl-long.309 A udio C hat L lama: Towards general-purpose speech abilities for LLM s . ...

    2024 doi

  6. [14]

    Liu, Leonid Karlinsky, and James R

    Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. 2024. https://openreview.net/forum?id=nBZBPXdJlC Listen, think, and understand . In The Twelfth International Conference on Learning Representations

    2024

  7. [15]

    Katharina H \"a mmerl, Jind r ich Libovick \'y , and Alexander Fraser. 2024. https://doi.org/10.18653/v1/2024.findings-acl.649 Understanding cross-lingual A lignment --- A survey . In Findings of the Association for Computational Linguistics: ACL 2024, pages 10922--10943, Bang...

    2024 doi

  8. [16]

    Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. 2024. https://arxiv.org/abs/2305.13009 Textually pretrained speech language models . Preprint, arXiv...

    2024 arXiv

  9. [17]

    William Held, Ella Li, Michael Ryan, Weiyan Shi, Yanzhe Zhang, and Diyi Yang. 2024. https://arxiv.org/abs/2410.02678 Distilling an end-to-end voice assistant without instruction training data . Preprint, arXiv:2410.02678

    2024 arXiv

  10. [18]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. https://doi.org/10.1109/TASLP.2021.3122291 Hubert: Self-supervised speech representation learning by masked prediction of hidden units . IEEE/ACM Transactions on ...

    2021

  11. [19]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  12. [20]

    Ludwig Kurzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. https://api.semanticscholar.org/CorpusID:220633469 Ctc-segmentation of large corpora for german end-to-end speech recognition . In International Conference on Speech and Computer

    2020

  13. [21]

    Yoach Lacombe, Vaibhav Srivastav, and Sanchit Gandhi. 2024. Data-speech. https://github.com/ylacombe/dataspeech

    2024

  14. [22]

    Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. https://arxiv.org/abs/2301.11716 Pre-training for speech translation: Ctc meets optimal transport . Preprint, arXiv:2301.11716

    2023 arXiv

  15. [23]

    Chia-Hsuan Lee, Szu-Lin Wu, Chi-Liang Liu, and Hung-yi Lee. 2018. Spoken squad: A study of mitigating the impact of speech recognition errors on listening comprehension. Proc. Interspeech 2018, pages 3459--3463

    2018

  16. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. https://api.semanticscholar.org/CorpusID:256390509 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . In International Conference on Machine Learning

    2023

  17. [25]

    Dan Lyth and Simon King. 2024. https://arxiv.org/abs/2402.01912 Natural language guidance of high-fidelity text-to-speech with synthetic annotations . Preprint, arXiv:2402.01912

    2024 arXiv

  18. [26]

    Potsawee Manakul, Guangzhi Sun, Warit Sirichotedumrong, Kasima Tharnpipitchai, and Kunat Pipatanakul. 2024. https://arxiv.org/abs/2409.10999 Enhancing low-resource language and instruction following capabilities of audio language models . Preprint, arXiv:2409.10999

    2024 arXiv

  19. [27]

    Ilaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas. 2022. Contrastive audio-language learning for music. In Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR)

    2022

  20. [28]

    Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoit Sagot, and Emmanuel Dupoux. 2024. https://ar...

    2024 arXiv

  21. [29]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  22. [30]

    Siqi Ouyang, Rong Ye, and Lei Li. 2023. https://doi.org/10.18653/v1/2023.acl-long.216 WACO : Word-aligned contrastive learning for speech translation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 389...

    2023 doi

  23. [31]

    Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, and Boris Ginsburg

    Yifan Peng, Krishna C. Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, and Boris Ginsburg. 2024. https://arxiv.org/abs/2410.17485 Voicetextblender: Augmenting large language models with speech capabilities via single-stage...

    2024 arXiv

  24. [32]

    Gabriel Peyré and Marco Cuturi. 2019. https://doi.org/10.1561/2200000073 Computational optimal transport: With applications to data science . Foundations and Trends® in Machine Learning, 11:355--206

    2019 doi

  25. [33]

    Khiem Pham, Khang Le, Nhat Ho, Tung Pham, and Hung Bui. 2020. https://arxiv.org/abs/2002.03293 On unbalanced optimal transport: An analysis of sinkhorn algorithm . Preprint, arXiv:2002.03293

    2020 arXiv

  26. [34]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  27. [35]

    Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023. https://arxiv.org/abs/2305.13516 Sca...

    2023 arXiv

  28. [36]

    Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large-scale multilingual dataset for speech research. ArXiv, abs/2012.03411

    2020 arXiv

  29. [37]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

    2022 arXiv

  30. [38]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://arxiv.org/abs/1606.05250 Squad: 100,000+ questions for machine comprehension of text . Preprint, arXiv:1606.05250

    2016 arXiv

  31. [39]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

    2020 doi

  32. [40]

    Vin Sachidananda, Shao-Yen Tseng, Erik Marchi, Sachin Kajarekar, and Panayiotis Georgiou. 2022. https://arxiv.org/abs/2202.03587 Calm: Contrastive aligned audio-language multirate and multimodal representations . Preprint, arXiv:2202.03587

    2022 arXiv

  33. [41]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. https://openreview.net/forum?id=yHdTscY6Ci Hugging GPT : Solving AI tasks with chat GPT and its friends in hugging face . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  34. [42]

    Richard Sinkhorn and Paul Knopp. 1967. https://api.semanticscholar.org/CorpusID:50329347 Concerning nonnegative matrices and doubly stochastic matrices . Pacific Journal of Mathematics, 21:343--348

    1967

  35. [43]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. https://arxiv.org/abs/2104.09864 Roformer: Enhanced transformer with rotary position embedding . Preprint, arXiv:2104.09864

    2023 arXiv

  36. [44]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. https://openreview.net/forum?id=14rn7HpKVk SALMONN : Towards generic hearing abilities for large language models . In The Twelfth International Conference on Learning ...

    2024

  37. [45]

    Ioannis Tsiamas, Gerard G \'a llego, Jos \'e Fonollosa, and Marta Costa-juss \`a . 2024. https://doi.org/10.18653/v1/2024.findings-acl.847 Pushing the limits of zero-shot end-to-end speech translation . In Findings of the Association for Computational Linguistics: ACL 2024, pa...

    2024 doi

  38. [46]

    U nl \"u Menev s e, Yusufcan Manav, Ebru Arisoy, and Arzucan \

    Merve \"U nl \"u Menev s e, Yusufcan Manav, Ebru Arisoy, and Arzucan \"O zg \"u r. 2024. https://aclanthology.org/2024.lrec-main.397 Dealing with data scarcity in spoken question answering . In Proceedings of the 2024 Joint International Conference on Computational Linguistics...

    2024

  39. [47]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. https://arxiv.org/abs/1807.03748 Representation learning with contrastive predictive coding . Preprint, arXiv:1807.03748

    2019 arXiv

  40. [48]

    Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, Sébastien Bratières, Paolo Merialdo, and Simone Scardapane. 2024. https://arxiv.org/abs/2409.17044 How to connect speech fou...

    2024 arXiv

  41. [49]

    Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. 2024 a . https://arxiv.org/abs/2309.00916 Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing . Preprint, arXiv:2309.00916

    2024 arXiv

  42. [50]

    Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. 2024 b . https://arxiv.org/abs/2411.00774 Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm . Preprint, arXiv:2411.00774

    2024 arXiv

  43. [51]

    Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. 2023. https://arxiv.org/abs/2307.03917 On decoder-only architecture for speech-to-text and large language model integration . Preprint, arXiv:2307.03917

    2023 arXiv

  44. [52]

    Chen Xu, Rong Ye, Qianqian Dong, Chengqi Zhao, Tom Ko, Mingxuan Wang, Tong Xiao, and Jingbo Zhu. 2023. https://doi.org/10.24963/ijcai.2023/761 Recent advances in direct speech-to-text translation . In Proceedings of the Thirty-Second International Joint Conference on Artificia...

    2023 doi

  45. [53]

    Rong Ye, Mingxuan Wang, and Lei Li. 2022. https://doi.org/10.18653/v1/2022.naacl-main.376 Cross-modal contrastive learning for speech translation . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2022 doi

  46. [54]

    Chenyu You, Nuo Chen, Fenglin Liu, Shen Ge, Xian Wu, and Yuexian Zou. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.91 End-to-end spoken conversational question answering: Task, dataset and model . In Findings of the Association for Computational Linguistics: NAACL 202...

    2022 doi

  47. [55]

    Jianing Zhou, Ziheng Zeng, Hongyu Gong, and Suma Bhat. 2024. https://doi.org/10.18653/v1/2024.findings-acl.684 CLASP : Cross-modal alignment using pre-trained unimodal models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 11518--11531, Bangkok,...

    2024 doi

  48. [56]

    Yi Zhu, Zexun Wang, Hang Liu, Peiying Wang, Mingchao Feng, Meng Chen, and Xiaodong He. 2022. https://doi.org/10.21437/Interspeech.2022-11378 Cross-modal transfer learning via multi-grained alignment for end-to-end spoken language understanding . In Interspeech 2022, pages 1131--1135

    2022 doi

  49. [57]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.