Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LiSTEN adapts a frozen LLM to speech and audio tasks by dynamically selecting soft prompts from a learned pool, outperforming LoRA fine-tuning with fewer trainable parameters.

desk verdict A useful prompt-pooling adaptation for audio LLMs that overclaims its own table; the retrieval mechanism is plausible but under-validated. read the letter →

arxiv 2505.18517 v1 pith:UF6UCJGY submitted 2025-05-24 cs.AI cs.LGcs.SDeess.AS

Pooneh Mousavi , Shubham Gupta , Cem Subakan , Mirco Ravanelli This is my paper · ORCID
classification cs.AIcs.LGcs.SDeess.AS
keywords audiolanguagemodelsprompttuningdynamicselectionparameter-efficientfine-tuningmultitasklearningspeechunderstandingsoftpromptsinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LiSTEN is a way to adapt a frozen text-only LLM to hear: it keeps the LLM untouched and instead learns a pool of soft prompts (key-value pairs) from which each input selects a few. The paper tries to establish that this dynamic selection beats the standard parameter-efficient alternative, LoRA fine-tuning, on a six-task audio benchmark while training in a single stage on far less data. If it is right, audio-capable language models become cheaper to build and easier to interpret, because related tasks visibly share the same prompt tokens. The central bet is that one averaged vector over the speech and text tokens is enough to pick the right prompts per instance.

What carries the argument

The load-bearing mechanism is the prompt pool: a set of $P$ learnable key-value pairs, where keys live in the LLM embedding space and values are soft-prompt tokens. Each input's speech and text tokens are mean-pooled into a single query vector $q = \frac{1}{T}\sum_{t=1}^T X_t$; the query is scored against the keys (cosine similarity, attention, or residual matching), and the values attached to the top-$k$ keys are prepended to the input as a dynamic soft prompt. An auxiliary key loss pulls selected keys toward $q$, and stochastic training samples the prompt size $k \sim U(1, P)$ per batch. The only trainable components are the Q-Former audio aligner and the prompt pool itself, so the frozen LLM keeps its text knowledge.

What would settle it

Train the identical pipeline but replace the query with the mean of the text tokens only, or with a random vector drawn once per batch, and compare test accuracy on emotion recognition and speech QA; if performance stays within a few points, the audio part of the query is not doing the routing, while a collapse would show the mean-pooling assumption is load-bearing. A second check is to shuffle the assignment of values to keys after training and measure the drop in accuracy.

Watch

Extended reading notes

Core claim

The central claim is that dynamic prompt selection with learnable key-value pairs, combined with stochastic prompt-size sampling during training, outperforms LoRA for multitask speech-and-audio adaptation of a frozen LLaMA-8B model. Specifically, the similarity-based selection strategy (scoring pooled input tokens against prompt keys with cosine similarity and taking the top-k) reaches the best or second-best score on six of eight tracked metrics, and the stochastic variant keeps nearly the same accuracy at inference with only 10 prompt tokens instead of 160. The method also removes the need for large ASR or captioning pre-training stages, trains everything at once, and yields an interpretable map of which tasks share which prompts.

Load-bearing premise

The method assumes that averaging all speech and text token vectors into one query vector preserves enough task-specific information to pick the right prompts; if mean pooling washes out the difference between, say, emotion recognition and speech question answering, prompt selection becomes near-random and the gains over a fixed prompt disappear.

Editorial extensions

If this is right

  • Compared with LoRA, prompt-pool methods improve or match scores on the evaluated tasks while using roughly 3.3 million trainable parameters, a number that can be reduced further without much loss.
  • Stochastic training permits inference with only 10 prompt tokens instead of 160, lowering computational cost while maintaining strong performance.
  • A single-stage training recipe on six tasks replaces the large-scale ASR and captioning pre-training that earlier audio LLMs required.
  • Prompt-sharing analysis gives an interpretable map: related tasks such as ASR and speech QA select overlapping prompt tokens, while unrelated tasks such as audio captioning select distinct ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The query design invites a direct stress test: replacing mean pooling with learned attention pooling (or with a text-only vector) would reveal how much of the routing signal actually comes from the audio stream.
  • Because prompts are selected per instance rather than per task, the same learned pool could plausibly transfer to unseen tasks by composing existing prompt tokens, a zero-shot extension the paper does not directly test.
  • Residual-based selection resembles a vector-quantization decomposition of the query, so the number of selected prompts could be tied to reconstruction error, turning prompt count into an adaptive confidence signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LiSTEN, a parameter-efficient adaptation framework for audio LLMs. It combines Whisper and BEATs audio encoders, a Q-Former alignment module, and a frozen LLaMA-3-8B-Instruct backbone. The core idea is a learnable prompt pool of key-value pairs: each input's mean-pooled audio/text representation serves as a query, and the top-k values selected by similarity, attention, or residual matching are prepended as soft prompts. A stochastic variant samples prompt length per batch. Experiments compare LoRA, fixed soft prompts, and DPS variants on ASR, En2Zh translation, emotion recognition, speaker verification, speech QA, and audio captioning. The main empirical claim is that similarity-based DPS, especially with stochastic training, improves over LoRA on most metrics and permits inference with 10 prompt tokens at small cost.

Significance. If the results are reproducible, LiSTEN is a useful contribution: it offers a single-stage, data-efficient alternative to LoRA for audio LLM adaptation, with interpretable task-specific token usage. The paper is transparent about the invalid SALMONN comparison, and the prompt diversity analysis in Figure 2 is a nice addition. However, the central empirical claim rests on single runs without error bars and on the untested informativeness of the mean-pooled query; these need to be addressed before the comparisons can be considered reliable.

major comments (4)
  1. [Section 4, Table 2] All results are reported from single runs with no seeds, error bars, or significance tests. Several of the differences that support the central claim are small; for example, similarity-based DPS and the fixed soft-prompt baseline differ by only 0.0015 in AAC Rouge-L (0.3578 vs 0.3557), 0.0022 in AAC BLEU (0.1288 vs 0.1310), and are reversed in QA Rouge-L (0.4924 vs 0.4997). The conclusion that similarity-based stochastic DPS "achieves the best performance across multiple tasks" is therefore not established against run-to-run variability. Please provide multiple seeds (at least 3) or confidence intervals for at least the main comparisons.
  2. [Section 2.1] The query q = (1/T) sum X_t is the only mechanism connecting an input instance to prompt selection. The paper provides no evidence that this mean-pooled representation retains task- and content-discriminative information. Figure 2 shows aggregate token usage, but it does not show that selected prompts vary correctly with q or that retrieval is necessary. A crucial control is to compare against random selection of k prompt values from the same trained pool; without this control, the reported gains over fixed soft prompts could be due to the larger learned prompt tensor rather than to dynamic retrieval. Please add a random-prompt baseline and, ideally, a task-classification probe on q.
  3. [Section 4.3, Table 2 rows 5-6] The inference-efficiency claim (10 tokens vs 160) is attributed to stochastic training, but the comparison only contrasts stochastic DPS evaluated at two prompt lengths. To establish that stochastic training is responsible, include a control where the non-stochastic similarity DPS (trained with fixed k=160) is evaluated at k=10, and preferably also stochastic DPS with different inference lengths. Otherwise the drop from 0.0421 to 0.0491 WER and the ER drop from 0.6119 to 0.5975 might reflect a generally k-sensitive prompt method rather than a benefit of stochasticity.
  4. [Section 1 and Section 3] The contribution claims "significantly less training data" and reduced dependence on large-scale ASR/captioning datasets, but no data-efficiency experiment is reported. Table 2 is generated from one training mixture (LibriSpeech train-clean-100 for ASR, six tasks), and there is no scaling curve or ablation with reduced data. A direct comparison at smaller training subsets, or at least a statement of total training data and a comparison to a LoRA baseline trained on the same amount, would substantiate this claim.
minor comments (5)
  1. [References] Reference [6] and reference [10] are the same Pengi paper and should be merged.
  2. [Section 2.1] The similarity softmax equation appears to have a formatting error: "exp(sim(q,ki))PP j=1 exp(sim(q,kj))" is missing the division bar; please re-set it.
  3. [Table 2] The table says "best and second-best values computed ignoring SALMONN row," but the table does not visually mark these values; please add bold or italics to make the comparison readable.
  4. [Section 3] The text states "we use a prompt pool of 400 tokens"; since the pool consists of key-value pairs, clarify whether this means 400 pairs or 400 key/value tokens each.
  5. [Section 4.2 and Figure 2] The claim that prompt sharing provides explainability for unseen tasks is speculative; the paper only analyzes six seen tasks, so soften this to a hypothesis. Also, the z-axis of Figure 2 is not fully described; please clarify what the color or height encodes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LiSTEN's dynamic prompt selection is a learned, externally evaluated mechanism, and the only self-citation ([20]) is an auxiliary stochastic-training variant, not load-bearing.

full rationale

The paper's contribution is architectural and empirical rather than derivational. The query is defined as the mean of concatenated speech and text tokens in Section 2.1, the prompt pool is trained with a next-token objective plus a key regularizer, and all reported results are measured with external metrics (WER, CER, accuracy, Rouge-L, BLEU) on standard held-out test sets. No fitted parameter is renamed as a prediction, and no test-set value is used to construct the model or its selection rule. The mean-pooling assumption is a design choice whose informativeness can be checked empirically; it is not a circular reduction because the claimed gains do not follow from the definition of q alone. The only self-citation is [20] (Gupta et al.), used as inspiration for stochastic prompt-size sampling. That citation supports an optional training variant, not the central dynamic-selection claim, and it is not invoked to rule out competing approaches. Consequently, the paper's derivation chain is self-contained against external benchmarks and no load-bearing step reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small set of hand-picked hyperparameters (pool size 400, prompt length 160, unreported alpha) and on the design assumption that a mean-pooled query vector selects useful prompts. The model introduces no new physical or mathematical entities; the prompt pool is a standard learnable component.

free parameters (4)
  • prompt pool size P = 400
    Chosen without ablation; the paper claims the pool can be reduced but presents no table or plot supporting that claim, so this hand-picked capacity affects the 3.3M parameter count.
  • prompt length k at inference = 160 (and 10 for the stochastic run)
    Hand-picked; training samples k uniformly from 1 to 400 (Section 2.1), so the fixed inference length of 160 (or 10) is a design choice that must be justified by the stochastic training.
  • key-loss scaling factor alpha = not reported
    Defined in L = L_next-token + alpha*L_key (Section 2.1) but never given numerically, so the balance between the LM loss and the prompt-key regularizer is unreported and unreproducible.
  • LoRA rank and soft-prompt length (baselines) = not reported
    The paper uses 'the same settings as SALMONN' for LoRA and Q-Former, but the backbone differs (LLaMA-3-8B vs Vicuna-13B) and no rank or length is reported, so baseline capacity is uncontrolled.
assumptions (4)
  • domain assumption Frozen Whisper and BEATs encoders provide sufficient audio features for all six tasks.
    Invoked in Section 2 without re-validation; if the encoders miss task-relevant acoustic cues, the prompt pool cannot recover them.
  • ad hoc to paper The mean-pooled query q preserves task-relevant information.
    Section 2.1 defines q as the average of all tokens; this is the central design assumption of the retrieval mechanism, with no theoretical or empirical justification beyond the final benchmark numbers.
  • domain assumption The mixed six-dataset training set is sufficient to learn a reusable prompt pool.
    Section 3 lists the datasets but no data-size ablation is run; the claim of data efficiency is asserted, not demonstrated.
  • domain assumption Stochastic prompt-length sampling transfers to small-k inference.
    Relies on the authors' prior work [20] to justify that k ~ U(1,P) training supports k=10 inference, a transfer not re-validated here beyond one table row.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs." pith.science (2026). https://pith.science/paper/UF6UCJGY

@misc{pith2026250518517,
  author       = {Pith},
  title        = {Pith review of: LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UF6UCJGY}},
  note         = {Machine review of arXiv:2505.18517}
}
read the original abstract

Foundation models based on large language models (LLMs) have shown great success in handling various tasks and modalities. However, adapting these models for general-purpose audio-language tasks is challenging due to differences in acoustic environments and task variations. In this work, we introduce LiSTEN Learning Soft Token Embeddings for Neural Audio LLMs), a framework for adapting LLMs to speech and audio tasks. LiSTEN uses a dynamic prompt selection strategy with learnable key-value pairs, allowing the model to balance general and task-specific knowledge while avoiding overfitting in a multitask setting. Our approach reduces dependence on large-scale ASR or captioning datasets, achieves competitive performance with fewer trainable parameters, and simplifies training by using a single-stage process. Additionally, LiSTEN enhances interpretability by analyzing the diversity and overlap of selected prompts across different tasks.

Figures

Figures reproduced from arXiv: 2505.18517 by the authors.

Figure 1
Figure 1. LiSTEN pipeline with Dynamic Prompt Selection (DPS). The prompt pool consists of key-value pairs. Audio is encoded using Whisper and BEATs, whose embeddings are concatenated and processed through a Q-Former. The task is represented as text and encoded with the backbone LLM tokenizer. The speech and text tokens are mean-pooled to obtain a query, which selects k values from the prompt pool. These selected values serve… view at source ↗
Figure 2
Figure 2. Token usage distribution across tasks in the test set. The z-axis represents token frequency, the x-axis shows token indices sorted by frequency for ASR, and the y-axis indicates the task. A prompt pool of 400 tokens was used, with each instance selecting 10 tokens at inference. period of 3000 steps and a maximum learning rate of 3 × 10−5 . We employ the LLaMA 8B instruct model2 as the LLM back￾bone. We use the same… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ALAS: An Automatic Latent Alignment Score for Audio Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    ALAS is a reference-based score for audio-text alignment in speech LLMs, computed from frozen hidden states and a Whisper-derived alignment path, with no training or fitted classifier.

Reference graph

Works this paper leans on

44 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    In recent years, text- based LLMs have achieved remarkable performance, serving as the backbone for multimodal systems

    Introduction Foundation models built on large language models (LLMs) have demonstrated strong capabilities in handling diverse tasks and modalities [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. In recent years, text- based LLMs have achieved remarkable performance, serving as the backbone for multimodal systems. A key factor behind their success is instruction tun...

    arXiv 2025

  2. [2]

    Inspired by L2P [25, 26], we implement a prompt selection mechanism where prompts are stored as learnable key-value pairs

    Model Design We introduce a new fine-tuning technique for training Audio Large Language Models (ALLM). Inspired by L2P [25, 26], we implement a prompt selection mechanism where prompts are stored as learnable key-value pairs. Our proposed architecture is illustrated in Figure 1. Our setup follows a similar structure to SALMONN 1, but differs in its fine-t...

    work page

  3. [3]

    The datasets used in our experiments, along with their evaluation metrics, are listed in Table 1

    Experiments The instruction tuning data used for training include multi- ple tasks, including Automatic Speech Recognition (ASR), au- tomatic speech translation from English to Chinese (En2Zh), emotion recognition (ER), speaker verification (SV), speech question answering (SQA), and audio captioning (ACAP). The datasets used in our experiments, along with...

    work page

  4. [4]

    We include SALMONN’s performance on our dataset for compari- son, with results aligning with its benchmarking in [38]

    Results and Analysis In Table 2, we compare LoRA, soft prompting (or prefix tun- ing), and our proposed dynamic prompt selection strategies for adapting a backbone LLM for audio-text multitask learning. We include SALMONN’s performance on our dataset for compari- son, with results aligning with its benchmarking in [38]. How- ever, a direct comparison with...

    work page

  5. [5]

    Conclusion In this work, we explored dynamic prompt selection as an ef- ficient alternative to LoRA and soft prompting for adapting LLMs to audio-language tasks. Our results show that similarity- based dynamic prompt selection, trained in a stochastic setting, achieves the best performance across multiple tasks while en- abling significantly lower inferen...

    work page

  6. [6]

    While performance decreases across all tasks, the drop is relatively small

    Row 5 corresponds to inference with a prompt length of 160, while row 6 represents inference with a significantly reduced prompt length of 10. While performance decreases across all tasks, the drop is relatively small. This suggests that stochas- tic training enables more efficient inference, reducing compu- tational costs while maintaining strong perform...

    work page

  7. [7]

    Pengi: An audio language model for audio tasks,

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,”Advances in Neural Infor- mation Processing Systems, vol. 36, pp. 18 090–18 108, 2023

    2023

  8. [8]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023

    arXiv 2023

Show all 44 references
  1. [9]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  2. [10]

    Anygpt: Unified multi- modal llm with discrete sequence modeling,

    J. Zhan, J. Dai, J. Ye, Y . Zhou, D. Zhang, Z. Liu, X. Zhang, R. Yuan, G. Zhang, L. Li et al. , “Anygpt: Unified multi- modal llm with discrete sequence modeling,” arXiv preprint arXiv:2402.12226, 2024

    2024 arXiv

  3. [11]

    Mini-omni2: Towards open-source gpt- 4o with vision, speech and duplex capabilities,

    Z. Xie and C. Wu, “Mini-omni2: Towards open-source gpt- 4o with vision, speech and duplex capabilities,” arXiv preprint arXiv:2410.11190, 2024

    2024 arXiv

  4. [12]

    Listen, think, and understand,

    Y . Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” arXiv preprint arXiv:2305.10790, 2023

    2023 arXiv

  5. [13]

    Language model can listen while speaking,

    Z. Ma, Y . Song, C. Du, J. Cong, Z. Chen, Y . Wang, Y . Wang, and X. Chen, “Language model can listen while speaking,” 2024. [Online]. Available: https://arxiv.org/abs/2408.02622

    2024 arXiv

  6. [14]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,

    Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro, “Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,” 2024. [Online]. Available: https://arxiv.org/abs/2402.01831

    2024 arXiv

  7. [15]

    Desta: Enhancing speech language models through descriptive speech-text alignment,

    K.-H. Lu, Z. Chen, S.-W. Fu, H. Huang, B. Ginsburg, Y .-C. F. Wang, and H. yi Lee, “Desta: Enhancing speech language models through descriptive speech-text alignment,” 2024. [Online]. Available: https://arxiv.org/abs/2406.18871

    2024 arXiv

  8. [16]

    Pengi: An audio language model for audio tasks,

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2305.11834

    2024 arXiv

  9. [17]

    Audiopalm: A large language model that can speak and listen,

    P. K. Rubenstein et al., “Audiopalm: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925, 2023

    2023 arXiv

  10. [18]

    Instruction tuning with gpt-4,

    B. Peng, C. Li, P. He, M. Galley, and J. Gao, “Instruction tuning with gpt-4,” arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  11. [19]

    Instruction tuning for large language mod- els: A survey,

    S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wuet al., “Instruction tuning for large language mod- els: A survey,” arXiv preprint arXiv:2308.10792, 2023

    2023

  12. [20]

    Alpaca: a strong, replicable instruction-following model; 2023,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpaca: a strong, replicable instruction-following model; 2023,” URL https://crfm. stanford. edu/2023/03/13/alpaca. html

    2023

  13. [21]

    Lora: Low-rank adaptation of large lan- guage models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large lan- guage models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  14. [22]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neu- ral Information Processing Systems, vol. 36, 2024

    2024

  15. [23]

    Wavllm: Towards ro- bust and adaptive speech large language model,

    S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran et al. , “Wavllm: Towards ro- bust and adaptive speech large language model,” arXiv preprint arXiv:2404.00656, 2024

    2024 arXiv

  16. [24]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  17. [25]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  18. [26]

    Last stop for modeling asynchronous time series,

    S. Gupta, T. Durand, G. Taylor, and L. W. Białokozowicz, “Last stop for modeling asynchronous time series,” 2025. [Online]. Available: https://arxiv.org/abs/2502.01922

    2025 arXiv

  19. [27]

    Speechprompt v2: Prompt tuning for speech classification tasks,

    K.-W. Chang, Y .-K. Wang, H. Shen, I.-t. Kang, W.-C. Tseng, S.-W. Li, and H.-y. Lee, “Speechprompt v2: Prompt tuning for speech classification tasks,” arXiv preprint arXiv:2303.00733 , 2023

    2023 arXiv

  20. [28]

    Chain-of- thought prompting for speech translation,

    K. Hu, Z. Chen, C.-H. H. Yang, P. ˙Zelasko, O. Hrinchuk, V . Lavrukhin, J. Balam, and B. Ginsburg, “Chain-of- thought prompting for speech translation,” arXiv preprint arXiv:2409.11538, 2024

    2024 arXiv

  21. [29]

    Speechgen: Un- locking the generative power of speech language models with prompts,

    H. Wu, K.-W. Chang, Y .-K. Wu, and H.-y. Lee, “Speechgen: Un- locking the generative power of speech language models with prompts,” arXiv preprint arXiv:2306.02207, 2023

    2023 arXiv

  22. [30]

    Speechprompt: An exploration of prompt tuning on generative spoken language model for speech processing tasks,

    K.-W. Chang, W.-C. Tseng, S.-W. Li, and H. yi Lee, “Speechprompt: An exploration of prompt tuning on generative spoken language model for speech processing tasks,” 2022. [Online]. Available: https://arxiv.org/abs/2203.16773

    2022 arXiv

  23. [31]

    Learning to prompt for continual learning,

    Z. Wang, Z. Zhang, C.-Y . Lee, H. Zhang, R. Sun, X. Ren, G. Su, V . Perot, J. Dy, and T. Pfister, “Learning to prompt for continual learning,” in Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2022, pp. 139–149

    2022

  24. [32]

    Dualprompt: Complemen- tary prompting for rehearsal-free continual learning,

    Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y . Lee, X. Ren, G. Su, V . Perot, J. Dy et al., “Dualprompt: Complemen- tary prompting for rehearsal-free continual learning,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 631–648

    2022

  25. [33]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  26. [34]

    Beats: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,” arXiv preprint arXiv:2212.09058, 2022

    2022 arXiv

  27. [35]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  28. [36]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  29. [37]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2022

    2022

  30. [38]

    Lib- rispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  31. [39]

    Com- mon voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Com- mon voice: A massively-multilingual speech corpus,” in Proc. of LREC, 2020

    2020

  32. [40]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  33. [41]

    V oxceleb: a large- scale speaker identification dataset,

    A. Nagrani, J. S. Chung, and A. Zisserman, “V oxceleb: a large- scale speaker identification dataset,” 2017

    2017

  34. [42]

    Lib- risqa: Pioneering free-form and open-ended spoken question an- swering with a novel dataset and framework,

    Z. Zhao, Y . Jiang, H. Liu, Y . Wang, and Y . Wang, “Lib- risqa: Pioneering free-form and open-ended spoken question an- swering with a novel dataset and framework,” arXiv preprint arXiv:2308.10390, 2023

    2023 arXiv

  35. [43]

    Clotho: An audio cap- tioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” in ICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 736–740

    2020

  36. [44]

    Audiobench: A universal benchmark for audio large language models,

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “Audiobench: A universal benchmark for audio large language models,” NAACL, 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.