Pith. sign in

REVIEW 3 major objections 5 minor 38 references

SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single end-to-end speech-language model can replace the dedicated models for voice-trigger detection, device-directed speech detection, and ASR in a virtual assistant, reporting a 64% relative improvement on voice-trigger EER and a 22%…

desk verdict A solid, incremental multi-task speech LLM paper whose headline gains hinge on baseline/test-set comparisons that the paper leaves under-anchored. read the letter →

arxiv 2501.19377 v2 pith:UFXKO7TL submitted 2025-01-31 cs.SD cs.CLcs.LGeess.AS

Dominik Wagner , Alexander Churchill , Siddharth Sigtia , Erik Marchi This is my paper · ORCID
classification cs.SDcs.CLcs.LGeess.AS
keywords multi-tasklearningmultimodallargelanguagemodelvirtualassistantvoicetriggerdetectiondevice-directedspeechautomaticrecognitionlow-rankadaptationspeech-enabled
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SELMA is a single end-to-end model that takes audio and text as inputs to a large language model and performs five virtual-assistant tasks at once: voice-trigger detection, device-directed speech detection, automatic speech recognition, text-based device-directed speech detection, and dialog-act classification. The paper argues that this one model matches or beats dedicated single-task models, reporting a 64% relative Equal-Error Rate improvement on voice-trigger detection, a 22% relative improvement on device-directed speech detection, and word error rates near a fine-tuned Whisper baseline. If true, the typical assistant pipeline that chains separate trigger, directedness, and ASR components could be collapsed into a single model, simplifying deployment and maintenance. The contribution is an architecture and training recipe, not a new theoretical result.

What carries the argument

The load-bearing component is the concatenated mean-pooled representation: the audio encoder's output sequence H is averaged over time to give a single vector R, which is then prepended to the sequence, producing a combined representation that gives the LLM simultaneous access to global utterance context and fine-grained temporal detail. The audio encoder and the LLM backbone are adapted with LoRA modules on their query and value matrices, so only 5.5M parameters (about 0.84% of the model) are trained. A gating network can optionally modulate the audio features, but the concatenation alone, without gating, is the configuration that achieves the headline results.

What would settle it

Rerun the UAD, ODLD, and [9] baselines on the exact VT and DDSD test sets used for SELMA; if their Equal-Error Rates fall to SELMA's levels on identical audio, the claimed relative gains would disappear.

Watch

Extended reading notes

Core claim

The paper shows that a single multimodal LLM, with the audio encoder and the language model jointly fine-tuned using low-rank adaptation, can handle the acoustic and semantic demands of all three primary assistant tasks simultaneously. The key to its reported success is the way audio is represented: the model keeps the full variable-length sequence of audio representations for tasks like ASR, and concatenates a mean-pooled global summary so tasks like voice-trigger and device-directedness detection can exploit the whole-utterance context. On the in-house test sets, this configuration reaches 7.78% EER on DDSD and 0.12% EER on VT detection, with a WER of 0.125, improving on dedicated baselines. Removing either the sequence or the pooled summary degrades performance, showing that both views are load-bearing.

Load-bearing premise

The reported 64% and 22% improvements assume that the in-house test sets used for SELMA and the published baselines are comparable, since the UAD and ODLD baselines were not rerun on SELMA's exact test audio.

Editorial extensions

If this is right

  • The virtual assistant input pipeline can be reduced from several specialised models to a single model, simplifying deployment and maintenance.
  • Joint training with ASR provides textual context that helps downstream VT and DDSD decisions, especially in noisy audio conditions.
  • LoRA adaptation keeps the trainable parameter count tiny, so the model can be switched between applications by swapping small adapter weights.
  • The concatenated pooled representation is a simple but effective alternative to learned aggregation like Q-Former, achieving comparable or better results without extra parameters.
  • Removing auxiliary tasks such as text-based DDSD or the standalone ASR task degrades VT detection or transcription quality, evidence that the multi-task design contributes to the results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pooled-plus-sequence trick could transfer to other audio-LLM tasks, such as speaker verification or emotion recognition, where both global and local acoustic cues matter.
  • A streaming variant that updates the pooled summary as audio arrives could extend SELMA to always-on wake-word scenarios without waiting for the full utterance.
  • The architecture's value is independent of the exact baseline percentages; a head-to-head re-run on identical test audio is what would pin down the true size of the gains.
  • If the multi-task design scales, the same approach could absorb downstream natural-language understanding, unifying the entire assistant loop in one model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SELMA is a speech-enabled language model that integrates a LoRA-adapted Whisper audio encoder with a Qwen LLM to perform five virtual-assistant tasks (ASR, voice-trigger detection, device-directed speech detection, text-based DDSD, and dialog-act classification) in a single end-to-end system. The architecture concatenates a mean-pooled global audio representation with the full sequence of audio tokens, optionally gated, and is trained with task-specific text prompts. The paper reports large relative EER improvements over dedicated baselines (64% for VT detection, 22% for DDSD) while maintaining competitive ASR WER, and ablation studies isolate the contributions of pooling, gating, auxiliary tasks, and ASR. The main claims are strength and simplicity of a single model replacing a multi-component pipeline.

Significance. If the reported gains hold under a controlled comparison, SELMA would be a significant step toward unifying several virtual-assistant front-end tasks in one multimodal LLM. The approach is parameter-efficient (5.5M trainable parameters out of an 8B-scale model) and uses a simple mean-pooling-plus-sequence design that clearly improves over the configuration with only one of the two. The ablation set is thorough, covering model components, data modalities, and task groupings, and the DET curves provide a more complete view than EER numbers alone. The main weakness is that the headline comparisons against UAD and ODLD rely on baseline numbers from prior work evaluated on test sets that are only 'similar to' or 'updated versions' of the SELMA test sets, and all metrics are single-run point estimates without uncertainty intervals. The paper is otherwise internally consistent, and the central architecture is plausible, so the claims are defensible but not yet fully anchored.

major comments (3)
  1. [Section III-B and Table II] The headline EER improvements (VT: 0.12% vs. UAD 0.33%, a claimed 64% relative gain; DDSD: 7.78% vs. GPT2-XL+Whisper 10.00% and ODLD 12.32%) depend on comparing SELMA to baseline numbers that were not re-computed on the same test distributions. Section III-B states that the VT test set is 'similar to' those in [3] and [17], and that the DDSD test is an 'updated version' of the in-house test from [9]; the text does not state that UAD (from [17]) and ODLD (from [1]) were evaluated on these exact test sets. Even for the DDSD baselines from [9], which share training data, the paper does not confirm they were re-evaluated on the updated test set. To support the relative-improvement claims, the authors should re-run all baselines on the exact SELMA test sets, or else report a detailed analysis of how scores shift across the compared test distributions.
  2. [Table II and Figure 3 (general evaluation)] No confidence intervals, bootstrap estimates, or significance tests are reported for any EER or WER in Table II or for the DET curves in Figure 3. All numbers appear to come from a single training run. The VT gap (0.12% vs. 0.33% EER) is sizable, but without uncertainty bounds the reader cannot judge whether the differences, particularly the smaller DDSD gaps (7.78% vs. 8.76%) and the ablation differences (e.g., SELMA 1 vs. SELMA 4 at 7.78% vs. 7.63% EER), are statistically reliable. The authors should provide confidence intervals for the primary results, for example via bootstrap re-sampling of test utterances or multiple seeds.
  3. [Section III-B (ASR transcript generation)] The ASR transcripts used as ground-truth text for the VT and DDSD training corpora are generated by an auxiliary model with the same architecture as the main SELMA model. This self-training setup is disclosed, but its impact on the VT/DDSD results is not analyzed. If the auxiliary model's transcripts contain systematic errors that correlate with its own weaknesses, the main model could be learning a biased text representation, and the conclusion that 'joint ASR improves VT/DDSD' (supported by the SELMA 1 vs. SELMA 8 ablation) would be less direct. I recommend adding an analysis of the sensitivity of VT/DDSD accuracy to transcript quality, for instance by comparing against human-annotated transcripts on a held-out subset, or by evaluating a version of SELMA that uses a different, non-SELMA ASR to generate the training transcripts.
minor comments (5)
  1. [Table I] Prompts #2 and #5 are identical ('What does the person say and what type of dialog act is this?'), which appears to be a typographical error; if two tasks are intended to share a prompt, this should be stated explicitly, otherwise one of the rows likely was meant to have a different label.
  2. [Section III-C (SELMA 1 description)] The text says SELMA 1 uses 'a concatenation of the mean pooled audio representation sequence and the sequence itself,' but in Section II-B the concatenation is defined as H_{1:K+1} = [R; H_{1:K}], i.e., the pooled vector is prepended to the sequence. Please align the wording with the formal definition.
  3. [Table II / References] The ODLD baseline is cited as '[1]', which in the reference list is the Siri team blog post on voice-trigger systems; if ODLD is described in more than one place, a more specific citation would help readers locate the exact model architecture.
  4. [Figure 3] The caption does not specify whether the DET curves are computed on the same test sets as the numbers in Table II; please add a sentence stating the test set and the number of queries used for each curve.
  5. [Section III-A] The paper states 'The overall system has 5.5M trainable parameters,' but it is not clear whether this figure refers to all SELMA variants or only SELMA 1. Since SELMA 4 adds a gating network and SELMA 5 replaces mean pooling with a Q-Former, the parameter counts may differ; please report the count for each variant.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SELMA's reported gains are empirical comparisons on held-out test sets, and no derived quantity is defined in terms of the quantity it is claimed to predict.

full rationale

The paper is an empirical systems paper, not a mathematical derivation. The central claim—that a LoRA-adapted audio encoder plus LLM with concatenated mean pooling improves VT EER and DDSD EER while keeping WER close to baseline—is evaluated on in-house held-out test sets, with ablations (SELMA 2–8) and independent ASR metrics on human-transcribed test data. The only self-referential elements are (i) an auxiliary SELMA-architecture model that generates ASR transcripts used as training labels for the VT/DDSD corpora and (ii) some baselines taken from the authors' prior work [9]. Neither is load-bearing circularity: the generated transcripts are not the evaluated VT/DDSD/ASR targets (those have separate task labels and human-transcribed test data), and the [9] baselines are externally published empirical results used as comparison points, not as premises that force SELMA's outcomes by construction. A genuine concern is that Section III-B describes the VT test set as only 'similar to' [17]/[3] and the DDSD test as an 'updated version' of [9], so the baseline comparability is a validity risk if the prior baselines were scored on different distributions. That is a correctness/measurement concern, not circularity under the definition used here, because neither the relative improvements nor the EER/WER figures reduce by definition to the paper's inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are postulated. The pooling and gating components are architectural modules, not entities requiring external evidence. The main load-bearing assumptions are the adequacy of the pretrained backbone, comparability of proprietary test sets, and the validity of automatically generated ASR transcripts as training labels for the joint tasks.

free parameters (2)
  • training task mixing weights = 80% main tasks (15% VT, 35% DDSD, 30% ASR), 20% auxiliary (5% text DDSD, 15% DA)
    Selected from preliminary experiments on the same validation data; affects all SELMA variants and is not swept in Table II.
  • LoRA hyperparameters = r=8, alpha=32, dropout=0.1
    Hand-set for all experiments with no sensitivity analysis; these control the parameter-efficient adaptation that is central to the method.
assumptions (4)
  • domain assumption Pretrained Qwen-Audio-Chat (Whisper-large-v2 encoder plus Qwen 7B LLM) provides a sufficiently strong starting point for VT, DDSD, and ASR after LoRA adaptation.
    The entire approach rests on this backbone; no experiments compare to other audio LLMs or backbones.
  • domain assumption In-house test sets and labels for VT, DDSD, and ASR are correct and comparable to the test sets used for the prior baselines.
    The headline EER and WER comparisons rely on this; Section III-B describes test sets only as 'similar' or 'updated versions'.
  • ad hoc to paper ASR transcripts generated by the auxiliary SELMA-style model are adequate ground-truth text for training the main model on VT and DDSD corpora.
    Section III-B states these generated transcripts 'serve as the ground truth labels during the main model's training process'; no human verification or quality metric is reported.
  • domain assumption The token probabilities p_theta(Y=yes) after the special tokens are usable as detection scores for EER thresholding.
    Section II-C uses these probabilities directly as detection scores; calibration is not examined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions." pith.science (2026). https://pith.science/paper/UFXKO7TL

@misc{pith2026250119377,
  author       = {Pith},
  title        = {Pith review of: SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UFXKO7TL}},
  note         = {Machine review of arXiv:2501.19377}
}
read the original abstract

In this work, we present and evaluate SELMA, a Speech-Enabled Language Model for virtual Assistant interactions that integrates audio and text as inputs to a Large Language Model (LLM). SELMA is designed to handle three primary and two auxiliary tasks related to interactions with virtual assistants simultaneously within a single end-to-end model. We employ low-rank adaptation modules for parameter-efficient training of both the audio encoder and the LLM. Additionally, we implement a feature pooling strategy enabling the system to recognize global patterns and improve accuracy on tasks less reliant on individual sequence elements. Experimental results on Voice Trigger (VT) detection, Device-Directed Speech Detection (DDSD), and Automatic Speech Recognition (ASR), demonstrate that our approach both simplifies the typical input processing pipeline of virtual assistants significantly and also improves performance compared to dedicated models for each individual task. SELMA yields relative Equal-Error Rate improvements of 64% on the VT detection task, and 22% on DDSD, while also achieving word error rates close to the baseline.

Figures

Figures reproduced from arXiv: 2501.19377 by the authors.

Figure 1
Figure 1. A typical pipeline for user input processing in virtual assistants (a) and our proposed approach (b). response. More recently, multimodal systems have been employed to jointly consider audio, text and visual modalities for the DDSD task [9], [19], [20]. Pretrained Large Language Models (LLMs) are increasingly deployed on devices, such as smartphones, serving as foundational backbones that can be directly used or fin… view at source ↗
Figure 2
Figure 2. illustrates SELMA, which consists of four main compo￾nents: The first main component 1 is an audio encoder E, which generates a sequence of latent audio representations from an input sequence given by log-Mel spectrogram features of length T. E consists of 1D convolutional layers responsible for downsampling the spectral features followed by S transformer blocks. The result is a high-dimensional tensor, denoted as H… view at source ↗
Figure 3
Figure 3. shows the Detection Error Trade-off (DET) curves for the VT detection (top) and DDSD (bottom) experiments. The EERs, marked by the intersection of the DET curve with the diagonal line correspond to the results presented in Table II. The SELMA models (solid lines) outperform the UAD baselines (dotted line) on the VT 0.00% 0.25% 0.50% 0.75% 1.00% 1.25% FAR 0.00% 0.25% 0.50% 0.75% 1.00% 1.25% FRR VT Detection 0.0% 3.0%… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 29 canonical work pages

  1. [9]

    A multimodal approach to device-directed speech detection with large language models,

    Dominik Wagner, Alexander Churchill, Siddharth Sigtia, Panayiotis Georgiou, Matt Mirsamadi, Aarshee Mishra, and Erik Marchi, “A multimodal approach to device-directed speech detection with large language models,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2024, pp. 10451–10455

    work page 2024

  2. [20]

    Multimodal large language models with fusion low rank adaptation for device directed speech detection,

    Shruti Palaskar, Ognjen Rudovic, Sameer Dharur, Florian Pesce, Gautam Krishna, Aswin Sivaraman, Jack Berkowitz, Ahmed Hussen Abdelaziz, Saurabh Adya, and Ahmed Tewfik, “Multimodal large language models with fusion low rank adaptation for device directed speech detection,” in Interspeech 2024, 2024, pp. 4778–4782

    work page 2024

  3. [17]

    Less is more: A unified architecture for device- directed speech detection with multiple invocation types,

    Oggi Rudovic et al., “Less is more: A unified architecture for device- directed speech detection with multiple invocation types,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023, pp. 1–5

    work page 2023

  4. [3]

    Multichannel Voice Trigger Detection Based on Transform-average-concatenate

    Takuya Higuchi, Avamarie Brueggeman, Masood Delfarah, and Stephen Shum, “Multichannel voice trigger detection based on transform-average-concatenate,” 2024, arXiv:2309.16036

    work page Pith review arXiv 2024

  5. [1]

    V oice trigger system for Siri,

    Siri Team, “V oice trigger system for Siri,” https://machinelearning. apple.com/research/voice-trigger, 2023

    work page 2023

  6. [2]

    Efficient V oice Trigger Detection for Low Resource Hardware,

    Siddharth Sigtia, Rob Haynes, Hywel Richards, Erik Marchi, and John Bridle, “Efficient V oice Trigger Detection for Low Resource Hardware,” in Proc. Interspeech 2018 , 2018, pp. 2092–2096

    work page 2018

  7. [4]

    Accurate Detection of Wake Word Start and End Using a CNN,

    Christin Jose, Yuriy Mishchenko, Thibaud S ´en´echal, Anish Shah, Alex Escott, and Shiv Naga Prasad Vitaladevuni, “Accurate Detection of Wake Word Start and End Using a CNN,” in Proc. Interspeech 2020, 2020, pp. 3346–3350

    work page 2020

  8. [5]

    Low-resource Low-footprint Wake-word Detec- tion using Knowledge Distillation,

    Arindam Ghosh, Mark Fuhs, Deblin Bagchi, Bahman Farahani, and Monika Woszczyna, “Low-resource Low-footprint Wake-word Detec- tion using Knowledge Distillation,” in Proc. Interspeech 2022 , 2022, pp. 3739–3743

    work page 2022

Show all 38 references
  1. [6]

    Convolutional neural networks for small-footprint keyword spotting,

    Tara N. Sainath and Carolina Parada, “Convolutional neural networks for small-footprint keyword spotting,” in Proc. Interspeech 2015, 2015, pp. 1478–1482

    2015

  2. [7]

    Keyword spotting for google assistant using contextual speech recognition,

    Assaf Hurwitz Michaely, Xuedong Zhang, Gabor Simko, Carolina Parada, and Petar Aleksic, “Keyword spotting for google assistant using contextual speech recognition,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2017, pp. 272–278

    2017

  3. [8]

    Contrastive speech mixup for low-resource keyword spotting,

    Dianwen Ng et al., “Contrastive speech mixup for low-resource keyword spotting,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023, pp. 1–5

    2023

  4. [10]

    Learning when to listen: detecting system-addressed speech in human-human-computer dialog,

    Elizabeth Shriberg, Andreas Stolcke, Dilek Hakkani-T ¨ur, and Larry Heck, “Learning when to listen: detecting system-addressed speech in human-human-computer dialog,” in Proc. Interspeech 2012, 2012, pp. 334–337

    2012

  5. [11]

    Device-directed utterance detection,

    Sri Harish Mallidi, Roland Maas, Kyle Goehner, Ariya Rastrow, Spyros Matsoukas, and Bj ¨orn Hoffmeister, “Device-directed utterance detection,” in Interspeech 2018, 2018, pp. 1225–1228

    2018

  6. [12]

    Device-directed speech detection: Regularization via distillation for weakly-supervised models,

    Vineet Garg, Ognjen Rudovic, Pranay Dighe, Ahmed Hussen Abde- laziz, Erik Marchi, Saurabh Adya, Chandra Dhir, and Ahmed Tewfik, “Device-directed speech detection: Regularization via distillation for weakly-supervised models,” in Interspeech 2022, 2022, pp. 1258–1262

    2022

  7. [13]

    Streaming Transformer for Hardware Efficient V oice Trigger Detection and False Trigger Mitigation,

    Vineet Garg, Wonil Chang, Siddharth Sigtia, Saurabh Adya, Pramod Simha, Pranay Dighe, and Chandra Dhir, “Streaming Transformer for Hardware Efficient V oice Trigger Detection and False Trigger Mitigation,” in Proc. Interspeech 2021 , 2021, pp. 4209–4213

    2021

  8. [14]

    Lattice-based improvements for voice triggering using graph neural networks,

    Pranay Dighe, Saurabh Adya, Nuoyu Li, Srikanth Vishnubhotla, De- vang Naik, Adithya Sagar, Ying Ma, Stephen Pulman, and Jason Williams, “Lattice-based improvements for voice triggering using graph neural networks,” in ICASSP 2020 - 2020 IEEE International Conference on Acousti...

    2020

  9. [15]

    V oice trigger detection from lvcsr hypothesis lattices using bidirectional lattice recurrent neural networks,

    Woojay Jeon, Leo Liu, and Henry Mason, “V oice trigger detection from lvcsr hypothesis lattices using bidirectional lattice recurrent neural networks,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 6356– 6360

    2019

  10. [16]

    Exploring attention mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directed,

    Atta Norouzian, Bogdan Mazoure, Dermot Connolly, and Daniel Wil- lett, “Exploring attention mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directed,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech ...

    2019

  11. [18]

    A Study for Improving Device-Directed Speech De- tection Toward Frictionless Human-Machine Interaction,

    Che-Wei Huang, Roland Maas, Sri Harish Mallidi, and Bj ¨orn Hoffmeister, “A Study for Improving Device-Directed Speech De- tection Toward Frictionless Human-Machine Interaction,” in Proc. Interspeech 2019, 2019, pp. 3342–3346

    2019

  12. [19]

    Modality dropout for multimodal device directed speech detection using verbal and non-verbal features,

    Gautam Krishna, Sameer Dharur, Oggi Rudovic, Pranay Dighe, Saurabh Adya, Ahmed Hussen Abdelaziz, and Ahmed H Tewfik, “Modality dropout for multimodal device directed speech detection using verbal and non-verbal features,” 2023, arXiv:2310.15261

    2023 arXiv

  13. [21]

    Speed is all you need: On-device acceleration of large diffusion models via gpu-aware optimizations,

    Yu-Hui Chen, Raman Sarokin, Juhyun Lee, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann, “Speed is all you need: On-device acceleration of large diffusion models via gpu-aware optimizations,” 2023, arXiv:2304.11267

    2023 arXiv

  14. [22]

    Apple intelligence foundation language models,

    Tom Gunter et al., “Apple intelligence foundation language models,” 2024, arXiv:2407.21075

    2024

  15. [24]

    Large language models are zero-shot reasoners,

    Takeshi Kojima, Shixiang Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa, “Large language models are zero-shot reasoners,” in Advances in Neural Information Processing Systems , 2022, vol. 35, pp. 22199–22213

    2022

  16. [25]

    Language models are few-shot learners,

    Tom B. Brown et al., “Language models are few-shot learners,” 2020, arXiv:2005.14165

    2020 arXiv

  17. [26]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro, “Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,” 2024, arXiv:2402.01831

    2024 arXiv

  18. [27]

    SALMONN: Towards generic hearing abilities for large language models,

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang, “SALMONN: Towards generic hearing abilities for large language models,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  19. [28]

    Qwen-audio: Advancing universal audio un- derstanding via unified large-scale audio-language models,

    Yunfei Chu et al., “Qwen-audio: Advancing universal audio un- derstanding via unified large-scale audio-language models,” 2023, arXiv:2311.07919

    2023 arXiv

  20. [29]

    Audiopalm: A large language model that can speak and listen,

    Paul K. Rubenstein et al., “Audiopalm: A large language model that can speak and listen,” 2023, arXiv:2306.12925

    2023 arXiv

  21. [30]

    Pengi: An audio language model for audio tasks,

    Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang, “Pengi: An audio language model for audio tasks,” in Advances in Neural Information Processing Systems , 2023, vol. 36, pp. 18090– 18108

    2023

  22. [31]

    Coding dialogs with the damsl annota- tion scheme,

    Mark Core and James Allen, “Coding dialogs with the damsl annota- tion scheme,” in AAAI Fall Symposium on Communicative Action in Humans and Machines , vol. 56, pp. 28–35. 1997

    1997

  23. [32]

    Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models,

    Dominik Wagner et al., “Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models,” in Third Workshop on Efficient Natural Language and Speech Processing (ENLSP-III) at NeurIPS 2023 , 2023

    2023

  24. [33]

    Attention is all you need,

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, vol. 30

    2017

  25. [34]

    Robust speech recognition via large- scale weak supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022, arXiv:2212.04356

    2022 arXiv

  26. [35]

    Qwen technical report,

    Jinze Bai et al., “Qwen technical report,” 2023, arXiv:2309.16609

    2023 arXiv

  27. [36]

    Quan- tizable transformers: Removing outliers by helping attention heads do nothing,

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort, “Quan- tizable transformers: Removing outliers by helping attention heads do nothing,” in Advances in Neural Information Processing Systems, 2023, vol. 36, pp. 75067–75096

    2023

  28. [37]

    LoRA: Low-rank adaptation of large language models,

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022

    2022

  29. [38]

    Decoupled weight decay regu- larization,

    Ilya Loshchilov and Frank Hutter, “Decoupled weight decay regu- larization,” in International Conference on Learning Representations , 2019

    2019

  30. [39]

    Phi-3 technical report: A highly capable language model locally on your phone,

    Marah Abdin et al., “Phi-3 technical report: A highly capable language model locally on your phone,” 2024, arXiv:2404.14219

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.