Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A CTC-guided adapter that shrinks speech embeddings to text-like lengths lets a frozen LLM follow instructions it never saw in training.

desk verdict The adapter is a neat engineering step, but the near-100% IFR is a format-compliance score, not evidence of zero-shot task performance, and the paper's own SQA accuracies are at chance. read the letter →

arxiv 2412.01145 v2 pith:RMYEUJES submitted 2024-12-02 eess.AS

Ruchao Fan , Bo Ren , Yuxuan Hu , Rui Zhao , Shujie Liu , Jinyu Li This is my paper · ORCID
classification eess.AS
keywords speech-LLMinstructionfollowingratemodalitymatchingCTCalignmentdynamic-windowQFormerzero-shotspeechtranslationquestionansweringfrozenLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the main obstacle to zero-shot instruction-following speech-LLMs is length mismatch between speech and text embeddings, not a lack of speech instruction data. It proposes AlignFormer, an adapter that uses CTC alignment to build dynamic windows over speech frames and QFormer cross-attention to merge each window into one embedding, so the speech sequence is shortened into something close to text length. With the LLM frozen and only paired speech-to-text ASR data used in training, the model is reported to perform zero-shot speech translation and speech question answering. The paper also introduces the instruction following rate (IFR) as a metric that separates 'the model heard the task' from 'the model solved the task.' If the central claim holds, expensive speech instruction data is not the only path to making a speech-LLM follow unseen instructions.

What carries the argument

The central mechanism is the CTC-derived dynamic window inside AlignFormer. A CTC layer computes a frame-level alignment between speech encoder outputs and the ASR transcription; each non-blank token in the CTC path defines a window of speech frames. Dynamic-window QFormer layers then apply an attention mask built from those windows, gathering all frames inside a window into one learned-query embedding, so the output sequence length approaches the text length. The paper compares greedy, forced, and mixed alignment strategies, where mixed alignment uses forced alignment early and mixes greedy paths later to handle the training-inference mismatch.

What would settle it

Run the audio-first AlignFormer pipeline with the greedy CTC alignment replaced by random windows of the same average length; if zero-shot IFR stays near 0.99, alignment is not carrying the result, and if it falls toward the baseline, alignment is essential.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that imperfect modality matching, not missing instruction data, is what keeps speech-LLMs from following unseen text instructions. AlignFormer replaces the generic adapter with a CTC layer plus dynamic-window QFormer: the CTC head produces a token-to-frame alignment, and the QFormer uses that alignment as an attention mask to merge each aligned window of speech encoder frames into a single embedding, shrinking the speech sequence toward text length. With the LLM frozen throughout ASR-only training, the resulting model performs zero-shot speech translation and speech question answering. The reported numbers are an average IFR of 0.99 in the audio-first setting and 0.49 in the instruction-first setting, versus 0.86 and 0.15 for the 80 ms baselines, with Gaokao and speaker-verification QA moving from zero to nonzero IFR. The paper also introduces IFR as a metric to separate 'the model heard the task' from 'the model solved the task.'

Load-bearing premise

The load-bearing premise is that the CTC head, at inference, marks the correct frame boundaries for each text token, so the QFormer windows merge the right frames rather than blending speech from different tokens.

Editorial extensions

If this is right

  • ASR-only training may be enough to make a speech-LLM follow instructions it has never been trained on, so collecting large amounts of speech instruction data is not the only route.
  • The IFR metric gives a way to tell apart cases where the model understood the speech but ignored the instruction from cases where it failed at the task.
  • Audio-first input ordering during training should be preferred when the goal is zero-shot instruction following on instruction-first prompts.
  • Because AlignFormer's windowing already shrinks input length to roughly text length, it can serve as a compression module for long-form ASR and summarization without much WER loss.
  • In principle, any text task the frozen LLM can perform, such as translation, QA, or reasoning, becomes available over speech once the alignment is good enough.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The alignment window is the real module; a stronger CTC or a distillation of alignment from a larger ASR model might push IFR further while preserving the frozen backbone.
  • The method suggests a recipe for other modalities: use a cheap alignment signal to build dynamic windows, then pool each window to one token before entering a frozen text model.
  • Since the speech encoder is ASR-supervised, non-semantic cues such as speaker identity are largely lost in the top-layer features; adding a speaker-embedding stream or a contrastive loss could recover them without changing AlignFormer's windowing.
  • IFR's prompt-sensitivity means cross-paper comparisons need a standardized prompt pool; otherwise gains in IFR may partly reflect prompt-detection choices rather than modality matching.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AlignFormer, a neural adapter for speech-LLMs that uses CTC alignments to define dynamic windows for a QFormer, reducing the length mismatch between speech and text sequences while keeping the LLM frozen. The model is trained on ASR data only, and the authors claim that it achieves zero-shot speech translation (ST) and speech question answering (SQA) by preserving the LLM's instruction-following ability. They introduce an instruction following rate (IFR) metric and report experiments on WavLLM evaluation data, finding that audio-first training yields higher IFR than instruction-first training and that AlignFormer attains near-100% IFR on ST and SQA tasks.

Significance. If the central claims are supported, the work would be significant: demonstrating that a speech-LLM trained only on ASR data can generalize zero-shot to other tasks is an important result. The proposed dynamic-window QFormer is a reasonable architectural contribution, and the systematic study of audio position is informative. However, the evidence is mixed: the ST results (BLEU scores of 14.76 and 15.44 on CoVoST2 and MuST-C) are suggestive of genuine zero-shot translation, whereas the SQA results show accuracies at or below chance despite near-perfect IFR. The IFR metric, as defined, only checks format or language, not whether the task was actually solved. Thus the central claim that AlignFormer 'can perform ST and SQA tasks' is only partially supported. The paper is transparent about several limitations (e.g., hallucinations, speaker-verification chance-level accuracy, low cosine similarity between speech and text embeddings), which is a strength, but those limitations also undercut the headline conclusion.

major comments (4)
  1. [Section V-B, Table III] The claim that AlignFormer 'can perform speech translation (ST) and speech question answering (SQA) tasks in a zero-shot manner' is not supported for SQA. In Table III, E1-audio-first + AlignFormer reports SQA IFR values of 1.0, 1.0, and 0.99 on Emotion, Speaker Verification, and Gaokao, respectively, but the corresponding accuracies are 31.18% (four-class, chance 25%), 50.15% (two-class, chance 50%), and 2.45% (three-choice, chance 33.3%). The metric IFR, as defined in Section III-C, detects only whether the response matches a prescribed format or target language; it does not verify that the speech content was used or that the question was answered correctly. The near-100% IFR therefore establishes format compliance, not zero-shot SQA performance. The paper's own discussion acknowledges that speaker verification accuracy is at chance and that the model is 'hard to infer speaker information from text,' which is inconsistent with the claim that AlignFormer performs SQA. The authors should either revise the central claim to focus on format-following or provide accuracy-based evidence that the model can solve SQA tasks above chance.
  2. [Section III-C] The definition of IFR is circular with respect to the paper's goal of measuring 'instruction following.' An instruction such as 'answer the question with choice A/B/C' contains a semantic requirement (choose the correct answer) as well as a formal one (output the specified format). IFR detects only the formal component. The paper itself states in Section III-C that 'low scores may come from the model not following the instructions,' but IFR cannot distinguish between a model that follows the format while ignoring the content and a model that actually solves the task. This limitation is not merely presentation; it directly affects the interpretation of Tables II-V. The paper should report IFR together with task metrics and clearly define what 'followed' means, or rename the metric to something like 'format compliance rate.'
  3. [Section V-A, Table II] The conclusion that 'audio-first training achieves higher IFR than instruction-first training' is confounded by the different prompts used in the two conditions. E1 uses 'Transcribe the audio clip into text.' while E2 uses 'Repeat exactly what the user says word by word.' These prompts differ not only in audio position but also in wording and difficulty. The E3/E4 experiments introduce audio versions of the instruction, but they do not include a control condition with instruction-first training using the same 'Transcribe' prompt. Without such a control, the observed IFR difference could be due to prompt wording rather than audio position. The authors should add a matched-prompt experiment or temper the claim accordingly.
  4. [Section V-D, Table V] The ablation study shows a strong dependence on CTC-head initialization: with a randomly initialized CTC head, Librispeech test-other WER degrades from 6.47% to 75.18% and the average WavLLM IFR drops from 0.99 to 0.87. This is acknowledged in the text, but it implies that the method's success relies heavily on the supervised ASR model used for initialization of the speech encoder and CTC head. The paper's claim that the model is trained 'with ASR data only' is true in a narrow sense, but the alignment capability comes from a pretrained ASR model rather than being learned from scratch. The authors should clearly separate the contribution of the AlignFormer architecture from the contribution of the ASR-pretrained initialization, and discuss whether the results would hold with a less capable or differently initialized CTC head.
minor comments (5)
  1. [Abstract] The phrase 'has gaining increased attention' is ungrammatical; it should be 'has gained increased attention.'
  2. [Section III-C] The text refers to 'BLUE scores' for speech translation; the correct acronym is 'BLEU.'
  3. [Table III caption] The caption of Table III is very long and contains methodological notes that belong in the body text; consider moving the explanations of SFT-LLM-Freeze and SFT-LLM-LoRA to Section V-B.
  4. [Abstract and Section V-B] The phrase 'game-changing improvements' is informal and subjective; replace it with a quantitative statement, for example 'improving IFR on Gaokao from 0% to 44%.'
  5. [Section V-D] The paper states that 'the model with greedy-alignment training cannot always follow the repeat prompt on Librispeech test-other data,' but no IFR number is reported for ASR tasks; please either provide a quantifiable metric or remove the unquantified claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: AlignFormer's IFR results are empirical measurements, with the IFR metric's format-detection design a validity caveat rather than a derived prediction.

full rationale

I find no significant circularity. The zero-shot results are measurements on held-out external benchmarks (WavLLM, Librispeech, CoVoST2, FLEURS), not derivations from fitted parameters: the CTC loss is supervised by ASR ground truth, the LLM is a frozen external Phi3.1 model, and no ST or SQA target is seen during training. The AlignFormer design draws on the authors' prior CTC-alignment work ([35], [43]), but those citations are architectural antecedents, not load-bearing uniqueness claims, and the paper also compares against WavLLM, Whisper+LLM, and CTC-out cascades. The one genuinely self-referential element is the IFR metric: Section III-C defines a 'followed' answer as one matching the prompt-specified format or target language, so near-100% IFR measures format compliance by construction. The paper itself flags the limits of IFR, noting it is not computed for ASR or open-ended tasks, and Table III reports SQA accuracies at or below chance despite high IFR. This is a measurement-validity caveat about the 'perform SQA' headline, not an equation-level circular reduction; the ST result retains independent content through BLEU scores.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the pretrained ASR encoder, the frozen Phi3 instruct model, and hand-chosen training constants such as the CTC loss weight and alignment mixing schedule. No target-task constants are fitted. The IFR metric is a new construct rather than an external benchmark, and the CTC alignment assumption is the most fragile piece.

free parameters (3)
  • CTC loss weight lambda = 0.3
    Equation (2) uses L = L_ntp + lambda * L_CTC; lambda is set to 0.3 in Section IV-A2 and is not swept or derived from the target tasks.
  • Greedy alignment mixing probability p_greedy = linearly increased to 0.5
    Section IV-A2: forced alignment is used for the first 7k steps, then greedy alignment is mixed with probability reaching 0.5 at the end. This schedule is hand-chosen.
  • Baseline QFormer window size = 4
    Section IV-A1: window size 4 groups four 80ms encoder outputs into one embedding at 320ms, chosen to approximate AlignFormer's token rate. It is a comparison setting, not a target-fitted value.
assumptions (6)
  • domain assumption Frozen Phi3.1-mini-instruct retains its text instruction-following behavior and can apply it to speech-derived embeddings.
    Section III-A: the whole zero-shot claim assumes that freezing the LLM preserves instruction following and that placing audio embeddings in the LLM input space triggers it.
  • domain assumption CTC alignment from the ASR-trained head is a valid token-to-frame segmentation for the dynamic windows.
    Section III-B builds attention masks from CTC paths; Section V-D shows random CTC initialization severely degrades results, so this premise is load-bearing.
  • domain assumption Speech encoder features carry enough semantic content for translation and QA despite ASR supervision.
    Section IV-A1 initializes the encoder from an AED ASR model; the zero-shot ST and SQA results rely on this semantic content.
  • ad hoc to paper At inference, instruction-first input order is the right probe for zero-shot capability.
    Section IV-B2 states Phi3.1 was post-trained instruction-first, so zero-shot evaluations use instruction-first; this is a stated choice, not a proven requirement.
  • ad hoc to paper Audio-first training is equivalent to training with many different instructions and therefore preserves instruction following.
    Section V-A states this hypothesis to explain E1's high IFR; it is post hoc and not directly measured.
  • ad hoc to paper Format matching in the response is a valid proxy for instruction following.
    Section III-C defines IFR by detecting requested answer formats and explicitly notes the metric is sensitive to prompts; no human validation of the detection is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM." pith.science (2026). https://pith.science/paper/RMYEUJES

@misc{pith2026241201145,
  author       = {Pith},
  title        = {Pith review of: AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMYEUJES}},
  note         = {Machine review of arXiv:2412.01145}
}
read the original abstract

Integrating speech into LLM (speech-LLM) has gaining increased attention recently. The mainstream solution is to connect a well-trained speech encoder and LLM with a neural adapter. However, the length mismatch between the speech and text sequences are not well handled, leading to imperfect modality matching between the speech and text. In this work, we propose a novel neural adapter, AlignFormer, to reduce the length gap between the two modalities. AlignFormer consists of CTC and dynamic-window QFormer layers, where the CTC alignment provides the dynamic window information for QFormer. The LLM backbone is frozen in training to preserve its text capability, especially the instruction following capability. When training with ASR data only, the proposed AlignFormer unlocks the instruction following capability for speech-LLM and the model can perform zero-shot speech translation (ST) and speech question answering (SQA) tasks. In fact, speech-LLM with AlignFormer can theoretically perform any tasks that the LLM backbone can deal with in the speech version. To evaluate the effectiveness of the instruction-following speech-LLM, we propose to use instruction following rate (IFR) and offer a systematic perspective for the IFR evaluation. In addition, we find that the audio position in training would affect the instruction following capability of speech-LLM and conduct an in-depth study on it. Our findings show that audio-first training achieves higher IFR than instruction-first training. The AlignFormer can achieve a near 100% IFR with audio-first training and game-changing improvements from zero to non-zero IFR on some evaluation data with instruction-first training. We believe that this study is a big step towards the perfect speech and text modality matching in the LLM embedding space.

Figures

Figures reproduced from arXiv: 2412.01145 by the authors.

Figure 1
Figure 1. (a) The architecture of speech-LLM with the ASR task prompt. (b) Different organizations of the LLM input embeddings: audio first or instruction [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities

    eess.AS 2025-05 conditional novelty 5.0 of 10

    Granite-speech-3.3-2b and Granite-speech-3.3-8b achieve competitive English ASR word error rates, with the 8B model beating several larger proprietary models on multiple public benchmarks while remaining fully open-source.

Reference graph

Works this paper leans on

48 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    Language models are few-shot learners,

    T. B. Brown, B. Mannet al., “Language models are few-shot learners,” inAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020

    work page 2020

  2. [2]

    Gpt-4 technical report,

    J. Achiam, S. Adleret al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Self-instruct: Aligning language models with self- generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), 2023, pp. 13 484–13 508

    work page 2023

  5. [5]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  6. [6]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  7. [7]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar ´e, M. Orsini, A. Royer, P. P ´erez, H. J ´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    arXiv 2024

  8. [8]

    Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm,

    X. Wang, Y . Li, C. Fu, L. Xie, K. Li, X. Sun, and L. Ma, “Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm,”arXiv preprint arXiv:2411.00774, 2024

    arXiv 2024

Show all 48 references
  1. [9]

    Mini-omni2: Towards open-source gpt-4o with vi- sion, speech and duplex capabilities,

    Z. Xie and C. Wu, “Mini-omni2: Towards open-source gpt-4o with vi- sion, speech and duplex capabilities,”arXiv preprint arXiv:2410.11190, 2024

    2024 arXiv

  2. [10]

    Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation,

    W. Yu, S. Wang, X. Yang, X. Chen, X. Tian, J. Zhang, G. Sun, L. Lu, Y . Wang, and C. Zhang, “Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation,”arXiv preprint arXiv:2411.18138, 2024

    2024 arXiv

  3. [11]

    Speechgpt: Empowering large language models with intrinsic cross- modal conversational abilities,

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y . Zhou, and X. Qiu, “Speechgpt: Empowering large language models with intrinsic cross- modal conversational abilities,”arXiv preprint arXiv:2305.11000, 2023

    2023 arXiv

  4. [12]

    Lauragpt: Listen, attend, understand, and regenerate audio with gpt,

    Z. Du, J. Wang, Q. Chen, Y . Chu, Z. Gao, Z. Li, K. Hu, X. Zhou, J. Xu, Z. Maet al., “Lauragpt: Listen, attend, understand, and regenerate audio with gpt,”arXiv preprint arXiv:2310.04673, 2023

    2023 arXiv

  5. [13]

    An embarrassingly simple approach for llm with strong asr capacity,

    Z. Ma, Y . Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang, and X. Chen, “An embarrassingly simple approach for llm with strong asr capacity,”ArXiv, vol. abs/2402.08846, 2024

    2024 arXiv

  6. [14]

    SALMONN: towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “SALMONN: towards generic hearing abilities for large language models,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024

    2024

  7. [15]

    Qwen2-audio technical report,

    Y . Chu, J. Xuet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  8. [16]

    On decoder-only architecture for speech- to-text and large language model integration,

    J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y . Wu, “On decoder-only architecture for speech- to-text and large language model integration,” inIEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, D...

    2023

  9. [17]

    COSMIC: data efficient instruction-tuning for speech in-context learn- ing,

    J. Pan, J. Wu, Y . Gaur, S. Sivasankaran, Z. Chen, S. Liu, and J. Li, “COSMIC: data efficient instruction-tuning for speech in-context learn- ing,”Interspeech, 2024

    2024

  10. [18]

    Prompting large language models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinliet al., “Prompting large language models with speech recognition abilities,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing ...

    2024

  11. [19]

    Speechverse: A large-scale generalizable audio language model,

    N. Das, S. Dingliwal, S. Ronanki, R. Paturi, Z. Huang, P. Mathur, J. Yuan, D. Bekal, X. Niu, S. M. Jayanthiet al., “Speechverse: A large-scale generalizable audio language model,”arXiv preprint arXiv:2405.08295, 2024

    2024 arXiv

  12. [20]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”Trans. Mach. Learn. Res., vol. 2023, 2023

    2023

  13. [21]

    High- fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High- fidelity audio compression with improved rvqgan,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  14. [22]

    Seed-asr: Understanding diverse speech and con- texts with llm-based speech recognition,

    Y . Bai, J. Chenet al., “Seed-asr: Understanding diverse speech and con- texts with llm-based speech recognition,”ArXiv, vol. abs/2407.04675, 2024

    2024 arXiv

  15. [23]

    Wavllm: Towards robust and adaptive speech large language model,

    S. Hu, L. Zhouet al., “Wavllm: Towards robust and adaptive speech large language model,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16,

    2024

  16. [24]

    Audiochatllama: Towards general-purpose speech abilities for llms,

    Y . Fathullah, C. Wu, E. Lakomkin, K. Li, J. Jia, S. Yuan, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, “Audiochatllama: Towards general-purpose speech abilities for llms,” inNorth American Chapter of the Association for Computational Linguistics, 2023. [Online]. Avai...

    2023

  17. [25]

    Blsp: Bootstrapping language-speech pre-training via behavior align- ment of continuation writing,

    C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y . Liu, C. Zong, and J. Zhang, “Blsp: Bootstrapping language-speech pre-training via behavior align- ment of continuation writing,”ArXiv, vol. abs/2309.00916, 2023

    2023 arXiv

  18. [26]

    Developing instruction-following speech language model without speech instruction-tuning data,

    K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, J. Balam, B. Ginsburg, Y .-C. F. Wang, and H. yi Lee, “Developing instruction-following speech language model without speech instruction-tuning data,”ArXiv, vol. abs/2409.20007, 2024

    2024 arXiv

  19. [27]

    Frozen large language models can perceive paralinguistic aspects of speech,

    W. Kang, J. Jia, C. Wu, W. Zhou, E. Lakomkin, Y . Gaur, L. Sari, S. Kim, K. Li, J. Mahadeokaret al., “Frozen large language models can perceive paralinguistic aspects of speech,”arXiv preprint arXiv:2410.01162, 2024

    2024 arXiv

  20. [28]

    Wav2Prompt: End-to-end speech prompt learning and task-based fine-tuning for text-based LLMs,

    K. Deng, G. Sun, and P. Woodland, “Wav2Prompt: End-to-end speech prompt learning and task-based fine-tuning for text-based LLMs,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techn...

    2025

  21. [29]

    Phi-3 technical report: A highly capable language model locally on your phone,

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behlet al., “Phi-3 technical report: A highly capable language model locally on your phone,”arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  22. [30]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture- of-loras,

    A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V . Chaudhary, C. Chenet al., “Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture- of-loras,”arXiv preprint arXiv:2503.01743, 2025

    2025 arXiv

  23. [31]

    Connec- tionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. J. Gomez, and J. Schmidhuber, “Connec- tionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” inMachine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh, Penn...

    2006

  24. [32]

    CASS-NAT: CTC alignment- based single step non-autoregressive transformer for speech recognition,

    R. Fan, W. Chu, P. Chang, and J. Xiao, “CASS-NAT: CTC alignment- based single step non-autoregressive transformer for speech recognition,” inIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021. IEEE, 2021,...

    2021

  25. [33]

    Unienc-cassnat: An encoder-only non-autoregressive asr for speech ssl models,

    R. Fan, N. B. Shankar, and A. Alwan, “Unienc-cassnat: An encoder-only non-autoregressive asr for speech ssl models,”IEEE Signal Processing Letters, vol. 31, pp. 711–715, 2024

    2024

  26. [34]

    Ctc-based compression for direct speech translation,

    M. Gaido, M. Cettolo, M. Negri, and M. Turchi, “Ctc-based compression for direct speech translation,” inProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: 10 Main V olume, EACL 2021, Online, April 19 - 23, 2021. Associa...

    2021

  27. [35]

    CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation,

    R. Zhao, J. Li, R. Fan, and M. Post, “CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation,”SLT, 2024

    2024

  28. [36]

    SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing,

    J. Ao, R. Wang, L. Zhou, C. Wanget al., “SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing,” inProc. ACL, 2022, pp. 5723–5738

    2022

  29. [37]

    SpeechLM: Enhanced speech pre-training with unpaired textual data,

    Z. Zhang, S. Chen, L. Zhou, Y . Wu, S. Ren, S. Liu, Z. Yao, X. Gong, L. Dai, J. Liet al., “SpeechLM: Enhanced speech pre-training with unpaired textual data,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  30. [38]

    Seamless: Multi- lingual expressive and streaming speech translation,

    L. Barrault, Y .-A. Chung, M. C. Meglioliet al., “Seamless: Multi- lingual expressive and streaming speech translation,”arXiv preprint arXiv:2312.05187, 2023

    2023 arXiv

  31. [39]

    M-adapter: Modality adaptation for end-to-end speech-to-text translation,

    J. Zhao, H. Yang, G. Haffari, and E. Shareghi, “M-adapter: Modality adaptation for end-to-end speech-to-text translation,” inProc. Inter- speech, 2022, pp. 111–115

    2022

  32. [40]

    MAESTRO: Matched speech text representa- tions through modality matching,

    Z. Chen, Y . Zhang, A. Rosenberg, B. Ramabhadran, P. J. Moreno, A. Bapna, and H. Zen, “MAESTRO: Matched speech text representa- tions through modality matching,” inProc. Interspeech, 2022, pp. 4093– 4097

    2022

  33. [41]

    Cjst: Ctc compressor based joint speech and text training for decoder-only asr,

    W. Zhou, J. Jia, L. Sari, J. Mahadeokar, and O. Kalinli, “Cjst: Ctc compressor based joint speech and text training for decoder-only asr,”

  34. [42]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022

    2022

  35. [43]

    Available: https://arxiv.org/abs/2411.07607

    [Online]. Available: https://arxiv.org/abs/2411.07607

    arXiv

  36. [44]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in21st Annual Conference of the International Speech Communication Association, Interspeech 2020, Vir...

    2020

  37. [45]

    A CTC alignment-based non- autoregressive transformer for end-to-end automatic speech recognition,

    R. Fan, W. Chu, P. Chang, and A. Alwan, “A CTC alignment-based non- autoregressive transformer for end-to-end automatic speech recognition,” IEEE ACM Trans. Audio Speech Lang. Process., vol. 31, pp. 1436–1448, 2023

    2023

  38. [46]

    Unsu- pervised cross-lingual representation learning at scale,

    A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzm´an, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsu- pervised cross-lingual representation learning at scale,” inProceedings of the 58th Annual Meeting of the Association for Computational Linguis...

    2020

  39. [47]

    Zero: Memory optimizations toward training trillion parameter models,

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “Zero: Memory optimizations toward training trillion parameter models,” inSC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020, pp. 1–16

    2020

  40. [2024]

    4552–4572

    Association for Computational Linguistics, 2024, pp. 4552–4572

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.