REVIEW 32 cited by
LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.
Forward citations
Cited by 32 Pith papers
-
Dual Information Speech Language Models for Emotional Conversations
A dual-adapter design with equivalence replacement regularization lets frozen LLMs perceive both paralinguistic and linguistic information from speech for emotional conversation.
-
AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation
With prompt engineering (audio concatenation plus in-context examples), large audio models rank speech synthesis systems in line with human preferences, reaching up to 0.91 Spearman correlation.
-
Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving
Bounding per-session resident KV eliminates the silent metastable latency cliff in real-time interaction serving and turns frame latency into a usable admission signal.
-
Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Cocktail-Talker uses three action tokens and GRPO to make a speech LLM decide whether to respond, keep listening, or ignore audio in noisy multi-speaker conversations.
-
TokAN: Accent Normalization Using Self-Supervised Speech Tokens
TokAN maps L2 speech tokens to L1-like tokens via an autoregressive converter plus GRPO rewards, cutting WER to 9.23% on seven English accents without natural parallel L1-L2 recordings.
-
Controllable Accent Normalization via Discrete Diffusion
Masked discrete diffusion over SSL speech tokens plus a Common Token Predictor yields the lowest WER among compared accent-normalization systems and continuous accent-strength control via source-token reuse.
-
Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
Language-aware query selection with a gated query bank improves multilingual, ASR-only-distilled speech LLMs on instruction following and spoken QA.
-
AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues
Current omni-modal LLMs underperform on audio-visual emotional reasoning, and automatic scores diverge from human perceptual judgments; AV-EMO-Reasoning provides a benchmark to measure this.
-
LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
LLaSO releases a 3.8B speech-language model, 25.5M training instances, and an evaluation benchmark, claiming a normalized score of 0.72.
-
Personalized Socially Assistive Robots With End-to-End Speech-Language Models For Well-Being Support
An end-to-end speech-language model integrated into a social robot was perceived as empathetic and natural in a small study, but movement and voice expressiveness remain weak.
-
KERAG_R: Knowledge-Enhanced Retrieval-Augmented Generation for Recommendation
KERAG_R improves LLM-based top-k recommendation by using a GAT to select relevant KG triples and incorporating them into instruction-tuned prompts, reporting gains over ten baselines on three datasets.
-
SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant
SOVA-Bench is a new evaluation framework for speech LLMs covering knowledge, recognition, linguistic and paralinguistic understanding, and semantic and acoustic generation quality.
-
StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion
StarVC is an autoregressive voice conversion model that generates text tokens before acoustic tokens, improving linguistic fidelity while retaining speaker similarity.
-
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
NTPP models dual-channel spoken dialogue by predicting both speakers' next speech tokens as a pair, achieving speaker-independent full-duplex generation in a decoder-only transformer.
-
Universal Visuo-Tactile Video Understanding for Embodied Interaction
VTV-LLM is a tactile-video large language model, trained on a new VTV150K dataset, that reasons about hardness, protrusion, elasticity, and friction in natural language.
-
Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling
SMLLE generates speech frame-by-frame using a Transducer for streaming semantic tokens plus a fully autoregressive mel-spectrogram model, reaching quality close to sentence-level zero-shot TTS.
-
MFA-KWS: Effective Keyword Spotting with Multi-head Frame-asynchronous Decoding
A joint CTC and token-and-duration transducer keyword spotter with frame-skipping and CDC-Last score fusion reports better recall on Snips, MobvoiHotwords, and LibriKWS-20 while decoding 1.47 to 1.63 times faster.
-
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation
VoiceStar uses a progress-based rotary position embedding and mixed prompt training to give zero-shot voice cloning precise duration control and much longer output than training clips.
-
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models
AJailBench is an open benchmark showing that large audio-language models can be jailbroken through TTS-converted text attacks and through subtle acoustic perturbations that preserve speech semantics.
-
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.
-
FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations
A full-duplex voice system with streaming personalized VAD and semantic end-of-turn detection reports fewer false barge-ins and latencies near commercial benchmarks.
-
Unlocking Speech Instruction Data Potential with Query Rewriting
A multi-LLM rewriting and multi-agent validation pipeline makes text-to-speech synthesized speech instruction data far more usable and improves downstream speech instruction following.
-
TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
TESU-LLM shows that a frozen LLM can answer spoken queries after training only a 13M-parameter projector on text, using SeamlessM4T's shared speech-text encoder.
-
Chain-of-Thought Training for Open E2E Spoken Dialogue Systems
Staged ASR-to-text-response-to-TTS training makes open end-to-end spoken dialogue systems trainable on 300 hours of public human-human data and more coherent than one-step speech-to-speech models.
-
OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction
OmniCharacter is a speech-language role-playing agent that generates character-specific voice responses with low latency, trained on a new 10K-dialogue, 135K-audio dataset of 20 game characters.
-
Speechless: Speech Instruction Training Without Speech for Low Resource Languages
Fine-tuning an LLM on text instructions converted to Whisper semantic tokens enables it to understand spoken instructions at inference, bypassing TTS and speech instruction data.
-
SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding
SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...
-
BoSS: Beyond-Semantic Speech
Current spoken-language models perform poorly on a new five-task evaluation of beyond-semantic speech signals, including dialect, emotion, age, and non-verbal cues.
-
Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models
A survey that organizes audio and video AI security research into adversarial, backdoor, and jailbreak attacks, with extra attention to multimodal large language models.
-
ModRWKV: Transformer Multimodality in Linear Time
A linear RNN backbone (RWKV7) with lightweight adapters can handle vision, speech, and time-series inputs, with competitive vision and speech results but unreliable time-series evaluation.
-
S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models
S2SBench quantifies intelligence degradation in speech-input LLMs by comparing perplexity between plausible and implausible continuations, and shows two-stage training of Baichuan-Audio reduces this degradation.
-
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
Llama-SMoP-DEDR, a sparse mixture of projectors with modality-specific experts and routers, lowers word error rate for LLM-based AVSR on LRS3, mainly with smaller LLMs.
Discussion (0). Sign in to comment.