REVIEW 13 cited by
Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce Baichuan-Audio, an end-to-end audio large language model that seamlessly integrates audio understanding and generation. It features a text-guided aligned speech generation mechanism, enabling real-time speech interaction with both comprehension and generation capabilities. Baichuan-Audio leverages a pre-trained ASR model, followed by multi-codebook discretization of speech at a frame rate of 12.5 Hz. This multi-codebook setup ensures that speech tokens retain both semantic and acoustic information. To further enhance modeling, an independent audio head is employed to process audio tokens, effectively capturing their unique characteristics. To mitigate the loss of intelligence during pre-training and preserve the original capabilities of the LLM, we propose a two-stage pre-training strategy that maintains language understanding while enhancing audio modeling. Following alignment, the model excels in real-time speech-based conversation and exhibits outstanding question-answering capabilities, demonstrating its versatility and efficiency. The proposed model demonstrates superior performance in real-time spoken dialogue and exhibits strong question-answering abilities. Our code, model and training data are available at https://github.com/baichuan-inc/Baichuan-Audio
Forward citations
Cited by 13 Pith papers
-
The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning
FLAIR enables spoken dialogue AI to conduct continuous latent reasoning while perceiving speech through recursive latent embeddings and an ELBO-based finetuning objective.
-
Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models
Spoken math models that emit a 40%-compressed reasoning trace between question and answer beat full-reasoning baselines by ~3 accuracy points while using roughly one third of the text tokens.
-
EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs
EchoingPixels prunes audio-visual LLM input tokens jointly across modalities and re-tunes RoPE frequencies so that 5–20% of tokens retain roughly full-model performance.
-
VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
A Chinese benchmark built on real human speech evaluates large audio language models across instruction following, knowledge, and robustness, revealing large performance gaps.
-
XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs
XY-Tokenizer is a 1 kbps dual-channel speech codec that reports simultaneously strong text alignment and high speaker similarity, comparable to specialized codecs at similar bitrates.
-
AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks
A cascaded open-weight speech agent using ReAct reasoning and external tools reaches 92.75% on VoiceBench OpenBookQA and 90% success on 30 multi-turn voice tasks.
-
Raon-Speech Technical Report
A 9B SpeechLM trained on 1.38M hours of English/Korean data plus a full-duplex chat extension trained on 119K hours of time-aligned dialogue outperform same-size audio models on speech tasks and FDB turn-taking metrics.
-
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.
-
MultiAPI Spoof: A Multi-API Dataset and Local-Attention Network for Speech Anti-spoofing Detection
A 30-API, 230-hour spoofed-speech dataset plus a local-attention tweak to Nes2Net improves anti-spoofing and enables API-level source tracing.
-
VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents
A 65.6-hour movie-dialogue benchmark for spoken role-playing agents, with an evaluation framework whose main judge is also an evaluated model.
-
CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation
CodecBench ranks 14 audio codecs on acoustic fidelity and semantic preservation across 19 datasets and four audio domains, revealing a reconstruction-versus-semantics tradeoff.
-
Breaking the Barriers of Text-Hungry and Audio-Deficient AI
A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.
-
S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models
S2SBench quantifies intelligence degradation in speech-input LLMs by comparing perplexity between plausible and implausible continuations, and shows two-stage training of Baichuan-Audio reduces this degradation.
Discussion (0). Continue with ORCID to comment.