REVIEW 8 cited by
AudioLM: a Language Modeling Approach to Audio Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
Forward citations
Cited by 8 Pith papers
-
Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving
Bounding per-session resident KV eliminates the silent metastable latency cliff in real-time interaction serving and turns frame latency into a usable admission signal.
-
SemanticAudio: Audio Generation and Editing in Semantic Space
SemanticAudio improves text-to-audio alignment by generating a compact semantic plan first with a Flow Matching planner and then rendering acoustic latents from that plan, and it performs training-free audio editing b...
-
CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents
CodecSep performs prompt-driven universal sound separation directly in neural audio codec latents by combining a frozen DAC backbone with a lightweight FiLM-conditioned Transformer masker driven by CLAP embeddings, yi...
-
Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan
Multi-TW is the first Traditional Chinese benchmark to evaluate multimodal models on both image-text and audio-text questions while also measuring inference latency.
-
EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing
Prompt-to-Prompt cross-attention control is adapted to autoregressive audio generation, enabling training-free music editing that outperforms a diffusion baseline.
-
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models
AJailBench is an open benchmark showing that large audio-language models can be jailbroken through TTS-converted text attacks and through subtle acoustic perturbations that preserve speech semantics.
-
OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction
OmniCharacter is a speech-language role-playing agent that generates character-specific voice responses with low latency, trained on a new 10K-dialogue, 135K-audio dataset of 20 game characters.
-
Ten Principles of AI Agent Economics
A conceptual framework of ten principles describing how AI agents might participate in economic systems, grounded mostly in existing economic theory and AI safety concerns.
Discussion (0). Continue with ORCID to comment.