Pith. sign in

REVIEW 17 cited by

Listen, Think, and Understand

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.10790 v3 pith:RWKJU5PL submitted 2023-05-18 eess.AS cs.SD

Yuan Gong , Hongyin Luo , Alexander H. Liu , Leonid Karlinsky , James Glass This is my paper · ORCID
classification eess.AScs.SD
keywords audioabilitymodelsexistinglistenperceptionreasoningthink
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The ability of artificial intelligence (AI) systems to perceive and comprehend audio signals is crucial for many applications. Although significant progress has been made in this area since the development of AudioSet, most existing models are designed to map audio inputs to pre-defined, discrete sound label sets. In contrast, humans possess the ability to not only classify sounds into general categories, but also to listen to the finer details of the sounds, explain the reason for the predictions, think about what the sound infers, and understand the scene and what action needs to be taken, if any. Such capabilities beyond perception are not yet present in existing audio models. On the other hand, modern large language models (LLMs) exhibit emerging reasoning ability but they lack audio perception capabilities. Therefore, we ask the question: can we build a model that has both audio perception and a reasoning ability? In this paper, we propose a new audio foundation model, called LTU (Listen, Think, and Understand). To train LTU, we created a new OpenAQA-5M dataset consisting of 1.9 million closed-ended and 3.7 million open-ended, diverse (audio, question, answer) tuples, and have used an autoregressive training framework with a perception-to-understanding curriculum. LTU demonstrates strong performance and generalization ability on conventional audio tasks such as classification and captioning. More importantly, it exhibits emerging audio reasoning and comprehension abilities that are absent in existing audio models. To the best of our knowledge, LTU is one of the first multimodal large language models that focus on general audio (rather than just speech) understanding.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 18 citations worldwide. Full citation record

  1. MixAssist: An Audio-Language Dataset for Co-Creative AI Assistance in Music Mixing

    cs.SD 2025-07 conditional novelty 7.0 of 10

    MixAssist is the first audio-grounded, multi-turn conversational dataset for co-creative music mixing instruction, and fine-tuning Qwen-Audio on it yields human-comparable mixing advice.

  2. SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

    cs.CV 2025-06 conditional novelty 7.0 of 10

    SAVVY-Bench tests audio-visual LLMs on dynamic 3D spatial questions, and the SAVVY pipeline, combining visual tracks with spatial audio and global mapping, lifts Gemini-2.5-pro accuracy from 50.9% to 58.0%.

  3. Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

    eess.AS 2026-07 conditional novelty 6.0 of 10

    AV-Flamingo is an open audio-visual language model that outperforms similarly sized open models on many long-video understanding benchmarks, using a new 7M-question dataset and timestamped reasoning.

  4. Unified Audio Intelligence Without Regressing on Text Intelligence

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified 30B MoE audio-text LLM achieves state-of-the-art audio understanding, generation, and speech tasks while preserving text reasoning comparable to its text-only backbone.

  5. Adaptive Perturbation Selection for Contrastive Audio Decoding

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    A learned per-example router over a 105-perturbation audio library improves contrastive decoding for audio-LLM hallucination, with task-dependent best distortions (e.g., reverse audio for temporal order).

  6. Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models

    eess.AS 2026-04 unverdicted novelty 6.0 of 10

    Self-distillation of Qwen3-Omni-Thinking on 545k Cogito-Pipe audio reasoning traces yields the best open-source MMAR CoT scores and top-tier challenge ranking.

  7. Teaching Physical Awareness to LLMs through Sounds

    cs.SD 2025-06 conditional novelty 6.0 of 10

    ACORN trains LLMs to answer acoustic physics questions from simulated channels and transfers zero-shot to a small real-vehicle test.

  8. ZeroSep: Separate Anything in Audio with Zero Training

    cs.SD 2025-05 conditional novelty 6.0 of 10

    Latent inversion of a mixed audio into a pretrained text-guided diffusion model, followed by denoising with classifier-free guidance weight 1, performs zero-training source separation.

  9. Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models

    cs.SD 2025-05 conditional novelty 6.0 of 10

    AJailBench is an open benchmark showing that large audio-language models can be jailbroken through TTS-converted text attacks and through subtle acoustic perturbations that preserve speech semantics.

  10. AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning

    eess.AS 2025-10 conditional novelty 5.0 of 10

    Test-time reinforcement learning against self-generated majority-vote pseudo-labels improves audio question answering accuracy on MMAU, MMAR, and MMSU.

  11. Improving Audio Event Recognition with Consistency Regularization

    cs.SD 2025-09 conditional novelty 5.0 of 10

    Consistency regularization improves audio event recognition on AudioSet by about 2 mAP, both in supervised and semi-supervised settings.

  12. Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition

    eess.AS 2025-06 conditional novelty 5.0 of 10

    A speech large language model trained on beamformed multi-channel audio performs directional speech recognition and source localization across 12 discrete angles on simulated smart glasses data.

  13. CoLMbo: Speaker Language Model for Descriptive Profiling

    cs.CL 2025-06 reject novelty 5.0 of 10

    CoLMbo pairs a fixed speaker encoder with a small language model to write descriptive profiles from voice, reporting high zero-shot accuracy for age, gender, ethnicity, and dialect.

  14. ALAS: An Automatic Latent Alignment Score for Audio Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    ALAS is a reference-based score for audio-text alignment in speech LLMs, computed from frozen hidden states and a Whisper-derived alignment path, with no training or fitted classifier.

  15. LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs

    cs.AI 2025-05 conditional novelty 5.0 of 10

    LiSTEN shows that dynamically selecting a few learnable prompt tokens from a shared pool can replace LoRA fine-tuning for audio-language models, matching or beating it with less training data.

  16. Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs

    cs.CL 2025-08 unverdicted novelty 4.0 of 10

    Under matched settings, continuous SSL speech features generally outperform discrete tokens on six spoken language understanding tasks in SpeechLLMs.

  17. When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs

    cs.CV 2025-02 conditional novelty 3.0 of 10

    A survey that classifies VLM attacks by goal and data manipulation strategy, and reviews defenses and metrics.

Pith tools