Pith. sign in

REVIEW 9 cited by

Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.15255 v4 pith:JANS4RF5 submitted 2023-05-24 cs.CL cs.LGcs.SDeess.AS

Eliya Nachmani , Alon Levkovitch , Roy Hirsch , Julian Salazar , Chulayuth Asawaroengchai , Soroosh Mariooryad , Ehud Rivlin , RJ Skerry-Ryan
show 1 more author
Michelle Tadmor Ramanovich
This is my paper · ORCID
classification cs.CLcs.LGcs.SDeess.AS
keywords speechspokencontinuationspectronansweringapproachgithubhttps
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present Spectron, a novel approach to adapting pre-trained large language models (LLMs) to perform spoken question answering (QA) and speech continuation. By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs. The entire system is trained end-to-end and operates directly on spectrograms, simplifying our architecture. Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass. Our method surpasses existing spoken language models in speaker preservation and semantic coherence. Furthermore, the proposed model improves upon direct initialization in retaining the knowledge of the original LLM as demonstrated through spoken QA datasets. We release our audio samples (https://michelleramanovich.github.io/spectron/spectron) and spoken QA dataset (https://github.com/google-research-datasets/LLAMA1-Test-Set).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

    eess.AS 2026-03 unverdicted novelty 7.0 of 10

    FLAIR enables spoken dialogue AI to conduct continuous latent reasoning while perceiving speech through recursive latent embeddings and an ELBO-based finetuning objective.

  2. Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Spoken math models that emit a 40%-compressed reasoning trace between question and answer beat full-reasoning baselines by ~3 accuracy points while using roughly one third of the text tokens.

  3. VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

    cs.SD 2025-10 conditional novelty 6.0 of 10

    A Chinese benchmark built on real human speech evaluates large audio language models across instruction following, knowledge, and robustness, revealing large performance gaps.

  4. NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction

    cs.CL 2025-06 conditional novelty 6.0 of 10

    NTPP models dual-channel spoken dialogue by predicting both speakers' next speech tokens as a pair, achieving speaker-independent full-duplex generation in a decoder-only transformer.

  5. Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models

    cs.SD 2025-05 conditional novelty 6.0 of 10

    AJailBench is an open benchmark showing that large audio-language models can be jailbroken through TTS-converted text attacks and through subtle acoustic perturbations that preserve speech semantics.

  6. RoboEgo System Card: An Omnimodal Model with Native Full Duplexity

    cs.AI 2025-06 conditional novelty 5.0 of 10

    RoboEgo combines a full-duplex audio/text backbone with vision and action heads to achieve an 80 ms theoretical response granularity, reporting competitive quality and better responsiveness than Qwen2.5-Omni in a smal...

  7. VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering

    cs.IR 2025-05 conditional novelty 5.0 of 10

    VoxRAG shows that a spoken query can retrieve topically relevant podcast segments via CLAP audio embeddings and FAISS search, with Recall@10 of 0.60 for somewhat relevant segments, though precise answers remain rare.

  8. Breaking the Barriers of Text-Hungry and Audio-Deficient AI

    cs.SD 2025-06 reject novelty 4.0 of 10

    A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.

  9. Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    In a three-stage end-to-end spoken language model, experience replay (mixing old data into later training) was the most effective mitigation against catastrophic forgetting, greatly outperforming model merging and LoR...

Pith tools