REVIEW 5 major objections 6 minor 1 cited by
S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read S2SBench quantifies intelligence degradation in end-to-end speech LLMs with a pairwise perplexity protocol and shows that freezing the language model during early audio training mitigates the loss.
desk verdict A useful, reusable benchmark idea whose headline claims currently rest on an undescribed TTS pipeline and a single unseeded run; worth refereeing, not worth believing as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of S2SBench is a pairwise perplexity protocol. For each test item the dataset provides a positive continuation or correct option and a negative continuation or wrong option sharing the same opening; the model is correct when the positive sample gets lower perplexity than the negative one, and accuracy is the fraction of items where that holds. Running identical items once as text tokens and once as audio tokens makes the accuracy gap a direct measure of what audio input costs in reasoning, and the difference between positive and negative perplexities gives a finer-grained confidence signal. The training-strategy comparison then uses the same protocol as a probe: freezing the language model during the first audio-training stage and unfreezing it later is the intervention whose effect the benchmark detects.
What would settle it
Record the S2SBench items with several different speakers, prosodies, and TTS systems and re-run the audio condition: if the speech-to-text accuracy gap changes by as much as the reported degradation, the benchmark is partly measuring the audio pipeline rather than the model's reasoning loss.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that intelligence degradation in fully end-to-end speech LLMs is real, measurable, and partially preventable. Using identical sentence-continuation and commonsense items in text and audio form, Baichuan-Audio scores lower in speech-to-text mode than in text-to-text mode on every dataset (for example, 77.5 versus 83.0 on English story continuation under single-stage training), so the gap is an effect of the audio input channel rather than of task difficulty. The pairwise perplexity protocol captures the same gap that accuracy does, and the positive-versus-negative loss separation tracks training progress more sensitively. When training is split into two stages, with the language model frozen while the audio embedding and head are learned and then unfrozen, speech-to-text accuracy rises on all three datasets (79.6, 72.4, and 69.3) and training becomes more stable. The paper reads these results as showing that S2SBench can diagnose degradation and that preserving pretrained text knowledge during audio adaptation is a workable remedy.
Load-bearing premise
The whole comparison rests on the assumption that each audio item and its text version are semantically equivalent, so the accuracy gap is caused by the input modality rather than by how the audio was spoken, recorded, or tokenized.
Editorial extensions
If this is right
- Any end-to-end speech LLM can be scored with S2SBench by comparing its perplexity-based accuracy on text versus audio versions of the same story-continuation and commonsense items; lower audio accuracy indicates intelligence degradation.
- Two-stage training, freezing the language model while the audio modules adapt, yields higher and more stable speech-to-text accuracy on all three datasets than single-stage training.
- The separation between positive and negative sample losses grows with training and is a more sensitive indicator of reasoning strength than accuracy alone.
- Speech input consistently underperforms text input on the same tasks, confirming that audio tokens, not task content, are what costs reasoning performance.
- Commonsense reasoning improves with continued training because pretrained text knowledge transfers, while Chinese discourse coherence remains harder and shows a larger speech-to-text gap.
Reading between the lines
- A natural extension is to use S2SBench as a training-time monitor, not just a final checkpoint evaluator; the loss-separation curves suggest that a plateau in positive-minus-negative loss could signal when to unfreeze the language model.
- Because the benchmark only measures speech-to-text comprehension, it says nothing about speech generation quality; a full speech-to-speech version would require a way to score the semantic content of generated audio, which the paper itself flags as an open problem.
- The two-stage benefit may depend on the order of audio modules (embedding first, head later); whether freezing all audio components simultaneously or unfreezing in a different order changes the result is a testable question the paper does not address.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces S2SBench, a benchmark intended to quantify 'intelligence degradation' in end-to-end speech LLMs by comparing reasoning accuracy under audio versus text input. The benchmark comprises three diagnostic datasets: English story continuation (sStoryCloze), Chinese story continuation (zh-sStoryCloze, a machine-translated version), and commonsense reasoning (sCMMLU, rewritten from CMMLU with GPT-4o). Evaluation uses a pairwise perplexity comparison: a sample is judged correct if the plausible ending or option receives lower perplexity than the implausible one. The authors apply the benchmark to Baichuan-Audio during training, compare single- and two-stage training strategies, and conclude that the two-stage strategy mitigates degradation. The datasets and evaluation code are released.
Significance. The benchmark addresses a real gap: there is no standard tool for measuring modality-induced reasoning loss in speech LLMs, and the pairwise PPL protocol is a simple, portable idea. The release of datasets and evaluation code is a concrete contribution. The two-stage training comparison is practically relevant for the speech-LLM community. However, the validity of the central claim depends on several unvalidated assumptions: that audio and text versions are semantically equivalent, that PPL is a valid proxy for reasoning, and that the reported single-run differences are statistically meaningful. If these are addressed, the benchmark could be widely adopted; in its current form, the main conclusion is not yet established.
major comments (5)
- [§3.1, §4.1] The paper asserts in §4.1 that 'the semantic content of both modalities is kept consistent,' but it never describes how the audio versions of zh-sStoryCloze and sCMMLU were generated, normalized, or quality-checked, and the only description in §3.1 is that each sample has both text and audio versions. Because the pairwise PPL protocol compares positive against negative items within each modality, any acoustic property that differs systematically between the positive and negative renderings (e.g., TTS prosody, duration, voice artifacts, or audio-tokenizer boundary effects) will directly shift the S→T accuracy and could reverse the reported two-stage-versus-single-stage comparison. The authors should specify the TTS engine, voice, sampling rate, and generation parameters, and provide validation evidence (e.g., ASR word-error rate, human naturalness/equivalence ratings, and duration/prosody statistics per condition).
- [§4.1, Table 1] Table 1 reports Baichuan-Audio's S→T accuracy but does not report the corresponding T→T accuracy for the same model at the same training stages; the only T→T numbers are for Qwen2.5, used as an upper bound. Since S2SBench is explicitly defined as degradation 'relative to text input,' the paired S→T-versus-T→T comparison on Baichuan-Audio is required to compute the degradation score. Without these paired numbers, the claim that two-stage training 'mitigates intelligence degradation' is not directly supported by the table.
- [§4.3, Appendix A] All results come from a single training run, so the reported differences (e.g., sStoryCloze 79.6 vs 77.5, zh-sStoryCloze 72.4 vs 70.1) carry no variance estimates. The appendix loss curves likewise show a single trajectory per condition. At least three seeds (or bootstrapping over items) are needed to establish that the two-stage advantage is not within run-to-run noise, especially because the benchmark is intended to guide training-strategy decisions.
- [§3.1] The construction of sCMMLU is not reproducible and its validity is not demonstrated: the paper states that GPT-4o 'rewrite[s] and filter[s]' CMMLU into 4,743 questions but gives no prompt, no filtering criteria, no examples, and no human or automatic verification that the four candidate statements are semantically equivalent in their first half and that only the correct option is plausible. If the generated statements differ in length, lexical frequency, or syntactic complexity across options, the pairwise PPL comparison could reflect surface statistics rather than commonsense reasoning. The authors should release the exact prompts, show sample items, and report agreement statistics from human or LLM annotators on whether the correct option is uniquely plausible.
- [§3.2, Figure 2] The evaluation protocol is underspecified: for the S→T setting, it is not stated whether the candidate ending/option is fed to the model as text tokens or as audio tokens, whether the perplexity is computed over the full candidate or only the last sentence, and whether perplexity is length-normalized. Audio-token boundary effects at the junction between the speech context and the candidate would affect S→T scores and could interact with the positive/negative contrast. A precise description of the input format and the exact probability used for scoring is necessary for reproducibility.
minor comments (6)
- [Table 1] The footnote mark '∗' for GLM-4-Voice is explained only as 'instruct model as no base model was provided'; please state explicitly that these numbers are not directly comparable to the base-model numbers, or move them to a separate column.
- [§3.1] 'Translation engine' is not identified; specify the translation service, version, and whether any manual post-editing was performed on zh-sStoryCloze.
- [§3.1] The sentence 'ultimately creating the sCMMLU dataset with 4,743 commonsense questions' should state how many original CMMLU questions were drawn from and how many were discarded in filtering.
- [Appendix A] The figures would benefit from consistent axis labels, units, and legends that distinguish Stage 1 and Stage 2; currently several panels are difficult to compare without zooming.
- [§1] The term 'intelligence degradation' is used throughout without a formal definition; please define it operationally (e.g., the gap in paired PPL accuracy between text and audio input) early in the paper.
- [Limitations] The Limitations section transparently notes the S→T-only scope; it would also be appropriate to list the single-run evaluation and the currently undocumented TTS pipeline as limitations, since they directly affect the strength of the conclusions.
Circularity Check
No significant circularity: S2SBench is externally anchored and the training-strategy conclusion is an empirical result, not a definitional consequence.
full rationale
The paper's central claim is that S2SBench can quantify intelligence degradation in speech LLMs and that two-stage training mitigates it. Neither claim reduces to the paper's inputs by construction. The benchmark datasets are adapted from external sources (sStoryCloze and CMMLU) with explicit transformations: zh-sStoryCloze is a translation of sStoryCloze, and sCMMLU is produced by rewriting CMMLU questions using GPT-4o. The pairwise perplexity protocol is a standard language-modeling evaluation applied symmetrically to text and audio inputs; it is not derived from Baichuan-Audio's parameters or training objective. The two-stage result is an empirical measurement reported in Table 1, comparing checkpoints under two training strategies; the outcome was not guaranteed by the benchmark's definitions and could in principle have favored single-stage training. The paper also reports external baselines (TWIST, Moshi, GLM-4-Voice, Qwen2.5), providing independent anchors that make the benchmark's rankings falsifiable rather than tautological. The limitations section explicitly acknowledges that the S→T evaluation does not capture speech generation, which further indicates that the authors are not equating the metric with the full phenomenon. The only self-referential element is that Baichuan-Audio is the authors' own model, but the paper does not cite a load-bearing theorem or prior result from the authors to justify its conclusions; the Baichuan-Audio reference is used to describe the model under evaluation, not as evidence for the benchmark's validity. The unvalidated audio-rendering pipeline is a validity threat, but it is not a circularity in the derivation chain. No fitted parameter is renamed as a prediction, and no equation equates the output with the input by construction.
Assumptions & free parameters
assumptions (4)
- domain assumption Perplexity difference between positive and negative candidate completions measures reasoning ability and commonsense knowledge.
- domain assumption Audio and text versions of each evaluation item are semantically equivalent, so performance differences are attributable to input modality.
- domain assumption GPT-4o rewriting of CMMLU preserves the original question semantics and the correctness of the designated answer.
- domain assumption Machine translation preserves story-continuation semantics in zh-sStoryCloze.
Cite this review
Pith. "Pith review of S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models." pith.science (2026). https://pith.science/paper/WJOAN6TV
@misc{pith2026250514438,
author = {Pith},
title = {Pith review of: S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/WJOAN6TV}},
note = {Machine review of arXiv:2505.14438}
}
read the original abstract
End-to-end speech large language models ((LLMs)) extend the capabilities of text-based models to directly process and generate audio tokens. However, this often leads to a decline in reasoning and generation performance compared to text input, a phenomenon referred to as intelligence degradation. To systematically evaluate this gap, we propose S2SBench, a benchmark designed to quantify performance degradation in Speech LLMs. It includes diagnostic datasets targeting sentence continuation and commonsense reasoning under audio input. We further introduce a pairwise evaluation protocol based on perplexity differences between plausible and implausible samples to measure degradation relative to text input. We apply S2SBench to analyze the training process of Baichuan-Audio, which further demonstrates the benchmark's effectiveness. All datasets and evaluation code are available at https://github.com/undobug/S2SBench.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.
Reference graph
Works this paper leans on
-
[3]
arXiv preprint arXiv:2410.00037
Moshi: a speech- text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037. Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng
-
[4]
arXiv preprint arXiv:2409.06666
Llama-omni: Seamless speech interaction with large language mod- els. arXiv preprint arXiv:2409.06666. Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, and 1 others
-
[6]
arXiv preprint arXiv:2306.09212
Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212. Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, and 1 others
-
[7]
arXiv preprint arXiv:2502.17239
Baichuan- audio: A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.17239. Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Gu- osheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, and 1 others
-
[8]
arXiv preprint arXiv:2410.08565, 3(7)
Baichuan-omni technical report. arXiv preprint arXiv:2410.08565, 3(7). Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan
-
[9]
arXiv preprint arXiv:2311.10122
Video-llava: Learn- ing united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Pop- uri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, and 1 others
-
[10]
Qwen2.5 technical report. Preprint, arXiv:2412.15115. Hankun Wang, Haoran Wang, Yiwei Guo, Zhihan Li, Chenpeng Du, Xie Chen, and Kai Yu. 2024a. Why do speech language models fail to generate semantically 5 coherent outputs? a modality evolving perspective. arXiv preprint arXiv:2412.17048. Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Lo...
-
[11]
arXiv preprint arXiv:2408.01800
Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800. Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yux- uan Wang, and Chao Zhang
Show all 13 references
-
[12]
arXiv preprint arXiv:2411.18138
Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation. arXiv preprint arXiv:2411.18138. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang
-
[13]
arXiv preprint arXiv:2412.02612
Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612. A Visualization of Intelligence Capability During Training To gain a deeper understanding of model behavior during training, we visualize the changes in intel- ligence c...
-
[2023]
arXiv preprint arXiv:2303.08774
Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and 1 oth- ers
-
[2024]
arXiv preprint arXiv:2407.10759
Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour
-
[2025]
arXiv preprint arXiv:2501.01957
Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957. Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexan- dre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, and 1 others
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.