REVIEW 3 major objections 5 minor 20 references
Qwen vs. Gemma Integration with Whisper: A Comparative Study in Multilingual SpeechLLM Systems
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims a Whisper-large-v3 encoder linked to a Gemma3-12B decoder through a lightweight projector reaches 16.63% average WER/CER on the MLC-SLM 2025 private test, beating the Qwen2.5-7B configuration by 1.97 points.
desk verdict The absolute result is solid, but the Gemma-vs-Qwen comparison is confounded and the 'significantly' claim needs controlled experiments before it can be taken seriously. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the three-stage training pipeline that connects a Whisper-large-v3 speech encoder, a two-layer SwiGLU perceptron projector, and an instruction-tuned decoder LLM adapted with LoRA. The projector performs temporal compression, reducing the 1,500 frames of a 30-second segment to 300 frames (5:1 compression) for Qwen2.5-7B or 375 frames (4:1 compression) for Gemma3-12B before the representation enters the LLM. Stage 1 trains only the encoder; stage 2 trains the encoder with the projector; stage 3 trains the projector together with the LoRA adapter while the rest of the LLM stays frozen. This staged alignment of speech features with the LLM's text space is the load-bearing mechanism that the paper claims makes the system work.
What would settle it
Rerun the experiment with the projector compression ratio, LoRA precision, and stage-2 encoder training held identical across Qwen2.5-7B and Gemma3-12B; if the 1.97-point gap shrinks or reverses, the paper's attribution of the improvement to the decoder choice is not supported.
Extended reading notes
Core claim
The paper reports that its Gemma3-12B configuration outperforms its Qwen2.5-7B configuration on the MLC-SLM Challenge 2025 development and private evaluation sets, with a private-test average WER/CER of 16.63% versus 18.60%. The authors attribute this 1.97-point absolute gap to the choice of language model, stating that the integration of Gemma3 helps to reduce the CER/WER significantly. The same experiments show that a cascaded setup in which the LLM corrects Whisper's output raises the error from 17.67% to 31.29% on the development set, while the end-to-end three-stage system keeps it lower, which the paper takes as evidence that joint optimization of the encoder, projector, and LLM is what makes the integration effective. In the challenge ranking, the Gemma3 system placed 15 out of 25.
Load-bearing premise
The comparison assumes the better result comes from choosing Gemma3, but the Qwen and Gemma systems differ in three other ways—how much the speech features are compressed, how precisely the model adapter is tuned, and whether the speech encoder is trained in the middle stage—so the improvement cannot be pinned to the language model alone.
Editorial extensions
If this is right
- A pretrained Whisper encoder can serve as a multilingual speech front-end for instruction-tuned LLMs, so a speech-LLM can be assembled from existing components rather than trained from scratch.
- The 1.97-point gap suggests that decoder choice can matter at the level of a few error-rate points within a fixed encoder-projector pipeline, at least on this dataset.
- End-to-end joint training beats a cascaded Whisper-plus-error-correction setup by a large margin, implying that post-hoc correction is not a substitute for aligned training.
- The private-test average of 16.63% places the system 15th among 25 submissions, showing the approach is competitive in a shared multilingual benchmark.
Reading between the lines
- The paper's own setup cannot isolate the decoder's contribution: the Qwen and Gemma runs differ in projector compression, LoRA precision, and whether the encoder is trained in stage 2, so the 'Gemma helps' claim is a hypothesis rather than a controlled result.
- A natural testable extension is to rerun both decoders with identical projector, precision, and stage schedule, and with multiple random seeds, to see whether the 1.97-point gap survives.
- Although the paper does not draw this conclusion, if the modular recipe generalizes, future systems could mix and match speech encoders and LLMs as drop-in parts, decoupling speech-model progress from LLM progress.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a modular multilingual SpeechLLM system submitted to the MLC-SLM Challenge 2025. It couples a fine-tuned Whisper-large-v3 encoder with a two-layer SwiGLU linear projector and a LoRA-tuned decoder LLM (either Qwen2.5-7B or Gemma3-12B), using a three-stage training schedule. On the challenge's private test set, it reports an average WER/CER of 16.63% with Gemma3-12B and 18.60% with Qwen2.5-7B, corresponding to rank 15/25, and it also reports per-language development-set results and ablations against Whisper-only, Qwen, Llama, and Phi-4 baselines. The main textual claim is that integrating Gemma3 significantly reduces WER/CER compared with Qwen2.5-7B.
Significance. The absolute result is a useful datapoint: a relatively simple, modular pipeline (fine-tuned encoder + linear projector + LoRA LLM) reaches competitive multilingual conversational ASR performance, and the paper includes per-language development-set tables, an error-correction ablation, and a public code repository. These are real strengths. The comparative significance of the Gemma3-vs-Qwen result is not yet established, because the two configurations differ in several design variables and no uncertainty estimates are given. As a system/competition report, the paper is worth publishing after the comparative claim is either tested in a controlled way or substantially weakened.
major comments (3)
- [§3.3.2, §3.3.3, §4.1] The claim that 'The integration of Gemma3 helps to reduce the CER/WER significantly' is not supported by the current experimental design. The Qwen2.5-7B and Gemma3-12B systems differ simultaneously in decoder family and size, projector compression ratio (Projector 5 vs Projector 4, §3.1.2), LoRA precision (16-bit vs 4-bit), and in whether stage 2 continues to train the speech encoder (explicitly stated only for Gemma3 in §3.3.3). Any of these factors, or their interaction, could account for the 1.97-point private-test gap. Please add controlled configurations that vary only the decoder (ideally also reporting matched projector and LoRA settings) or, if that is not feasible within the competition setting, revise the conclusion so that the improvement is attributed to the Gemma3-based configuration rather than to Gemma3 as a decoder family.
- [§4.1, Table 1] All results appear to come from a single run, with no confidence intervals, standard deviations, or repeated-seed experiments. The development-set gap between the two systems is only 0.63 points (21.31 vs 20.68), while the private-test gap is 1.97 points (18.60 vs 16.63); without variance information, the word 'significantly' in §4.1 is not justified statistically. Please report at least a small number of independent runs or an equivalent uncertainty estimate for the central comparison, and otherwise use a non-statistical phrasing such as 'in our experiments, the Gemma3-based configuration achieved a lower WER/CER.'
- [§4.1, Table 1] The 'Baseline' row (20.17) is used to compute relative improvements of 7.78% and 17.55%, but the manuscript never states what this baseline is (official challenge baseline, fine-tuned Whisper, or some other model). Without this definition, the relative gains in §4.1 cannot be interpreted. Please identify the baseline system and confirm that the comparison is on the same development/private test conditions.
minor comments (5)
- [§3.3] Please state explicitly, for each configuration, which components are frozen vs trainable in each of the three stages; the current text implies Qwen and Gemma differ in stage 2 but does not spell out the Qwen stage-2 setup.
- [Table 2] The columns 'Baseline LargeV3' and 'LargeV3-I' both appear to refer to fine-tuned Whisper-large-v3; rename them so the reader can tell organizer baselines from the authors' reproductions.
- [References] Reference [3] is cited for Llama3.1-8B but the reference points to Llama-Adapter v2; add the correct citation for Llama 3.1.
- [§4.2] The sentence about Gemma3-12B-4bit-III vs Baseline-Qwen says the model 'slightly underperforms Baseline-Qwen with a marginal increase of 0.29%,' but Table 2 shows Gemma3-12B-4bit-III at 20.68% vs Baseline-Qwen at 21.31%, i.e., Gemma is better; the stated numbers appear inconsistent.
- [Table 2] Table 2 contains implausible entries for Baseline-Qwen on Russian (608.22) and Korean (420.00); if these are formatting artifacts or special cases, please explain them or correct them.
Circularity Check
No circularity: the paper's claims are empirical evaluation results, not predictions derived from assumptions that contain the target.
full rationale
This is an empirical system paper. The central results are measured WER/CER values on the challenge development set and private evaluation set, reported in Table 1 and Table 2. There is no derivation chain in which an output is defined in terms of its own target, no fitted parameter that is later renamed as a prediction, and no 'uniqueness theorem' imported from the authors' prior work. The few citations are to external model and method papers (Whisper, Qwen, Gemma, FlashAttention-2, SpecAugment, meeteval), and none is load-bearing in a circular way: the cited models and toolkits provide components or evaluation metrics, not the conclusion that one decoder outperforms another. The concern that the Qwen/Gemma comparison changes multiple variables at once (decoder family and size, projector compression ratio, LoRA precision, and whether the encoder is trained in stage 2) is a legitimate threat to the causal attribution stated in Section 4.1, but it is a validity issue about confounding variables and statistical significance, not a circularity issue. No step in the paper reduces, by construction or by self-citation, to its own input, so the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Projector compression ratio =
5:1 (Qwen), 4:1 (Gemma)
- LoRA alpha =
32
- Learning rate =
3e-5
- Number of training stages =
3 (Qwen), 2 or 3 for Gemma variants
assumptions (3)
- domain assumption Whisper-large-v3 features are an adequate representation for multilingual speech in this challenge
- domain assumption WER/CER on the challenge's development and private test sets is a reliable measure of system performance
- ad hoc to paper The three-stage training procedure is beneficial
Cite this review
Pith. "Pith review of Qwen vs. Gemma Integration with Whisper: A Comparative Study in Multilingual SpeechLLM Systems." pith.science (2026). https://pith.science/paper/TR2X4UW7
@misc{pith2026250613596,
author = {Pith},
title = {Pith review of: Qwen vs. Gemma Integration with Whisper: A Comparative Study in Multilingual SpeechLLM Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/TR2X4UW7}},
note = {Machine review of arXiv:2506.13596}
}
read the original abstract
This paper presents our system for the MLC-SLM Challenge 2025, focusing on multilingual speech recognition and language modeling with large language models (LLMs). Our approach combines a fine-tuned Whisper-large-v3 encoder with efficient projector architectures and various decoder configurations. We employ a three-stage training methodology that progressively optimizes the encoder, projector, and LLM components. Our system achieves competitive performance with a private test average WER/CER result of 16.63% using the Gemma3-12B and 18.6% using the Qwen2.5-7B as decoder-only language model.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction In recent years, Large Language Models (LLMs) have emerged as transformative tools across a wide range of nat- ural language processing (NLP) applications, including ma- chine translation, question answering, summarization, and dialogue systems [1, 2, 3]. Their ability to model long- range dependencies and generate coherent, contextually rich...
-
[2]
System Architecture The architecture of our system is illustrated in Figure 1, in- cluding three main components. From the raw waveform O, aspeech encoderSE(·)is utilized to extract speech representations from the raw waveform ˜S=SE(O)∈ RTs×Ds, whereT s is the number of speech frames and Ds is the output dimension of the speech encoder. Subse- quently, th...
work page Pith review arXiv 2025
-
[3]
Experiment Setup 3.1. Models 3.1.1. Speech encoder We investigate the use of Whisper as a speech encoder, specifically the large-v3 version. Whisper is a Transformer- based encoder-decoder model, trained on 680k hours of la- belled speech of multiple languages. The large version has 1.5B parameters. 3.1.2. Projector The projector architecture is a two-lay...
-
[4]
Main results The main results are illustrated in Table 1
Experimental Results 4.1. Main results The main results are illustrated in Table 1. In relative, our proposed systems outperform the baseline by 7.78% and 17.55% for Whisper+Qwen2.5-7B and Whisper+Gemma3- 12B respectively. The integration of Gemma3 helps to re- duce the CER/WER significantly, with an absolute reduc- tion of 1.97% compared to using Qwen2.5...
-
[5]
Conclusions We present a system for the MLC-SLM Challenge 2025 that effectively combines state-of-the-art speech encoder and language modeling components. Our three-stage train- ing methodology and modular architecture enable flexible experimentation and optimization of different system con- figurations. The experimental results demonstrate the effective-...
work page 2025
-
[6]
Acknowledgements The authors would like to thank the MLC-SLM Challenge 2025 organizers for providing the dataset and evaluation framework. We also acknowledge the computational re- sources provided by the Institute for Infocomm Research (I2R), A∗STAR, Singapore, which made this research pos- sible
work page 2025
-
[7]
J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inInternational conference on ma- chine learning. PMLR, 2023, pp. 19 730–19 742
work page 2023
-
[8]
Visual instruction tuning,
H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023
2023
Show all 20 references
-
[9]
Llama-adapter v2: Parameter-efficient visual instruction model,
P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yueet al., “Llama-adapter v2: Parameter-efficient visual instruction model,”arXiv preprint arXiv:2304.15010, 2023
2023 arXiv
-
[10]
An embarrassingly sim- ple approach for llm with strong asr capacity,
Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhanget al., “An embarrassingly sim- ple approach for llm with strong asr capacity,”arXiv preprint arXiv:2402.08846, 2024
2024 arXiv
-
[11]
Audiopalm: A large lan- guage model that can speak and listen,
P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonovet al., “Audiopalm: A large lan- guage model that can speak and listen,”arXiv preprint arXiv:2306.12925, 2023
2023 arXiv
-
[12]
Meralion-audiollm: Technical report,
Y . He, Z. Liu, S. Sun, B. Wang, W. Zhang, X. Zou, N. F. Chen, and A. T. Aw, “Meralion-audiollm: Technical report,” arXiv preprint arXiv:2412.09818, 2024
2024 arXiv
-
[13]
Robust speech recognition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518
2023
-
[14]
Qwen2 technical report,
Q. Team, “Qwen2 technical report,”arXiv preprint arXiv:2412.15115, 2024
2024 arXiv
-
[15]
Gemma 3 technical report,
G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieil- lard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram ´e, M. Rivi`ereet al., “Gemma 3 technical report,”arXiv preprint arXiv:2503.19786, 2025
2025 arXiv
-
[16]
Glu variants improve transformer,
N. Shazeer, “Glu variants improve transformer,”arXiv preprint arXiv:2002.05202, 2020
2002 arXiv
-
[17]
Flashattention-2: Faster attention with bet- ter parallelism and work partitioning,
T. Dao, “Flashattention-2: Faster attention with bet- ter parallelism and work partitioning,”arXiv preprint arXiv:2307.08691, 2023
2023 arXiv
-
[18]
Decoupled weight decay regu- larization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regu- larization,”arXiv preprint arXiv:1711.05101, 2017
2017 arXiv
-
[19]
Specaugment: A simple data aug- mentation method for automatic speech recognition,
D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data aug- mentation method for automatic speech recognition,” inIn- terspeech 2019, 2019, pp. 2613–2617
2019
-
[20]
Phi-4 technical report,
M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmannet al., “Phi-4 technical report,”arXiv preprint arXiv:2412.08905, 2024
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.