{"id":"5065b4fb-13b7-46c6-bdde-3e2c21d170f6","arxiv_id":"2505.11352","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using CTC posterior scores from a speech encoder as soft weights over a language model's token embeddings lets a frozen encoder and a fine-tuned LLM work together, with a 49% average word error reduction on MLS.","lead":"This paper connects a speech recognition model to a large language model by feeding it word-score probabilities instead of raw audio. The approach reportedly cuts word errors by about half on eight languages and lets different speech encoders be swapped without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot modularity claim is supported only by swapping USM-CTC checkpoints that share the same base encoder; switching to a genuinely different speech encoder is untested, so the central modularity claim is narrower than stated.","rationale":"The reader's weakest assumption (codebook sufficiency) is reasonable, but it is partially tested: Table 5 uses a randomly initialized 16K-codebook and matches matched-vocab performance, and Table 12 shows freezing the LLM embedding table costs only 0.2 WER, so Eq. (4) does not critically depend on the semantic structure of E. The less tested part of the central claim is the breadth of zero-shot modularity, which is specifically advertised as an advantage. The paper's internal logic is coherent, and the headline ASR/AST numbers are consistent with the tables. My concern is external validity: all swaps are within the USM family, so the strongest 'lego' property is asserted, not demonstrated. This does not overturn the conditional verdict; it sharpens the condition. The reader's conditional verdict already flags missing artifacts and statistical detail, so no verdict change. I would keep the verdict conditional, with the condition expanded to include a non-USM zero-shot transfer test and a stated temperature-selection protocol.","tokens_in":16968,"tokens_out":8047,"duration_ms":87778,"concrete_test":"Retrain a CTC head on a non-USM speech encoder, e.g., Whisper or wav2vec2, using the same 256K Gemma vocabulary and the same mls-en training data; feed its posteriors through Eq. (4) into the already fine-tuned Gemma decoder from the mls-en setup without any retraining, and evaluate WER on the MLS_en test set with τ chosen on a held-out dev split. If the WER is close to that encoder's CTC baseline (or worse), Section 3.4's modularity claim fails for genuinely different encoders.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 promises that the LLM can accept outputs from 'a different speech encoder as long as the model also operates on the same CTC vocabulary,' and the abstract claims the speech encoder 'can be switched' zero-shot. Every zero-shot experiment (Tables 3 and 7, plus Appendix C) swaps USM-CTC models that share the same USM architecture and pre-trained base, differing only in the ASR data used for CTC fine-tuning. Shared base means frame-level features, blank posterior statistics, and error patterns are likely much more similar than across arbitrary encoder families. The demonstrated transfer may therefore be an artifact of sibling checkpoints rather than evidence for the CTC-posterior interface itself. A second, compounding issue is that Table 7 reports 'the best BLEU score achieved with the optimal temperature value' without stating whether the temperature was selected on a held-out set; if selected on the test set, the zero-shot AST gains are optimistic. The central claim about modularity is thus broader than the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LegoSLM, a method to connect a pre-trained CTC-based speech encoder with a decoder-only LLM. Instead of feeding continuous encoder representations or ASR hypotheses, the encoder is fine-tuned to produce CTC posteriors over the LLM's vocabulary; the posteriors are then linearly combined with the LLM's input embedding table (Eq. 4) to form pseudo-speech embeddings, which are concatenated with text embeddings for the LLM input. The speech encoder is frozen during LLM fine-tuning. Experiments on ASR (MLS, LibriSpeech, SpeechStew) and AST (CoVoST 2) show that the method outperforms ASR error correction baselines and is competitive with or better than simple speech-prompt baselines, with a reported 49% average WERR over the USM-CTC baseline on 8 MLS languages. The paper also introduces a temperature parameter to control the acoustic/language influence at decode time, and demonstrates zero-shot combination of LLMs with USM-CTC encoders trained on different data.","tokens_in":17164,"tokens_out":8196,"duration_ms":72149,"significance":"If the claims hold, LegoSLM is a simple and modular interface between speech encoders and LLMs, with potential advantages in privacy (only posteriors exchanged), cost (no beam search or N-best generation), and flexibility (encoder can be swapped). The paper provides extensive experiments across multiple languages and datasets, and the headline results are internally consistent; for example, the 49% WERR matches the average of the per-language WERRs derivable from Table 2. The proposed method and ablations are clearly described, and the comparison to AEC and speech prompts is useful. However, the modularity claim is only supported for sibling USM-CTC checkpoints, and the selection protocol for key hyperparameters (blk_downscale and temperature) is not reported, which tempers the strength of the conclusions.","major_comments":[{"comment":"The zero-shot modularity claim is broader than the evidence. Every zero-shot experiment swaps USM-CTC models that share the same USM architecture and pre-trained base, differing only in the supervised ASR data (mls-en, multi, public, lbs). The claim in Section 3.4 that the LLM can accept outputs from a different speech encoder as long as the model also operates on the same CTC vocabulary is not tested with a genuinely different encoder family. Shared frame-level features, blank-posterior statistics, and error patterns across sibling checkpoints may make transfer much easier than across arbitrary encoders. Please either add an experiment with a different encoder (e.g., a different architecture such as Whisper, or a CTC model initialized from a different pretrained checkpoint) or explicitly limit the claim to swapping USM-CTC checkpoints fine-tuned on different data.","section":"Section 3.4 and Tables 3, 5, 7 (also Appendix C)"},{"comment":"The selection protocol for blk_downscale and temperature is not reported. The headline 49% WERR (Table 2) uses LegoSLM* with blk_downscale=1e4 in Eq. (5), and Table 7 reports the best BLEU score achieved with the optimal temperature value without stating whether the temperature was chosen on a held-out set. If these values were selected using the test set, the reported gains are optimistic. Please state whether blk_downscale=1e4 and the optimal tau values were chosen on a held-out development set, and if so, report the held-out selection details. If not, report results at a fixed default (e.g., blk_downscale=1 and tau=1) alongside the tuned results.","section":"Sections 3.3, 5.1, 5.3 and Tables 2, 7"}],"minor_comments":[{"comment":"The first paragraph says 'Table 10 presents the detailed WER results' but the correct table reference is Table 15; additionally, in the last row of Table 15, '5,7' should be '5.7'.","section":"Appendix C"},{"comment":"The notation it = argmax_k(zt) is nonstandard; it should refer to the indices of the top-K elements. Consider defining it = top-k(zt) for clarity.","section":"Equation (7)"},{"comment":"The comparison between LegoSLM* and SP (P+G) relies on WER differences as small as 0.1-0.3 (e.g., Table 2, en: 5.7 vs 5.5; es: 5.8 vs 5.4). Without error bars or significance tests, the relative ordering of these systems should be presented more cautiously.","section":"Tables 1 and 2"},{"comment":"It would be helpful to clarify whether the reconstructed speech embeddings for blank frames are used as the randomly initialized vector during training or whether those frames are masked, since the random embedding may inject noise early in adaptation.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The core ASR/AST results are sound and internally consistent, but the modularity claim is overstated relative to the evidence: all zero-shot experiments swap sibling USM-CTC checkpoints rather than genuinely different encoders. The hyperparameter selection issue for blk_downscale and temperature also needs clarification. With a scoped claim or an additional cross-encoder experiment, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ma et al. propose LegoSLM: fine-tune a speech encoder (USM) with CTC over the LLM's vocabulary, then reconstruct the LLM's input embeddings as a weighted sum of the embedding table using the CTC posteriors. That's a genuinely new and elegant way to connect a speech encoder to an LLM, and it works. On MLS multilingual ASR they get 49% average WERR over USM-CTC (Table 2: 17.8 to 9.1, internally consistent), matching a tuned speech-prompt baseline on average and beating AEC. On CoVoST2 AST, LegoSLM* outperforms all baselines including AEC n=10. The ablations (blank downscaling, temperature control, top-K variants, vocab mismatch) are sensible and clearly reported. The paper is honest about training details and includes a thorough appendix.\n\nThe main soft spot is the scope of the zero-shot modularity claim. Every swapped encoder in Tables 3 and 7 is a USM-CTC checkpoint: same architecture, same pre-trained base, differing only in ASR fine-tuning data. The abstract says 'the speech encoder can be switched' which sounds general, and Section 3.4 promises any encoder operating on the same CTC vocabulary. That is not demonstrated. The result is still useful—it shows the LLM can be reused when its own encoder is updated or adapted, which is a real practical scenario—but the claim should be narrowed.\n\nSecond, Table 7 reports 'the best BLEU score achieved with the optimal temperature value' without saying whether the temperature was selected on a held-out set. Appendix C shows test-set curves; if the temperature is chosen on the test set, the zero-shot AST comparison is optimistic. This is a moderate concern, not a fatal one: the main ASR results use default temperature, and the temperature trends are otherwise clear.\n\nMinor: no error bars or significance tests, and no code release. Also, the speech-prompt baseline is a simple projection layer; the comparison is fair enough given they also report U+P+G.\n\nAll that said, the central idea is sound and the empirical execution is above average. The paper deserves serious peer review; I'd ask the authors to (a) re-scope the zero-shot claim or test a genuinely different encoder family, and (b) specify how temperature was chosen. For a reading group, it's worth a slot as a good example of a modular speech-LLM interface.","headline":"A clean new interface for speech-LLM integration with solid experiments; the zero-shot swapping claim overreaches the evidence.","tokens_in":17701,"tokens_out":3312,"would_cite":true,"duration_ms":31596,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen speech encoder can feed an LLM through CTC posteriors and be swapped zero-shot.","keywords":["CTC posteriors","speech-LLM interface","automatic speech recognition","speech translation","zero-shot encoder swap","embedding-table reconstruction","modular ASR","softmax temperature control"],"falsifier":"A decisive test: train the same LegoSLM pipeline but replace the pretrained LLM embedding table $E$ with a randomly initialized table, keeping the rest of the LLM pretrained and training the system as in the paper. If WER stays near 5.6% on MLS_en, the linguistic structure of the embeddings is not what carries the signal; if WER degrades sharply, the codebook hypothesis is supported.","tokens_in":1850,"feed_emoji":"🎤","tokens_out":1757,"duration_ms":58598,"temperature":0.7,"pith_summary":"The paper proposes LegoSLM, a way to connect a pre-trained speech encoder to a large language model without training a special mapping network or decoding into text first. The speech encoder is fine-tuned with CTC loss over the LLM's own token vocabulary, and each frame's posterior distribution is used as weights to combine the LLM's input embeddings into a pseudo-audio embedding. These reconstructed embeddings are concatenated with text embeddings and fed to the LLM, which is fine-tuned to transcribe or translate. On multilingual MLS ASR the method reports an average 49% word-error-rate reduction over the USM-CTC baseline, matches a speech-prompt baseline on ASR while keeping the encoder frozen, and beats AEC and speech-prompt baselines on speech translation. The deeper claim is modularity: because only CTC posteriors cross the interface, a differently-trained speech encoder can be plugged into the already-fine-tuned LLM with no retraining.","feed_headline":"CTC posteriors let speech encoders plug into LLMs","feed_subtitle":"Weighted text embeddings carry audio to an LLM, giving 49% WERR and zero-shot encoder swaps.","key_machinery":"The load-bearing object is Eq. (4): $s_t = E \\cdot o_t$, where $E$ is the LLM's input embedding table and $o_t$ is the CTC posterior vector over the LLM vocabulary (including a special blank mapped to a newly initialized embedding). This turns the LLM's text embeddings into a codebook for speech: the encoder never sends continuous features, only a distribution over text tokens, and the LLM's own embeddings reconstruct an audio-like input. The CTC blank token, the optional top-K variants, and the temperature $\\tau$ in the softmax all modify this one reconstruction step.","core_discovery":"The central claim is that a CTC posterior matrix over the LLM's word-piece vocabulary is a sufficient interface between speech and language. Formally, with embedding table $E$ and per-frame posteriors $o_t$, the speech representation is $s_t = E \\cdot o_t$; this weighted sum is then concatenated with text embeddings for the LLM. The paper shows this representation carries enough information to support both ASR and speech translation, and that because the interface is a probability distribution over tokens rather than continuous encoder features, a different encoder trained on the same CTC vocabulary can be substituted after LLM fine-tuning with no weight updates. The paper also reports that downweighting the CTC blank token (LegoSLM*) improves ASR, and that a softmax temperature on the CTC posteriors acts as an acoustic-model/language-model weight for zero-shot combinations.","pith_inferences":["If the reconstruction step is as information-preserving as the experiments suggest, the same interface could be used for tasks beyond ASR and translation, such as speech summarization or spoken language understanding, without retraining the encoder.","The success of the top-K softmax variant suggests the effective interface is sparse; a testable extension is whether keeping far fewer than 100 tokens per frame, or selecting them dynamically, can cut compute while preserving WER.","Because the blank token dominates CTC posteriors and carries little content, its downweighting is a hidden hyperparameter; other encoders with different alignment statistics may need different blank weights, and the temperature control partially automates this.","The vocabulary-mismatch experiment (16K vs 256K tokens with an extra learned embedding table) implies the approach does not require shared tokenizers, so it could connect encoders and LLMs in different languages or with different subword units through a small learned map."],"forward_implications":["On MLS-en ASR, LegoSLM* reaches 5.6% WER with a frozen USM encoder, close to the 5.5% of the speech-prompt system and well below the 7.8% of 10-best AEC.","On multilingual MLS ASR across 8 languages, LegoSLM* gives an average 49% WERR over the USM-CTC baseline, versus 50% for speech prompts and 22% for AEC.","In zero-shot encoder swapping, a Gemma fine-tuned on MLS-en USM outputs transfers to public, multi, and LibriSpeech encoders with 32-37% WERR, while speech-prompt systems fail with WER above 165.","On CoVoST 2 speech translation, LegoSLM* achieves the best BLEU in fr->en, de->en, and en->de, and keeps competitive BLEU when the encoder is swapped.","A softmax temperature tau around 0.6-1.1 improves zero-shot combinations, with lower tau favoring a stronger encoder and higher tau giving the LLM more freedom."],"supporting_citations":[{"why":"Supplies the USM speech encoder that is fine-tuned with CTC loss and later frozen in LegoSLM.","marker":"Zhang et al., 2023"},{"why":"Supplies the Gemma 2B LLM whose embedding table is used as the codebook for reconstructing speech embeddings.","marker":"Team et al., 2024"},{"why":"Defines CTC loss, posteriors, and the blank token behavior that LegoSLM relies on and modifies.","marker":"Graves et al., 2006"},{"why":"Provides the speech-prompt baseline using a linear projection layer, which the paper compares against.","marker":"Fathullah et al., 2024"},{"why":"Provides the N-best T5 ASR error correction approach and the N-best input formulation used as an AEC baseline.","marker":"Ma et al., 2023a"},{"why":"Supplies the MLS dataset used for all multilingual ASR training and evaluation.","marker":"Pratap et al., 2020"},{"why":"Supplies the CoVoST 2 dataset used for the speech translation experiments.","marker":"Wang et al., 2021"},{"why":"Describes the BEST-RQ self-supervised pretraining used to build the USM encoder before CTC fine-tuning.","marker":"Chiu et al., 2022"}],"fun_headline_variants":["CTC posteriors bridge speech encoders and LLMs","49% WERR: CTC posteriors link speech to LLMs","Zero-shot encoder swap via CTC posterior interface","Speech-to-LLM via weighted word embeddings","CTC posteriors enable modular speech-LLM connection"],"cache_read_input_tokens":19840,"weakest_assumption_plain":"The load-bearing premise is that a probability distribution over the LLM's text tokens, averaged through the token embedding table, is a rich enough representation of speech for the LLM to recover the utterance; the paper validates this by experiment, not by analysis.","fun_headline_variants_meta":{"raw":{"variants":["CTC posteriors bridge speech encoders and LLMs","49% WERR: CTC posteriors link speech to LLMs","Zero-shot encoder swap via CTC posterior interface","Speech-to-LLM via weighted word embeddings","CTC posteriors enable modular speech-LLM connection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1553,"prompt_tokens":998,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":614,"tokens_out":555,"duration_ms":4903,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:28.444993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test: train the same LegoSLM pipeline but replace the pretrained LLM embedding table $E$ with a randomly initialized table, keeping the rest of the LLM pretrained and training the system as in the paper. If WER stays near 5.6% on MLS_en, the linguistic structure of the embeddings is not what carries the signal; if WER degrades sharply, the codebook hypothesis is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CTC loss, posteriors, and the blank token behavior that LegoSLM relies on and modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the speech-prompt baseline using a linear projection layer, which the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MLS dataset used for all multilingual ASR training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CoVoST 2 dataset used for the speech translation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the BEST-RQ self-supervised pretraining used to build the USM encoder before CTC fine-tuning."}],"review_version":1}