{"id":"216fa923-9f12-4644-864e-b006356a8935","arxiv_id":"2412.16464","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A streaming ASR architecture that adds a frozen LLM as the internal language model of a factorized transducer, with vocabulary adaptation and weak-to-strong LM swapping, reports large WER reductions.","lead":"This paper builds a streaming speech recognizer by plugging a frozen LLM into a factorized transducer and training it with a weak-to-strong language model swap plus minimum word error rate tuning. The method reports 17% fewer errors than a strong FT baseline and 32% fewer than an RNN-T baseline on LibriSpeech and multilingual LibriSpeech.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 17% WERR result uses Llama3, but the weak-to-strong swap ablations in Table II only cover Llama2/FFT Llama2; no Llama3 ablation isolates the swap from the 8B model's raw LM strength and MWER.","rationale":"The reader's weakest assumption is that the weak-to-strong swap is justified only by the Table II ablations and lacks diagnostic support. I agree that the training-dynamics assumption is the load-bearing point, but I identify a more specific gap: Table II does not include Llama3, even though the headline result uses Llama3. The paper provides indirect evidence (Table III's no-swap/no-MWER Llama3 rows vs Table I's final Llama3 row) that the full pipeline helps, but this cannot isolate the swap from the effect of MWER training. Given that the swap is the paper's novel contribution and that the 8B Llama3 differs substantially from the ablated Llama2 models, a Llama3-specific swap ablation is necessary to attribute the 17% WERR to the proposed method rather than to the LM's raw capacity or to MWER. This does not overturn the reader's CONDITIONAL verdict; it sharpens the condition that should be met before the central claim is fully trusted. The concern is concrete and testable, and it does not rely on any speculative assumption about author conduct or external consensus.","tokens_in":9066,"tokens_out":12758,"duration_ms":104515,"concrete_test":"Run the Table II ablation grid with Llama3 as the test-time non-blank predictor: (a) train stateless, test Llama3, no MWER; (b) train stateless, test Llama3, MWER; (c) train Llama3, test Llama3, no MWER; (d) train Llama3, test Llama3, MWER, all on LibriSpeech with the same 5k vocabulary and hyperparameters. If (b) beats (d) by a margin comparable to the Llama2 rows (roughly 5-9% relative), the swap is validated for the headline model. If (b) is approximately equal to (d) or worse, the 17% claim should be attributed to LLM strength and/or MWER training, not to the weak-to-strong swap itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is the 17% WERR over the FT baseline (test-clean 2.47 vs 2.97) achieved by Transducer-Llama with Llama3 (Table I). The paper's key novel component is the weak-to-strong LM swap (Section III-B). However, the ablation study in Table II that validates this swap uses Llama2-0.5B and FFT Llama2 as the strong predictor; no row uses Llama3-8B. Table III reports Llama3 results but explicitly without the swap and without MWER training (3.02/7.44 with the Llama3 tokenizer, 2.76/7.36 with the ASR vocabulary), so the 10.5% test-clean gain from 2.76 to 2.47 could be attributed to MWER alone rather than to the swap. Thus the reader's weakest assumption (an encoder trained with a weak predictor remains compatible with a much stronger LLM) is not directly tested for the exact model configuration behind the headline number. Since 8B Llama3 is much larger than 0.5B Llama2, the distribution mismatch between the stateless bigram predictor and Llama3 may be qualitatively different; the swap's success on Llama2 does not guarantee it holds for Llama3.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Transducer-Llama, a factorized transducer (FT) architecture in which a frozen LLM serves as the non-blank predictor, together with a vocabulary adaptation scheme, a weak-to-strong LM swap, and MWER training. On LibriSpeech, the system with Llama3-8B reports 2.47% WER on test-clean and 6.53% on test-other, which the paper translates into a 17% relative WER reduction over the strongest FT baseline and 32% over an RNN-T baseline. The paper also reports multilingual LibriSpeech (MLS) results and an offline comparison against a speech LLM. The central claim is that frozen LLMs can be integrated into a streaming transducer while improving accuracy and preserving streaming capability.","tokens_in":9350,"tokens_out":7155,"duration_ms":62034,"significance":"If the reported results hold, the paper offers a useful recipe for inserting frozen LLMs into an already-streamable transducer without retraining the encoder with the LLM, which is a practical contribution to streaming ASR. The strengths of the paper are its consistent held-out evaluation on standard LibriSpeech/MLS test sets, the ablation table (Table II) that isolates the weak-to-strong swap and MWER for Llama2-based predictors, and the vocabulary adaptation result showing a large training-speed advantage (Table III). The main empirical claim is large in magnitude (17% WERR), and the architecture is plausible. However, the key mechanism is not directly ablated for the exact model configuration behind the headline number (Llama3-8B), and the paper provides no latency or uncertainty analysis, so the headline result is not yet fully pinned down.","major_comments":[{"comment":"The weak-to-strong LM swap is not ablated for Llama3-8B, the model used for the headline 17% WERR. Table II covers only LSTM, stateless, Llama2-0.5B, and FFT Llama2 as test-time predictors; Table III reports Llama3 without the swap and without MWER (2.76/7.36), while Table I reports Llama3 with both the swap and MWER (2.47/6.53). There is therefore no Llama3 row that isolates the swap from MWER, and the 10.5% test-clean gain from 2.76 to 2.47 could be wholly attributable to MWER. Since Llama3 is 16x larger than Llama2 and uses a 128k tokenizer, the assumption that an encoder trained with a bigram predictor remains compatible with Llama3 is nontrivial. Please add a Llama3 no-swap row (Llama3 used at both training and test, with and without MWER) or explicitly restrict the swap claim to the Llama2 results.","section":"Section IV-C2 / Table II and Table III"},{"comment":"The WER comparisons are based on a single decoding pass with no confidence intervals or significance tests. Several adjacent rows in Table I differ by only 0.07 WER (e.g., FFT Llama2 vs Llama3 on test-clean), and the text's conclusion that 'as more powerful LMs are used, the WER continues to decrease' depends on these small differences. Repeated runs or a test-set significance test would support this monotonicity claim; at minimum, the absence of such statistics should be acknowledged in the experimental section.","section":"Section IV-C1 / Table I"},{"comment":"The paper's title and abstract emphasize streaming, but no latency, real-time factor, or memory measurements are reported for Transducer-Llama. The streaming property is architectural (the encoder uses Emformer/Conformer with chunk masks), but with an 8B-parameter LLM as the autoregressive predictor, practical streaming feasibility is not self-evident. Please report an RTF or latency figure for the streaming models, or explicitly state that streaming capability is inherited from the FT architecture and was not evaluated empirically.","section":"Section IV-B / IV-C"}],"minor_comments":[{"comment":"The symbols '%' and '!' in the MWER column are not defined in the caption; please state explicitly that '%' denotes RNN-T loss only and '!' denotes MWER training.","section":"Table II"},{"comment":"'hyper-perameters' should be 'hyper-parameters'.","section":"Eq. (1)"},{"comment":"'5.3m total batch size' should be '5.3M' for consistency with the model-size notation, and '0.2k hrs' / '1.1k hrs' are awkward units; consider writing the durations in hours or thousands of hours consistently.","section":"Section IV-A"},{"comment":"'Tokenize' should be 'Tokenizer', and the meaning of the 'Train Speed' value (8.05) is not defined; state what it is relative to and in what units.","section":"Table III"},{"comment":"The comparison with the offline Speech LLM [14] in Table IV uses a different encoder and offline decoding; the text acknowledges this, but a sentence clarifying that the comparison is indicative rather than a controlled ablation would help.","section":"Section IV-C4"},{"comment":"No code, model checkpoints, or data splits are released, which limits reproducibility; a reproducibility statement would be welcome.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a practical engineering contribution, and honestly a good one. The idea is to take the Factorized Transducer from [8], freeze a Llama model as the non-blank predictor, shrink its vocabulary to the ASR token set via embedding/output-layer initialization, train the encoder with a stateless predictor, then swap in the LLM and run MWER. The reported 17% WERR over a strong FT baseline and 32% over RNN-T on LibriSpeech are consistent with the ablations. The vocabulary adaptation table (Table III) is a nice result on its own: 8x training speedup and slightly better WER than the native 128k Llama tokenizer.\n\nWhat's genuinely new is the combination: vocabulary transfer for LLM-in-transducer, the weak-to-strong swap, and MWER after the swap. Each piece has precursors, but the package is new and the ablations mostly support the claims. The multilingual MLS results also hold up, and the offline comparison against a published speech LLM is a useful sanity check.\n\nThe soft spots are real but not disqualifying. First, no code, no data, no significance tests. The tables are clean, but these are single runs, and ASR numbers on LibriSpeech can move by a couple tenths with seed changes. Second, alpha and beta are copied from [8] with no sensitivity analysis. Minor, since they follow the prior work. Third, the weak-to-strong swap is justified heuristically. The paper says it prevents the encoder from relying on the strong LM, but there is no diagnostic—no analysis of encoder representations or training dynamics. That's a gap, but the ablation rows do show the swap helps across several predictor pairs, so the heuristic has empirical support.\n\nThe stress-test concern about Table II is worth flagging: the swap ablations use Llama2 and FFT Llama2, not Llama3. For Llama3, the improvement from 2.76 (no swap, no MWER) to 2.47 (swap + MWER) is not broken down, so strictly speaking the headline number could be driven by MWER alone rather than the swap. That doesn't sink the paper—the full recipe works—but it does mean the paper overstates slightly when it credits the swap for the Llama3 result. A careful referee should ask for one extra row: Llama3 with swap but no MWER, or with MWER but no swap.\n\nWho is this for? Engineers building streaming ASR systems, and researchers working on internal LM fusion. It's a serious empirical paper, not a theoretical one. I'd send it to peer review. The missing Llama3 ablation is fixable in revision.\n\nRecommendation: engage with it. If the authors add the missing ablation and release code, this becomes a reference point for frozen-LLM streaming ASR.","headline":"Solid engineering recipe for streaming ASR with a frozen LLM, but the headline Llama3 result lacks an ablation isolating the weak-to-strong swap from MWER.","tokens_in":9905,"tokens_out":2217,"would_cite":true,"duration_ms":18493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A streaming speech recognizer can carry a frozen Llama inside it, cutting word error by 17 percent relative to a strong factorized-transducer baseline.","keywords":["streaming speech recognition","factorized transducer","large language models","weak-to-strong LM swap","minimum word error rate training","vocabulary adaptation","LibriSpeech","RNN-T"],"falsifier":"Retrain the full pipeline but swap in a randomly initialized Llama3-sized transformer with fixed weights as the non-blank predictor before MWER; if WER stays near the Llama3 result, the gain does not come from the LLM's pre-trained text knowledge, and if WER collapses, the language-model prior is essential to the reported improvement.","tokens_in":8854,"feed_emoji":"🎙️","tokens_out":5834,"duration_ms":48463,"temperature":0.7,"pith_summary":"Transducer-Llama claims that a frozen large language model can act as the internal language model of a streaming factorized transducer, improving speech recognition accuracy without sacrificing online operation. The recipe has three parts: shrink the LLM tokenizer to a 5,000-token ASR vocabulary, train the encoder with a weak bigram-like predictor, then swap in a frozen Llama and fine-tune with a word-error-rate loss. On LibriSpeech this yields 2.47% WER on test-clean and 6.53% on test-other, a 17% relative reduction over the factorized-transducer baseline and 32% over an RNN-T baseline. On the multilingual LibriSpeech corpus the same strategy reduces average WER from 10.73% to 9.24%. The point is that streaming ASR and large text LMs are not incompatible.","feed_headline":"Streaming ASR gains 17% by swapping in a frozen Llama","feed_subtitle":"Train with a weak predictor, then swap in Llama3 and fine-tune on word error: online speech recognition improves with no extra latency.","key_machinery":"The load-bearing mechanism is the weak-to-strong LM swap, combined with vocabulary adaptation and ILM-aware MWER training. A small stateless non-blank predictor (effectively a bigram token predictor) is used during RNN-T loss training; once the encoder and blank predictor have converged, the small predictor is replaced by a frozen LLM whose probabilities enter the factorized-transducer decoding equation. The vocabulary adaptation step initializes new embedding and output layers for a compact 5,000-token ASR vocabulary from the LLM's original 128,000-token vocabulary, allowing the LLM's transformer layers to stay fixed. MWER training then optimizes the integration of the LLM with the word-level edit distance, which is what makes the swap effective.","core_discovery":"The paper establishes that a text-only LLM, frozen during ASR training, can serve as the non-blank predictor of a streaming factorized transducer and materially improve accuracy. The critical finding is that simply training the transducer with a strong LLM predictor under the RNN-T loss gives only small gains, whereas training first with a weak predictor and then swapping in the LLM for decoding, followed by minimum word error rate (MWER) fine-tuning, gives large gains. With Llama3 as the swapped-in predictor, the model reaches 2.47% WER on LibriSpeech test-clean and 6.53% on test-other, beating both the strong factorized-transducer baseline and a fully fine-tuned smaller Llama2. The authors interpret this as the encoder being forced to rely on acoustic information during training, so the strong LM is used as a genuine complement rather than a crutch.","pith_inferences":["If the swap is general, the same recipe should work with other frozen text LMs and other streaming encoders, since the central assumption is only that the encoder learns acoustic representations that are largely independent of the predictor used during RNN-T training.","The paper's comparison to offline speech-LLM baselines suggests that a streaming Transducer-Llama may already be competitive with non-streaming LLM-prompted ASR; a latency-controlled head-to-head benchmark would make that trade-off explicit.","A stress test on acoustically noisy or out-of-domain audio, where LM priors can mislead, would clarify whether MWER's gains come from better acoustic encoding or from the language-model prior injected at decoding.","The weak-to-strong recipe resembles classical sequence-discriminative training with an external LM, so the same strategy may transfer to other end-to-end architectures that factor out an internal LM component."],"forward_implications":["Streaming deployment can use the full predictive power of an 8B-parameter frozen LLM while keeping the 160 ms segment-based online decoding used by the Emformer encoder.","Because the LLM transformer stack stays frozen, a trained Transducer-Llama can be reused for other downstream tasks without retraining the speech model, and the ASR-specific parameters remain compact.","Vocabulary adaptation cuts the output dimension roughly 26 times, giving an 8x faster training step and lower WER than running the LLM's native tokenizer.","The weak-to-strong swap plus MWER beats both training directly with the LLM under RNN-T loss and fully fine-tuning a smaller LLM, as shown in the ablation table.","Multilingual gains replicate on French, Italian, and Dutch in the MLS corpus, so the approach is not English-specific."],"supporting_citations":[{"why":"Defines the factorized-transducer training and fusion equations plus the ILM-aware MWER loss that Transducer-Llama builds on.","marker":"[8]"},{"why":"Supplies Llama2, the 0.5B frozen or fully fine-tuned LLM used as a strong non-blank predictor.","marker":"[11]"},{"why":"Supplies Llama3, the 8B frozen LLM that gives the best WER results in the paper.","marker":"[12]"},{"why":"Documents that strong internal LMs give limited gains in modular transducers, motivating the weak-to-strong swap.","marker":"[7]"},{"why":"Provides the MWER sequence-discriminative training loss used to fine-tune the LLM integration.","marker":"[20]"},{"why":"Introduces the factorized-transducer structure that separates blank and non-blank predictors, which the paper extends.","marker":"[5]"},{"why":"Provides the offline speech-LLM baseline on MLS that the paper compares against.","marker":"[14]"},{"why":"Supplies the LibriSpeech corpus used for the main streaming ASR experiments.","marker":"[21]"}],"fun_headline_variants":["Weak-to-strong LM swap cuts streaming ASR WER by 17%","Train with weak predictor, decode with Llama3: 17% WER cut","Frozen Llama swap in transducer gives 17% streaming WER reduction","Streaming ASR: swap in Llama after training, error drops 17%","Weak predictor training then Llama swap: streaming ASR improved 17%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that an encoder trained with a weak bigram-like predictor learns acoustic representations that stay compatible with a much stronger language model when that model is swapped in at test time.","fun_headline_variants_meta":{"raw":{"variants":["Weak-to-strong LM swap cuts streaming ASR WER by 17%","Train with weak predictor, decode with Llama3: 17% WER cut","Frozen Llama swap in transducer gives 17% streaming WER reduction","Streaming ASR: swap in Llama after training, error drops 17%","Weak predictor training then Llama swap: streaming ASR improved 17%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3308,"prompt_tokens":989,"completion_tokens":2319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2213}},"tokens_in":605,"tokens_out":2319,"duration_ms":14189,"temperature":1.0,"reasoning_tokens":2213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:33:26.413762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the full pipeline but swap in a randomly initialized Llama3-sized transformer with fixed weights as the non-blank predictor before MWER; if WER stays near the Llama3 result, the gain does not come from the LLM's pre-trained text knowledge, and if WER collapses, the language-model prior is essential to the reported improvement.","supporting_citations":[{"cited_title":"Effective internal language model training and fusion for factorized transducer model,","cited_arxiv_id":null,"evidence_quote":"Defines the factorized-transducer training and fusion equations plus the ILM-aware MWER loss that Transducer-Llama builds on."},{"cited_title":"LLaMa 2: Open foundation and fine-tuned chat models,","cited_arxiv_id":null,"evidence_quote":"Supplies Llama2, the 0.5B frozen or fully fine-tuned LLM used as a strong non-blank predictor."},{"cited_title":"Modular hybrid autoregressive transducer,","cited_arxiv_id":null,"evidence_quote":"Documents that strong internal LMs give limited gains in modular transducers, motivating the weak-to-strong swap."},{"cited_title":"Efficient minimum word error rate training of RNN- transducer for end-to-end speech recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the MWER sequence-discriminative training loss used to fine-tune the LLM integration."},{"cited_title":"Factorized neural transducer for efficient language model adaptation,","cited_arxiv_id":null,"evidence_quote":"Introduces the factorized-transducer structure that separates blank and non-blank predictors, which the paper extends."},{"cited_title":"Prompting large language models with speech recognition abilities,","cited_arxiv_id":null,"evidence_quote":"Provides the offline speech-LLM baseline on MLS that the paper compares against."},{"cited_title":"LibriSpeech: an ASR corpus based on public domain audio books,","cited_arxiv_id":null,"evidence_quote":"Supplies the LibriSpeech corpus used for the main streaming ASR experiments."}],"review_version":1}