{"id":"ff58f5d9-ebed-4504-a1c0-cc0428a86efb","arxiv_id":"2505.13036","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"KIT combines LLM-based ASR fusion with quality-filtered fine-tuning and post-editing for offline speech translation, and a contrastively pretrained SpeechLLM with post-editing for multilingual instruction following.","lead":"This paper describes KIT's systems for the IWSLT 2025 offline English-to-German speech translation and constrained long-form instruction-following tracks, using LLMs in both cascaded and end-to-end designs. The offline system fuses multiple ASR outputs with an LLM and post-edits translations; the instruction-following system pairs a SeamlessM4T encoder with a LLaMA-3.1 model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SQA/SSUM and en-it ST claims rest on references generated by the same LLaMA/Seamless models used in the system; the reported final-model superiority may be an artifact of train/eval distribution overlap.","rationale":"I read the paper as a system description whose core technical claims are modest and mostly supported by ablations on development sets. The Offline-track claims (LLM fusion and APE improve MetricX/COMET on ITV and ACL) rely on human-reference evaluation and are internally consistent. The IF-track contrastive-pretraining benefit is supported by ASR and ST results on ACL 60/60, which are human-based. The weakest point is the SQA/SSUM and en-it ST evaluation, where generated references overlap with the training-data generation pipeline. The paper is transparent about the asterisked test sets, but the conclusion in Section 3.7 does not carry that caveat strongly enough. The reader's weakest assumption identifies exactly this issue, and my stress-test refines it: the problem is not merely that references are model-generated, but that they are generated by the same model and prompt family used to construct the training data, creating a train/eval distribution leak that systematically favors the IF model over baselines not trained on that distribution. This does not require rejection: the ASR/ST evidence stands, and the SQA/SSUM concern can be resolved with independent references. Therefore I keep the reader's conditional verdict unchanged.","tokens_in":15588,"tokens_out":9356,"duration_ms":93249,"concrete_test":"Recompute Table 8 SQA/SSUM and en-it ST scores using references that do not come from the training-data generator: either the official IWSLT 2025 test set with human references, or a 250-sample subset of NUTSHELL test segments with SQA answers and SSUM summaries written by human annotators (and en-it ST translations produced by a professional translator). If IF contr. wasser. tag + post-edit no longer outperforms Phi-4, Qwen2-Audio, and Whisper+LLaMA on SQA/SSUM, the reported final-model superiority is an artifact of the synthetic reference distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central IF-track claim that the final submitted model 'offers the best performance for ASR, SQA, and SSUM, and is competitive with the other IF models in ST' (Section 3.7) is most vulnerable on SQA and SSUM, where the evaluation references are not human annotations. Section 3.1 states that the 250-sample SQA test set is generated with LLaMA-3.1-8B-Instruct from NUTSHELL test data, and the multilingual SSUM references are LLaMA-3.1 translations of NUTSHELL. The en-it ST reference is likewise generated by SeamlessM4T-Large. Crucially, the same LLaMA-3.1 model and same prompts are used for the augmented SQA/SSUM training data, so the test set is drawn from the exact distribution the IF model was trained to mimic. The model's LLM backbone is LLaMA-3.1-8B-Instruct and its speech encoder is SeamlessM4T-Large, so the same model families appear on both sides of evaluation. Baselines such as Phi-4 and Qwen2-Audio were not trained on this generated distribution, so their lower BERTScore/COMET values in Table 8 may reflect distribution mismatch rather than worse instruction following. Even the Whisper+LLaMA baseline, though using LLaMA, is not fine-tuned on the generated SQA/SSUM data. This does not undermine the contrastive-pretraining ablation on ASR and en-de/en-zh ST, where references come from ACL 60/60, but it does undermine the final-model selection and the SQA/SSUM superiority claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes KIT's submissions to the IWSLT 2025 Offline Speech Translation track (English-to-German) and the Constrained Long Instruction-Following (IF) track. For the Offline track, the system combines VAD-based segmentation, multiple ASR systems whose outputs are fused by an LLM at document level, a quality-filtered fine-tuned Tower 7B translation model, and a Tower 13B automatic post-editing stage. For the IF track, the authors build an end-to-end SpeechLLM by connecting a SeamlessM4T-Large encoder to LLaMA-3.1-8B-Instruct through a Q-Former, and they compare direct fine-tuning, ASR pretraining, and contrastive pretraining with cosine or Wasserstein losses. They further study VAD segmentation lengths, a chain-of-thought 'tag' variant for SQA, and document-level post-editing. The central claims are that contrastive pretraining substantially improves IF performance over no pretraining and ASR pretraining (Table 5), that LLM fusion and APE improve Offline ST quality (Table 4), and that the final submitted IF model 'offers the best performance for ASR, SQA, and SSUM, and is competitive with the other IF models in ST' (Section 3.7).","tokens_in":15979,"tokens_out":8041,"duration_ms":73676,"significance":"If the results hold, the paper provides useful evidence for two practical design choices: contrastive pretraining of a frozen-encoder SpeechLLM helps in a constrained setting, and a cascaded pipeline with multi-ASR fusion plus automatic post-editing improves noisy speech translation. The ablations in Tables 5-7 are clearly structured and the augmented dataset is released on HuggingFace, which is valuable for reproducibility. However, the SQA, SSUM, and en-it ST evaluations rely on references generated by the same model families (LLaMA-3.1-8B-Instruct and SeamlessM4T-Large) that are used inside the submitted system, so the headline superiority claims on those tasks are not yet established. The core ASR and en-de/en-zh ST results on ACL 60/60 are not affected by this concern, but the paper's final-model selection and some cross-model comparisons need additional support.","major_comments":[{"comment":"The SQA, SSUM, and en-it ST evaluation sets are generated by the same model families used in the submitted system. The 250-sample SQA/SSUM test sets are built from NUTSHELL data using LLaMA-3.1-8B-Instruct-generated questions/answers and LLaMA-3.1 translations (Section 3.1), and the en-it ACL 60/60 references are produced by SeamlessM4T-Large. Since the IF model's LLM is LLaMA-3.1-8B-Instruct, its speech encoder is SeamlessM4T-Large, and the SQA/SSUM augmentation training data are generated with the same LLaMA-3.1 prompts and distribution, the IF models are evaluated on a distribution they were explicitly trained to mimic, while baselines such as Phi-4 and Qwen2-Audio were not. The reported BERTScore/COMET advantages in Table 8 for SQA, SSUM, and en-it ST may therefore reflect distribution familiarity rather than task competence. This directly affects the Section 3.7 claim that the final model 'offers the best performance for ASR, SQA, and SSUM.' The paper should either add a human-annotated evaluation sample for these tasks, compare baselines on references not produced by the authors' own models, or explicitly restrict the superiority claims to exclude these generated-reference settings.","section":"Section 3.1, Section 3.7, Table 8"},{"comment":"All experimental results are reported from single runs without error bars, confidence intervals, or significance tests. Several of the decisions that determine the final submission rest on small differences: in Table 5, contrastive cosine versus Wasserstein gives 18.82 vs. 19.07 WER and 77.31 vs. 77.33 COMET for en-de ST; in Table 7, ST en-de COMET improves from 71.61 with no post-editing to 71.91 with a context size of 3, a gain of 0.30 that is likely within run-to-run noise. Given that the paper selects one of several closely matched configurations as the final model, single-run reporting without variance estimates makes it difficult to know whether the selected configuration is genuinely better than the alternatives. The authors should add multiple seeds or significance tests for the key comparisons, or soften the claims that depend on these differences.","section":"Section 3.2, Section 3.7, Tables 5 and 7"},{"comment":"The statement that the submitted model 'IF contr. wasser. tag + post-edit ... offers the best performance for ASR, SQA, and SSUM' is not fully supported by Table 8. For SQA en-en, IF contr. cos. tag achieves BERTScore 82.86, while the submitted model reaches 82.80; for SQA en-zh, IF contr. cos. tag achieves 80.53 vs. 80.32 for the submitted model. If the authors intend 'best among the VAD-segmented variants' or 'best according to a different aggregation criterion,' that criterion should be stated explicitly. As written, the claim contradicts the numbers in the paper's own results table and should be corrected.","section":"Section 3.7 vs. Table 8"}],"minor_comments":[{"comment":"The figure caption contains the text 'We use bird score and blue score,' which appears to be a typo for 'BERTScore and BLEU score'; please correct it.","section":"Figure 1"},{"comment":"There are several typos, including 'submision' and 'plently'; please proofread the manuscript.","section":"Section 2.3.2"},{"comment":"'optimalchunk' should read 'optimal chunk size'; also update the sentence for clarity.","section":"Section 2.1"},{"comment":"Please clarify how the 250-sample test sets are sampled from NUTSHELL test data and confirm that they are disjoint from the augmented training data, so that readers can assess potential train/eval leakage.","section":"Section 3.1"},{"comment":"Please specify the exact BERTScore configuration (e.g., model and F1 variant) used for SQA and SSUM, and the COMET checkpoint used for ST, to improve reproducibility.","section":"Section 3.6"},{"comment":"The table is very wide and the '+ post-edit' rows are easy to misalign with their parent rows; consider repeating the model name on each row or using clearer separators.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"This is a system description for a shared task, and the core ASR and en-de/en-zh ST results rest on the ACL 60/60 test set, so the main pipeline claims are probably salvageable. The generated-reference issue is serious but fixable: the authors can add a small human evaluation, rerun the SQA/SSUM/en-it comparisons on independent human references, or explicitly narrow the claims. The paper fits the venue, but the current overclaiming in Section 3.7 should be corrected in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a system description, not a research breakthrough, and it is honest about that. Two things to know: the Offline ST pipeline (ASR fusion + APE) looks real and the en-de gains on ACL 60/60 (MetricX 2.27 to 1.84) are credible because that test set is human-annotated. The IF-track ASR/ST ablations are also well done and internally consistent: contrastive pretraining beats direct fine-tuning and ASR pretraining on Table 5. Those results survive scrutiny.\n\nWhere it gets shaky is the SQA/SSUM and en-it ST evaluations. Section 3.1 explains that the SQA test set is generated by LLaMA-3.1-8B-Instruct, the multilingual SSUM references are LLaMA-3.1 translations, and the en-it ST reference comes from SeamlessM4T-Large. Your own model uses LLaMA-3.1-8B-Instruct as the LLM and SeamlessM4T-Large as the speech encoder. You have effectively trained and evaluated on the same model-generated distribution. That does not invalidate the contrastive-pretraining ablation, but it does mean the final-model claim (\"best performance for ASR, SQA, SSUM\") is not justified for SQA/SSUM. The baselines (Phi-4, Qwen2) were not fine-tuned on that generated data, so their lower BERTScore is partly a distribution mismatch, not necessarily worse instruction following.\n\nThe paper discloses these asterisked test sets clearly, which is good practice, but the text still overstates the conclusion. Also, everything is single-run with no significance tests—minor for a workshop system paper, but it means gaps like 18.82 vs 19.07 WER could be noise. There is a typo in Table 8 where '79.88' and '86.83' run together, plus a few other proofing slips.\n\nSo the reader's take is about right. The Offline track and the IF ASR/ST ablations are worth taking seriously. The SQA/SSUM numbers should be treated as indicative, not evidence of superiority. For a shared-task system description, this deserves referee time—it contains enough careful engineering and a few useful insights (e.g., fusion helps ST even when WER does not improve). I would recommend accept-with-revisions: ask the authors to either re-evaluate SQA/SSUM on human-annotated data or soften the claims and explicitly discuss the distribution overlap.\n\nNot something I would cite next year, but I would point students toward it as a solid example of constrained-system building with honest ablations.","headline":"A decent shared-task system paper whose ASR/ST results hold up, but whose SQA/SSUM superiority claims rest on self-generated references from the same model family the system uses.","tokens_in":16550,"tokens_out":4048,"would_cite":false,"duration_ms":37777,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that contrastive pretraining—not data or scale—is what makes a frozen-encoder speech LLM suddenly good at four instruction-following tasks, and that LLM fusion plus post-editing completes the offline translation system.","keywords":["speech translation","instruction following","speech LLM","contrastive pretraining","ASR fusion","automatic post-editing","spoken question answering","speech summarization"],"falsifier":"Take the ACL 60/60 audio, obtain human-written answers, summaries, and en-it translations for the same 250-sample subsets, and rerun the Table 8 comparison; if the contrastive-pretrained IF models no longer beat the baselines on those human references, the central claim fails. For the offline track, a comparable check is to replace the MetricX evaluation with a human annotation task on the final translations and see whether the 1.84 versus 2.27 gap persists.","tokens_in":15393,"feed_emoji":"🎤","tokens_out":6472,"duration_ms":58238,"temperature":0.7,"pith_summary":"This paper reports a pair of systems for two IWSLT 2025 tracks, asking whether a single speech-conditioned LLM can follow varied spoken-language instructions and whether LLMs assembled around a cascade improve offline translation. For the instruction-following track, it tries to establish that contrastive pretraining of the projector that bridges a frozen speech encoder and a frozen LLM is the decisive ingredient, beating direct fine-tuning and ASR pretraining on every task. For the offline translation track, it tries to establish that fusing the hypotheses of several ASR systems with an LLM at document level, and then post-editing the translation, pushes quality beyond any single system. The reader should care because both results point to a cheap way to get more from frozen foundation models: train only a small connector and let an LLM reconcile multi-system outputs in context.","feed_headline":"Contrastive pretraining lifts a speech LLM across four tasks","feed_subtitle":"Its follow-up document-level fusion and post-editing push offline translation to MetricX 1.84.","key_machinery":"For the IF track, the load-bearing object is a Q-Former projector—four transformer layers with four learnable query tokens—trained to map frozen SeamlessM4T speech features into frozen LLaMA-3.1 text space; the paper's claim is that contrastive pretraining of this bridge, using cosine similarity or Wasserstein distance, creates a representation that downstream fine-tuning can exploit across all four tasks. For the offline track, the load-bearing mechanism is document-level LLM fusion of ASR hypotheses, where a fine-tuned Llama-3-8B with LoRA sees several system transcripts and predicts the reference transcription, followed by automatic post-editing of the machine translation output, both run with surrounding context rather than sentence-by-sentence.","core_discovery":"The central claim of the instruction-following half is that contrastive pretraining of the trainable connector between a frozen speech encoder (SeamlessM4T-Large) and a frozen LLM (LLaMA-3.1-8B-Instruct) is the decisive improvement over direct fine-tuning and ASR pretraining. On the ACL 60/60 test set, the contrastive-cosine model reaches 18.82 WER for English ASR versus 25.1 with no pretraining and 21.42 with ASR pretraining, and it improves COMET for en→de, en→it, and en→zh translation as well as BERTScore for SQA and SSUM. The submitted model, the Wasserstein-contrastive variant with answer tagging and document-level post-editing, gives the best ASR, SQA, and SSUM results among the system's own models and is competitive in ST. The offline half claims that fusing four ASR hypotheses with a fine-tuned LLM at document level, then translating with a quality-filtered fine-tuned Tower 7B and post-editing with Tower 13B, lowers MetricX on ACL 60/60 from 2.27 for the best single pipeline to 1.84.","pith_inferences":["Editorial inference: the reported IF gains are measured against test sets whose SQA/SSUM references were generated by LLaMA-3.1-8B-Instruct and whose en-it ST reference was generated by SeamlessM4T-Large; if human-created references disagree with those generations, the margin between the submitted system and the baselines could shrink.","Editorial inference: the key result likely transfers to other frozen encoder/LLM pairs with Q-Former style bridges, since the paper's ablation shows the benefit comes from the pretraining objective, not from model scale.","Editorial inference: a direct test of the offline pipeline's robustness would be to run it on meetings or lectures with human-annotated transcripts and translations, where domain shift and segmentation noise would stress the fusion and APE components.","Editorial inference: the paper's observation that lower WER does not predict translation quality suggests shared-task scoring should weight downstream metrics, not ASR accuracy, when ranking speech translation systems."],"forward_implications":["If the IF claim holds, a single frozen speech encoder plus frozen LLM can cover ASR, ST, SQA, and SSUM with only a small trainable bridge, which makes multi-task speech assistants much cheaper to build.","Contrastive pretraining should become the default initialization for this kind of SpeechLLM, not direct fine-tuning: it cuts WER by about six points and lifts every downstream metric reported in the paper.","Document-level LLM fusion can improve translation even when the fused transcript is not the lowest-WER hypothesis, because translation quality depends on punctuation, casing, and context that WER ignores.","Post-editing with surrounding context partially reverses the damage done by segmenting long audio, so long-form spoken translation systems should treat segmentation and post-editing as one joint design problem.","In offline translation, the full pipeline reduces MetricX from 2.27 to 1.84 on ACL 60/60, suggesting that refinement layers add more than swapping ASR models."],"supporting_citations":[{"why":"Supplies the contrastive pretraining objective and the hyperparameter configuration adopted for the IF SpeechLLM.","marker":"Zü fle and Niehues (2024)"},{"why":"SeamlessM4T-Large provides the frozen speech encoder and also generates the en-it reference translations used for evaluation.","marker":"Communication et al. (2023)"},{"why":"LLaMA-3.1-8B-Instruct is the frozen LLM in the constrained IF setting and the base model for the post-editing stage.","marker":"Grattafiori et al. (2024)"},{"why":"Whisper v2 and v3 supply the ASR hypotheses for fusion and serve as the strongest ASR baselines in the IF track.","marker":"Radford et al. (2023)"},{"why":"Provides the sentence- and document-level automatic post-editing approach reused for translation refinement in both tracks.","marker":"Koneru et al. (2024b)"},{"why":"Motivates the cascaded LLM-in-the-loop design and the use of long-form context when segmenting noisy audio.","marker":"Koneru et al. (2024a)"},{"why":"XCOMET quality estimation is used to filter the parallel data for fine-tuning the Tower 7B translation model.","marker":"Guerreiro et al. (2024)"},{"why":"Tower 7B is the base translation LLM and Tower 13B is the base for the offline track's APE model.","marker":"Xu et al. (2024a)"}],"fun_headline_variants":["Contrastive pretraining is the key to a better speech LLM","Document-level fusion of ASR outputs improves translation","End-to-end speech instruction following with a frozen encoder and LLM","KIT's pipeline fuses multiple ASRs and refines with LLM","Offline ST improves with document-level fusion and post-editing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The IF track's reported advantages are measured on machine-generated references produced by the same model families used inside the system, so if those references diverge from human judgment, the reported ranking of models may be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive pretraining is the key to a better speech LLM","Document-level fusion of ASR outputs improves translation","End-to-end speech instruction following with a frozen encoder and LLM","KIT's pipeline fuses multiple ASRs and refines with LLM","Offline ST improves with document-level fusion and post-editing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000993,"raw_usage":{"total_tokens":4234,"prompt_tokens":997,"completion_tokens":3237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":3148}},"tokens_in":613,"tokens_out":3237,"duration_ms":23728,"temperature":1.0,"reasoning_tokens":3148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:21:16.585002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the ACL 60/60 audio, obtain human-written answers, summaries, and en-it translations for the same 250-sample subsets, and rerun the Table 8 comparison; if the contrastive-pretrained IF models no longer beat the baselines on those human references, the central claim fails. For the offline track, a comparable check is to replace the MetricX evaluation with a human annotation task on the final translations and see whether the 1.84 versus 2.27 gap persists.","supporting_citations":[],"review_version":1}