{"id":"53e57ffd-8568-4c90-99ad-8b13aa2da525","arxiv_id":"2504.18012","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Pre-trained decoders consistently improve multimodal translation, while pre-trained encoders help only when visual-text alignment is strong.","lead":"This paper tests whether pre-trained language models improve translation when images are added as extra input, comparing many model setups on English-German and English-French tasks. It reports that pre-trained decoders help reliably, while pre-trained encoders only help when images and text are well matched.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported near-perfect BLEU/METEOR/COMET scores on Multi30K (98.7–99.2 for mBART/Qwen/LLaMA, vs ~40 for published MMT systems) indicate the evaluation pipeline is not measuring held-out translation quality; the central asymmetry claim rests on this uncalibrated pipeline.","rationale":"I read the paper in good faith as a systematic empirical study of pre-trained components in MMT. For the central asymmetry claim to hold, the reported BLEU, METEOR, and COMET scores must be valid measures of held-out translation quality on Multi30K and CoMMuTE. The near-perfect scores in Table 2 make this condition implausible: BLEU 98.7–99.2 for mBART/Qwen/LLaMA on Multi30K En–De, with METEOR 99+ and COMET 91–93, far exceeds any credible published result on this benchmark. This is not merely a calibration mismatch; it is the exact signature of evaluating on the training split or otherwise leaking reference information. The reader's weakest_assumption correctly identifies this. I considered whether Table 3 alone could support the asymmetry claim, since its scores are in a plausible range. However, the paper provides no code, seeds, hyperparameters, external baselines, or error bars, and Figure 3—the key evidence for encoder sensitivity to alignment—is presented without numeric tables. Section 4.2's wording about training 'from scratch' and Figure 2's caption add further ambiguity about which split produced the near-saturated curves. None of this is a statement about author intent; it is a request for a standard evaluation check. The proposed test—recomputing one row under sacreBLEU and comparing train versus test—would settle whether the concern lands. Until such a check is reported, the reader's REJECT verdict is appropriate, so my read does not change it.","tokens_in":13986,"tokens_out":7167,"duration_ms":76806,"concrete_test":"Using the authors' checkpoint and evaluation script (which must be released), recompute the En–De mBART-Large row of Table 2 on the official Multi30K Test2016 split, scoring with sacreBLEU under standard 13a tokenization and with the reported COMET model; then score the same trained model on the Multi30K training split. If the test split also yields ~99 BLEU while the training split yields ~99, the evaluation used training data or leaked references and the asymmetry claim collapses. If the test BLEU drops to roughly 40, the published 98.7 was an artifact of nonstandard scoring, and all tables must be regenerated before the central claim can be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that pre-trained decoders consistently help while pre-trained encoders help only under good visual-text alignment—is an empirical generalization from Tables 2–4. Its load-bearing assumption is that all reported metrics are computed on the official held-out test splits with standard tokenization. That assumption is contradicted by the numbers themselves. On Multi30K En–De, published MMT systems typically score roughly 40 BLEU; Table 2 reports 98.7–99.2 BLEU, 99+ METEOR, and 91–93 COMET for mBART, Qwen, and LLaMA. Near-ceiling scores on a 1,000-sentence test set are exactly what would be observed if the pipeline evaluated on the training split, leaked references into the candidate, or used nonstandard scoring. The same table's Transformer-Large baseline (44.5 BLEU) is plausible, making the jump to 92–99 from swapping in a pre-trained model internally suspicious. Section 4.2's phrase 'trained from scratch on Multi30K' and Figure 2's caption add ambiguity about which split was scored. No code, checkpoints, hyperparameters, or external baselines are provided in Sections 3–4, so the numbers cannot be calibrated. If the metric pipeline is wrong, every comparative statement about decoders, encoders, and alignment inherits the error; Table 3's more plausible scores do not rescue the claim because the same unvalidated pipeline produced them and the headline asymmetry is stated across all tables.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of how pre-trained encoders and decoders affect multimodal machine translation (MMT) performance. Using a unified framework with CLIP image features, the authors compare Transformer-from-scratch baselines with models built on T5, mBART, Qwen2.5, and LLaMA3.2 across English-German and English-French on Multi30K and CoMMuTE, reporting BLEU, METEOR, and COMET scores. The central claim is that pre-training plays an asymmetric role: pre-trained decoders consistently improve output quality, while pre-trained encoders help only when visual-text alignment is strong. The paper also reports experiments on visual modality removal and shuffled image-text alignment, and concludes with guidance for future MMT architecture design.","tokens_in":14308,"tokens_out":2590,"duration_ms":27418,"significance":"If the empirical findings were reliable, the paper would provide useful practical guidance for MMT system design, namely that decoder pre-training is the highest-leverage component and that encoder pre-training should be paired with strong vision-language alignment. The paper also covers a broad model zoo and multiple evaluation metrics, which is a reasonable scope for a systematic study. However, the significance is currently undermined by the implausibility of the reported metric values: BLEU scores between 98.7 and 99.2 on Multi30K are far outside the range achieved by published MMT systems, and the paper provides no external baselines, code, or evaluation scripts to calibrate these numbers. Because the central asymmetry claim is derived directly from the reported tables, the contribution cannot be assessed until the evaluation pipeline is verified and corrected.","major_comments":[{"comment":"The reported BLEU scores for mBART-Large, Qwen2.5-0.5B, and LLaMA3.2-1B on Multi30K En-De and En-Fr are 98.7-99.2, with METEOR scores above 99 and COMET scores above 91. These values are far outside the plausible range for this benchmark: published MMT systems typically score around 40 BLEU on Multi30K, and the paper's own Transformer-Large baseline is 44.5 BLEU. Near-ceiling scores on a 1,000-sentence test set are exactly what one would observe if the model was evaluated on the training split, if references leaked into the candidate, or if a nonstandard scoring procedure was used. Because the abstract's central claim about asymmetric pre-training is supported only by these tables, this issue is load-bearing and must be resolved before any comparative conclusions can be drawn.","section":"Section 4.1, Table 2"},{"comment":"The evaluation protocol is insufficiently specified. The paper does not state which tokenization was used for BLEU computation (e.g., sacreBLEU with standard tokenization versus model-specific subword tokenization), whether test sets were the official held-out splits, how checkpoints were selected, what beam size was used, or whether any deduplication or postprocessing was applied. The suspiciously high scores in Table 2 cannot be diagnosed or calibrated without this information. I request the exact evaluation commands, the output files for at least one condition, and comparisons against published MMT results on the same splits.","section":"Section 3.3 and Section 4.1"},{"comment":"The caption of Figure 2 says the models are 'trained from scratch on Multi30K,' while Section 3.2 and Table 2 describe full-parameter fine-tuning of pre-trained models. These descriptions are contradictory. If the large models are initialized with pre-trained weights, the rapid convergence described as 'memory reviving' is expected and is not evidence of a new phenomenon; if they are truly trained from scratch, the near-ceiling BLEU scores become even more implausible. The authors must clarify which setting was used and align the text with the experimental setup.","section":"Section 4.2 and Figure 2"},{"comment":"Several conclusions about the effect of visual modality are based on differences of 0.1-0.3 BLEU points (e.g., mBART-Large En-De Test2016: 98.7 vs. 98.8). With a 1,000-sentence test set, such differences are within normal noise, and the paper reports no variance, no multiple runs, no significance testing, and no confidence intervals. The claim that decoder-only models benefit from visual input while encoder-decoder models do not is therefore not supported by the reported numbers.","section":"Table 4 and Section 4.4"}],"minor_comments":[{"comment":"The terms 'memory reviving' and 'continuing learning' are used in the title and Section 4.2 but are never formally defined or distinguished from standard fine-tuning and catastrophic forgetting, which makes the framing hard to evaluate.","section":"Section 1"},{"comment":"The case study claims that Qwen2.5-0.5B 'successfully reproduces the reference translation' and that mBART and LLaMA produce hallucinated forms, but the actual model outputs are not shown; only paraphrased descriptions are given. Showing the generated sentences would make the analysis verifiable.","section":"Section 4.6"},{"comment":"The dataset name is inconsistently written as 'Multi30k' and 'Multi30K' across the paper; please standardize. The MSCOCO test set is also referred to as 'MSCOCO' rather than the conventional 'MS COCO' or 'COCO'.","section":"Section 3.1"},{"comment":"The BLEU formula omits the standard brevity penalty definition and does not specify the n-gram range N; adding these details would improve reproducibility.","section":"Equation (1)"}],"recommendation":"reject","confidential_remarks":"The near-ceiling BLEU/METEOR scores in Table 2 are so far outside the expected range for Multi30K that the evaluation pipeline is almost certainly measuring something other than held-out translation quality. Since the paper's central empirical claim rests on these numbers and no code or external baselines are provided, this cannot be fixed by a revision short of re-running all experiments. I would recommend rejection unless the authors can demonstrate, with released code and standard scoring, that the scores are genuine."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline is simple: this paper asks the right question but its measuring stick is broken. The reported BLEU numbers on Multi30K—98.7 to 99.2 for mBART, Qwen, LLaMA—are outside the plausible range for held-out translation on that benchmark (published systems sit around 40). The same table shows Transformer-Large at 44.5 BLEU, which is plausible, and then T5-Small jumps to 92.3. That pattern is exactly what you'd expect from evaluating on the training split or leaking references into scoring. Without code, baselines, or hyperparameters, the numbers cannot be calibrated.\n\nWhat is genuinely new: the paper systematically varies pre-trained encoders versus decoders under one unified MMT framework, includes decoder-only LLMs (Qwen, LLaMA) as translation decoders, and adds a shuffled-alignment sensitivity test. That is a useful experimental design, and attributing the 'does vision help?' question to encoder versus decoder roles is a meaningful refinement. The related work is well covered, and the case study is at least illustrative.\n\nBut the soft spots are load-bearing. Section 4.2 says the models in Figure 2 were 'trained from scratch on Multi30K,' while Section 4.1 describes full fine-tuning; that internal inconsistency matters because it changes what the curves mean. More fundamentally, every comparative claim—decoders help consistently, encoders help only under good alignment—is drawn from Tables 2–4, which all inherit the same unvalidated scoring pipeline. The near-perfect METEOR scores (99+) are equally suspicious. The paper offers no external anchor: no published baseline, no reproduced standard system, no released code.\n\nSo: the research question is worth asking, and the experimental skeleton is sensible, but the evidence as presented does not support the conclusions. This is a paper that needs a full redo of the evaluation pipeline, external baselines, and artifact release before it can be assessed. For a reader, it might be worth skimming for the design idea, but I would not cite the results. I would not send this to peer review in its current form; I'd recommend desk reject with an invitation to resubmit after correcting the evaluation.\n\nBest,","headline":"Sensible question, broken yardstick: the paper's near-perfect BLEU scores on Multi30K invalidate its central asymmetry claim, despite a well-designed comparison of pre-trained encoders vs decoders.","tokens_in":14821,"tokens_out":4803,"would_cite":false,"duration_ms":40032,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-trained decoders, not encoders, reliably improve multimodal machine translation. The paper claims the benefit of pre-training is asymmetric: decoders give consistent gains while encoders depend on image-text alignment.","keywords":["multimodal machine translation","pre-trained language models","decoder-only LLMs","vision-language alignment","transfer learning","BLEU","Multi30K","CoMMuTE"],"falsifier":"Run the paper's fine-tuned mBART-Large, Qwen2.5-0.5B, and LLaMA3.2-1B models on Multi30K test2016 English-German with sacreBLEU and the official reference splits; if the scores land near 40 BLEU rather than 99, the asymmetry claim is an artifact of the evaluation pipeline. A second check is to verify that the training split did not overlap the test references and that COMET was computed with the published reference-based model rather than a variant that takes only the source.","tokens_in":1712,"feed_emoji":"🌐","tokens_out":1998,"duration_ms":74245,"temperature":0.7,"pith_summary":"This paper asks how pre-trained language models should be inserted into multimodal machine translation, translation that also takes an image as input, and answers with a systematic comparison under one unified framework. It claims that the benefit of pre-training is asymmetric: a pre-trained decoder consistently makes translations more fluent and accurate, whereas a pre-trained encoder helps only when the visual and textual information are well aligned. On standard benchmarks, Multi30K for English-German and English-French plus the ambiguity-focused CoMMuTE set, decoder-only models such as Qwen2.5 and LLaMA3.2 are the most stable, while encoder-decoder models degrade when images are shuffled or semantically redundant. A sympathetic reader would care because this gives architecture guidance: spend pre-training or fine-tuning budget on decoders, and treat image features as flexible prompts rather than trusted semantic context.","feed_headline":"Pre-trained decoders, not encoders, drive MMT gains","feed_subtitle":"Decoder pre-training reliably lifts multimodal translation; encoder gains hinge on image-text alignment.","key_machinery":"The load-bearing machinery is a unified MMT baseline that fixes the visual encoder as frozen CLIP-ViT, varies the text encoder and decoder across Transformer-Small, Transformer-Base, Transformer-Large, T5-Small, mBART-Large, Qwen2.5-0.5B, and LLaMA3.2-1B, and combines them with different training strategies, from scratch, fully fine-tuned, or partially frozen. The comparison is carried by three metrics, BLEU, METEOR, and COMET, on three Multi30K test splits and CoMMuTE. The key manipulation is the shuffled-alignment condition, which randomly pairs sentences with unrelated images to expose how much each architecture depends on vision-language alignment; that condition is what separates encoder-sensitive from decoder-robust behavior.","core_discovery":"The central discovery is that pre-training contributes to multimodal machine translation through an asymmetry between encoder and decoder. Under a shared baseline in which a frozen CLIP vision encoder supplies image features, replacing the text encoder with a pre-trained encoder, T5 or mBART, helps on clean, well-aligned image-text pairs but becomes unstable, sometimes harmful, when the visual signal is noisy, redundant, or shuffled. Replacing the decoder with a pre-trained decoder or decoder-only LLM, T5, mBART, Qwen2.5, or LLaMA3.2, gives consistent gains in BLEU, METEOR, and COMET across all test sets, and these models are more robust to misaligned images. The paper also reports that large pre-trained models converge quickly by reviving stored linguistic knowledge, yet still need continued fine-tuning to handle fine-grained multimodal grounding. In its case studies, the decoder-only Qwen2.5 model resolves lexical ambiguity correctly, for example translating 'mole' as 'Leberfleck', while encoder-decoder models hallucinate words under visual distraction.","pith_inferences":["The paper does not state it, but its results suggest that if the evaluation is valid, a large part of MMT's difficulty collapses once the decoder is pre-trained; a natural test is whether the same asymmetry holds on lower-resource or more distant language pairs.","The shuffled-alignment robustness of decoder-only models implies a practical defense against noisy vision: feed image features as soft prompts rather than fusing them into the encoder.","For sentences whose meaning is fully determined by text, the paper's results imply a text-only decoder may match or beat a multimodal encoder-decoder, so future work could benchmark text-only versus image-prompted versions of the same decoder."],"forward_implications":["If the central claim is right, multimodal translation systems should allocate pre-training resources to the decoder side, since decoder pre-training reliably improves fluency and accuracy.","Encoder pre-training should be deployed only where image-text alignment is known to be strong; with noisy or unrelated images it can hurt.","Decoder-only LLMs are the most robust choice under misaligned or ambiguous visual input, because strong language modeling lets them ignore misleading image cues.","Adding images is not universally beneficial: for encoder-decoder models visual input often acts as noise, while for decoder-only models it can help disambiguate.","The observed fast convergence of large pre-trained models means memory revival is not enough; continued task-specific training is still required."],"supporting_citations":[{"why":"Supplies the Multi30K dataset and task setup used for the main English-German and English-French experiments.","marker":"(Elliott et al., 2016)"},{"why":"Supplies the CoMMuTE evaluation set that tests lexically ambiguous sentences with contrastive image pairs.","marker":"(Futeral et al., 2023)"},{"why":"Provides the frozen CLIP vision encoder used to extract image features.","marker":"(Radford et al., 2021)"},{"why":"Defines the Transformer encoder and decoder baselines that the pre-trained components replace.","marker":"(Vaswani et al., 2017)"},{"why":"Provides the mBART-Large pre-trained encoder-decoder model central to the comparison.","marker":"(Liu et al., 2020b)"},{"why":"Provides the T5-Small pre-trained encoder-decoder model used in the comparison.","marker":"(Raffel et al., 2019)"},{"why":"Provides the Qwen2.5-0.5B decoder-only LLM that exhibits the strongest robustness.","marker":"(Qwen et al., 2025)"},{"why":"Provides the LLaMA3.2-1B decoder-only LLM used in the comparison.","marker":"(Grattafiori et al., 2024)"},{"why":"Defines the COMET metric used to measure semantic quality of translations.","marker":"(Rei et al., 2020)"},{"why":"Defines the BLEU metric used to measure n-gram overlap with references.","marker":"(Papineni et al., 2002)"}],"fun_headline_variants":["Pre-trained decoders boost multimodal translation, encoders don't","Decoder pre-training wins in MMT; encoder gains depend on alignment","Why pre-trained decoders, not encoders, lift multimodal MT","MMT asymmetry: Decoder pre-training consistently helps, encoder not","Pre-trained decoders shine in multimodal translation, encoders vary"],"cache_read_input_tokens":16896,"weakest_assumption_plain":"The asymmetry claim rests on the reported BLEU, METEOR, and COMET numbers being genuine held-out test scores with standard tokenization; since BLEU values near 99 on Multi30K are far above the roughly 40 BLEU that established systems report on the same benchmark, any leak or nonstandard scoring would void every comparative conclusion.","fun_headline_variants_meta":{"raw":{"variants":["Pre-trained decoders boost multimodal translation, encoders don't","Decoder pre-training wins in MMT; encoder gains depend on alignment","Why pre-trained decoders, not encoders, lift multimodal MT","MMT asymmetry: Decoder pre-training consistently helps, encoder not","Pre-trained decoders shine in multimodal translation, encoders vary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1442,"prompt_tokens":959,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":575,"tokens_out":483,"duration_ms":4266,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:26:40.719017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's fine-tuned mBART-Large, Qwen2.5-0.5B, and LLaMA3.2-1B models on Multi30K test2016 English-German with sacreBLEU and the official reference splits; if the scores land near 40 BLEU rather than 99, the asymmetry claim is an artifact of the evaluation pipeline. A second check is to verify that the training split did not overlap the test references and that COMET was computed with the published reference-based model rather than a variant that takes only the source.","supporting_citations":[],"review_version":1}