{"id":"75500aad-892f-47d5-8adf-fa289873eda7","arxiv_id":"2412.02101","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage decoder-only architecture with instruction-level contrastive learning improves zero-shot multilingual translation and closes most of the gap to encoder-decoder models.","lead":"The paper proposes a two-stage decoder-only architecture plus instruction-level contrastive learning to make decoder-only multilingual translation models transfer source representations into the target language subspace, improving zero-shot translation. A generalist reader might care because it shows how to recover much of the encoder-decoder advantage without the encoder, using fewer parameters and better zero-shot performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"On OPUS-100, TDO alone does not improve zero-shot BLEU over the vanilla prefix decoder-only baseline; the reported gains are driven by InstruCL, so the claimed two-stage mechanism for language transfer is not supported by the paper's own ablations.","rationale":"The reader identified the mechanism of Equation 4 as the weakest assumption. My stress-test sharpens this with a concrete empirical inconsistency: on OPUS-100, the TDO architecture alone does not deliver the claimed zero-shot improvement, and the headline gains come from InstruCL. This does not invalidate the empirical contribution of the combined method, but it does undercut the paper's central explanatory claim that target-token exclusion in the first stage is what enables language transfer. The similarity metric used to support the mechanism is unvalidated and based only on TED-19, so it cannot rescue the interpretation. This is a load-bearing concern because the paper's novelty and framing rest on the language-transfer story; if the mechanism is instead the contrastive loss or simply a better source-encoder-like first stage, the architectural insight is overstated. I recommend keeping the reader's conditional verdict: the paper should either add the proposed ablation (removing ly from the first stage) or explicitly weaken the causal language about language transfer. No adversarial intent is implied; the empirical tables are valuable regardless, but the central claim needs this check.","tokens_in":23694,"tokens_out":9416,"duration_ms":98908,"concrete_test":"On OPUS-100 (N=12,d=512), run TDO+Prefix and TDO+Prefix+CL with the target-language instruction ly in the first stage replaced by the source-language tag lx, while keeping ly in the second stage. If the language-transfer mechanism is real, zero-shot BLEU for TDO+Prefix should drop from 5.32 below the 6.62 vanilla baseline, and the InstruCL gain should shrink; if zero-shot BLEU is unchanged or improves, the first stage is not using ly to align into the target subspace. Also report the Figure-4 similarity scores for this variant; if the score rises despite the removed instruction, the metric does not measure target-language transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is that excluding target tokens in the first M layers of TDO enables language transfer into the target-language subspace and that this is why decoder-only translation improves. The paper's own ablations contradict this for the larger-scale OPUS-100 setting. In Table 1 (OPUS-100, N=12, d=512), the plain prefix decoder-only baseline scores 6.62 zero-shot BLEU; TDO+Prefix alone scores 5.32, and TDO+Prefix+Adaption scores 5.97. Only when InstruCL is added do zero-shot scores jump to 7.93-8.52, producing the headline +3.39 over Enc-Dec. The same pattern appears in the OPUS N=6,d=1024 block: 8.15 baseline vs. 7.06 for TDO+Prefix, 9.28 for TDO+Prefix+CL. Thus the two-stage architecture, the paper's proposed 'implicit alignment' mechanism, does not by itself confer zero-shot language transfer on the large dataset; the contrastive loss is the effective ingredient. The only direct evidence for the mechanism is the Figure 4 similarity curve, computed on TED-19 with an unvalidated SVCCA-based score (Appendix A) that is never shown to correlate with translation quality and is reported without variance. The paper's conclusion that 'improvements derive from enhanced language transfer' (Section 7) is therefore an interpretation that its own large-scale ablations weaken.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that decoder-only multilingual NMT underperforms because source-token representations are not transferred into the target-language subspace, and proposes a Two-stage Decoder-only (TDO) architecture that excludes target tokens for the first M layers, with adaption modules at the stage boundary, plus Instruction-level Contrastive Learning (InstruCL) on the translation-instruction token. Experiments on TED-19 and OPUS-100, in both from-scratch and fine-tuning settings, report that TDO+InstruCL is competitive with encoder-decoder baselines on supervised directions and improves zero-shot translation by up to 3.39 BLEU, 6.99 chrF++, 3.22 BERTScore, and 4.81 COMET. A layer-wise similarity analysis (Appendix A) is offered as evidence that the proposed methods increase target-language preference in source representations.","tokens_in":23968,"tokens_out":6524,"duration_ms":67459,"significance":"If the causal interpretation is correct, TDO+InstruCL provides a decoder-only architecture that matches or exceeds encoder-decoder MNMT at comparable or slightly lower parameter counts, with practical zero-shot gains. The evaluation is genuinely broad: two corpora, training from scratch and fine-tuning, four automatic metrics plus target-off ratio, ablations over architecture components, M, and the InstruCL layer index, and released code. The main weakness is that the evidence for the architectural mechanism is incomplete and partly contradicted by the paper's own large-scale ablations; moreover, the representation metric used to support the mechanism is not independently validated. These issues affect the central attribution of the gains, so the paper needs substantial revision before the claims can be accepted.","major_comments":[{"comment":"The ablation does not support the causal claim that the two-stage split repairs decoder-only language transfer. With N=12, d=512, zero-shot BLEU is 6.62 for the prefix decoder-only baseline, 5.32 for TDO(+Prefix), 5.97 for TDO+Adaption, and only 7.93/8.52 after adding InstruCL; the N=6, d=1024 block shows the same pattern (8.15 vs 7.06, then 9.28 with CL). Since Section 7 concludes that the improvements derive from TDO-enhanced language transfer, the OPUS-100 results attribute the zero-shot gains to the contrastive objective rather than to the architecture itself. Please either restrict the architectural claim to TED-19 or provide OPUS-100 representation evidence that TDO improves transfer even when BLEU does not.","section":"§5.3, Table 1 (OPUS-100 blocks)"},{"comment":"Figure 4's similarity score is the only direct evidence that TDO realizes the alignment in Eq. 4, but Eq. 11 is an average over parallel sentences and is not shown to track translation quality. The score is computed with an SVCCA procedure taken from prior work, and no variance or per-pair distribution is reported. A concrete test would be to correlate the similarity score with held-out BLEU/COMET across the models in Table 1, or to probe Eq. 4 directly by measuring pairwise similarity of the encoder outputs of parallel sentences sharing a target language; as it stands, the representation analysis is interpretive rather than confirmatory.","section":"§6.1 and Appendix A"},{"comment":"The M sweep on OPUS-100 shows that zero-shot BLEU is best at M=3 and declines as M grows, while the main experiments use M=N and claim that a larger first stage improves transfer. This non-monotonicity is consistent with the 'second stage benefits linguistic diversity' explanation, but it undercuts the simple statement that excluding target tokens in the first stage is what transfers source representations into the target subspace. Please reconcile the M-sweep behavior with the proposed mechanism and state which M value is recommended for large corpora.","section":"§6.2, Figure 5"},{"comment":"The paper reports single point estimates without confidence intervals or multiple seeds. Several fine-tuning differences are small, for example Table 2, NLLB-600M: TDO+CL vs TDO zero-shot BLEU is 15.48 vs 15.48, yet the text draws conclusions about InstruCL effectiveness from differences of this size. Please report variance or significance at least for the headline comparisons and for the component ablations in Table 1.","section":"Tables 1–3 and Appendix I"}],"minor_comments":[{"comment":"The caption says 'Abbreviations align with Table 2,' but it should refer to Table 1.","section":"Table 2 caption"},{"comment":"The label 'InstrucCL' is used once in the enumeration of analyzed models; elsewhere the method is called InstruCL.","section":"§6.1"},{"comment":"The notation after singular value decomposition is unclear: h_a ∈ R^{d_a} is introduced without defining d_a and d_b, and the relationship between the token-level H and the sentence-level h should be stated more carefully.","section":"Appendix A"},{"comment":"The term 'variation ratio' should be defined explicitly, including the reference model against which the variation is computed and the handling of negative values.","section":"Figure 5 caption"},{"comment":"There is a typo in 'Tabel 3 shows the experimental results,' and the placeholder '-' in the idx column would be clearer as 'n/a'.","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing before you read this one. First, the combined system works: Two-stage Decoder-only plus instruction-level contrastive learning beats encoder-decoder baselines on zero-shot translation on both datasets, with gains up to 3.39 BLEU and 6.99 chrF++, and the fine-tuning experiments show you can convert an encoder-decoder checkpoint into a decoder-only model without sacrificing quality. Second, the headline mechanism story does not survive the paper's own large-scale ablations. On OPUS-100, TDO without InstruCL gets 5.32 zero-shot BLEU against 6.62 for a vanilla prefix decoder-only model; the two-stage architecture alone makes zero-shot worse. The jump to 7.93-8.52 comes only when InstruCL is added. On TED-19 the architecture does help on its own (13.56 vs 8.18), so the implicit-alignment idea is plausible at small scale, but at the scale where MNMT is actually used, the contrastive loss is doing the heavy lifting.\n\nWhat's new: nobody has split the decoder-only stack into two explicitly separated stages for MNMT before, and the instruction-level contrastive objective with the identity-pair anchor is a clever, simple idea. The experiments are thorough: two datasets, scratch and fine-tuning, four metrics plus target-off ratio, ablations on M and on the layer index. Appendix J is a nice control, showing the adaptation modules are not just extra parameters.\n\nSoft spots, in order of weight. The mechanistic interpretation outruns the evidence: the only direct support is an SVCCA-based similarity score from the authors' prior work, never validated against translation quality and reported without uncertainty intervals. There is no significance testing anywhere; the fine-tuning gains are under 1 BLEU in most cells and could easily be noise. Section 6.2 partially acknowledges the OPUS-100 pattern, but the conclusion still claims the gains 'derive from enhanced language transfer,' conflating the architecture with the loss. A careful revision should separate those two and soften the claim.\n\nThis paper is for people building decoder-only MT systems and anyone doing representation analysis in multilingual models. It deserves a serious referee: the recipe is concrete, the negative result buried in the ablations is informative, and the method could matter to the field. I'd send it to review with a request to temper the mechanism claim and add error bars.","headline":"A worthwhile empirical recipe for decoder-only multilingual translation whose central mechanism claim is contradicted by its own large-scale ablations.","tokens_in":24541,"tokens_out":4025,"would_cite":true,"duration_ms":38840,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decoder-only translation models underperform because they fail to transfer source representations into the target language, and a two-stage design plus instruction contrastive learning fixes this.","keywords":["multilingual neural machine translation","decoder-only architecture","zero-shot translation","language transfer","contrastive learning","translation instruction","two-stage decoder","representation analysis"],"falsifier":"Train a TDO variant with the same parameter count but no target-token exclusion, either by letting target tokens attend from the start or by replacing the split with a learned routing that keeps source and target information separate, and check whether the zero-shot gains persist; if they do, the two-stage exclusion is not the cause. A cheaper check is to feed two parallel source sentences in different languages with the same target language through the first $M$ layers and compute the cosine similarity of the resulting representations; if it is not higher than the vanilla decoder-only model's similarity, the claimed alignment of Equation 4 has not occurred.","tokens_in":23469,"feed_emoji":"🌐","tokens_out":8233,"duration_ms":70515,"temperature":0.7,"pith_summary":"The paper claims that decoder-only translation models fall behind encoder-decoder models in multilingual settings because they never push source-token representations toward the target language; unlike an encoder, a decoder-only stack has no intermediate state that can be aligned across source languages. To fix this, the authors split the decoder-only stack into two stages: the first $M$ layers see only source tokens plus the target-language tag, and the remaining layers fuse in target tokens as usual, with a shared adapter at the boundary. They add Instruction-level Contrastive Learning, which pulls the representation of the target-language instruction toward an identity pair that acts as a proxy for the target language, to stop the source representations from drifting back to the source side. Across TED-19 and OPUS-100, the resulting TDO model matches encoder-decoder baselines on supervised pairs and exceeds them on zero-shot pairs by up to 3.39 BLEU, 6.99 chrF++, 3.22 BERTScore, and 4.81 COMET, while using fewer parameters. The paper's own representation analysis shows layer-wise similarity scores crossing the target-language threshold earlier in TDO than in a plain decoder-only model.","feed_headline":"Two-stage decoder beats encoder-decoder on zero-shot translation","feed_subtitle":"Splitting decoding into two stages plus contrastive instruction learning gains up to 3.39 BLEU on unseen pairs.","key_machinery":"The central object is the representation-alignment identity $\\mathrm{encoder}(l_y, x_a) = \\mathrm{encoder}(l_y, x_b)$, which says that an ideal encoder maps two source sentences with the same target language and parallel semantics into the same target-language subspace. TDO implements this by excluding target tokens from the first $M$ decoder-only layers, then fusing them in the remaining layers; a shared FFN adapter at each boundary compensates for the fact that the target tokens enter later. InstruCL is the second mechanism: for each instance it builds a positive identity pair $(l_y, y, y)$, translating the target sentence to itself, and uses a contrastive loss on the instruction token at layer $1.5N$ so that instruction representations from different source languages but the same target language are pulled together. Together, these mechanisms carry the argument: the similarity score measuring target-language preference, the ratio of similarity to the identity pair over similarity to the source-side pair, rises from about layer 4 and stays high into the second stage.","core_discovery":"The central claim is that the decoder-only architecture is not inherently weak at multilingual translation; it lacks the explicit language-transfer step that encoder-decoder models get for free. In an encoder-decoder model, the encoder output is an intermediate state $H$ that is implicitly aligned across source languages because the decoder must produce the same target sentence regardless of source language, formalized as $\\mathrm{encoder}(l_y, x_a) = \\mathrm{encoder}(l_y, x_b)$. Decoder-only models fuse source and target tokens in one joint masked attention pass, so no such aligned state exists and source representations keep mostly source-language features. The paper's Two-stage Decoder-only architecture splits the stack so that target tokens are withheld from the first $M$ layers, making those layers behave like an encoder, and only then are target tokens fused; two shared feed-forward adapters smooth the boundary. Instruction-level Contrastive Learning adds a loss on the translation-instruction token that pulls each instance's instruction representation toward a same-language identity pair, providing direct supervision for language transfer in the second stage. The authors conclude from their experiments and layer-wise similarity measurements that TDO achieves the alignment of Equation 4 and that this, not added parameters, is what improves zero-shot translation.","pith_inferences":["If the mechanism is confirmed, the same two-stage separation could be applied to the attention patterns of large language models used for translation, since the paper explicitly leaves LLM-scale decoder-only translation as future work; the expected test is whether excluding target tokens in early layers helps LLMs transfer to low-resource pairs.","The paper's similarity metric is computed on averaged sentence representations; a sharper test would track whether individual source tokens, not just sentence averages, land in the target subspace, since token-level alignment is what the cross-attention analogy predicts.","The contrastive loss relies on within-batch negatives, so the gains may depend on batch composition; a testable extension is curriculum or hard-negative mining over languages, which could further reduce the off-target ratio.","Because the first stage is described as an implicit encoder, a natural next question is whether the learned alignment is compositional, that is, whether instruction representations for language pairs never seen together are recognized in the subspace; the OPUS-100 results with 95 languages make this testable."],"forward_implications":["Decoder-only multilingual models can match encoder-decoder models on supervised directions and beat them on zero-shot directions while using about 10% fewer parameters, under the tested settings.","Zero-shot transfer is governed by whether source representations occupy the target-language subspace; measuring layer-wise target-language preference can predict which architecture will transfer better.","The first stage of TDO is about language transfer and the second about linguistic diversity; on small datasets more layers should go to the first stage, on large datasets the balance shifts, and InstruCL matters most in the second stage.","InstruCL also helps encoder-decoder and plain decoder-only models; it is most effective when applied in the middle of the decoder-only second stage, not in the first stage, and it reduces the off-target ratio.","Pre-trained encoder-decoder translation models can be converted into TDO by initializing from the decoder and freezing embeddings, and TDO then outperforms the encoder-decoder initialization on all four metrics in the fine-tuning experiments."],"supporting_citations":[{"why":"Establishes that the encoder of an MNMT model transfers source representations toward the target side, supplies the layer-wise similarity score used in the analysis, and provides the identity-pair proxy for a target language.","marker":"(Qu et al., 2024)"},{"why":"Introduced multilingual NMT with a target language tag prepended to source tokens, the translation-instruction scheme TDO and InstruCL rely on.","marker":"(Johnson et al., 2017)"},{"why":"Empirically shows decoder-only architectures underperform encoder-decoder in MNMT and that supervising source-token representations with language modeling loss does not help.","marker":"(Gao et al., 2022)"},{"why":"Compares LM and encoder-decoder architectures for machine translation scaling and transfer, providing baseline evidence that decoder-only underperforms despite large scale.","marker":"(Zhang et al., 2022)"},{"why":"Shows MNMT is sensitive to language-tag placement, motivating the design of the translation instruction and the contrastive objective on the instruction token.","marker":"(Wu et al., 2021)"},{"why":"Finds that sentence representations translating into different target languages separate as layers increase, supporting the view that transfer happens by aligning representations to the target language.","marker":"(Kudugunta et al., 2019)"},{"why":"Examines knowledge transfer in multilingual MT through a representational lens, providing additional support for measuring language transfer at the representation level.","marker":"(Stap et al., 2023)"},{"why":"Distinguishes causal from prefix decoder-only attention and shows prefix attention is stronger in this setting, guiding the masked-attention variant used in TDO.","marker":"(Raffel et al., 2023)"}],"fun_headline_variants":["Two-stage decoder-only lifts zero-shot translation","Decoder-only gains target-language features via two-stage decoding","Zero-shot MNMT improved by two-stage decoder-only","Splitting decoder stack boosts zero-shot multilingual translation","Two-stage decoding plus contrastive learning hike zero-shot BLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that simply withholding target tokens from the first $M$ layers and adding a shared adapter is enough to make those layers behave like an encoder and align different source languages in the target-language subspace; the paper supports this with similarity measurements but does not prove it formally, so if the gains actually come from the contrastive loss, the adapters, or the extra parameters, the architectural claim weakens.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage decoder-only lifts zero-shot translation","Decoder-only gains target-language features via two-stage decoding","Zero-shot MNMT improved by two-stage decoder-only","Splitting decoder stack boosts zero-shot multilingual translation","Two-stage decoding plus contrastive learning hike zero-shot BLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000668,"raw_usage":{"total_tokens":3075,"prompt_tokens":1000,"completion_tokens":2075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":1998}},"tokens_in":616,"tokens_out":2075,"duration_ms":13956,"temperature":1.0,"reasoning_tokens":1998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:48:37.926690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a TDO variant with the same parameter count but no target-token exclusion, either by letting target tokens attend from the start or by replacing the split with a learned routing that keeps source and target information separate, and check whether the zero-shot gains persist; if they do, the two-stage exclusion is not the cause. A cheaper check is to feed two parallel source sentences in different languages with the same target language through the first $M$ layers and compute the cosine similarity of the resulting representations; if it is not higher than the vanilla decoder-only model's similarity, the claimed alignment of Equation 4 has not occurred.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Compares LM and encoder-decoder architectures for machine translation scaling and transfer, providing baseline evidence that decoder-only underperforms despite large scale."}],"review_version":1}