Pith. sign in

REVIEW 4 major objections 5 minor 51 references

Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Decoder-only translation models underperform because they fail to transfer source representations into the target language, and a two-stage design plus instruction contrastive learning fixes this.

desk verdict A worthwhile empirical recipe for decoder-only multilingual translation whose central mechanism claim is contradicted by its own large-scale ablations. read the letter →

arxiv 2412.02101 v1 pith:WZIFDKSD submitted 2024-12-03 cs.CL

Zhi Qu , Yiran Wang , Chenchen Ding , Hideki Tanaka , Masao Utiyama , Taro Watanabe This is my paper · ORCID
classification cs.CL
keywords multilingualneuralmachinetranslationdecoder-onlyarchitecturezero-shotlanguagetransfercontrastivelearninginstructiontwo-stagedecoderrepresentationanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that decoder-only translation models fall behind encoder-decoder models in multilingual settings because they never push source-token representations toward the target language; unlike an encoder, a decoder-only stack has no intermediate state that can be aligned across source languages. To fix this, the authors split the decoder-only stack into two stages: the first $M$ layers see only source tokens plus the target-language tag, and the remaining layers fuse in target tokens as usual, with a shared adapter at the boundary. They add Instruction-level Contrastive Learning, which pulls the representation of the target-language instruction toward an identity pair that acts as a proxy for the target language, to stop the source representations from drifting back to the source side. Across TED-19 and OPUS-100, the resulting TDO model matches encoder-decoder baselines on supervised pairs and exceeds them on zero-shot pairs by up to 3.39 BLEU, 6.99 chrF++, 3.22 BERTScore, and 4.81 COMET, while using fewer parameters. The paper's own representation analysis shows layer-wise similarity scores crossing the target-language threshold earlier in TDO than in a plain decoder-only model.

What carries the argument

The central object is the representation-alignment identity $\mathrm{encoder}(l_y, x_a) = \mathrm{encoder}(l_y, x_b)$, which says that an ideal encoder maps two source sentences with the same target language and parallel semantics into the same target-language subspace. TDO implements this by excluding target tokens from the first $M$ decoder-only layers, then fusing them in the remaining layers; a shared FFN adapter at each boundary compensates for the fact that the target tokens enter later. InstruCL is the second mechanism: for each instance it builds a positive identity pair $(l_y, y, y)$, translating the target sentence to itself, and uses a contrastive loss on the instruction token at layer $1.5N$ so that instruction representations from different source languages but the same target language are pulled together. Together, these mechanisms carry the argument: the similarity score measuring target-language preference, the ratio of similarity to the identity pair over similarity to the source-side pair, rises from about layer 4 and stays high into the second stage.

What would settle it

Train a TDO variant with the same parameter count but no target-token exclusion, either by letting target tokens attend from the start or by replacing the split with a learned routing that keeps source and target information separate, and check whether the zero-shot gains persist; if they do, the two-stage exclusion is not the cause. A cheaper check is to feed two parallel source sentences in different languages with the same target language through the first $M$ layers and compute the cosine similarity of the resulting representations; if it is not higher than the vanilla decoder-only model's similarity, the claimed alignment of Equation 4 has not occurred.

Watch

Extended reading notes

Core claim

The central claim is that the decoder-only architecture is not inherently weak at multilingual translation; it lacks the explicit language-transfer step that encoder-decoder models get for free. In an encoder-decoder model, the encoder output is an intermediate state $H$ that is implicitly aligned across source languages because the decoder must produce the same target sentence regardless of source language, formalized as $\mathrm{encoder}(l_y, x_a) = \mathrm{encoder}(l_y, x_b)$. Decoder-only models fuse source and target tokens in one joint masked attention pass, so no such aligned state exists and source representations keep mostly source-language features. The paper's Two-stage Decoder-only architecture splits the stack so that target tokens are withheld from the first $M$ layers, making those layers behave like an encoder, and only then are target tokens fused; two shared feed-forward adapters smooth the boundary. Instruction-level Contrastive Learning adds a loss on the translation-instruction token that pulls each instance's instruction representation toward a same-language identity pair, providing direct supervision for language transfer in the second stage. The authors conclude from their experiments and layer-wise similarity measurements that TDO achieves the alignment of Equation 4 and that this, not added parameters, is what improves zero-shot translation.

Load-bearing premise

The load-bearing assumption is that simply withholding target tokens from the first $M$ layers and adding a shared adapter is enough to make those layers behave like an encoder and align different source languages in the target-language subspace; the paper supports this with similarity measurements but does not prove it formally, so if the gains actually come from the contrastive loss, the adapters, or the extra parameters, the architectural claim weakens.

Editorial extensions

If this is right

  • Decoder-only multilingual models can match encoder-decoder models on supervised directions and beat them on zero-shot directions while using about 10% fewer parameters, under the tested settings.
  • Zero-shot transfer is governed by whether source representations occupy the target-language subspace; measuring layer-wise target-language preference can predict which architecture will transfer better.
  • The first stage of TDO is about language transfer and the second about linguistic diversity; on small datasets more layers should go to the first stage, on large datasets the balance shifts, and InstruCL matters most in the second stage.
  • InstruCL also helps encoder-decoder and plain decoder-only models; it is most effective when applied in the middle of the decoder-only second stage, not in the first stage, and it reduces the off-target ratio.
  • Pre-trained encoder-decoder translation models can be converted into TDO by initializing from the decoder and freezing embeddings, and TDO then outperforms the encoder-decoder initialization on all four metrics in the fine-tuning experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is confirmed, the same two-stage separation could be applied to the attention patterns of large language models used for translation, since the paper explicitly leaves LLM-scale decoder-only translation as future work; the expected test is whether excluding target tokens in early layers helps LLMs transfer to low-resource pairs.
  • The paper's similarity metric is computed on averaged sentence representations; a sharper test would track whether individual source tokens, not just sentence averages, land in the target subspace, since token-level alignment is what the cross-attention analogy predicts.
  • The contrastive loss relies on within-batch negatives, so the gains may depend on batch composition; a testable extension is curriculum or hard-negative mining over languages, which could further reduce the off-target ratio.
  • Because the first stage is described as an implicit encoder, a natural next question is whether the learned alignment is compositional, that is, whether instruction representations for language pairs never seen together are recognized in the subspace; the OPUS-100 results with 95 languages make this testable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper claims that decoder-only multilingual NMT underperforms because source-token representations are not transferred into the target-language subspace, and proposes a Two-stage Decoder-only (TDO) architecture that excludes target tokens for the first M layers, with adaption modules at the stage boundary, plus Instruction-level Contrastive Learning (InstruCL) on the translation-instruction token. Experiments on TED-19 and OPUS-100, in both from-scratch and fine-tuning settings, report that TDO+InstruCL is competitive with encoder-decoder baselines on supervised directions and improves zero-shot translation by up to 3.39 BLEU, 6.99 chrF++, 3.22 BERTScore, and 4.81 COMET. A layer-wise similarity analysis (Appendix A) is offered as evidence that the proposed methods increase target-language preference in source representations.

Significance. If the causal interpretation is correct, TDO+InstruCL provides a decoder-only architecture that matches or exceeds encoder-decoder MNMT at comparable or slightly lower parameter counts, with practical zero-shot gains. The evaluation is genuinely broad: two corpora, training from scratch and fine-tuning, four automatic metrics plus target-off ratio, ablations over architecture components, M, and the InstruCL layer index, and released code. The main weakness is that the evidence for the architectural mechanism is incomplete and partly contradicted by the paper's own large-scale ablations; moreover, the representation metric used to support the mechanism is not independently validated. These issues affect the central attribution of the gains, so the paper needs substantial revision before the claims can be accepted.

major comments (4)
  1. [§5.3, Table 1 (OPUS-100 blocks)] The ablation does not support the causal claim that the two-stage split repairs decoder-only language transfer. With N=12, d=512, zero-shot BLEU is 6.62 for the prefix decoder-only baseline, 5.32 for TDO(+Prefix), 5.97 for TDO+Adaption, and only 7.93/8.52 after adding InstruCL; the N=6, d=1024 block shows the same pattern (8.15 vs 7.06, then 9.28 with CL). Since Section 7 concludes that the improvements derive from TDO-enhanced language transfer, the OPUS-100 results attribute the zero-shot gains to the contrastive objective rather than to the architecture itself. Please either restrict the architectural claim to TED-19 or provide OPUS-100 representation evidence that TDO improves transfer even when BLEU does not.
  2. [§6.1 and Appendix A] Figure 4's similarity score is the only direct evidence that TDO realizes the alignment in Eq. 4, but Eq. 11 is an average over parallel sentences and is not shown to track translation quality. The score is computed with an SVCCA procedure taken from prior work, and no variance or per-pair distribution is reported. A concrete test would be to correlate the similarity score with held-out BLEU/COMET across the models in Table 1, or to probe Eq. 4 directly by measuring pairwise similarity of the encoder outputs of parallel sentences sharing a target language; as it stands, the representation analysis is interpretive rather than confirmatory.
  3. [§6.2, Figure 5] The M sweep on OPUS-100 shows that zero-shot BLEU is best at M=3 and declines as M grows, while the main experiments use M=N and claim that a larger first stage improves transfer. This non-monotonicity is consistent with the 'second stage benefits linguistic diversity' explanation, but it undercuts the simple statement that excluding target tokens in the first stage is what transfers source representations into the target subspace. Please reconcile the M-sweep behavior with the proposed mechanism and state which M value is recommended for large corpora.
  4. [Tables 1–3 and Appendix I] The paper reports single point estimates without confidence intervals or multiple seeds. Several fine-tuning differences are small, for example Table 2, NLLB-600M: TDO+CL vs TDO zero-shot BLEU is 15.48 vs 15.48, yet the text draws conclusions about InstruCL effectiveness from differences of this size. Please report variance or significance at least for the headline comparisons and for the component ablations in Table 1.
minor comments (5)
  1. [Table 2 caption] The caption says 'Abbreviations align with Table 2,' but it should refer to Table 1.
  2. [§6.1] The label 'InstrucCL' is used once in the enumeration of analyzed models; elsewhere the method is called InstruCL.
  3. [Appendix A] The notation after singular value decomposition is unclear: h_a ∈ R^{d_a} is introduced without defining d_a and d_b, and the relationship between the token-level H and the sentence-level h should be stated more carefully.
  4. [Figure 5 caption] The term 'variation ratio' should be defined explicitly, including the reference model against which the variation is computed and the handling of negative values.
  5. [Appendix I] There is a typo in 'Tabel 3 shows the experimental results,' and the placeholder '-' in the idx column would be clearer as 'n/a'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the TDO and InstruCL results are measured against external benchmarks, and the self-cited similarity analysis is interpretive rather than a fitted prediction.

full rationale

The paper's central comparison is against external benchmarks (TED-19, OPUS-100) with four standard metrics, and the reported gains are direct test-set scores of the proposed architecture, not quantities reconstructed from the inputs. The main load-bearing design—excluding target tokens for M layers and adding InstruCL—is evaluated by ablations against prefix decoder-only and encoder-decoder baselines; the hyperparameters M and the layer index for InstruCL are varied in Sections 6.2 and 6.3, and the headline numbers in Table 1 are not fitted to the test set. The representation analysis (Figures 1b and 4) uses a similarity score from the authors' prior work (Qu et al., 2024), and the identity-pair proxy for the target language in Section 4.2 is also attributed to that work; these self-citations are interpretive supports for the mechanism story, but they are not used as a substitute for the translation evaluation, and the central claim that TDO plus InstruCL improves zero-shot translation does not reduce to the metric by construction. The ablation pattern on OPUS-100, where TDO alone does not improve zero-shot BLEU over the prefix decoder-only baseline, is a legitimate internal-validity concern about which component drives the gain, but it is not circularity: the paper does not define the prediction in terms of the fitted component, and no load-bearing argument reduces to a self-citation chain or to the metric's definition. No circular step satisfying the quotation-and-reduction requirement was found.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The two named free parameters (M, InstruCL layer index) are the main tuned choices. The assumptions are domain-level: the SVCCA similarity measure, the adapter sufficiency, and the identity-pair proxy. No new particles or external entities are introduced. The central claim rests on these assumptions plus the standard transformer training setup.

free parameters (2)
  • M, the number of first-stage layers = M = N (balanced) in main experiments; swept over 1..11 on TED-19 and 3..21 on OPUS-100
    M controls where target tokens are fused. The paper tunes it per dataset (Figure 5) and shows optimal M differs between TED-19 and OPUS-100. The central results in Tables 1 and 2 use M = N, which is a chosen, not derived, hyperparameter.
  • Layer index for InstruCL = 1.5N (main experiments); swept over layers 1..12
    The contrastive loss is applied at a specific layer index. The paper sweeps the index (Figure 6) and reports that the optimal position is the middle layer of the second stage, so this is a hand-tuned hyperparameter.
assumptions (3)
  • domain assumption The representation analysis of Qu et al. (2024), using SVCCA-based similarity, reliably measures target-language vs source-language features. The paper's motivation and its verification of TDO (Figures 1b, 4) depend on this measure.
    Section 3.2 and Appendix A adopt the SVCCA similarity score from Qu et al. (2024). The paper assumes that a score above 0.5 genuinely indicates target-language features and that changes in this score cause translation quality changes. This is an interpretive assumption, as the score is not validated against any ground-truth feature label.
  • domain assumption A shared FFN adapter at the M-layer boundary is sufficient to bridge the representational gap between untransformed source tokens and newly introduced target tokens.
    Section 4.1 introduces the adapter as a design choice. The paper does not test alternatives (e.g., layer norm, learned position embeddings, or no adapter) inside the main tables, so the adequacy of this specific module is assumed.
  • domain assumption The identity pair (ly, y, y) is a valid proxy for the target-language representation, so pulling the instruction token toward it improves transfer.
    Section 4.2 defines the positive instance as the identity pair, citing Qu et al. (2024). The paper does not independently verify that this proxy is better than alternatives (e.g., parallel sentences in the same target language).
invented entities (1)
  • Translation instruction token representation as an alignment anchor
    purpose: InstruCL uses the representation of the language tag ly as the anchor for contrastive learning, pulling it toward the identity-pair representation.
    This is a new use of the instruction token as a training anchor. The paper provides in-domain improvements (Tables 1 and 3) but no external falsifiable handle beyond those test-set scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation." pith.science (2026). https://pith.science/paper/WZIFDKSD

@misc{pith2026241202101,
  author       = {Pith},
  title        = {Pith review of: Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZIFDKSD}},
  note         = {Machine review of arXiv:2412.02101}
}
read the original abstract

Existing multilingual neural machine translation (MNMT) approaches mainly focus on improving models with the encoder-decoder architecture to translate multiple languages. However, decoder-only architecture has been explored less in MNMT due to its underperformance when trained on parallel data solely. In this work, we attribute the issue of the decoder-only architecture to its lack of language transfer capability. Specifically, the decoder-only architecture is insufficient in encoding source tokens with the target language features. We propose dividing the decoding process into two stages so that target tokens are explicitly excluded in the first stage to implicitly boost the transfer capability across languages. Additionally, we impose contrastive learning on translation instructions, resulting in improved performance in zero-shot translation. We conduct experiments on TED-19 and OPUS-100 datasets, considering both training from scratch and fine-tuning scenarios. Experimental results show that, compared to the encoder-decoder architecture, our methods not only perform competitively in supervised translations but also achieve improvements of up to 3.39 BLEU, 6.99 chrF++, 3.22 BERTScore, and 4.81 COMET in zero-shot translations.

Figures

Figures reproduced from arXiv: 2412.02101 by the authors.

Figure 1
Figure 1. Comparison between different architectures in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the encoder-decoder architec [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of proposed methods. Notably, the term, Token, not only means the real token before and after [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration of linguistic preference, which [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Variation in different values of M. The y-axis [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Variation in different layer index of InstruCL. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Averaged BLEU scores in different architec [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 9
Figure 9. Figure 9: Different manners of the masked self-attention [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 12 canonical work pages

  1. [1]

    Ankur Bapna and Orhan Firat. 2019. https://doi.org/10.18653/v1/D19-1165 Simple, scalable adaptation for neural machine translation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538--1548, Hong Kong, China. Associat...

    doi:10.18653/v1/d19-1165 2019

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  3. [3]

    Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, and Baobao Chang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.608 On the off-target problem of zero-shot multilingual neural machine translation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9542--9558, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-acl.608 2023

  4. [4]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116

    arXiv 2019

  5. [5]

    Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. https://doi.org/10.1145/3406095 A survey of multilingual neural machine translation . ACM Comput. Surv., 53(5)

    doi:10.1145/3406095 2020

  6. [6]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    arXiv 2018

  7. [7]

    Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. https://arxiv.org/abs/1905.03197 Unified language model pre-training for natural language understanding and generation . Preprint, arXiv:1905.03197

    arXiv 2019

  8. [8]

    Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. https://arxiv.org/abs/2010.11125 Beyond english-centric multilingual machine translation ...

    arXiv 2020

Show all 51 references
  1. [9]

    Yarman Vural, and Kyunghyun Cho

    Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016. https://doi.org/10.18653/v1/D16-1026 Zero-resource translation with multi-lingual neural machine translation . In Proceedings of the 2016 Conference on Empirical Methods in Natura...

    2016 doi

  2. [10]

    Yingbo Gao, Christian Herold, Zijian Yang, and Hermann Ney. 2022. https://aclanthology.org/2022.aacl-main.43 Is encoder-decoder redundant for neural machine translation? In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lingu...

    2022

  3. [11]

    Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021. Larger-scale transformers for multilingual masked language modeling. arXiv preprint arXiv:2105.00572

    2021 arXiv

  4. [12]

    Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc ' Aurelio Ranzato, Francisco Guzm \'a n, and Angela Fan. 2022. https://doi.org/10.1162/tacl_a_00474 The F lores-101 evaluation benchmark for low-resource and multilingua...

    2022 doi

  5. [13]

    Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2019. https://doi.org/10.18653/v1/P19-1121 Improved zero-shot neural machine translation via ignoring spurious correlations . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pa...

    2019 doi

  6. [14]

    Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. 2021. https://arxiv.org/abs/2011.00362 A survey on contrastive self-supervised learning . Preprint, arXiv:2011.00362

    2021 arXiv

  7. [15]

    Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean

    Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. https://doi.org/10.1162/tacl_a_00065 G oogle ' s multilingual neural machine translat...

    2017 doi

  8. [16]

    Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah Smith. 2021. https://openreview.net/forum?id=KpfasTaLUpq Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation . In International Conference on Learning Representations

    2021

  9. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . Preprint, arXiv:1412.6980

    2017 arXiv

  10. [18]

    Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/v1/D18-2012 S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processin...

    2018 doi

  11. [19]

    Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. https://doi.org/10.18653/v1/D19-1167 Investigating multilingual NMT representations at scale . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationa...

    2019 doi

  12. [20]

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. https://arxiv.org/abs/2001.08210 Multilingual denoising pre-training for neural machine translation . Preprint, arXiv:2001.08210

    2020 arXiv

  13. [21]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. https://doi.org/10.18653/v1/N19-4009 fairseq: A fast, extensible toolkit for sequence modeling . In Proceedings of the 2019 Conference of the North A merican Chap...

    2019 doi

  14. [22]

    Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. https://doi.org/10.18653/v1/2021.acl-long.21 Contrastive learning for many-to-many multilingual neural machine translation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the ...

    2021 doi

  15. [23]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  16. [24]

    Telmo Pires, Robin Schmidt, Yi-Hsiu Liao, and Stephan Peitz. 2023. https://doi.org/10.18653/v1/2023.acl-long.825 Learning language-specific layers for multilingual machine translation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023 doi

  17. [25]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  18. [26]

    Maja Popovi \'c . 2017. https://doi.org/10.18653/v1/W17-4770 chr F ++: words helping character n-grams . In Proceedings of the Second Conference on Machine Translation, pages 612--618, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  19. [27]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  20. [28]

    Zhi Qu, Chenchen Ding, and Taro Watanabe. 2024. https://arxiv.org/abs/2406.08092 Languages transferred within the encoder: On representation transfer in zero-shot multilingual translation . Preprint, arXiv:2406.08092

    2024 arXiv

  21. [29]

    Zhi Qu and Taro Watanabe. 2022. https://aclanthology.org/2022.coling-1.467 Adapting to non-centered languages for zero-shot multilingual translation . In Proceedings of the 29th International Conference on Computational Linguistics, pages 5251--5265, Gyeongju, Republic of Kore...

    2022

  22. [30]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf Improving language understanding by generative pre-training

    2018

  23. [31]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Preprint, arXiv:1910.10683

    2023 arXiv

  24. [32]

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings of the 31st International Conference on Neural Information Processing Systems, N...

    2017

  25. [33]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52 COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Pro...

    2022

  26. [34]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

    2020 doi

  27. [35]

    David Stap, Vlad Niculae, and Christof Monz. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.998 Viewing knowledge transfer in multilingual machine translation through a representational lens . In Findings of the Association for Computational Linguistics: EMNLP 2023, pag...

    2023 doi

  28. [36]

    Shaomu Tan and Christof Monz. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.836 Towards a better understanding of variations in zero-shot neural machine translation performance . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pag...

    2023 doi

  29. [37]

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...

    2022 arXiv

  30. [38]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Neural Informa...

    2017

  31. [39]

    Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. https://arxiv.org/abs/2204.05832 What language model architecture and pretraining objective work best for zero-shot generalization? Preprint, arXiv:2204.05832

    2022 arXiv

  32. [40]

    Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. 2021. https://doi.org/10.18653/v1/2021.findings-acl.264 Language tags matter for zero-shot neural machine translation . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001--3007, Online. A...

    2021 doi

  33. [41]

    Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. https://openreview.net/forum?id=farT6XXntP A paradigm shift in machine translation: Boosting translation performance of large language models . In The Twelfth International Conference on Learning Representations

    2024

  34. [42]

    Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. https://arxiv.org/abs/2305.18098 Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages . Preprint, arXiv:2305.18098

    2023 arXiv

  35. [43]

    Yilin Yang, Akiko Eriguchi, Alexandre Muzio, Prasad Tadepalli, Stefan Lee, and Hany Hassan. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.578 Improving multilingual translation by representation and gradient regularization . In Proceedings of the 2021 Conference on Empiric...

    2021 doi

  36. [44]

    Qi Ye, Sachan Devendra, Felix Matthieu, Padmanabhan Sarguna, and Neubig Graham. 2018. When and why are pre-trained word embeddings useful for neural machine translation. In HLT-NAACL

    2018

  37. [45]

    Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. 2022. https://proceedings.mlr.press/v162/zhang22h.html Examining scaling and transfer of language model architectures for machine translation . In Proceedings of the 39th Inte...

    2022

  38. [46]

    Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020 a . https://doi.org/10.18653/v1/2020.acl-main.148 Improving massively multilingual neural machine translation and zero-shot translation . In Proceedings of the 58th Annual Meeting of the Association for Computati...

    2020 doi

  39. [47]

    Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023. https://arxiv.org/abs/2306.10968 Bayling: Bridging cross-lingual alignment and instruction following through interactive t...

    2023 arXiv

  40. [48]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020 b . https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  41. [49]

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. https://arxiv.org/abs/2304.04675 Multilingual machine translation with large language models: Empirical results and analysis . Preprint, arXiv:2304.04675

    2023 arXiv

  42. [50]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.