Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Registering: inserting target-language tokens and masking target attention to only see them nearly eliminates the off-target problem in multilingual machine translation and lets a 913M-parameter decoder-only model outperform a…

desk verdict A genuinely useful architecture with a real empirical win; just scope the NLLB comparison to what the evidence actually shows. read the letter →

arxiv 2501.02979 v3 pith:3NOAW2YM submitted 2025-01-06 cs.CL

Zhi Qu , Yiran Wang , Jiannan Mao , Chenchen Ding , Hideki Tanaka , Masao Utiyama , Taro Watanabe This is my paper · ORCID
classification cs.CL
keywords multilingualneuralmachinetranslationoff-targetproblemzero-shotdecoder-onlyarchitectureattentionmaskingregistertokenstargetlanguagespacelow-resource
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the off-target failure of multilingual neural machine translation—decoding into the wrong language—can be nearly eliminated by a purely architectural change: insert one artificial target-language token (a "register") per source token, and mask the decoder so target words can only attend to these registers. The registers are expected to act as containers that carry each source word's meaning into the target language's representational space. On EC-40 this reduces the average off-target ratio to 3.65% (from 26.69% for a vanilla encoder-decoder) and raises spBLEU across 1,640 directions. The paper then pre-trains two decoder-only models, MITRE-466M and MITRE-913M, and reports that the 913M model outperforms NLLB-3.3B on spBLEU and chrF++ and is competitive with commercial large language models despite having far fewer parameters. The authors' conclusion is that a routing bottleneck over register activations, not raw parameter count, is what unlocks competitive multilingual translation in compact models.

What carries the argument

The load-bearing object is the register: an artificial token that carries no semantics of its own, initialized as a duplicate of the target-language tag and inserted once per source token. The load-bearing mechanism is the modified attention mask, which applies the prefix decoder-only scheme (sources attend bidirectionally; registers attend to sources and to each other; target tokens attend only to registers and previous targets). This mask creates a hard information bottleneck: the target language identity and the source content must both pass through the register activations. The paper's analyses of attention weights and representational similarity support the intended operation—registers mirror their positionally aligned source token while moving into the target language space as layers deepen. Registering is the named method, and the one-to-one register-source length ratio is what makes each register a representational container for a specific source token.

What would settle it

Run MITRE-913M and NLLB-3.3B on a held-out test that covers languages outside MITRE's 24 supported languages, for instance the Arabic, Hindi, Bengali, or Somali directions from EC-40, and compare spBLEU, chrF++, and off-target ratio; if the off-target ratio rises sharply or NLLB-3.3B's advantage grows on those directions, the claim that registering strictly constrains generation to the target language space and lets a 913M model beat NLLB-3.3B is contradicted.

Watch

Extended reading notes

Core claim

Registering is a parameter-free change to decoder-only sequence-to-sequence training. A set of artificial tokens, initialized by duplicating the target-language tag, is inserted between the source sequence and the target sequence; the attention mask then forbids target tokens from attending to source tokens directly and instead lets each target token attend only to the register activations and previously generated target tokens. Because each register is positionally aligned with a source token and registers can attend to the whole source, the model must learn to encode source meaning into the target-language register space. The paper argues that this strictly constrains generation to the target language space and thereby nearly resolves the off-target problem. Empirically, the paper reports off-target ratios falling to 3.65% on EC-40, and after pre-training on 9.3 billion sentence pairs across 24 languages, it says MITRE-913M surpasses NLLB-3.3B on spBLEU and chrF++ while remaining competitive with commercial LLMs; the paper also notes that COMET scores, which reward human-like fluency, still slightly favor NLLB-3.3B and GPT-4o mini.

Load-bearing premise

The load-bearing premise is that the 24-language, 194-direction bridge-language evaluation set is representative of general multilingual translation quality: the paper's own Limitations section concedes that NLLB was trained with many more languages and directions plus back-translation and distillation, and on COMET the NLLB-3.3B baseline already edges ahead (85.96 vs. 85.88).

Editorial extensions

If this is right

  • Off-target translation is nearly eliminated: the averaged off-target ratio on EC-40 drops from 26.69% to 3.65% in 24-layer models, and into single digits across both model sizes.
  • A small decoder-only MNMT model can beat a much larger encoder-decoder one: MITRE-913M exceeds NLLB-3.3B by 1.14 spBLEU points on average across the 24-language test set despite using roughly a quarter of the parameters.
  • The benefit scales: registering improves zero-shot spBLEU more consistently than prior off-target remedies as model depth grows, with gains of 5.19 points (12-layer) and 3.62 points (24-layer).
  • Fine-tuning inherits the advantage: MITRE-913M shows the largest gains over its pre-trained checkpoint in both full-parameter and LoRA fine-tuning across the 5-, 25-, and 100-direction scenarios.
  • The mechanism is inspectable: attention and representation analyses show that the highest target-token attention lands on the register aligned with the corresponding source token, so register activations can be read as source meaning placed in target language space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bottleneck interpretation is right, off-target errors are primarily an attention-routing failure rather than a representation gap; a natural testable extension is to add the same register mask to existing instruction-strengthening or contrastive-alignment methods and check whether the gains are additive.
  • The method's "strict constraint" is architectural but not semantic: nothing forces a register to encode all source information losslessly, and the COMET gap the paper reports suggests fluency may be the residual cost, so stress tests on distant language pairs or mixed-code inputs could reveal where register compression loses content.
  • Because registers are just duplicated language tags with a mask change, the same trick could be applied when fine-tuning large language models for translation, potentially constraining output language without extra parameters; this is an implication the paper does not pursue.
  • The paper's data-efficiency argument suggests that with a better data-collection strategy than the bridge-language sampling used here, an even smaller or higher-coverage MITRE-style model might close the COMET gap; the open-sourced models make that directly testable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes "registering," an architecture-level change for decoder-only multilingual NMT: artificial target-language tokens (registers) are inserted between source and target, and the attention mask is modified so target tokens attend only to register activations and previously generated target tokens. The authors claim this bottleneck strictly constrains generation to the target language space, nearly eliminates the off-target problem, and enables a compact model (MITRE-913M, 913M params) to outperform NLLB-3.3B on spBLEU/chrF++ and compete with GPT-4o mini. They report EC-40 benchmark results, an ablation isolating the mask, attention/representation analyses supporting the register mechanism, pre-trained open-source models, and fine-tuning results.

Significance. If the central claim holds, the paper makes a useful contribution: it shows that a simple attention-bottleneck change, rather than additional parameters, can substantially improve zero-shot multilingual translation in decoder-only models. The EC-40 experiments are internally consistent across spBLEU, chrF++, COMET, and off-target ratio, and the ablation in Table 5 demonstrates that the mask, not merely the added tokens, drives the improvement. The open-sourcing of MITRE-466M and MITRE-913M is a concrete practical asset, and the token-level analyses in Section 6.2 provide plausible evidence for the proposed mechanism. The significance is moderated, however, by the fact that the headline comparison to NLLB-3.3B is limited to the 24 languages and 194 directions used to train MITRE and is not corroborated by COMET.

major comments (3)
  1. [Section 5.3, Tables 3 and 13; Abstract; Conclusion] The blanket statement that MITRE-913M "outperforms NLLB-3.3B" is not supported by the evidence as stated. The reported spBLEU advantage of 1.14 points is averaged over 194 directions in the same 24-language pool used to train MITRE, Table 13 shows that COMET actually slightly favors NLLB-3.3B (85.96 vs. 85.88), and all comparisons come from single runs with no variance estimates or significance testing. The abstract and conclusion repeat the unqualified claim. Please either temper the claim to the specific evaluation conditions, add held-out languages/directions, or provide error bars and significance tests before asserting superiority over NLLB-3.3B.
  2. [Section 5.4, Table 4] The statement that "MITRE outperforms NLLB in both scenarios" is contradicted by the 5-direction column of Table 4: NLLB-3.3B LoRA reaches 29.05 spBLEU versus MITRE-913M LoRA at 28.14, and NLLB-1.3B full fine-tuning reaches 28.50 versus MITRE-466M full fine-tuning at 28.19. The claim appears to depend on selecting the largest MITRE model and the full-fine-tuning setting. Please specify the intended comparison protocol (e.g., best model per family, same parameter budget, or same fine-tuning method) and revise the claim accordingly.
  3. [Limitations; Section 5.3] The discussion of why the NLLB comparison is "relatively fair" addresses training-data size, supervision coverage, and techniques such as back-translation and distillation, but it does not address representativeness. Because the evaluation set is drawn from the same 24-language, 194-direction set as MITRE's training, the comparison measures fit to MITRE's chosen language set, not general multilingual ability. The claim in Section 5.3 that MITRE "demonstrates better generalization than NLLB" is therefore unsupported. At minimum, present results separately for language groups not used in MITRE's training, or evaluate held-out directions/languages, before making generalization claims.
minor comments (5)
  1. [Abstract; Section 3.2; Figure 2] The paper says registers have "the same length as the source tokens," but Section 3.2 defines the register length as matching x′, which includes the language tag and possibly the source EOS token. Please make the length definition consistent.
  2. [Abstract] There is a typo in the abstract: "introduceregistering" should read "introduce registering."
  3. [Section 4.3] The off-target ratio is computed with fast-langdetect, which the paper itself notes is not fully accurate. The claim that registering "nearly resolves the off-target problem" should be accompanied by an explicit reminder of this evaluation limitation.
  4. [Section 6.2] The t-SNE analysis and the attention-weight analysis are based on 100 randomly selected instances with no reported random seeds or multiple trials; please state whether these selections are fixed and reproducible.
  5. [References] The GPT-4o mini results are attributed to the GPT-4 technical report (OpenAI, 2024); please cite the specific model documentation or release note for GPT-4o mini if available.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the method is an empirical architecture change, with design choices validated by ablation rather than assumed into the result.

full rationale

The paper's central claims (off-target reduction, spBLEU/chrF++ improvements, competitive performance with NLLB-3.3B and GPT-4o mini) are supported by direct experiments on EC-40 and on a 9.3B-pair pretraining corpus; they are not derived from a fitted parameter renamed as a prediction. The register length and initialization are design choices, and Section 6.1 ablates them (register vs. mask, length ratio) rather than defining them into the outcome. The phrase 'generation is strictly constrained to the target language space' (Section 3.2) is a description of the attention-mask design rather than an empirical result, but the paper does not use that phrasing as evidence for the measured gains; those gains come from the benchmark tables. Self-citations to Qu et al. (2024b) motivate the off-target analysis and provide the TDO baseline, but they are not the evidence for the effectiveness of registering, which is benchmarked against independent baselines and NLLB. The only mild caveat is that the headline comparison to NLLB-3.3B is evaluated on a 24-language, 194-direction set aligned with MITRE's training coverage, and COMET slightly favors NLLB-3.3B; the paper discloses this in its Limitations and Appendix I and softens the GPT-4o mini claim accordingly. This is a scope/representativeness concern, not circularity. Overall, no load-bearing step reduces to its own input by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The register length ratio is a tuned hyperparameter; the bridge language choice is a data-selection decision that directly shapes the comparison. The axioms are mostly domain assumptions from prior literature plus the paper-specific initialization choice. Registers are the sole invented entity, and their semantic mirroring claim currently rests on the paper's own attention and representation analyses.

free parameters (2)
  • register length ratio = 1.0 (len(x')/len(r))
    Selected as optimal in Figure 3 on EC-40 and then fixed for all MITRE models. This is a hyperparameter tuned on the benchmark.
  • bridge languages = de, nl, fr, es, ru, cs, id, ms, ja, zh
    Hand-chosen as the two resource-rich languages per group; determines which directions are supervised and therefore shapes the NLLB comparison.
assumptions (5)
  • domain assumption The off-target problem is a dominant cause of MNMT performance loss.
    Motivates the entire method; cited to Chen et al. 2023 and Tan and Monz 2023 in Section 1.
  • domain assumption A target language tag embedding can define a target language space for registers.
    Core to the mechanism; supported only indirectly by t-SNE and cosine similarity analyses in Section 6.2.
  • ad hoc to paper Registers are initialized by duplicating the target language tag l_y.
    Section 3.2 states this design choice; no ablation compares random or learned register embeddings.
  • domain assumption The bridge-language data collection yields sufficient coverage for zero-shot translation.
    Section 5.1; the paper's Limitations admit fewer directions and fewer samples per direction than NLLB.
  • standard math Standard Transformer training practices from cited literature are sufficient.
    Sections C and D use Fairseq defaults, Adam, label smoothing, and temperature sampling; no new mathematical background is introduced.
invented entities (1)
  • registers (target-language artificial tokens)
    purpose: Positionally-aligned containers that are supposed to carry source token semantics into the target language space; target generation attends only to these tokens.
    The tokens are explicit in the released models, so external testing is possible, but the paper itself provides the only evidence that they specifically mirror source tokens; no outside corroboration is presented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation." pith.science (2026). https://pith.science/paper/3NOAW2YM

@misc{pith2026250102979,
  author       = {Pith},
  title        = {Pith review of: Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3NOAW2YM}},
  note         = {Machine review of arXiv:2501.02979}
}
read the original abstract

The multilingual neural machine translation (MNMT) aims for arbitrary translations across multiple languages. Although MNMT-specific models trained on parallel data offer low costs in training and deployment, their performance consistently lags behind that of large language models (LLMs). In this work, we introduce registering, a novel method that enables a small MNMT-specific model to compete with LLMs. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method advances the state-of-the-art of MNMT. We further pre-train two models, namely MITRE (multilingual translation with registers), by 9.3 billion sentence pairs across 24 languages collected from public corpora. One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: https://github.com/zhiqu22/mitre.

Figures

Figures reproduced from arXiv: 2501.02979 by the authors.

Figure 1
Figure 1. Illustration of the attention view among differ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of registering. The example depicts a translation from English to German. The illustrated [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The spBLEU score variations on EC-40 where [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: 2D distribution of token-level representations [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Token-level attention weights illustration, where the weight of each token is averaged across all heads of a [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Layer-wise cosine similarity among sentence [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The training and validation loss in pre-training [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Three cases of attention analysis on MITRE [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Data distribution of our pre-training dataset. Figure [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: 2D distributions of token-level representations extracted from the different layers of a model trained on [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 27 canonical work pages

  1. [1]

    High Med Low Extra Low #params Method→ ← → ← → ← → ← sup. zero. avg. Enc-dec242M vanilla 50.21 49.61 42.07 44.93 28.62 28.15 32.94 31.17 77.26 52.29 53.72 +CL 55.33 52.42 46.33 47.60 30.00 31.42 34.17 34.40 77.33 56.75 57.93 +LCS 50.27 52.22 43.19 47.73 31.88 30.19 37.09 32.5077.5255.44 56.71 259M +LA VS 52.89 51.57 44.30 46.13 28.81 28.71 32.97 32.16 77....

    work page arXiv

  2. [3]

    Asian Model → ← → ← → ← → ← → ← → ← avg

    English Germanic Romance Slavic Mal.-Polyn. Asian Model → ← → ← → ← → ← → ← → ← avg. M2M483M 81.63 81.40 78.90 77.03 80.48 78.49 79.85 80.31 68.34 72.75 78.67 78.57 77.74615M 81.16 82.15 81.42 79.98 82.00 80.50 81.29 82.43 72.56 74.62 80.02 79.96 79.791.2B 85.93 85.17 84.15 82.87 84.46 83.12 83.87 85.41 75.91 77.83 82.95 82.58 82.66 NLLB615M 86.61 86.76 8...

    work page arXiv

  3. [4]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 21142–21157, Miami, Florida, USA

    Exploring intrinsic language-specific sub- spaces in fine-tuning multilingual neural machine translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 21142–21157, Miami, Florida, USA. As- sociation for Computational Linguistics. Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, and Baobao Chang

    work page 2024

  4. [5]

    drink” points to r6, while “friend

    outperforms GPT-4o mini, but rather that MITRE- 913M outperforms NLLB 3.3B and can compete with GPT-4o mini. J Supporting Experiments for Comparing Enc-dec and Dec-only In Section 3.2, we explain our reason for imple- menting registering in Dec-only. Specifically, Dec- only offers better parameter efficiency than Enc- dec, where the encoder learns the rep...

    work page arXiv

  5. [8]

    The data distribution is visualized at the family level in Figure 9a and at the language level in Fig- ure 9b

    As described in Section 5.1, our data collection strategy results in 9.3 billion trans- lation instances across 194 translation directions. The data distribution is visualized at the family level in Figure 9a and at the language level in Fig- ure 9b. Additionally, Figure 9b highlights which translation directions are supervised and which are zero-shot. No...

    work page 2023

  6. [11]

    In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium

    SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. Xiang Lisa Li and Percy Liang

    work page 2018

  7. [12]

    Preprint, arXiv:2207.04672

    No language left behind: Scal- ing human-centered machine translation. Preprint, arXiv:2207.04672. OpenAI

    arXiv

  8. [13]

    Preprint, arXiv:2303.08774

    Gpt-4 technical report. Preprint, arXiv:2303.08774. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli

    arXiv

Show all 34 references
  1. [14]

    In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota

    fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics....

    2019

  2. [15]

    In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium

    A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computa- tional Linguistics. Zhi Qu, Chenchen Ding, and Taro Watanabe. 2024a. Languages transferred w...

    arXiv

  3. [16]

    In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid)

    COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. Ricardo Rei, Craig Stewart, Ana C Farin...

    2022

  4. [17]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online

    COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Annette Rios, Mathias Müller, and Rico Sennrich

    2020

  5. [18]

    In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 14973–14987, Singapore

    Viewing knowledge transfer in multilingual machine translation through a representational lens. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 14973–14987, Singapore. Association for Computational Linguistics. Zengkui Sun, Yijin Liu, Fandon...

    2023

  6. [19]

    In Findings of the Association for Compu- tational Linguistics ACL 2024 , pages 9201–9214, Bangkok, Thailand and virtual meeting

    LCS: A language converter strategy for zero-shot neural machine trans- lation. In Findings of the Association for Compu- tational Linguistics ACL 2024 , pages 9201–9214, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. Shaomu Tan and Christof Monz

    2024

  7. [20]

    In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13553–13568, Singa- pore

    Towards a better understanding of variations in zero-shot neural ma- chine translation performance. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13553–13568, Singa- pore. Association for Computational Linguistics. Laurens van...

    2023

  8. [21]

    Preprint, arXiv:2307.12966

    Aligning large lan- guage models with human: A survey. Preprint, arXiv:2307.12966. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou

    arXiv

  9. [23]

    Preprint, arXiv:2305.18098

    Bigtranslate: Augmenting large lan- guage models with multilingual translation capability over 100 languages. Preprint, arXiv:2305.18098. Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat

    arXiv

  10. [24]

    In Findings of the Association for Computa- tional Linguistics: NAACL 2024, pages 2765–2781, Mexico City, Mexico

    Multilingual machine translation with large language models: Empirical results and anal- ysis. In Findings of the Association for Computa- tional Linguistics: NAACL 2024, pages 2765–2781, Mexico City, Mexico. Association for Computational Linguistics. A Description of EC-40 EC...

    2023

  11. [26]

    We train on 8 Tesla V100 GPUs, setting memory-efficient- fp16 in Fairseq, with a maximum input of 2048 source tokens per GPU and a gradient accumu- lation of 16 steps

    Note that we include only super- vised translation directions in validation. We train on 8 Tesla V100 GPUs, setting memory-efficient- fp16 in Fairseq, with a maximum input of 2048 source tokens per GPU and a gradient accumu- lation of 16 steps. Both input and output token leng...

    2021

  12. [27]

    We include only supervised transla- tion directions in validation

    with a size of 160,000 by 150 million sentences randomly sampled from the training set. We include only supervised transla- tion directions in validation. Then, we train on 80 Tesla V100 GPUs, setting memory-efficient-fp16 in Fairseq, with a maximum input of 1408 source tokens...

    2017

  13. [28]

    It is important to note that random.sample causes the 5 and 25 directions to be subsets of the 100 directions

    We define three scenarios, including 5, 25, and 100 translation di- rections. It is important to note that random.sample causes the 5 and 25 directions to be subsets of the 100 directions. Specifically, the first 5 and first 25 directions in the 100-direction set correspond to...

    2022

  14. [29]

    F Prompts for GPT Our prompts for GPT series follow: Translating the following sentence from [SRC] to [TGT]: [INPUT]

    As a result, the learn- able parameters of NLLB series are 1.18M, 2.36M, and 4.72M, respectively, and the learnable param- eters of MITRE series are 0.78M and 1.47M, re- spectively. F Prompts for GPT Our prompts for GPT series follow: Translating the following sentence from [S...

    2023

  15. [32]

    Asian Model → ← → ← → ← → ← → ← → ← avg

    English Germanic Romance Slavic Mal.-Polyn. Asian Model → ← → ← → ← → ← → ← → ← avg. M2M483M 50.43 54.36 44.84 46.24 43.96 48.26 43.36 43.06 37.54 41.77 39.83 27.85 42.53615M 49.97 54.74 46.77 48.62 45.34 49.83 44.70 44.42 40.17 43.59 41.04 28.84 44.121.2B 54.80 54.44 49.02 47...

  16. [34]

    English* Germanic Romance Slavic Mal.-Pol

    Given the different trend compared to Tables 3 and 12, we not only mention it in Section 5.3, but also provide an additional discussion in Appendix I. English* Germanic Romance Slavic Mal.-Pol. Asian* Source Family English* Germanic Romance Slavic Mal.-Pol. Asian* Target Famil...

    2023

  17. [2015]

    Preprint, arXiv:1503.02531

    Distilling the knowledge in a neural network. Preprint, arXiv:1503.02531. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    arXiv

  18. [2016]

    In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing, pages 268–277, Austin, Texas

    Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing, pages 268–277, Austin, Texas. Associa- tion for Computational Linguistics. Yingbo Gao, Christian Herold, Zijian...

    2016

  19. [2017]

    Preprint, arXiv:1412.6980

    Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980. Taku Kudo and John Richardson

    arXiv

  20. [2018]

    In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium

    Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics. Akiko Eriguchi, Shufang Xie, Tao Qin, and Hany Has- san

    2018

  21. [2019]

    Preprint, arXiv:1907.05019

    Massively multilingual neural machine translation in the wild: Findings and chal- lenges. Preprint, arXiv:1907.05019. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandh...

    1907 arXiv

  22. [2020]

    In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901

    Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. Mengyu Bu, Shuhao Gu, and Yang Feng

    1901

  23. [2021]

    In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001–3007, Online

    Language tags matter for zero-shot neural machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001–3007, Online. Association for Computa- tional Linguistics. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen...

    2021

  24. [2022]

    Building multilingual machine transla- tion systems that serve arbitrary XY translations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 600–606, Seattle, United States....

    2022

  25. [2023]

    In Findings of the Association for Computational Linguistics: ACL 2023, pages 9542–9558, Toronto, Canada

    On the off-target problem of zero-shot multilingual neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9542–9558, Toronto, Canada. Association for Computational Linguistics. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xi- ao...

    2023

  26. [2024]

    In Find- ings of the Association for Computational Linguistics ACL 2024, pages 10410–10423, Bangkok, Thailand and virtual meeting

    Im- proving multilingual neural machine translation by utilizing semantic and linguistic features. In Find- ings of the Association for Computational Linguistics ACL 2024, pages 10410–10423, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. Zhe ...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.