{"id":"1db488a5-f3f0-40d6-aeb6-b50023788d59","arxiv_id":"2412.17537","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Forgetting in domain-adapted NMT is strongly linked to the adaptation dataset's target vocabulary coverage, and a minimal coverage-based mix-in of generic data prevents most of the loss.","lead":"This paper studies why neural machine translation models lose generic quality when fine-tuned on a specialized domain. It finds that the amount of forgetting tracks how much of the model's general vocabulary appears in the adaptation data, and that adding a small set of generic sentences that cover missing vocabulary prevents most of the loss.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix B's Random #Minimal control undermines the coverage-specific mechanism: the causal claim rests on a post-hoc redefinition of coverage, leaving generic-data quantity as a viable alternative.","rationale":"The paper makes a valuable empirical contribution: a new token-level measure of detrimental vocabulary shift, a systematic comparison of domain heuristics, and a transparent report of a control experiment that limits overclaiming. The qualitative findings in Section 2 (e.g., in-domain tokens appearing in out-of-domain contexts, subword-level triggers) are supported by illustrative examples and are likely robust independent of the causal claim. However, the central causal claim—that target-side generic vocabulary coverage is the driver of forgetting and that Minimal Mix-in works by maximizing that coverage—is not securely established. The strongest evidence for this mechanism is the intervention, but Appendix B's Random #Minimal control, which matches the size of the Minimal Mix-in set but not its full-vocabulary coverage, performs nearly identically. The authors' reinterpretation of coverage in terms of the generic test set is plausible but is introduced after the fact, and the original correlation metric (Table 7) is not this revised construct. Because the sample size is eight domains, a small number of confounded observations, the correlational evidence cannot distinguish coverage from generic-data quantity. A matched control manipulating test-set coverage at fixed size would resolve the ambiguity. The reader's conditional verdict is appropriate; this concern does not change it.","tokens_in":16073,"tokens_out":5389,"duration_ms":49134,"concrete_test":"Run a control arm of the Section 4 experiment in which, for each domain, a random generic subset is selected to have the same size as Minimal Mix-in but with deliberately low coverage of the generic test vocabulary (e.g., exclude generic sentences containing any of the 100 most frequent test-set target tokens). Compare ΔBLEU and ΔCOMET against Minimal Mix-in and Random #Minimal. If the low-coverage random subset still mitigates forgetting to the same degree, then the mechanism is generic-data quantity, not vocabulary coverage; if it mitigates less, the test-set-coverage construct is causal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 presents Minimal Mix-in as evidence that target-side generic vocabulary coverage drives forgetting, claiming it mitigates at least 80% of Random 1:1 forgetting while using less than 10% of the generic data. The control in Appendix B, however, shows that Random #Minimal, a randomly sampled generic set of the same size, reduces forgetting almost as much, despite far lower generic-vocabulary coverage (0.65 vs 0.97 for de-en; 0.70 vs 0.95 for en-ja). The authors reconcile this only by redefining coverage relative to the generic test set, where both methods reach ~0.98–0.99. This post-hoc metric shift means the measured construct—generic vocabulary coverage—is not the causal variable; the intervention's success is equally explained by injecting a modest amount of generic data. The original correlation (Table 7) uses full generic-vocabulary coverage, not the test-set-relative construct that the appendix invokes, and with only 8 domains Kendall's tau cannot separate coverage from dataset size or generic-data quantity. Thus the load-bearing premise, that coverage itself rather than amount of generic mixing or test-set overlap mitigates forgetting, is not secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies catastrophic forgetting in NMT domain adaptation. It introduces a token-level metric, ForgetGenUse, to measure vocabulary-shift forgetting, and applies it to eight domains across two language pairs (de-en and en-ja) by fine-tuning pre-trained models for a fixed 20K steps. The authors examine correlations between forgetting metrics and several domain heuristics, and report that target-side generic vocabulary coverage of the adaptation set is strongly inversely correlated with ΔBLEU (τ=0.7) and ΔCOMET (τ=0.9). They then propose Minimal Mix-in, which adds generic sentences that cover missing target vocabulary, and claim that it mitigates at least 80% of the forgetting reduction achievable by standard 1:1 random mix-in while using less than 10% of the generic data. Appendix B includes a Random #Minimal control matched for the number of generic segments, which appears to perform similarly to Minimal Mix-in despite much lower full-vocabulary coverage.","tokens_in":16281,"tokens_out":5150,"duration_ms":49141,"significance":"If the central claim holds, the paper offers both a diagnostic tool (vocabulary coverage) for predicting forgetting and a practical, low-cost mitigation strategy (coverage-based mix-in). The ForgetGenUse metric is a genuine contribution: it separates vocabulary shift from corpus-level quality decline and is defined independently of the outcome metrics. The paper is also transparent about its limitations and includes a control condition that partially undercuts its own causal interpretation, which is scientifically commendable. However, the correlational evidence is based on a very small number of domains and single runs, and the intervention, as analyzed, does not cleanly isolate coverage from the quantity of generic data. The strength of the contribution therefore depends on whether the authors can reconcile or sharpen the causal claim; the current manuscript is better suited as a correlational study with a plausible mechanism than as a demonstration of why forgetting occurs.","major_comments":[{"comment":"The claim that Minimal Mix-in demonstrates a causal role for target-side generic vocabulary coverage is not secured. The Random #Minimal control in Appendix B, matched for the number of generic segments, reduces forgetting nearly as much as Minimal Mix-in despite much lower full-vocabulary coverage (0.65–0.70 vs 0.95–0.97 for de-en/en-ja). The reconciliation in the appendix redefines coverage relative to the generic test-set vocabulary, whereas Section 3.3 and Table 7 define coverage over the full generic training vocabulary. This metric shift means the intervention does not isolate coverage from generic-data quantity. The paper should either adopt the test-set-relative construct as the primary definition and validate it consistently, or design a comparison that varies coverage while holding mix-in size constant (e.g., random subsets of the same size with different test-set coverage). Without this, the causal 'why' in the title is not established beyond correlation.","section":"§4.1 and Appendix B"},{"comment":"The correlation analysis rests on only eight domain-level observations (five de-en, three en-ja) from single fine-tuning runs. The reported Kendall's tau values (e.g., ΔCOMET vs target coverage τ=0.9, ΔBLEU vs target coverage τ=0.7) are not accompanied by confidence intervals, and may be driven by the extreme points (Kor for de-en, BSD for en-ja). To support the phrase 'strong relationship,' the paper should report bootstrap confidence intervals, run multiple seeds, or add more domains. Without such evidence, the coverage–forgetting link remains suggestive rather than established.","section":"§3.3, Table 7"},{"comment":"The conclusion that detrimental vocabulary shift can occur even when the source sentence has no in-domain triggers is based on a small set of manually inspected examples and fast-align-based alignments with no reported reliability measure. The paper states that these replacements 'tend to occur in very different contexts' but does not provide a systematic quantification across the full set of forgotten tokens. A more systematic analysis, or at least an inter-annotator agreement check on a larger sample, is needed to support this general claim, which is central to the paper's 'what is forgotten' contribution.","section":"§2.3"}],"minor_comments":[{"comment":"The notation 'F orgetGenU se' appears throughout the manuscript with spacing artifacts; it should be rendered consistently as 'ForgetGenUse'.","section":"Throughout"},{"comment":"The column headers of Table 3 are difficult to follow; please clarify that the left-side counts are from the generic model outputs and the right-side counts are from the adapted model, with the frequency in the adaptation dataset shown in parentheses.","section":"Table 3"},{"comment":"The text reads 'Datasets size is recognized' and should be 'Dataset size is recognized'.","section":"§3.1"},{"comment":"The term 'enja' appears where 'en-ja' is intended.","section":"Appendix B"},{"comment":"The paper does not mention whether code or trained models will be released; given the experimental nature, a statement on data/code availability would help reproducibility.","section":"Appendix A / Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a sincere empirical study with a useful new metric and a transparent presentation of a confounding control. The central causal claim requires revision: either the paper should be reframed as a correlational study with a plausible mechanism, or supplemented with a controlled intervention that varies coverage while holding mix-in size fixed. The title's 'why' currently overpromises relative to the evidence. The paper is within scope for a CL journal and, with the coverage construct clarified, could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for the ForgetGenUse metric alone. The paper correctly points out that BLEU/COMET drops conflate quality loss with vocabulary shift, and the metric cleanly isolates the forgetting of previously-correct generic tokens. The qualitative analysis of what replaces those tokens (in-domain synonyms, same-root subwords, even in contextually inappropriate settings) is concrete and well-illustrated. That part is a real contribution, and the paper is honest about its limitations.\n\nThe bigger claim—that forgetting varies with the adaptation set's generic vocabulary coverage, and that a coverage-targeted mix-in recovers most generic performance with little data—is plausible but not nailed down. The correlations rest on 8 domains, no multi-seed runs, no confidence intervals. That is a soft spot but not fatal for a first investigation. What bothers me more is the relationship between the headline correlation and the intervention. The authors say both target coverage and mix-in size matter, but Appendix B shows that a randomly sampled generic set of the same size as Minimal Mix-in reduces forgetting almost as much, despite much lower global coverage (0.65 vs 0.97 de-en). Their reconciliation—recompute coverage relative to the generic test set, where both are ~0.98–0.99—is reasonable but post hoc. It means the measured construct is not the one used in the original correlation, and the intervention's success is equally explained by simply adding a bit of generic data. The Minimal Test vs Random #Minimal Test oracle does favor targeted coverage, which rescues the idea somewhat, but it is still a single experiment.\n\nFor a practitioner the practical takeaway survives: you don't need a huge generic mix-in to blunt forgetting. That is useful. But the paper's central causal story, that coverage itself is the driver, is not secured. It should be presented as a predictive heuristic pending more domains, multi-seed runs, and a cleaner control that varies coverage while holding generic-data amount truly constant.\n\nSend this to peer review. The metric and the qualitative findings deserve a serious referee, even though the causal claim needs reframing and re-validation. No code or data release is a mark against it, but not disqualifying.\n\nI'd bring it to reading group and would probably cite the ForgetGenUse metric in future forgetting work.","headline":"A genuinely useful new metric and a candid empirical study, but the coverage-causality claim is shakier than the abstract suggests and Appendix B undercuts it more than the authors let on.","tokens_in":16801,"tokens_out":2204,"would_cite":true,"duration_ms":22217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Target vocabulary coverage predicts how much domain-adapted MT forgets","keywords":["catastrophic forgetting","neural machine translation","domain adaptation","vocabulary coverage","mixed fine-tuning","vocabulary shift","ForgetGenUse","quality metrics"],"falsifier":"Fix one domain and dataset size while varying only target coverage, for example by deleting rare target tokens or by adding generic sentences that cover only the generic test-set vocabulary; if two such adaptation sets with different global coverage produce the same $\\Delta$BLEU and $\\Delta$COMET on the generic test set, then global coverage is not the driver. Alternatively, rerun the eight adaptations across several seeds and bootstrap the Kendall's $\\tau$ values; if the $\\tau=0.7$ and $\\tau=0.9$ correlations do not survive, the paper's central claim fails.","tokens_in":15840,"feed_emoji":"🧠","tokens_out":10480,"duration_ms":90963,"temperature":0.7,"pith_summary":"Neural machine translation models that are fine-tuned to a new domain do not forget uniformly; this paper claims they forget generic vocabulary roughly in proportion to how little of that vocabulary appears in the adaptation data. It introduces ForgetGenUse, a token-level measure showing that what is forgotten is mostly out-of-domain vocabulary, replaced by adaptation-set tokens that are often semantically related but contextually wrong, sometimes changing meaning (billion to million). Across eight domains in German-English and English-Japanese, forgetting measured by $\\Delta$BLEU correlates with target-side vocabulary coverage at Kendall's $\\tau=0.7$, and $\\Delta$COMET at $\\tau=0.9$, while common heuristics such as dataset size, segment length, likelihood, and vocabulary divergence do not. As an intervention, Minimal Mix-in adds generic sentences only until the target vocabulary is covered; it uses less than 10% of the generic data of a standard 1:1 mix-in while restoring at least 80% of the achievable forgetting mitigation on most domains and preserving in-domain quality. If this is right, practitioners could predict and prevent much of catastrophic forgetting from properties of the adaptation data alone.","feed_headline":"Low target vocabulary coverage predicts NMT catastrophic forgetting","feed_subtitle":"Adaptation data that covers the generic target vocabulary forgets less; a minimal mix-in does most of the work with under 10% of generic…","key_machinery":"The central object is target vocabulary coverage, defined as the proportion of the pre-trained model's target-side subword vocabulary that appears at least once in the adaptation corpus, with source coverage measured separately. The paper's new measure ForgetGenUse counts, per reference segment, how many occurrences of a token the original model produced correctly but the adapted model no longer produces, normalized over the reference counts, so it isolates detrimental vocabulary shift rather than net quality change. Minimal Mix-in is the intervention that operationalizes the mechanism: it adds generic training sentences only until every missing target subword has been seen, thereby setting coverage near 100% while changing nothing about model architecture, adaptation, or inference. Coverage supplies the explanatory variable, ForgetGenUse supplies the vocabulary-level readout, and Minimal Mix-in converts the correlation into a controlled comparison across domains.","core_discovery":"The paper's central discovery is that catastrophic forgetting in domain-adapted NMT is governed by the adaptation dataset's target-side vocabulary coverage: across five de-en and three en-ja domains, forgetting measured by $\\Delta$BLEU correlates with target coverage at $\\tau=0.7$ and $\\Delta$COMET at $\\tau=0.9$, while data size, segment length, generic likelihood, and vocabulary divergence do not significantly explain it. The paper also discovers that the forgetting itself is predominantly a shift away from out-of-domain vocabulary: out-of-domain tokens are forgotten at a higher rate than in-domain tokens, and the replacements are adaptation-set tokens used in inappropriate or even meaning-changing contexts, sometimes triggered by a single subword in the target vocabulary. It shows that this vocabulary shift is not captured by BLEU or COMET, since ForgetGenUseAll does not significantly correlate with either metric. The causal reading is supported by Minimal Mix-in: raising target coverage to 100% with a small generic set reduces forgetting to roughly the level of 1:1 generic mixing, with less than 10% of the generic data for most domains, and without sacrificing in-domain scores.","pith_inferences":["The paper's Appendix B leaves implicit a sharper version of the claim: a random generic set of the same size as Minimal Mix-in reduces forgetting almost as much despite much lower global coverage, because both reach around 99% coverage of the generic test set. The operative quantity may therefore be coverage of the evaluation distribution, not of the full model vocabulary, and this distinction sho","If coverage is the driver, then adaptation-set curation could be inverted: instead of mixing generic data in afterward, one could select in-domain sentences whose target tokens complement the generic vocabulary, potentially preventing forgetting at the data-selection stage.","The limitations note that the authors see the same adaptation-set token preference in LLM-based MT in concurrent work; re-running the coverage correlations and the Minimal Mix-in intervention on a prompted or fine-tuned LLM would show whether the mechanism extends beyond the Transformer NMT setting tested here.","Practical use of the $\\tau=0.7$ and $\\tau=0.9$ correlations should wait for repeated-seed and more-language confirmation, since the statistics come from eight domain points without confidence intervals; a production rule such as 'cover 100% of the target vocabulary' could be validated by bootstrap resampling across many more domains."],"forward_implications":["Adaptation risk becomes measurable in advance: a low target-coverage score on the adaptation data predicts larger drops in generic BLEU and COMET, so practitioners can decide whether mixing in generic data is necessary before fine-tuning.","Coverage-driven mixing can replace a 1:1 generic mix for most domains: using under 10% of the generic data recovers at least 80% of the achievable forgetting mitigation, with in-domain scores close to unmixed fine-tuning.","Quality metrics alone underestimate forgetting: a model can hold or improve BLEU/COMET while replacing correct generic tokens with wrong adaptation-set tokens, so evaluations should check token-level vocabulary shift.","Forgetting can fire without any in-domain trigger in the source, because a single target subword that is frequent in the adaptation set can bias the decoder, so source-side domain detection will miss some cases.","Equalizing target coverage to 100% makes forgetting comparable across very different domains, suggesting that the adaptation data's vocabulary, rather than the domain label itself, is what drives the loss."],"supporting_citations":[{"why":"Introduces catastrophic interference, the phenomenon the paper sets out to explain and measure.","marker":"McCloskey and Cohen (1989)"},{"why":"Defines forgetting in NMT as degradation of general-domain performance, the baseline framing the paper refines.","marker":"Thompson et al. (2019)"},{"why":"Provides mixed fine-tuning, the standard mitigation that the paper's Random 1:1 baseline follows and Minimal Mix-in modifies.","marker":"Chu et al. (2017)"},{"why":"Defines BLEU, one of the two quality metrics used to quantify forgetting on generic test sets.","marker":"Papineni et al. (2002)"},{"why":"Defines COMET, the neural quality metric whose drop correlates most strongly with target coverage.","marker":"Rei et al. (2020)"},{"why":"Supplies the de-en OPUS multi-domain datasets and test splits used for five of the eight adaptation domains.","marker":"Aharoni and Goldberg (2020)"},{"why":"Establishes the 20K-step fine-tuning setup whose forgetting levels the experiments replicate.","marker":"Hasler et al. (2021)"},{"why":"Provides the symmetric alignment tool used to pair original and adapted translation tokens for the replacement analysis.","marker":"Dyer et al. (2013)"},{"why":"Defines BPE subword units, the vocabulary objects over which coverage and ForgetGenUse are computed.","marker":"Sennrich et al. (2016)"},{"why":"Reports that decoder embeddings best predict preserved generic performance, the finding the paper connects to its data-side coverage result.","marker":"Gu and Feng (2020)"}],"fun_headline_variants":["NMT forgetting? It's target vocabulary coverage","Domain adaptation forgets out-of-domain words first","What NMT forgets: out-of-domain vocabulary shift","Target vocab coverage predicts catastrophic forgetting in NMT","Why NMT forgets: target vocabulary coverage, not data size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the observed correlation between forgetting and target vocabulary coverage reflects coverage itself, rather than dataset size, the random composition of the mix-in set, or chance, since the correlations rest on eight domains with no repeated seeds and the paper's own control shows a same-size random generic set performs nearly as well.","fun_headline_variants_meta":{"raw":{"variants":["NMT forgetting? It's target vocabulary coverage","Domain adaptation forgets out-of-domain words first","What NMT forgets: out-of-domain vocabulary shift","Target vocab coverage predicts catastrophic forgetting in NMT","Why NMT forgets: target vocabulary coverage, not data size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000997,"raw_usage":{"total_tokens":4203,"prompt_tokens":906,"completion_tokens":3297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":3220}},"tokens_in":522,"tokens_out":3297,"duration_ms":24813,"temperature":1.0,"reasoning_tokens":3220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:25:56.826608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix one domain and dataset size while varying only target coverage, for example by deleting rare target tokens or by adding generic sentences that cover only the generic test-set vocabulary; if two such adaptation sets with different global coverage produce the same $\\Delta$BLEU and $\\Delta$COMET on the generic test set, then global coverage is not the driver. Alternatively, rerun the eight adaptations across several seeds and bootstrap the Kendall's $\\tau$ values; if the $\\tau=0.7$ and $\\tau=0.9$ correlations do not survive, the paper's central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides mixed fine-tuning, the standard mitigation that the paper's Random 1:1 baseline follows and Minimal Mix-in modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the 20K-step fine-tuning setup whose forgetting levels the experiments replicate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the symmetric alignment tool used to pair original and adapted translation tokens for the replacement analysis."}],"review_version":1}