{"id":"01d0b16f-5fd1-4173-acc6-fb5ea5be61e3","arxiv_id":"2412.05862","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For medical translation, NLLB-200 3.3B matches or beats 7-8B open-source LLMs in most directions, and fine-tuning it is often as good as fine-tuning much larger LLMs.","lead":"This paper compares open-source language models and a dedicated translation model for medical translation across four language pairs. It finds that the dedicated model, NLLB-200, is often better or comparable, especially for lower-resource languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning comparison is likely confounded: all 7-8B LLMs use QLoRA while NLLB-200 3.3B appears fully fine-tuned, so the claimed superiority of fine-tuned NLLB is not established.","rationale":"The reader's concern about missing significance tests and small Swahili test sets is valid, especially for the zero-shot ordering where decisive gaps are under 1 BLEU point. However, the more load-bearing issue is the apparent asymmetry in fine-tuning methods: if NLLB-200 was fully fine-tuned while all decoder-only models received only QLoRA, the central 'fine-tuned NLLB outperforms fine-tuned Llama-3 8B' claim is not an architecture-level finding but an artifact of adaptation budget. This directly affects the abstract, the conclusions, and the practical recommendation that task-oriented MT models remain necessary even when a domain dataset is available. The zero-shot pillar would survive such a confound, which is why I do not recommend changing the overall CONDITIONAL verdict: the paper's zero-shot results still have indicative value, but the fine-tuning pillar must be re-run with matched adaptation methods or clearly restated as a QLoRA-vs-full-fine-tuning comparison rather than an LLM-vs-NLLB comparison. The reader's significance concern and this confound are complementary, but the confound is the single most load-bearing threat to the strongest claim as phrased.","tokens_in":22507,"tokens_out":5582,"duration_ms":50954,"concrete_test":"Inspect the public fine-tuning scripts to confirm the optimizer and parameter-update strategy for NLLB-200 3.3B versus the 7-8B LLMs. If NLLB was fully fine-tuned while LLMs used QLoRA, re-run the comparison with Llama-3 8B full-parameter fine-tuning (or with LoRA rank matched to NLLB's trainable parameters) on the same medium [m+1] medical data, evaluating on the same EN-FR, EN-PT, EN-SW, and SW-EN test sets with the reported BLEU, chrF++, and COMET/AfriCOMET metrics. If the fine-tuned LLM reaches or exceeds NLLB-200 3.3B, the fine-tuning superiority claim is confounded and should be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two pillars: zero-shot superiority and fine-tuned superiority. The first is plausible, though significance-limited. The second, which supports the 'diminishing returns' takeaway and the abstract's 'still underperform', rests on an asymmetric adaptation setup. In Section 3, QLoRA is introduced specifically for 'autoregressive decoder-only models' (Mistral, Mixtral, Llama-3, Gemma); NLLB-200 3.3B fine-tuning is described separately with no mention of any parameter-efficient method, and the released scripts (github.com/ymoslem/Adaptive-MT-LLM-Fine-tuning) are the only way to confirm. The natural reading is that NLLB-200 received full-weight fine-tuning while all 7-8B LLMs were QLoRA-adapted (one epoch, 4-bit base). QLoRA updates a small fraction of weights; full fine-tuning can absorb a 10k/100k domain corpus much more effectively. If that is what was done, 'fine-tuned NLLB beats fine-tuned Llama/Mistral' conflates architecture with training budget, and the paper's conclusion that encoder-decoder MT models remain necessary after fine-tuning is unsupported. Section 7 discusses data contamination but never mentions this asymmetry, and no multiple seeds or significance tests are reported, so the <1 BLEU gaps (e.g., EN-PT zero-shot NLLB 39.32 vs Llama-3.1 8B 38.77) remain fragile even for the zero-shot pillar.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical comparison of open-source decoder-only LLMs against the encoder-decoder model NLLB-200 3.3B for medical-domain translation in four directions (EN-FR, EN-PT, EN-SW, SW-EN), across zero-shot, retrieval-augmented one-shot, and fine-tuned settings. The authors report that NLLB-200 3.3B outperforms 7-8B LLMs in most zero-shot conditions, that fine-tuned NLLB-200 generally matches or beats fine-tuned 7-8B LLMs, and that very large LLMs (Llama-3.1 405B, DeepSeek-V3 685B) achieve top scores at much higher computational cost. Public code and data-preparation scripts are provided, and the paper includes useful engineering details on QLoRA, packing, quantization, and inference engines.","tokens_in":22779,"tokens_out":6280,"duration_ms":54982,"significance":"If the results hold, the paper provides a useful resource-oriented baseline for practitioners deciding between general-purpose LLMs and task-oriented MT models for domain-specific translation, particularly for medium- and low-resource languages. The study's breadth is a genuine strength: four language directions, several model families, three automatic metrics, and a public repository for both training and inference. However, the ranking conclusions currently rest on point estimates without uncertainty quantification, and the fine-tuning comparison appears to compare full-weight NLLB-200 tuning against QLoRA adaptation of decoder-only LLMs. The core question is important and the evidence is substantial, but the headline claims are not yet fully established as stated.","major_comments":[{"comment":"The central ranking of models is not backed by significance tests, confidence intervals, or multiple seeds. Several decisive gaps are small, e.g. Table 2 shows zero-shot EN-PT BLEU of 39.32 for NLLB-200 3.3B versus 38.77 for Llama-3.1 8B, and Table 1 shows zero-shot EN-FR COMET of 59.32 for NLLB-200 versus 59.36 for Llama-3 8B. With the Swahili medical test set containing only 1,000 segments (§2.2), the reported ordering cannot be considered established. I recommend reporting bootstrap confidence intervals for BLEU, chrF++, and COMET, and, where feasible, results over multiple fine-tuning seeds.","section":"§4, Tables 1–4"},{"comment":"The fine-tuning comparison appears to compare a full-weight fine-tuned NLLB-200 3.3B against QLoRA-adapted 7-8B decoder-only LLMs. Section 3 introduces QLoRA specifically for 'autoregressive decoder-only models' and describes NLLB-200 fine-tuning separately without any parameter-efficient method; Appendix B.2.2 reports a learning-rate search for NLLB without mentioning QLoRA. Since QLoRA updates only a small fraction of weights, the claim that fine-tuned NLLB-200 outperforms fine-tuned Llama-3/Mistral (Section 4.1.2 and the Abstract) conflates architecture with adaptation budget. The authors should either equalize the adaptation setup (e.g. LoRA/QLoRA for NLLB-200, or full fine-tuning for the LLMs), or explicitly report and justify the trainable-parameter counts and training budgets for both sides. Section 7 should acknowledge this asymmetry, which is at least as relevant as the data-contamination concern discussed there.","section":"§3 and Appendix B.2.2"},{"comment":"The headline 'three out of four language directions' is metric-dependent and underspecified. On zero-shot BLEU, NLLB-200 3.3B wins all four directions among the 7-8B models, whereas on COMET, Llama-3 8B edges NLLB-200 in EN-FR (59.36 vs. 59.32, Table 1). The paper reports three metrics with no stated aggregation rule; the abstract and Section 4 should therefore state the metric or predefined aggregation used for the 'three out of four' claim, and should not imply a uniform conclusion across all metrics.","section":"Abstract and §4.1"}],"minor_comments":[{"comment":"The text in Section 4.2 says NLLB-200 3.3B is the top-performing model for SW-EN, but Table 3 shows Llama-3.1 405B (one-shot) with higher BLEU and chrF++ in both the medical and generic domains; the text should qualify the claim as applying to the 8B-class models or explicitly exclude the massive-model class.","section":"§4.2 and Table 3"},{"comment":"The claim that Gemma 7B has the worst zero-shot performance is not supported by Table 1, where BioMistral has lower zero-shot BLEU (24.13 vs. 26.21) and the two models trade places on chrF++; the statement should be tied to a specific metric or otherwise revised.","section":"§4.1.1 and Table 1"},{"comment":"The Swahili row in Table 8 is difficult to parse ('9,000+ 1,000n/a' and '10,415+ 514'); please format the training-domain mixture and context counts as separate numeric columns.","section":"Appendix Table 8"},{"comment":"The packing=True choice can change scores substantially (e.g., EN-SW medical zero-shot BLEU drops from 18.04 to 13.38 in Table 12), so the main text should state that packing was used for all reported fine-tuned results; otherwise readers may not realize the setup differs from the unpacked baseline.","section":"Appendix B.3"},{"comment":"Table captions refer to 'NLLB-200 3.3' while the text and other tables use 'NLLB-200 3.3B'; please make the model name consistent throughout.","section":"Tables 1 and 3"}],"recommendation":"major_revision","confidential_remarks":"The empirical resource-oriented contribution is real and the public release of code and scripts is valuable. The main risk is that the fine-tuning asymmetry and the lack of uncertainty quantification will be seen as undermining the headline; a careful revision with equalized adaptation budgets and significance-aware reporting could make this suitable for publication. I do not see a circularity problem: the comparisons are against external test sets and standard metrics, and the ordering is largely reproducible from the tables for the larger gaps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The zero-shot comparison is worth having, but the fine-tuning conclusion is not established: NLLB-200 appears fully fine-tuned while all 7-8B LLMs get QLoRA, so 'fine-tuned NLLB beats fine-tuned Llama' conflates model architecture with adaptation budget.\n\nWhat is actually new here is a resource-oriented medical MT benchmark covering four directions (EN-FR, EN-PT, EN-SW, SW-EN) with zero-shot, fuzzy-match one-shot prompting, and small/medium fine-tuning sets. The authors evaluate a spread of open LLMs (Mistral, Llama, Gemma, Mixtral, Qwen, DeepSeek) and release data-prep and training code, which makes the measurements reproducible. The finding that NLLB-200 3.3B holds its own against 8B LLMs on zero-shot across most directions is plausible and practically useful.\n\nThe soft spots, in order. First, the fine-tuning asymmetry. Section 3 introduces QLoRA specifically for autoregressive decoder-only models; the NLLB fine-tuning paragraph has no mention of parameter-efficient methods, and the compute notes say QLoRA was used for LLM fine-tuning. So NLLB got full-weight updates while Llama/Mistral/Gemma got one epoch of QLoRA. That is not an even contest. To support the 'specialized MT models remain necessary' takeaway, the authors need to match the adaptation budget, e.g., QLoRA for NLLB or full fine-tuning for LLMs. Second, no significance tests, confidence intervals, or multiple seeds. Several key gaps are under 1 BLEU (EN-PT zero-shot NLLB 39.32 vs Llama-3.1 8B 38.77), and the Swahili medical test set is only 1,000 segments. Third, the headline 'three out of four' depends on the metric: on BLEU, NLLB wins all four zero-shot directions; on COMET, Llama-3 8B edges it on EN-FR by 0.04. That is not a contradiction, but the intro bullet should say which metric.\n\nThe paper does acknowledge data contamination risk in Section 7, and the released code qualifies as a real artifact. But the confound above is load-bearing for the central claim about fine-tuned performance. A serious referee would ask for a corrected comparison and at least a paired bootstrap before accepting the ranking.\n\nThis paper is for practitioners comparing MT options in the medical domain, and for researchers designing domain-adaptation experiments. It deserves peer review, not desk rejection, but it needs major revision. I would take it to a reading group and cite it, with a caveat, as a case study in how adaptation methods can flip the apparent ordering.","headline":"Zero-shot benchmark is useful, but the fine-tuning comparison conflates architecture with adaptation budget and the ranking lacks any significance testing.","tokens_in":23317,"tokens_out":5923,"would_cite":true,"duration_ms":47624,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source 3.3B dedicated translation model outperforms 7-8B general-purpose LLMs on medical-domain translation in three of four language directions, and remains competitive after fine-tuning.","keywords":["machine translation","large language models","domain-specific translation","medical domain","NLLB-200","low-resource languages","fine-tuning","retrieval-augmented translation"],"falsifier":"Run the same zero-shot and fine-tuned NLLB-200 3.3B against Llama-3 8B comparisons with multiple seeds or bootstrap resampling of the test sets, focusing on the sub-one-BLEU gaps such as EN-PT zero-shot NLLB 39.32 vs Llama-3.1 8B 38.77; if the confidence intervals overlap or the ordering flips, the three-of-four claim fails. A second check would use a fresh, never-published medical test set to rule out the data contamination the paper itself flags.","tokens_in":22301,"feed_emoji":"🏥","tokens_out":8573,"duration_ms":67887,"temperature":0.7,"pith_summary":"This paper tests whether general-purpose open-source large language models have made dedicated machine-translation systems unnecessary for specialized, domain-specific translation. Using medical-domain data in English-to-French, English-to-Portuguese, English-to-Swahili, and Swahili-to-English, it compares NLLB-200 3.3B, a task-oriented encoder-decoder MT model, with open decoder-only LLMs from 7B to 685B parameters. The paper claims that NLLB-200 3.3B beats the 7-8B LLMs in zero-shot translation in three of the four directions, and that after fine-tuning on a small or medium medical dataset it matches or exceeds fine-tuned Llama-3 8B. The conclusions matter for practitioners choosing between specialized MT models and general LLMs under data-privacy and compute constraints. The paper concludes that task-oriented MT models remain a core component for high-quality domain-specific translation, while very large LLMs are best used offline for data augmentation and distillation.","feed_headline":"3.3B MT model beats 8B LLMs on medical translation in 3 of 4 pairs","feed_subtitle":"Specialized encoder-decoder MT models still beat 8B general LLMs on medical translation after fine-tuning.","key_machinery":"The mechanism carrying the argument is a controlled comparison protocol built around NLLB-200 3.3B, a 3.3B-parameter encoder-decoder Transformer trained for multilingual translation. Every model is evaluated in the same two inference modes: zero-shot sentence translation, and one-shot translation where the prompt is augmented with a fuzzy match, i.e. the most semantically similar translation pair retrieved from a context dataset by embedding similarity. A subset of models, including Mistral 7B, Llama-3 8B, Gemma 7B, and Mixtral 8x7B, is then fine-tuned on small (10,000 segments) and medium (100,000 segments) medical datasets, with LLMs fine-tuned via parameter-efficient low-rank adaptation and NLLB fine-tuned fully; NLLB is also fine-tuned with fuzzy-match concatenation and teacher-forced target prefixes. Translation quality is measured with BLEU, chrF++, and COMET, with AfriCOMET-MTL substituted for Swahili.","core_discovery":"The central claim, stated as the authors would state it, is that scale and generality are not enough for specialized translation: a 3.3B-parameter encoder-decoder model built for translation, NLLB-200 3.3B, outperforms open decoder-only LLMs in the 7-8B parameter range for zero-shot medical translation in three of the four language pairs tested, and after fine-tuning on small-to-medium medical datasets it achieves results comparable to or better than fine-tuned LLMs such as Llama-3 8B. Larger LLMs such as Llama-3 70B, Llama-3.1 405B, and DeepSeek-V3 685B do reach higher baseline scores, especially with one-shot retrieval-augmented prompting, but at deployment costs the paper argues are prohibitive for most real-time production use. The discovery therefore points in two directions at once: dedicated MT models still earn their place in domain-specific workflows, and the quality of very large LLMs makes them attractive as teachers for distilling domain-specific medium-sized models.","pith_inferences":["If the ranking survives significance testing, the cost-per-quality ratio favors training or fine-tuning task-oriented models of a few billion parameters rather than relying on 8B general LLMs for domain-specific translation; this is an inference about deployment economics the paper gestures at but does not quantify.","The sub-one-BLEU gaps in some decisive comparisons (e.g. EN-PT zero-shot NLLB 39.32 vs Llama-3.1 8B 38.77) could reverse under bootstrap or repeated runs, so the three-of-four claim should be read as provisional until variance is reported.","The paper's own data-contamination warning suggests a direct falsification route: build a fresh, unpublished medical test set and rerun the same comparisons; if 8B LLMs then match or beat NLLB, contamination rather than architecture may explain the gap.","Fine-tuned NLLB-200 3.3B outperforming much larger LLMs on EN-SW hints that, in low-resource directions, task specialization plus moderate scale may beat generalist scale; this suggests testing distillation from 405B-class teachers into medium encoder-decoder models as a concrete next step."],"forward_implications":["Organizations with a small or medium domain-specific parallel dataset can get competitive or better quality by fine-tuning NLLB-200 3.3B than by fine-tuning an 8B-class open LLM, at a fraction of the parameter count.","For zero-shot medical translation, 7-8B open LLMs are not yet a drop-in replacement for dedicated MT models, particularly for medium- and low-resource directions involving Swahili.","One-shot prompting with a fuzzy match consistently improves decoder-only LLMs over zero-shot, so retrieval-augmented prompting is a low-cost lever when fine-tuning is not available.","Very large open LLMs deliver the best baseline quality, but their hardware and latency requirements make them more realistic for offline data augmentation and knowledge distillation than for real-time production translation.","NLLB-200 3.3B's fine-tuned sentence-level variant is enough for Portuguese and Swahili; fuzzy-match concatenation gives no consistent extra gain, so simpler fine-tuning can suffice."],"supporting_citations":[{"why":"Supplies the adaptive MT fine-tuning setup and the fuzzy-match prompt recipe the experiments build on.","marker":"(Moslem et al., 2023a)"},{"why":"Provides the OPUS repository from which the French, Portuguese, and Swahili parallel data are sourced.","marker":"(Tiedemann, 2012)"},{"why":"Contributes TICO-19, the medical-domain COVID-19 corpus used for training and test data.","marker":"(Anastasopoulos et al., 2020)"},{"why":"Defines BLEU, one of the three primary evaluation metrics.","marker":"(Papineni et al., 2002)"},{"why":"Defines chrF++, the character-level metric used alongside BLEU.","marker":"(Popović, 2017)"},{"why":"Defines COMET, the neural metric used for semantic evaluation on French and Portuguese.","marker":"(Rei et al., 2020)"},{"why":"Provides AfriCOMET-MTL, the COMET variant used because COMET does not fully support Swahili.","marker":"(Wang et al., 2024)"},{"why":"Introduces the Llama 3 and Llama 3.1 model family, the main decoder-only baselines compared.","marker":"(Dubey et al., 2024)"},{"why":"Introduces Mistral 7B, a key fine-tuned decoder-only baseline.","marker":"(Jiang et al., 2023)"}],"fun_headline_variants":["3.3B translation model beats 8B LLMs on medical text","Specialized MT beats 8B LLMs in medical translation","LLMs fall short of dedicated MT models on medical translation","NLLB-200 3.3B edges out 8B LLMs in medical translation","Medical translation: 3.3B MT model tops 8B LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central ranking assumes that single-run differences in BLEU, chrF++, and COMET/AfriCOMET scores are meaningful, even though no statistical significance tests or repeated runs are reported and some decisive gaps are under one BLEU point.","fun_headline_variants_meta":{"raw":{"variants":["3.3B translation model beats 8B LLMs on medical text","Specialized MT beats 8B LLMs in medical translation","LLMs fall short of dedicated MT models on medical translation","NLLB-200 3.3B edges out 8B LLMs in medical translation","Medical translation: 3.3B MT model tops 8B LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2826,"prompt_tokens":1008,"completion_tokens":1818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":1732}},"tokens_in":624,"tokens_out":1818,"duration_ms":10667,"temperature":1.0,"reasoning_tokens":1732,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:15:17.196416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same zero-shot and fine-tuned NLLB-200 3.3B against Llama-3 8B comparisons with multiple seeds or bootstrap resampling of the test sets, focusing on the sub-one-BLEU gaps such as EN-PT zero-shot NLLB 39.32 vs Llama-3.1 8B 38.77; if the confidence intervals overlap or the ordering flips, the three-of-four claim fails. A second check would use a fresh, never-published medical test set to rule out the data contamination the paper itself flags.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OPUS repository from which the French, Portuguese, and Swahili parallel data are sourced."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes TICO-19, the medical-domain COVID-19 corpus used for training and test data."}],"review_version":1}