{"id":"6e6c9780-77f8-4f6b-a9d8-6dfa927b7c5a","arxiv_id":"2412.04003","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Marco-LLM, a Qwen2-based model continually pretrained on 300B multilingual tokens, reports large gains on low-resource language benchmarks and any-to-any translation.","lead":"A team at Alibaba trained Qwen2 models on 300 billion tokens of web, parallel, and synthetic data in 29 languages, then tuned them with multilingual instruction data and preference alignment. The resulting Marco-LLM models report large gains on low-resource language tests and any-to-any translation, though no code, data, or checkpoints are released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on uncontaminated, protocol-matched benchmark comparisons; the paper provides no contamination audit for SFT data (Aya, MetaMathQA, Orca, Belle) and no evaluation prompts/decoding/BLEU details, so the reported multilingual gains are not yet established.","rationale":"The reader's weakest assumption is exactly the condition that must hold for the central claim: benchmark comparisons are valid and uncontaminated. I agree that this is the most load-bearing point. The paper's internal disclaimer in Section 3.1.3 only covers one data category, and Section 4.1.1 lists SFT sources that are known to contain or derive from instruction/benchmark collections; without a leak audit, the post-training gains in MMMLU, Belebele, and AGIEval cannot be trusted. The evaluation protocol is also underspecified: no prompts, decoding settings, or BLEU tokenization are given, which makes the cross-model numbers impossible to reproduce and leaves room for pipeline-driven artifacts. I considered the missing Table?? for any-to-any translation as an alternative concern; it is a genuine omission, but it affects only a subset of the claims, whereas contamination or protocol mismatch threatens every headline margin. The two-stage continual pretraining recipe, the parallel-data ablation, and the learning-rate analysis are useful and presented with enough specificity to be plausible; if the audit and released harness come back clean, the conditional verdict should be upgraded to accept. As it stands, the evidence is not independently verifiable, so the reader's CONDITIONAL verdict is appropriate and unchanged.","tokens_in":43439,"tokens_out":8247,"duration_ms":79491,"concrete_test":"Run an exact and fuzzy contamination audit (13-gram overlap plus embedding similarity) between every test item in MMMLU, AGIEval, Belebele, Flores devtest, XCOPA, XStoryCloze, and TyDiQA and the full 300B continual-pretraining corpus plus the SFT/DPO data described in Sections 4.1.1 and 4.2.2. If any eval item or near-duplicate is found, recompute Tables 6-9 and 11-12 after removing leaked examples; if Marco's margins over Qwen2.5 shrink materially, the headline claim is not established. In parallel, release the exact prompts, decoding parameters, and BLEU tokenizer used for Flores and rerun Marco-72B and Qwen2.5-72B through that same harness to confirm the +10.0 Flores gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline numbers (Tables 6-11) are the entire evidence that 300B tokens of continual pretraining plus multilingual SFT/DPO transfer to low-resource languages. For those numbers to support the claim, every baseline must be scored under the same prompt, decoding, and metric pipeline, and no evaluation instance may appear in training. The paper asserts in Section 3.1.3 only that high-quality knowledge data excludes benchmark training sets; it gives no such guarantee for the SFT/DPO corpora described in Section 4.1.1, which include Aya, MetaMathQA, MathInstruct, Belle, and Orca, nor for parallel/synthetic data. Section 3.3.1 lists benchmark splits and shot counts but no prompts, answer-extraction rules, or BLEU tokenization, so the reader cannot check whether Marco and Qwen2.5 were evaluated identically. A single overlap between, e.g., MMMLU or Belebele test items and Aya/Orca would inflate the post-training margins; an inconsistent BLEU pipeline could explain the suspicious 0.1 score for Llama3-70B on En->Ko in Table 11 and the -8.6 Dutch drop for Qwen2.5-7B in Table 6. Until a contamination audit is run and the eval harness is released, the central claim is conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Marco-LLM is a recipe for extending an existing multilingual base model (Qwen2) to 29 languages by (i) continual pretraining on a curated 300B-token mixture with a two-stage curriculum and a lowered learning rate, and (ii) multilingual SFT and DPO. The paper reports large average gains: Marco-7B reaches 75.5 versus 69.1 for Qwen2.5-7B across 29 languages, and Marco-72B reaches 87.9 versus 85.2 for Qwen2.5-72B; the largest margins are in low-resource languages such as Nepali and Kazakh. It also reports improvements on MMMLU, Belebele, TyDiQA, and Flores, including English-pivot and any-to-any translation.","tokens_in":43768,"tokens_out":5970,"duration_ms":51843,"significance":"If the numbers are trustworthy, this is a practically valuable demonstration that a comparatively small amount (300B tokens) of well-curated multilingual continual pretraining plus multilingual post-training can substantially close the low-resource gap of a strong open model. The two-stage continual-pretraining design and the parallel-data filtering ablation (Section 3.5) are useful contributions, and the data collection pipeline is described in rare detail. However, the evidence is entirely benchmark-based, and the manuscript currently does not supply enough protocol detail or contamination checks to verify the headline margins. No code, model weights, or evaluation harness is promised, so independent verification is not possible from the paper alone.","major_comments":[{"comment":"The evaluation protocol is under-specified. The paper lists datasets, splits, shots, and metrics, but not the exact prompts, answer extraction rules, decoding hyperparameters, or BLEU tokenization/normalization used for any model. Because the central comparison is across different base models with possibly different tokenizers, small protocol differences can move scores by several points. Two values in the tables suggest protocol issues: Llama3-70B obtains 0.1 BLEU on En to Ko in Table 11, and Qwen2.5-7B drops from 80.2 to 71.6 on Dutch in Table 6. Provide the exact harness or a public reference, and report at least one per-language input/output example for each benchmark.","section":"Section 3.3.1, Tables 6 and 11"},{"comment":"No contamination audit is given for the SFT and DPO corpora. The only exclusion claim concerns high-quality knowledge data (Section 3.1.3); no such guarantee is made for the SFT mixture, which explicitly includes Aya collection, MetaMathQA, MathInstruct, Belle, Orca, WMT dev sets, WikiMatrix, translated preference data, and synthetic data. At least one of these sources has been reported to contain benchmark items, and parallel dev sets can overlap with test sets used for WMT16. Run an n-gram or embedding-based overlap analysis between every training component and every evaluation benchmark (MMMLU, AGIEval, CEval, Belebele, TyDiQA, Flores-200 devtest, XCOPA, XStoryCloze, XWinograd) and report the maximum overlap per benchmark.","section":"Section 4.1.1 and Section 3.1.3"},{"comment":"The any-to-any translation section is internally inconsistent. The text refers to Table??, quotes averages of 19.5 and 14.4, while Table 12 reports averages of 19.7 and 14.6. Moreover, Table 12 contains only 7B models, so the abstract's claim of substantial enhancements in any-to-any machine translation tasks is not supported for the 72B model. Fix the reference, correct the numbers, and add the 72B any-to-any results or explicitly limit the claim to the 7B model.","section":"Section 4.1.5, Table 12"},{"comment":"All reported scores are single-run values without error bars or significance tests. While most claimed gains are large, several comparisons are close, for example Marco-72B versus Qwen2.5-72B on Polish in Table 7 (88.2 versus 88.8) and Marco-7B versus Qwen2.5-7B on Thai in Table 6 (72.9 versus 73.7). For such entries the textual claim of consistent outperformance is not supported without repeated evaluations or a paired test over the per-language subtasks.","section":"Tables 6-11"}],"minor_comments":[{"comment":"Model names are used inconsistently: Marco, Marco-7B, Marco-Chat-7B, Marco-72B, and Marco-Chat appear for what seem to be the same models. Please fix the terminology once and for all.","section":"Section 4.1.5 and Appendix A.2"},{"comment":"There are several typos and formatting errors: re-warned should be re-warmed in Section 3.2; Averge in Figure 7; Kazakh(he) in Section 4.2.3; Macro model in Section 4; truction (existing preference dataset) in Section 4.1.5; and the ratio of digits„ in Section 3.1.2.","section":"Throughout"},{"comment":"The relationship between X-MMLU (13 languages, Section 3.3.1) and MMMLU (14 languages, Section 4.1.3) is unclear. Clarify which dataset is used in which table, since both appear in the evaluation suite.","section":"Section 3.3.1 and Section 4.1.3"},{"comment":"The multilingual MT-bench comparison reports win/loss/tie rates from GPT-4o-mini but does not state the number of prompts per language, the judge prompt, the decoding temperature, or the tie-breaking rule. Add these details so the pairwise comparison is reproducible.","section":"Section 4.2.3"},{"comment":"The ablation on parallel data filtering is described as showing significant improvements, but Figure 5 has no numerical values and no statistical test. Report the underlying numbers and the number of evaluation examples used.","section":"Section 3.5, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"This is a systems-style report from an industry group, and the main risk is not the method but the verifiability of the headline claims. The missing evaluation protocol and contamination audit are critical for a paper whose evidence is entirely benchmark-based, and the Table?? reference plus numeric mismatches in the any-to-any section suggest a rushed revision. I would ask for a public evaluation harness or detailed protocol, a contamination audit, and corrected any-to-any results before reconsidering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible, detailed industrial report on continual pretraining of Qwen2 for low-resource languages, with large claimed gains that are internally consistent. But the evaluation section is not complete enough to trust the headline margins, and there is no contamination audit or released artifacts. It deserves a serious referee, not a desk reject, and the main revision requests are concrete.\n\nWhat's genuinely useful here: the paper documents a full recipe — 300B token mixture, two-stage continual pretraining with different LR schedules, parallel data filtering, SFT and DPO — and runs ablations on the small model that show why they chose those hyper-parameters. The per-language numbers across 29 languages are extensive and the gains in Kazakh, Nepali, Azerbaijani etc. are large and consistent across tables. That pattern is real evidence the recipe does something. The parallel-data filtering ablation and the learning-rate analysis are honest and informative. Citations build appropriately on CulturaX, PolyLM, Aya, and the usual continual-pretraining line.\n\nThe soft spots are about verification, not about the idea. The evaluation protocol is under-specified: no prompts, no answer extraction rules, no BLEU tokenization, no decoding parameters. A couple of numbers look like pipeline artifacts (Llama3-70B En→Ko at 0.1 BLEU; Qwen2.5-7B Dutch dropping 8.6 points from Qwen2-7B for no obvious reason). The `Table??` reference for the any-to-any results is unresolved, which is strange for an arXiv submission. More importantly, there is no contamination audit for the SFT and DPO data (Aya, MetaMathQA, Orca, Belle, etc.), and the paper only claims benchmark exclusion for the 'high-quality knowledge data' in pretraining. Since the SFT data includes common instruction sets, a single overlap with MMMLU or Belebele test items would inflate the post-training margins. The MMMLU comparison against GPT-4o-mini and GPT-4 also looks non-standard. No error bars, no significance tests, no model release, no eval harness release. There are also minor editing artifacts (e.g., 'truction', 'Kazakh(he)') that suggest a rushed submission.\n\nNone of this makes the central claim implausible. The directional result — continual pretraining on a diverse multilingual corpus improves low-resource performance — is consistent with prior work and with the internal dynamics shown in Figure 4. I would bet the effect is real. The size of the effect, and the superiority over Qwen2.5 on high-resource languages, is what needs independent confirmation.\n\nWho this is for: people working on multilingual continual pretraining, data curation, or low-resource LLM deployment. A competent referee can extract the recipe and the ablations without trusting every table.\n\nRecommendation: send to peer review, but require the authors to release the evaluation code/prompts and a contamination audit, and fix the unresolved any-to-any table, before acceptance. If they won't release artifacts, the paper should be reframed as a technical report with clearly provisional numbers.","headline":"A credible industrial recipe with large claimed multilingual gains, but the evaluation protocol is under-specified and the results are not yet independently verifiable.","tokens_in":44362,"tokens_out":2824,"would_cite":false,"duration_ms":26482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Marco-LLM claims that a two-stage multilingual continual-pretraining and post-training recipe lifts Qwen2's average score across 29 languages from 69.1 to 75.5 at 7B scale and from 85.2 to 87.9 at 72B scale while also improving direct…","keywords":["multilingual LLM","continual pretraining","low-resource languages","cross-lingual transfer","machine translation","supervised fine-tuning","preference alignment","data curation"],"falsifier":"Score Marco-7B, Marco-72B, and all baselines on a freshly produced parallel test set in the 19 low-resource languages using one fixed prompt, one decoder, and one tokenizer, and also search the training corpora for near-duplicates of Flores, Belebele, and MMMLU items; the central claim fails if the margins vanish under that protocol or if contamination turns up.","tokens_in":43240,"feed_emoji":"🌐","tokens_out":8701,"duration_ms":71254,"temperature":0.7,"pith_summary":"The paper is trying to show that a strong multilingual base model can be extended to underrepresented languages without retraining from scratch. The recipe is continual pretraining on a curated 300B-token corpus covering 29 languages in two stages, then multilingual supervised fine-tuning and preference alignment. The authors report that this lifts the average score across 29 languages from 69.1 to 75.5 at 7B scale and from 85.2 to 87.9 at 72B scale, with the largest gains in low-resource languages such as Nepali and Kazakh, while keeping English and Chinese performance intact. If the reported margins hold, the implication is that data curation plus targeted continual training can narrow the gap between high- and low-resource language capabilities.","feed_headline":"Multilingual retraining lifts 29-language average by 6.4 points","feed_subtitle":"Averaging 75.5 across 29 languages, it beats Qwen2.5-7B by 6.4 points without losing English","key_machinery":"The load-bearing mechanism is a two-stage continual pretraining schedule on a curated 300B-token multilingual corpus, followed by multilingual supervised fine-tuning and direct preference optimization. Stage-I uses 160B tokens at a peak learning rate of 1e-5 with a mixture that keeps 32% English and 17% Chinese to limit catastrophic forgetting; Stage-II uses 140B tokens at 6e-6 and raises the low-resource share from 9% to 15% to push multilingual capability. The corpus work that makes this work is heavy filtering, MinHash deduplication, and the inclusion of parallel data wrapped in diverse translation templates to create cross-lingual alignment. The authors attribute part of the efficiency to Qwen2's 150k-token vocabulary, which keeps low-resource text highly compressible.","core_discovery":"The paper's central claim is that its two-stage continual pretraining and post-training recipe turns Qwen2 into a model whose low-resource language performance is substantially better than the base and than comparable open models, without sacrificing high-resource performance. The supporting evidence is an average of 75.5 for Marco-7B across 29 languages versus 69.1 for Qwen2.5-7B, and 87.9 for Marco-72B versus 85.2 for Qwen2.5-72B, plus large gains on non-English-pivot Flores translation (19.7 versus 14.6 BLEU at 7B scale). It also reports that Marco-72B beats GPT-4 on many MMMLU languages and beats Google Translate on several Flores directions.","pith_inferences":["A natural test of generalizability would be to apply the same two-stage recipe to a different base model with a smaller vocabulary; if the gains shrink, the 150k vocabulary is doing much of the work.","The 5.9% overall data utilization rate suggests the bottleneck is selection rather than crawl size; an explicit tokens-per-language saturation curve would show where extra low-resource data stops paying.","The authors' observed gap between high- and low-resource languages after SFT suggests that more pretraining tokens for low-resource languages, not longer SFT, is the lever for closing the remaining gap; this is an inference from their Figure 7, not a claim they test."],"forward_implications":["A 7B-parameter model built with this recipe can outperform much larger general-purpose models on low-resource language benchmarks.","English, Chinese, and other high-resource languages do not have to be traded away when extending a model to low-resource languages.","Direct translation between non-English pairs becomes practical without an English pivot, which is relevant for language pairs that commercial systems serve poorly.","The parallel-data ablation implies that data filtering is load-bearing at 72B scale, so small-scale pilots may not predict what large models need."],"supporting_citations":[{"why":"It supplies the Qwen2 base model and architecture that Marco-LLM continually pretrains.","marker":"[Yang et al., 2024b]"},{"why":"It provides the statistical filtering thresholds and cleaning pipeline used to curate the multilingual web corpus.","marker":"[Nguyen et al., 2024]"},{"why":"It supplies the deduplication pipeline (MinHash and exact-match) used on the filtered corpus.","marker":"[Penedo et al., 2023]"},{"why":"It is the OPUS source of parallel sentences used to build the cross-lingual training pairs.","marker":"[Tiedemann, 2012]"},{"why":"It is the CCAligned source of parallel and comparable web-document pairs used alongside OPUS.","marker":"[Chaudhary et al., 2019]"},{"why":"It motivates diverse translation templates and multilingual instruction tuning in the SFT stage.","marker":"[Üstün et al., 2024]"},{"why":"It is the Belebele reading-comprehension benchmark that anchors the low-resource language gains.","marker":"[Bandarkar et al., 2024]"},{"why":"It is the Flores benchmark used to measure English-pivot and any-to-any machine translation.","marker":"[Team et al., 2022]"},{"why":"It is the original MMLU benchmark whose multilingual extension MMMLU is a central evaluation.","marker":"[Hendrycks et al., 2021]"},{"why":"It provides the X-MMLU multilingual benchmark used in the continual-pretraining evaluation.","marker":"[Dac Lai et al., 2023]"}],"fun_headline_variants":["Multilingual retraining lifts low-resource languages by 6.4 points","Qwen2 gets a 6.4-point multilingual boost without English loss","Marco-LLM: 6.4-point gain across 29 languages over Qwen2.5","Multilingual training bridges low-resource gaps in LLMs","Cross-lingual LLM beats Qwen2.5 by 6.4 points on 29 languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reported margin depends on the baselines being evaluated under the identical prompt, decoding, and scoring protocol, and on none of the training corpora containing test-set sentences.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual retraining lifts low-resource languages by 6.4 points","Qwen2 gets a 6.4-point multilingual boost without English loss","Marco-LLM: 6.4-point gain across 29 languages over Qwen2.5","Multilingual training bridges low-resource gaps in LLMs","Cross-lingual LLM beats Qwen2.5 by 6.4 points on 29 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2601,"prompt_tokens":962,"completion_tokens":1639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1538}},"tokens_in":578,"tokens_out":1639,"duration_ms":10755,"temperature":1.0,"reasoning_tokens":1538,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:51:40.644660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score Marco-7B, Marco-72B, and all baselines on a freshly produced parallel test set in the 19 low-resource languages using one fixed prompt, one decoder, and one tokenizer, and also search the training corpora for near-duplicates of Flores, Belebele, and MMMLU items; the central claim fails if the margins vanish under that protocol or if contamination turns up.","supporting_citations":[],"review_version":1}