EuroLLM-9B: Technical Report

Alexandra Birch; Amin Farajian; Andr\'e F. T. Martins; Barry Haddow; Duarte M. Alves; Fran\c{c}ois Yvon; Jo\~ao Alves; Jos\'e G. C. de Souza; Jos\'e Pombal; Manuel Faysse

REVIEW 3 major objections 5 minor 5 cited by

EuroLLM-9B: Technical Report

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EuroLLM-9B claims to be the leading open European-made 9B model, with its instruction-tuned variant topping WMT24++ machine translation by more than three COMET points over Gemma-2-9B-IT in both directions.

desk verdict A solid, artifact-rich systems report whose headline MT lead is plausible but not yet metric-independent; worth peer review with a demand for alternative-metric rescoring. read the letter →

arxiv 2506.04079 v2 pith:FX523G5Y submitted 2025-06-04 cs.CL cs.AIcs.LG

Pedro Henrique Martins , João Alves , Patrick Fernandes , Nuno M. Guerreiro , Ricardo Rei , Amin Farajian , Mateusz Klimaszewski , Duarte M. Alves

show 9 more authors

José Pombal Nicolas Boizard Manuel Faysse Pierre Colombo François Yvon Barry Haddow José G. C. de Souza Alexandra Birch André F. T. Martins

This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords EuroLLM-9BEuropeanlanguagesmultilinguallanguagemodelmachinetranslationdatafilteringsyntheticinstructionBPEtokenizerWMT24++

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This technical report presents EuroLLM-9B, a large language model trained from scratch on roughly 4 trillion tokens across all 24 official European Union languages and 11 additional languages. Its central claim is that both the base model and its instruction-tuned variant are the leading open European-made LLMs of their size, and specifically that EuroLLM-9B-Instruct achieves the best WMT24++ machine translation results among all compared European and non-European models, in both translation directions, by more than three COMET points over the second-best model, Gemma-2-9B-IT. The paper also introduces and releases two reusable pieces: EuroFilter, an AI-based multilingual data filter, and EuroBlocks-Synthetic, a synthetic post-training dataset that strengthens instruction-following in European languages. A sympathetic reader would take this as evidence that an open, European-language-first model at the 9B scale can compete with or beat general-purpose open models on multilingual benchmarks and translation.

What carries the argument

The argument is carried by an integrated data-and-training pipeline. EuroFilter is a multilingual classifier that transfers educational-quality scores from English web text to other languages, trained on translated FineWeb-edu annotations, so lower-resource European languages get quality-filtered web data. EuroBlocks-Synthetic is a post-training dataset built by prompting a strong model with a monolingual document to create an instruction, then using the same document as context to produce an answer in the target language, which expands instruction coverage to less-resourced EU languages. The tokenizer is a byte-fallback BPE with 128,000 pieces, giving token fertility close to that of 256,000-token models while using half the embedding parameters. The three-phase pre-training schedule starts with 50 percent English, reduces English to 32.5 percent while boosting multilingual data, and ends with code and mathematics raised to 23 percent during an annealing-to-zero learning-rate phase, a configuration the paper ties to late-training reasoning gains. Parallel training data is also filtered with COMETKIWI-22 at a threshold of 0.7, and machine translation is scored with COMET-22, the same metric family.

What would settle it

Re-score all WMT24++ outputs with human post-edited judgments or an independent metric family such as chrF or BLEURT, and re-translate the MMLU-Pro and MUSR test sets with a third-party translation provider instead of the authors' own translation model; if EuroLLM-9B-IT's lead over Gemma-2-9B-IT falls below the reported three-plus COMET points, the headline claim would not hold.

Watch

Extended reading notes

Core claim

The discovery, stated on the paper's own terms, is that a 9B-parameter open model trained from scratch with a carefully staged data pipeline can become the most capable open European-made LLM of its size at the time of release. EuroLLM-9B-Instruct scores 84.19 COMET-22 on en-to-xx and 83.94 on xx-to-en translation over the WMT24++ test set, while Gemma-2-9B-IT scores 80.47 and 80.39, a gap of more than three points in both directions. On multilingual general benchmarks averaged across EU languages, the base model posts the best Borda count among European-made pre-trained models and performs comparably to Gemma-2-9B, while the instruct model repeats that pattern and also outperforms all European models on nearly every language-pair translation direction, with Greek-to-English as the sole exception.

Load-bearing premise

The ranking depends on the evaluation being neutral: the metric family used to filter the parallel training data also scores the test translations, and the translation tool used to build parts of the multilingual evaluations also generated post-training data.

Editorial extensions

If this is right

EuroLLM-9B-Instruct, at 9B parameters and about 4 trillion training tokens, is the best open model in its size class for European-language machine translation, ahead of Gemma-2-9B-IT by more than three COMET-22 points in both translation directions.
The base model leads European-made models of similar size on the averaged multilingual benchmarks, with the best Borda count among pre-trained European models and performance comparable to Gemma-2-9B.
The 128k-tokenizer reaches fertility close to 256k-token models while saving half the embedding parameters, which lowers the memory cost of broad language coverage.
The public release of EuroFilter, EuroBlocks-Synthetic, the base model, and the instruction-tuned model lets other teams reproduce or modify the full pipeline instead of treating the 9B model as a black box.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported three-point translation lead may partly reflect a feedback loop: the same COMET metric family used to filter parallel training data is used to score the test translations, and the paper does not quantify how much of the margin would survive a switch to human judgments or an independent metric.
If EuroFilter generalizes beyond the 35 languages tested, the transfer-by-translation recipe could be applied to other low-resource language families; that extension is implicit in the method but not run here.
The sharp late-training gains from the code-and-math-heavy annealing phase suggest the final-phase data mixture is a promising lever for further scaling, yet the paper tests only three candidate mixtures.
The consistent fourth-place TruthfulQA result across languages hints that multilingual instruction tuning may trade away some truthfulness, but the report does not analyze why that happens.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is the kind of technical report that should exist. It trains and releases a 9B model covering 24 EU languages, and it gives away the useful pieces: EuroFilter, the synthetic EuroBlocks-SFT data, the models, and evaluation code. The genuinely new content is EuroFilter—transferring FineWeb-Edu quality scores to non-English data via translation and an mDeBERTa classifier—plus the RAG-style synthetic instruction generation and the annealing-to-zero third phase. That is real engineering value, and the report is transparent about data sources, thresholds, hyperparameters, and ablations.

The evaluation is broader than most: EU20 and Okapi are external translated benchmarks, WMT24++ uses post-edited references, and Borda counts are a sensible way to rank across noisy single-run numbers. The authors also dated the "leading open European-made LLM" claim to release date, which is the right way to phrase it.

Now the soft spots, in order of weight. The headline MT claim—more than 3 COMET-22 points ahead of Gemma-2-9B-IT—rests entirely on COMET-22, and the parallel training data was filtered with COMETKIWI-22, a sibling metric from the same group. Training on data selected by one learned metric family can preferentially improve that same family. WMT24++ also had contributions from this group, which weakens the independence further. I would not call the result fabricated; the model is also strong on non-MT benchmarks, so it is clearly competitive. But the specific 3-point margin should not be treated as metric-independent until someone re-scores outputs with at least one other metric (chrF, BLEURT, or a different learned metric) or a human sample, and reports variance. That is an addressable flaw, not a fatal one.

Second, MMLU-Pro and MUSR translations were produced with Tower v2, the same tool used to synthesize parts of the post-training data. That is a bias risk for those two benchmarks, though the direction and size are unknown and the effect is likely smaller than the MT issue. Third, all scores are single-run point estimates; no significance tests. Given the scale, that is common in this literature, but it matters because the paper does ranking claims.

Finally, the coverage claim (24 EU languages) exceeds evaluation coverage for several languages; the EU20 benchmark itself skips Irish, Maltese, and Croatian, and the per-language tables do not include all languages for every benchmark. The authors should state this gap explicitly.

Verdict: worth a serious referee. I would ask for alternative-metric MT rescoring and a clearer limitation paragraph before acceptance, but the artifacts and the transparency justify the review cycle. I would also cite it if I work on multilingual data filtering.

Referee Report

3 major / 5 minor

Summary. This paper presents EuroLLM-9B, a 9B-parameter multilingual large language model trained from scratch on roughly 4 trillion tokens in three phases, covering all 24 official EU languages plus 11 additional languages. The report describes the tokenizer, architecture, data collection and filtering (including the new EuroFilter classifier and COMETKIWI-22 filtering at threshold 0.7), the three-phase pre-training schedule, and the post-training procedure that produces EuroLLM-9B-Instruct using the EuroBlocks dataset. Evaluation on external EU20 and Okapi multilingual benchmarks and on WMT24++ machine translation shows that EuroLLM-9B is competitive among European-made models, with the instruction-tuned model claiming a COMET-22 margin of more than three points over Gemma-2-9B-IT in both MT directions. The paper also releases the models, EuroFilter, EuroBlocks-Synthetic, and evaluation code.

Significance. The work is a substantial engineering contribution: it openly releases two 9B models, a multilingual data filter, a synthetic post-training dataset, and evaluation code, which will be useful for future European-language LLM research. The use of externally sourced EU20 and Okapi benchmarks for the general multilingual evaluation, and of WMT24++ with post-edited references for MT, gives the evaluation meaningful breadth. However, the headline machine-translation claim rests entirely on COMET-22, a learned metric from the same group that built the COMETKIWI-22 filter applied to the training data, and no independent metric or human evaluation is reported. If the MT margin is confirmed by an independent evaluation, the paper's claim of a leading open European-made LLM in its size class becomes well supported.

major comments (3)

[§2.4.1, §4.1, Table 5] The headline MT claim is scored exclusively with COMET-22, a learned metric developed by the same group (Unbabel/IST) as COMETKIWI-22, which was used to filter the parallel training data at a threshold of 0.7 in §2.4.1. Since the model was trained on data selected by a sibling metric, the 3+ COMET-22 point margin over Gemma-2-9B-IT in Table 5 may partly reflect optimization toward the metric family rather than general translation quality. An independent evaluation (chrF/BLEU, an alternative learned metric such as BLEURT, or a human evaluation on a subset) is needed before the abstract's 'best results' claim can be accepted.
[§4.3, Table 5] All reported scores are single-point estimates without confidence intervals or significance tests. The per-language tables show that the MT margin over the second-best model varies widely (e.g., 1.07 points for Bulgarian xx→en versus 4.91 for en→xx in Table 7), so the averaged 'more than three points' difference in §4.3 is not established as a stable, statistically reliable effect. A bootstrap or paired significance test should be reported for the WMT24++ COMET-22 averages.
[§3.1, §4.1] The MMLU-Pro and MUSR multilingual evaluations use translations produced by Tower v2, the same translation system used to generate parts of the post-training data, including the translated Cosmopedia documents and translated prompt-answer pairs in §3.1. This creates a potential evaluator-alignment risk for those two benchmarks, analogous to the COMET issue. Since these benchmarks contribute to the Borda counts supporting the 'leading open European-made LLM' claim, the paper should either use an independent translation source for these benchmarks or demonstrate that the ranking is robust to the translation tool.

minor comments (5)

[Appendix A.2.4] The captions for the Chinese (ZH) pre-trained and post-trained tables (Tables 54 and 55) say 'Arabic benchmarks'; they should say 'Chinese benchmarks'.
[§2.4.1] The phrase 'TOWER V2-supported languages' should use the consistent spelling 'Tower v2' to match the rest of the paper and the cited reference.
[§4.1] The benchmark name is written inconsistently as 'MMLU-PRO', 'MMLU-PRO', and 'MMLU-Pro'; please unify the capitalization.
[Table 5] The header of Table 5 uses 'WMT24++ WMT24++' without naming the metric; adding 'COMET-22' to the header would make the table self-contained.
[§5.1.1, §5.1.2] The phase-2 and phase-3 mixture ablations are conducted on reduced-scale settings (80B tokens on the 1.7B model for phase 2), and the paper should state explicitly that these are proxy experiments rather than full-scale validations of the final training setup.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark-based claims rest on external test sets and identical scoring protocols; metric/tool overlaps are validity concerns, not by-construction reductions.

full rationale

The report's central quantitative claims are benchmark scores, not quantities derived from the report's own assumptions. The EU-language general-benchmark tables rest on EU20-Benchmarks (openGPT-X) and Okapi translations, which are external; the machine-translation headline rests on WMT24++ scored with COMET-22 under the same protocol for every compared model. Although the parallel-data filter uses COMETKIWI-22 and the MT scorer COMET-22 is from the same research group, filtering on a quality-estimation score does not by construction determine a downstream learned-metric score, and the identical COMET-22 protocol is applied to all baselines, so relative rankings are not forced by the training-filter choice. The use of Tower v2 for MMLU-Pro/MUSR translations and for some post-training data is a possible evaluation-bias risk, but these translations are not EuroLLM outputs, and the primary EU general-benchmark tables are anchored to external EU20/Okapi translations. Self-citations to EuroLLM-1.7B for data-mixture scaling laws and to COMET/Tower as metric/translation tools are not load-bearing: no central result is reduced to a self-cited theorem or to a fitted parameter renamed as a prediction. The identified concerns are best framed as robustness and independence issues rather than circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central ranking claim rests on dataset construction choices (mixtures, filter thresholds, synthetic generation recipes) whose support comes from ablations on the reported benchmarks or from the authors' prior EuroLLM-1.7B report, plus evaluation-tool assumptions (COMET family validity, translation-based benchmark equivalence). No mathematical axioms beyond standard transformer training assumptions are involved.

free parameters (5)

tokenizer vocabulary size = 128,000 pieces
Selected 'through experimentation' (§2.1) to balance fertility against embedding parameter count; no stated criterion or derivation.
phase 1 data mixture (English, other languages, code/math) = 50% / 45% / 5%
Justified by 'careful scaling laws ... described in Martins et al. (2024)', the authors' prior EuroLLM-1.7B report, not shown or summarized in this paper (§2.5).
phase 2 and phase 3 data mixtures = English 32.5%, code/math 7% (phase 2); code/math 23% (phase 3)
Selected by ablations (§5.1.1, §5.1.2) that maximize Arc-Challenge, Hellaswag, and MMLU, the same benchmark families used in the headline evaluation tables.
COMETKIWI-22 parallel-data filter threshold = 0.7 (with Bicleaner 0.5, 0.6 for Portuguese)
Hand-set quality threshold for parallel training data (§2.4.1); the metric family is the same one used for the MT evaluation (COMET-22).
post-training data filtering thresholds = ArmoRM 0.08; complexity/readability score 4; LLM-as-a-judge 9/10
Hand-set thresholds for EuroBlocks quality filtering (§3.1) and for synthetic math data (§2.4.1); the report itself notes reward-model scores are uncalibrated.

assumptions (4)

domain assumption FineWeb-Edu educational scores transfer to non-English languages through translation and an mDeBERTa classifier (EuroFilter).
§2.4.1: the multilingual filter is trained on English FineWeb-Edu scores translated with Tower v2; per-language validity of the transferred scores is not validated.
domain assumption Translated benchmark versions measure the same capability as the English originals.
§4.1: EU20, Okapi, and in-house Tower translations of MMLU-Pro/MUSR are used as multilingual evaluation; translation equivalence for multiple-choice tests is assumed without analysis.
domain assumption COMET-22 scores are a sufficient proxy for translation quality on WMT24++.
§4.1: MT conclusions rely solely on COMET-22, from the same group whose COMETKIWI-22 filtered the training data; no secondary metric or human evaluation is reported.
domain assumption Scaling-law conclusions from EuroLLM-1.7B transfer to the 9B model.
§2.5 defers data-mix justification to Martins et al. (2024) without re-derivation or sensitivity analysis at 9B scale.

invented entities (2)

EuroFilter-v1 independent evidence
purpose: Multilingual web-document quality classifier used to filter pre-training data for non-English languages.
Released at utter-project/EuroFilter-v1 (§1, §2.4.1); as a shipped artifact it is independently usable, though its per-language accuracy is not evaluated in this report.
EuroBlocks-Synthetic independent evidence
purpose: Synthetic multilingual instruction-following dataset (1.95M examples after filtering) for post-training EuroLLM-9B-Instruct.
Released at utter-project/EuroBlocks-SFT-Synthetic-1124 (§3.1); independently usable, though the generation pipeline relies on the authors' own models and judges.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EuroLLM-9B: Technical Report." pith.science (2026). https://pith.science/paper/FX523G5Y

@misc{pith2026250604079,
  author       = {Pith},
  title        = {Pith review of: EuroLLM-9B: Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FX523G5Y}},
  note         = {Machine review of arXiv:2506.04079}
}

read the original abstract

This report presents EuroLLM-9B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11 additional languages. EuroLLM addresses the issue of European languages being underrepresented and underserved in existing open large language models. We provide a comprehensive overview of EuroLLM-9B's development, including tokenizer design, architectural specifications, data filtering, and training procedures. We describe the pre-training data collection and filtering pipeline, including the creation of EuroFilter, an AI-based multilingual filter, as well as the design of EuroBlocks-Synthetic, a novel synthetic dataset for post-training that enhances language coverage for European languages. Evaluation results demonstrate EuroLLM-9B's competitive performance on multilingual benchmarks and machine translation tasks, establishing it as the leading open European-made LLM of its size. To support open research and adoption, we release all major components of this work, including the base and instruction-tuned models, the EuroFilter classifier, and the synthetic post-training dataset.

Figures

Figures reproduced from arXiv: 2506.04079 by the authors.

**Figure 2.** Scheme of the learning rate scheduler. on 400 Nvidia H100 GPUs from the MareNostrum 5 supercomputer, maintaining a constant batch size of 2,800 sequences (approximately 12 million tokens), and employing the Adam optimizer (Kingma & Ba, 2014), with bfloat16 mixed precision. The training process consists of three distinct phases, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Percentage attributed to each data category in the 3 pre-training phases. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Percentage of the training corpus attributed to each language, excluding English and code [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Results on Arc Challenge (left), Hellaswag (middle), and MMLU (right) throughout the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Results on Arc Challenge (left), Hellaswag (middle), and MMLU (right) with different [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Results on Arc Challenge (left), Hellaswag (middle), and MMLU (right) with different 3 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Prompt used to generate synthetic instructions from monolingual web data. [PITH_FULL_IMAGE:figures/full_fig_p055_8.png]

**Figure 9.** Figure 9: Prompt used to generate answers for the synthethic instructions created. The prompt uses [PITH_FULL_IMAGE:figures/full_fig_p056_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
cs.CL 2026-04 unverdicted novelty 7.0 of 10

Gemma 3 27B and Aya Expanse 32B are the strongest multilingual synthetic-data teachers; model scale does not predict effectiveness while prompt diversity, length and response fluency do.
Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIA
cs.CL 2026-07 conditional novelty 6.0 of 10

Portugal's 9B-parameter national language model AMALIA agrees with human annotators on coding moral authority but fails a construct-validity test showing it reaches correct codes via surface correlates rather than the...
From RAG to Agentic RAG for Faithful Islamic Question Answering
cs.CL 2026-01 conditional novelty 6.0 of 10

An agentic retrieval-augmented system that searches the Quran in steps before answering outperforms single-shot retrieval and plain models on a new bilingual Islamic QA benchmark.
Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian
cs.CL 2025-09 conditional novelty 6.0 of 10

Llama-GENBA-10B is a 10B-parameter trilingual model that reports top Bavarian scores among sub-10B models on a machine-translated benchmark the authors built.
A Sovereign, Open-Source Foundation Model for German and English
cs.CL 2026-07 conditional novelty 5.5 of 10

Soofi S 30B-A3B, a hybrid Mamba-MoE model pretrained on ~27T tokens with deliberately up-weighted German, reports the highest English and German aggregate scores among fully open base models in its comparison while ma...

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages · cited by 5 Pith papers

[1]

Directs the model to perform a specific action or task

work page
[2]

Provides context for the desired behavior or output

work page
[3]

May include constraints or specific requirements

work page
[4]

Is typically phrased in natural language

work page
[5]

They are crucial in shaping the AI assistant’s capabilities and ensuring it aligns with the intended use case

Can be general or domain-specific Instructions are used to fine-tune the model’s behavior, helping it understand and respond appropri- ately to various user inputs and scenarios. They are crucial in shaping the AI assistant’s capabilities and ensuring it aligns with the intended use case. Instructions typically fall into one of the categories below:

work page
[6]

Roleplay and Simulation: Inhabiting a character/persona

work page
[7]

Problem Solving: Coding, Mathematical reasoning, Knowledge and reasoning

work page
[8]

Creative Tasks: Creative writing, Brainstorming

work page

Show all 15 references

[9]

Information Processing: Summarization, Extraction, Classification, Translation
[10]

Question Answering: Open-ended, Closed-ended, Multiple Choice
[11]

Text Transformation: Rewriting
[13]

Advisory: Asking for advice
[14]

Domain-Specific Knowledge: Humanity, history, and social studies, Other (specific domains could be added here as needed)
[15]

Summary:

General / Miscellaneous ## Task Description: Considering the definition of an Instruction, analyze the following{language}web document and perform a three-step analysis: <document>{text}<document/ > Firstly, try to give a short summary in{language}of the document. This should ...
[2017]

Instruction

Association for Computational Linguistics. URLhttps://aclanthology.org/ W17-0235. Víctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramírez-Sánchez. Prompsit’s submission to WMT 2018 parallel corpus filtering shared task. InProceedings of the Third Confere...

2018 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Directs the model to perform a specific action or task

work page

[2] [2]

Provides context for the desired behavior or output

work page

[3] [3]

May include constraints or specific requirements

work page

[4] [4]

Is typically phrased in natural language

work page

[5] [5]

They are crucial in shaping the AI assistant’s capabilities and ensuring it aligns with the intended use case

Can be general or domain-specific Instructions are used to fine-tune the model’s behavior, helping it understand and respond appropri- ately to various user inputs and scenarios. They are crucial in shaping the AI assistant’s capabilities and ensuring it aligns with the intended use case. Instructions typically fall into one of the categories below:

work page

[6] [6]

Roleplay and Simulation: Inhabiting a character/persona

work page

[7] [7]

Problem Solving: Coding, Mathematical reasoning, Knowledge and reasoning

work page

[8] [8]

Creative Tasks: Creative writing, Brainstorming

work page