Pith. sign in

REVIEW 4 major objections 8 minor 69 references

Assessing the Role of Data Quality in Training Bilingual Language Models

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Unequal data quality, not data quantity, drives bilingual language model performance gaps, and an English-only quality filter can close most of the gap in French, German, and Chinese.

desk verdict Solid controlled experiments on data quality vs. quantity in bilingual models; the English-only filtering recipe is useful but its cross-lingual generality is validated via translated benchmarks, so treat the 1% gap claim as French-confirmed and promising elsewhere. read the letter →

arxiv 2506.12966 v1 pith:K4UXB3CX submitted 2025-06-15 cs.CL

Skyler Seto , Maartje ter Hoeve , Maureen de Seyssel , David Grangier This is my paper
classification cs.CL
keywords dataqualitybilinguallanguagemodelsmultilingualpretrainingcross-lingualfilteringSentence-BERTclassifiercurationperformancegap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the familiar 'curse of multilinguality' performance drops are driven less by adding a second language than by feeding that language lower-quality data. Its controlled experiments hold data constant by translating a single corpus, and the bilingual gap disappears. The paper then shows that a quality classifier trained only on English labels can select better French, German, and Chinese pretraining data from large raw corpora, improving monolingual performance by 2-4% and shrinking bilingual-monolingual gaps to about 1%. If correct, the practical consequence is that balanced bilingual models can be built through data curation rather than longer training or bigger models.

What carries the argument

The load-bearing mechanism is the multilingual Sentence-BERT embedding space used as a shared geometry: comparable documents in different languages are mapped close together, so a logistic-regression quality classifier trained on English 'high quality' (OpenHermes/ELI5-style) versus 'low quality' (RefinedWeb) labels can rank documents in French, German, and Chinese by their proximity to the English high-quality region. Selecting the 90th percentile of this score yields the training set. The paper also uses translated corpora as a control to hold document content constant while varying language and quality.

What would settle it

Compare the Sentence-BERT embedding clusters of English high- and low-quality documents with those of German and Chinese web data: if the target-language documents do not separate along the same axis, the filter would select a different distribution and the reported gains would not reproduce. A second concrete check is to run the French, German, and Chinese filtered models on native target-language benchmarks such as CMMLU; if the filtered model fails to beat the unfiltered baseline there, the quality-transfer claim collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that data quality, not language identity or dataset size, is a primary cause of bilingual model degradation. Matching quality between languages—by translating one corpus and training monolingual and bilingual models on the same content—removes the performance gap that appears when models are trained on unequal native corpora. Building on this, a lightweight logistic-regression classifier over multilingual Sentence-BERT embeddings, trained only on English high- and low-quality document labels, selects the top decile of raw French, German, and Chinese web data; the filtered data improves target-language performance and brings bilingual models within about 1% of monolingual models across six common-sense-reasoning tasks and MMLU-style knowledge tasks.

Load-bearing premise

The method assumes that the multilingual sentence-embedding space aligns English quality with quality in the target language, so a classifier trained only on English labels ranks French, German, and Chinese documents correctly; the direct alignment evidence is shown for French but not for German or Chinese.

Editorial extensions

If this is right

  • Bilingual performance gaps can be reduced to roughly 1% by curating the non-English data, without extra training steps or larger models.
  • Filtering raw multilingual corpora with an English-trained quality score improves monolingual French, German, and Chinese performance by 2-4% over the unfiltered corpus.
  • A 1.3B model trained on filtered data matches or beats public bilingual models such as CroissantLLM on French benchmarks, despite being trained on far fewer tokens.
  • Filtering acts like training acceleration: the same benchmark score is reached about 5x faster for monolingual and 2x faster for bilingual models.
  • High-quality English data alone, even translated, is not enough for strong non-English performance; native high-quality curation still matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: If the embedding-space transfer holds beyond the three tested languages, one English quality seed could curate pretraining data for the 50+ languages covered by the embedding model, centralizing data curation.
  • Extension: The paper's evaluations for German and Chinese rely on translated English benchmarks; native-language benchmarks would test whether the quality gains reflect genuine target-language understanding rather than translation artifacts.
  • Extension: A direct alignment check before applying the filter in a new language—comparing the embedding clusters of English high-quality data with the target language's raw data—would cheaply predict whether the filter will transfer.
  • Extension: Because the filter selects by document-level embedding similarity, it may systematically prefer text that resembles English-style writing; culturally distinct but high-quality target-language documents could be filtered out.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper studies whether data quality, rather than data quantity alone, accounts for monolingual-vs-bilingual performance gaps in language models. Section 3 presents controlled experiments on 1.3B models trained on a 2x2 grid of data quality (mC4 low vs FineWebEDU high) and language (native English vs translated French/German/Chinese), finding that high-quality bilingual models match monolingual models within about 1% at 200K steps, while gaps of 2-2.5% appear at 30K steps. Section 4 proposes a language-agnostic filter: a logistic regression trained on multilingual Sentence-BERT embeddings of English DCLM-quality labels (OpenHermes/ELI5 vs RefinedWeb), applied to RedPajama2 and FineWeb2. Section 5 reports French experiments showing filtering improves monolingual and bilingual performance and is competitive with public models; Section J extends the filter to German and Chinese with improvements on Core benchmarks translated from English. The abstract claims 2-4% monolingual gains and reduction of bilingual gaps to 1% for French, German, and Chinese.

Significance. If the stated results hold, the paper makes a useful practical contribution: data quality can be curated with English-only labels, and filtering reduces the bilingual penalty. The controlled experiments in Section 3.3 are well designed, particularly the quality-by-language grid and the 30K-step analysis, and the paper is honest in its Limitations section. The comparison with public models in Table 2 and the ablations with FineWeb2 HQ in Appendix I are valuable reference points. However, the cross-lingual claim for German and Chinese rests on translated benchmarks only, and the headline '1% gap' is obtained after post hoc selection of the filter percentile. The core French evidence is credible, but the broader claim is not yet supported.

major comments (4)
  1. [Section J, Tables 8-9; Section 7] The German and Chinese filtering results are evaluated only on Core benchmarks translated from English; no native German or Chinese benchmark is used to evaluate the Section 4 filter. The paper's own Limitations section states that translated evaluations can reward training on English-derived or translated data. The gains in Tables 8-9 are therefore consistent with the filter selecting English-like documents rather than higher target-language quality, and the abstract's 'improves monolingual performance by 2-4% and reduces bilingual model performance gaps to 1%' is not established for German and Chinese. Reporting native benchmark results (e.g., German NLI or regional tasks, Chinese CMMLU for the filtered models) or explicitly limiting the cross-lingual claim to French would address this.
  2. [Section 5.4, Figure 6] The 90% filter percentile is selected after examining evaluation results for the 30th, 60th, 90th, and 95th percentiles, with the 95% run discarded due to 'over-filtering.' Because the same evaluation set is used for both selection and reporting, the 1% gap figure is a post hoc optimum and is likely optimistic. Please report results for all percentiles without selecting one, or pre-specify the threshold on a validation split.
  3. [Appendix D, Section 4] The only evidence for the embedding-space transfer that underlies the language-agnostic filter is a cluster histogram comparing English and translated French data (Figure 8). No alignment check is shown for German or Chinese original corpora before the filter is applied in Section J. Since the filter's applicability to these languages is load-bearing for the paper's main claim, please add a direct alignment evaluation (e.g., cluster overlap, retrieval accuracy, or similarity distributions) for German and Chinese data, or state the transfer assumption explicitly as a limitation.
  4. [Abstract; Sections 3.4 and 5.4] The claim that bilingual performance gaps are reduced to 1% is conditional on 200K training steps: at 30K steps, the same experiments show gaps of 2-2.5% (Figures 2 and 10). The abstract states the 1% figure without this qualifier, which overstates the recipe's effect for shorter training runs that are common in practice. Please qualify the claim with the training regime.
minor comments (8)
  1. [Section 3, first paragraph] Typo: 'trainng' should be 'training.'
  2. [Appendix B.2] Typo: 'Tabl 5' should be 'Table 5.'
  3. [Throughout] The dataset name is spelled inconsistently as 'RedPajama2' and 'Redpajamav2' (e.g., Section 5.1 vs. Appendix B.1); please standardize.
  4. [Figure 5 and Figure 13 captions] Abbreviations such as 'SBert OH FR,' 'FT FR,' 'FT-M FR,' and 'fwe_fr' are not defined in the captions or the surrounding text; please define them at first use.
  5. [Table 4] The token counts for 'RPJ2 (90%)' (260B French, 248B German) are inconsistent with a 10% retention filter applied to the base pools (310B and 297B). Please clarify whether these counts include repeated epochs or whether the percentile notation refers to something else.
  6. [Section 3.2, methodology] The phrase 'totaling 100K steps each' is ambiguous: it is unclear whether the monolingual models train for 100K or 200K steps and whether the bilingual model trains for 200K total steps with 100K per language; please restate.
  7. [Section 5.5, first paragraph] The statement 'up to 15× fewer' than other models lacks a unit; please specify that it refers to training tokens.
  8. [Appendix D, Figure 8] The claim that English and French distributions are 'similar' is based on visual inspection of histograms; please add a quantitative divergence or overlap measure.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: central claims rest on controlled pretraining experiments and external benchmarks, with only minor non-load-bearing self-citations.

full rationale

The paper's derivation chain is empirical rather than definitional. Section 3 establishes the quality/language interaction by training 1.3B models under controlled combinations of mC4 (low DCLM quality) and FineWebEDU (high DCLM quality), using translations to hold content constant; these are genuine controlled experiments, not tautologies. Section 4's filter is trained on English OpenHermes/ELI5 labels and applied to non-English corpora, but the filter parameters are not fitted to the evaluation benchmarks, and the French results include native evaluations (Regional and NLI tasks in Table 25) while Chinese quality effects are shown on native CMMLU in Figure 3. The self-citations (Seto et al. 2024 for the translation system and for consistency with auxiliary-data findings; Grangier et al. 2024a/b for the clustering/classification recipe) are methodological or consistency references and are not load-bearing evidence for the central transfer claim. The paper's Section 7 Limitations explicitly concedes that 'our evaluations languages other than English rely on translated evaluation sets' and that 'certain aspects of the evaluation may lead to improved performance when using English auxiliary or translated data.' This weakens the cross-lingual generality for German and Chinese, but it is an acknowledged external-validity threat rather than a step where the prediction reduces to its inputs by construction. The filter's gain on translated benchmarks could partly reflect selection of English-like documents, but the paper does not define its quality measure in terms of those benchmark scores, and independent native-language evidence exists for French and Chinese. No circular step can be exhibited from the paper's equations or self-citations, so the score is 2 only to mark minor, non-load-bearing self-citations.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three load-bearing choices. First, the definition of quality is inherited from the DCLM classifier training data (RefinedWeb low, OpenHermes/ELI5 high), which is external but paper-selected. Second, the filtering threshold is set to keep roughly 10% of data and the 90th percentile is chosen after inspecting evaluation results. Third, the assumption that English-trained quality labels transfer across languages in a multilingual embedding space is only preliminarily checked for French. The paper does not fit a predictive law with numeric free parameters, but the threshold selection and the binary score threshold in the FineWebEDU ablation are data-dependent choices affecting the headline numbers.

free parameters (3)
  • Quality filter percentile (threshold tau) = 90th percentile (top 10%)
    Chosen to keep enough data for pretraining, following Li et al. (2024); the 90% level is reported as strongest in Section 5.4 after comparing 30/60/90 on the Core eval, so the headline gain is partially selected on the test set.
  • FineWebEDU binary quality threshold = annotation score >= 2
    Ad hoc choice in Appendix H for the textbook-quality ablation; the paper notes score >=3 is easier (82% accuracy) and that score >=2 may admit more low-quality examples.
  • Bilingual data mixing ratio = 50/50 per language
    All bilingual models use equal proportion of data from each language (Section 3.1), which is a design choice, though equal data does not control for equal tokens per language given tokenizer differences.
assumptions (4)
  • domain assumption Quality labels transfer across languages via the multilingual SBERT embedding space.
    Section 4 and Appendix D provide support only for English vs translated French cluster histograms; no embedding-space validation is shown for German or Chinese before filtering.
  • domain assumption Translated evaluation benchmarks measure target-language ability.
    Core and MMLU evaluations in French and German are translations (Section 5.5, Limitations Section 7); the headline 1% gap is on these translated sets, with only limited native eval checks.
  • ad hoc to paper FineWebEDU is 'high quality' and mC4 is 'low quality' for the purposes of the controlled comparisons.
    Section 3.3 defines quality by the DCLM classifier scores (0.023 vs 0.1127) and downstream benchmark performance, but the datasets also differ in domain/educational content, confounding quality with content.
  • ad hoc to paper DCLM classifier training data (RefinedWeb vs OpenHermes/ELI5) defines the target quality distribution.
    Section 4 uses this English seed to train the filter; the definition of quality is external but the choice of seed set is a paper-specific design decision.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing the Role of Data Quality in Training Bilingual Language Models." pith.science (2026). https://pith.science/paper/K4UXB3CX

@misc{pith2026250612966,
  author       = {Pith},
  title        = {Pith review of: Assessing the Role of Data Quality in Training Bilingual Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4UXB3CX}},
  note         = {Machine review of arXiv:2506.12966}
}
read the original abstract

Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more languages can degrade performance for some languages (such as English), while improving others (typically more data constrained languages). In this work, we investigate causes of these inconsistencies by comparing bilingual and monolingual language models. Our analysis reveals that unequal data quality, not just data quantity, is a major driver of performance degradation in bilingual settings. We propose a simple yet effective data filtering strategy to select higher-quality bilingual training data with only high quality English data. Applied to French, German, and Chinese, our approach improves monolingual performance by 2-4% and reduces bilingual model performance gaps to 1%. These results highlight the overlooked importance of data quality in multilingual pretraining and offer a practical recipe for balancing performance.

Figures

Figures reproduced from arXiv: 2506.12966 by the authors.

Figure 1
Figure 1. Performance with varying data quality and language. Models are trained on combinations of mC4 (low) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance with varying data quality and language. Models are trained on combinations of mC4 (low) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance with varying data quality and lan [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: (a) Multilingual Language Representations: Build a universal sentence embedding that maps multilin￾gual data to the same embedding space. (b) English High Quality Training and Multilingual Data Filtering: Classifier is trained on the embeddings of a small amount of hig…
Figure 5
Figure 5. Figure 5: Quality vs. accuracy on Core tasks for filtered [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Bilingual vs. Monolingual performance on [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Performance at intermediate checkpoints during training for 1.3B models for Core EN and FR benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Cluster histograms for distribution of different [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Performance with varying data quality and language. Models are trained on combinations of mC4 (low) [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Performance with varying data quality and language. Models are trained on combinations of mC4 (low) [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Quality vs. Zero-shot Accuracy on Core tasks for filtered RedPajama2 in French (SBert OH FR) compared with TransWebEDU (fwe_de) and base Red￾Pajamav2 in French (rpj_fr) after 200K steps. steps on TransWebEDU (German) on translated benchmarks consistent with our experi…
Figure 15
Figure 15. Figure 15: We find that the DCLM classifier data performs better, while the FineWebEDU data attains the same performance as the base model. There are a few possibilities that we leave to future work: (i) The original FineWebEDU classifier scores be￾tween 0 and 5. When training a…
Figure 14
Figure 14. Figure 14: Continued pretraining experiments for 1.3B [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 16
Figure 16. Figure 16: Monolingual model performance comparing filtering on the Core FR benchmarks for various model sizes. on the same data pools for both the base distribution of RedPajamav2 FR and filtered version at 90% filtering. Note that the 2.7B model has twice the context length an…
Figure 17
Figure 17. Figure 17: Bilingual model performance comparing filtering on the Core EN and FR benchmarks for various model sizes. Findings: For monolingual models, we see 2% improvement for 350M and 1.3B, and 1% for the 2.7B model. With more data, it’s possible to see greater improvements on…
Figure 18
Figure 18. Figure 18: Performance at intermediate checkpoints during training for different model sizes models for Core FR [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 16 canonical work pages

  1. [1]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. 2024. A survey on data selection for language models. arXiv preprint arXiv:2402.16827

    arXiv 2024

  2. [2]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  3. [3]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432--7439

    2020

  4. [4]

    Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A Smith, and Luke Zettlemoyer. 2024. Breaking the curse of multilinguality with cross-lingual expert language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10822--10837

    work page 2024

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  6. [6]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

    arXiv 2023

  7. [7]

    Tyler Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024. When is multilinguality a curse? language modeling for 250 high-and low-resource languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074--4096

    work page 2024

  8. [8]

    Tyler A Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K Bergen. 2023. When is multilinguality a curse? language modeling for 250 high-and low-resource languages. arXiv preprint arXiv:2311.09205

    arXiv 2023

Show all 69 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  2. [10]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, \'E douard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual...

    2020

  3. [11]

    Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. Advances in neural information processing systems, 32

    2019

  4. [12]

    Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053

    2018 arXiv

  5. [13]

    Wietse De Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch bert model. arXiv preprint arXiv:1912.09582

    2019 arXiv

  6. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...

    2019

  7. [15]

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2023. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393

    2023 arXiv

  8. [16]

    Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, Ant \'o nio Loison, Duarte Alves, Caio Corro, Nicolas Boizard, Jo \ a o Alves, Ricardo Rei, Pedro Martins, et al. 2024. Croissantllm: A truly bilingual french-english language model. arXiv preprint arXiv:2402.00786

    2024 arXiv

  9. [17]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  10. [18]

    David Grangier, Simin Fan, Skyler Seto, and Pierre Ablin. 2024 a . Task-adaptive pretrained language models via clustered-importance sampling. arXiv preprint arXiv:2410.03735

    2024 arXiv

  11. [19]

    David Grangier, Angelos Katharopoulos, Pierre Ablin, and Awni Hannun. 2024 b . Specialized language models with cheap inference from limited domain data. arXiv preprint arXiv:2402.01093

    2024 arXiv

  12. [20]

    Nika Haghtalab, Michael Jordan, and Eric Zhao. 2022. On-demand sampling: Learning optimally from multiple distributions. Advances in Neural Information Processing Systems, 35:406--419

    2022

  13. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  14. [22]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  15. [23]

    Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291

    2019 arXiv

  16. [24]

    Hang Le, Lo \" c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabb \'e , Laurent Besacier, and Didier Schwab. 2019. Flaubert: Unsupervised language model pre-training for french. arXiv preprint arXiv:1912.05372

    2019 arXiv

  17. [25]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. 2024. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794

    2024 arXiv

  18. [26]

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463

    2023 arXiv

  19. [27]

    Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380

    2024 arXiv

  20. [28]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Su \'a rez, Yoann Dupont, Laurent Romary, \'E ric Villemonte de La Clergerie, Djam \'e Seddah, and Beno \^ t Sagot. 2019. Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894

    2019 arXiv

  21. [29]

    Pedro Henrique Martins, Patrick Fernandes, Jo \ a o Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, Jos \'e Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, et al. 2024. Eurollm: Multilingual language models for europe. arXiv preprint arXiv:2409.16235

    2024 arXiv

  22. [30]

    Bettina Messmer, Vinko Sabol c ec, and Martin Jaggi. 2025. Enhancing multilingual llm pretraining with model-based data selection. arXiv preprint arXiv:2502.10361

    2025

  23. [31]

    Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning--based text classification: a comprehensive review. ACM computing surveys (CSUR), 54(3):1--40

    2021

  24. [32]

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36

    2024

  25. [33]

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2024. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint Internati...

    2024

  26. [34]

    OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774

    2023 arXiv

  27. [35]

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024 a . The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557

    2024 arXiv

  28. [36]

    Guilherme Penedo, Hynek Kydl cek, Vinko Sabolcec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. 2024 b . Fineweb2: A sparkling update with 1000s of languages, december 2024b. URL https://huggingface. co/datasets/HuggingFaceFW/fineweb-2

    2024

  29. [37]

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. ar...

    2023 arXiv

  30. [38]

    Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computati...

    2022

  31. [39]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  32. [40]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

    2021 arXiv

  33. [41]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485--5551

    2020

  34. [42]

    Nils Reimers and Iryna Gurevych. 2019. http://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  35. [43]

    Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512--4525

    2020

  36. [44]

    Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Alfonso Amayuelas, et al. 2024. Include: Evaluating multilingual language understanding with regional knowledge. arXiv pr...

    2024 arXiv

  37. [45]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and t...

    2021

  38. [46]

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. 2024. How to train data-efficient llms. arXiv preprint arXiv:2402.09668

    2024 arXiv

  39. [47]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  40. [48]

    Israfel Salazar, Manuel Fern \'a ndez Burda, Shayekh Bin Islam, Arshia Soltani Moakhar, Shivalika Singh, Fabian Farestam, Angelika Romanou, Danylo Boiko, Dipika Khullar, Mike Zhang, et al. 2025. Kaleidoscope: In-language exams for massively multilingual vision evaluation. arXi...

    2025 arXiv

  41. [49]

    Raphael Scheible, Fabian Thomczyk, Patric Tippmann, Victor Jaravine, and Martin Boeker. 2020. Gottbert: a pure german language model. arXiv preprint arXiv:2012.02110

    2020 arXiv

  42. [50]

    Skyler Seto, Maartje ter Hoeve, Richard He Bai, Natalie Schluter, and David Grangier. 2024. Training bilingual lms with data constraints in the targeted language. arXiv preprint arXiv:2411.12986

    2024 arXiv

  43. [51]

    Shivalika Singh, Angelika Romanou, Cl \'e mentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in mul...

    2024 arXiv

  44. [52]

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. 2017. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843--852

    2017

  45. [53]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  46. [54]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  47. [55]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  48. [56]

    Jiayi Wang, Yao Lu, Maurice Weber, Max Ryabinin, David Adelani, Yihong Chen, Raphael Tang, and Pontus Stenetorp. 2025. Multilingual language model pretraining using machine-translated data. arXiv preprint arXiv:2502.13252

    2025 arXiv

  49. [57]

    Zirui Wang, Zachary C Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual models: Findings and a meta-learning treatment. arXiv preprint arXiv:2010.03017

    2020 arXiv

  50. [58]

    Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. 2024. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37:116462--116492

    2024

  51. [59]

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei L \"u , Rui Hu, et al. 2023 a . Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341

    2023 arXiv

  52. [60]

    Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023 b . Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018

    2023 arXiv

  53. [61]

    Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209

    2017 arXiv

  54. [62]

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm \'a n, Armand Joulin, and \'E douard Grave. 2020. Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Co...

    2020

  55. [63]

    BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:...

    2022 arXiv

  56. [64]

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36

    2024

  57. [65]

    Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Yuqi Ye, and Hanwen Gu. 2024. A survey on multilingual large language models: Corpora, alignment, and bias. arXiv preprint arXiv:2404.00929

    2024 arXiv

  58. [66]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association...

    2021

  59. [67]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  60. [68]

    Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. 2025. https://arxiv.org/abs/2501.08197 Opencsg chinese corpus: A series of high-quality chinese datasets for llm training . Preprint, arXiv:2501.08197

    2025 arXiv

  61. [69]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.