Pith. sign in

REVIEW 4 cited by

An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.10267 v3 pith:BMASMK2V submitted 2025-03-13 cs.CL

Laurie Burchell , Ona de Gibert , Nikolay Arefyev , Mikko Aulamo , Marta Bañón , Pinzhen Chen , Mariia Fedorova , Liane Guillou
show 27 more authors
Barry Haddow Jan Hajič Jindřich Helcl Erik Henriksson Mateusz Klimaszewski Ville Komulainen Andrey Kutuzov Joona Kytöniemi Veronika Laippala Petter M{ae}hlum Bhavitvya Malik Farrokh Mehryary Vladislav Mikhailov Nikita Moghe Amanda Myntti Dayyán O'Brien Stephan Oepen Proyag Pal Jousia Piha Sampo Pyysalo Gema Ramírez-Sánchez David Samuel Pavel Stepachev Jörg Tiedemann Dušan Variš Tereza Vojtěchová Jaume Zaragoza-Bernabeu
This is my paper · ORCID
classification cs.CL
keywords datahpltlanguagemultilingualcontainscoveringlanguagesmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Disentangling Language Modeling and Boundaries

    cs.CL 2026-08 reject novelty 6.0 of 10

    The paper hypothesizes that next-byte and boundary distributions in byte-level LMs can be disentangled, proposes two experiments to test it, but provides no experimental results.

  2. Mangosteen: An Open Thai Corpus for Language Model Pretraining

    cs.CL 2025-07 conditional novelty 6.0 of 10

    An open 47B-token Thai pre-training corpus and a Thai-adapted data cleaning pipeline, with ablations showing quality gains and an 8B model that improves on Thai benchmarks.

  3. FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...

  4. Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    JQL trains small multilingual quality scorers from LLM judgments and human annotations, and filtering pretraining data with them improves downstream multilingual model performance over heuristic baselines.

Pith tools