Pith. sign in

REVIEW 4 major objections 4 minor 44 references

Mangosteen: An Open Thai Corpus for Language Model Pretraining

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The authors claim that Thai-specific cleaning of web text — script-based language identification, Thai-rooted quality thresholds, and filters for adult and gambling content — yields a 47.4-billion-token corpus whose continually…

desk verdict A transparent Thai corpus and pipeline worth building on; the ablations are underpowered single runs, and the safety regression needs a real look, but the resource release carries it. read the letter →

arxiv 2507.14664 v2 pith:BCPZQOFJ submitted 2025-07-19 cs.CL

Wannaphong Phatthiyaphaibun , Can Udomcharoenchaikit , Pakpoom Singkorapoom , Kunat Pipatanakul , Ekapol Chuangsuwanich , Peerat Limkonchotiwat , Sarana Nutanong This is my paper
classification cs.CL
keywords Thaipre-trainingcorpusdatacleaningpipelinelanguagemodelcontinualidentificationcontentfilteringCommonCrawlSEA-HELM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that Thai-specific data curation, not just corpus scale, is what makes a Thai language model good. It builds Mangosteen, a 47.4-billion-token Thai pre-training corpus, by adapting the Dolma web-cleaning pipeline to Thai script and culture: a rule-based language identifier replaces FastText, C4 and Gopher quality rules get Thai thresholds (including a 200-word minimum document length), and dedicated filters remove adult and gambling content that is illegal in Thailand. In GPT-2 ablations at 10 billion tokens each, the fully cleaned data beats raw Thai web text on SEA-HELM and the Thai LLM benchmark while shrinking the corpus from 202 million to 25 million documents. An 8-billion-parameter SEA-LION model continually pre-trained on Mangosteen then outperforms SEA-LION-v3, Llama-3.1, and Typhoon-2 on Thai benchmarks, with the abstract reporting roughly a four-point gain. If the claim holds, the recipe matters: mid-resource languages can buy quality with cleaning rather than scale.

What carries the argument

The mechanism is the Thai-adapted Dolma pipeline, a four-stage assembly where each stage is re-tuned for Thai. Language identification uses a rule-based ThaiCharRatioTagger that keeps documents with at least 50 percent Thai Unicode characters, chosen because FastText assigned high Thai confidence to documents with under 40 percent Thai characters. Quality filtering keeps C4 rules but drops the punctuation rule (Thai sentences do not end with periods) and adds corrupted-Unicode handling, while Gopher rules get a raised minimum document length of 200 words — justified by lower WangchanBART perplexity, with a Kruskal-Wallis test reported as significant — an 80 percent Thai-letter requirement, Thai stopwords, an ellipsis rule that counts three dots, and a rejection of documents containing 'read more' truncation markers. Deduplication keeps Dolma's Bloom-filter URL and document-level overlap removal but drops paragraph-level deduplication because newlines are not reliable paragraph boundaries in Thai. Content filtering trains two FastText binary classifiers on Thai-labeled adult and gambling documents, with gambling labels supplemented by an LLM and a human-validated subsample. The ablation design itself carries the argument: each stage is added incrementally to 10-billion-token GPT-2 runs, and the pipeline is also re-applied to the already-cleaned FineWeb2 to test whether Thai-specific cleaning adds value on top of generic cleaning.

What would settle it

Run the same five GPT-2 configurations (baseline through content filters) several times at 10 billion tokens with different random seeds and check whether the per-step score ordering — including the deduplication dip and the content-filter recovery — is stable across seeds; if the stage-to-stage gaps move by more than a couple of SEA-HELM points between runs, the claimed attribution of gains to individual pipeline steps is not established. Alternatively, train the 8-billion-parameter model on an uncleaned Thai web corpus of comparable size with the same continual pre-training and SFT recipe and check whether the reported roughly four-point SEA-HELM edge over SEA-LION-v3 shrinks or vanishes.

Watch

Extended reading notes

Core claim

The central claim is that every stage of a pre-training data pipeline needs to be reworked for the target language and culture, and that the reworked pipeline produces a corpus that beats both raw web text and generically cleaned web text. On raw CommonCrawl, the Thai-adapted pipeline raises a GPT-2 model's SEA-HELM average from 7.04 to 17.80 and its Thai LLM benchmark average from 2.08 to 2.53 while removing about 88 percent of documents; on already-cleaned FineWeb2, the same pipeline halves the Thai subset and improves average SEA-HELM. The downstream demonstration is WangchanLION-V3-8B: an 8-billion-parameter SEA-LION model continually pre-trained on Mangosteen's 47.4 billion tokens and supervised-fine-tuned on Thai instructions, which the paper reports scoring 43.42 on SEA-HELM average versus 39.31 for SEA-LION-v3 and 31.54 for Llama-3.1, while leading the Thai LLM Leaderboard NLG task at 54.84. The paper also reports that the gains concentrate in Thai cultural knowledge, roleplay, and reasoning on MT-Bench, and acknowledges an NLU trade-off because world knowledge from the original English-trained base is overwritten. All artifacts — pipeline code, cleaning manifests, corpus snapshot, and checkpoints — are released so the recipe can be inspected and re-run.

Load-bearing premise

The step-by-step effectiveness of the pipeline rests on 124-million-parameter GPT-2 models trained once per configuration on 10 billion tokens each, with no repeated runs and no error bars, and the paper assumes that this ranking of cleaning steps carries over to the 8-billion-parameter model.

Editorial extensions

If this is right

  • On raw CommonCrawl, adding the pipeline stages step by step lifts a GPT-2 model's SEA-HELM average from 7.04 to 17.80 and its Thai LLM benchmark average from 2.08 to 2.53, while shrinking the corpus from 202 million documents to 25.1 million.
  • Passing the already-cleaned FineWeb2 Thai subset through the same pipeline halves it (35.9 million to 17.1 million documents) and improves average SEA-HELM, which the paper reads as evidence that a generic pipeline misses Thai-specific unwanted content such as gambling and adult pages.
  • The 8-billion-parameter WangchanLION-V3-8B model scores 43.42 on the SEA-HELM average with the same SFT recipe, against 39.31 for SEA-LION-v3, 41.46 for Typhoon-2, and 31.54 for Llama-3.1, and leads the Thai LLM Leaderboard NLG task with 54.84.
  • On Thai MT-Bench, the Mangosteen-trained model's largest gains over the baselines sit in the cultural-knowledge, roleplay, and reasoning categories, suggesting the cleaned Thai data contributes culturally specific knowledge rather than generic fluency.
  • Since the pipeline code, cleaning manifests, corpus snapshot, and all checkpoints are released openly, the corpus functions as a reproducibility artifact: any team can re-run the recipe, inspect which documents were removed at each stage, and adapt it to neighboring languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-run 10-billion-token ablations leave open the possibility that the per-stage ordering — including the visible drop in SEA-HELM after deduplication and its recovery after content filtering — reflects training noise; a multi-seed replication at the same scale would reveal whether the gaps between stages exceed run-to-run variance.
  • The 200-word minimum document length was validated by perplexity measured on the same corpus it filters, which creates a mild circularity; an independent check would hold a 50-word or 100-word floor fixed and compare downstream scores at small and large scale to see how much short-document diversity is actually being traded away.
  • Because a large share of the removed documents are gambling and adult pages that are illegal in Thailand, the pipeline's safety payoff may exceed its benchmark payoff; a direct audit counting how much gambling and adult content survives into the final 25.1-million-document corpus would test that separately from downstream scores.
  • The script-ratio language identifier is transparent and cheap, and the same trick could plausibly be reused for other script-distinct languages, but the fixed 50-percent Thai-character threshold could silently discard legitimate mixed-language documents such as technical or code-heavy pages, which deserves a targeted check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces Mangosteen, a 47.4B-token Thai pretraining corpus assembled from Common Crawl, FineWeb2, and curated non-web sources (Wikipedia, Royal Gazette, OCR books, CC-licensed YouTube subtitles), together with a Thai-adapted version of the Dolma cleaning pipeline. The pipeline adds a rule-based Thai language identifier, modified C4/Gopher quality filters, document-level deduplication, and Thai-trained adult/gambling content filters. The authors evaluate pipeline components via GPT-2 124M ablations on 10B-token subsets and present an 8B-parameter continual-pretrained model (WangchanLION-V3-8B) that outperforms SEA-LION-v3, Llama-3.1, and Typhoon-2 on SEA-HELM and the Thai LLM Benchmark. The paper releases pipeline code, cleaning manifests, the corpus, and checkpoints.

Significance. If the pipeline's effectiveness is confirmed, this is a valuable contribution to open Thai-language resources: it is one of the few works to release the full data-cleaning pipeline, corpus snapshot, and model checkpoints, and it documents design decisions in unusual detail (Appendices B and C). The comparison of language identifiers and the correlation analyses of Gopher rules are useful empirical documentation. The central quantitative claim—that the Thai-adapted components cause the observed gains—is currently supported only by single-run small-scale ablations and a single 8B pretraining trial, so the significance of the causal story is contingent on additional variance control.

major comments (4)
  1. [Section 6.1, Table 3] Each ablation configuration is a single GPT-2 124M run on 10B tokens, with no repeated seeds and no error bars. The reported pattern is non-monotonic in a way that is consistent with seed-level noise: adding deduplication drops the SEA-HELM average from 17.64 to 12.29, and adding content filters recovers to 17.80, which is only 0.16 above the +Quality Filters stage; on the Thai LLM Benchmark the full pipeline improves by only 0.03 over the previous stage. Without variance estimates or multiple runs, the paper cannot distinguish the contribution of individual pipeline steps from chance, and the claim that each component is effective is not supported.
  2. [Section 4.3 and Appendix C.1] The 200-word minimum document length is a load-bearing design choice, but it is not validated by a downstream ablation. The perplexity experiment in Appendix C.1 compares lower bounds of 50 and 171 words, not 200, and the perplexity is computed on the same 200,000-document subsample from which the threshold was derived, so the selection is partly circular. The authors themselves note that 'Ideally, the optimal value for the lower bound should be determined through a series of data ablation experiments' but that resources prevented this. The paper should either report an ablation over candidate thresholds or explicitly frame the 200-word threshold as a heuristic rather than an empirically justified optimum.
  3. [Section 5.2 and Section 6.1.1] The GPT-2 evaluation excludes NLU, MT-bench, and safety tasks because the models produce normalized scores of zero, and the SEA-HELM evaluation removes NLR and XNLI due to translation-quality concerns. This means the headline improvement 'SEA-HELM NLG from 3 to 11' is measured on a modified benchmark whose composition differs from the published SEA-HELM suite. Reporting zero suppression and raw scores for all tasks would allow readers to assess whether the reported gains are robust to the choice of included subtasks.
  4. [Pre-training Cost and Shared Knowledge paragraph and abstract] The final 8B comparison rests on a single training trial, as the paper states the compute budget allowed 'effectively one full trial.' Consequently, the headline advantage over baselines has no variance estimate. In addition, the abstract's claim of improvement 'by about four points' is ambiguous: Table 5 shows an average lead of 4.11 over SEA-LION-v3 but only 1.96 over Typhoon-2, while the conclusion says 'approximately 2 points above recent Thai baselines.' The claims should be reconciled and the uncertainty acknowledged.
minor comments (4)
  1. [Abstract and Conclusion] The abstract says the model 'surpasses SEA-LION-v3 and Llama-3.1 by about four points,' but the Conclusion says 'approximately 2 points above recent Thai baselines.' Please make the comparison target explicit and consistent.
  2. [Appendix C] The opening sentence references 'Section 3.1.3,' which does not exist in the manuscript; the relevant section appears to be Section 4.3.
  3. [Figure 8] The labels contain typos: 'Basline' should be 'Baseline' and 'Goher rules' should be 'Gopher rules.' Similar typos appear elsewhere, e.g., 'Tabel 15' in Appendix C.2.
  4. [Section 6.1.2] The text says 'most of the text was removed by our C4, Gopher, and gambling-related content filters,' but Table 4 shows that the largest removal happens at the Quality Filters step (from 175M to 35.6M documents), while the content filter step removes only about 0.9M documents. Please make the attribution to individual steps precise.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the central Mangosteen quality claim is tested on external benchmarks, not derived from the corpus's own statistics.

full rationale

The paper's central claim—that its Thai-adapted Dolma pipeline improves pretraining data quality—is supported by GPT-2 ablations evaluated on SEA-HELM and the Thai LLM Benchmark, which are independent of the Mangosteen corpus and were not fitted to it. The 200-word quality-filter threshold is chosen post hoc from WangchanBART perplexity on the same 200,000-document subsample (Appendix C.1), so that particular threshold is internally tuned rather than externally predicted; however, this does not make the downstream SEA-HELM/Thai LLM comparison circular, because the benchmarks are not constructed from the perplexity signal or from the pipeline's filter rules. Similarly, the paper openly states in 'Pre-training Cost and Shared Knowledge' that compute allowed 'effectively one full trial,' so the 8B result is a single run without variance estimates; that is a statistical robustness limitation, not a reduction of the conclusion to its inputs. Self-citations (PyThaiNLP/nlpO3, SEA-LION, SEA-HELM) are to tools, a base model, and an evaluation benchmark, all external artifacts that do not assert the paper's conclusion. No equation or construction in the paper defines pipeline quality in terms of the benchmark scores it later reports; no fitted parameter is renamed as a prediction. The circularity burden is therefore low.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several hand-chosen thresholds and proxy assumptions rather than a theoretical derivation. The free parameters are the pipeline thresholds; the axioms are the methodological assumptions that the chosen ablation and benchmark setup faithfully measures corpus quality.

free parameters (6)
  • Thai character ratio threshold (LID) = 0.5
    Rule-based LID keeps documents where at least half of characters are Thai (Section 4.2). Threshold chosen by hand, not optimized.
  • Minimum document length (Gopher word count) = 200 words
    Raised from Gopher's 50 to 200 based on perplexity comparisons on the same sample (Appendix C.1); this cutoff directly affects the 202M-to-25M document reduction.
  • Maximum document length = 100,000 words
    Retained from Gopher; stated as cap in Section 4.3.
  • Thai letter fraction threshold = 0.80
    Modified Gopher rule: documents where Thai consonants are less than 80% of characters are excluded (Section 4.3).
  • Ellipsis line fraction threshold = 0.30
    Documents where over 30% of lines end with ellipsis are discarded (Section 4.3).
  • Content filter label threshold = 3 words
    Docs are labeled as adult/gambling if they contain 3+ distinct words from a predefined list (Section 4.5).
assumptions (6)
  • domain assumption Thai Unicode character ratio is a sufficient language identifier
    Section 4.2 uses ratio >= 0.5 to detect Thai; the custom tagger is not validated against gold mixed-language labels.
  • domain assumption GPT-2 124M ablations on 10B tokens transfer to larger models
    Section 6.1 uses GPT-2 to rank pipeline steps; no evidence this ranking holds for 8B models.
  • domain assumption Benchmark subsetting is valid
    Section 5.2 removes NLR/XNLI and GPT-2 evaluation drops NLU, MT-bench, and safety; gains are reported on the selected subset.
  • domain assumption Perplexity (WangchanBART) is a valid proxy for document quality
    Appendix C.1 uses perplexity to justify the 200-word threshold without external quality annotations.
  • domain assumption Keyword-based labels for content filters are reliable
    Section 4.5 labels docs by 3+ keyword matches, then trains FastText classifiers on them; human validation only on a random subset.
  • domain assumption Common Crawl 2018-30 to 2023-23 plus FineWeb2 covers Thai web
    Section 3.1 selects these sources; representativeness of Thai web content is not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mangosteen: An Open Thai Corpus for Language Model Pretraining." pith.science (2026). https://pith.science/paper/BCPZQOFJ

@misc{pith2026250714664,
  author       = {Pith},
  title        = {Pith review of: Mangosteen: An Open Thai Corpus for Language Model Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCPZQOFJ}},
  note         = {Machine review of arXiv:2507.14664}
}
read the original abstract

Pre-training data shapes a language model's quality, but raw web text is noisy and demands careful cleaning. Existing large-scale corpora rely on English-centric or language-agnostic pipelines whose heuristics do not capture Thai script or cultural nuances, leaving risky material such as gambling content untreated. Prior Thai-specific efforts customize pipelines or build new ones, yet seldom release their data or document design choices, hindering reproducibility and raising the question of how to construct a transparent, high-quality Thai corpus. We introduce Mangosteen: a 47 billion-token Thai corpus built through a Thai-adapted Dolma pipeline that includes custom rule-based language ID, revised C4/Gopher quality filters, and Thai-trained content filters, plus curated non-web sources such as Wikipedia, Royal Gazette texts, OCR-extracted books, and CC-licensed YouTube subtitles. Systematic ablations using GPT-2 show the pipeline trims CommonCrawl from 202M to 25M documents while raising SEA-HELM NLG from 3 to 11; an 8B-parameter SEA-LION model continually pre-trained on Mangosteen then surpasses SEA-LION-v3 and Llama-3.1 by about four points on Thai benchmarks. We release the full pipeline code, cleaning manifests, corpus snapshot, and all checkpoints, providing a fully reproducible foundation for future Thai and regional LLM research.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  4. [4]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  5. [5]

    SCB 10X, VISTEC, and SEACrowd. 2024. https://huggingface.co/spaces/ThaiLLM-Leaderboard/leaderboard Thai llm leaderboard

    work page 2024

  6. [6]

    Ashish Agrawal, Barah Fazili, and Preethi Jyothi. 2024. https://aclanthology.org/2024.eacl-short.28/ Translation errors significantly impact low-resource languages in cross-lingual learning . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 319--329, St. Julian '...

    work page 2024

  7. [7]

    Giusepppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor

    work page 2015

  8. [8]

    Adrien Barbaresi. 2021. https://aclanthology.org/2021.acl-demo.15 Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction . In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System...

    work page 2021

Show all 44 references
  1. [9]

    Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Pette...

    2025 arXiv

  2. [10]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  3. [11]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods ...

    2018 doi

  4. [12]

    Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. https://arxiv.org/abs/2404.05829 Sambalingo: Teaching large language models new languages . Preprint, arXiv:2404.05829

    2024 arXiv

  5. [13]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://openreview.net/forum?id=OUIFPHEgJU QL o RA : Efficient finetuning of quantized LLM s . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  6. [14]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    2018 arXiv

  7. [15]

    Julen Etxaniz, Oscar Sainz, Naiara Miguel, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. 2024. https://doi.org/10.18653/v1/2024.acl-long.799 Latxa: An open language model and evaluation suite for B asque . In Proceedings of the 62n...

    2024 doi

  8. [16]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The P ile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  9. [17]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  10. [18]

    Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. https://aclanthology.org/L18-1550/ Learning word vectors for 157 languages . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018) , Miya...

    2018

  11. [19]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. https://aclanthology.org/E17-2068/ Bag of tricks for efficient text classification . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume...

    2017

  12. [20]

    Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad B, Varun G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra. 2024. https://doi.org/10.18653/v1/2024.acl-long.843 I ndic LLMS u...

    2024 doi

  13. [21]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, and 40 o...

    2025 arXiv

  14. [22]

    Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius Hadiwijaya, Joanito Agili Lopo, William Nixon, B \"o rje F

    Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James Validad Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Monta...

    2024 doi

  15. [23]

    Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki ...

    2023 doi

  16. [24]

    Chenghao Mou, Chris Ha, Kenneth Enevoldsen, and Peiyuan Liu. 2023. https://doi.org/10.5281/zenodo.8364980 Chenghaomou/text-dedup: Reference snapshot

    2023 doi

  17. [25]

    Raymond Ng, Thanh Ngan Nguyen, Yuli Huang, Ngee Chia Tai, Wai Yi Leong, Wei Qi Leong, Xianbin Yong, Jian Gang Ngui, Yosephine Susanto, Nicholas Cheng, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Adithya Venkatadri Hulagadri, Kok Wai Teng, Yeo Yeow Tong, Bryan Siow, Wei Yi...

    2025

  18. [26]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. https://arxiv.org/abs/2309.09400 Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages . Preprin...

    2023 arXiv

  19. [27]

    Pedro Javier Ortiz Su 'a rez, Laurent Romary, and Benoit Sagot. 2020. https://www.aclweb.org/anthology/2020.acl-main.156 A monolingual approach to contextualized word embeddings for mid-resource languages . In Proceedings of the 58th Annual Meeting of the Association for Compu...

    2020

  20. [28]

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024 a . https://arxiv.org/abs/2406.17557 The fineweb datasets: Decanting the web for the finest text data at scale . Preprint, arXiv:2406.17557

    2024 arXiv

  21. [29]

    Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. 2024 b . https://doi.org/10.57967/hf/3744 Fineweb2: A sparkling update with 1000s of languages

    2024 doi

  22. [30]

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. https://arxiv.org/abs/2306.01116 The refinedweb dataset for falcon llm: Outperforming curated corpora wit...

    2023 arXiv

  23. [31]

    Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, Pattarawat Chormai, Peerat Limkonchotiwat, Thanathip Suntorntip, and Can Udomcharoenchaikit. 2023. https://doi.org/10.18653/v1/2023.nlposs-1.4 P y T hai NLP : T h...

    2023 doi

  24. [32]

    Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarnmongkol, Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Tharnpipitchai. 2023. https://arxiv.org/abs/2312.13951 Typhoon: Thai large language models . Preprint, arXiv:2312.13951

    2023 arXiv

  25. [33]

    Kunat Pipatanakul, Potsawee Manakul, Natapong Nitarach, Warit Sirichotedumrong, Surapon Nonesung, Teetouch Jaknamon, Parinthapat Pengpun, Pittawat Taveekitworachai, Adisai Na-Thalang, Sittipong Sripaisarnmongkol, Krisanapong Jirayoot, and Kasima Tharnpipitchai. 2024. https://a...

    2024 arXiv

  26. [34]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  27. [35]

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks,...

    2021 arXiv

  28. [36]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1)

    2020

  29. [37]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Preprint, arXiv:1910.10683

    2023 arXiv

  30. [38]

    Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut, Alic...

    2025 arXiv

  31. [39]

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennigh...

    2024 doi

  32. [40]

    Thanathip Suntorntip, Arthit Suriyawongkul, and Wannaphong Phatthiyaphaibun. 2024. https://doi.org/10.5281/zenodo.14082449 nlpo3

    2024 doi

  33. [41]

    Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xian Bin Yong, Weiqi Leong, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Yifan Mai, and William Chandra Tjhi. 2025. https://arxiv.org/abs/2502.14301 Sea-helm: Southeast asian holistic ev...

    2025 arXiv

  34. [42]

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm \'a n, Armand Joulin, and Edouard Grave. 2020. https://aclanthology.org/2020.lrec-1.494 CCN et: Extracting high quality monolingual datasets from web crawl data . In Proceedings of the Twel...

    2020

  35. [43]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  36. [44]

    Sumeth Yuenyong, Kobkrit Viriyayudhakorn, Apivadee Piyatumrong, and Jillaphat Jaroenkantasima. 2025. https://arxiv.org/abs/2411.07238 Openthaigpt 1.5: A thai-centric open source large language model . Preprint, arXiv:2411.07238

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.