Pith. sign in

REVIEW 4 major objections 5 minor 78 references

Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An English-trained Llama 3.1 8B can be adapted to Korean with ~100B tokens and beat comparable 8B models in Korean benchmarks, with a 1.4x FP8 training speedup at no accuracy cost

desk verdict Useful low-budget Korean adaptation recipe and six new benchmarks, but the headline 'best Korean performance' claim rests on leaked training splits and a flawed few-shot evaluation protocol. read the letter →

arxiv 2506.21595 v1 pith:TL5O6UHM submitted 2025-06-18 cs.CL

Jinpyo Kim , Gyeongje Cho , Chanwoo Park , Jongwon Park , Jongmin Kim , Yeonkyoun So , Jaejin Lee This is my paper
classification cs.CL
keywords KoreanLLMadaptationcontinualpre-trainingtokenizerextensionFP8precisionlow-resourcelanguagemodelingbenchmarksbilingualpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a capable Korean-English bilingual LLM can be built from an English model on a low budget. The authors start from Llama 3.1 8B, add roughly 72,000 Korean tokens to its tokenizer, continually pre-train on about 102B tokens (half Korean, half English), and then apply SFT and DPO with a few million post-training examples. Their resulting models, Thunder-LLM and Thunder-LLM-Ins, post the highest average Korean benchmark score among the compared 8B-scale models while staying within a few points of the best English scores, and they do this with far less data than the multi-trillion-token training of the comparison models. They also report a 1.43x end-to-end training speedup from applying FP8 precision only to the layers that were verified to remain stable, with no measured accuracy loss. The paper's aim is a reproducible recipe that other researchers can copy for Korean or adapt to other low-resource languages.

What carries the argument

The load-bearing mechanism is the combination of a tokenizer extension and a data-efficient training schedule. The tokenizer extension builds a Korean-optimized Unigram vocabulary of 72,000 tokens, appends them to Llama's original 128,000 tokens, and initializes each new embedding as the average of the sub-token embeddings from the base tokenizer, preserving English behavior while roughly halving the token count of Korean text. Continual pre-training on a 1:1 Korean/English mixture of about 102B tokens transfers Korean competence without destroying English skills, and post-training with SFT and DPO on a few million examples — benchmark training sets plus synthetic responses generated by larger models — supplies instruction-following and reasoning. The FP8 scheme is the speed mechanism: a stability screen over layer types shows attention matrix multiplications cannot tolerate FP8, while linear layers and the LM head can, and keeping FP8 to the stable layers yields a 1.43x training speedup with no measured accuracy change.

What would settle it

Take a newly written Korean evaluation set — for instance a reading-comprehension and reasoning test built after Thunder-LLM-Ins was released, covering domains absent from its post-training data (fresh news, legal or medical text, or new common-sense questions) — and run Thunder-LLM-Ins, EXAONE-3.5-7.8B-Instruct, and Qwen2.5-7B-Instruct through it with identical prompting. If Thunder-LLM-Ins no longer leads that held-out Korean benchmark, the paper's central performance claim is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that each stage of the pipeline contributes a measurable gain and together they close most of the Korean gap at 8B scale. Extending the Llama 3.1 tokenizer with 72,000 Korean Unigram tokens (embeddings initialized by averaging the base tokenizer's sub-token embeddings) nearly halves the number of tokens required for Korean text and cuts Korean inference time by 18%. Continual pre-training on 102B tokens of roughly balanced Korean and English text raises the average Korean benchmark score from 48.7 to 52.7, while leaving English scores largely intact. Selective FP8 training — FP8 for linear layers and the LM head but not for attention, which the authors found unstable — achieves a 1.43x end-to-end training speedup with no accuracy drop. Post-training with SFT and DPO on benchmark training sets plus synthetic data lifts the average Korean score to 65.0 and the English average to 74.7, putting Thunder-LLM-Ins at the top of the Korean suite and second on the English suite among the compared 8B models. The authors emphasize that they spent about 3,150 H100 hours on continual pre-training plus roughly 61 H100 hours on post-training, versus the several-trillion-token budgets behind the compared models.

Load-bearing premise

The claim that Thunder-LLM-Ins is the best Korean model of its scale depends on treating test splits of benchmarks whose training splits were used in post-training as a fair measure, and the paper itself reports the largest gains on exactly those benchmarks (Section 6.2), so the advantage may reflect overfitting to benchmark distributions rather than general Korean ability.

Editorial extensions

If this is right

  • Korean adaptation of an 8B English model is feasible with about 102B tokens of mixed continual pre-training plus a few million post-training examples, i.e., roughly 3,200 H100 hours rather than a from-scratch multi-trillion-token budget.
  • The extended Unigram tokenizer roughly halves Korean token counts, giving an 18% Korean inference speedup, with the cost of a small English slowdown that shrinks at larger model sizes.
  • Applying FP8 only to layers proven stable (linear layers and the LM head, not attention) gives a 1.43x end-to-end training speedup with no measured accuracy loss, a cheaper route than architecture-level FP8 stabilization.
  • The six new Korean benchmarks — five localized translations plus Ko-LAMBADA, redesigned because Korean sentences end in verbs — give future Korean LLM work a common evaluation yardstick.
  • Post-training on benchmark training sets and synthetic data lifts Korean average scores by roughly 12 points (from 52.7 to 65.0) and English by roughly 9 points (from 65.4 to 74.7), while English scores remain second among the compared models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own Section 6.2 observation — that gains are largest on benchmarks whose training sets were used in post-training — implies the true general Korean advantage is probably smaller than the headline 65.0 average; a held-out Korean test set would be needed to measure it.
  • The recipe's portability to other low-resource languages is plausible but untested; languages with very different scripts or little token overlap with English would not get the same benefit from the embedding-averaging initialization, since the base tokenizer's sub-tokens would be less informative.
  • A cheap ablation the authors do not run — varying the English share in continual pre-training while holding total tokens fixed — would reveal how much of the admitted English regression is caused by the 1:1 Korean/English mix, which directly informs the transfer of this recipe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper describes an end-to-end pipeline for adapting Llama-3.1-8B to Korean: crawling about 3 TB of Korean web text, applying rule-based/deduplication/KenLM filtering, extending the Llama tokenizer with 72K Korean tokens via a Unigram-based method, continual pre-training on 102B tokens with a 1:1 Korean-to-English ratio, and SFT+DPO post-training on benchmark training sets plus synthetic data. The authors also introduce six new Korean benchmarks, analyze which Transformer components can be trained in FP8, and report that the resulting Thunder-LLM-Ins achieves the best average Korean score among 8B-scale models (65.0 vs. 63.9 for EXAONE-3.5-7.8B-Instruct), comparable English performance, a 1.43x training speedup from FP8, and an 18% inference speedup from the tokenizer extension.

Significance. If the performance and efficiency claims held, this would be a valuable case study for low-resource language adaptation: it is unusually complete in documenting data collection, preprocessing, training, and evaluation, and it contributes new Korean benchmarks plus a simple layer-wise FP8 stability analysis. The concrete cost accounting (48 H100s, 3,150 hours for continual pre-training) and the commitment to release code are strengths. However, the headline 'best Korean performance' claim is not currently supported because the evaluation is partly circular, two Korean benchmarks draw few-shot examples from the test split, the FP8 accuracy claim is not demonstrated, and the key margin rests on single-run results with no variance estimates. The underlying pipeline is salvageable, but the evidence as presented does not establish the central claim.

major comments (4)
  1. [Section 3.3 / Table 10 / Section 6.2 / Table 5] The evaluation of the central claim is partly circular. Table 10 shows that the post-training dataset includes the training splits of KoBEST-HellaSwag, KMMLU, HellaSwag, WinoGrande, OBQA, MMLU, GSM8K, ARC-Easy, and ARC-Challenge, and Section 6.2 concedes that gains are largest on benchmarks whose training sets were included. In Table 5, the 1.1-point average margin of Thunder-LLM-Ins (65.0) over EXAONE-3.5-7.8B-Instruct (63.9) depends on KoBEST-HellaSwag (+12.4) and KMMLU (+2.5). Excluding those two columns, the average over the remaining seven Korean benchmarks is 66.4 for Thunder-LLM-Ins versus 67.2 for EXAONE, so the 'best Korean performance' claim is not supported by the current comparison. The paper should report held-out benchmarks separately and compare against baselines that also post-train on the same benchmark training splits, or remove the overlapping benchmarks from the headline average.
  2. [Appendix D.2.3 and D.2.6] The evaluation protocols for Ko-ARC-Easy/Challenge and Ko-GSM8K explicitly sample few-shot demonstrations from the test split. This is invalid regardless of any training-set overlap, because the model is given answer-annotated instances from the evaluation set when computing the score. These two benchmarks contribute to the averages in Table 5, so the reported numbers are inflated. The authors should re-evaluate with demonstrations sampled from a train or dev split, or use zero-shot evaluation.
  3. [Section 6.3 / Table 7 / Section 4.4] The claim that FP8 training provides a 1.43x speedup 'without sacrificing model accuracy' is not supported by the evidence shown. Table 7 reports only tokens per second; no accuracy comparison between BF16 and FP8 training is presented anywhere in the paper. The stability test in Table 4 shows that a 360M model's loss converges for certain FP8 component configurations, but that is not the same as demonstrating accuracy equivalence for the final 8B model. The authors should report final benchmark scores (or at least training loss curves) for BF16 and FP8 runs under the same configuration.
  4. [Appendix C3 / Table 5] Appendix C3 states that all model training was performed once due to computational constraints. Given that the headline Korean advantage over EXAONE-3.5-7.8B-Instruct is an average margin of about 1.1 points, single-run results without error bars or multiple evaluation seeds are not sufficient to establish superiority. At minimum, the paper should report variance over multiple few-shot sampling seeds and evaluation runs, and should soften any claim that the model is 'best' if the margin is within noise.
minor comments (5)
  1. [Section 6 heading] The heading 'Evalution' should be 'Evaluation'.
  2. [Table 8 caption] The word 'desribed' is a typo for 'described'.
  3. [Abstract / Section 7 / Limitations] The abstract and conclusion claim 'superior Korean performance compared to state-of-the-art models', while the Limitations section states 'We do not aim to build a state-of-the-art LLM.' This tension should be resolved by either revising the headline claim or clarifying the intended contribution.
  4. [Section 4.4 / Table 7] Please specify exactly which components (Attention, Linear, LM Head) were run in FP8 for the final Thunder-LLM training runs whose speed is reported in Table 7, so that the 1.43x speedup is reproducible.
  5. [Appendix D] The paper states that the evaluation pipeline 'most of which originated from lm-evaluation-harness' was modified, but the exact modifications (beyond import paths) are not described. Listing the modifications would improve reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Korean 'best' claim partly reduces to post-training on the benchmarks that supply the winning margin; excluding KoBEST-HellaSwag and KMMLU flips the ranking, though independent Korean gains remain.

  1. fitted input called prediction [Appendix B3; §3.3; Table 10; §6.2; Table 5]
    "For 9 benchmarks(HellaSwag, WinoGrande, ARC Easy/Challenge, MMLU, OpenbookQA, GSM8K, KoBEST-HellaSwag, and KMMLU), we utilize its train set as post-training data. For evaluating models, we utilize test set of each benchmark. ... We observe a significant performance improvement in the benchmarks that include their training set in our post-training dataset."

    The headline 'Thunder-LLM-Ins outperforms other models in Korean' rests on the Table 5 average (65.0 vs 63.9 for EXAONE). The two Korean benchmarks with the largest Thunder-LLM-Ins margins, KoBEST-HellaSwag (+12.4) and KMMLU (+2.5), are exactly the benchmarks whose training splits were placed into post-training (Table 10). Excluding those two columns, the remaining seven Korean benchmarks average about 66.4 for Thunder-LLM-Ins vs 67.2 for EXAONE, flipping the ranking. Thus the claimed 'prediction' of best-in-class Korean performance is substantially a measure of fit to the benchmark training distribution rather than an independent prediction; the paper's own §6.2 acknowledges the special improvement on benchmarks whose training sets were included.

full rationale

The paper is largely a transparent empirical engineering report, and most of its technical claims are not circular: the tokenizer extension, FP8 layer-selection stability test, and continual pre-training are evaluated on held-out test splits, and the external citations (Llama, EXAONE, Qwen, etc.) are independent artifacts. The main circularity concern is the evaluation design for the central 'best Korean performance' claim. Section 3.3 and Appendix B3 state that training sets of several Korean and English benchmarks were used for SFT/DPO post-training, while test sets of the same benchmarks were used for evaluation. The paper's own §6.2 observes the largest improvements on benchmarks whose training sets were included. Numerically, the 1.1-point average lead over EXAONE in Table 5 is carried by KoBEST-HellaSwag and KMMLU, both of which appear in the post-training data of Table 10; removing only those two columns makes EXAONE's average higher. This makes the headline comparison partly a fitted-input result rather than a clean held-out prediction. There is still independent grounding: Thunder-LLM-Ins also shows large gains on Ko-WinoGrande and Ko-LAMBADA, benchmarks not used in post-training, so the central claim is not wholly forced. Separately, Appendix D.2.3 and D.2.6 sample few-shot demonstrations from the test split itself for Ko-ARC and Ko-GSM8K; this is a self-referential evaluation protocol that can inflate those scores, although it is a correctness risk rather than a derivation-circularity step. On balance, the Korean 'best' claim is partially circular, but not reducible to a single fitted parameter or self-citation chain, so a score of 4 is appropriate.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

No invented physical or conceptual entities. The central claim rests on roughly five hand-chosen thresholds and design ratios shaping the corpus and training, plus several domain assumptions about transfer (360M to 8B, translated benchmarks, Llama base) that are empirically checked only once.

free parameters (5)
  • English:Korean data ratio = 1:1
    Chosen empirically in Section 4.2 to improve Korean quickly while limiting English degradation; affects all downstream results.
  • KenLM perplexity filtering threshold = not reported numerically
    Set to match the token processing budget in Section 3.2, so the threshold encodes a compute constraint rather than a quality target.
  • Korean tokenizer vocabulary size = 72,000
    Chosen for the extended tokenizer in Section 4.1; drives inference speed and LM head cost.
  • Rule-based filtering thresholds = 10-10M words, avg word length 2-10, >=80% Hangul, 5-gram repetition <=15%
    Hand-chosen criteria in Section 3.2 that determine the training corpus composition.
  • EMA accessibility threshold for cafe crawling = 0.15
    Stopping criterion for board crawling in Appendix B.3; affects data composition.
assumptions (6)
  • domain assumption Llama-3.1-8B is a suitable base for Korean adaptation
    The entire method builds on this model in Section 4; its English capabilities are expected to be preserved.
  • domain assumption Continual pretraining with a 1:1 Korean:English mix improves Korean without collapsing English
    Stated as an empirical choice in Section 4.2; validated only by their single run.
  • domain assumption Translated benchmarks (Ko-ARC, Ko-GSM8K, Ko-WinoGrande, Ko-IFEval) measure the same capabilities as the originals
    Translation and localization in Section 5 may change difficulty or introduce errors.
  • domain assumption FP8 stability results from a 360M model transfer to the 8B model
    Layer-wise FP8 stability test used a 360M Llama-like model in Section 4.4 and was applied to 8B training.
  • domain assumption Perplexity under a 5-gram KenLM trained on Korean Wikipedia is a valid quality filter
    Used to discard high-perplexity web documents in Section 3.2; threshold set by compute budget rather than measured quality.
  • domain assumption The crawled Naver, Daum, and Tistory web text is representative and adequately safe
    No PII or offensive-content filtering was performed (Appendix E2); the paper assumes platform moderation suffices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources." pith.science (2026). https://pith.science/paper/TL5O6UHM

@misc{pith2026250621595,
  author       = {Pith},
  title        = {Pith review of: Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TL5O6UHM}},
  note         = {Machine review of arXiv:2506.21595}
}
read the original abstract

Since state-of-the-art LLMs often underperform in languages other than English or Chinese, improving the capability of LLMs in new languages has become an essential task. Moreover, LLMs' entire end-to-end training process remains largely unknown to the public due to proprietary reasons, technical complexity, inconsistent documentation, and ethical considerations. The complete picture remains a closely guarded secret within the industry. This paper presents methods to adapt an existing English-based LLM to Korean in a low-budget scenario. We describe the entire end-to-end process: collecting Korean datasets, preprocessing the data, training the model, creating downstream benchmarks, and conducting evaluations. The evaluation results indicate that our method can effectively and cost-efficiently add new language capabilities to existing LLMs. Our new bilingual models, Thunder-LLM and Thunder-LLM-Ins, achieve superior Korean performance compared to state-of-the-art models while utilizing minimal data and computational resources. We share our comprehensive experience and make the code publicly available.

Figures

Figures reproduced from arXiv: 2506.21595 by the authors.

Figure 1
Figure 1. Illustration of the overall process. ered from three popular Korean websites: Naver2 , Daum3 , and Tistory4 . We focused on three types of online content: blogs, online communities (often referred to as cafés in Korea), and news articles. A blog is an informational website consisting of dis￾crete, often informal, posts. An online community (café) is a platform that facilitates discussions on specific topics. News ar… view at source ↗
Figure 2
Figure 2. Extending the original Llama tokenizer with [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 25 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [4]

    Yunju Bak, Hojin Lee, Minho Ryu, Jiyeon Ham, Seungjae Jung, Daniel Wontae Nam, Taegyeong Eo, Donghun Lee, Doohae Jung, Boseop Kim, and 1 others. 2025. Kanana: Compute-efficient bilingual language models. arXiv preprint arXiv:2502.18934

    arXiv 2025

  5. [5]

    Beomi. 2023. Koalpaca: Korean alpaca model based on stanford alpaca (feat. llama and polyglot-ko). https://github.com/Beomi/KoAlpaca

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  7. [7]

    ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, and KyungTae Lim. 2024 a . https://aclanthology.org/2024.lrec-main.1095/ Optimizing language augmentation for multilingual large language models: A case study on K orean . In Proceedi...

    work page 2024

  8. [8]

    ChangSu Choi, Yongbin Jeong, Seoyoon Park, InHo Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, and 1 others. 2024 b . Optimizing language augmentation for multilingual large language models: A case study on korean. arXiv preprint arXiv:2403.10882

    arXiv 2024

Show all 78 references
  1. [9]

    ChuGyouk. 2024. Numinamath cot korean. https://huggingface.co/datasets/ChuGyouk/AI-MO-NuminaMath-CoT-Ko

    2024

  2. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  3. [12]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021 b . Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [13]

    Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177

    2023 arXiv

  5. [14]

    Michal Danilák. 2013. https://github.com/Mimino666/langdetect Langdetect . May

    2013

  6. [15]

    Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. 2024 a . Sailor: Open language models for south-east asia. arXiv preprint arXiv:2404.03608

    2024 arXiv

  7. [16]

    Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Xin Mao, Ziqi Jin, Wei Lu, and Min Lin. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-demo.45 Sailor: Open language models for south- E ast A sia . In Proceedings of the 2024 Conference on Empirical Methods in Natura...

    2024 doi

  8. [17]

    Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, and 1 others. 2025. Sailor2: Sailing in south-east asia with inclusive multilingual llms. arXiv preprint arXiv:2502.12982

    2025 arXiv

  9. [18]

    Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. 2024. Scaling fp8 training to trillion-token llms. arXiv preprint arXiv:2409.12517

    2024 arXiv

  10. [19]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  11. [20]

    Rebecca Gelles, Veronica Kinoshita, Micah Musser, and James Dunham. 2024. Resource democratization: is compute the binding constraint on ai research? In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applic...

    2024

  12. [21]

    Gemma-Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:...

    2024 arXiv

  13. [22]

    Georg Glasze, Ama \"e l Cattaruzza, Fr \'e d \'e rick Douzet, Finn Dammann, Marie-Gabrielle Bertran, Clotilde B \^o mont, Matthias Braun, Didier Danet, Alix Desforges, Aude G \'e ry, and 1 others. 2023. Contested spatialities of digital sovereignty. Geopolitics, 28(2):919--958

    2023

  14. [23]

    Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, and 1 others. 2024. Bilingual adaptation of monolingual foundation models. arXiv preprint arXiv:2407.12869

    2024 arXiv

  15. [24]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  16. [25]

    Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.39 K or NLI and K or STS : New benchmark datasets for K orean natural language understanding . In Findings of the Association for Computational Linguis...

    2020 doi

  17. [26]

    Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187--197

    2011

  18. [27]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

  19. [28]

    Peter Izsak, Moshe Berchansky, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.831 How to train BERT with an academic budget . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10644--10652, Online and Punta Cana,...

    2021 doi

  20. [29]

    Myeongjun Jang, Dohyung Kim, Deuk Sin Kwon, and Eric Davis. 2022. https://aclanthology.org/2022.coling-1.325/ K o BEST : K orean balanced evaluation of significant tasks . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3697--3708, Gyeon...

    2022

  21. [30]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  22. [31]

    Deokyeong Kang and Taeuk Kim. 2024. Analysis of language models in korean program synthesis based on the kr-humaneval benchmark. In Annual Conference on Human and Language Technology, pages 245--250. Human and Language Technology

    2024

  23. [32]

    Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2024. https://doi.org/10.18653/v1/...

    2024 doi

  24. [33]

    Artur Kiulian, Anton Polishko, Mykola Khandoga, Yevhen Kostiuk, Guillermo Gabrielli, ukasz Gaga a, Fadi Zaraket, Qusai Abu Obaida, Hrishikesh Garud, Wendy Wing Yee Mak, and 1 others. 2024. From english-centric to effective bilingual: Llms with custom tokenizers for underrepres...

    2024 arXiv

  25. [34]

    Hyunwoong Ko, Kichang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, Jiwung Hyun, Sungho Park, and Kyubyong Park. 2023. A technical report for polyglot-ko: Open-source large-scale korean language models. arXiv preprint arXiv:2306.02254

    2023 arXiv

  26. [35]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  27. [36]

    Jungyup Lee, Jemin Kim, Sang Park, and SeungJae Lee. 2025. https://arxiv.org/abs/2501.10648 Dna 1.0 technical report . Preprint, arXiv:2501.10648

    2025 arXiv

  28. [37]

    LG-Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, and 1 others. 2024. Exaone 3.5: Series of large language models for real-world use cases. arXiv preprint arXiv:2412.04862

    2024

  29. [38]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, and 1 others. 2024. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Informat...

    2024

  30. [39]

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. [https://huggingface.co/AI-MO/Numi...

    2024

  31. [40]

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023 a . Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/datasets/Open-Orca/OpenOrca

    2023

  32. [41]

    Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023 b . https://https://huggingface.co/Open-Orca/SlimOrca Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification

    2023

  33. [42]

    Soyoung Lim, Heecheol Cho, Taeil Hur, Jiyeon Yim, Taeyoung Ko, Tae-Hyun Chun, Eunjin Choi, Jiyoung Jeong, Yonggyun Yu, Donghyun Shin, GyeongHwan Jang, Minjong Kim, and Sangwon Lee. 2022. Mwp\_kr\_data, dataset for math word problems in korean language. https://github.com/jkc-a...

    2022

  34. [43]

    Le, Barret Zoph, Jason Wei, and Adam Roberts

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. https://arxiv.org/abs/2301.13688 The flan collection: Designing data and methods for effective instruction tuning . Preprint, arXiv:2301.13688

    2023 arXiv

  35. [44]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/e099c1c9699814af0be873a175361713-Paper-Conference.pdf Simpo: Simple preference optimization with a reference-free reward . In Advances in Neural Information Processing Syste...

    2024

  36. [45]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  37. [46]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2021. Wechsel: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. arXiv preprint arXiv:2112.06598

    2021 arXiv

  38. [47]

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830

    2024 arXiv

  39. [48]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. https://arxiv.org/abs/2306.02707 Orca: Progressive learning from complex explanation traces of gpt-4 . Preprint, arXiv:2306.02707

    2023 arXiv

  40. [49]

    Nvidia h100 tensor core gpu architecture

    NVIDIA. Nvidia h100 tensor core gpu architecture. https://resources.nvidia.com/en-us-data-center-overview/gtc22-whitepaper-hopper

  41. [50]

    NVIDIA. 2025. Transformer engine. https://github.com/NVIDIA/TransformerEngine. Accessed: 2025-05-12

    2025

  42. [51]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

    2022

  43. [52]

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. 2016. https://doi.org/10.18653/v1/P16-1144 The LAMBADA dataset: Word prediction requiring a broad discourse cont...

    2016 doi

  44. [53]

    Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, and 1 others. 2021. Klue: Korean language understanding evaluation. arXiv preprint arXiv:2105.09680

    2021 arXiv

  45. [54]

    A Paszke. 2019. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703

    2019 arXiv

  46. [55]

    Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, and 1 others. 2023. Fp8-lm: Training fp8 large language models. arXiv preprint arXiv:2310.18313

    2023 arXiv

  47. [56]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

    2025 arXiv

  48. [57]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and 1 others. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

    2021 arXiv

  49. [58]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741

    2023

  50. [59]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

    2020

  51. [60]

    Leonard Richardson. 2007. https://www.crummy.com/software/BeautifulSoup/bs4/doc/ Beautiful soup documentation . April

    2007

  52. [61]

    Huw Roberts, Emmie Hine, and Luciano Floridi. 2023. Digital sovereignty, digital expansionism, and the prospects for global ai governance. In Quo Vadis, Sovereignty? New Conceptual and Regulatory Boundaries in the Age of Digital China, pages 51--75. Springer

    2023

  53. [62]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  54. [63]

    Carlos Saura Garc \' a. 2024 a . Datafeudalism: the domination of modern societies by big tech companies. Philosophy & Technology, 37(3):90

    2024

  55. [64]

    Carlos Saura Garc \' a. 2024 b . Digital expansionism and big tech companies: consequences in democracies of the european union. Humanities and Social Sciences Communications, 11(1):1--8

    2024

  56. [65]

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, and 1 others. 2024. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arX...

    2024 arXiv

  57. [66]

    Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548

    2024 arXiv

  58. [67]

    Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2025 a . https://aclanthology.org/2025.naacl-long.206/ KMMLU : Measuring massive multitask language understanding in K orean . In Proceedin...

    2025

  59. [68]

    Youngjun Son, Chaewon Kim, and Jaejin Lee. 2025 b . Fed: Fast and efficient dataset deduplication framework with gpu acceleration. arXiv preprint arXiv:2501.01046

    2025 arXiv

  60. [69]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  61. [70]

    Leandro Von Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, and Helen Ngo. 2022. https://doi.org/10.18653/v1/2022.emnlp-demos.13 Evaluate & evaluation on the hub: Better best practices for d...

    2022 doi

  62. [71]

    Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, and 1 others. 2024. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37...

    2024

  63. [72]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022 a . https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In The Tenth International Conference on Learning R...

    2022

  64. [73]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  65. [74]

    https://dumps.wikimedia.org Wikimedia downloads

    Wikimedia. https://dumps.wikimedia.org Wikimedia downloads

  66. [75]

    Ningyuan Xi, Yetao Wu, Kun Fan, Teng Chen, Qingqing Gu, Peng Yu, Jinxian Qu, Chenxi Liu, Zhonglin Jiang, Yong Chen, and 1 others. 2024. A practice of post-training on llama-3 70b with optimal selection of additional language mixture ratio. arXiv preprint arXiv:2409.06624

    2024 arXiv

  67. [76]

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. 2025. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint arXiv:2504.14655

    2025 arXiv

  68. [77]

    Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, and 1 others. 2024. Hyperclova x technical report. arXiv preprint arXiv:2404.01954

    2024 arXiv

  69. [78]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  70. [79]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.