Pith. sign in

REVIEW 4 major objections 5 minor 18 references

Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Llama-3-Motif is a 102B model grown from Llama 3 70B by depth and width expansion and continually pre-trained on 194B tokens at a 9:1 Korean-to-English ratio; the paper reports it matches or exceeds GPT-4 on the KMMLU and KorMedMCQA Korean

desk verdict A plausibly novel scaling recipe for Korean LLMs, but the GPT-4-comparable claim currently rests on un-decontaminated benchmark scores and no English evaluation. read the letter →

arxiv 2509.03972 v1 pith:F7JE4VLR submitted 2025-09-04 cs.CL cs.AIcs.LG

Junghwan Lim , Gangwon Jo , Sungmin Lee , Jiyoung Park , Dongseok Kim , Jihwan Kim , Junhyeok Lee , Wai Ting Cheung
show 11 more authors
Dahye Choi Kibong Choi Jaeyeon Huh Beomgyu Kim Jangwoong Kim Taehyun Kim Haesol Lee Jeesoo Lee Dongpin Oh Changseok Song Daewon Suh
This is my paper
classification cs.CLcs.AIcs.LG
keywords Llama-3-MotifKoreanlanguagemodelingcontinualpretrainingdepthexpansionwidthKMMLUKorMedMCQAKTOalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Llama-3-Motif is a 102-billion-parameter model built by expanding Llama 3 70B to 96 layers and a hidden size of 9,216, then continually pre-trained on about 194 billion tokens with a 9:1 Korean-to-English ratio. The paper reports that this model scores 64.74 on KMMLU-direct (5-shot), above the authors' measurement of GPT-4o (64.11), and 77.19/83.34 on KorMedMCQA Doctor/average, above GPT-4-base (76.49/83.06), while claiming English performance is retained. If true, the work shows a practical recipe for raising a less-resourced language's capability from an existing open checkpoint with relatively modest additional training data, using depth/width expansion plus targeted post-training. The authors position it as a step toward closing the gap for under-resourced languages, with a medical consultation service as one application. The main caveat is the absence of reported decontamination between the web-crawled corpus and benchmark items.

What carries the argument

The load-bearing mechanism is structured parameter growth with initialization preservation. Depth expansion (LlamaPro) inserts layers initialized to copy existing blocks, so the model's behavior is initially unchanged; width expansion (Masked Structure Growth) adds hidden and FFN units whose outputs are masked at first and unmasked gradually each training step. This lets the enlarged model absorb new Korean-domain knowledge during 194B tokens of continual pretraining (9:1 Korean:English) while retaining the base model's English and general abilities. Post-training uses NEFTune (noisy embeddings during SFT) and KTO (unpaired preference optimization) to align the model at lower data-collection

What would settle it

Take the 35,030 KMMLU questions and the KorMedMCQA exam items, run exact and fuzzy string matching against the 194B-token training corpus after the paper's deduplication pipeline; any nontrivial match count means the benchmark gap is partly memorization. Then re-administer both benchmarks on a newly released Korean exam that postdates the training data cutoff and compare Llama-3-Motif against GPT-4o and GPT-4-base under identical 5-shot conditions.

Watch

Extended reading notes

Core claim

The central claim is that a 70B English-centric checkpoint can be grown into a 102B model that is competitive with GPT-4 on Korean understanding without abandoning the Transformer architecture or starting from scratch. Growth is done in two stages: LlamaPro adds 20% more layers, and Masked Structure Growth widens hidden and FFN dimensions, with new neurons initially masked and progressively unmasked during continual pretraining. All weights are then tuned on a Korean-heavy corpus of 194B tokens, followed by NEFTune-based instruction fine-tuning and KTO preference optimization. The paper reports the resulting model outperforms existing Korean-specialized open models by 9–40% on KMMLU and 4.9–

Load-bearing premise

The headline comparison to GPT-4 holds only if the 194B-token crawl that trained the model never overlapped the KMMLU or KorMedMCQA test questions and every baseline was measured under the same prompt-and-scoring protocol; the paper reports no decontamination check and mixes community reports with author-run numbers.

Editorial extensions

If this is right

  • The expansion-plus-continual-pretraining recipe can be repeated from an existing open checkpoint, so other languages with scarce high-quality data could get a foundation model tuned to their domains without training from scratch.
  • A 194B-token, 9:1 Korean:English mix appears sufficient to lift KMMLU-direct from about 54.5 (reported for Llama-3-70B-Instruct) to 64.74.
  • The KorMedMCQA results suggest the model can serve as a base for Korean medical consultation services, the use case the authors state they are developing.
  • The 0.63-point edge over GPT-4o on KMMLU and the 0.28/0.70-point edges over GPT-4-base on KorMedMCQA are best read as parity claims, not dominance claims, given the mixed provenance of the baseline numbers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: no decontamination step is reported, so the headline comparison should be treated as provisional until the 194B-token corpus is checked for overlap with KMMLU and KorMedMCQA test items.
  • Editorial inference: the 9:1 Korean:English ratio is likely language-specific; other languages may need a different balance between new-language exposure and preservation of English general knowledge, and sweeping this ratio would be a natural extension.
  • Editorial inference: because the reported parity rests on two benchmarks and 5-shot prompting, the paper's more durable contribution may be the engineering recipe (structured growth plus continual pretraining plus low-cost alignment) rather than a broad claim about GPT-4-level Korean ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Llama-3-Motif, a 102B-parameter dense Transformer built by expanding Llama-3-70B through depth (LlamaPro) and width (Masked Structure Growth) expansion, followed by continual pretraining on about 194B tokens with a 9:1 Korean:English ratio, SFT with NEFTune, and preference alignment with KTO. The authors report KMMLU-direct 5-shot score of 64.74 and KorMedMCQA scores of 77.19 (Doctor) / 83.34 (average), claiming the model outperforms existing Korean-specialized models and is comparable to GPT-4, while 'retaining strong performance in English.' The central evidence is Tables 2 and 3, which compare against a mix of community-reported and author-measured baselines. No English evaluation is reported, and no decontamination analysis is described for the training corpus versus the benchmark test sets.

Significance. If the empirical claims are reliable, the paper would demonstrate a practical recipe for producing a Korean-specialized 102B model from an existing 70B checkpoint using relatively few additional tokens, and it would provide useful engineering evidence for LlamaPro and MSG at scale. The reported 102B parameter count is consistent with the architecture in Table 1 (96 layers, dim 9,216, FFN 30,720, vocab 128k), and the use of external benchmarks (KMMLU, KorMedMCQA) is appropriate. The main contributions are not machine-checked proofs or released code/weights; the value is in the empirical scaling and data-curation result. However, the headline comparison to GPT-4 rests entirely on small margins (0.63 points on KMMLU-direct; 0.28/0.73 points on KorMedMCQA) that are vulnerable to training-data contamination and uncontrolled evaluation protocols. The absence of any English benchmark also leaves the 'retaining strong performance in English' claim unsupported.

major comments (4)
  1. [§4.2-4.3, §6.1-6.2] No decontamination analysis is reported for the 194B-token web-crawled corpus against the KMMLU and KorMedMCQA test sets. Section 6.1 states that KMMLU questions come from Korean standardized exams and that the model's strong performance is 'largely attributed to the composition of our training dataset,' which included 'domestic academic papers, research reports, and patents.' KorMedMCQA is built from public Korean licensing-exam questions. Because the reported margins over GPT-4 are sub-point (0.63 KMMLU, 0.28/0.73 KorMedMCQA), even minor overlap/memorization could account for the advantage. The paper needs to report an n-gram/substring overlap analysis, exclude any overlapping items, and re-run the evaluation before 'comparable to GPT-4' can be accepted as a capability claim.
  2. [Abstract, §1, §6] The central claim includes 'retaining strong performance in English,' but Section 6 contains only Korean benchmarks. No English evaluation (e.g., MMLU, HellaSwag, ARC, or any held-out English task) is reported. Without a comparison to the original Llama-3-70B or to other English-strong models, the claim of retained English capability is unsupported. Please add standard English benchmark results, ideally with the same evaluation protocol used for baselines.
  3. [Tables 2 and 3] The baseline scores are not produced under a single comparable protocol: Table 2 mixes community reports (†), author measurements (‡), and different instruction-tuned/base variants; Table 3 uses a report from [9] for GPT-4-base and author measurements for other models. No decoding parameters, prompt templates, or variance/confidence intervals are given. The 0.63-point KMMLU gap and 0.28-point KorMedMCQA-average gap are small enough that protocol differences or sampling noise could reverse the comparison. A controlled re-evaluation of all models with identical 5-shot prompts and greedy decoding, or at least a sensitivity analysis, is needed to support the 'comparable to GPT-4' statement.
  4. [§2] The paper states that four preliminary experiments on Qwen 1.8B were conducted and that LlamaPro was selected 'based on our preliminary results,' but no results, tables, or quantitative comparisons are provided. Since the model-scaling methodology is a listed contribution, the absence of these experimental data makes the choice of depth/width expansion strategy unverifiable. Please report at least a summary table of the four methods' performance and the criterion used for selection.
minor comments (5)
  1. [§5] Typo: 'validaters' should be 'validators.'
  2. [§5] 'Direct Policy Optimization (DPO)' should be 'Direct Preference Optimization (DPO).'
  3. [§5] The final KTO hyperparameters (batch size 128, LR 1e-6, NEFT alpha 0, KTO lambda 1.375/1) are stated, but the SFT/NEFTune setup lacks training length, LR, and dataset size; please add these for reproducibility.
  4. [Table 2] The 'Community report' reference [4] is a LinkedIn post; please replace with an archival or citable source, or describe the evaluation protocol used for those entries.
  5. [§4.1] The 9:1 Korean:English ratio is stated, but the final token counts per language (approximately 174.6B Korean and 19.4B English if the total is 194B) could be made explicit to clarify the balance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark results are independent measurements, not re-derivations of training inputs.

full rationale

The paper's central claims are empirical benchmark scores on KMMLU and KorMedMCQA. These are external, published benchmarks [13, 9], and the paper does not fit any parameter to them or define them in terms of the model's own outputs. The model expansion methods (LlamaPro [15], MSG [16]) are cited external techniques, and no load-bearing argument reduces to a self-citation by the present authors. Section 6.1 says the model's KMMLU performance 'can be largely attributed to the composition of our training dataset,' but that is an interpretive attribution, not a definitional equivalence: the reported scores are still measurements of the trained model on held-out benchmark questions. The absence of a reported decontamination analysis between the 194B-token Korean/English corpus and KMMLU/KorMedMCQA is a legitimate data-validity and benchmark-integrity concern, and it could weaken the GPT-4 comparison if test items were memorized. However, that concern is not circularity: even under contamination, the scores are observed outcomes, not quantities constructed from the model's own fitted parameters or from the paper's own assertions. No self-citation chain is invoked as proof, no uniqueness theorem is imported from the authors' prior work, and no known result is renamed. The derivation chain is therefore self-contained in the relevant sense, and the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the Llama-3-70B base checkpoint, two cited growth methods, hand-tuned data ratios and hyperparameters, and an unexamined cleanliness assumption about the evaluation. None of these are independently established inside the paper; the ablations that would justify the key choices are referenced but not shown. No new theoretical entities are postulated; the trained model is an engineering artifact, not an invented entity in the ledger sense.

free parameters (6)
  • Korean:English data ratio = 9:1
    Section 4.1; hand-chosen to trade Korean gains against English retention; no ablation reported for this ratio.
  • Depth expansion factor = +20% (80 to 96 layers)
    Section 2 and Table 1; chosen after four Qwen 1.8B experiments whose results are not reported.
  • Width expansion sizes = model dim 8192 to 9216; FFN 28672 to 30720
    Section 2 and Table 1; no ablation reported for the width choices.
  • NEFTune alpha (SFT) = 8
    Section 5; chosen hyperparameter; NEFT alpha set to 0 during KTO.
  • KTO hyperparameters = batch 128, LR 1e-6, lambda_desired 1.375, lambda_undesired 1
    Section 5; tuned with cached policy logits to control toxicity; no sensitivity analysis shown.
  • Data filtering thresholds = 83.59% sample drop, 40.13% volume drop
    Section 4.3; pipeline unspecified and thresholds not justified.
assumptions (5)
  • domain assumption Llama-3-70B base weights remain a valid prior for the expanded model
    Section 2: the whole expansion assumes the pretrained checkpoint's capabilities survive LlamaPro depth and MSG width growth.
  • domain assumption KMMLU and KorMedMCQA are valid measures of Korean ability and generalize to non-benchmark Korean text
    Section 6; the paper validates on no other Korean tasks beyond these two benchmarks.
  • ad hoc to paper The 194B-token Korean corpus does not overlap the benchmark test sets
    Sections 4.2-4.3 vs 6.1-6.2; no decontamination reported, and the corpus includes exam-adjacent professional documents.
  • domain assumption English ability is retained with only a 10% English continued-pretraining share
    Section 4.1; untested because no English benchmark is reported anywhere.
  • standard math Standard Transformer training and optimization machinery behaves as in the cited literature
    Implicit throughout Sections 4-5 (optimizers, attention, RMSNorm, learning-rate schedules).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study." pith.science (2026). https://pith.science/paper/F7JE4VLR

@misc{pith2026250903972,
  author       = {Pith},
  title        = {Pith review of: Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7JE4VLR}},
  note         = {Machine review of arXiv:2509.03972}
}
read the original abstract

We introduce Llama-3-Motif, a language model consisting of 102 billion parameters, specifically designed to enhance Korean capabilities while retaining strong performance in English. Developed on the Llama 3 architecture, Llama-3-Motif employs advanced training techniques, including LlamaPro and Masked Structure Growth, to effectively scale the model without altering its core Transformer architecture. Using the MoAI platform for efficient training across hyperscale GPU clusters, we optimized Llama-3-Motif using a carefully curated dataset that maintains a balanced ratio of Korean and English data. Llama-3-Motif shows decent performance on Korean-specific benchmarks, outperforming existing models and achieving results comparable to GPT-4.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 5 canonical work pages

  1. [9]

    Kormedmcqa: Multi- choice question answering benchmark for korean healthcare professional licensing examinations

    Sunjun Kweon, Byungjin Choi, Minkyu Kim, Rae Woong Park, and Edward Choi. Kormedmcqa: Multi- choice question answering benchmark for korean healthcare professional licensing examinations. arXiv preprint arXiv:2403.01469, 2024

    arXiv 2024

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Exaone 3.0 7.8 b instruction tuned language model

    Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, et al. Exaone 3.0 7.8 b instruction tuned language model. arXiv e-prints, pages arXiv–2408, 2024

    work page 2024

  4. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [4]

    Accessed: 2024-09-28

    https://kr.linkedin.com/posts/jdh3577_llama31-gemma2-qwen2-%EB%93%B1-%EC%B5%9C% EA%B7%BC%EC%97%90%EB%8A%94-%ED%95%9C%EA%B5%AD%EC%96%B4%EB%8F%84-%EC%B6%A9%EB% B6%84%ED%9E%88-%EC%9E%98%ED%95%98%EB%8A%94-activity-7222200121843752961-aMwi? utm_source=li_share&utm_content=feedcontent&utm_medium=g_dt_web&utm_campaign= copy. Accessed: 2024-09-28

    work page 2024

  6. [5]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  7. [6]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    arXiv 2024

  8. [7]

    Neftune: Noisy embeddings improve instruction finetuning

    Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint arXiv:2310.05914, 2023

    arXiv 2023

Show all 18 references
  1. [8]

    Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling

    Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166, 2023

    2023 arXiv

  2. [10]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  3. [11]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 6

    2017 arXiv

  4. [12]

    Staged training for transformer language models

    Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. In International Conference on Machine Learning, pages 19893–19908. PMLR, 2022

    2022

  5. [13]

    Kmmlu: Measuring massive multitask language understanding in korean

    Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548, 2024

    2024 arXiv

  6. [14]

    Attention is all you need.(nips), 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.(nips), 2017. arXiv preprint arXiv:1706.03762, 10:S0140525X16001837, 2017

    2017 arXiv

  7. [15]

    Llama pro: Progressive llama with block expansion

    Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415, 2024

    2024 arXiv

  8. [16]

    Masked structural growth for 2x faster language model pre-training

    Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. Masked structural growth for 2x faster language model pre-training. arXiv preprint arXiv:2305.02869, 2023

    2023 arXiv

  9. [17]

    Hyperclova x technical report

    Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. Hyperclova x technical report. arXiv preprint arXiv:2404.01954, 2024

    2024 arXiv

  10. [18]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. A Appendix ...

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.