Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Krikri: Advancing Open Large Language Models for Greek

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Greek-focused 8B model, built from Llama 3.1 by continual pretraining and synthetic-data post-training, outperforms open multilingual rivals on Greek tasks and holds its own against models three to four times its size.

desk verdict The base-model Greek gains are the credible story here; the chat-benchmark wins are real but rest on translated tests and a judge the paper itself admits may be biased. read the letter →

arxiv 2505.13772 v2 pith:RQTAL4YC submitted 2025-05-19 cs.CL

Dimitris Roussis , Leon Voukoutis , Georgios Paraskevopoulos , Sokratis Sofianopoulos , Prokopis Prokopidis , Vassilis Papavasileiou , Athanasios Katsamanis , Stelios Piperidis
show 1 more author
Vassilis Katsouros
This is my paper · ORCID
classification cs.CL
keywords GreeklanguagemodelcontinualpretrainingtokenizerexpansioninstructiontuningdirectpreferenceoptimizationsyntheticdatabenchmarksmultilingualLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Llama-Krikri-8B is an open Greek-language model built by taking Meta's Llama 3.1-8B and continually pretraining it on roughly 110 billion tokens, about 60 percent of them Greek, after expanding the tokenizer with 20,992 Greek-domain tokens. The paper's central claim is that this adaptation, followed by a multi-stage post-training pipeline using synthetic instruction data and preference optimization, makes an 8-billion-parameter model the strongest open choice for Greek, ahead of both the previous Greek model and general multilingual models of similar size. The instruct version also posts competitive scores against models with three to four times more parameters on Greek benchmarks. To make the case, the paper introduces three new Greek benchmarks built by translating and post-editing IFEval, MT-Bench, and Arena-Hard. A fair reader would care because low- and medium-resource languages rarely get open models with this level of capability, and the recipe could be reused.

What carries the argument

The mechanism is a full adaptation pipeline: a domain-expanded tokenizer (149,248 tokens; Greek fertility falls from 2.73 to 1.65 tokens per word), embedding initialization by averaging the old tokenizer's embeddings over each new token, mixed-curriculum continual pretraining on a 110B-token mix with English and code replay segments, an annealing phase on curated and synthetic question-answer data, then two-stage supervised fine-tuning and direct preference optimization with length normalization for the chat version. The evaluation machinery consists of three newly translated and post-edited Greek benchmarks (IFEval Greek, MT-Bench Greek, Arena-Hard Greek) plus six existing Greek tasks and six English tasks; the Greek Arena-Hard and MT-Bench use GPT-4o family models as judge.

What would settle it

Have native Greek speakers blindly rank Llama-Krikri-8B-Instruct, Aya Expanse 32B, and Gemma 2 27B on the same MT-Bench Greek and Arena-Hard Greek prompts, then compare those rankings with GPT-4o's scores; a ranking reversal would show the headline chat-model lead is partly a judge artifact.

Watch

Extended reading notes

Core claim

According to the paper, continual pretraining on a Greek-heavy corpus is enough to make Llama 3.1-8B substantially better at Greek without giving back English competence. The base model averages 59.5 percent accuracy on six Greek benchmarks versus 48.7 percent for Llama-3.1-8B and 47.9 percent for Meltemi-7B, and it even edges the base model on English by 0.8 points on average. The instruct variant reaches 67.5 percent on IFEval Greek, 7.96 on MT-Bench Greek, and a 31.8 percent win rate on Arena-Hard Greek, beating Aya Expanse 8B and Llama-3.1-70B-Instruct while matching Gemma 2 27B on the Greek Arena-Hard. The paper reads these results as evidence that data synthesis and careful post-training can close the resource gap for a medium-resource language.

Load-bearing premise

The load-bearing premise is that the translated Greek benchmarks and the GPT-4o judge measure Greek capability fairly; if the judge favors responses resembling its own training style, or if translation artifacts make the tests easier for models trained on English-distilled data, the reported leads over other open models shrink.

Editorial extensions

If this is right

  • An 8B Greek model reaches 67.5 percent on IFEval Greek, surpassing Llama-3.1-8B-Instruct by 21.7 points and the previous Greek model Meltemi by 34.8 points.
  • On Arena-Hard Greek, the instruct model's 31.8 percent win rate puts it ahead of Llama-3.1-70B-Instruct at 27.4 percent despite being roughly one-ninth the size.
  • The base model improves Greek benchmark accuracy by 10.8 points over Llama-3.1-8B while retaining and slightly improving English performance by 0.8 points, so catastrophic forgetting is not observed in this setup.
  • The expanded tokenizer halves Greek token consumption, directly lowering inference cost for Greek users.
  • The model extends to polytonic and Ancient Greek, with a reported 54.66 BLEU score for ancient-to-modern Greek translation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the three new chat benchmarks are translations of English tests, part of the measured gain may reflect improved English-style instruction following rather than Greek-specific linguistic skill; a fully original Greek benchmark would isolate the language-specific contribution.
  • The same pipeline of tokenizer expansion, curriculum continual pretraining, synthetic instruction data, and DPO should transfer to other medium-resource European languages that share a strong multilingual base model.
  • The paper's own caveat about judge bias implies that the Arena-Hard Greek margins over larger models could shrink if a judge not derived from GPT-4-family training were used; a human-preference replication would settle how much of the lead is real.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Llama-Krikri-8B, a Greek-focused large language model obtained by continual pretraining of Meta's Llama 3.1-8B on a curated 110B-token corpus, with tokenizer and embedding expansion, an annealing phase, and a multi-stage SFT/DPO post-training pipeline. The paper also presents three new Greek chat benchmarks (IFEval Greek, MT-Bench Greek, Arena-Hard Greek) and evaluates base and instruct models on Greek and English tasks, reporting substantial gains over existing open multilingual models and competitiveness with models 3-4 times larger, as well as an Ancient-Modern Greek translation experiment. The central claim is that Llama-Krikri-8B and Llama-Krikri-8B-Instruct significantly outperform comparable open models on Greek tasks while retaining English capabilities.

Significance. If the evaluation concerns are addressed, this is a useful contribution to Greek NLP: the paper releases open models, a detailed training recipe, tokenizer statistics, and three public Greek evaluation benchmarks, and the base-model Greek improvements (Table 3, +10.8 average over Llama-3.1-8B) and English retention (Table 4, +0.8) are concrete and reproducible with established benchmarks. The strengths are the transparency of the corpus and pipeline description, the public release of benchmarks, and the careful attention to tokenizer fertility for Greek. The main weaknesses are that the headline chat-model superiority rests on translated and LLM-judged benchmarks whose validity is not yet established, and no uncertainty quantification is provided for any of the reported scores.

major comments (4)
  1. [Section 4.2, Table 7] The Greek Arena-Hard win rates (31.8 for Llama-Krikri-8B-Instruct vs 4.0 for Llama-3.1-8B-Instruct) are computed with GPT-4o-Mini as the baseline and GPT-4o as the judge, and the paper itself cites Li et al. (2025) showing that judges are biased toward student models trained on distilled teacher data. Since Llama-Krikri-8B-Instruct was post-trained on MAGPIE and other synthetic data from GPT-4-class teachers, this judge-bias confound directly affects the headline gap; the claim of significant outperformance on this benchmark is not load-bearing without a human-preference check or a second judge with an independent decision rule. I would ask for a robustness analysis on a random subset of prompts (e.g., 50-100) with human ratings, or at minimum a sensitivity analysis using a different judge and an explicit agreement metric.
  2. [Section 4.2, IFEval Greek / MT-Bench Greek / Arena-Hard Greek] The three newly proposed benchmarks are translations and post-edits of English benchmarks, and Section 6 acknowledges that future evaluation should include original Greek datasets to minimize translation effects. The manuscript reports human post-editing but provides no inter-annotator agreement, no translation quality metrics, and no comparison with an original-Greek benchmark; translated prompts can change difficulty because Greek morphology and word-order constraints differ from English. Therefore the size of the Greek IFEval gap (+21.7 over Llama-3.1-8B-Instruct) and the MT-Bench Greek lead are not yet established as true Greek-capability differences. I would ask for translation quality statistics, a back-translation consistency check, or a comparison against a non-translated Greek instruction-following set before treating these benchmarks as decisive.
  3. [Tables 3-7] All reported scores are single-run point estimates with no error bars, confidence intervals, or significance tests; differences such as the MT-Bench Greek 7.96 vs 7.68 for Aya Expanse 8B, or the Open LLM Leaderboard average 24.18 vs 23.76, may be within run-to-run noise. The word "significantly" is used repeatedly (Section 4.2, Section 5) without statistical support; I would ask for repeated evaluations or bootstrap CIs on the main comparisons, or a downgrading of the wording to "directionally outperforms."
  4. [Section 4.2, Ancient-Modern Greek translation] The claim of highly accurate handling of Ancient Greek rests on a 100-sentence set with only BLEU scores (54.66 grc->ell, 20.41 ell->grc) and no baseline scores from Llama-3.1-8B-Instruct, Meltemi, or a translation system, and no human evaluation. This is too small and too weakly benchmarked to support the contribution bullet about Ancient Greek; I would ask for baselines on the same set, a larger evaluation sample, and ideally human judgments of adequacy and fluency.
minor comments (6)
  1. [Section 3.2, Appendix A.3, Table 8] The caption contains the typo "extention" which should read "extension."
  2. [Section 4.2, Table 5] The row labeled "Llama-Krikri-8B" should be labeled "Llama-Krikri-8B-Instruct" to match the other chat models in the table and avoid ambiguity with the base model.
  3. [Section 4.2, MT-Bench Greek] The description says the benchmark contains 80 multi-turn conversations but does not state whether the full MT-Bench set was used or how many prompts were discarded during post-editing; please specify the exact source and filtering process.
  4. [Section 4.2, Arena-Hard Greek] The manuscript mentions using the style-control version of Arena-Hard but does not give the exact revision or style-control parameters; please add this to the appendix for reproducibility.
  5. [Section 1 and Section 4] The introduction and contributions mention function calling and agentic behavior, but no evaluation of function calling is reported; either add such an evaluation or soften the claim.
  6. [Section 6, Limitations] The sentence stating that future benchmarks should include original Greek datasets that are not the result of machine translation and post-editing directly qualifies the central comparative claims and should be moved or expanded in the main evaluation section, not only in the limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported scores are trained outputs measured on held-out benchmarks; author-constructed Greek chat benchmarks and LLM-judge effects are validity risks, not circular reductions.

full rationale

The paper's central claims are empirical: Llama-Krikri-8B is continually pretrained on a large Greek corpus, post-trained with SFT and DPO, and then evaluated on Greek and English benchmarks. The evaluation numbers (Tables 3-7) are model outputs on test sets, not quantities derived from the model's own training objective or from fitted parameters renamed as predictions. There is no equation in the paper in which a claimed result is defined in terms of the quantity it is supposed to predict, and no fitted parameter is presented as an independent prediction. The three new Greek chat benchmarks (IFEval Greek, MT-Bench Greek, Arena-Hard-Auto Greek) are translated and post-edited versions of existing English benchmarks; they were not mentioned as training inputs, and the paper does not claim that excluding benchmark translations was part of the training protocol explicitly, but neither does it state that these benchmarks were used for training. The acknowledged risk that GPT-4o as judge may favor models trained on teacher-distilled data (Section 4.2, citing Li et al., 2025) is a benchmark-validity concern, not a circularity: it does not make the evaluation score equal to any training input by construction. Similarly, the Limitations section's statement that future work should include original Greek datasets 'not the result of machine translation and post-editing' concedes a measurement concern, but conceding a limitation is not circular reasoning. The self-citations to Meltemi (Voukoutis et al., 2024) provide the prior Greek model and a Greek evaluation suite, but the present model's advantages over Meltemi are measured on shared benchmarks with reported numbers; the comparison is not load-bearing in the sense of assuming the conclusion. Overall, the derivation chain is self-contained: training data and procedures are described, and evaluation is conducted on held-out tasks with concrete scores. Therefore no circular step meeting the quoted-evidence standard is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central comparisons rest on a set of hand-selected training choices (corpus mix, token allocations, data volumes) and external assumptions about base model transfer, synthetic data quality, and evaluation validity. None of these choices is fitted to the evaluation benchmarks, but they determine the magnitude of the reported gains.

free parameters (4)
  • Pretraining corpus composition = Greek 62.3%, English 23.1%, parallel 6.0%, math/code 8.6% (upsampled to 110.2B tokens)
    Hand-chosen mix to balance Greek adaptation with retention of English, math, and code capabilities.
  • Tokenizer token allocation per domain = General 15,000, Legal 4,000, Scientific 1,000, Literature 500, Ancient Greek 492 (total 20,992)
    Manually allocated to reduce fertility across domains; not optimized on downstream benchmarks.
  • Annealing dataset size = 3.5B curated + 189M synthetic QA tokens
    Chosen based on ablation (Table 11) showing +2.1 Greek and +0.8 English over continual pretraining.
  • SFT/DPO data sizes = 856K pairs (SFT1), 638K pairs (SFT2), 92K preference triplets (DPO)
    Quantities selected by the authors; no search reported.
assumptions (4)
  • domain assumption Llama-3.1-8B is a strong multilingual base model whose knowledge can be extended to Greek without destroying English capabilities.
    The entire approach relies on the base model retaining and transferring knowledge during continual pretraining (Section 3.4).
  • domain assumption MAGPIE-style synthetic instruction and preference data, filtered by a reward model, improve instruction following and alignment for Greek.
    Synthetic data is used at scale for SFT and DPO (Section 3.6); the paper provides no controlled comparison showing synthetic data alone drives the gains.
  • domain assumption GPT-4o provides valid, unbiased judgments of Greek response quality in MT-Bench and Arena-Hard.
    All chat quality scores in Tables 5 and 7 come from GPT-4o as judge; the paper acknowledges possible judge bias.
  • domain assumption Translated and post-edited versions of IFEval, MT-Bench, and Arena-Hard preserve the properties of the original benchmarks.
    The three new Greek benchmarks are translations; validity is assumed rather than demonstrated (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Krikri: Advancing Open Large Language Models for Greek." pith.science (2026). https://pith.science/paper/RQTAL4YC

@misc{pith2026250513772,
  author       = {Pith},
  title        = {Pith review of: Krikri: Advancing Open Large Language Models for Greek},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQTAL4YC}},
  note         = {Machine review of arXiv:2505.13772}
}
read the original abstract

We introduce Llama-Krikri-8B, a cutting-edge Large Language Model tailored for the Greek language, built on Meta's Llama 3.1-8B. Llama-Krikri-8B has been extensively trained on high-quality Greek data to ensure superior adaptation to linguistic nuances. With 8 billion parameters, it offers advanced capabilities while maintaining efficient computational performance. Llama-Krikri-8B supports both Modern Greek and English, and is also equipped to handle polytonic text and Ancient Greek. The chat version of Llama-Krikri-8B features a multi-stage post-training pipeline, utilizing both human and synthetic instruction and preference data, by applying techniques such as MAGPIE. In addition, for evaluation, we propose three novel public benchmarks for Greek. Our evaluation on existing as well as the proposed benchmarks shows notable improvements over comparable Greek and multilingual LLMs in both natural language understanding and generation as well as code generation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MORFES: A Benchmark for Productive Inflectional Competence in Modern Greek

    cs.CL 2026-07 conditional novelty 6.0 of 10

    MORFES is the first expert-verified Modern Greek productive-inflection benchmark; Sophea-Genesis-1 leads it at 84% per-item production without losing general capability.

Reference graph

Works this paper leans on

23 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Greek Texts (56.7B tokens): The Greek part of the dataset was sourced from publicly available resources spanning a wide range of domains and sources: Wikipedia, ELRC-SHARE (Lösch et al., 2021), EUR-LEX & MultiEUR-LEX (Chalkidis et al., 2019, 2021), MaCoCu (Bañón et al., 2022), CLARIN-EL (Gavriilidou et al., 2023), EMEA21, parliamentary proceedings (Erjave...

    work page 2021

  2. [2]

    English Texts (21B tokens):A subset of high-quality English data was mixed into the training corpus. This subset ensures that the model is continually trained on English data, and is drawn from sources that were also used for the Greek data, such as Wikipedia, Wikisource, Project Gutenberg (post- 1900), EUR-LEX, EMEA, Greek academic repositories, etc. We ...

    work page 1900

  3. [3]

    Parallel Data (5.5B tokens): We compiled a diverse parallel corpus with language pairs covering multiple languages: Greek, English, French, Portuguese, German, Spanish, and Italian. The decision to add parallel data which covers other European languages (i.e., German, French, Italian, Portuguese, and Spanish) is informed from the languages that have been ...

    work page 2024

  4. [4]

    Code and Math (7.8B tokens): We also integrated datasets containing text with code and mathemat- ics, leveraging Stack Overflow30, Python-Edu which is a subset of the SmolLM corpus (Allal et al.,

    work page

  5. [5]

    Ancient Greek which contains texts only in Ancient Greek. • Filtering and Preprocessing: Each dataset underwent sequential processing and filtering including language identification verification with FastText (Joulin et al., 2017), application of regular expres- sions to remove URLs and other anomalies, symbol-to-word ratio filtering to remove outliers, a...

    work page 2017

  6. [6]

    In Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC- COLING 2024), pages 5473–5483

    Ecthr-pcr: A dataset for precedent understand- ing and prior case retrieval in the european court of human rights. In Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC- COLING 2024), pages 5473–5483. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N...

    arXiv 2024

  7. [7]

    Preprint, arXiv:2407.20743

    Meltemi: The first open large language model for greek. Preprint, arXiv:2407.20743. Zhilin Wang, Alexander Bukharin, Olivier Delal- leau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Olek- sii Kuchaiev, and Yi Dong. 2024a. Helpsteer2- preference: Complementing ratings with preferences. Preprint, arXiv:2410.01257. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Z...

    arXiv

  8. [8]

    In Proceedings of the 2022 ACM Conference on Fair- ness, Accountability, and Transparency, FAccT ’22, page 214–229, New York, NY , USA

    Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fair- ness, Accountability, and Transparency, FAccT ’22, page 214–229, New York, NY , USA. Association for Computing Machinery. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yun- tian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data syn...

    arXiv 2022

Show all 23 references
  1. [13]

    AutoMathText has undergone an automatic selection process using Qwen-72B (Bai et al.,

    originating from The Stack V2 dataset (Lozhkov et al., 2024) and having been scored with an educational code classifier, and the AutoMathText dataset (Zhang et al., 2024b), which is a collection of math-related documents originating from web data, papers on arXiv, and code/not...

    2024

  2. [14]

    lorem ipsum

    for relevancy to the mathematical domain and the educational value of each document. Code and Mathematics data, although not specific to Greek, were included to preserve and enhance the model’s ability to handle coding tasks, math problems and formal language. Maintaining thes...

    2017

  3. [15]

    General domain which reuses a sampled portion of the data used to train the tokenizer of Meltemi (V oukoutis et al., 2024) and covers diverse domains,

    2024

  4. [16]

    Legal domain which includes legal texts

  5. [17]

    Scientific domain which uses publicly available articles, theses, and dissertations

  6. [18]

    Literature domain from public-domain literature, poetry, and other original writings across various variants of Greek (e.g., Koine Greek, Medieval Greek, Modern Greek, etc.),

  7. [21]

    ΄Εστω x ο αρχικός αριθμός μήλων που είχε η Αλίκη

  8. [22]

    Γνωρίζουμε ότι: * Δίνει 2/5 του x στην αδερφή της * Δίνει 1/5 του x στον αδερφό της * Μετά από όλα αυτά της μένουν 6 μήλα

  9. [23]

    Ας κάνουμε την εξίσωση : * Αρχικά είχε: x μήλα * Της μένουν: 6 μήλα * ΄Αρα: x - (2/5)x - (1/5)x = 6 * Απλοποιώντας: x - (3/5)x = 6 * Συνδυάζοντας όρους: (2/5)x = 6 * Πολλαπλασιάζοντας και τα δύο μέλη με 5/2: * x = 15 Ας επαληθεύσουμε: * Αρχικά είχε 15 μήλα * Δίνει: 2/5 * 15 = ...

  10. [2017]

    "" # ΄Ελεγχος εγκυρότητας εισόδου if not isinstance(n, int) or n < 0: raise ValueError (

    and we masked the cross-entropy loss of prompt tokens so the model would only learn based on the completions. For the DPO phase, we applied length normalization as described by Rafailov et al. (2024). At its core the DPO objective maximizes the likelihood of preferred response...

    2024

  11. [2021]

    In International Conference on Learning Representations

    Measuring massive multitask language under- standing. In International Conference on Learning Representations. Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger kernel: Efficie...

    2024 arXiv

  12. [2022]

    The ParlaMint corpora of parliamentary pro- ceedings. Lang. Resour . Eval., 57(1):415–448. Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface. co/spaces/open-llm-leaderboard/open_llm_ leaderbo...

    2024 arXiv

  13. [2023]

    In The Eleventh International Confer- ence on Learning Representations

    Quantifying memorization across neural lan- guage models. In The Eleventh International Confer- ence on Learning Representations. OpenReview. Ilias Chalkidis, Manos Fergadiotis, and Ion Androut- sopoulos. 2021. Multieurlex – a multi-lingual and multi-label legal document class...

    2021 arXiv

  14. [2024]

    Preprint, arXiv:2305.18290

    Direct preference optimization: Your lan- guage model is secretly a reward model. Preprint, arXiv:2305.18290. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, A...

    2022 arXiv

  15. [2025]

    Preprint, arXiv:2502.02737

    Smollm2: When smol goes big – data- centric training of a small language model. Preprint, arXiv:2502.02737. Duarte M Alves, José Pombal, Nuno M Guerreiro, Pe- dro H Martins, João Alves, Amin Farajian, Ben Pe- ters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. 2024. To...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.