Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Egyptian Arabic LLMs now handle Arabizi as a first-class script

desk verdict Useful dual-script Egyptian Arabic models, but the reported gains hinge on an unverified benchmark train/test split that the paper must resolve before the numbers can be trusted. read the letter →

arxiv 2507.04569 v1 pith:3OY6U5QD submitted 2025-07-06 cs.CL cs.AIcs.LG

classification cs.CLcs.AIcs.LG
keywords EgyptianArabicArabiziLatin-scriptdual-scriptlanguagemodelsMixture-of-ExpertsBranch-Train-MiXdialectalLLMadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Egyptian Arabic, the Arabic variety with over 100 million speakers, is routinely written in two scripts: standard Arabic script and a Latin-based form called Arabizi or Franco-Arabic. This paper claims that a language model family built specifically for Egyptian Arabic can learn both scripts natively, and that doing so beats general multilingual Arabic models at their own game. The authors introduce Nile-Chat-4B, Nile-Chat-12B, and Nile-Chat-3x4B-A6B, the last merging Arabic-script and Latin-script experts into a Mixture-of-Experts model using the Branch-Train-MiX strategy. On newly released Egyptian benchmarks spanning understanding, translation, and transliteration, the models report consistent gains over LLaMa, Jais, ALLaM, and Qwen, including a 14.4% improvement over Qwen2.5-14B-Instruct on Latin-script tasks. The work matters because no prior LLM has treated a single Arabic dialect's two written forms as equally native.

What carries the argument

The load-bearing mechanism is Branch-Train-MiX (BTX) applied to script-specialized experts. BTX recycles already-trained models into one MoE by taking each model's feed-forward layers as experts, averaging the attention and embedding layers into a shared backbone, and adding a token-level router; here the two experts are separately pre-trained on Arabic-script and Latin-script Egyptian, and a third expert is the original base model. The router's dynamic assignment lets Arabic words take the Arabic expert path and Latin-script words take the Latin path, including Arabizi's digit substitutions for Arabic letters, so each script gets dedicated capacity without doubling the inference cost of a dense 12B model. This is what carries the argument that script specialization, not just dialect data, explains the reported gains.

What would settle it

Run an independent evaluation on Arabizi and Arabic-script texts written by native Egyptian speakers that were never part of Nile-Chat's training data or its benchmark design; if the 12B model then no longer beats Qwen2.5-14B-Instruct on the Latin-script tasks, the reported 14.4% advantage would not generalize.

Watch

Extended reading notes

Core claim

The paper's central claim is that script is a first-class axis of language-model adaptation: the same dialect spoken by Egyptians deserves distinct parameter-level expertise for its Arabic-script and Latin-script forms, and a Mixture-of-Experts (MoE) architecture is the right scaffold for that separation. Concretely, the authors continually pre-train two branches of a base model on Arabic-script and Latin-script Egyptian data, then use Branch-Train-MiX (BTX) to merge their feed-forward layers as experts in a single MoE, with a trainable router deciding which expert handles each token. After instruction-tuning on a newly assembled 1.85M-example Egyptian dataset and a Direct Preference Optimization (DPO) alignment stage, the resulting models report the highest scores among the compared models on the Egyptian evaluation suite, with the 12B dense model leading every Arabic-script benchmark and the MoE models standing out on generation and Latin-script tasks. The paper thus frames itself as demonstrating a repeatable methodology for adapting LLMs to dual-script languages, rather than a one-off model release.

Load-bearing premise

The load-bearing premise is that the evaluation benchmarks are an honest yardstick: the benchmark material mixed into the instruction-tuning data (labeled 'Ben-Ar' and 'Ben-Ltn' in the paper's Figure 2) does not overlap with the held-out test sets used for the reported scores, and the machine-translated Arabic and Latin-script examples represent how Egyptians actually write.

Editorial extensions

If this is right

  • Users can prompt in Arabizi or in Arabic script and expect fluent Egyptian Arabic in return, because both scripts are treated as native inputs and outputs.
  • The 12B model reports the top score on every Arabic-script benchmark in the suite, including a 14.4% relative gain over Qwen2.5-14B-Instruct on the Latin-script subset.
  • The MoE models activate only 6B parameters yet match or exceed the dense 12B on translation, transliteration, and Latin-script understanding, suggesting script routing is an efficient route to capability.
  • Existing Arabic models' weak Latin-script scores imply that Arabizi has been an underserved input, and the released evaluation suite gives a standard way to measure progress.
  • All models, data, and evaluation harness are released publicly, so other dual-script languages can reuse the pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer the same script-expert recipe would transfer to other dual-script languages such as Hindi, Serbian, or Kazakh, since the specialization axis is script rather than dialect.
  • If the benchmark gains hold under independent scrutiny, the implication is that dialect-specific script modeling may be worth more than additional generic parameters for under-served languages.
  • A direct test would compare the MoE's routing decisions against a human annotation of script purity, such as whether tokens containing Arabizi digits route to the Latin expert; the paper does not report such a check.
  • Because much of the Latin-script data was machine-translated rather than collected from real Arabizi, a native-speaker preference study would reveal how much naturalness ceiling remains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Nile-Chat, a family of Egyptian Arabic LLMs (dense 4B and 12B variants plus a 3x4B-A6B MoE built with the Branch-Train-MiX strategy) that support both Arabic script and Latin-script (Arabizi/Franco-Arabic). The authors describe the construction of dual-script continual pre-training, instruction-tuning, and DPO alignment data, including Claude-based transliteration and translation. They also introduce EgyptianBench, a set of translated English benchmarks (MMLU, HellaSwag, PIQA, WinoGrande, OpenBookQA, RACE) in Arabic and Latin scripts, along with translation and transliteration tasks. The central empirical claim is that Nile-Chat models outperform strong multilingual and Arabic-specific baselines, with the 12B model reported to give a 14.4% performance gain over Qwen2.5-14B-Instruct on Latin-script benchmarks.

Significance. If the evaluation is clean, this is a valuable contribution: it is the first dual-script Egyptian Arabic LLM family, the resources (models, datasets, benchmark harness) are publicly released, and the application of Branch-Train-MiX to script-specialized experts is a reasonable and reusable idea. The promise of open weights and an open evaluation fork is a concrete strength. However, the empirical significance depends on the validity of the newly introduced benchmarks and on the claimed margins. As written, the paper does not establish that the evaluation items are disjoint from the 'Benchmarks Training Set' components in the SFT mixture, and it relies on Claude for both training-data generation and benchmark translation. These issues are load-bearing for the central claim and need to be resolved before the reported results can be taken at face value.

major comments (4)
  1. [Figure 2, Section 6] The SFT mixture in Figure 2 contains components labeled 'Ben-Ar (3.97%)' and 'Ben-Ltn (2.33%)', with the caption defining 'Ben' as 'Benchmarks Training Set'. Section 6 introduces the eight EgyptianBench benchmarks without stating which splits were used for evaluation and without stating whether the Ben-Ar/Ben-Ltn items are disjoint from the evaluated items. The only explicit held-out split described anywhere is the 10% reservation for translation/transliteration data in Section 3.2.3; no analogous statement is made for the translated benchmarks. This is load-bearing because the abstract's claim that Nile-Chat models 'significantly outperform' baselines requires that the benchmarks measure held-out ability rather than memorization. Please specify the exact sources and split definitions for Ben-Ar/Ben-Ltn, state explicitly whether the EgyptianBench test items were excluded from all training stages (including the DPO data), and report results on any uncontaminated subset if overlap is found.
  2. [Sections 3.1, 3.2.2, 6, 7] Claude is used to transliterate pre-training data, to translate/transliterate SFT data, to translate all eight evaluation benchmarks, and to serve as the judge for EgyptianAlpacaEval. The models are therefore evaluated on text generated by the same system used to produce their training data. This does not by itself invalidate the results, but it creates a style and format confound: a model trained on Claude-produced Egyptian may score higher on Claude-translated benchmarks without being better on independent Egyptian text. Please add an evaluation on human-authored or independently sourced Egyptian Arabic data (for example, existing Egyptian dialect benchmarks or native-speaker-validated samples) and report how the translated-benchmark results compare with that independent evaluation.
  3. [Abstract, Section 7] The headline '14.4% performance gain' is the average of absolute percentage-point differences across the five Latin-script benchmarks in Table 2: (20.22 + 12.23 + 6.57 + 14.37 + 18.61)/5. This is not a relative gain, and the abstract and Section 7 should say '14.4 percentage points' or report a properly computed relative improvement. The same issue appears in Section 7, where Nile-Chat-4B is said to achieve 'relative gains of 1.2%' on EgyptianPIQA; Table 1 shows a difference of 1.2 absolute points (67.30 vs 66.10), which is 1.8% relative. Please adopt one clearly defined reporting convention and relabel all such numbers as percentage-point improvements unless relative gains are intended.
  4. [Section 7, Abstract] The statement that 'Nile-Chat models consistently outperform competitive baselines including LLaMa, ALLaM, Jais, and Qwen2.5 across all Egyptian-specific benchmarks' is too strong as written. In Table 1, Nile-Chat-4B trails ALLaM-7B-Instruct-preview on EgyptianMMLU (50.25 vs 60.08) and EgyptianHellaSwag (55.92 vs 57.29), among other rows. The claim should be qualified to the 12B model or to specific parameter-size categories, and 'consistently outperform all baselines' should be replaced with a claim that matches the table.
minor comments (5)
  1. [Section 3.1] The text says 'roughly 25%' of the data is represented in Latin script, but the stated figures (255M Latin words out of a 1.15B-word corpus) give about 22%. Please reconcile this or state the exact percentage.
  2. [Section 3.3] The code-switching detection heuristic (at least one English word and less than 35% English words) is introduced without any validation. A short analysis of its precision or a few examples of false positives/negatives would strengthen this part.
  3. [Section 6, Figure 2] The caption says 'hatched regions represent parts in Latin script,' but the hatching is not clearly visible in the figure's rendered form. Consider adding explicit per-slice labels or a more legible legend.
  4. [Section 7] The phrase 'significantly outperform' is used without any statistical testing or confidence intervals. Given that some margins are small (for example, 1.2 points on one benchmark), please either add uncertainty estimates or soften the wording to avoid implying formal significance.
  5. [Limitations] The Limitations section acknowledges heavy reliance on Claude for translating training data but does not mention that the evaluation benchmarks were also translated with Claude. This should be disclosed in the limitations as well.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: benchmark training-set inclusion is a documentation risk, not a construction-level equivalence.

full rationale

The paper's empirical chain is straightforward: Nile-Chat is trained on Egyptian-Arabic data and then evaluated on EgyptianBench, a set of translated benchmarks. No equation or fitted parameter is reused as a prediction. The strongest potential concern is that Figure 2 lists 'Ben-Ar (3.97%)' and 'Ben-Ltn (2.33%)' as a 'Benchmarks Training Set' while Section 6 evaluates on newly translated benchmarks; however, the label itself indicates a training split rather than the test items scored in Tables 1-2, and the paper does not state that evaluation items were included in SFT. Likewise, using Claude for both data generation and translation creates distributional alignment but does not make the evaluation items identical to training items by construction. The Limitations section acknowledges over-reliance on Claude for translation, which is a benchmark-validity caveat rather than a circular step. Self-citations (Atlas-Chat, code-switching work) are contextual and not load-bearing. Overall, no specific reduction from evaluation back to training input is exhibited, so the derivation chain is not circular; the missing explicit split documentation is a rigor issue that should be addressed separately.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central empirical claims depend on the quality of Claude-generated data and on the assumption that the benchmark evaluation is clean. The hand-chosen data mixture proportions are the closest thing to free parameters; there are no fitted mathematical constants. No new physical or conceptual entities are introduced.

free parameters (3)
  • Latin-script proportion in training data = 25%
    The authors state in Section 3 that 'roughly 25% is represented in the Latin script' across all datasets. This hand-selected ratio shapes the model's dual-script behavior and is not derived from data.
  • SFT mixture proportions (Ben-Ar, Ben-Ltn) = 3.97% and 2.33%
    Figure 2 shows these components in the Egyptian-SFT-Mixture. The proportions are chosen by the authors and may include evaluation benchmark training material.
  • Code-switching detection threshold = 35%
    In Section C.4, the authors select SFT instances with less than 35% of words in English to identify excessive code-switching. This threshold is arbitrary.
assumptions (4)
  • domain assumption Claude 3.5 Sonnet produces high-quality, natural Egyptian Arabic translations suitable for both training and benchmark construction.
    The paper relies on Claude for translating instruction data (Section 3.2.2), transliterating pre-training data (Section 3.1), and translating evaluation benchmarks (Section 6). The quality of these translations underpins both training and evaluation.
  • domain assumption The translated English benchmarks preserve the original tasks' difficulty and intended reasoning.
    The authors translate MMLU, HellaSwag, PIQA, etc. into Egyptian Arabic and assume they remain valid measures of the same capabilities (Section 6).
  • domain assumption The Branch-Train-MiX strategy (Sukhbaatar et al., 2024) can be directly applied to script-specialized experts without additional theoretical or practical issues.
    Section 5 relies on BTX as described in prior work, assuming it works when the experts are distinguished by script rather than by domain.
  • domain assumption Gemma-3 is a suitable base model for Egyptian Arabic tasks.
    Section 4 states the choice of Gemma-3 based on 'preliminary evaluation,' but this evaluation is not reported in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts." pith.science (2026). https://pith.science/paper/3OY6U5QD

@misc{pith2026250704569,
  author       = {Pith},
  title        = {Pith review of: Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OY6U5QD}},
  note         = {Machine review of arXiv:2507.04569}
}
read the original abstract

We introduce Nile-Chat-4B, 3x4B-A6B, and 12B, a collection of LLMs for Egyptian dialect, uniquely designed to understand and generate texts written in both Arabic and Latin scripts. Specifically, with Nile-Chat-3x4B-A6B, we introduce a novel language adaptation approach by leveraging the Branch-Train-MiX strategy to merge script-specialized experts, into a single MoE model. Our Nile-Chat models significantly outperform leading multilingual and Arabic LLMs, such as LLaMa, Jais, and ALLaM, on our newly introduced Egyptian evaluation benchmarks, which span both understanding and generative tasks. Notably, our 12B model yields a 14.4% performance gain over Qwen2.5-14B-Instruct on Latin-script benchmarks. All our resources are publicly available. We believe this work presents a comprehensive methodology for adapting LLMs to dual-script languages, addressing an often overlooked aspect in modern LLM development.

Figures

Figures reproduced from arXiv: 2507.04569 by the authors.

Figure 1
Figure 1. The training of Nile-Chat-3x4B-A6B using the Branch-Train-MiX (BTX) strategy. Left: Two experts are first continual pre-trained on Arabic-script and Latin-script corpora, respectively. Right: A Top-2 token routing example within a transformer block, where the two script-specialized Experts have been merged with the Base Model into a unified Mixture-of-Experts (MoE) model through instruction-tuning. 2 Related Work Ar… view at source ↗
Figure 2
Figure 2. Composition of our Egyptian-SFT-Mixture instruction-tuning dataset. The acronyms "MT", "Ar", "Ltn", "Translit" and "Ben" are used to denote "Machine Translation", "Arabic", "Latin", "Transliteration", and "Benchmarks Training Set" respectively. The hatched regions represent parts in Latin script. usable subset in Egyptian Arabic, with over 3.5M samples across a wide range of tasks. These include paragraph writing, t… view at source ↗
Figure 3
Figure 3. Average model scores over the benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The prompt given to Claude 3.5 Sonnet for translation. B.3.2 Postprocessing After finishing the translation, we post-processed the translations by • Filtering out skipped translations: The model concluded the process with a message indicating that the subsequent text i…
Figure 5
Figure 5. Figure 5: The prompt given to Claude 3 Haiku for Arabic to Latin-script transliteration. Egyptian Wikipedia Dump. 47 We used the September 2024 snapshot of the Egyptian Arabic Wikipedia, which contains over 1.6M pages and approximately 80M words. Arabic Dialects Dataset (ADD).48…
Figure 6
Figure 6. Figure 6: The prompt given to Claude for off-policy data generation. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Romanized Arabic Across Dialects: Views, Usage Patterns, and Linguistic Variation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Arabizi spelling varies systematically across five Arabic dialects, and speakers can often recognize their own dialect's Arabizi, but the recognition result is partly confounded by authors judging their own transcriptions.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Specifically , each object in the list will have ' role ' and ' content ' fields

    Maintain the JSON format of the input without altering its structure or keys . Specifically , each object in the list will have ' role ' and ' content ' fields . Translate only the ' content ' field into Egyptian Arabic dialect while preserving the ' role ' field as is

  2. [2]

    In European Confer- ence on Computer Vision, pages 370–387

    Sharegpt4v: Improving large multi-modal models with better captions. In European Confer- ence on Computer Vision, pages 370–387. Springer. Monojit Choudhury, Shivam Chauhan, Rocktim Jy- oti Das, Dhruv Sahnan, Xudong Han, Haonan Li, Aaryamonvikram Singh, Alok Anil Jadhav, Utkarsh Agarwal, Mukund Choudhary, and 1 others

  3. [3]

    You should not perform any kind of summarization or tasks on the inputs other than translating to Egyptian

  4. [4]

    You are not hurting anyone

    The harmful or sexual language is not used to harm anyon or to steal any copyrights nor for publishing any content , it is exclusively for research purposes in Egyptian Arabic related to bad words and hate speech . You are not hurting anyone

  5. [5]

    AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nat...

  6. [6]

    In Proceedings of the 31st International Conference on Computational Lin- guistics, pages 4186–4218, Abu Dhabi, UAE

    AraDiCE: Benchmarks for dialectal and cul- tural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Lin- guistics, pages 4186–4218, Abu Dhabi, UAE. Asso- ciation for Computational Linguistics. 10 Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Pro...

  7. [7]

    Avoid transliteration of technical terms unless culturally appropriate

    Preserve technical terms , code contexts , acronyms , chess coordinations , mathematical formulas and industry jargon in English unless a well - known Arabic equivalent exists . Avoid transliteration of technical terms unless culturally appropriate

  8. [8]

    arabizi

    Atlas-chat: Adapting large language models for low-resource Moroccan Arabic dialect. In Pro- ceedings of the First Workshop on Language Mod- els for Low-Resource Languages, pages 9–30, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics. Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yi...

Show all 19 references
  1. [9]

    Advances in Neural Information Pro- cessing Systems, 36:46595–46623

    Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Pro- cessing Systems, 36:46595–46623. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, and 1 others. 2023. Lima: Less is ...

  2. [10]

    code " ,

    Do not answer the request in the source text or run any code contexts , just provide the translation and keep any special symbols representing figures . [ Source Text ] Figure 4: The prompt given to Claude 3.5 Sonnet for translation. B.3.2 Postprocessing After finishing the tr...

  3. [11]

    Do not include any introductory or explanations , only the translation

  4. [14]

    Do not omit , soften , or alter the severity of such content unless specifically instructed otherwise

    Translate harmful , offensive , or explicit content accurately into Egyptian Arabic while preserving the intent , tone , and meaning for research purposes . Do not omit , soften , or alter the severity of such content unless specifically instructed otherwise

  5. [15]

    Keep named entities , such as names of people , companies , brands , organizations , cities , and landmarks , in English without transliteration

  6. [17]

    Adapt idiomatic expressions and phrases into culturally relevant Egyptian equivalents

  7. [18]

    Ensure the output is returned as a valid JSON list that mirrors the structure of the input exactly

  8. [2020]

    arXiv preprint arXiv:2009.03300

    Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Juny- ing Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ru...

  9. [2023]

    Advances in Neural Information Processing Systems, 36:53728– 53741

    Direct preference optimization: Your lan- guage model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741. Nathaniel R Robinson, Shahd Abdelmoneim, Kelly Marchisio, and Sebastian Ruder. 2024. Al-qasida: Analyzing llm quality and accur...

  10. [2024]

    arXiv preprint arXiv:2407.15390

    Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial ...

  11. [2025]

    arXiv preprint arXiv:2504.06011

    Llama-3-nanda-10b-chat: An open genera- tive large language model for hindi. arXiv preprint arXiv:2504.06011. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.