{"id":"2e999505-6738-46f6-999a-80d6d0220e31","arxiv_id":"2412.04277","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 1.6B Arabic language model trained with synthetic multiple-choice instruction data outperforms 7B-13B models on several Arabic multiple-choice benchmarks.","lead":"The authors fine-tuned the small Stable LM 2 1.6B model on 114 billion Arabic tokens, then instruction-tuned it with synthetic dialogue data to create Arabic Stable LM. The resulting 1.6B chat model beats models 8 times larger on several Arabic multiple-choice benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No contamination analysis is reported; with 114B tokens of web-crawl pretraining and rephrased instruction data, benchmark items may be memorized, so the 1.6B-vs-13B claim is unsecured.","rationale":"The reader's weakest assumption matches my read: the claim that a 1.6B chat model outperforms 7–13B models stands or falls on whether the public benchmarks leaked into the 114B-token web-crawl corpus or the 183k rephrased dialogues. This is not an accusation of bad faith; it is an unaddressed confound in the reported pipeline. I considered two other candidate concerns and find them secondary. First, the cloze-format evaluation may favor a model fine-tuned on MCQ-style rephrased data, but the authors disclose this and ablate it (Table 8), so it is a transparency issue rather than a hidden flaw—though it makes the contamination question sharper, since the MCQ rephrasing is generated from the same documents as pretraining. Second, the baseline evaluation procedure is underspecified (no eval code, unknown prompt details), but the paper's own CF-vs-MCF plots suggest the authors evaluated baselines in both formats; this is less decisive than a missing decontamination check. The proposed test—exact n-gram overlap plus score recomputation after removing leaked items—would settle the concern. Because the evidence as presented does not rule out contamination, the CONDITIONAL verdict is appropriate; I do not recommend changing it.","tokens_in":19305,"tokens_out":4446,"duration_ms":45085,"concrete_test":"Compute exact 13-gram overlap (Arabic-normalized) between each eval benchmark's prompts/answers and (a) the cleaned 114B-token Arabic corpus and (b) the 183k rephrased dialogues; then re-run the Table 6 evaluation excluding any test items with hits. If the hit rate is nonzero and removing leaked items drops ar-stablelm-2-chat below jais-family-13b on ArabicMMLU or AceGPT-7B-chat on AlGhafa, the headline claim fails. If hit rate is zero, contamination is ruled out and the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 builds the Arabic pretraining mix from CulturaX (74M web documents, 158B tokens pre-cleaning; 114B after cleaning) plus SANAD and an e-book corpus. Section 4 then generates the 183k rephrased instruction dialogues from those cleaned documents, including an MCQ template (Appendix B). The evaluation benchmarks—ArabicMMLU, AlGhafa, ACVA, CIDAR—are all public HuggingFace datasets (Table 12). No step in Sections 3–6 measures n-gram overlap or removes eval items from the crawl or the rephrased data. Web crawls contain benchmark Q&A; because the MCQ rephrasing pipeline converts arbitrary document text into multiple-choice questions, a single leaked document can enter both pretraining and instruction tuning. The central comparison—45.5% vs 41.9% on ArabicMMLU and 50.1% vs 45.1% on AlGhafa—is therefore consistent with memorization of answer strings. Table 8 shows that adding MCQ synthetic data is worth +1.4 average points, so the evaluation format is sensitive to exactly the kind of training data that would be contaminated. Without an overlap audit, the 'beats 8x' claim is not interpretable as a capability result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Arabic Stable LM 1.6B, a base and chat model obtained by continuing pretraining of Stable LM 2 1.6B on 114B Arabic tokens (CulturaX, SANAD, e-books) mixed with English, followed by instruction tuning on a synthetic rephrased dialogue dataset, Instar-500k, and Aya. The authors evaluate on ArabicMMLU in cloze format (CF), CIDAR-MCQ-100, ACVA, and AlGhafa, and report that the chat model achieves the highest average scores, outperforming models with up to 13B parameters on ArabicMMLU and AlGhafa.","tokens_in":19466,"tokens_out":6034,"duration_ms":55663,"significance":"If the reported results are contamination-free and reproducible, the paper would make a useful contribution: it demonstrates that a 1.6B model can be competitive with 7-13B Arabic-centric LLMs after continued pretraining on a moderately sized Arabic corpus, and that synthetic MCQ-style instruction data provides a measurable boost. The release of the models and the detailed ablations (learning-rate cool-down, synthetic data) are valuable. However, the central superiority claim is currently unsecured due to absent decontamination analysis, under-specified evaluation prompts, and lack of statistical uncertainty quantification.","major_comments":[{"comment":"No contamination analysis is reported. The pretraining corpus includes 114B tokens of web-crawled text from CulturaX (Table 3), and the instruction-tuning dataset is generated by rephrasing documents from that same corpus (§4, Appendix B). All four evaluation benchmarks are public datasets (Table 12). Because web crawls are known to contain benchmark questions, and because the MCQ rephrasing pipeline converts document text into multiple-choice items, a leaked benchmark item could appear in both pretraining and instruction tuning. Table 8 shows that adding MCQ synthetic data improves the average by 1.4 points, which indicates that the evaluation is sensitive to exactly the kind of data that would be contaminated. The paper must provide an n-gram overlap analysis (e.g., 8-13 gram hits) between the training corpora and the evaluation sets, remove any overlapping items, and re-report Tables 5 and 6 before the 'beats up to 8x' claim can be interpreted as a capability result.","section":"§3.2, §4, §5"},{"comment":"The evaluation protocol is under-specified. The text states that ACVA uses 5-shot and F1 macro, but for ArabicMMLU, AlGhafa, and CIDAR-MCQ-100 no few-shot count or prompt template is given. This matters because cloze-format accuracy can vary substantially with prompt wording and example choices. In addition, CIDAR-MCQ-100 contains only 100 items, and no confidence intervals or significance tests are reported anywhere; differences of 1-2 percentage points on that set (e.g., 46.0 vs 45.0 in Table 6) are within sampling noise. The authors should specify all evaluation prompts and shot counts, report variance (e.g., bootstrap CIs) or repeated runs, and avoid claiming superiority over baselines whose margins are not statistically distinguishable.","section":"§5"},{"comment":"The headline claim is format-dependent, but the paper does not give this caveat enough prominence. In the cloze format, ar-stablelm-2-chat beats larger models (Tables 5-6), but in the multiple-choice format it is the second-best small model, not a leader (Figure 6). The authors justify CF as more robust (Appendix A), but because most prior Arabic benchmark reports use MCF, the comparison to the literature is oblique. The claim 'beating multiple models with up to 8x the parameters' in the abstract should be explicitly qualified as 'in cloze format on ArabicMMLU and AlGhafa' and ideally accompanied by MCF results for all evaluated models, so readers can judge whether the advantage is specific to the answer format.","section":"§6, Appendix A"}],"minor_comments":[{"comment":"'around 74K documents' contradicts Table 3, which lists 74.0M documents for CulturaX before cleaning.","section":"§3.2"},{"comment":"'Gropher' should be 'Gopher'.","section":"§3.3"},{"comment":"The column headers list 't5' twice; the sequence should probably be t1-t9.","section":"Table 11"},{"comment":"'a 4 % higher accuracy on average' is a 3.6 percentage-point difference from Table 5; please rephrase.","section":"§6"},{"comment":"The caption says 'with and without MCQ synthetic data,' but the 'w/o MCQ' model still includes standard (non-MCQ) rephrased data; please clarify.","section":"Table 8"},{"comment":"Figures 7 and 8 are reproduced at a size that makes the Arabic text nearly illegible; please enlarge or provide text versions.","section":"Appendix B"},{"comment":"The Koto et al. entry has stray quotation marks from the source file.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The contamination concern is the main barrier to accepting the central claim. The authors should also verify that all baseline numbers in Tables 5 and 6 were produced by the same evaluation harness; if any were taken from other papers, they need to be re-evaluated. The fact that the first author is a creator of the CIDAR-MCQ-100 evaluation set is not a flaw per se, but the small size of that set makes it a weak basis for the cultural-alignment claim. Given the resources of the authors, a decontamination audit and full evaluation-protocol disclosure should be feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll cut to it: this paper is a competent, useful model release, but the central claim is not yet interpretable as a capability result because the authors never check whether the web-crawl pretraining data (CulturaX, 114B token) contains ArabicMMLU, AlGhafa, ACVA, or CIDAR. All four benchmarks are public HuggingFace datasets, and the training mix is a web crawl. The absence of any n-gram overlap audit or decontamination step is a genuine hole, and the paper's own Table 8 makes it worse: adding MCQ-style synthetic data (rephrased from the same cleaned documents) is worth +1.4 points on average, which means the evaluation format is sensitive to exactly the kind of data that could leak.\n\nWhat's good: the authors release the base and chat checkpoints, so independent verification is possible. The ablation on early vs. late cool down is a thoughtful addition, and the tokenizer fertility analysis is relevant. The comparison across a wide range of Arabic models (from 135M to 13B) is more thorough than most papers in this space. The rephrased-MCQ recipe, while built on prior work from the same team, is a concrete new application to Arabic and the models are openly available. That counts for something.\n\nThe soft spots are real but not fatal to the paper's overall usefulness. Besides contamination, there are no error bars or significance tests, and one benchmark (CIDAR) has only 100 items, so the 46.0 vs. 45.0 differences there are noise. The evaluation prompts and few-shot settings are underspecified for ArabicMMLU and AlGhafa. The paper's choice of cloze format is defensible, but the fact that the model cannot follow the multiple-choice format (Figure 5) while doing well in cloze is exactly the pattern you'd see from surface-level memorization of answer choices. The limitations section is honest about overtokenization, benchmark scarcity, and synthetic data risks, but conspicuously missing any mention of contamination.\n\nBottom line: this paper deserves a serious referee, and I'd send it to peer review, but with the explicit expectation that the authors either run a proper decontamination check on the pretraining and instruction data against all evaluation benchmarks, or reframe the claims as 'not yet audited.' I would not desk-reject it, and I would not cite it as evidence of capability until the leakage question is addressed. For practitioners who want a small Arabic model, the release itself is worth evaluating, but the benchmark claims need the audit.","headline":"Solid empirical contribution with a real gap: no contamination check, so the 'beats 8x larger models' headline is unsecured even though the models are released and the recipe is credible.","tokens_in":20175,"tokens_out":2119,"would_cite":true,"duration_ms":25413,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 1.6-billion-parameter Arabic model outperforms Arabic LLMs up to eight times its size on cloze-format benchmarks.","keywords":["Arabic LLM","low-resource NLP","continued pre-training","instruction tuning","synthetic data","cloze format evaluation","model efficiency","ArabicMMLU"],"falsifier":"Compute exact or near-duplicate n-gram overlap between the test questions in ArabicMMLU, AlGhafa, ACVA, and CIDAR and the 114B cleaned training tokens; if a substantial fraction of questions (or their answer strings) appear in the crawl, the benchmark superiority could come from memorization rather than general capability. A cleaner test would re-run the comparisons on a newly written Arabic benchmark released after training.","tokens_in":18998,"feed_emoji":"🗣️","tokens_out":4343,"duration_ms":38275,"temperature":0.7,"pith_summary":"Arabic Stable LM 1.6B is an attempt to answer a concrete question: can a small Arabic-focused language model compete with much larger ones? The authors fine-tune the 1.6B Stable LM 2 model on roughly 114 billion Arabic tokens, then instruction-tune it with a mix of existing datasets and 183k synthetic question-answer dialogues generated from its own training documents. Their central claim is that the resulting chat model, ar-stablelm-2-chat, beats every model they compare against on ArabicMMLU in cloze format, including 13B models, and sets the best average on AlGhafa. If correct, this matters because it suggests strong Arabic NLP performance does not require 7-13B parameters, cutting hardware cost and inference latency.","feed_headline":"1.6B Arabic model beats 13B rivals on Arabic tests","feed_subtitle":"Trained on 114B Arabic tokens plus synthetic dialogues, the chat model tops benchmarks despite being 8x smaller.","key_machinery":"The load-bearing mechanism is continued pre-training plus synthetic instruction tuning. Starting from Stable LM 2 1.6B, the authors train for 500k steps on a mixture of 619B English and 115B Arabic tokens (sampled so Arabic gets 82% of steps), with a warm-up, cosine/inverse-square-root schedule, and an early linear cool-down that they show improves ArabicMMLU. The Arabic cleaning pipeline keeps 114B of 158B raw tokens. For the chat model they add 677,746 instruction samples, of which 182,505 come from using Qwen2-7B-Instruct to rephrase training documents into dialogues and multiple-choice tests; the MCQ portion alone raises the average score by 1.5 points. Evaluation uses the cloze format with normalized accuracy, which the paper argues is more robust than letter-based multiple choice.","core_discovery":"The paper's central discovery is that a 1.64B chat model, ar-stablelm-2-chat, achieves the highest average accuracy among all compared models on ArabicMMLU (cloze format) at 45.5%, ahead of the best 13B model (jais-family-13b) at 41.9%, and scores 50.1% on AlGhafa versus 45.1% for AceGPT-7B-chat. The base model also finishes near or above most larger models. The authors attribute the gains to continued pre-training on 114B cleaned Arabic tokens from CulturaX, SANAD news, and an Arabic e-book corpus, a learning-rate cool-down schedule, and instruction tuning that includes synthetic dialogues with multiple-choice questions. They also report that the cloze format gives more reliable and more favorable measurements for their models than the multiple-choice-letter format.","pith_inferences":["A natural next experiment, not run here, is tokenizer transfer: replacing the high-fertility Stable LM tokenizer with an Arabic-trained one would likely cut per-token cost and raise inference speed.","Because the MCQ-format gap is large for several baselines, leaderboard rankings for Arabic models may depend heavily on evaluation format; this suggests future Arabic benchmarks should report both CF and MCF.","The rephrasing pipeline, which uses Qwen2-7B-Instruct to generate dialogues from arbitrary documents, should transfer to other low-resource languages with small adaptation cost.","The absence of a decontamination study means the headline numbers should be treated as upper bounds until overlap checks are published."],"forward_implications":["If the result holds, Arabic NLP can be served with a 1.6B model, lowering GPU memory and inference latency relative to 7-13B Arabic models.","The cloze-format evaluation protocol, with normalized accuracy, becomes the recommended way to compare Arabic MCQ models, since letter-based formats penalize models that do not follow the letter-answer convention.","Synthetic instruction data generated by rephrasing the model's own training documents is a cheap way to build large Arabic chat-training sets.","The fine-tuning recipe (continued pre-training, cool-down schedule, MCQ rephrasing) transfers to other base models and possibly to larger Stable LM versions."],"supporting_citations":[{"why":"Provides the Stable LM 2 1.6B base model and hyperparameters used as the starting point.","marker":"(Bellagente et al., 2024)"},{"why":"Supplies CulturaX, the main Arabic pre-training source with 158B raw tokens before cleaning.","marker":"(Nguyen et al., 2023)"},{"why":"Introduces the fertility score and the Jais models, and reports that mixing English improves Arabic results.","marker":"(Sengupta et al., 2023)"},{"why":"Defines the ACVA benchmark and provides the AceGPT baselines that the paper compares against.","marker":"(Huang et al., 2023)"},{"why":"Defines ArabicMMLU, the primary benchmark where the chat model claims the best average.","marker":"(Koto et al., \"2024\")"},{"why":"Defines AlGhafa, the second benchmark where the chat model claims the best average.","marker":"(Almazrouei et al., 2023)"},{"why":"Contributes the Instar-500k instruction dataset and the GemmAr-7B baseline.","marker":"(Chouikhi et al., 2024)"},{"why":"Describes the rephrasing method used to generate the synthetic dialogue data.","marker":"(Pieler et al., 2024)"},{"why":"Provides the CIDAR dataset used for the CIDAR-MCQ-100 evaluation.","marker":"(Alyafeai et al., 2024)"}],"fun_headline_variants":["1.6B Arabic model beats 13B on Arabic benchmarks","Small Arabic LM tops 13B models despite 8x size gap","1.6B Arabic chat model outranks 13B rivals","Arabic Stable LM: 1.6B outshines 13B","Efficient Arabic LM: 1.6B beats 13B on tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper performs no contamination analysis, so the central claim rests on the assumption that ArabicMMLU, AlGhafa, ACVA, and CIDAR test items were not present in the crawled training data.","fun_headline_variants_meta":{"raw":{"variants":["1.6B Arabic model beats 13B on Arabic benchmarks","Small Arabic LM tops 13B models despite 8x size gap","1.6B Arabic chat model outranks 13B rivals","Arabic Stable LM: 1.6B outshines 13B","Efficient Arabic LM: 1.6B beats 13B on tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2892,"prompt_tokens":890,"completion_tokens":2002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1904}},"tokens_in":506,"tokens_out":2002,"duration_ms":13497,"temperature":1.0,"reasoning_tokens":1904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:33:47.585025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute exact or near-duplicate n-gram overlap between the test questions in ArabicMMLU, AlGhafa, ACVA, and CIDAR and the 114B cleaned training tokens; if a substantial fraction of questions (or their answer strings) appear in the crawl, the benchmark superiority could come from memorization rather than general capability. A cleaner test would re-run the comparisons on a newly written Arabic benchmark released after training.","supporting_citations":[{"cited_title":"GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning","cited_arxiv_id":"2407.02147","evidence_quote":"Contributes the Instar-500k instruction dataset and the GemmAr-7B baseline."}],"review_version":1}