{"id":"2a4761e7-a956-455f-87de-00feb623df51","arxiv_id":"2507.15850","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"3LM provides open Arabic benchmarks for native and synthetic STEM multiple-choice questions and translated HumanEval/MBPP code tasks, with evaluations of 40 models.","lead":"This paper introduces 3LM, a set of three Arabic benchmarks for evaluating large language models on STEM questions and code generation, with data released publicly. It reports results from over 40 Arabic and multilingual models, finding that English and Arabic code abilities are closely correlated while STEM skills vary by model and format.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"STEM benchmark validity hinges on unmeasured QA-pair error rates; code benchmark rests on ROUGE-L backtranslation without semantic human audit.","rationale":"The reader's weakest assumption is essentially identical to the load-bearing concern here: unvalidated item quality in the LLM-assisted STEM pipeline and unverified semantic fidelity in the ROUGE-L-gated code translation. My independent reading of the full text confirms this is the main threat to the central claim. The paper's release of public datasets and code is a genuine contribution and the correlation analysis is internally coherent, so rejection is not warranted; however, the absence of any quantitative quality metric (error rate, agreement, human-review counts) means the benchmark's validity rests on an unsupported assertion. A CONDITIONAL verdict is appropriate: the benchmarks can be accepted as resources, but the reported model rankings should not be relied upon until item-level quality is quantified. I do not see a separate fatal flaw; the circularity (Qwen used to build and test) is real but is a validity risk rather than proof of corruption, and it is also covered by the same error-rate test. The concrete test proposed is one specific check that would settle the concern: measure key-accuracy error rates on a sampled subset with independent annotators and check whether rankings move when erroneous items are removed.","tokens_in":24877,"tokens_out":1742,"duration_ms":17046,"concrete_test":"Sample, say, 100 items from each of NativeQA and SyntheticQA, and have two independent native-Arabic STEM educators label each item's answer key as correct/incorrect and the question as answerable without error. Report per-benchmark error rates and Cohen's kappa. If the answer-key error rate exceeds roughly 2-5% or agreement is low, recompute Table 1 ranking deltas after excluding erroneous items; if rankings change materially, the validity claim fails. For code, independently translate 50 above-threshold HumanEval-Ar/MBPP-Ar prompts back to English and check semantic equivalence of the task specification (not lexical overlap), or run the original hidden test suites against solutions written from the Arabic prompts by bilingual programmers; report the pass-rate drop relative to English prompts.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that 3LM yields valid rankings of 40+ models on Arabic STEM and code. That claim requires the benchmark items and answer keys to be correct. The paper asserts \"manual verification by the research team\" (Section 3.1.2) and \"rigorous filtering\" for synthetic items (Section 3.1.3), but reports no annotator counts, no inter-annotator agreement, no per-item error rate, and no example of a rejected item. The pipeline is also circular: Qwen3-235B-A22B performs extraction, classification, MCQ conversion, and answer assignment for the native benchmark, and YourBench uses the same model family that is later evaluated; if Qwen3 introduces systematic answer-label errors, rankings on native and synthetic STEM benchmarks shift in ways the paper cannot detect.\n\nFor code, translations are accepted when backtranslation ROUGE-L F1 exceeds 0.85 (HumanEval) or 0.8 (MBPP), and only below-threshold items receive human review. ROUGE-L measures surface n-gram overlap, not semantic equivalence; a translation can pass the threshold while changing the required function behavior, and the paper reports no human review rate or error counts for above-threshold items. The reported near-perfect English-Arabic correlation (r=0.97, Table 4/5) is consistent with the Arabic tasks being nearly identical to English tasks, but it also masks item-level mistranslations that preserve ordering. The \"several rounds of review\" wording suggests additional human passes, yet the methods describe human review only below threshold. This is the weakest load-bearing point: model rankings on STEM benchmarks are only meaningful if the answer keys are right, and the paper provides no quantitative evidence of key accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 3LM, a suite of three Arabic benchmarks: 865 native STEM multiple-choice question-answer pairs extracted from Arabic textbooks and worksheets, 1,744 synthetic STEM MCQs generated with the YourBench pipeline using Qwen3-235B-A22B, and 542 Arabic-translated code prompts (HumanEval-Ar and MBPP-Ar) produced by translating EvalPlus benchmarks with GPT-4o and backtranslation-based quality filtering. The authors evaluate over 40 base and instruction-tuned Arabic-centric and multilingual LLMs on these benchmarks in MCQ and completion formats, and on code via pass@1, also reporting cross-task correlations and a distractor-perturbation robustness analysis. All datasets and evaluation code are released publicly.","tokens_in":25303,"tokens_out":6914,"duration_ms":68983,"significance":"If the benchmark items are accurate, 3LM would be a valuable open resource for Arabic STEM and code evaluation, filling an acknowledged gap: existing Arabic benchmarks emphasize linguistic/cultural content, and AraSTEM is not publicly available. The code suite extends widely used EvalPlus benchmarks with Arabic prompts, and the evaluation across 40+ models is a useful contribution. Strengths include the public release of datasets and code, the native Arabic sourcing for STEM items, anchoring the code half to external HumanEval/MBPP tasks with EvalPlus test suites, and the robustness perturbation analysis. However, the central validity claim depends on benchmark-item correctness, and the paper provides no quantitative evidence of that correctness: manual verification is asserted without annotator counts, agreement scores, or error rates, and the code translation quality gate is based on a surface n-gram metric with no reported audit of accepted translations. These gaps make the reported model rankings potentially unreliable and need to be addressed before the benchmarks can be accepted as valid assessment tools.","major_comments":[{"comment":"The accuracy of the native and synthetic QA pairs is asserted but not quantified. The manual verification in §3.1.2 is described without annotator counts, inter-annotator agreement, or error rates, and the synthetic filtering in §3.1.3 is called \"rigorous\" without reporting rejection rates or any sample of rejected items. Since the paper's central claim is that 3LM yields valid rankings of 40+ models, the authors should report a human audit on a random sample (e.g., 100 items per benchmark) with per-domain correctness rates, and either correct or exclude erroneous items.","section":"§3.1.2, §3.1.3"},{"comment":"The abstract claims a \"human-in-the-loop process with several rounds of review,\" but the methods state that human review is applied only to translations below the ROUGE-L thresholds; above-threshold translations are accepted without human review. ROUGE-L measures surface n-gram overlap, not semantic equivalence, so a translated prompt could pass the threshold while changing the required function behavior. The authors should report the number of prompts that fell below threshold, the human review outcomes, and a sample audit of above-threshold prompts with semantic equivalence judgments.","section":"§3.2, Abstract"},{"comment":"The claim that \"Models consistently perform better in MCQ format compared to completion format across all scales\" is contradicted by the instruction-tuned results in Table 8, particularly for Qwen3 models: Qwen3-8B-Instruct scores 20.23 in MCQ versus 47.63 in completion, Qwen3-14B 39.54 versus 50.98, and Qwen3-30B-A3B 17.57 versus 53.53. This anomaly is not discussed and suggests a problem with the MCQ scoring protocol for these models (e.g., Arabic-letter likelihood scoring under a chat template). The authors should explain or correct this inconsistency.","section":"§5.3.1, Table 8"},{"comment":"Several reported code scores appear internally inconsistent. In Table 3, Qwen2.5-32B-Base shows Arabic MBPP+ at 82.8, which is 35.8 points higher than its Arabic MBPP (47.0) and also higher than its English MBPP+ (69.0), an implausible pattern. In Table 10, aya-expanse-32B-Instruct shows Arabic MBPP as 5.5, likely a typo for 55.5. These data errors undermine the reliability of the code evaluation results, which are a central contribution; the authors should verify all tables and correct data-entry errors.","section":"Table 3, Table 10"},{"comment":"The synthetic benchmark is generated by Qwen3-235B-A22B, and models from the Qwen3 family are evaluated on it. This is not circular in the strict sense because the benchmark items are not derived from the evaluated models' outputs, but it creates a risk of inflated scores for the generator family if the generator's question style and answer-label distribution match its own preferences. The authors should assess this risk by comparing the relative performance of Qwen3-family models on the synthetic benchmark versus the native benchmark; if the performance gap is systematically larger for Qwen3 models than for other families, this would indicate a generation bias. The paper currently reports no such analysis.","section":"§3.1.3, Tables 8-9"}],"minor_comments":[{"comment":"The AraSTEM citation is listed as Mustapha et al. (2024), but the arXiv identifier (2501.00559) is dated January 2025; please correct the year and ensure consistency in the reference list.","section":"§2"},{"comment":"The text refers to \"Table 9\" for instruction-tuned models' overall averages, but Table 9 reports domain-wise synthetic scores; the overall averages appear in Table 8. Please fix the cross-reference.","section":"§5.3.1"},{"comment":"The Arabic script in the translated instruction and response prompts appears garbled (e.g., reversed or mirrored letter forms such as \"ىجري\" and \"ةيلاتلﺍ\"). This is likely an OCR artifact and should be cleaned before publication.","section":"Appendix A.2"},{"comment":"The tables report a column labeled \"Score 25%\" without defining whether this is the score on the 25% perturbed subset or the full benchmark with a 25% perturbation rate. Please clarify the metric and describe how the 25% subset was selected (e.g., stratified by domain or difficulty).","section":"§6, Tables 11-12"},{"comment":"The jais-family-8k 30B row shows Arabic MBPP (12.4) and MBPP+ (10.3) far below its English scores (46.6 and 38.1). If this is not a typo, the result is striking and should be discussed; otherwise it should be corrected.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's main value is the public release of Arabic STEM and code benchmarks, and the evaluation of many models is a useful reference point. However, the absence of item-level quality validation is a genuine obstacle to accepting the ranking claims: without measured error rates in the QA pairs and some semantic audit of the code translations, the leaderboard results could be artifacts of benchmark noise. The internal inconsistencies in Tables 3 and 8 also need careful checking. I would like the editor to ensure that the authors provide the requested human-audit data and corrected tables in the revision, rather than only prose responses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper delivers what it promises—three open Arabic evaluation resources (native STEM MCQs, synthetic STEM QA, and HumanEval/MBPP translations with plus tests) and a 40+ model evaluation across them. That is genuinely useful. The strongest parts are the native question collection (865 pairs from Arabic school materials, OCR'd and cleaned) and the EvalPlus-Ar code suite, which extends HumanEval and MBPP to Arabic with the expanded test sets. I believe the datasets are public on HF, which is the right move.\n\nWhere it gets soft: the quality of the STEM items is asserted, not measured. Section 3.1.2 says \"manual verification by the research team,\" but gives no annotator counts, no agreement scores, no per-item error rate. The synthetic set is built by YourBench using Qwen3-235B-A22B with a difficulty filter, and the paper later evaluates Qwen3-family models on these same questions. That circularity is real. It does not kill the benchmark—the datasets are inspectable and the pipeline is described in enough detail to reproduce—but it means the reported model rankings on STEM are only as trustworthy as the unmeasured answer-label accuracy. If the authors had reported a clean human audit on a random sample, that concern would mostly disappear.\n\nThe code half has a similar issue. Translations are accepted when backtranslation ROUGE-L F1 is above 0.85 (HumanEval) or 0.8 (MBPP), with human review only for below-threshold items. ROUGE-L is a surface metric; a prompt can pass while changing the function's required behavior. The paper doesn't report the human review rate or any error counts. The tight English-Arabic correlation (r=0.97) is consistent with the Arabic tasks being near-identical to English, which is good for test validity, but it also means item-level mistranslations would be masked.\n\nThe limitations section is honest about the school-level scope and the visual gap, and it admits the synthetic set inherits LLM biases. That helps.\n\nBottom line: this is a bench-construction paper, not a modeling paper, and its value is mostly in the released datasets. I'd send it to peer review, but require the authors to add quantitative validation of the STEM QA pairs and report the human-review statistics for the code translations. With those numbers, the paper is a solid community resource. Without them, the rankings are hard to trust.\n\nRecommendation: accept for review with a request for major revision focused on validation transparency.","headline":"Open Arabic STEM and code benchmarks that fill a real gap, but the paper leans on unmeasured QA-pair quality and a ROUGE-L-based translation gate; useful and worth peer review with a demand for validation numbers.","tokens_in":25792,"tokens_out":2476,"would_cite":true,"duration_ms":26728,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three open benchmarks test Arabic LLMs on science and code.","keywords":["Arabic LLM evaluation","STEM question answering","code generation benchmark","machine-translated prompts","multiple-choice benchmark","pass@1","synthetic question generation","multilingual model evaluation"],"falsifier":"Take a random sample of 200 native and 200 synthetic STEM items, have independent Arabic-speaking STEM educators re-answer them and flag ambiguous or wrong items, and check whether the original answer key matches. If the key-mismatch rate exceeds a few percent, the reported rankings would be unreliable. For the code benchmarks, have native-Arabic-speaking programmers judge whether a random sample of the 542 Arabic prompts, including items above the ROUGE-L thresholds, preserve the original English intent; if a noticeable fraction are semantically unfaithful, the threshold-based acceptance of untranslated items is invalid.","tokens_in":24696,"feed_emoji":"🧪","tokens_out":8979,"duration_ms":82775,"temperature":0.7,"pith_summary":"The paper sets out to fill a gap in Arabic LLM evaluation: existing Arabic benchmarks emphasize language, dialect, and culture, while science and programming, the domains most relevant to practical LLM use, are barely covered. It introduces 3LM, a suite of three public benchmarks built from natively Arabic sources: 865 multiple-choice questions extracted from Arabic textbooks and worksheets, 1,744 synthetic STEM questions generated from the same material, and 542 Arabic-translated code prompts adapted from established English code benchmarks. To show the benchmarks work, the paper evaluates more than 40 Arabic and multilingual models and reports a consistent pattern, with MCQ scores far exceeding completion-format scores and English and Arabic code-generation scores almost perfectly correlated. A sympathetic reader would care because 3LM offers the first open, native Arabic test bed for scientific reasoning and bilingual programming, turning a blind spot into a measurable target.","feed_headline":"Three open benchmarks test Arabic LLMs on science and code","feed_subtitle":"More than 40 models scored on 3,151 native, synthetic, and translated Arabic items.","key_machinery":"The load-bearing object is the benchmark suite itself, whose three components are built by three distinct mechanisms. Native STEM items come from an OCR pipeline for Arabic math-heavy PDFs, followed by a four-stage LLM extraction-and-formatting pipeline that classifies question type, filters visually dependent items, rewrites non-MCQs into MCQs, and is followed by manual team verification. Synthetic items are produced by an adapted YourBench pipeline, with ingestion, summarization, chunking, question generation, and analysis stages, using the same textbook sources and a difficulty threshold of 6 or higher. Code items are produced by translating HumanEval+ and MBPP+ prompts with GPT-4o, validating via backtranslation with ROUGE-L thresholds of 0.85 for HumanEval and 0.8 for MBPP, and sending below-threshold items to native-Arabic-speaking Python experts. These mechanisms carry the paper's claims because they determine the exact items, answers, and tests on which model rankings rest.","core_discovery":"The paper's central claim is that 3LM provides valid, reproducible evaluation of Arabic LLMs in STEM knowledge and code generation. It argues this on three fronts: the native STEM items are authentic because they come from real Arabic educational materials; the synthetic items extend coverage while controlling difficulty; and the code prompts, though translated, preserve the semantics and test suites of the original English benchmarks. Based on evaluations of over 40 models, the paper further claims that these benchmarks reveal substantive findings about model behavior, most notably that multiple-choice accuracy systematically overstates ability measured without a visible answer choice, and that Arabic code ability tracks English code ability almost exactly, while Arabic NLP ability is only weakly related to either.","pith_inferences":["Implicit consequence: because the paper excludes diagrams and figures from the benchmarks, an image-inclusive Arabic STEM benchmark is the natural next step, and the original PDF corpus still contains the visual items that were filtered out.","Testable extension: the ROUGE-L thresholds could be calibrated against human semantic-fidelity ratings on the 542 translated prompts, which would turn the current rule of thumb into a validated filter for future multilingual code benchmarks.","Editorial connection: since native and synthetic items draw on the same textbook sources, one could directly test for generation bias by comparing model accuracy on native-versus-synthetic versions of the same concept, revealing whether the LLM generator changes difficulty."],"forward_implications":["If 3LM is correct, Arabic LLM teams gain three public benchmarks that can be plugged into training and evaluation pipelines, making progress in Arabic scientific reasoning measurable.","The consistent MCQ-over-completion gap across model sizes implies that MCQ-only Arabic evaluations overstate model ability, so future Arabic benchmark reports should treat completion-format scores as the sterner test.","The near-perfect correlation between English and Arabic code pass@1 scores suggests that translating prompts is a reliable way to extend English code benchmarks to Arabic, but that the Arabic score mostly re-measures the same underlying code ability.","The weak Arabic-NLP-to-code correlation for instruct models indicates that Arabic language proficiency and programming proficiency diverge after fine-tuning, so they should be evaluated and optimized separately."],"supporting_citations":[{"why":"Provides HumanEval, the base code benchmark whose docstrings are translated into Arabic and whose pass@1 metric is used.","marker":"Chen et al., 2021"},{"why":"Provides MBPP, the base code benchmark whose task statements are translated for MBPP-Ar.","marker":"Austin et al., 2021"},{"why":"Provides EvalPlus and the expanded HumanEval+/MBPP+ test suites, the source of the Arabic code prompts.","marker":"Liu et al., 2023"},{"why":"Supplies YourBench, the pipeline adapted to generate the 1,744 synthetic STEM questions.","marker":"Shashidhar et al., 2025"},{"why":"AraSTEM, the only prior dedicated Arabic STEM benchmark, which remains inaccessible and thus defines the gap 3LM fills.","marker":"Mustapha et al., 2024"},{"why":"Provides the Open Arabic LLM Leaderboard scores used for the cross-task correlations with Arabic code performance.","marker":"El Filali et al., 2025"}],"fun_headline_variants":["3LM: New benchmarks for Arabic LLMs in STEM and code","Arabic AI gets 3LM tests for science and programming","3LM reveals Arabic code skill mirrors English, MC overstates ability","Open 3LM benchmarks gauge Arabic LLMs on STEM and code","3LM: Arabic LLM benchmarks expose multiple-choice inflation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 2,609 STEM items' answer keys and wording are correct, and that the 542 translated code prompts mean the same thing as their English sources, because the paper asserts manual verification and ROUGE-L backtranslation thresholds without reporting annotator counts, agreement scores, or error rates, so any hidden errors would silently skew the model rankings.","fun_headline_variants_meta":{"raw":{"variants":["3LM: New benchmarks for Arabic LLMs in STEM and code","Arabic AI gets 3LM tests for science and programming","3LM reveals Arabic code skill mirrors English, MC overstates ability","Open 3LM benchmarks gauge Arabic LLMs on STEM and code","3LM: Arabic LLM benchmarks expose multiple-choice inflation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1444,"prompt_tokens":838,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":454,"tokens_out":606,"duration_ms":7521,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:21:30.641238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 200 native and 200 synthetic STEM items, have independent Arabic-speaking STEM educators re-answer them and flag ambiguous or wrong items, and check whether the original answer key matches. If the key-mismatch rate exceeds a few percent, the reported rankings would be unreliable. For the code benchmarks, have native-Arabic-speaking programmers judge whether a random sample of the 542 Arabic prompts, including items above the ROUGE-L thresholds, preserve the original English intent; if a noticeable fraction are semantically unfaithful, the threshold-based acceptance of untranslated items is invalid.","supporting_citations":[],"review_version":1}