ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark

Hisham Cholakkal; Jorma Laaksonen; Ketan More; Omkar Thawakar; Rao Muhammad Anwer; Salman Khan; Sara Ghaboura; Wafa Alghallabi

REVIEW 3 major objections 6 minor 56 references

ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces ARB, the first benchmark to score step-by-step reasoning in Arabic across both text and images, and shows twelve leading multimodal models produce fluent Arabic reasoning yet frequently land on the wrong final answer.

desk verdict A genuinely useful first Arabic multimodal step-reasoning benchmark, with reference-chain validation weaker than the claims. read the letter →

arxiv 2505.17021 v1 pith:XXKLVIGA submitted 2025-05-22 cs.CV

Sara Ghaboura , Ketan More , Wafa Alghallabi , Omkar Thawakar , Jorma Laaksonen , Hisham Cholakkal , Salman Khan , Rao Muhammad Anwer This is my paper

classification cs.CV

keywords Arabicmultimodalreasoningstep-by-stepbenchmarkchain-of-thoughtevaluationlargemodelsculturalgroundingLLM-as-judgeunderrepresentedlanguagesreasoning-answergap

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ARB, a benchmark of 1,356 Arabic multimodal questions whose answers are paired with 5,119 human-curated step-and-action reasoning chains across 11 domains, from math and OCR to cultural and historical interpretation. Its central claim is that ARB is the first benchmark to score step-by-step reasoning, not just final answers, in Arabic, and that this finer measurement exposes failures that accuracy alone hides. Across 12 state-of-the-art open and closed multimodal models, the paper finds a persistent gap: models produce structured, coherent-looking Arabic reasoning yet frequently fail to reach the correct answer, and the same model produces correct English reasoning but flawed Arabic reasoning on an identical visual task. If the claim is right, it means current models' reasoning skills do not transfer reliably to Arabic, and safe deployment in Arabic-speaking settings needs step-level, culturally grounded evaluation.

What carries the argument

The carrying object is the step-action pair: each of the 1,356 ARB samples couples an image, an Arabic question, and a reference solution broken into explicit reasoning steps, each step bound to the action that executes it, so a model output can be matched against a reference chain rather than a single answer label. Around that structure the paper builds an Arabic ten-dimension rubric, scoring faithfulness at step and token level, informativeness, repetition, hallucination, redundancy, semantic coverage, reasoning alignment, commonsense, and missing steps, with GPT-4o acting as an LLM-as-judge on a 1–10 scale and inter-annotator agreement of 83.56% among humans and 87.62% once the judge model is included to show the rubric is stable. The reference chains themselves are generated through five sourcing categories (translated English reasoning data, Arabic QA benchmarks, caption-derived pairs, synthetic samples, and tool-augmented chart and table content) and pass native-speaker review, with over 200 samples discarded for cultural misalignment or shallow reasoning.

What would settle it

Independently reconstruct the reference chains: have native speakers solve each of the 1,356 questions from scratch, without seeing the model-generated drafts, and compare their chains to the published references; if a substantial fraction (for instance, more than 10%) of reference chains contain material errors, missing steps, or culturally wrong readings, then the step-quality scores and the reported reasoning-answer gap would have to be recomputed. A cheaper, domain-local check: recompute every arithmetic step in the Math & Logic and chart domains and count how often the reference chain's own numbers produce the stated final answer.

Watch

Extended reading notes

Core claim

ARB's central claim is that it is the first benchmark purpose-built to evaluate step-by-step reasoning in Arabic across textual and visual modalities, and that step-level evaluation shows current LMMs reasoning less reliably in Arabic than their final-answer or English-language performance suggests. The paper reports a consistent reasoning-answer gap: GPT-4.1 and o4-mini score above 80% on Arabic reasoning-step quality while answering only 58–60% of questions correctly, and open-weight models show the same pattern at lower levels (Qwen2.5-VL-7B at 64.03% steps versus 37.02% answers; Llama-3.2-11B at 25.58% answers). A controlled comparison of the same model on identical visual input finds correct, constraint-aware reasoning in English and flawed reasoning in Arabic, including miscounted elements and ignored structural constraints. The paper's conclusion is that Arabic, with its linguistic and cultural specificity, requires its own step-level, culturally grounded evaluation, and that coherent-looking Arabic reasoning cannot be trusted as evidence of correct reasoning.

Load-bearing premise

The load-bearing premise is that the 5,119 reference reasoning steps, drafted largely by GPT-4o and Claude and then reviewed by native speakers, are correct and complete enough to serve as ground truth for step-level scoring, and the paper's inter-annotator study covers only 5% of the dataset and rates model outputs, not the reference chains themselves.

Editorial extensions

If this is right

Final-answer accuracy understates the problem: a model can score above 80% on Arabic step quality while answering only about 59% of questions correctly, so evaluations that skip the steps will misread model competence.
The consistent gap between all models' reasoning scores and their final-answer accuracy means well-structured Arabic reasoning does not guarantee correct conclusions, so step-level scoring is needed to distinguish fluent chains from valid ones.
Because the same model solves the same visual task correctly in English and wrongly in Arabic, English-language benchmark results should not be assumed to transfer to Arabic, and language-specific, culturally grounded evaluation is required.
Domain breakdowns show the gap persists across task categories, with perceptual-heavy domains (complex visual perception, remote sensing) hardest and OCR and scientific reasoning relatively strongest, identifying where Arabic-specific training data is most needed.
The released benchmark, rubric, and evaluation suite let other groups score new models against the same step-level reference chains, turning ARB into a reusable diagnostic rather than a one-off study.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the reference chains were drafted mostly by GPT-4o and Claude and only reviewed, not independently reconstructed, by native speakers, the benchmark's step scores should be read with the caveat that hidden reference errors would inflate judged coherence; a cheap audit of the math and chart domains against independent recomputation would settle how much this matters.
Editorial inference: the coherence-without-correctness pattern echoes behaviors observed in chain-of-thought research on English tasks, and ARB's contribution is to measure the same failure at larger scale in a language with different morphology and tokenization, which suggests testable extensions comparing Arabic- and English-prompted reasoning under identical step-level rubrics.
Editorial inference: a natural next use of ARB is as training signal, for instance preference data for an Arabic-native model that rewards step-level faithfulness, in which case the reported reasoning-answer gap makes a specific prediction: step scores should improve before final-answer accuracy does.
Editorial inference: the dataset's composition (41% math and logic, 24% charts and tables) means aggregate scores mostly reflect structured-reasoning domains, so conclusions about conversational, dialectal, or free-form Arabic would be an overreach, a limitation the paper itself acknowledges.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — the useful thing to know about ARB is that it is a real resource, not a paper-shaped object. It is the first Arabic multimodal benchmark for step-by-step reasoning, ships 1,356 samples with 5,119 step–action pairs across 11 domains, and the authors release the data, rubric, and evaluation code. That alone makes it worth refereeing. The construction pipeline is described in unusual detail: five sourcing categories, tool-augmented generation, and a human-in-the-loop review with native speakers. The 12-model evaluation is standard but competently done, and the qualitative examples (Figures 17 and 18) genuinely illustrate the reasoning–answer gap.

The soft spot is exactly where the stress-test note puts its finger: the reference chains are the ground truth for every step-level score, but their correctness is only internally reviewed. The reported Krippendorff alpha of 87.62% covers 5% of the dataset and measures human–LLM agreement on a 1–5 compatibility rating of model outputs — not on the 1–10 multi-attribute rubric used for the actual scores, and not on whether the reference steps themselves are correct and complete. If a reference has an error, every model penalized for not matching it inherits that error. The 41% Math & Logic and 24% CDT share further means the "cultural grounding" conclusion rests on a relatively small slice of the data; the Hist., Soc.Cult., and MED domains together are maybe 12–16% of samples. That is not fatal to the benchmark as a resource, but the paper's headline claim about persistent cultural-grounding failures is stronger than the evidence supports.

Other nits: no error bars on final-accuracy or step scores, so the model ordering (e.g., GPT-4.1 vs o4-mini) is not statistically grounded; the evaluation prompt is adapted from LLamaV-o1, which is fine but should be more explicit about what changed; the title says "Comprehensive" while the domain distribution is clearly skewed — that is marketing, not a flaw.

Who is this for? Anyone building or evaluating Arabic LMMs, and anyone working on multilingual multimodal reasoning. The benchmark will be used even if the paper's stronger claims get softened in revision. A serious referee should engage, and the likely outcome is a conditional accept with a request for direct validation of the reference chains (e.g., re-annotate a sample of steps with a separate set of native speakers and report step-level agreement), error bars, and a domain-stratified analysis of the cultural-grounding claim.

My vote: send it to review. I'd cite it if I worked on Arabic multimodal eval, and I'd take it to reading group.

Referee Report

3 major / 6 minor

Summary. The paper introduces ARB, a benchmark for step-by-step multimodal reasoning in Arabic, consisting of 1,356 image-question samples with 5,119 reference reasoning steps and actions across 11 domains. The construction pipeline combines translated English reasoning benchmarks, Arabic QA benchmarks, English caption-derived data, synthetic data, and tool-augmented chart/table data; all content is reported to pass native-speaker and domain-expert review. The paper evaluates 12 open- and closed-weight LMMs using final-answer accuracy, lexical and semantic similarity metrics, and a GPT-4o-based LLM-as-a-judge that scores model reasoning against the reference steps on a 1-10 multi-attribute rubric. The central reported finding is a consistent gap between high reasoning-step scores and lower final-answer accuracy, along with cross-lingual and domain-specific qualitative differences.

Significance. If the reference reasoning chains can be shown to be correct and complete, ARB would be the first step-level Arabic multimodal reasoning benchmark, filling a real gap in the evaluation of underrepresented languages. The release of the dataset, rubric, and evaluation suite is a concrete contribution, and the detailed pipeline description, the cross-lingual comparison in Figure 6, and the qualitative error analyses in Figures 17 and 18 are valuable diagnostic illustrations. However, the current evidence for reference-chain validity is narrower than the claims require, so the quantitative headline numbers should be treated as provisional pending additional validation.

major comments (3)

[§3.2, Table 3] The ground-truth reasoning chains are the load-bearing component for every step-level score, but their correctness and completeness are not directly validated. The chains were generated mainly by GPT-4o and Claude (with GPT-4o for translation in Category 1, GPT-4o for most reasoning generation, and Claude-3.5 for table data), then reviewed by native speakers and domain experts. The only quantitative reliability evidence, Krippendorff's alpha over 5% of the dataset, rates model outputs on a 1-5 compatibility scale (Figure 8), not the correctness, necessity, or completeness of the reference chains themselves, and it does not use the 1-10 multi-attribute rubric applied by the actual judge (Figure 9). Consequently, every step-quality score and the reported reasoning-answer gap inherit any errors, omissions, or culturally misaligned assumptions in the references. Section 6 acknowledges reliance on human judgment and prompt-induced biases, but this limitation is not operationalized. Please add a direct human-validation study of the reference chains on a representative stratified sample, independent of the annotators involved in construction, and report agreement on step correctness and completeness.
[§3.2, Table 3] The LLM-as-judge validation involves circularity pressure and a task mismatch. GPT-4o helped generate the reference steps and also serves as the judge scoring model outputs against those references; the reported 87.62% Krippendorff's alpha is obtained by including GPT-4o's judgments in the agreement calculation on a 5% sample rated with a different 1-5 scale. The paper does not report human-judge agreement on the actual 1-10 Arabic rubric, nor per-domain agreement, nor confidence intervals or significance tests for the final-answer and reasoning-step numbers in Table 3. Without this, it is difficult to know whether the differences between models, such as the 80.41% versus 61.02% reasoning-step scores for GPT-4.1 and GPT-4o-mini, are meaningful. Please report judge-human agreement on the real rubric on a stratified sample and provide confidence intervals or statistical tests for the headline comparisons.
[Section 4, Figure 19] The claim of persistent challenges in cultural grounding rests heavily on domains that are small after the Math & Logic and CDT dominance. With 1,356 total samples and a 41% Math & Logic split and 24% CDT split, culturally rich domains such as Hist. (4%), Soc.Cult. (8%), and MED (4%) contain roughly 50-110 samples each. A small number of reference errors or evaluation-instability artifacts in these domains could drive the culture-related conclusions. The domain-level analyses in Figures 13-16 should report per-domain sample sizes, confidence intervals, and ideally per-domain judge reliability so the reader can assess how much weight to place on the domain-specific claims.

minor comments (6)

[Table 4] The model name 'IntenVL3' in Table 4 should be 'InternVL3' to match the text and other tables.
[Figure 21] The caption for Figure 21(b) contains the placeholder text '[describe the second figure]' and must be completed before publication.
[§3.2, Table 3] Table 3 labels step scores as percentages ('Reasoning Steps (%)'), although the underlying judge scores are on a 1-10 scale; please clarify how the normalization and aggregation to percentages were performed.
[§3.2, Table 4] The text reports LaBSE averages as '81.5%±2' and '71.5%±5', but Table 4 gives only point estimates; please include the variances or confidence intervals, or state the number of samples over which the intervals were computed.
[Figures 13-16] The abbreviation 'Argo' in the domain abbreviations appears to be intended as 'Agro' for Agricultural Image Understanding; please harmonize the abbreviation across figures and the appendix table.
[References] The reference for Anthropic lists the year as '20254'; this appears to be a typo and should be corrected to '2025'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the benchmark construction and the reasoning-answer gap are grounded in independent measurements; minor self-citation and judge/reference overlap do not force the conclusions.

full rationale

ARB's central claims are the construction of a 1,356-sample Arabic multimodal reasoning benchmark and the empirical finding that LMMs show a reasoning-answer gap. Neither claim is circular by construction. The 5,119 reference steps are drafted by GPT-4o/Claude but then pass through native-speaker review, iterative human-in-the-loop feedback, and multi-stage filtering (Sections 2.2 and 2.3); the released references are not fitted to any evaluated model's outputs. Final-answer accuracy in Table 3 is computed independently of the LLM-as-judge step scores, and GPT-4o—the model that helped generate references and also serves as judge—finishes behind GPT-4.1 and o4-mini on reasoning-step quality, so the judge does not simply rank itself first. The evaluation prompt is adapted from the authors' own LlamaV-o1 suite (Thawakar et al., 2025), and data sources include CAMEL-Bench, TimeTravel, and VRC-Bench from the same research group; this is self-citation, but it is not load-bearing because the headline comparison tables and final-answer numbers are external to those citations. The strongest validity concern—that the reported 5% inter-annotator agreement study rates model outputs rather than validating every reference chain—is an evidence limitation, not a circular reduction: no benchmark score is defined in terms of the judge's own output, and no fitted parameter is renamed as a prediction. Accordingly, there is no equation-level or definitional circularity; the score reflects only the minor self-citation and the overlap between the reference-generation model and the judge.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark itself is the contribution; it rests on assumptions about representativeness, the correctness of LLM-generated human-reviewed references, and the validity of LLM-as-judge evaluation. No free parameters are fitted and no theoretical entities are introduced.

assumptions (4)

domain assumption The 11-domain taxonomy and the actual 41% Math and Logic and 24% Charts, Diagrams, and Tables distribution are representative enough to support the claim of a comprehensive Arabic multimodal reasoning benchmark.
Stated in Section 2.1 and Figure 19; the benchmark's own distribution is heavily skewed toward structured reasoning, so claims of comprehensiveness and cultural coverage rest on this assumption.
ad hoc to paper GPT-4o and Claude generated reasoning chains that pass native-speaker review constitute correct and complete ground-truth reasoning steps.
Data generation in Section 2.2 and filtering in Section 2.3; no error-rate statistics are provided for the review process, and only 5% of samples are used in the inter-annotator agreement study.
ad hoc to paper GPT-4o as an LLM-as-judge with Arabic prompts provides a valid proxy for human judgment of reasoning quality across all 11 domains.
Described in Section 3.2; validated on 5% of the dataset with a Krippendorff alpha of 87.62 including the model, but per-domain reliability is not reported.
domain assumption Source benchmarks such as VRC-Bench, CAMEL-Bench, AgriCLIP, TimeTravel, and AI2D provide accurate visual content and questions for the new domains.
Used in Section 2.2 Categories 1, 2, 3, and 5; no independent re-verification of source labels is described.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark." pith.science (2026). https://pith.science/paper/XXKLVIGA

@misc{pith2026250517021,
  author       = {Pith},
  title        = {Pith review of: ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXKLVIGA}},
  note         = {Machine review of arXiv:2505.17021}
}

read the original abstract

As Large Multimodal Models (LMMs) become more capable, there is growing interest in evaluating their reasoning processes alongside their final outputs. However, most benchmarks remain focused on English, overlooking languages with rich linguistic and cultural contexts, such as Arabic. To address this gap, we introduce the Comprehensive Arabic Multimodal Reasoning Benchmark (ARB), the first benchmark designed to evaluate step-by-step reasoning in Arabic across both textual and visual modalities. ARB spans 11 diverse domains, including visual reasoning, document understanding, OCR, scientific analysis, and cultural interpretation. It comprises 1,356 multimodal samples paired with 5,119 human-curated reasoning steps and corresponding actions. We evaluated 12 state-of-the-art open- and closed-source LMMs and found persistent challenges in coherence, faithfulness, and cultural grounding. ARB offers a structured framework for diagnosing multimodal reasoning in underrepresented languages and marks a critical step toward inclusive, transparent, and culturally aware AI systems. We release the benchmark, rubric, and evaluation suit to support future research and reproducibility. Code available at: https://github.com/mbzuai-oryx/ARB

Figures

Figures reproduced from arXiv: 2505.17021 by the authors.

**Figure 1.** ARB Dataset Diversity. ARB comprises a wide array of multimodal reasoning samples, each combining a visual input with an Arabic question and detailed step-by-step reasoning with actions taken by step. The dataset spans 11 distinct domains, including visual reasoning, OCR and document understanding, chart and diagram interpretation, mathematical and logical inference, scientific and medical analysis, cultural and his… view at source ↗

**Figure 2.** The ARB Dataset Pipeline. The figure illustrates the ARB pipeline for evaluating Arabic multimodal reasoning in LMMs. It begins with data collection across 11 domains—such as medical imaging, historical interpretation, visual reasoning, and agriculture—sourced from curated datasets (e.g., VRC-Bench, CAMEL-Bench), synthetic content, tool-augmented outputs, and web scraping. Data is generated across five categories: E… view at source ↗

**Figure 3.** Overview of the ARB Data Collection, Generation and Verification Framework. The ARB benchmark is constructed from five primary data sources: (1) English reasoning benchmarks, (2) Arabic question–answer benchmarks, (3) English-captioned datasets, (4) Synthetic data, and (5) Tool-augmented data. All data undergoes iterative refinement through human-in-the-loop feedback and validation by native Arabic speakers to ensur… view at source ↗

Figures from the paper (18 more)

**Figure 4.** Figure 4: ARB Evaluation Prompt. The standardized Arabic prompt used across all ARB domains to elicit structured, curriculum-based reasoning steps from evaluated models during inference. The English version is provided in Appendix E. measure semantic similarity between English …

**Figure 5.** Figure 5: Arabic Reasoning Evaluation Metrics. We assess step-by-step reasoning using five core Arabicspecific dimensions: Faithfulness (At-Tatabuq) ¯ , Informativeness (Al-Ithra’ Al-Ma’l ¯ um¯ at¯ ¯ı), Coherence (At-Tawafuq) ¯ , Commonsense (Al-Mantiq Al-’Amm) ¯ , and Reasonin…

**Figure 6.** Figure 6: Cross-Lingual Reasoning Comparison (Arabic vs. English). This figure compares LMMs (GPT-4o) reasoning steps in Arabic and English for the same visual task. In the Arabic version, the model misinterprets structural constraints, yellow highlights incorrect assumptions ab…

**Figure 7.** Figure 7: Filtering and Verification User Interface. The interface enables annotators to manually edit content when applicable and assign quality ratings to guide subsequent controller review and final approval [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Inter-Annotator Agreement Interface. The interface allows annotators to evaluate each sample by assessing the compatibility of the model’s step/action chain with the provided image, question, and choices (when applicable). Annotators assign a score by comparing the mo…

**Figure 10.** Figure 10: English Translation of the Arabic Evaluation Prompt. A translated version of the prompt used to evaluate reasoning steps in ARB (see [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: English Translation of ARB Evaluation Metrics. An English version of the Arabic reasoning evaluation rubric used in ARB (see [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 12.** Figure 12: English Version of the ARB Prompt. This figure presents the English translation of the original Arabic prompt (see [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

**Figure 13.** Figure 13: Domain-Level Performance of Closed-Source Models. Bar charts comparing final answer accuracy and average reasoning step quality across ARB domains for each closed-source model. GPT-4.1 and o4-mini show strong reasoning in domains like Sci.R, CDT, and Hist., while nota…

**Figure 14.** Figure 14: Domain-Level Performance of Open-Source Models. Comparison of final answer accuracy and reasoning step scores across ARB domains for six open-source models. LLaMA-4 and AIN perform well in Sci.R and OCR but struggle in RS and VR. Qwen2.5-VL and LLaMA-3.2 show large ga…

**Figure 15.** Figure 15: Stepwise Attribute-Level Evaluation of Closed-Source Models. Heatmaps illustrating the average scores (1–10 scale) across key reasoning attributes—faithfulness, coherence, informativeness, and other diagnostic criteria—within each ARB domain for six closed-source mode…

**Figure 16.** Figure 16: Stepwise Attribute-Level Evaluation of Open-Source Models. Heatmaps visualizing average attributelevel scores (1–10 scale) across ARB domains for six open-source models, based on the LLM-as-Judge framework using Arabic prompts. Each cell reflects the model’s performa…

**Figure 17.** Figure 17: Qualitative Errors in Open-Source Models. This figure showcases common reasoning flaws in open-source LMMs across diverse Arabic multimodal tasks. Errors include incomplete reasoning steps, inconsistent logic, and hallucinated interpretations not grounded in the input…

**Figure 18.** Figure 18: Qualitative Errors in Closed-Source Models. This figure highlights reasoning failures by closed-source LMMs across various Arabic multimodal tasks. Common issues include incorrect numerical comparisons, invalid assumptions, misinterpreted constraints, and logically in…

**Figure 19.** Figure 19: Domain Distribution in ARB. The figure shows the distribution of ARB samples across 11 domains. Math & Logic (41%) and Charts, Diagrams, & Tables (24%) dominate, reflecting the dataset’s emphasis on structured reasoning. Other domains, including Social & Cultural, S…

**Figure 20.** Figure 20: Step Count Distribution by Domain. This figure shows the frequency distribution of reasoning steps per sample across the 11 ARB domains. Most domains exhibit a concentration between 2 and 6 steps, with Math & Logic, History, and Remote Sensing containing a larger shar…

**Figure 21.** Figure 21: Question token analysis in ARB: (a) token length by domain, and (b) [describe the second figure]. [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]

**Figure 22.** Figure 22: Question-to-Reasoning Token Ratio by Domain. The figure illustrates the average ratio between question token lengths and reasoning step token lengths across domains. Higher ratios in domains like Argo and MED indicate that these tasks require significantly more elabor…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 22 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, et al. 2025. Palm: A culturally inclusive and linguistically diverse dataset for arabic llms. arXiv preprint arXiv:2503.00151

work page Pith review arXiv 2025
[4]

Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of arabic multimodal large language models and benchmarks. arXiv preprint arXiv:2403.01031

arXiv 2024
[5]

Anthropic . 20254. https://www.anthropic.com Claude . AI assistant

work page
[6]

Ekaba Bisong and Ekaba Bisong. 2019. Matplotlib and seaborn. Building machine learning and deep learning models on google cloud platform: A comprehensive guide for beginners, pages 151--165

work page 2019
[7]

Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. 2024 a . Measuring and improving chain-of-thought reasoning in vision-language models. In NAACL-HLT

work page 2024
[8]

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024 b . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

arXiv 2024

Show all 56 references

[9]

Cohere-Labs. 2025. Aya vision 8b: A multilingual vision-language model. https://huggingface.co/CohereForAI/aya-vision-8b. Accessed: 2025-05-03

2025
[10]

Google DeepMind. 2024. Gemini 2.0 flash thinking: Unlocking transparent reasoning in ai. https://deepmind.google/technologies/gemini/flash-thinking/. Accessed: 2025-05-03

2024
[11]

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852

2020 arXiv
[12]

Google Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. https://arxiv.org/abs/2403.05530. Accessed: 2025-05-03

2024 arXiv
[13]

Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Husain Salem Abdulla Alharthi, Ines Riahi, Abduljalil Saif, Jorma Laaksonen, Fahad Shahbaz Khan, Salman H Khan, and Rao Muhammad Anwer. 2025 a . Camel-bench: A comprehensive arabic lmm benchmark. NAACL

2025
[14]

Sara Ghaboura, Ketan More, Ritesh Thawkar, Wafa Alghallabi, Omkar Thawakar, Fahad Shahbaz Khan, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. 2025 b . Time travel: A comprehensive benchmark to evaluate lmms on historical and cultural artifacts. arXiv preprint arXiv:2502.14865

2025 arXiv
[15]

Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919

2022 arXiv
[16]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

2025 arXiv
[17]

Ahmed Heakl, Sara Ghaboura, Omkar Thawkar, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, and Salman Khan. 2025. Ain: The arabic inclusive large multimodal model. arXiv preprint arXiv:2502.00094

2025 arXiv
[18]

Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah AlShamlan, Omar Knio, and George Turkiyyah. 2024. Arablegaleval: A multitask benchmark for assessing arabic legal knowledge in large language models. arXiv preprint arXiv:2408.07983

2024
[19]

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

2024 arXiv
[20]

Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925

2024 arXiv
[21]

Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, and Yova Kementchedjhieva. 2025. Jeem: Vision-language understanding in four arabic dialects. arXiv preprint arXiv:2503.21910

2025 arXiv
[22]

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pages 2...

2016
[23]

Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications

2018
[24]

Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. 2025. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321

2025 arXiv
[25]

Salima Lamsiyah, Kamyar Zeinalipour, Matthias Brust, Marco Maggini, Pascal Bouvry, Christoph Schommer, et al. 2025. Arabicsense: A benchmark for evaluating commonsense reasoning in arabic with large language models. In Proceedings of the 4th Workshop on Arabic Corpus Linguisti...

2025
[26]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[27]

Meta-AI. 2024. Llama 3.2 11b vision instruct. https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct. Accessed: 2025-05-03

2024
[28]

Meta-AI. 2025. Llama-4-scout-17b-16e-instruct. https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct. Accessed: 2025-05-03

2025
[29]

Kyle Moore, Jesse Roberts, Thao Pham, and Douglas Fisher. 2024. Reasoning beyond bias: A study on counterfactual prompting and chain of thought reasoning. arXiv preprint arXiv:2408.08651

2024 arXiv
[30]

Basel Mousi, Nadir Durrani, Fatema Ahmad, Md Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2024. Aradice: Benchmarks for dialectal and cultural capabilities in llms. arXiv preprint arXiv:2409.11404

2024 arXiv
[31]

Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. arXiv preprint arXiv:2501.00559

2024 arXiv
[32]

Umair Nawaz, Awais Muhammad, Hanan Gani, Muzammal Naseer, Fahad Shahbaz Khan, Salman Khan, and Rao Anwer. 2025. Agriclip: Adapting clip for agriculture and livestock via domain-specialized cross-model alignment. In Proceedings of the 31st International Conference on Computatio...

2025
[33]

OpenAI. 2024 a . https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: Advancing cost-efficient intelligence . Accessed: 2025-05-03

2024
[34]

OpenAI. 2024 b . https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

2024 arXiv
[35]

OpenAI. 2025 a . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api . Accessed: 2025-05-03

2025
[36]

OpenAI. 2025 b . https://openai.com/index/o3-o4-mini-system-card/ Openai o3 and o4-mini system card . Accessed: 2025-05-03

2025
[37]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[38]

Pinterest. 2025. Pinterest platform. https://www.pinterest.com/

2025
[39]

Qwen-Team. 2025. https://qwenlm.github.io/blog/qwen2.5-vl/ Qwen2.5-vl . Accessed: 2025-05-03

2025
[40]

Leonardo Ranaldi and Andr \`e Freitas. 2024. Self-refine instruction-tuning for aligning reasoning in language models. arXiv preprint arXiv:2405.00402

2024 arXiv
[41]

Mohammed Al-Maghrabi Research. 2025. Allam-thinking: Arabic large language model with enhanced reasoning capabilities. https://huggingface.co/almaghrabima/ALLaM-Thinking

2025
[42]

Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, and Fajri Koto. 2025. Commonsense reasoning in arab culture. arXiv preprint arXiv:2502.12788

2025 arXiv
[43]

Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv pre...

2023 arXiv
[44]

Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Sys...

2024
[45]

Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Synthetic prompting: generating chain-of-thought demonstrations for large language models. In Proceedings of the 40th International Conference on Machine Learning, pages 30706--30775

2023
[46]

Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944

2025 arXiv
[47]

Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. 2025. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186

2025 arXiv
[48]

Emily Vaillancourt and Christopher Thompson. 2024. Instruction tuning on large language models to improve reasoning performance. Authorea Preprints

2024
[49]

Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin...

2023
[50]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

2022 arXiv
[51]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[52]

Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. 2025. https://arxiv.org/abs/2411.10440 Llava-cot: Let vision language models reason step-by-step . Preprint, arXiv:2411.10440

2025 arXiv
[53]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

2023
[54]

Zheng-Xin Yong, M Farid Adilazuarda, Jonibek Mansurov, Ruochen Zhang, Niklas Muennighoff, Carsten Eickhoff, Genta Indra Winata, Julia Kreutzer, Stephen H Bach, and Alham Fikri Aji. 2025. Crosslingual reasoning through test-time scaling. arXiv preprint arXiv:2505.05408

2025 arXiv
[55]

Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. 2024. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198

2024 arXiv
[56]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, et al. 2025. Palm: A culturally inclusive and linguistically diverse dataset for arabic llms. arXiv preprint arXiv:2503.00151

work page Pith review arXiv 2025

[4] [4]

Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of arabic multimodal large language models and benchmarks. arXiv preprint arXiv:2403.01031

arXiv 2024

[5] [5]

Anthropic . 20254. https://www.anthropic.com Claude . AI assistant

work page

[6] [6]

Ekaba Bisong and Ekaba Bisong. 2019. Matplotlib and seaborn. Building machine learning and deep learning models on google cloud platform: A comprehensive guide for beginners, pages 151--165

work page 2019

[7] [7]

Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. 2024 a . Measuring and improving chain-of-thought reasoning in vision-language models. In NAACL-HLT

work page 2024

[8] [8]

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024 b . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

arXiv 2024