Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

AIN: The Arabic INclusive Large Multimodal Model

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AIN, a 7B English-Arabic multimodal model, outperforms GPT-4o by 3.4 percent on the 38-subdomain CAMEL-Bench Arabic benchmark while retaining strong English ability.

desk verdict Useful data pipeline, but the headline GPT-4o comparison rests on the authors' own benchmark with no contamination check. read the letter →

arxiv 2502.00094 v2 pith:CAFRVPEC submitted 2025-01-31 cs.CV cs.AIcs.CLcs.HCcs.LG

Ahmed Heakl , Sara Ghaboura , Omkar Thawkar , Fahad Shahbaz Khan , Hisham Cholakkal , Rao Muhammad Anwer , Salman Khan This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.HCcs.LG
keywords ArabicmultimodalmodelbilingualLMMCAMEL-BenchOCRdatatranslationpipelinelow-resourcelanguageQwen2-VLfine-tuningvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AIN is a 7B-parameter English-Arabic multimodal model built by fully fine-tuning Qwen2-VL-7B on 3.6 million carefully filtered bilingual samples. The paper's central claim is that this open-weights model surpasses the closed frontier model GPT-4o by an absolute 3.4 percent averaged over eight domains and 38 sub-domains of the CAMEL-Bench Arabic benchmark. A sympathetic reader should care because Arabic is under-served by large multimodal models, and the result suggests a small open model can close the gap with far larger proprietary systems when training data is curated specifically for the language. The authors also report gains over the base model on ArabicMMLU and retained or improved English benchmark scores, plus a 76 percent preference over GPT-4o and LLaVA in a blind human survey with native Arabic speakers.

What carries the argument

The load-bearing mechanism is the bilingual data-construction pipeline: GPT-4o-mini translates English multimodal data into Arabic using Arabic prompts; LaBSE cosine similarity with an 80 percent threshold rejects weak translations; BLEU, METEOR, and ROUGE checks verify quality; and LLavaGuard plus GPT-4o filters unsafe images. The resulting 3.6 million samples feed a one-epoch full-parameter fine-tune of Qwen2-VL-7B. This pipeline is what the paper credits for concentrating Arabic linguistic and cultural competence into a small open model.

What would settle it

Run a near-duplicate search between AIN's 3.6 million training samples and CAMEL-Bench questions and images, then re-run GPT-4o under a standardized prompt and decoding protocol; if overlap is non-trivial or GPT-4o's score shifts enough to erase the 3.4 percent gap, the central claim fails.

Watch

Extended reading notes

Core claim

The discovery AIN argues for is that high-quality bilingual multimodal data, not model scale, is the main lever for Arabic visual understanding. Using 35 percent authentic Arabic data and 65 percent machine-translated English multimodal data, refined through semantic similarity filtering, quality metrics, and toxicity screening, a 7B model reaches 63.77 total on CAMEL-Bench versus GPT-4o's 60.13, with particularly large margins on OCR and document understanding and remote sensing. The same model improves on 14 of 19 ArabicMMLU categories over its base and holds or improves English scores on ten benchmarks. In a blind human evaluation, native Arabic speakers chose AIN's answers 76 percent of the time against GPT-4o and LLaVA.

Load-bearing premise

The claim stands or falls on CAMEL-Bench being an uncontaminated, fair measure of Arabic multimodal ability; the paper reports no test-train overlap check, and the benchmark was built by five of the paper's own authors.

Editorial extensions

If this is right

  • Arabic-speaking users gain a practical open model for OCR, medical imaging, remote sensing, cultural visual question answering, and video understanding.
  • A modest 7B open model can beat closed frontier models in a low-resource language when training data is curated with language-specific care.
  • Machine-translated English multimodal data plus a smaller share of authentic Arabic data may be enough to bootstrap capable Arabic multimodal models without massive native corpora.
  • English capabilities are not sacrificed: the fine-tuned model improves over its base on MMBench, MME, MMMU, POPE, SEED, MathVista, ScienceQA, ChartQA, AI2D, and MMT-Bench.
  • The same bilingual data recipe is a reusable template for other Arabic dialects and related languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same recipe likely transfers to other low-resource languages with similar script complexity; a direct test would be training the same pipeline on, say, Swahili or Indonesian and measuring gains over the base model.
  • Because CAMEL-Bench was authored by five of AIN's own authors and the paper reports no test-train overlap check, the 3.4 percent margin should be re-verified by an independent group using a standardized evaluation protocol.
  • A translation-heavy training set may inherit GPT-4o-mini's translation style and errors, so probing on dialectal Arabic or adversarial OCR could reveal boundaries the benchmark misses.
  • The 35 versus 65 percent authentic-to-translated data split raises a scaling question: ablating this ratio would show how much of the gain comes from translation volume versus authentic Arabic content.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AIN, a 7B-parameter English-Arabic bilingual multimodal model obtained by full fine-tuning Qwen2-VL-7B on 3.6 million curated Arabic-English multimodal samples, with a data pipeline that includes GPT-4o-mini translation, LaBSE semantic verification, BLEU/METEOR/ROUGE quality checks, and LLavaGuard-based toxicity filtering. The central claim is that AIN-7B surpasses GPT-4o on the CAMEL-Bench Arabic multimodal benchmark by an absolute 3.4 points (63.77 vs. 60.13, Table 1), while also improving over Qwen2-VL-7B on ArabicMMLU and ten English benchmarks and receiving 76% preference in a 10-question human survey of over 200 Arabic native speakers.

Significance. If the CAMEL-Bench result is valid, the paper is significant: it would show that an open 7B model fine-tuned on 3.6M bilingual samples can beat a frontier closed model on Arabic multimodal understanding, a low-resource domain with clear practical value. The paper also documents a careful data curation and safety pipeline, and it ships a demo, webpage, and GitHub repository, which supports reproducibility if the data and evaluation code are released. The ArabicMMLU and English-benchmark improvements over the base model are encouraging and provide some independent evidence of capability gains. However, the headline claim is anchored to CAMEL-Bench, a benchmark co-authored by five of the seven AIN authors, and the paper reports no contamination check and no detailed evaluation protocol for baselines; the external benchmarks only compare against the base model, not against the closed models that the headline claim targets.

major comments (4)
  1. [Section 2 and Table 1] The headline 3.4% gain over GPT-4o on CAMEL-Bench cannot be assessed without ruling out test-train contamination. Section 2 describes the training data only as 'publicly available MSA Arabic and English datasets' and does not list the datasets, provide hashes, or report any overlap check against CAMEL-Bench. Since five of the seven AIN authors are also authors of CAMEL-Bench (Ref [1]), there is a direct channel through which test examples could enter the 3.6M training samples. Please provide a complete data provenance list and a quantitative contamination analysis (for example, n-gram or embedding similarity between training samples and the CAMEL-Bench test set), or the central claim remains unverified.
  2. [Section 3] The evaluation protocol for baselines on CAMEL-Bench is unspecified. The paper does not state whether prompts were given in Arabic or English (or both), the decoding temperature, the number of runs, or the answer extraction and parsing method. Figure 5 shows GPT-4o producing 'No Answer' and format violations, which suggests the scoring pipeline may penalize non-conforming outputs in a way that could inflate AIN's relative score if AIN's outputs are format-tuned. Please specify the exact evaluation protocol for all models and release the evaluation code or prompts so the comparison can be reproduced.
  3. [Section 1.3] The human evaluation is presented as a blind comparison, but the manuscript does not state whether the order of the three models was randomized across participants or whether each question was shown with a fixed model order. Without randomization, the reported 76% preference for Model 1 could reflect presentation order or position bias rather than genuine quality. In addition, no confidence intervals or significance tests are reported for the preference percentages. Please describe the randomization procedure and provide statistical analysis of the survey results.
  4. [Tables 2 and 3] The external benchmarks provide only partial support for the claim of state-of-the-art Arabic performance. ArabicMMLU and the ten English benchmarks are reported only as improvements over Qwen2-VL-7B, the base model, not against GPT-4o, Gemini-1.5-Pro, or other Arabic-capable LMMs. Thus the 'state-of-the-art' claim rests entirely on CAMEL-Bench, whose fair evaluation is exactly what is in question. Please add comparisons on these external benchmarks against at least the closed models used in Table 1, or temper the claim to 'improvement over the base model' and 'best on CAMEL-Bench'.
minor comments (5)
  1. [Figure 1 and Table 1] The benchmark name appears as 'Camel-Bench' in Figure 1 but as 'CAMEL-Bench' in the text, Table 1, and the abstract; please use a consistent capitalization.
  2. [References] References [3] and [24] both cite the same URL for GPT-4o under different titles, and references [5] and [8] both describe Gemini 1.5; please unify these citations to avoid confusion.
  3. [Section 1.3 and Table 1] The model name 'LLaVA' is written inconsistently as 'LlaV A' in Figures 5 and Figure 2 and as 'LLaVa-NeXt' and 'LLaVa-OneVision' in Table 1; please use the correct official names throughout.
  4. [Figure 13] The pipeline diagram includes a branch labeled 'Consider data re-translation' and a loop 'All Satisfied?', but the accompanying text does not explain when re-translation is triggered or how many iterations were used; please clarify the procedure.
  5. [Section 3] The fine-tuning details are incomplete: the paper mentions 'hyper-parameter configurations established by LLaMA-Factory' but does not report the learning rate, batch size, optimizer, or sequence length; please provide these values or cite a specific configuration file.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark-author overlap is a validity concern, not a circular derivation.

full rationale

The paper reports empirical measurements of a fine-tuned model on fixed benchmarks; there is no derivation chain in which an output quantity is defined in terms of an input quantity, a fitted parameter is relabeled as a prediction, or a conclusion is entailed by its own premises. The central comparison in Table 1 uses CAMEL-Bench (ref [1]), whose author list overlaps with the present paper; this raises legitimate concerns about benchmark independence and possible contamination, but it is not circularity: the benchmark scores are not used to construct the training data or the model, and no equation in the paper maps CAMEL-Bench labels into the model weights. The paper additionally reports improvements on ArabicMMLU (an externally authored benchmark) and on ten English benchmarks, showing that the reported capability gains are not solely defined by the authors' own benchmark. No self-definitional, fitted-input-called-prediction, uniqueness-importation, ansatz-smuggling, or renaming step is present. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on empirical design choices (translation thresholds, data ratios, training settings) and on the base model and benchmark from prior work. No new theoretical entities are introduced; the invented element is the data pipeline itself, which is an engineering artifact rather than a postulated entity.

free parameters (5)
  • LaBSE similarity threshold = 0.80
    Translations with cosine similarity below 0.80 were excluded (Section 2.2, Figure 13). This is a hand-chosen cutoff that shapes the training data; no sensitivity analysis is reported.
  • Quality metric thresholds = BLEU: 60%, METEOR: 80%, ROUGE: 80%
    Used on 50 back-translated samples to validate the translation pipeline (Section 2.2, Table 9). These thresholds are chosen post hoc and only checked on a small sample.
  • Training epochs = 1 epoch
    Full fine-tuning for one epoch (Section 3); no comparison of training durations is reported.
  • Authentic Arabic data ratio = 35%
    The dataset is 35% authentic Arabic and 65% machine-translated (Section 2). The ratio is a design choice without reported ablation.
  • Base hyperparameters = LLaMA-Factory defaults (not specified)
    The paper defers to LLaMA-Factory hyperparameters (Section 3) but does not list learning rate, batch size, weight decay, or warmup.
assumptions (5)
  • domain assumption Qwen2-VL-7B provides a strong English vision-language base that can be adapted to Arabic via fine-tuning.
    The paper uses Qwen2-VL-7B as the base without justifying its suitability for Arabic (Section 3, ref [2]).
  • domain assumption Machine translation by GPT-4o-mini, after filtering, preserves the semantic and cultural content needed to train an Arabic multimodal model.
    The entire data pipeline relies on translated data (65% of corpus); verification uses automated similarity metrics and a small 50-sample human-assisted check (Sections 2.1 and 2.2).
  • domain assumption LaBSE cosine similarity is a valid proxy for translation quality for this purpose.
    LaBSE was selected after comparing a few models on 21 sentence pairs and 100 additional samples; the 0.80 threshold is applied to the full dataset (Section 2.2).
  • ad hoc to paper The CAMEL-Bench evaluation is uncontaminated and comparable across models.
    The paper reports CAMEL-Bench scores for AIN and baselines without any discussion of test-train leakage. CAMEL-Bench is from the same research group (ref [1]), raising contamination risk. This assumption is load-bearing for the headline claim.
  • domain assumption Automated text metrics (BLEU, METEOR, ROUGE) correlate with human judgment of Arabic translation quality.
    Used to verify the translation pipeline (Section 2.2), but no human evaluation of the final filtered dataset is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AIN: The Arabic INclusive Large Multimodal Model." pith.science (2026). https://pith.science/paper/CAFRVPEC

@misc{pith2026250200094,
  author       = {Pith},
  title        = {Pith review of: AIN: The Arabic INclusive Large Multimodal Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CAFRVPEC}},
  note         = {Machine review of arXiv:2502.00094}
}
read the original abstract

Amid the swift progress of large language models (LLMs) and their evolution into large multimodal models (LMMs), significant strides have been made in high-resource languages such as English and Chinese. While Arabic LLMs have seen notable progress, Arabic LMMs remain largely unexplored, often narrowly focusing on a few specific aspects of the language and visual understanding. To bridge this gap, we introduce AIN-the Arabic Inclusive Multimodal Model-designed to excel across diverse domains. AIN is an English-Arabic bilingual LMM designed to excel in English and Arabic, leveraging carefully constructed 3.6 million high-quality Arabic-English multimodal data samples. AIN demonstrates state-of-the-art Arabic performance, while also possessing strong English-language visual capabilities. On the recent CAMEL-Bench benchmark comprising 38 sub-domains including, multi-image understanding, complex visual perception, handwritten document understanding, video understanding, medical imaging, plant diseases, and remote sensing-based land use understanding, our AIN demonstrates strong performance with the 7B model outperforming GPT-4o by an absolute gain of 3.4% averaged over eight domains and 38 sub-domains. AIN's superior capabilities position it as a significant step toward empowering Arabic speakers with advanced multimodal generative AI tools across diverse applications.

Figures

Figures reproduced from arXiv: 2502.00094 by the authors.

Figure 1
Figure 1. Cross-domain performance analysis on the Camel-Bench Benchmark. Our AIN-7B achieves promising performance compared to significantly bigger models (GPT-4o and Gemini-1.5-Pro) in both domain￾specific and aggregate settings. Despite its smaller size, our AIN-7B achieves competitive performance across all 38 sub-domains with significantly superior capabilities on OCR & document understanding. Abstract Amid the swift pro… view at source ↗
Figure 2
Figure 2. AIN: A versatile LMM excelling in visual and contextual understanding across diverse domains, including VQA on complex topics, OCR for various fonts and handwriting, cultural insights (traditions, food, places), agricultural tasks (crop identification, fruit classification, disease detection), remote sensing (multi-scale objects), medical imaging (various modalities), and video analysis (animation, human activities)… view at source ↗
Figure 3
Figure 3. AIN compared to existing LMMs across CAMEL-Bench benchmark [ [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Qualitative results demonstrating AIN’s comprehensive capabilities across diverse domains. The results show its proficiency in handling both multiple-choice and open-ended questions. Our proposed AIN exhibits robust performance in addressing queries related to visual a…
Figure 6
Figure 6. Figure 6: AIN human evaluation survey, illustrating assessment criteria and multi-domain questions [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Nationality of AIN Survey Participants: Participants represent 17 Arab nations, with the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: User Model Preferences. Participant preferences for the three models in the survey, with [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: User Preferences for MSA and Local Dialects: The majority (74.3%) preferred MSA for [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Survey Feedback - Part 1: Questions 1 to 4 explored diverse domains, including agriculture, [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Survey Feedback - Part 2: Questions 5 to 8 focus on domains such as traffic sign [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Survey Feedback - Part 3: Questions 9 and 10 focus on domains such as celebrities, [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Data verification and filtering pipeline for textual and visual data. Textual data underwent semantic similarity checks using LaBSE [29] (80% threshold) and quality evaluation using BLEU [31] (60% threshold), METEOR [32] (80% threshold), and ROUGE [33] (80% threshold)…
Figure 14
Figure 14. Figure 14: Similarity scores for diverse settings, including direct correct translation, incorrect [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Comparison of LaBSE and Paraphrase-XLM-R to identify the optimal model. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Visual Data Toxicity Filtering. Using GPT-4o [ [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SARD: A Large-Scale Synthetic Arabic OCR Dataset for Book-Style Text Recognition

    cs.CV 2025-05 conditional novelty 6.0 of 10

    SARD is a new synthetic dataset of 843,622 Arabic book pages with ten fonts, plus baseline OCR benchmarks showing large differences between modern vision-language models and traditional engines.

  2. ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ARB provides 1,356 Arabic multimodal questions with 5,119 human-reviewed reasoning steps and shows leading models score much higher on reasoning fluency than on correct answers.

  3. QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation

    cs.CV 2025-06 reject novelty 4.0 of 10

    Fine-tuning Qwen2-VL on synthetic Arabic data yields QARI v0.2 with CER 0.061 and WER 0.160 on the authors' private test set, but public SARD results show Mistral OCR is more accurate.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    Camel-bench: A comprehensive arabic lmm benchmark

    Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Alharthi, Ines Riahi, Abduljalil Saif, Jorma Laaksonen, Fahad S Khan, Salman Khan, and Rao M Anwer. Camel-bench: A comprehensive arabic lmm benchmark. arXiv preprint arXiv:2410.18976, 2024

    arXiv

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  3. [3]

    Gpt-4o model

    OpenAI. Gpt-4o model. https://openai.com, 2024. Accessed: 2024

    work page 2024

  4. [4]

    Gpt-4o-mini model

    OpenAI. Gpt-4o-mini model. https://openai.com, 2024. Accessed: 2024-10-14

    work page 2024

  5. [5]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  6. [6]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  7. [7]

    Pangea: A fully open multilingual multimodal llm for 39 languages

    Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neu- big. Pangea: A fully open multilingual multimodal llm for 39 languages. arXiv preprint arXiv:2410.16153, 2024

    arXiv

  8. [8]

    Gemini: A family of highly capable multimodal models, 2023

    Google AI. Gemini: A family of highly capable multimodal models, 2023

    work page 2023

Show all 35 references
  1. [9]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv prepri...

    2023 arXiv

  2. [10]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  3. [11]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  4. [12]

    Maya: An instruction finetuned multilingual multimodal model

    Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze-Yin Chan, SM Uddin, Shayekh Bin Islam, et al. Maya: An instruction finetuned multilingual multimodal model. arXiv preprint arXiv:2412.07112, 2024

    2024 arXiv

  5. [13]

    Arabicmmlu: Assessing massive multitask language understanding in arabic.arXiv preprint arXiv:2402.12840, 2024

    Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. Arabicmmlu: Assessing massive multitask language understanding in arabic.arXiv preprint arXiv:2402.12840, 2024

    2024 arXiv

  6. [14]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025

    2025

  7. [15]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  8. [16]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  9. [17]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  10. [18]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024

    2024

  11. [19]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  12. [20]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2...

    2022

  13. [21]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  14. [22]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. ...

    2016

  15. [23]

    MMT-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, jiayi lei, Quanfeng Lu, Peng Gao, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. MMT-bench: A compr...

    2024

  16. [24]

    Gpt-4 model

    OpenAI. Gpt-4 model. https://openai.com, 2024. Accessed: 2024

    2024

  17. [25]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  18. [26]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1. Minneapolis, Minnesota, 2019

    2019

  19. [27]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019

    2019

  20. [28]

    Mpnet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems, 2020

    2020

  21. [29]

    Language- agnostic bert sentence embedding

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852, 2020

    2007 arXiv

  22. [30]

    Arabert: Transformer-based model for arabic language understanding

    Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104, 2020

    2003 arXiv

  23. [31]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  24. [32]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages 65–72, 2005. 19

    2005

  25. [33]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  26. [34]

    Llavaguard: Vlm-based safeguards for vision dataset curation and safety assessment

    Lukas Helff, Felix Friedrich, Manuel Brack, Kristian Kersting, and Patrick Schramowski. Llavaguard: Vlm-based safeguards for vision dataset curation and safety assessment. arXiv preprint arXiv:2406.05113, 2024

    2024 arXiv

  27. [35]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 3: System...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.