Pith. sign in

REVIEW 4 major objections 5 minor 46 references

AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces a 12-type Arabic hallucination evaluation framework and claims Allam, an Arabic-pretrained model, hallucinates less than multilingual models and matches reasoning models on Arabic QA and summarization.

desk verdict Useful Arabic hallucination benchmark with a real annotation resource, but the headline claim that factual hallucinations dominate 'across all models and tasks' is contradicted by the paper's own Table 2 for summarization. read the letter →

arxiv 2509.04656 v2 pith:GNNIN4LK submitted 2025-09-04 cs.CL

Aisha Alansari , Hamzah Luqman This is my paper
classification cs.CL
keywords ArabicLLMhallucinationtaxonomygenerativequestionansweringabstractivesummarizationfactualityvsfaithfulnessmanualannotationAllamTruthfulQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that Arabic LLM hallucination can be measured with a fine-grained, task-specific framework, and that on that measurement Arabic-pretrained models—especially Allam—produce fewer hallucinations than multilingual models on Arabic generative QA and summarization. It also claims factual hallucinations dominate over faithfulness errors in both tasks, and that even reasoning-focused models hallucinate more when answering in Arabic than in English on the same TruthfulQA questions. A reader should care because most hallucination benchmarks are English-only, leaving Arabic users and application builders without a basis for choosing or auditing models. If the framework holds up, it gives Arabic NLP a reusable annotation taxonomy and a first comparative ranking of 12 open and API models.

What carries the argument

A 12-indicator hallucination taxonomy split into factuality and faithfulness dimensions, with seven factuality and two faithfulness indicators for generative QA and five factuality and three faithfulness indicators (plus a 5-point human rating and hallucination density) for summarization. The framework's load-bearing mechanism is manual annotation by three native Arabic speakers, double-annotating each of 5,600 outputs with third-expert adjudication, because automatic metrics like ROUGE cannot capture factual consistency. Hallucination density—hallucinations normalized by total factual units—is what makes model comparisons fair across different summary lengths.

What would settle it

Compute Cohen's kappa (or a similar agreement measure) on the double-annotated subsets for each hallucination type; if kappa falls below roughly 0.6, the sub-0.2 gaps between adjacent models in Tables 1 and 2 are not resolvable. A cheaper check: re-annotate a random 100-response sample with a fresh team and see whether Allam still ranks first in both tasks.

Watch

Extended reading notes

Core claim

The central discovery is a comparative result: across 5,600 manually annotated outputs on 300 Arabic GQA questions and 100 Arabic summarization articles, the Arabic-pretrained Allam achieves the lowest average hallucination scores among non-reasoning models (0.382 in GQA, 0.215 in summarization) and performs on par with reasoning models like DeepSeek-R1, while models like Noon, Jais, Bloom, and Fanar hallucinate substantially more. Factual hallucination types (factual contradiction, named-entity, value, generic; in summarization, fabrication and context inconsistency) occur more often than faithfulness errors across every model and task. The paper further reports that four reasoning models—G

Load-bearing premise

The whole ranking rests on the consistency of the manual annotations; the paper reports double annotation with adjudication but never reports an inter-annotator agreement statistic, so the small score gaps between models could partly reflect annotator subjectivity.

Editorial extensions

If this is right

  • Factual hallucination, not faithfulness failure, is the main reliability risk in Arabic LLM outputs across both QA and summarization.
  • Language-specific pretraining can match or beat much larger reasoning-oriented models on Arabic factuality: Allam (7B-class) trails GPT-4o but beats DeepSeek-R1 and QwQ on summarization density and rivals them on GQA.
  • Arabic output is intrinsically harder for current reasoning models: the same TruthfulQA questions produce higher hallucination rates in Arabic than in English.
  • The released annotations of 5,600 outputs can be reused to train or fine-tune automatic Arabic hallucination detectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Arabic half of the hallucination gap likely tracks the share of Arabic text in pretraining corpora rather than model reasoning ability; a direct test would be to regress per-model hallucination rate on Arabic token proportion.
  • The taxonomy's counterexample-based guidelines could be turned into a few-shot annotation prompt for an LLM judge, giving a cheap automatic approximation of the manual scores—though the paper itself does not validate such a judge.
  • Because all models were decoded greedily at temperature 0.0, the rankings say nothing about sampling behavior; a decoding-strategy extension (temperature sweep, contrastive decoding) would show whether the ordering is stable under generation settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AraHalluEval, a framework for evaluating hallucination in Arabic LLM outputs using a 12-type fine-grained taxonomy that separates factuality errors from faithfulness errors. The authors manually annotate outputs of 12 models (4 Arabic, 4 multilingual, 4 reasoning-based) on 300 TyDiQA-GoldP-AR questions and 100 XLSum Arabic summaries, and also compare five models on Arabic/English TruthfulQA. The central claims are that factual hallucinations are more prevalent than faithfulness errors across all models and tasks, and that the Arabic model Allam has lower hallucination rates than multilingual models with performance comparable to reasoning-based models.

Significance. The framework and the manually annotated dataset of 5,600 outputs are potentially valuable resources for Arabic hallucination research, especially because the taxonomy distinguishes factuality from faithfulness and the authors provide code and deterministic decoding details. If the annotation reliability and the density measure are clarified, the dataset could support future work on Arabic hallucination detection and model comparison. However, the paper's headline finding is currently contradicted by its own Table 2, and the absence of inter-annotator agreement statistics makes the model-level rankings unverifiable. The contribution is therefore provisional but worth pursuing.

major comments (4)
  1. [Abstract, §4, Table 2, Conclusion] The headline claim that 'factual hallucinations are more prevalent than faithfulness errors across all models and tasks' is directly contradicted by Table 2 for the summarization task. For Allam, Jais, Bloom-7b, Llama, Qwen2.5, DeepSeek-R1, GPT-4o, and QwQ, the Total Faithfulness column exceeds the Total Factual column (e.g., Allam 0.220 vs 0.210, Bloom 1.020 vs 0.780, QwQ 1.040 vs 0.420). The aggregate across all summarization models is 7.238 faithfulness vs 6.403 factual. This is not an annotation-noise issue; it is an internal inconsistency between the reported data and the central empirical claim. The claim must be restricted to GQA, or the totals and the conclusion must be corrected.
  2. [§3.3.2, §6, Tables 1-2] The model-level rankings are the paper's core result, but the manual annotations are not accompanied by any inter-annotator agreement statistic (e.g., Cohen's kappa or Krippendorff's alpha). The paper reports double annotation with third-expert adjudication and §6 acknowledges human subjectivity, but the reported gaps between adjacent models are very small — for example, Allam's GQA average is 0.382 versus DeepSeek-R1's 0.377, and GPT-4o's summarization average is 0.105 versus GPT-o3's 0.145. Without agreement statistics, these fine-grained differences cannot be distinguished from annotator subjectivity. Please report agreement per task and per indicator, and restrict fine-grained ordering claims if agreement is not strong.
  3. [§3.3.1, Table 2, Appendix G] The 'hallucination density' measure is load-bearing for fairness and for the statistical tests in §4, but it is underspecified. For summarization it is described as 'the proportion of correct and incorrect facts in each summary', with no explicit formula, numerator, denominator, or definition of a 'factual unit'. In Appendix G, a different density is defined for GQA as the average number of hallucination types marked 'Yes' per response. Since Table 2 reports density values and §4 uses them in t-tests, the exact computation must be stated and justified.
  4. [§4, Conclusion, Tables 1-2] The claim that 'Arabic models consistently produced fewer hallucinations compared to their multilingual counterparts' is not supported by the tables. In GQA, Jais (0.777) and Noon (0.763) have higher average hallucination scores than every multilingual model in Table 1. In summarization, Fanar (1.172), Jais (0.638), and Noon (0.743) exceed Llama (0.515) and Qwen2.5 (0.477). Additionally, the Mann-Whitney test in §4 pools individual responses across models while the model is the sampling unit, which inflates the effective sample size and overstates significance for a model-level conclusion. The supported statement is that Allam, not the Arabic group as a whole, outperforms the multilingual models; please rephrase or use a model-level analysis.
minor comments (5)
  1. [§1, Abstract] The in-text callout 'Figure ??' in §1 is unresolved, and the Abstract contains 'Github link' as a placeholder instead of the actual repository URL.
  2. [Tables 1-2] The check-mark model-language legend is not explained, and model names are inconsistent (e.g., 'Llama' vs 'LLama3-8b', 'Qwen2.5' vs 'qwen2.5-7b'). Please unify names and add a legend for the Arabic/Multilingual/Reasoning columns.
  3. [§3.3.1, §4, Appendix E] There are several typos: 'indictaor' should be 'indicator', 'gpt-4o demonstrate' should be 'demonstrates', and the experimental-setup subsections numbered 5.1 and 5.2 in Appendix E conflict with Section 5 'Conclusion'. Please fix the numbering.
  4. [Figure 4] Figure 4 is said to use a log10 scale, but the axes are unlabeled and no numeric values are shown. This makes the distribution figure difficult to interpret; please label axes or provide the underlying frequencies.
  5. [Appendix D, Table 3] The TruthfulQA translation process removes questions that cannot be translated correctly. Please report the number of parallel questions actually used for Table 3 and confirm whether the filtering was symmetric across Arabic and English.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an empirical measurement grounded in external datasets and manual annotation; the only self-citation is procedural and non-load-bearing.

full rationale

The paper does not derive any result from a fitted parameter or from a self-citation chain. Its framework is a manually applied taxonomy (12 indicators), and its findings are counts/rates from annotations of 5,600 outputs against TyDiQA-GoldP-AR, XLSum, and TruthfulQA. The one overlap with the authors' prior work (Hasanaath et al. 2025, cited in Appendix F for the API inference setup of reasoning-based models) is procedural: it supplies no labels, scores, or conclusions, so it is not load-bearing. The abstract's claim that factual hallucinations dominate is, on inspection, contradicted by the Table 2 totals for summarization; however, an internal inconsistency or an unequal number of factuality vs faithfulness indicators is a correctness/measurement-design concern, not a circularity in which a reported result is equivalent by construction to its input. No equation in the paper defines a reported result in terms of the thing it purports to predict, and no fitted input is relabeled as a prediction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters: the evaluation uses hand-crafted prompts and a fixed taxonomy, but no numerical parameters are fit to data. No new physical or theoretical entities are introduced; the 12 indicators are an annotation taxonomy, not postulated entities. The axioms above are the assumptions the central findings depend on, chief among them the reliability of the manual annotations.

assumptions (3)
  • domain assumption Manual annotations of hallucination types are reliable ground truth.
    All model rankings rest on human labels by three native speakers; no inter-annotator agreement is reported (Section 3.3.2, Appendix A).
  • domain assumption TyDiQA-GoldP-AR and XLSum Arabic test sets are representative of Arabic GQA and summarization.
    Used as the source of evaluation inputs; conclusions are assumed to generalize beyond these 300 questions and 100 articles (Section 3.1).
  • ad hoc to paper The 12 hallucination indicators form an exhaustive, non-overlapping taxonomy.
    The taxonomy is defined for this study (Figure 3); categories such as factual contradiction, knowledge-source conflict, and response conflict overlap in practice, and no mutual-exclusivity test is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs." pith.science (2026). https://pith.science/paper/GNNIN4LK

@misc{pith2026250904656,
  author       = {Pith},
  title        = {Pith review of: AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GNNIN4LK}},
  note         = {Machine review of arXiv:2509.04656}
}
read the original abstract

Recently, extensive research on the hallucination of the large language models (LLMs) has mainly focused on the English language. Despite the growing number of multilingual and Arabic-specific LLMs, evaluating LLMs' hallucination in the Arabic context remains relatively underexplored. The knowledge gap is particularly pressing given Arabic's widespread use across many regions and its importance in global communication and media. This paper presents the first comprehensive hallucination evaluation of Arabic and multilingual LLMs on two critical Arabic natural language generation tasks: generative question answering (GQA) and summarization. This study evaluates a total of 12 LLMs, including 4 Arabic pre-trained models, 4 multilingual models, and 4 reasoning-based models. To assess the factual consistency and faithfulness of LLMs' outputs, we developed a fine-grained hallucination evaluation framework consisting of 12 fine-grained hallucination indicators that represent the varying characteristics of each task. The results reveal that factual hallucinations are more prevalent than faithfulness errors across all models and tasks. Notably, the Arabic pre-trained model Allam consistently demonstrates lower hallucination rates than multilingual models and a comparative performance with reasoning-based models. The code is available at: https://github.com/aishaalansari57/AraHalluEval

Figures

Figures reproduced from arXiv: 2509.04656 by the authors.

Figure 1
Figure 1. An example of LLM hallucination errors in the GQA task. Named-entity error denotes in￾correct names of people, places, or organizations, value error denotes wrong dates, ages, or time refer￾ences, factual contradiction represents information not present in the real-world, whereas response conflict represents contradicting information within the response itself. issue not only undermines their trustworthiness but als… view at source ↗
Figure 2
Figure 2. The AraHalluEval pipeline. (2) Fanar-1-9b (Team et al., 2025) developed by pre-training the google/gemma-2-9b model on 1 trillion Arabic and English tokens; (3) Jais-6.7b (Sengupta et al., 2023), which is a bilingual Arabic￾English LLM, optimized for proficiency in Arabic while demonstrating robust capabilities in English; (4) Noon-7b (Naseej for Technology, 2023), which is an Arabic LLM based on BLOOM, trained usin… view at source ↗
Figure 3
Figure 3. Definitions and examples of the hallucination indicators used to measure the hallucination of each LLM. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Frequency of hallucination types (log10- scaled) generated by evaluated LLMs across (a) GQA and (b) text summarization tasks. erally more robust in reducing factual hallucina￾tions in the Arabic GQA task compared to their multilingual counterparts. More details about s…
Figure 5
Figure 5. Figure 5: Distribution of hallucination density across [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The annotation platform C Annotation Examples [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Prompts used for GQA and summarization coding with a temperature of 0.0, disabling top-k and top-p sampling to produce deterministic out￾puts. We set the maximum number of tokens to 128 for summarization and 64 for GQA. A rep￾etition penalty of 1.2 was applied, and no …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 26 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Samir Abdaljalil, Hasan Kurban, and Erchin Serpedin. 2025. https://arxiv.org/abs/2503.07833 Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations . Preprint, arXiv:2503.07833

    arXiv 2025

  4. [4]

    M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan AlRashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, and 1 others. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390

    arXiv 2024

  5. [5]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

    arXiv 2023

  6. [6]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1--45

    work page 2024

  7. [7]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

    arXiv 2023

  8. [8]

    Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454--470

    work page 2020

Show all 46 references
  1. [9]

    Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Improved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2022

  2. [10]

    Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1--22

    2009

  3. [11]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--630

    2024

  4. [12]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554

    2023 arXiv

  5. [13]

    Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592--3603

    2020

  6. [14]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  7. [15]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  8. [16]

    Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers

    2010

  9. [17]

    Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics...

    2021

  10. [18]

    Ahmed Hasanaath, Aisha Alansari, Ahmed Ashraf, Chafik Salmane, Hamzah Luqman, and Saad Ezzini. 2025. Arareasoner: Evaluating reasoning-based llms for arabic nlp. arXiv preprint arXiv:2506.08768

    2025

  11. [19]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions ...

    2025

  12. [20]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  13. [21]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1--38

    2023

  14. [22]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  15. [23]

    Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332--9346

    2020

  16. [24]

    Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163--177

    2022

  17. [25]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \'e , and 1 others. 2023. Bloom: A 176b-parameter open-access multilingual language model

    2023

  18. [26]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th I...

    2020

  19. [27]

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449--6464

    2023

  20. [28]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252

    2022

  21. [29]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023 a . Mqag: Multiple-choice question answering and generation for assessing information consistency in summarization. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Confere...

    2023

  22. [30]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023 b . Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004--9017

    2023

  23. [31]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906--1919

    2020

  24. [32]

    Hamdy Mubarak, Hend Al-Khalifa, and Khaloud Suliman Alkhalefah. 2024. https://aclanthology.org/2024.lrec-main.705/ Halwasa: Quantify and analyze hallucinations in large language models: A rabic as a case study . In Proceedings of the 2024 Joint International Conference on Comp...

    2024

  25. [33]

    Naseej for Technology . 2023. Naseej launches its innovative arabic ai language model “noon” as an open-source initiative. https://naseej.com/news/2023/06/. Accessed: 2025-07-02

    2023

  26. [34]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024....

    2024 arXiv

  27. [35]

    OpenAI . 2025. https://openai.com/index/introducing-o3-and-o4-mini/ Introducing openai o3 and o4-mini . OpenAI Blog

    2025

  28. [36]

    Qwen Qwen-Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  29. [37]

    Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922

    2023 arXiv

  30. [38]

    Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ar...

    2023 arXiv

  31. [39]

    Alharbi, Samar Ahmed, Omar Nacar, Lahouri Ghouti, and Anis Koubaa

    Serry Taiseer Sibaee, Abdullah I. Alharbi, Samar Ahmed, Omar Nacar, Lahouri Ghouti, and Anis Koubaa. 2024. https://aclanthology.org/2024.osact-1.17/ ASOS at A rabic LLM s hallucinations 2024: Can LLM s detect their hallucinations :) . In Proceedings of the 6th Workshop on Open...

    2024

  32. [40]

    Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. Finesure: Fine-grained summarization evaluation using llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906--922

    2024

  33. [41]

    Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. 2024. Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 14379--14391

    2024

  34. [42]

    Melanie Subbiah, Faisal Ladhak, Akankshya Mishra, Griffin Adams, Lydia Chilton, and Kathleen Mckeown. 2024. Storysumm: Evaluating faithfulness in story summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9988--10005

    2024

  35. [43]

    Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, and 1 others. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944

    2025 arXiv

  36. [44]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  37. [45]

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and 1 others. 2024. Freshllms: Refreshing large language models with search engine augmentation. In Findings of the Association for Computational Linguistics A...

    2024

  38. [46]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.