Pith. sign in

REVIEW 3 major objections 5 minor 39 references

IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IslamicTurathBench is an expert-built Arabic benchmark of 3,465 questions that ties language-model scores to specific classical source works, disciplines, difficulty tiers, and answer formats in Islamic scholarship.

desk verdict Credible, well-documented benchmark release; the dataset construction is the strong part, and the evaluation layer has a contained circularity plus a Shafi'i-heavy fiqh slice that the paper discloses but should frame more carefully. read the letter →

arxiv 2608.04703 v1 pith:IYCIZFM3 submitted 2026-08-05 cs.CL

Shahd Gaben , Heba Sbahi , Samer Rashwani , Abdessalam Bouchekif , Mutaz Al-Khatib , Emad Mohamed , Somaya Eltanbouly , Mohammed Ghaly This is my paper · ORCID
classification cs.CL
keywords LLMevaluationbenchmarkIslamicscholarlytradition(turath)ArabicNLPmultiple-choiceQAreadingcomprehensionopen-endedknowledgereligiousandculturaldomainshumanreferencepanel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IslamicTurathBench (ISTB) is a new Arabic benchmark for measuring how well large language models handle the classical Islamic scholarly tradition (turath). It contains 3,465 expert-written question-answer items drawn from 35 recognized source works spanning more than twelve centuries and seven disciplines: Quran sciences, Hadith sciences, theology, jurisprudence, principles of jurisprudence, Sufism, and Prophetic biography. The benchmark's central wager is that a 3 x 3 design -- three scholarly-demand tiers (Beginner, Intermediate, Advanced) crossed with three task formats (multiple choice, passage-based comprehension, and closed-book open-ended questions) -- can separate what a model knows from how the question is asked. A 148-item scholarly human reference panel and zero-shot baselines from ten systems accompany the release. If the benchmark works as intended, it gives researchers a reproducible instrument for diagnosing where models fail on source-grounded religious scholarship rather than on generic web-style religious talk.

What carries the argument

The load-bearing object is the bounded gold-answer schema. Multiple-choice items carry a single keyed option; comprehension (COMP) items tether the answer to a supplied source passage; and closed-book knowledge (KNOW) items encode reference answers as finite accepted sets (possible_item_1 through possible_item_8 plus a no_requested_items threshold demanding one, several, or all elements). This determinacy constraint is what makes open-ended scoring tractable in a tradition with legitimate scholarly plurality: when disagreement is real, the question must name its source, author, or school. On top of this, the aggregate metric $$\mathrm{ISTB}(m)=\frac{1}{|C|}\sum_{(d,t,k)\in C} S_{d,t,k}(m)$$ averages per-discipline, per-format, per-tier cell means so that no large item group (MCQ holds 2,276 of the 3,465 items) dominates the headline score. Open-ended answers are scored by a reference-guided LLM-as-a-judge pipeline with two judges and a head judge for disagreements above 0.2, with inter-judge agreement near 0.94.

What would settle it

Take a stratified random sample of about two hundred items and have an independent panel of Islamic Studies scholars, chosen to cover multiple jurisprudential schools and not involved in the dataset's creation, answer them directly from the cited source works without seeing the gold answers. If a substantial fraction of gold answers (say more than five percent) are judged unsupported by the named source text, or if the accepted-answer sets of KNOW items prove incomplete against the sources, then the ground-truth layer is not stable enough to support the reported comparisons.

Watch

Extended reading notes

Core claim

The central claim is that language-model performance on the classical Islamic scholarly tradition is measurable, source-attributable, and systematically task-dependent, and that the field has lacked a resource able to show this. The paper argues that ISTB supplies that resource: every item is traceable to a named source work, each question is constrained to a determinate reference answer so that scoring stays stable despite legitimate scholarly plurality, and the aggregate ISTB score is a cell-balanced macro-average giving equal weight to every discipline-format-demand cell. On the paper's own reading of its baselines, the data show three things: passage-grounded comprehension scores highest (mean 0.843 across systems), closed-book open-ended knowledge questions score lowest (0.574), and performance falls monotonically from Beginner (0.819) to Advanced (0.672). Against the matched human subset, models exceed the scholarly panel on passage-grounded questions, while on closed-book knowledge questions only the strongest reported system tops the panel, and the paper takes this as evidence that single-number leaderboards would hide the real behaviour.

Load-bearing premise

The benchmark's validity rests on the assumption that the expert-authored gold answers and their difficulty labels faithfully represent what the selected turath works actually say, and that bounding every answer to a determinate reference has not erased legitimate scholarly plurality.

Editorial extensions

If this is right

  • Discipline, tier, format, and source-work filters let researchers draw a performance profile of any system against specific scholarly fields and specific historical texts.
  • Because format marginals diverge so sharply (COMP 0.843 vs KNOW 0.574 on average), a benchmark or leaderboard that reports a single number without format control will misrepresent where a model's competence actually lies.
  • The source-linked design turns ISTB into an evaluation harness for retrieval-augmented generation: one can index the documented corpus and compare closed-book against retrieval-supported answering on the same questions.
  • The matched human subset lets future work report human-versus-model comparisons holding task format fixed, rather than one aggregate ranking.
  • The 112 audit-corrected items, 32 controlled-plurality KNOW items, and published refinement protocol give users a visible quality-control trail for auditing the gold standard itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The consistent KNOW deficit across all ten systems suggests that parametric memory of the classical tradition -- vintage- and source-specific knowledge -- is the binding constraint; a testable prediction is that retrieval augmentation will narrow the KNOW gap far more than the MCQ or COMP gaps.
  • The jurisprudence items are built on a Shafi'i-majority source list, so a 'Fiqh' marginal should be read as a school-specific sample; reading it as cross-school coverage would overstate the benchmark's reach.
  • The per-source author death dates in the metadata make possible a historical-gradient analysis -- plotting model accuracy against author death date to test whether later commentaries are systematically harder for models than foundational works.
  • The confidence-calibration results imply that deployed Islamic question-answering systems should not treat verbalized confidence as a reliability signal; abstention or source-citation policies could be evaluated with the same instrument.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents IslamicTurathBench (ISTB), an Arabic question-answering benchmark consisting of 3,465 expert-authored items linked to 35 classical and contemporary Islamic scholarly works across seven disciplines (Quran sciences, Hadith sciences, theology, jurisprudence, principles of jurisprudence, Sufism, and prophetic biography). Items are organized along a scholarly-demand axis (Beginner, Intermediate, Advanced) and a task-format axis (MCQ, passage-based comprehension, open-ended knowledge questions). The release includes metadata, validation artifacts, aggregated scores from a 148-item human reference panel, and zero-shot baselines from ten LLM-based systems, with an LLM-judge pipeline for open-ended scoring. The paper claims the dataset supports reproducible, multi-granular evaluation of LLM behavior in the Islamic scholarly tradition.

Significance. If the dataset's validity holds, ISTB is a valuable resource: it is source-linked, discipline-balanced, metadata-rich, and publicly released with integrity manifests, schemas, a loader, and evaluation code. The authors are transparent about construction choices, including AI-assisted drafting as a bounded aid, a documented refinement audit, and explicit source-bounding of the Fiqh items. The inclusion of a human reference panel and a judge-selection screen with expert review are additional strengths. The main risks are external: the gold answers have not been independently audited, and the human reference scores are generated by the same LLM-judge pipeline used for model evaluation. These issues are fixable and do not invalidate the dataset's potential, but they need to be addressed before the benchmark can support fine-grained capability claims.

major comments (3)
  1. [Section 2.4, Section 4.3, Table 6] The central validity claim—that gold answers are source-faithful and that legitimate scholarly plurality has been adequately controlled—rests entirely on internal review. The refinement audit in Section 4.3 corrected 112 items using flags generated by the authors' own LLM pipeline (e.g., absolute judge disagreement ≥0.5 between GPT-5.2 and Gemini-2.5-Flash, low-score consensus, and passage mismatches), and no independent external expert audit verifies that the KNOW reference lists (possible_item_* and no_requested_items) do not omit legitimate answers from recognized schools or authors. Because every downstream metric treats these gold answers as ground truth, I recommend an external, non-author audit of a stratified sample of items, with the audit protocol and any disagreements reported in the paper.
  2. [Section 2.5, Section 5.3, Table 13] The human reference scores are generated by the same LLM-judge protocol used to score model responses, and raw human responses are excluded from the release. This makes the headline human–model comparison (Table 13) dependent on the judge models' scoring preferences rather than on an independent measure of scholarly quality. If the judge pipeline has systematic biases, those biases can affect human and model responses differently, so the claim that only Gemini-3-Pro exceeds the human panel on KNOW may be an artifact of judge behavior. Please re-score a sample of human and model KNOW/COMP responses with blinded external experts and report agreement, or release de-identified raw human responses so the community can independently re-score them.
  3. [Table 2, Figure 5, Table 10] The Fiqh discipline is almost entirely Shafi'i-school material: five of the six listed Fiqh sources are Shafi'i or Shafi'i-oriented (Minhaj al-Talibin, Ans al-Matalib, Hashiyat al-Ramli, Mughni al-Muhtaj, al-Fiqh al-Manhaji), and the sixth (Ihya' 'Ulum al-Din) is not a Fiqh manual. Yet results are reported under the unqualified label 'Fiqh' in Table 10 and Figure 5. The caveat in Section 2.2 that Fiqh items are strictly bound by the selected sources is not carried through to the results, so users may read Shafi'i-specific knowledge as general jurisprudence. Please rename the axis (e.g., 'Fiqh (Shafi'i school)') or add a prominent note to all Fiqh results and to the dataset README.
minor comments (5)
  1. [Section 5.4, Table 10] Please state how the 95% confidence intervals in Table 10 were computed (e.g., bootstrap over items, over cells, or over design cells) and whether the cell-balanced macro-average variance was estimated accordingly.
  2. [Section 4.3] Please clarify the unit of the reported interval Krippendorff's α (0.81): was it computed over flagged items, over correction-fix categories, or over expert judgments of the 462 candidate questions?
  3. [Section 5.3] Please define the sample sizes n=3,966 (COMP) and n=7,662 (KNOW) explicitly; if these are numbers of judged responses, it would also be helpful to report per-judge-pair agreement so readers can interpret the disagreement-escalation threshold |j1−j2|>0.2.
  4. [Section 7] The evaluation code and aggregate result files are released, but raw model responses and intermediate LLM-judge outputs are not; releasing at least per-response judge scores would substantially strengthen the reproducibility of Tables 10–13.
  5. [Title and Abstract] The transliteration of the tradition term is inconsistent ('turath' in the title and once in the abstract, 'turāth' elsewhere); please standardize it.

Circularity Check

1 steps flagged · score 4.0 of 10

The benchmark artifact itself is externally grounded and non-circular, but the evaluation layer is self-referential: the 'scholarly human reference panel' COMP/KNOW scores are produced by the same LLM-judge pipeline that scores the models, so the human-model comparison reduces by construction to judge-relative closeness to team-authored gold answers.

  1. other [Supplementary S6; Section 5.7; Section 2.5; Section 5.3]
    "This section reproduces the judge prompts used to score COMP and KNOW responses against gold reference answers. The same prompt templates are used for model and human responses."

    The abstract and Section 2.5 present the 148-item panel of three professors as an external 'scholarly baseline for comparison.' Per S6, however, COMP/KNOW human responses are scored by the same reference-guided LLM-judge protocol (Section 5.3) that scores model responses. The human-panel scores in Table 13 are therefore outputs of the same function f(gold, response) applied to human text and to model text, judged against the same team-authored gold answers — gold that Section 4.3 itself refined using flags from the same judge pair (including 78 gold-answer revisions and 58 KNOW scoring-rule adjustments).

full rationale

The central dataset contribution is externally grounded and not circular: the 3,465 items are expert-authored by named domain experts (Section 2.1), linked to 35 specific source works with bibliographic metadata and URLs (Sections 2.2, 3.1, 4.1), and are independently auditable against the turath corpus. The discipline taxonomy is adopted from classical external taxonomies (Ibn al-Akfani, Ibn Khaldun, Hajji Khalifah, Section 2.2), not from the authors' prior work. The only self-citations (QIAS 2025 and MAWARITH, references 6-7, sharing co-author Bouchekif) appear in Table 1 as benchmark positioning and are not load-bearing. The model baseline numbers (Section 5.5) are ordinary LLM-judge measurements against released gold answers and do not reduce to their inputs. The genuine self-referentiality is confined to the evaluation layer: the human reference panel's COMP/KNOW scores are produced by the same LLM judges that score the models (Supplementary S6), so Table 13's human-model gaps (e.g., COMP: humans 0.632 vs. Gemini-3-Pro 0.928) measure how closely each text approximates the team-authored gold as judged by team-validated judges, not an independent human measurement. A secondary but disclosed loop is the Section 4.3 refinement audit, where the same two production judges' disagreements flagged 112 items for correction; the later judge-agreement statistics (alpha = 0.937/0.936) are partially co-adapted with the gold they grade, though final corrections were human-made and transparently reported. Because the core dataset is independent and externally checkable, and the circularity affects only the supporting human-reference comparison, the score is 4 rather than 6-8.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The dataset's claim of expert grounding invokes domain authority (professors and recognized texts) and an LLM-judge-based evaluation layer. These are stated assumptions rather than derivation inputs. No new particles, forces, or unobservable entities are introduced.

free parameters (1)
  • LLM judge disagreement threshold = 0.2
    Chosen by hand for the open-ended scoring pipeline; if the absolute difference between the two production judges exceeds 0.2, a head judge overrides. This affects the adjudication rate (18.4% of open-ended responses) and the final COMP/KNOW scores.
assumptions (5)
  • domain assumption The seven-discipline taxonomy from Ibn al-Akfani, Ibn Khaldun, and Hajji Khalifah captures the structure of the classical Islamic scholarly tradition.
    Section 2.2: if this taxonomy is not representative, the claim of multi-discipline coverage weakens.
  • domain assumption The selected 35 works are recognized and authoritative sources for their disciplines.
    Table 2: source selection was made by three professors with no external audit; if the corpus is skewed, benchmark results may not generalize.
  • domain assumption The Beginner/Intermediate/Advanced tiers are valid operationalizations of pedagogical progression and Bloom's taxonomy.
    Section 2.3 and Table 3: this mapping is a design assumption, not an empirical result.
  • domain assumption Boundedness of reference answers is a valid quality criterion and does not systematically erase legitimate plurality.
    Section 4.2 and Table 6: the paper argues plurality is controlled by source/school specification, but this is an evaluative assumption about the domain.
  • ad hoc to paper LLM judges (GPT-5.2, Gemini-2.5-Flash, gemini-3.1-pro-preview) produce scores that approximate expert assessment for open-ended answers.
    Section 5.3 and S7: judges were screened on 20 KNOW items only; if judge scores diverge from expert judgment, the open-ended baselines and human-panel scores lose meaning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)." pith.science (2026). https://pith.science/paper/IYCIZFM3

@misc{pith2026260804703,
  author       = {Pith},
  title        = {Pith review of: IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IYCIZFM3}},
  note         = {Machine review of arXiv:2608.04703}
}
read the original abstract

Large language models (LLMs) are increasingly used for question answering, education, and research, including in religious and cultural domains where answers depend on specialised source traditions. Yet in Islamic Studies, key concepts, methods, and debates preserved in the authoritative scholarly tradition, known as turath, lack high-quality annotated resources. We introduce IslamicTurathBench (ISTB), a multi-task, multi-discipline dataset for evaluating LLMs on classical Islamic scholarship. Developed and reviewed by domain experts, ISTB contains 3,465 question-answer items drawn from 35 recognised source works spanning more than 12 centuries of scholarship across seven key fields of Islamic Studies. To enable comprehensive profiling of model capabilities, ISTB is structured along two axes: scholarly demand (Beginner, Intermediate, and Advanced) and task format (multiple-choice questions, passage-based comprehension, and open-ended knowledge questions). ISTB includes aggregated scores from a scholarly human reference panel and zero-shot baselines from ten systems. The dataset supports reproducible evaluation of language-model behaviour across source works, disciplines, scholarly-demand levels, and question formats in a historically layered scholarly domain.

Figures

Figures reproduced from arXiv: 2608.04703 by the authors.

Figure 1
Figure 1. IslamicTurathBench construction pipeline. The dataset was produced [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Repository layout of the IslamicTurathBench v1.0 Zenodo [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Refinement audit triage pipeline. Automated diagnostic signals isolated 462 unique candidate questions. Expert re [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Task- [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Discipline-level closed-book performance gap by system. Heatmap showing, for each system and discipline, the difference between closed-book open-ended performance (KNOW) and the stronger of the two supported task formats, MCQ or COMP. Values closer to zero indicate sma…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 31 canonical work pages

  1. [1]

    Brown, T. B. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020)

    work page 2020

  2. [2]

    Bommasani, R. et al. On the opportunities and risks of foundation models. arXiv https://arxiv.org/abs/2108.07258 (2021)

    arXiv 2021

  3. [3]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. arXiv https://arxiv.org/abs/2303.08774 (2023)

    arXiv 2023

  4. [4]

    & Iqbal, W

    Elmahjub, E., Qadir, J., Mushtaq, A., Naeem, R., Ghaznavi, I. & Iqbal, W. IslamicLegalBench : Evaluating LLMs' knowledge and reasoning of Islamic law across 1,200 years of Islamic pluralist legal traditions. arXiv https://arxiv.org/abs/2602.21226 (2026)

    work page arXiv 2026

  5. [5]

    IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

    Abdelaal, A., Al Haffar, M. N., Fawzi, M. & Magdy, W. IslamicMMLU: A benchmark for evaluating LLMs on Islamic knowledge. arXiv https://arxiv.org/abs/2603.23750 (2026)

    work page Pith review arXiv 2026

  6. [6]

    Bouchekif, A. et al. QIAS 2025: Overview of the shared task on Islamic inheritance reasoning and knowledge assessment. In Proc. Third Arabic Natural Language Processing Conference (2025)

    work page 2025

  7. [7]

    Bouchekif, A. et al. MAWARITH: A dataset and benchmark for legal inheritance reasoning with LLMs. arXiv https://arxiv.org/abs/2603.07539 (2026)

    arXiv 2026

  8. [8]

    Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions

    Atif, F., Askarbekuly, N., Darwish, K. & Choudhury, M. Sacred or synthetic? Evaluating LLM reliability and abstention for religious questions. arXiv https://arxiv.org/abs/2508.08287 (2025)

    work page Pith review arXiv 2025

Show all 39 references
  1. [9]

    Bhatia, G. et al. From RAG to agentic RAG for faithful Islamic question answering. arXiv https://arxiv.org/abs/2601.12809 (2026)

    2026 arXiv

  2. [10]

    Kitāb al-ʿIbar wa-dīwān al-mubtadaʾ wa-l-khabar fī tārīkh al-ʿArab wa-l-Barbar wa-man ʿāṣarahum min dhawī al -shaʾn al-akbar

    Ibn Khaldūn, ʿAbd al-Raḥmān ibn Muḥammad. Kitāb al-ʿIbar wa-dīwān al-mubtadaʾ wa-l-khabar fī tārīkh al-ʿArab wa-l-Barbar wa-man ʿāṣarahum min dhawī al -shaʾn al-akbar. Ed. Khalīl Shaḥāda; rev. Suhayl Zakkār. (Dār al -Fikr, Beirut, 1st ed., 1401 AH / 1981 CE)

    1981

  3. [11]

    Irshād al -qāṣid ilā asnā al -maqāṣid

    Ibn al -Akfānī, Muḥammad ibn Ibrāhīm ibn Sāʿid al -Anṣārī al-Sinjārī al-Bukhārī. Irshād al -qāṣid ilā asnā al -maqāṣid. Corr. Muḥammad Salīm al-Āmidī al-Bukhārī. (Asʿad Bik Ḥaydar, Beirut, 1322 AH)

  4. [12]

    Kashf al -ẓunūn ʿan asāmī al -kutub wa-l-funūn

    Ḥājjī Khalīfah, Muṣṭafā ibn ʿAbd Allāh, known as Kātib Çelebi. Kashf al -ẓunūn ʿan asāmī al -kutub wa-l-funūn. Corr. and annot. Muḥammad Sharaf al-Dīn Yaltqāyā and Rifat Bilge. (Wakālat al-Maʿārif, Istanbul, 1360–1362 AH / 1941–1943 CE)

    1941

  5. [13]

    Taʿlīm al -mutaʿallim ṭarīq al -taʿallum

    al -Zarnūjī, Burhān al -Islām. Taʿlīm al -mutaʿallim ṭarīq al -taʿallum. Ed. Marwān Qayānī. (al -Maktab al -Islāmī, Beirut and Damascus, 1st ed., 1401 AH / 1981 CE)

    1981

  6. [14]

    Anderson, L. W. & Krathwohl, D. R. A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. (Longman, 2001)

    2001

  7. [15]

    Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv. Neural Inf. Process. Syst. 33, 9459– 9474 (2020)

    2020

  8. [16]

    & Shazeer , N

    Roberts, A., Raffel, C. & Shazeer , N. How much knowledge can you pack into the parameters of a language model? In Proc. 2020 Conference on Empirical Methods in Natural Language Processing 5418 –5426 (Association for Computational Linguistics, 2020)

    2020

  9. [17]

    & Hajishirzi , H

    Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D. & Hajishirzi , H. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proc. 61st Annual Meeting of the Association for Computational Linguistics (Association for Com...

    2023

  10. [18]

    & Hruschka, E

    Pezeshkpour, P. & Hruschka, E. Large language models' sensitivity to the order of options in multiple -choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024 (Association for Computational Linguistics, 2024)

    2024

  11. [19]

    & Suhr, A

    Sclar, M., Choi, Y., Tsvetkov, Y. & Suhr, A. Quantifying language models' sensitivity to spurious features in prompt desi gn or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations (2024)

    2024

  12. [20]

    & Huang, M

    Zheng, C., Zhou, H., Meng, F., Zhou, J. & Huang, M. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (2024)

    2024

  13. [21]

    Gemini 2.5 Pro

    Google. Gemini 2.5 Pro. Gemini API documentation. https://ai.google.dev/gemini -api/docs/models/gemini-2.5-pro (accessed 16 June 2026)

    2026

  14. [22]

    Gemini 2.5 Pro model card

    Google DeepMind. Gemini 2.5 Pro model card. https://storage.googleapis.com/deepmind -media/Model-Cards/Gemini-2-5- Pro-Model-Card.pdf (2025)

    2025

  15. [23]

    Google AI Studio

    Google. Google AI Studio. https://aistudio.google.com/ (accessed 16 June 2026)

    2026

  16. [24]

    Gaben, S., Sbahi, H., Rashwani, S., Bouchekif, A., Ghaly, M., al -Khatib, M., Mohamed, E.& Eltanbouly, S. (2026). IslamicTurathBench (ISTB): a benchmark for the Islamic Scholarly Tradition in Arabic (Version 1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.20674930

    2026 doi

  17. [25]

    GPT-5.1: A smarter, more conversational ChatGPT

    OpenAI. GPT-5.1: A smarter, more conversational ChatGPT. OpenAI. https://openai.com/index/gpt-5-1/ (2025)

    2025

  18. [26]

    A new era of intelligence with Gemini 3

    Google. A new era of intelligence with Gemini 3. Google Blog. https://blog.google/products -and- platforms/products/gemini/gemini-3/ (2025)

    2025

  19. [27]

    DeepSeek-V3 Technical Report

    DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv https://arxiv.org/abs/2412.19437 (2024)

    2024 arXiv

  20. [28]

    DeepSeek -R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek -AI. DeepSeek -R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv https://arxiv.org/abs/2501.12948 (2025)

    2025 arXiv

  21. [29]

    Bari, M. S. et al. ALLaM: Large Language Models for Arabic and English. arXiv https://arxiv.org/abs/2407.15390 (2024)

    2024 arXiv

  22. [30]

    ALLaM -2-7b-instruct

    Saudi Data and AI Authority. ALLaM -2-7b-instruct. Azure AI Foundry model catalog. https://ai.azure.com/catalog/models/ALLaM-2-7b-instruct (accessed 22 June 2026)

    2026

  23. [31]

    Anwar, M. et al. Jais 2: A Family of Arabic-Centric Open Large Language Models. Technical report, Inception, MBZUAI and Cerebras. https://huggingface.co/inceptionai/Jais-2-70B-Chat (2025)

    2025

  24. [32]

    Fanar 2.0: Arabic Generative AI Stack

    FANAR TEAM et al. Fanar 2.0: Arabic Generative AI Stack. arXiv https://arxiv.org/abs/2603.16397 (2026)

    2026

  25. [33]

    Abbas, U. et al. Fanar -Sadiq: A Multi-Agent Architecture for Grounded Islamic QA. arXiv https://arxiv.org/abs/2603.08501 (2026)

    2026 arXiv

  26. [34]

    Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Adv. Neural Inf. Process. Syst. 36 (2023). arXiv https://arxiv.org/abs/2306.05685

    2023 arXiv

  27. [35]

    Kim, S. et al. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proc. 2024 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, 2024)

    2024

  28. [36]

    & Ermon, S

    Kuleshov, V., Fenner, N. & Ermon, S. Accurate uncertainties for deep learning using calibrated regression. In Proc. 35th International Conference on Machine Learning, PMLR 80, 2796–2804 (2018)

    2018

  29. [37]

    Nixon, J. et al. Measuring calibration in deep learning. In Proc. CVPR Workshops (2019). Author Contributions Shahd Gaben: Writing, Methodology, Data Curation, Software, Formal Analysis, Validation, Visualization, Project Administration. Heba Sbahi: Data Curation, Investigatio...

    2019

  30. [38]

    legal precondition

    Dataset-related materials S1. Scholarly-demand tier examples Table S1. Scholarly-demand examples from Jurisprudence. The examples show how the tier label changes the required operation, not merely the source title. Tier Example question Beginner الصائم؟ على الفطر يجب متى When ...

  31. [39]

    :الرسول وكقول رضاعا، الابن كابن الأصلاب من ليسوا الذين الأبناء حلائل المخالفة مفهوم ،]٣٢ :النساء} [أصلابكم من الذين أبنائكم وحلائل {

    Evaluation-related materials S5. Default prediction prompt templates All subject-system prompts follow the zero -shot protocol described in the main article. This section reproduces one complete rendered example per task format. The discipline-specific role instruction is inst...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.