Pith. sign in

REVIEW 4 major objections 5 minor 22 references

From RAG to Agentic RAG for Faithful Islamic Question Answering

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that giving a model tools to search and read the Qur'an before answering—agentic RAG—produces the largest and most language-balanced gains in grounded Islamic question answering.

desk verdict Useful bilingual Islamic QA benchmark and an honest empirical sweep, but the agentic-vs-standard RAG headline is confounded by unequal evidence budgets and a weakly calibrated LLM judge. read the letter →

arxiv 2601.07528 v2 pith:WIN6CH4Q submitted 2026-01-12 cs.CL cs.AI

Gagan Bhatia , Hamdy Mubarak , Mustafa Jarrar , George Mikros , Fadi Zaraket , Mahmoud Alhirthani , Mutaz Al-Khatib , Logan Cochrane
show 3 more authors
Kareem Darwish Rashid Yahiaoui Firoj Alam
This is my paper
classification cs.CLcs.AI
keywords IslamicquestionansweringagenticRAGretrieval-augmentedgenerationhallucinationmeasurementabstentionbilingualbenchmarkQur'angroundingLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds a measure-first resource for Islamic question answering: 3,810 bilingual questions with single atomic gold answers, graded as Correct, Incorrect, or Not_Attempted so that hallucination and abstention are directly visible. It then asks which intervention—supervised tuning, preference alignment, standard retrieval, or agentic retrieval—most improves grounded correctness. The answer is agentic retrieval: forcing the model to search the Qur'an, inspect verses, and iterate before answering gives the largest gains, and it makes a small model nearly language-balanced. This matters because ungrounded religious answers can be consequential, and the result suggests tool-mediated evidence-seeking is a practical path to more faithful assistants.

What carries the argument

The load-bearing component is the agentic RAG loop built on a verse-level Qur'an corpus of 6,236 atomic ayat and a small toolset: semantic search, surah metadata lookup, direct verse reading, and within-surah search. The model follows a constrained two-turn protocol—an evidence-seeking turn that invokes tools and inspects candidate verses, then a final answer turn that cites sources. The benchmark's atomic single-gold answers and strict three-way grading (Correct/Incorrect/Not_Attempted) make hallucination and abstention measurable, which is what lets the authors attribute gains to grounding rather than conversational fluency.

What would settle it

Regrade a random sample of roughly 1,000 benchmark items using trained human Islamic-law annotators who are blind to which system produced each answer and who do not privilege verse-citation style; if the agentic-RAG advantage over single-shot RAG shrinks to near zero, the reported gain is partly a judge artifact. A simpler check is to swap the judge for a different model family and compare the ordering of Base, RAG, and Agentic RAG.

Watch

Extended reading notes

Core claim

On a new bilingual benchmark of 3,810 atomic Islamic questions, retrieval grounding consistently improves correctness, and an agentic setup that lets the model search, read verses, and revise before answering produces the largest gains: Qwen3-4B-2507 rises from 21.85 (base) to 38.85 (standard RAG) to 48.90 (agentic RAG), while the Arabic–English gap narrows from about 12 points to about 1.4 points. The best overall configuration is the largest Arabic-centric model with agentic RAG, reaching 57.30 average accuracy. The authors interpret this as evidence that iterative evidence seeking and explicit verse inspection, not merely retrieving a fixed context, are what convert retrieval into reliabl

Load-bearing premise

The entire accuracy scale rests on the validity of the automatic judge's labels, which were calibrated on only 200 human-annotated items (79% agreement, κ=0.51) and which were also used to rewrite the benchmark's gold answers, so judge preference could masquerade as correctness.

Editorial extensions

If this is right

  • Agentic RAG is the single most effective intervention among those tested, adding about 10 points over standard RAG for a small model and producing the best overall result of 57.30.
  • Retrieval, in either form, improves correctness across all tested backbones, confirming that the benchmark is knowledge-intensive and that parametric memory alone is insufficient.
  • Agentic RAG narrows the Arabic–English performance gap, indicating that anchoring generation to a shared canonical retrieval base improves cross-lingual robustness.
  • Generative evaluation with abstention reveals large reliability gaps that MCQ-style benchmarks hide: most generalist multilingual models score below 30% under strict grading.
  • Supervised tuning and reward-based alignment give smaller and less consistent gains than retrieval, so they are complements rather than substitutes for grounding.
  • Because the judge can label a response Not_Attempted, abstention becomes a measurable and rewarded behavior, enabling systems to decline rather than hallucinate.
  • The gain from agentic RAG is concentrated in iterative evidence collection and verse inspection, which suggests that retrieval quality and tool orchestration, not model size, are the limiting factors.
  • If the judge's preference for verse citations inflates the agentic advantage, then the ordering of interventions could change under a different grading protocol; this is testable by re-grading with independent human experts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces IslamicFaithQA, a bilingual (Arabic/English) generative QA benchmark of 3,810 items with atomic gold answers and a strict Correct/Incorrect/Not_Attempted LLM-judge protocol. The authors also release 25K Arabic SFT reasoning pairs, 5K bilingual preference samples, and a 6,236-ayah Qur'an retrieval corpus. Using these resources, they compare Base, +SFT, +RL, +RAG, and +Agentic RAG configurations across several models and report that retrieval is the most consistently effective intervention, that agentic RAG yields the largest gains beyond standard RAG (Qwen3-4B-2507: 38.85 to 48.90; Fanar-2-27B: 51.50 to 57.30), and that agentic RAG narrows the Arabic–English accuracy gap.

Significance. If the central claim holds, this is a useful contribution: the benchmark targets important but under-measured failure modes (hallucination and abstention) in a high-stakes domain, and the released data suite is a substantial resource. The paper also explicitly acknowledges limitations of LLM judging and grounding scope, which is commendable. However, the headline claim that tool-mediated iterative evidence seeking—rather than simply access to more evidence—causes the observed gains is not established by the current experimental design. The LLM-as-judge setup also rests on weak human calibration and possible circularity. The resources are valuable, but the mechanistic claim needs additional evidence before the paper can be accepted.

major comments (4)
  1. [§4, Appendix C, Table 4] The standard-RAG vs agentic-RAG comparison does not control the evidence budget. Standard RAG retrieves top-5 verses once, while agentic RAG receives the same top-5 but can additionally call search_quran, read_ayah, search_surah, and get_surah_info over a two-turn budget. The agentic condition therefore has access to strictly more evidence and can reformulate queries. The reported +10.05 (Qwen3-4B) and +5.80 (Fanar-2-27B) gains could reflect additional or better-targeted evidence reaching the generator, not the value of iterative reasoning. An evidence-matched ablation is needed—e.g., feeding the exact final evidence set collected by the agent to a single-shot RAG generator, or allowing standard RAG the same number of retrieval calls—before the causal claim can be supported.
  2. [§3.3, §4, Appendix A.5] The accuracy scale rests on GPT-4.1-as-judge, and the same model was used in the curation pipeline to reformulate benchmark items and gold answers (Section 3.3, Appendix A.1). Although RL rewards come from a separate LLM judge (Qwen3-235B), the final benchmark gold answers were produced by GPT-4.1, so the grader is not independent of the answer source. Human calibration is limited to N=200 with 79% agreement and Cohen's κ=0.51, with no breakdown by language, difficulty, or output style. Given that agentic outputs often include verse citations (Figure 5), judge preference for citation-like phrasing is a concrete risk. Please report judge error patterns by language and difficulty, and re-grade a sample with human raters or a second independent judge.
  3. [§4 (last paragraph) and Table 4] The text states that agentic RAG setups were used for Fanar-1-9B, ALLaM-7B, and Qwen3-4B after training, and for Fanar-2-27B without retraining. However, Table 4 reports agentic rows only for Qwen3-4B and Fanar-2-27B. The general claim that agentic RAG yields the largest gains across backbones is therefore supported by only two configurations. Either report the missing agentic rows for ALLaM-7B and Fanar-1-9B, or qualify the claim to the models actually evaluated.
  4. [§5.1–§5.2, Tables 3–4] All results are single-run point estimates with no variance, confidence intervals, or significance tests. Generation uses T=0.7 and grading uses an LLM, so small differences may be noise; for example, ALLaM-7B +RL (39.55) vs +RAG (40.76) and Fanar-1-9B +RL (38.18) vs +RAG (41.20) are modest. To support the ranking of interventions, please provide multiple runs or bootstrap confidence intervals on the key comparisons.
minor comments (5)
  1. [Abstract] Typo: 'We made the datasets are publicly available' should be 'We make the datasets publicly available.' The Hugging Face URL should also be included in the reference list.
  2. [Appendix A.1] The prompt text says 'IMPORTANT: Both the question and gold_answer should be in Arabic' even though the surrounding instructions describe generating English questions. Clarify the intended language of the reformulated benchmark items.
  3. [§3.3 vs Appendix F] The difficulty distribution numbers are inconsistent: Section 3.3 reports Level 3 at 31.2%, Level 4 at 21.8%, Level 1 at 22.8%, while Appendix F reports Level 3 at 26.90%, Level 2 at 23.03%, Level 4 at 20.03%, Level 1 at 17.67%. Please reconcile.
  4. [Tables 3, 4, and 7] The model name is inconsistent: Section 4 and Tables 3–4 refer to SeaLLM-7B-v3, but Table 7 lists SeaLLM-7B-v2.5. Use a single identifier.
  5. [Appendix C] Please state the exact tool-call and passage limits within each of the two turns of the agentic RAG budget, in addition to the two-turn overall budget, for reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Benchmark gold answers and the final grader are both GPT-4.1, making the accuracy scale partially self-referential; the central RAG-vs-agentic comparison remains empirical.

  1. other [Section 3.3 (ISLAMICFAITHQA Curation Pipeline) and Appendix C (LLM-as-a-Judge Grading)]
    "Before being reformulated by GPT-4.1 into short, fact-based generative questions with atomic gold answers. ... For automatic grading under the Correct/Incorrect/Not_Attempted protocol, we use GPT-4.1 as the judge model."

    The benchmark's gold answer—the definition of 'correct'—is a GPT-4.1 reformulation, and the grader deciding CORRECT/INCORRECT/NOT_ATTEMPTED is also GPT-4.1. Thus the reported %Correct is, by construction, a measure of agreement between model outputs and GPT-4.1's own rewording style, not an independent measure of Islamic correctness. The paper's own limitation statement concedes dependence on 'the correctness of the LLM judge and a limited human-calibration subset.' Human calibration (N=200, 79% agreement, Cohen's k=0.51) provides only partial external grounding. The relative RAG-vs-agentic comparison is not logically forced by this loop, so this is a partial circularity in the evaluation signal rather than a derivation of the central claim.

full rationale

The paper's central claims—retrieval improves correctness and agentic RAG yields the largest gains beyond standard RAG—come from empirical comparisons in Table 4, not from equations that reduce to fitted inputs. Standard RAG retrieves top-5 verses once, while agentic RAG has a two-turn budget with additional tool calls; this is an experimental confound (unequal evidence access) but not a circular derivation, so it does not raise the circularity score. The main genuine circularity is evaluative: GPT-4.1 is used both to reformulate the benchmark's gold answers and to grade model answers, making the accuracy scale partially self-referential. The human-validated subset and adjudication mitigate this, but the 79% agreement and k=0.51 leave substantial judge-dependent variance. The SFT/RL data and Quran retrieval corpus are external resources; no load-bearing self-citation chain or imported uniqueness theorem appears. The central agentic-vs-standard comparison is independent of the judge loop in the sense that both conditions are graded by the same judge, yet the absolute gains could be inflated by the judge favoring GPT-4.1-style phrasing (e.g., verse citations). Overall, this is partial circularity in the benchmark's correctness signal, not a derivation forced by construction; score 4.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; the agentic 'tools' (search_quran, read_ayah, etc.) are software functions, not invented entities. The main assumptions are about the validity of LLM-based evaluation, the atomic-answer framing, and Qur'an-only grounding.

free parameters (5)
  • retrieval_top_k = 5
    Top-5 verses are retrieved in standard RAG and exposed to agentic tools; chosen by hand, no sweep reported.
  • agentic_interaction_budget = two turns
    Agentic RAG is limited to exactly two turns (evidence-seeking + final answer); the size of the gains depends on this budget.
  • evaluation_temperature = 0.7
    Decoding temperature for benchmark generation; no justification or sensitivity analysis is given.
  • SFT_learning_rate = 5e-5
    Learning rate chosen for supervised fine-tuning; reported in Appendix C.
  • RL_learning_rate = 3e-6
    Learning rate for GSPO alignment; reported in Appendix C.
assumptions (4)
  • domain assumption LLM-as-a-judge labels are a valid proxy for human correctness
    All accuracy scores are GPT-4.1 labels; human calibration is limited (N=200 in §4, 79% agreement, κ=0.51), so benchmark validity depends on judge reliability. Invoked in §4 and Appendix A.5.
  • domain assumption Atomic single-gold answers exist for all benchmark items
    The benchmark design assumes each question has one decisive Qur'an-grounded answer, despite acknowledged multiplicity across madhahib. Stated in §3.2 and Limitations.
  • domain assumption The 6,236-ayah corpus is sufficient grounding for benchmark answers
    Grounding is Qur'an-only; questions requiring Hadith or fiqh consensus may be disadvantaged, as acknowledged in Limitations.
  • domain assumption Dense retrieval with mE5-base identifies relevant verses
    Retrieval quality is assumed from the encoder choice; no retrieval evaluation (recall/accuracy of retrieved verses) is reported. §4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From RAG to Agentic RAG for Faithful Islamic Question Answering." pith.science (2026). https://pith.science/paper/WIN6CH4Q

@misc{pith2026260107528,
  author       = {Pith},
  title        = {Pith review of: From RAG to Agentic RAG for Faithful Islamic Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIN6CH4Q}},
  note         = {Machine review of arXiv:2601.07528}
}
read the original abstract

Large Language Models (LLMs) are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations (MCQ: Multiple choice questions, MRC: Machine Reading Comprehension) do not capture key real-world failure modes, notably free-form hallucinations and the ability to abstain when evidence is insufficient. To address this gap, we introduce IslamicFaithQA, a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modeling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur'an retrieval corpus of ~6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic-English robustness even with a small model (i.e., Qwen3 4B). We made the datasets are publicly available. https://huggingface.co/datasets/QCRI/IslamicFaithQA

Figures

Figures reproduced from arXiv: 2601.07528 by the authors.

Figure 1
Figure 1. Current–Proposed–Outcome. (a) Current Islamic QA. (b) We combine ISLAMICFAITHQA, LLM judging, Quran retrieval, and agentic evidence seeking. (c) This yields more faithful, citation-backed responses. Islamic question answering is a particularly challenging testbed for this reliability problem. Deployed Islamic QA systems2 indicate strong demand, yet their proprietary evaluations high￾light the need for shared benchma… view at source ↗
Figure 2
Figure 2. The construction pipeline for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Statistical analysis of ISLAMICFAITHQA. (a) Difficulty Distribution: The dataset exhibits a balanced spread across five difficulty levels. (b) Topic Diversity: The benchmark covers a wide range of theological do￾mains. tance Law (26.4%) and Jurisprudence (17.4%) con￾stitute the largest categories, followed by Prophetic Biography (11.4%), Islamic Creed (9.8%), and Quranic Studies (9.4%). Regarding cognitive de￾mands,… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: End-to-end development and evaluation work [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Example item from ISLAMICFAITHQA (Islamic Jurisprudence). We show the query, the atomic gold target, and a model predic￾tion. The LLM-as-a-judge assigns a discrete la￾bel (Correct/Incorrect/Not_Attempted) based on se￾mantic equivalence to the single-gold answer. are em…
Figure 9
Figure 9. Figure 9: Difficulty distribution across top categories [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 6
Figure 6. Figure 6: Category distribution (counts) in the Islamic [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Overall difficulty distribution and reasoning [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Reasoning complexity and step distribution [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 10
Figure 10. Figure 10: ISLAMICFAITHQA sample. set to 5 × 10−5 . All other hyperparameters (e.g., optimizer choice, batch size, warmup schedule, gra￾dient clipping, and number of epochs) use standard settings. Preference-Based Alignment (RL). For align￾ment, we optimize the policy using the …
Figure 11
Figure 11. Figure 11: Bukhar¯ ¯ı an [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: RL preference sample. An RL instance specifies a question derived from canonical text and an atomic gold target. During RL, candidate model re￾sponses are scored by an LLM-as-a-judge against this gold target to produce scalar rewards for policy optimisa￾tion (English …
Figure 13
Figure 13. Figure 13: Annotation Interface and level 5 (12.37%). For category assignment, we consolidate heterogeneous source-specific tags into 12 unified category_type labels, with the largest classes being Inheritance Law, Islamic Finance and Economics, Hadith Studies, Qur’anic Studies …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 4 linked inside Pith

  1. [1]

    Islamic Creed

    "Islamic Creed" - Questions about belief in Allah, prophets, angels, books, Day of Judgment, divine decree

  2. [2]

    In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32854–32883, Suzhou, China

    CARE: Multilingual human preference learn- ing for cultural awareness. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32854–32883, Suzhou, China. Association for Computational Linguistics. Lukas Haas, Gal Yona, Giovanni D’Antonio, Sasha Goldshtein, and Dipanjan Das. 2025. Simpleqa ver- ified: A reliable fa...

    arXiv 2025

  3. [3]

    Inheritance Law

    "Inheritance Law" - Questions about Islamic inheritance calculations and distributions

  4. [4]

    Hadith Studies

    "Hadith Studies" - Questions about prophetic traditions, their authentication, and narrators

  5. [5]

    2502.12769v3

    How much do llms hallucinate across lan- guages? on multilingual estimation of llm hallucina- tion in the wild. 2502.12769v3. Changyue Wang, Weihang Su, Qingyao Ai, and Yiqun Liu. 2025. Joint evaluation of answer and reason- ing consistency for hallucination detection in large reasoning models. 2506.04832v1. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Ya...

    arXiv 2025

  6. [6]

    id": "MIZAN-001

    Medethicsqa: A comprehensive question an- swering benchmark for medical ethics evaluation of llms. 2506.22808v1. 11 An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Day- iheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 202...

    arXiv 2025

  7. [7]

    Islamic History

    "Islamic History" - Questions about Islamic historical events, figures, and civilizations

  8. [8]

    Jurisprudence

    "Jurisprudence" - Questions about worship rituals, purification, prayer, fasting, hajj, transactions 12

Show all 22 references
  1. [9]

    Islamic Finance and Economics

    "Islamic Finance and Economics" - Questions about halal transactions, banking, business contracts

  2. [10]

    Islamic Family Law

    "Islamic Family Law" - Questions about marriage, divorce, child custody, family rights

  3. [11]

    Qur 'anic Studies

    "Qur 'anic Studies" - Questions about Qur'anic verses, tafsir, themes, stories, and interpretation

  4. [12]

    Prophetic Biography

    "Prophetic Biography" - Questions about the life of Prophet Muhammad and his companions

  5. [14]

    Islamic Ethics and Morality

    "Islamic Ethics and Morality" - Questions about moral principles, character, social interactions

  6. [17]

    Comparative Religion

    "Comparative Religion" - Questions about other religions from Islamic perspective

  7. [18]

    Contemporary Issues

    "Contemporary Issues" - Questions about modern applications of Islamic rulings Evaluate the question based on: - Depth of knowledge required - Complexity of reasoning needed - Number of sources and concepts involved - Level of nuance and ethical complexity - Whether simple rec...

  8. [19]

    search_quran(query: str) - Search for relevant Quran verses by semantic similarity

  9. [20]

    get_surah_info(surah_number: int) - Get information about a specific surah

  10. [21]

    read_ayah(surah: int, ayah: int) - Read a specific ayah with full details

  11. [22]

    name": "tool_name

    search_surah(surah_number: int, query: str) - Search within a specific surah To use a tool, respond with: <tool_call> {"name": "tool_name", "arguments": {"arg1": "value1"}} </tool_call> After gathering information from the Quran, provide your final answer with: <answer>Your fi...

    1998

  12. [2022]

    In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection

    Qur’an qa 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection. Pedro Henrique Martins, Jo...

    2022 arXiv

  13. [2023]

    2306.11507v1

    Trustgpt: A benchmark for trustworthy and responsible large language models. 2306.11507v1. Zheng Hui, Yijiang River Dong, Ehsan Shareghi, and Nigel Collier. 2025. TRIDENT: Benchmarking llm safety in finance, medicine, and law. arXiv preprint arXiv:2507.21134. Albert Q. Jiang, ...

    2025 arXiv

  14. [2025]

    2506.18199v2

    Prompt engineering techniques for mitigating cultural bias against arabs and muslims in large lan- guage models: A systematic review. 2506.18199v2. Farah Atif, Nursultan Askarbekuly, Kareem Darwish, and Monojit Choudhury. 2025. Sacred or synthetic? evaluating llm reliability a...

    2025 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.