Pith. sign in

REVIEW 5 major objections 4 minor 27 references

Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning on an expanded 1,895-question dataset lifts Qur'anic verse retrieval, raising AraBERT's mean average precision at 10 from 0.22 to 0.36.

desk verdict Useful dataset expansion, but the causal claim about dataset size is untested because fine-tuning and dataset size are confounded. read the letter →

arxiv 2412.11431 v1 pith:CBX7UFNP submitted 2024-12-16 cs.CL cs.IR

Mohamed Basem , Islam Oshallah , Baraa Hikal , Ali Hamdi , Ammar Mohamed This is my paper · ORCID
classification cs.CLcs.IR
keywords QuranquestionansweringpassageretrievaldatasetexpansionArabictransformersfine-tuningzero-answerhandling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the bottleneck in Qur'anic question answering is not only model choice but the small size and narrow phrasing of the training data. By expanding the shared-task dataset from 251 to 1,895 questions through rephrasing and categorization, and fine-tuning Arabic transformer models on that set, the authors report large gains in verse-retrieval ranking and in correctly declining to answer unanswerable questions. If the result holds, it means a modest investment in dataset expansion can unlock practical accuracy in a low-resource, classical-language QA setting. The most striking numbers are AraBERT-base's MAP@10 rising from 0.22 to 0.36 and no-answer handling rising from 25% to 75%.

What carries the argument

The machinery is a data-expansion and fine-tuning loop. The authors take the 251-question Qur'an QA dataset, expand it to 629 questions by reformulation, then rephrase each question twice to obtain 1,895 training questions labeled as single-answer, multi-answer, or zero-answer. They fine-tune several Arabic transformer models (AraBERT-base/large, CAMeLBERT, AraELECTRA, RoBERTa, BERT-squad) on this set, then add ensemble prediction and a confidence-thresholding step that discards low-confidence retrievals. The threshold is what lets the system answer 'no answer' instead of guessing, and the dataset diversity is what lets the models generalize across phrasings.

What would settle it

Run the identical fine-tuning and thresholding pipeline on the original 174-question training split and evaluate on the same 52-question test set; if MAP@10 and MRR do not drop materially, the expanded dataset is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper claims that expanding the Qur'an QA 2023 dataset from 251 to 1,895 questions, by rephrasing and categorizing questions, and then fine-tuning transformer language models on this enlarged set, substantially improves Qur'anic passage retrieval. On the shared-task evaluation, AraBERT-base raises MAP@10 from 0.22 to 0.36 and MRR from 0.37 to 0.59, and the handling of unanswerable questions rises from a 25% to a 75% success rate across several models. The authors attribute these gains to the larger, more diverse training set and to thresholding of low-confidence predictions for zero-answer cases.

Load-bearing premise

The reported improvements are computed against a baseline that is never defined in the paper, so the gains attributed to dataset expansion could instead reflect fine-tuning or thresholding choices.

Editorial extensions

If this is right

  • AraBERT-base fine-tuned on the expanded set becomes a practical retrieval model for Qur'anic QA, with top-10 precision of 0.36 and a first-hit rate of 0.59.
  • Zero-answer questions can be handled reliably through confidence thresholding, raising the success rate from 25% to 75% for models like BERT-squad-accelerate.
  • Other low-resource QA tasks in classical or historical languages can use the same recipe: expand the seed questions by paraphrase, categorize answer types, then fine-tune.
  • The gains are not limited to Arabic-specific models; even general BERT-squad improves MAP@10 from 0.07 to 0.25 after fine-tuning, suggesting data quality drives much of the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An ablation that fine-tunes on the 629 non-rephrased questions would separate the contribution of dataset size from the contribution of paraphrase diversity; the paper does not isolate these two.
  • The no-answer threshold is a binary cutoff; in practice, a calibrated confidence score would let a deployed system choose its own precision/recall trade-off for abstention.
  • The same expansion-before-fine-tuning sequence could be tried on other scriptural QA corpora, where zero-answer and multi-answer labels are often sparse.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper presents an expanded Quran question-answering dataset, growing the original 251 questions to 1,895 through rephrasing, diversification, and integration of external resources. It then fine-tunes six transformer models (AraBERT-base, AraBERT-large, CAMeLBERT, AraELECTRA, RoBERTa, BERT-squad-accelerate) on the expanded dataset and evaluates passage retrieval on the shared-task test set. The abstract claims that AraBERT-base achieves MAP@10 of 0.36 and MRR of 0.59, improving over a baseline by 63% and 59%, and that the method achieves a 75% success rate on no-answer questions. The paper's central claim is that dataset expansion and fine-tuning improve retrieval and zero-answer handling.

Significance. If the causal claim were supported, the expanded 1,895-question dataset would be a useful resource for the Quran QA community, and the systematic comparison of six Arabic transformer models would provide practical guidance for passage retrieval in this domain. The paper also demonstrates the use of thresholding for no-answer detection and combines external resources (Tafseer, Kaggle, Hugging Face) with manual rephrasing. However, the current experimental design conflates fine-tuning with dataset expansion, and several reported results are inconsistent or misattributed. These issues are fixable but currently prevent the central claim from being accepted on the evidence presented.

major comments (5)
  1. [Section 5, Tables 1-3] The 'Baseline' columns in Tables 1-3 are never defined. If 'Baseline' refers to the pre-trained models without fine-tuning, the comparison conflates the effect of fine-tuning with the effect of the expanded dataset. If 'Baseline' refers to models fine-tuned on the original 251-question dataset, that condition is never run or reported. Without a control model fine-tuned on the original training set, the paper's causal claim that 'dataset expansion led to improvements' is unsupported. The authors should either define the baseline explicitly and add the missing control experiment, or soften the causal language to describe the combined effect of fine-tuning and dataset enlargement.
  2. [Abstract and Section 1] The abstract states that the best model is AraBERT-base with MAP@10 0.36 and MRR 0.59, while the contributions list in Section 1 says 'particularly with the AraBERT-large model.' Section 3.4 also states that AraBERT-large 'showed significant advancements in semantic matching and in identifying instances where there is no answer,' but Section 5 reports that AraBERT-large 'remained relatively stable, with only slight increases.' These statements are contradictory. The paper must clarify which model is the claimed best, and the abstract and contributions must match the results actually presented.
  3. [Abstract, Section 5, Tables 2 and 3] The abstract and conclusion attribute the 75% success rate on no-answer questions to 'the proposed approach.' However, in Tables 2 and 3, the No Answer Recall and No Answer Precision of 0.75 (against a baseline of 0.25) belong to bert-squad-accelerate, not to AraBERT-base, which is the headline model. For AraBERT-base, the no-answer performance is 0.25 against a baseline of 0.00 (or 0.00 in Table 3). The abstract's claim is therefore misattributed and should be corrected to refer to a specific model, not the 'proposed approach' as a whole.
  4. [Section 3.1 and Figure 1] The test set contains only 52 questions. The reported MAP@10 improvement for AraBERT-base from 0.22 to 0.36 corresponds to a small absolute number of re-ranked queries on a 52-question test set. The paper provides no confidence intervals, bootstrap estimates, or significance testing. Given the small sample, the observed improvements may be within sampling variability. The authors should report confidence intervals or perform a significance test (e.g., paired bootstrap over queries) to support the claim that the improvements are not due to chance.
  5. [Section 4] The thresholding mechanism for zero-answer cases is described only qualitatively as 'filter out low-confidence predictions.' The paper does not report the threshold value, how it was selected, or whether it was tuned on the development set or the test set. If the threshold was optimized on the test set, the no-answer results are optimistic. The authors should specify the threshold selection procedure and, ideally, report results across a range of thresholds to show robustness.
minor comments (4)
  1. [Title] The title contains a typo: 'Fine-T uned' should be 'Fine-Tuned'.
  2. [Section 3.2 and Figure 1] The relationship between the total dataset size and the train/test split is unclear. Section 3.2 says the dataset was expanded to 1,895 questions, while Figure 1 shows 1,895 training questions plus 52 test and 25 development questions. The paper should clarify whether the 1,895 total includes the test and development sets, and how the split was constructed.
  3. [Section 3.1] The data cleaning process mentions duplicate removal and formatting standardization, but it does not describe how the newly generated rephrased questions were validated for semantic equivalence. It would be helpful to know whether multiple annotators reviewed the expanded questions and what the inter-annotator agreement was.
  4. [Tables 1-3] The tables would benefit from explicit definitions of the 'Baseline' and 'Ours' conditions in the captions, as well as the number of queries used for each metric. This would prevent the current ambiguity about what is being compared.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical evaluations against a fixed test split, not derivations from the paper's own inputs.

full rationale

The paper's headline numbers (MAP@10, MRR, recall, precision) are obtained by fine-tuning standard transformer models on an expanded 1,895-question training set and evaluating on a retained 52-question test set with a 25-question development set, as shown in Figure 1. There is no equation or fitted parameter whose value is then renamed as a prediction. The 'Baseline' vs. 'Ours' columns in Tables 1-3 are empirical before/after comparisons; the fact that the baseline is not fully defined and that fine-tuning and dataset size are confounded is a threat to the causal claim 'dataset expansion led to improvements,' but it is an experimental-design concern, not circularity. The thresholding mechanism for zero-answer cases is mentioned without reported threshold values, so no specific reduction of the no-answer success rate to a tuned parameter can be exhibited. The only self-citations are Hamdi et al. (refs 12 and 13) in a general statement about imbalanced training datasets; they are not load-bearing for the central claim. No enumerated circularity pattern is supported by quoted evidence.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new entities, forces, or conserved quantities are introduced. The framework is standard supervised fine-tuning. The key unstated inputs are the threshold parameter, unreported hyperparameters, and the unverified equivalence between rephrased questions and original answer annotations.

free parameters (2)
  • zero-answer threshold = not reported
    Section 4 mentions a thresholding mechanism to filter low-confidence predictions for zero-answer cases. No threshold value or selection procedure is given; it is presumably tuned on the development set and directly affects the no-answer recall figures.
  • fine-tuning hyperparameters (learning rate, batch size, epochs) = not reported
    Section 3.4 states 'extensive experimentation with different hyperparameters' but does not report final values. The reported scores cannot be reproduced or attributed to the dataset without these settings.
assumptions (3)
  • domain assumption Rephrasing a question twice preserves the original annotated answer
    Section 3.2 expands 629 questions to 1,895 by rephrasing each twice. The model's training labels assume the paraphrases refer to the same Qur'anic passages. If rephrasing alters meaning, the training data is mislabeled and the evaluation would be unreliable.
  • domain assumption The added external sources follow the same annotation schema as the Qur'an QA test set
    Section 3.1 integrates questions from Tafseer books, Kaggle, Hugging Face, and plant citations into the training set. The evaluation on the 52-question test set assumes label compatibility across these heterogeneous sources, which is not verified.
  • ad hoc to paper The 'Baseline' in Tables 1-3 differs from 'Ours' only in the training dataset
    The paper never defines what the baseline is. The causal claim that dataset expansion drives the improvements requires this assumption, but the paper provides no evidence about what the baseline models were trained on or evaluated under.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models." pith.science (2026). https://pith.science/paper/CBX7UFNP

@misc{pith2026241211431,
  author       = {Pith},
  title        = {Pith review of: Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CBX7UFNP}},
  note         = {Machine review of arXiv:2412.11431}
}
read the original abstract

Understanding the deep meanings of the Qur'an and bridging the language gap between modern standard Arabic and classical Arabic is essential to improve the question-and-answer system for the Holy Qur'an. The Qur'an QA 2023 shared task dataset had a limited number of questions with weak model retrieval. To address this challenge, this work updated the original dataset and improved the model accuracy. The original dataset, which contains 251 questions, was reviewed and expanded to 629 questions with question diversification and reformulation, leading to a comprehensive set of 1895 categorized into single-answer, multi-answer, and zero-answer types. Extensive experiments fine-tuned transformer models, including AraBERT, RoBERTa, CAMeLBERT, AraELECTRA, and BERT. The best model, AraBERT-base, achieved a MAP@10 of 0.36 and MRR of 0.59, representing improvements of 63% and 59%, respectively, compared to the baseline scores (MAP@10: 0.22, MRR: 0.37). Additionally, the dataset expansion led to improvements in handling "no answer" cases, with the proposed approach achieving a 75% success rate for such instances, compared to the baseline's 25%. These results demonstrate the effect of dataset improvement and model architecture optimization in increasing the performance of QA systems for the Holy Qur'an, with higher accuracy, recall, and precision.

Figures

Figures reproduced from arXiv: 2412.11431 by the authors.

Figure 1
Figure 1. Architecture Diagram: The workflow for dataset expansion and model fine￾tuning. The old dataset is manipulated to create a larger set, which is then fed into various Language Modelss for fine-tuning, resulting in improved question-answer pairs. 3.1 Data Collection The dataset was collected from various trustworthy sources to provide a wide range of questions and related Qur’anic verses. The aim was to gather reli￾ab… view at source ↗
Figure 2
Figure 2. A sample from shared task A. We retrieve the most relevant Qur’anic segment [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 13 canonical work pages

  1. [1]

    In: Proceedings of ArabicNLP 2023, pp

    Alawwad, H., Alawwad, L., Alharbi, J., Alharbi, A.: Ahjl at qur’an qa 2023 shared task: Enhancing passage retrieval using sentence transformer and translation. In: Proceedings of ArabicNLP 2023, pp. 702–707 (2023)

    2023

  2. [2]

    case study: Arabic fake news detection

    Aljamel, A., Khalil, H., Aburawi, Y.: Comparative study of fine-tuned bert-based models and rnn-based models. case study: Arabic fake news detection. The International Journal of Engineering and Information Technology (IJEIT)12(1), 56–64 (2024)

    work page 2024

  3. [3]

    arXiv preprint arXiv:2003.00104 (2020)

    Antoun, W., Baly, F., Hajj, H.: Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104 (2020). URL https://arxiv.org/abs/2003.00104

    arXiv 2020

  4. [4]

    In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp

    Antoun, W., Baly, F., Hajj, H.: Araelectra: Pre-training text discriminators for arabic language understanding. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 191–195. Association for Computational Linguistics (2021). URL https://aclanthology.org/2021.wanlp-1.21/

    work page 2021

  5. [5]

    Noor Book (2023)

    Ashor, Q.: 1000 QAs from the Holy Qur’an. Noor Book (2023). URL https://quranpedia.net/book/451/1/259

    2023

  6. [6]

    Transactions of the ACL (2020)

    Clark, J., et al.: Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the ACL (2020)

    work page 2020

  7. [7]

    In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186. Association for Computational Linguistics (2019). URLhttps://ac...

    2019

  8. [8]

    GitHub Repository

    Elkomy, M.: Quran qa 2022 dataset (2022). GitHub Repository

    work page 2022

Show all 27 references
  1. [9]

    In: Proceedings of ArabicNLP 2023, pp

    Elkomy, M., Sarhan, A.: Tce at qur’an qa 2023 shared task: Low resource enhanced transformer-based ensemble approach for qur’anic qa. In: Proceedings of ArabicNLP 2023, pp. 728–742. Association for Computational Linguistics, Singapore (Hybrid) (2023)

    2023

  2. [10]

    Artificial Intelligence Review57(9), 1–37 (2024)

    Essam, M., Deif, M., Elgohary, R.: Deciphering arabic question: A dedicated survey on arabic question analysis methods, challenges, limitations and future pathways. Artificial Intelligence Review57(9), 1–37 (2024)

    2024

  3. [11]

    GARDEN, Q.B.: List of plants citation in quran and hadith v5.pdf (2024)

    2024

  4. [12]

    Journal of Computer Science (2016)

    Hamdi, A., Shaban, K., Zainal, A.: A review on challenging issues in arabic sentiment analysis. Journal of Computer Science (2016)

    2016

  5. [13]

    ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 17(4), 1–28 (2018)

    Hamdi, A., Shaban, K., Zainal, A.: Clasenti: a class-specific sentiment analysis framework. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 17(4), 1–28 (2018)

    2018

  6. [14]

    Springer (2019)

    Hillman, J., Baydoun, E.: Quality assurance and relevance in academia: a review. Springer (2019)

    2019

  7. [15]

    URL https://huggingface.co/datasets/ImruQays/ Quran-Classical-Arabic-English-Parallel-texts 12 Authors Suppressed Due to Excessive Length

    ImruQays: Quran-classical-arabic-english parallel texts dataset on hugging face (2024). URL https://huggingface.co/datasets/ImruQays/ Quran-Classical-Arabic-English-Parallel-texts 12 Authors Suppressed Due to Excessive Length

    2024

  8. [16]

    In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp

    Inoue, G., Habash, N.: Camelbert: A language model for arabic. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 270–278. Association for Computational Linguistics (2021). URL https://aclanthology.org/2021.wanlp-1.29/

    2021

  9. [17]

    arXiv preprint arXiv:1907.11692 (2019)

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). URL https://arxiv.org/abs/1907.11692

    2019 arXiv

  10. [18]

    arXiv preprint arXiv:2308.13821 (2023)

    Liu, Z., Li, Y., Chen, N., Wang, Q., Hooi, B., He, B.: A survey of imbalanced learning on graphs: Problems, techniques, and future directions. arXiv preprint arXiv:2308.13821 (2023)

    2023 arXiv

  11. [19]

    In: Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023)

    Mahmoudi, G., Eetemadi, S., Morshedzadeh, Y.: A multi-task transfer learning approach for qur’an-related question answering. In: Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023). ACL Anthology (2023)

    2023

  12. [20]

    In: ArabicNLP-WS 2023, pp

    Malhas, M., et al.: Qur’an qa 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur’an. In: ArabicNLP-WS 2023, pp. 1–13. Association for Computational Linguistics (2023)

    2023

  13. [21]

    Malhas, R.: Arabic question answering on the holy qur’an. Ph.D. thesis, Ph.D. thesis (2023)

    2023

  14. [22]

    URL https://www.kaggle.com/datasets/mobassir/quranqa/code

    Mobassir: Quran qa dataset on kaggle (2024). URL https://www.kaggle.com/datasets/mobassir/quranqa/code

    2024

  15. [23]

    Preprint submitted to Elsevier (2024)

    Qamar, F., Latif, S., Latif, R.: A benchmark dataset with larger context for non-factoid question-answering over islamic text. Preprint submitted to Elsevier (2024)

    2024

  16. [24]

    Rashad, M.: Quran-tafseerbook dataset on hugging face (2024)

    2024

  17. [25]

    Oxford University Press (2017)

    Sardar, Z.: Reading the Qur’an: The contemporary relevance of the sacred text of Islam. Oxford University Press (2017)

    2017

  18. [26]

    arXiv preprint arXiv:2010.02394 (2020)

    Sun, L., Xia, C., Yin, W., Liang, T., Yu, P.S., He, L.: Mixup-transformer: Dynamic data augmentation for nlp tasks. arXiv preprint arXiv:2010.02394 (2020)

    2020 arXiv

  19. [27]

    arXiv preprint arXiv:2310.08184 (2023)

    Zheng, H., Shen, L., Tang, A., Luo, Y., Hu, H., Du, B., Tao, D.: Learn from model beyond fine-tuning: A survey. arXiv preprint arXiv:2310.08184 (2023)

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.