{"id":"4d0c3245-fb4b-4cd5-893a-4c72ac1ecd8d","arxiv_id":"2412.11431","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning Arabic BERT variants on an expanded, rephrased Qur'an QA dataset improves passage retrieval scores, with AraBERT-base reaching MAP@10 0.36 and MRR 0.59.","lead":"This paper expands a Qur'anic question-answering dataset from 251 to 1,895 questions by rephrasing and adding external sources, then fine-tunes several Arabic transformer models for passage retrieval. The best reported model, AraBERT-base, improves MAP@10 from 0.22 to 0.36 and MRR from 0.37 to 0.59, but the baseline is undefined, the dataset is not released, and the best no-answer result belongs to a different model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed causal effect of dataset expansion on retrieval is untested: Tables 1–3 compare pre-fine-tuned baselines to models fine-tuned on the expanded set, so fine-tuning and dataset size are confounded.","rationale":"The reader identified the undefined baseline as the weakest assumption, and this stress-test finds the same load-bearing flaw, sharpened to a missing control: even if the baseline were precisely defined as the pre-fine-tuned model, the comparison would still not isolate dataset expansion because fine-tuning itself is a confound. The paper's own Section 4 says fine-tuning 'utilizes expanded dataset of 1895 questions,' and Section 5's tables compare 'Baseline' to 'Ours' without ever reporting a model fine-tuned on the original 174 training questions. Without that ablation, the 63% MAP@10 gain and 59% MRR gain cannot be attributed to the expanded dataset. The paper does use standard metrics and fine-tunes several known transformer models, which are real elements, but no code, data, or significance tests are provided. The internal inconsistencies—abstract naming AraBERT-base as best, contribution bullet naming AraBERT-large, and the 75% no-answer result belonging to a different model—reinforce that the quantitative claims are not reliable as presented. Therefore the reader's REJECT verdict is appropriate and unchanged.","tokens_in":8184,"tokens_out":4547,"duration_ms":40938,"concrete_test":"Run the missing ablation: fine-tune each model with identical hyperparameters, thresholding, and evaluation protocol on the original 174-question training set, and compare against the same models fine-tuned on the 1,895-question set, on the same 52-question test set. If MAP@10, MRR, and no-answer gains over the old-data model are within bootstrap confidence intervals over the 52 test queries, the dataset-expansion claim fails. Also check whether rephrased versions of test questions leaked into the expanded training set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that expanding the training set from 251 questions (174 train) to 1,895 questions improved passage retrieval and no-answer handling. The only evidence is the 'Baseline' vs. 'Ours' columns in Tables 1–3, and Section 5 explicitly frames these as 'before and after adjustment.' Under that reading, the comparison conflates two variables: applying any fine-tuning at all, and using the larger dataset. A model fine-tuned on the original 174-question training set is never evaluated, so there is no evidence that dataset expansion, rather than fine-tuning or the added external sources, caused the gains. The no-answer comparison has the same defect, and the 75% no-answer value in the abstract actually belongs to bert-squad-accelerate in Table 3, not to the headline model AraBERT-base. With only 52 test questions and no significance testing, the headline MAP@10 improvement from 0.22 to 0.36 also lies within plausible sampling noise. The causal claim is therefore not supported by the reported experimental design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an expanded Quran question-answering dataset, growing the original 251 questions to 1,895 through rephrasing, diversification, and integration of external resources. It then fine-tunes six transformer models (AraBERT-base, AraBERT-large, CAMeLBERT, AraELECTRA, RoBERTa, BERT-squad-accelerate) on the expanded dataset and evaluates passage retrieval on the shared-task test set. The abstract claims that AraBERT-base achieves MAP@10 of 0.36 and MRR of 0.59, improving over a baseline by 63% and 59%, and that the method achieves a 75% success rate on no-answer questions. The paper's central claim is that dataset expansion and fine-tuning improve retrieval and zero-answer handling.","tokens_in":8403,"tokens_out":4573,"duration_ms":36938,"significance":"If the causal claim were supported, the expanded 1,895-question dataset would be a useful resource for the Quran QA community, and the systematic comparison of six Arabic transformer models would provide practical guidance for passage retrieval in this domain. The paper also demonstrates the use of thresholding for no-answer detection and combines external resources (Tafseer, Kaggle, Hugging Face) with manual rephrasing. However, the current experimental design conflates fine-tuning with dataset expansion, and several reported results are inconsistent or misattributed. These issues are fixable but currently prevent the central claim from being accepted on the evidence presented.","major_comments":[{"comment":"The 'Baseline' columns in Tables 1-3 are never defined. If 'Baseline' refers to the pre-trained models without fine-tuning, the comparison conflates the effect of fine-tuning with the effect of the expanded dataset. If 'Baseline' refers to models fine-tuned on the original 251-question dataset, that condition is never run or reported. Without a control model fine-tuned on the original training set, the paper's causal claim that 'dataset expansion led to improvements' is unsupported. The authors should either define the baseline explicitly and add the missing control experiment, or soften the causal language to describe the combined effect of fine-tuning and dataset enlargement.","section":"Section 5, Tables 1-3"},{"comment":"The abstract states that the best model is AraBERT-base with MAP@10 0.36 and MRR 0.59, while the contributions list in Section 1 says 'particularly with the AraBERT-large model.' Section 3.4 also states that AraBERT-large 'showed significant advancements in semantic matching and in identifying instances where there is no answer,' but Section 5 reports that AraBERT-large 'remained relatively stable, with only slight increases.' These statements are contradictory. The paper must clarify which model is the claimed best, and the abstract and contributions must match the results actually presented.","section":"Abstract and Section 1"},{"comment":"The abstract and conclusion attribute the 75% success rate on no-answer questions to 'the proposed approach.' However, in Tables 2 and 3, the No Answer Recall and No Answer Precision of 0.75 (against a baseline of 0.25) belong to bert-squad-accelerate, not to AraBERT-base, which is the headline model. For AraBERT-base, the no-answer performance is 0.25 against a baseline of 0.00 (or 0.00 in Table 3). The abstract's claim is therefore misattributed and should be corrected to refer to a specific model, not the 'proposed approach' as a whole.","section":"Abstract, Section 5, Tables 2 and 3"},{"comment":"The test set contains only 52 questions. The reported MAP@10 improvement for AraBERT-base from 0.22 to 0.36 corresponds to a small absolute number of re-ranked queries on a 52-question test set. The paper provides no confidence intervals, bootstrap estimates, or significance testing. Given the small sample, the observed improvements may be within sampling variability. The authors should report confidence intervals or perform a significance test (e.g., paired bootstrap over queries) to support the claim that the improvements are not due to chance.","section":"Section 3.1 and Figure 1"},{"comment":"The thresholding mechanism for zero-answer cases is described only qualitatively as 'filter out low-confidence predictions.' The paper does not report the threshold value, how it was selected, or whether it was tuned on the development set or the test set. If the threshold was optimized on the test set, the no-answer results are optimistic. The authors should specify the threshold selection procedure and, ideally, report results across a range of thresholds to show robustness.","section":"Section 4"}],"minor_comments":[{"comment":"The title contains a typo: 'Fine-T uned' should be 'Fine-Tuned'.","section":"Title"},{"comment":"The relationship between the total dataset size and the train/test split is unclear. Section 3.2 says the dataset was expanded to 1,895 questions, while Figure 1 shows 1,895 training questions plus 52 test and 25 development questions. The paper should clarify whether the 1,895 total includes the test and development sets, and how the split was constructed.","section":"Section 3.2 and Figure 1"},{"comment":"The data cleaning process mentions duplicate removal and formatting standardization, but it does not describe how the newly generated rephrased questions were validated for semantic equivalence. It would be helpful to know whether multiple annotators reviewed the expanded questions and what the inter-annotator agreement was.","section":"Section 3.1"},{"comment":"The tables would benefit from explicit definitions of the 'Baseline' and 'Ours' conditions in the captions, as well as the number of queries used for each metric. This would prevent the current ambiguity about what is being compared.","section":"Tables 1-3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially useful dataset contribution, but the evaluation design needs a control experiment (fine-tuning on the original 251-question dataset) to disentangle the effect of dataset expansion from fine-tuning itself. The abstract and contributions also need correction regarding the best model and the no-answer statistic. If the authors can supply the missing control and fix these reporting issues, the paper could be suitable for publication in a workshop or a venue accepting incremental QA-resource papers. The small test set remains a concern that should be addressed with significance testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper expands the Quran QA dataset from 251 to 1,895 questions by rephrasing and adding external sources, then fine-tunes six Arabic transformer models and reports solid-looking gains (AraBERT-base MAP@10 0.22→0.36). The expansion is real work, but the experimental design doesn't support the claim that dataset size caused the improvement.\n\nWhat's genuinely useful: the expanded dataset is a plausible resource for a very low-resource niche, Qur'anic QA. The authors try several models and address zero-answer handling with a thresholding mechanism, which matters for this task. They also engage the relevant prior work (Mahmoudi, Alawwad, Sarhan/Elkomy). If the dataset had been released, this paper would be a useful data paper.\n\nThe soft spots are load-bearing. \"Baseline\" in Tables 1–3 is never defined. Reading Section 5, \"before and after adjustment\" appears to compare off-the-shelf models to models fine-tuned on the expanded set. That conflates two variables: applying any fine-tuning, and using 1,895 questions instead of the original 174 training questions. There is no experiment that fine-tunes on the original training set, so the causal claim about dataset expansion is untestable. The abstract names AraBERT-base as the best model, while the contributions list says AraBERT-large. The 75% no-answer success quoted in the abstract belongs to bert-squad-accelerate, not the headline model. With only 52 test questions and no significance testing, the MAP gap from 0.22 to 0.36 could easily be noise.\n\nThe dataset is not released, so the main artifact is inaccessible. As presented, the results aren't reproducible and the central claim is unsupported. A proper revision would need a controlled baseline (fine-tune on the original 174 questions), consistent reporting of the best model, release of data and code, and uncertainty estimates.\n\nWho it's for: people working on Arabic low-resource QA or Qur'anic NLP. They'd learn more from the dataset-expansion methodology than from the numbers. General NLP readers won't find much here.\n\nShould it go to peer review? I'd say yes—the subfield is underserved and the dataset effort deserves a serious look with major revisions. But in its current form I would not accept it. Recommendation: engage with it as a potential dataset paper, but require the fixes above before anything is published.\n\nBest.","headline":"Useful dataset expansion, but the causal claim about dataset size is untested because fine-tuning and dataset size are confounded.","tokens_in":8938,"tokens_out":3557,"would_cite":false,"duration_ms":27839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on an expanded 1,895-question dataset lifts Qur'anic verse retrieval, raising AraBERT's mean average precision at 10 from 0.22 to 0.36.","keywords":["Quran question answering","passage retrieval","dataset expansion","Arabic transformers","fine-tuning","zero-answer handling"],"falsifier":"Run the identical fine-tuning and thresholding pipeline on the original 174-question training split and evaluate on the same 52-question test set; if MAP@10 and MRR do not drop materially, the expanded dataset is not what drives the reported gains.","tokens_in":7968,"feed_emoji":"📖","tokens_out":6320,"duration_ms":51093,"temperature":0.7,"pith_summary":"This paper argues that the bottleneck in Qur'anic question answering is not only model choice but the small size and narrow phrasing of the training data. By expanding the shared-task dataset from 251 to 1,895 questions through rephrasing and categorization, and fine-tuning Arabic transformer models on that set, the authors report large gains in verse-retrieval ranking and in correctly declining to answer unanswerable questions. If the result holds, it means a modest investment in dataset expansion can unlock practical accuracy in a low-resource, classical-language QA setting. The most striking numbers are AraBERT-base's MAP@10 rising from 0.22 to 0.36 and no-answer handling rising from 25% to 75%.","feed_headline":"Expanded Qur'an QA data lifts verse retrieval by 63%","feed_subtitle":"AraBERT-base's MAP@10 rises from 0.22 to 0.36 and no-answer success from 25% to 75%.","key_machinery":"The machinery is a data-expansion and fine-tuning loop. The authors take the 251-question Qur'an QA dataset, expand it to 629 questions by reformulation, then rephrase each question twice to obtain 1,895 training questions labeled as single-answer, multi-answer, or zero-answer. They fine-tune several Arabic transformer models (AraBERT-base/large, CAMeLBERT, AraELECTRA, RoBERTa, BERT-squad) on this set, then add ensemble prediction and a confidence-thresholding step that discards low-confidence retrievals. The threshold is what lets the system answer 'no answer' instead of guessing, and the dataset diversity is what lets the models generalize across phrasings.","core_discovery":"The paper claims that expanding the Qur'an QA 2023 dataset from 251 to 1,895 questions, by rephrasing and categorizing questions, and then fine-tuning transformer language models on this enlarged set, substantially improves Qur'anic passage retrieval. On the shared-task evaluation, AraBERT-base raises MAP@10 from 0.22 to 0.36 and MRR from 0.37 to 0.59, and the handling of unanswerable questions rises from a 25% to a 75% success rate across several models. The authors attribute these gains to the larger, more diverse training set and to thresholding of low-confidence predictions for zero-answer cases.","pith_inferences":["An ablation that fine-tunes on the 629 non-rephrased questions would separate the contribution of dataset size from the contribution of paraphrase diversity; the paper does not isolate these two.","The no-answer threshold is a binary cutoff; in practice, a calibrated confidence score would let a deployed system choose its own precision/recall trade-off for abstention.","The same expansion-before-fine-tuning sequence could be tried on other scriptural QA corpora, where zero-answer and multi-answer labels are often sparse."],"forward_implications":["AraBERT-base fine-tuned on the expanded set becomes a practical retrieval model for Qur'anic QA, with top-10 precision of 0.36 and a first-hit rate of 0.59.","Zero-answer questions can be handled reliably through confidence thresholding, raising the success rate from 25% to 75% for models like BERT-squad-accelerate.","Other low-resource QA tasks in classical or historical languages can use the same recipe: expand the seed questions by paraphrase, categorize answer types, then fine-tune.","The gains are not limited to Arabic-specific models; even general BERT-squad improves MAP@10 from 0.07 to 0.25 after fine-tuning, suggesting data quality drives much of the improvement."],"supporting_citations":[{"why":"Defines the Qur'an QA 2023 shared task, whose limited dataset and low baseline retrieval motivate the expansion.","marker":"[20]"},{"why":"Reports the prior ensemble method and the 25.05% MAP that this work aims to surpass with a larger dataset.","marker":"[9]"},{"why":"Proposes multi-task transfer learning for Qur'anic QA and concludes that larger, more diverse datasets are needed.","marker":"[19]"},{"why":"Provides the AraBERT model family fine-tuned in the experiments.","marker":"[3]"},{"why":"Source of the 1,000 Tafseer-based question-answer pairs used to expand the training data.","marker":"[5]"}],"fun_headline_variants":["Expanded Qur'an QA 7.5x, retrieval up 63%","AraBERT on 1,895 Qur'an questions raises MAP to 0.36","Qur'an QA expansion lifts no-answer success to 75%","251 to 1,895 questions: Qur'an retrieval up 63%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported improvements are computed against a baseline that is never defined in the paper, so the gains attributed to dataset expansion could instead reflect fine-tuning or thresholding choices.","fun_headline_variants_meta":{"raw":{"variants":["Expanded Qur'an QA 7.5x, retrieval up 63%","AraBERT on 1,895 Qur'an questions raises MAP to 0.36","Qur'an QA expansion lifts no-answer success to 75%","251 to 1,895 questions: Qur'an retrieval up 63%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1428,"prompt_tokens":955,"completion_tokens":473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":387}},"tokens_in":571,"tokens_out":473,"duration_ms":4508,"temperature":1.0,"reasoning_tokens":387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:56:04.445275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical fine-tuning and thresholding pipeline on the original 174-question training split and evaluate on the same 52-question test set; if MAP@10 and MRR do not drop materially, the expanded dataset is not what drives the reported gains.","supporting_citations":[{"cited_title":"In: ArabicNLP-WS 2023, pp","cited_arxiv_id":null,"evidence_quote":"Defines the Qur'an QA 2023 shared task, whose limited dataset and low baseline retrieval motivate the expansion."}],"review_version":1}