REVIEW 4 major objections 4 minor 21 references
IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read IndicSQuAD delivers translated SQuAD datasets for ten Indic languages, claiming the largest multi-Indic extractive QA resource to date.
desk verdict A genuinely useful translated QA resource for ten Indic languages, but the paper's central claim about answer-span accuracy is unverified; needs human evaluation before it can be fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a similarity-based answer-span alignment algorithm. After sentence segmentation, the sentence containing the answer and the answer string are translated independently; then every substring of the translated sentence is compared against the translated answer using embedding-based similarity scores. The best substring is chosen as the base answer, and adjacent words are appended iteratively as long as the score stays within 1% of the maximum, with named entities and numbers transliterated in a final step. This mechanism is what is supposed to guarantee that translated answers line up with their passages despite word-order and morphological differences.
What would settle it
Have native speakers annotate a random sample of, say, 300 answer spans per language from the released test sets, marking whether the stored span is the exact and only correct answer to the question within the passage; if the exact-match rate is substantially below the near-perfect alignment the pipeline claims, the central fidelity claim is weakened.
Extended reading notes
Core claim
The paper's central claim is that a robust, automated translation-and-alignment recipe, originally built for Marathi, scales to a ten-language Indic QA resource. For each language, the recipe segments English contexts into sentences, translates the answer-bearing sentence and the answer separately, then uses embedding similarity to select the substring of the translated sentence that best matches the translated answer, expanding it word by word while similarity stays within 1% of the maximum. Named entities and numbers are transliterated to maintain script consistency. The paper reports that this produces, for each of the ten languages, a dataset identical in size to the Marathi one and presents the collection as the largest multi-Indic QA resource to date, with baseline results favoring language-specific BERT models over MuRIL-BERT.
Load-bearing premise
The dataset's usefulness rests on the assumption that the 1% similarity-based substring selection finds the correct answer span in every translated passage, even though this was only manually verified for Marathi and not for the nine new languages.
Editorial extensions
If this is right
- Each of the ten languages gains 118,516 training examples, a scale that previously existed for only a few of them.
- Researchers can fine-tune language-specific BERT models on identical data sizes, making cross-language comparisons of model ability more direct.
- The pipeline, if truly language-agnostic, can be rerun on additional low-resource languages to produce comparable QA sets.
- Because all languages are derived from the same English source, the ten datasets form a controlled testbed for measuring how well multilingual models transfer across Indic languages.
- The reported baselines indicate that monolingual models remain the stronger choice when per-language training data is available, while multilingual models lag on the lower-resource languages.
Reading between the lines
- A per-language audit of answer-span boundaries would test whether the 1% similarity acceptance threshold is equally appropriate for Dravidian and Indo-Aryan languages, since the reported human verification is limited to Marathi.
- Because all ten datasets derive from a common English source, cross-language performance gaps mix genuine linguistic difficulty with translation artifacts; treating them purely as language difficulty would overread the benchmark.
- The transliteration of named entities into Devanagari may introduce script-mixed passages in languages that use other scripts, so script normalization is a cheap, testable modification that could shift the reported scores.
- A small, manually verified gold set per language, along the lines of the one reported for Marathi, would let future work quantify span-alignment noise and could serve as a filtered training subset.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces IndicSQuAD, an extractive question-answering dataset constructed by translating SQuAD 2.0 into ten Indic languages (Marathi plus nine new languages) using a pipeline adapted from the MahaSQuAD work. The paper reports splits of 118,516 training, 11,873 validation, and 11,803 test examples per language, evaluates monolingual BERT models and MuRIL-BERT on each language, and claims that the translation procedure preserves linguistic fidelity and accurate answer-span alignment. It further claims that IndicSQuAD is the largest multi-Indic QA resource to date and that monolingual models generally outperform the multilingual baseline.
Significance. If the span-alignment and linguistic-fidelity claims were verified, IndicSQuAD would be a substantial resource: roughly 118k training examples per language for ten languages would support training and benchmarking of extractive QA in languages that currently lack large-scale resources. The public release of the dataset and model links is a practical strength, and building on the documented MahaSQuAD pipeline gives a concrete starting point. However, the strongest claims in the abstract are not backed by human verification or error analysis for the newly added languages, and the methodology section contains language-specific artifacts that call into question whether the described procedure was actually applied uniformly to all target languages.
major comments (4)
- [Section 3.3, Steps 3 and 6] The algorithm is written entirely for Marathi: Step 3 says the sentence and answer are 'translated into Marathi (target language)', and Step 6 says named entities and numerical values are 'transliterated into the Devanagari script using the AI4Bharat Transliteration Engine'. Table 2, however, lists Tamil, Telugu, Kannada, and Malayalam, which use non-Devanagari scripts. As written, applying this pipeline to those languages would produce Devanagari text instead of the target script, which contradicts the dataset's stated script coverage and leaves the 'accurate answer-span alignment' claim unsupported for those languages. The manuscript must specify how the translation and transliteration steps are specialized for each target language and script.
- [Section 3.3, Steps 4-5; Section 2.1] The paper reports no manual verification or error analysis of the automatically selected answer spans for the nine languages added in this work. Section 2.1 notes that MahaSQuAD included a gold test set of 500 manually verified examples, but no equivalent is reported for IndicSQuAD. Because the span-selection method is an unsupervised similarity search (Step 4) with a fixed 1% extension tolerance (Step 5), and because morphologically rich Dravidian languages may challenge substring matching, the central claim of 'accurate answer-span alignment across diverse languages' is not currently supported. A language-wise alignment accuracy estimate on a manually checked sample, or a substantial weakening of the claim, is needed.
- [Section 4.3, Table 1] The baseline comparisons are presented without error bars, multiple random seeds, or significance tests. Several differences are small (e.g., Bengali EM 50.07 vs 49.36; Punjabi EM 51.04 vs 50.80), and the conclusion that monolingual models outperform MuRIL-BERT across most languages could be an artifact of run-to-run variance. The authors should report standard deviations and statistical tests, or soften the comparative claims accordingly.
- [Abstract; Section 3.2; Table 2; Conclusion] The language count is inconsistent: the abstract and conclusion describe 'nine major Indic languages', while contribution 1 and Table 2 list ten languages including Marathi, and Section 3.2 says the dataset covers 9 languages but then discusses ten. Since 'nine new languages plus Marathi' versus 'ten total' is a basic factual claim about the resource, the manuscript should state the count consistently.
minor comments (4)
- [Section 2.2 and Section 2.3] The same dataset is introduced twice as 'ChaII Dataset' and 'ChAII Dataset', and Section 2.3 attributes it to Singh et al. (2025) although the surrounding text and the original reference are Thirumala and Ferracane (2022). Please merge the duplicate descriptions and correct the citation.
- [Section 4.2] The hyperparameter n_best_size is set to 2, which is unusually low for extractive QA and may strongly restrict the candidate answer spans. Please clarify whether this is the intended value and whether it affects the reported baseline numbers.
- [Section 3.4 and Table 3] The rows of Table 3 are labeled 'entities' rather than 'instances' or 'examples', and Section 3.1 says the source is SQuAD 2.0 but does not specify whether the counts correspond to the full SQuAD 2.0 split. Please align the terminology and identify the exact source split.
- [References] The MuCoT work appears twice as Kumar et al. 2022a and 2022b with slightly different venue formatting; the duplicate entry should be removed or merged.
Circularity Check
No significant circularity: the dataset construction and baselines are self-contained and do not derive their central claims from fitted parameters or self-citations.
full rationale
IndicSQuAD's construction is a translation-and-span-alignment pipeline, not a derivation of predictions from fitted inputs. The answer-span alignment in Section 3.3 uses a fixed 1% similarity threshold inherited from the authors' prior MahaSQuAD work; this is a design heuristic applied to produce the dataset, not a parameter fitted to the paper's own evaluation targets. The Abstract's claim of 'accurate answer-span alignment' is an asserted quality property, and even if it is under-verified for the nine new languages, that is a validation or correctness concern rather than circularity. Self-citations to MahaSQuAD, MahaNLP, and L3Cube monolingual BERT models are legitimate references to externally available, code-reproduced prior artifacts; none of the paper's central claims reduces to those citations alone. The monolingual model baselines are trained on the released dataset, but the paper does not use those baseline results to justify the dataset's construction, so there is no statistically forced 'prediction.' No load-bearing step in the paper is equivalent to its own input by construction, and no uniqueness theorem or ansatz is smuggled in via self-citation. The main risks are empirical (label alignment quality and threshold transfer) and belong in correctness review, not circularity review.
Assumptions & free parameters
free parameters (1)
- Similarity threshold for span extension =
1%
assumptions (4)
- domain assumption Google Translate produces translations that preserve question-answer semantics
- domain assumption Similarity-based substring matching correctly locates the answer span in the translated sentence
- domain assumption Unanswerable questions in SQuAD 2.0 remain unanswerable after translation
- ad hoc to paper The 1% similarity threshold generalizes across all ten languages
Cite this review
Pith. "Pith review of IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages." pith.science (2026). https://pith.science/paper/KSWLOS5Z
@misc{pith2026250503688,
author = {Pith},
title = {Pith review of: IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages},
year = {2026},
howpublished = {\url{https://pith.science/paper/KSWLOS5Z}},
note = {Machine review of arXiv:2505.03688}
}
read the original abstract
The rapid progress in question-answering (QA) systems has predominantly benefited high-resource languages, leaving Indic languages largely underrepresented despite their vast native speaker base. In this paper, we present IndicSQuAD, a comprehensive multi-lingual extractive QA dataset covering nine major Indic languages, systematically derived from the SQuAD dataset. Building on previous work with MahaSQuAD for Marathi, our approach adapts and extends translation techniques to maintain high linguistic fidelity and accurate answer-span alignment across diverse languages. IndicSQuAD comprises extensive training, validation, and test sets for each language, providing a robust foundation for model development. We evaluate baseline performances using language-specific monolingual BERT models and the multilingual MuRIL-BERT. The results indicate some challenges inherent in low-resource settings. Moreover, our experiments suggest potential directions for future work, including expanding to additional languages, developing domain-specific datasets, and incorporating multimodal data. The dataset and models are publicly shared at https://github.com/l3cube-pune/indic-nlp
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. https://arxiv.org/abs/1910.11856 On the cross-lingual transferability of monolingual representations . CoRR, abs/1910.11856
arXiv 2019
-
[4]
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. https://arxiv.org/abs/2003.05002 Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages . Preprint, arXiv:2003.05002
arXiv 2020
-
[5]
Samruddhi Deode, Janhavi Gadre, Aditi Kajale, Ananya Joshi, and Raviraj Joshi. 2023. L3cube-indicsbert: A simple approach for learning cross-lingual sentence representations using multilingual bert. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 154--163
2023
-
[6]
Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. 2018. MMQA: A Multi-domain Multi-lingual Question-Answering Framework for English and Hindi . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA)
work page 2018
-
[7]
Raviraj Joshi. 2022 a . L3cube-hindbert and devbert: Pre-trained bert transformer models for devanagari based hindi and marathi languages. arXiv preprint arXiv:2211.11418
arXiv 2022
-
[8]
Raviraj Joshi. 2022 b . L3cube-mahacorpus and mahabert: Marathi monolingual corpus, marathi bert language models, and resources. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference, pages 97--101
work page 2022
Show all 21 references
-
[9]
Raviraj Joshi. 2022 c . L3cube-mahanlp: Marathi natural language processing datasets, models, and library. arXiv preprint arXiv:2205.14728
2022 arXiv
-
[10]
Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. https://arxiv.org/abs/2103.107...
2021 arXiv
-
[11]
Gokul Karthik Kumar, Abhishek Gehlot, Sahal Shaji Mullappilly, and Karthik Nandakumar. 2022 a . Mucot: Multilingual contrastive training for question-answering in low-resource languages. In Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian La...
2022
-
[12]
Gokul Karthik Kumar, Abhishek Singh Gehlot, Sahal Shaji Mullappilly, and Karthik Nandakumar. 2022 b . https://arxiv.org/abs/2204.05814 Mucot: Multilingual contrastive training for question-answering in low-resource languages . Preprint, arXiv:2204.05814
2022 arXiv
-
[13]
Patrick S. H. Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. https://arxiv.org/abs/1910.07475 MLQA: evaluating cross-lingual extractive question answering . CoRR, abs/1910.07475
2019 arXiv
-
[14]
Vidula Magdum, Omkar Jayant Dhekane, Sharayu Sandeep Hiwarkhedkar, Saloni Sunil Mittal, and Raviraj Joshi. 2023. mahanlp: A marathi natural language processing library. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Confere...
2023
-
[15]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392
2016
-
[16]
Ryokan Ri, Shun Kiyono, and Sho Takase. 2024. https://arxiv.org/abs/2407.00454 Self-translate-train: Enhancing cross-lingual transfer of large language models via inherent capability . Preprint, arXiv:2407.00454
2024 arXiv
-
[17]
Pritika Rohera, Chaitrali Ginimav, Akanksha Salunke, Gayatri Sawant, and Raviraj Joshi. 2024. L3cube-indicquest: A benchmark question answering dataset for evaluating knowledge of llms in indic context. arXiv preprint arXiv:2409.08706
2024 arXiv
-
[18]
Md Rashad Al Hasan Rony, Sudipto Kumar Shaha, Rakib Al Hasan, Sumon Kanti Dey, Amzad Hossain Rafi, Amzad Hossain Rafi, Ashraf Hasan Sirajee, and Jens Lehmann. 2024. https://arxiv.org/abs/2410.10229 Banglaquad: A bengali open-domain question answering dataset . Preprint, arXiv:...
2024
-
[19]
Ghatage Ruturaj, Kulkarni Aditya Ashutosh, Patil Rajlaxmi, Endait Sharvi, and Joshi Raviraj. 2023. Mahasquad: Bridging linguistic divides in marathi question-answering. In Proceedings of the 20th International Conference on Natural Language Processing (ICON), pages 497--505
2023
-
[20]
Abhishek Kumar Singh, Vishwajeet kumar, Rudra Murthy, Jaydeep Sen, Ashish Mittal, and Ganesh Ramakrishnan. 2025. https://arxiv.org/abs/2407.13522 Indic qa benchmark: A multilingual benchmark to evaluate question answering capability of llms for indic languages . Preprint, arXi...
2025 arXiv
- [21]
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.