REVIEW 5 major objections 6 minor 30 references
Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read For finding Sanskrit texts with English queries, translating the documents to English beats direct cross-lingual retrieval, the paper reports.
desk verdict A valuable new English-Sanskrit CLIR dataset whose headline DT-over-DR/QT result is confounded by labels built from the same English translations that DT uses. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Anveshana dataset: 3,400 manually crafted English queries, roughly ten per document, over 334 Srimadbhagavatam chapters scraped from the Vedabase, with each query paired to a single gold chapter. Around this dataset, the paper builds a three-armed evaluation framework — Document Translation (DT), Query Translation (QT), and Direct Retrieve (DR) — with the Google Translate API as the translation engine for DT and QT. The argument is carried by comparing BM25, ColBERT, Contriever, XLM-R, multilingual-e5, mDPR, GPT-2, and REPLUG-LSR across these arms using NDCG, MAP, Recall, and Precision at k=1,3,5,10.
What would settle it
Have two independent Sanskrit scholars annotate, for each of the 340 test queries, all Srimadbhagavatam chapters that actually answer the query, then recompute NDCG@10 for DT-BM25 and the best direct-retrieval model; if DT-BM25 no longer clearly beats direct retrieval under this multi-relevance gold standard, the central claim is an artifact of single-relevance annotation.
Extended reading notes
Core claim
The paper claims that in the English-query-to-Sanskrit-document setting, the dominant retrieval strategy is document translation followed by English monolingual retrieval, and specifically that BM25 on Google-translated Sanskrit chapters is far more effective than any of the evaluated dense cross-lingual or query-translation approaches. Across all test metrics and cutoffs, the DT framework outperforms DR and QT; at k=10, DT-BM25 reaches 62.46% NDCG while the best DR model (multilingual-e5-base) achieves 10.74% and the best QT model (BM25 on Sanskrit-translated queries) achieves 6.86%. The authors attribute this to the maturity of English retrieval models and the fact that translated documents preserve enough content for lexical matching, whereas direct embedding spaces and Sanskrit-side retrieval are poorly served by current multilingual models.
Load-bearing premise
Each English query is assumed to be relevant to exactly one Sanskrit chapter, and the manual pairing made from English translations is assumed to carry over to the Sanskrit originals; if the correspondence is loose, every reported recall, precision, NDCG, and MAP number loses its meaning.
Editorial extensions
If this is right
- If DT+BM25 is the best recipe, then building Sanskrit CLIR systems today should prioritize high-quality document translation over training cross-lingual embedders.
- The Anveshana dataset provides a standardized testbed for future English-Sanskrit retrieval research, with train/validation/test splits and negative sampling.
- Fine-tuned dense models like ColBERT and Contriever on translated documents substantially beat zero-shot alternatives (e.g., 41.70% vs 30.48% NDCG@10), so task-specific fine-tuning still adds value in the DT setting.
- Query translation into Sanskrit is currently a weak route; improving Sanskrit-side translation or monolingual Sanskrit retrieval would be a direct lever on QT performance.
- The gap between DT and DR suggests that current multilingual embedding models do not align Sanskrit well with English for retrieval purposes.
Reading between the lines
- If the single-relevance annotation is relaxed to allow multiple relevant chapters, the reported absolute NDCG numbers would drop, but the relative ordering (DT ≫ DR ≈ QT) may be robust because lexical overlap in translated English is a stronger signal than cross-lingual semantic similarity for this corpus.
- A testable extension is to run the same three-armed comparison with human-translated documents instead of Google Translate; if the DT advantage shrinks, part of the result is an artifact of translation quality rather than a property of the retrieval task.
- The result may generalize to other low-resource classical languages where parallel translations exist: the paper's recipe is essentially 'use the existing translation to move the retrieval problem into a high-resource language,' which is cheaper than building new embedding spaces.
- The paper notes translation-error cascading as a limitation but does not quantify how retrieval performance changes as translation quality varies, so a sensitivity analysis on translation quality would be a natural next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Anveshana, a new English-to-Sanskrit cross-lingual information retrieval (CLIR) benchmark built from 3,400 manually authored English query–Sanskrit chapter pairs over 334 Srimadbhagavatam chapters. The authors compare three retrieval frameworks—direct retrieval (DR), document translation (DT), and query translation (QT)—using BM25, dense retrievers (Contriever, ColBERT, XLM-RoBERTa, multilingual-E5, mDPR), GPT-2, and REPLUG LSR, with both fine-tuned and zero-shot configurations. The central claim, stated in the abstract and repeated in Sections 6–8, is that DT methods outperform DR and QT, with DT-BM25 achieving the best results (e.g., NDCG@10 = 62.46% in Table 6 versus 10.74% for the best DR model and 6.86% for the best QT model). The paper also reports a zero-shot monolingual-Contriever English model as strongest among zero-shot systems.
Significance. If the benchmark is valid and the evaluation protocol is sound, Anveshana would be a valuable first public resource for English-to-Sanskrit CLIR, filling an acknowledged gap for a low-resource classical language. The paper deserves credit for releasing the dataset publicly, for spanning three clearly separated retrieval paradigms, and for including multiple modern retrieval models with fine-tuning. The empirical ranking is internally consistent across Tables 3–6 in the sense that DT-BM25 is always best, which is a useful sanity check. However, the significance of the central comparative claim is currently conditional on two unresolved issues: the gold relevance labels are single-source and were authored from English translations of the same chapters that DT translates back into English, and the zero-shot evaluation protocol is ambiguous about what document language is being retrieved. The dataset itself is a contribution regardless, but the paper's headline finding—that DT is superior for this task—cannot be interpreted as evidence about cross-lingual retrieval ability until these issues are addressed.
major comments (5)
- [§3.1 and §3.4] The gold relevance labels are not independently validated. Section 3.1 states that queries were manually crafted by examining English translations of each document and then pairing each query with that single document as relevant. Section 3.4 then presents a sample query whose paired Sanskrit document is admitted to be only 'indirectly related' to the query. Because NDCG, MAP, Recall, and Precision are computed against these binary labels, a high score such as DT-BM25's 62.46% NDCG@10 (Table 6) may indicate that the system recovers the source chapter rather than that it finds content relevant to the query's information need. Please provide independent relevance judgments for at least the 340 test queries (or a representative sample), report inter-annotator agreement, and re-evaluate the main comparisons using those judgments.
- [§4.2, §3.1, Tables 3–6] The DT pipeline translates Sanskrit documents into English via Google Translate, while the queries themselves were written from English translations of those same chapters. This creates a lexical-overlap advantage for DT-BM25 that is not shared by DR or QT: the query and the translated document share vocabulary from a common English rendering of the source. The reported 62.46% NDCG@10 may therefore reflect translation-memory effects rather than cross-lingual retrieval skill. The manuscript should include a control that removes this confound, for example by evaluating DT-BM25 with queries that are paraphrases independent of the Vedabase English translation, or by measuring and reporting the query–translated-document lexical overlap and comparing against a simple 'translate document then BM25' baseline on a held-out set of queries not derived from those translations.
- [§4.3 and Tables 3–6 (Zero-shot rows)] The zero-shot evaluation is ambiguous with respect to document language. Section 4.3 says Contriever was used for 'zero-shot retrieval of English documents,' and Section 6 reports the best zero-shot model as 'monolingual-Contriever eng.' But the task is English-query-to-Sanskrit-document retrieval. If these zero-shot models are evaluated on English translations of the Sanskrit documents, then the 'Zero-shot' rows in Tables 3–6 belong to the DT family and cannot be compared as a separate framework in support of the abstract's DT-versus-DR conclusion. Please state explicitly for every row in Tables 3–6 what document representation was indexed (original Sanskrit, English translation, or both), and, if English translations are used, re-frame those results accordingly.
- [§6, QT results paragraph] The text in the QT paragraph says BM25 'demonstrated a uniform performance across all metrics at approximately 2.95%,' but the tables report NDCG@3 = 5.18%, NDCG@5 = 5.94%, and NDCG@10 = 6.86% (Tables 4–6). This is a direct numerical contradiction between the results narrative and the reported data. Please correct the text to match the tables and ensure all QT numbers are reported consistently across the section.
- [Tables 3–6] No variance estimates, standard errors, or significance tests are reported for any of the retrieval results. Several comparisons that are discussed as meaningful differences (e.g., ColBERT-fine-tunedDOT NDCG@10 = 40.78% versus contriever-fine-tunedDOT = 41.70%; intfloat/multilingual-e5-base DR NDCG@10 = 10.74% versus mDPR-BM35-HN1 = 4.74%) could easily be within query-level noise. Please report per-query score distributions, bootstrap confidence intervals, or a paired significance test (e.g., paired bootstrap or Wilcoxon) for the key comparisons that support the central claim.
minor comments (6)
- [§3.2] The description of the regex substitution for poetic markers (e.g., '——1.1.3——') is vague; please provide the actual preprocessing rule and give an example of a document after preprocessing.
- [§4.1] The sentence describing the scaling is garbled: 'scaled by 1√length of embeddings' should be written as 'scaled by the inverse square root of the embedding length,' and the surrounding math notation should be cleaned up.
- [§4.2 and Tables 3–6] The naming of the Contriever variants is confusing: the text refers to 'mjwong/contriever-mnli fine-tuned (CONCAT and DOT)' and later to 'contriever - fine-tunedDOT' and 'contriever - fine-tuned' without defining the exact model checkpoints and pooling used; please standardize the model names and specify the configuration for each table row.
- [§4.3 and Table 3] The model 'mDPR-BM35-HN1' appears in Table 3 without being defined in the text; please explain what BM35 and HN1 refer to (e.g., BM25 hard negatives, number of negatives) and how this model differs from the mDPR model described in §4.3.
- [Table 3 caption] The caption states 'For k = 1, NDCG, MAP, Precision, Recall will yield same value,' which holds only because each query has exactly one gold document; please state this assumption explicitly in the caption.
- [§3.4 and Table 2] Table 2 is described as showing a sample query and its corresponding Sanskrit document, but the excerpt in the text does not display the actual Sanskrit text; please include the full paired document snippet so readers can evaluate the relevance relationship for themselves.
Circularity Check
No significant circularity: this is an empirical benchmark study with held-out evaluation, and the main comparison is not forced by construction.
full rationale
Anveshana is a newly constructed CLIR benchmark, and the paper's central claim that Document Translation (DT) methods outperform Direct Retrieve (DR) and Query Translation (QT) is an empirical result obtained on a held-out test split (Section 3.2: 90:10 train/test split; Section 6 reports results on the test dataset). The dataset construction is described as manually crafting queries from English translations of Srimadbhagavatam chapters (Section 3.1), which raises a legitimate validity concern about whether each query has exactly one relevant chapter; the paper itself admits in Section 3.4 that one paired Sanskrit document is only 'indirectly related' to its query. However, this is a question of label quality and benchmark validity, not circularity: the ranking of DT BM25 over DR and QT models is not logically entailed by the way the queries were written, nor is any parameter fitted to the test labels and then renamed a prediction. The fine-tuned models are trained on a separate train split with negative sampling and evaluated on unseen test queries, so there is no fitted-input-called-prediction pattern. The paper also contains several self-citations (Krishna et al. 2019, 2021; Sandhan et al. 2022a,b; Sujoy et al. 2023), but none is load-bearing for the retrieval comparison; they are background references for prior Sanskrit NLP work. No equation or definition in the paper reduces the reported DT advantage to the benchmark construction, and no external 'uniqueness theorem' is invoked to force a choice. The appropriate verdict is therefore no significant circularity.
Assumptions & free parameters
free parameters (4)
- train/test split ratio =
90:10
- negative-to-positive sampling ratio =
2:1
- REPLUG retrieval depth =
top 20
- evaluation cutoffs =
k=1,3,5,10
assumptions (4)
- domain assumption English-to-Sanskrit chapter correspondence is faithful
- domain assumption Each query has exactly one relevant document
- domain assumption Google Translate provides usable translations for Sanskrit
- standard math Standard IR metrics are appropriate
Cite this review
Pith. "Pith review of Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents." pith.science (2026). https://pith.science/paper/Y56QGN7T
@misc{pith2026250519494,
author = {Pith},
title = {Pith review of: Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y56QGN7T}},
note = {Machine review of arXiv:2505.19494}
}
read the original abstract
The study presents a comprehensive benchmark for retrieving Sanskrit documents using English queries, focusing on the chapters of the Srimadbhagavatam. It employs a tripartite approach: Direct Retrieval (DR), Translation-based Retrieval (DT), and Query Translation (QT), utilizing shared embedding spaces and advanced translation methods to enhance retrieval systems in a RAG framework. The study fine-tunes state-of-the-art models for Sanskrit's linguistic nuances, evaluating models such as BM25, REPLUG, mDPR, ColBERT, Contriever, and GPT-2. It adapts summarization techniques for Sanskrit documents to improve QA processing. Evaluation shows DT methods outperform DR and QT in handling the cross-lingual challenges of ancient texts, improving accessibility and understanding. A dataset of 3,400 English-Sanskrit query-document pairs underpins the study, aiming to preserve Sanskrit scriptures and share their philosophical importance widely. Our dataset is publicly available at https://huggingface.co/datasets/manojbalaji1/anveshana
Figures
Reference graph
Works this paper leans on
-
[1]
Monika Arora, Uma Kanjilal, and Dinesh Varshney. 2016. Evaluation of information retrieval: precision and recall. International Journal of Indian Culture and Business Management , 12(2):224--236
work page 2016
-
[2]
Tianchi Bi, Liang Yao, Baosong Yang, Haibo Zhang, Weihua Luo, and Boxing Chen. 2020. Constraint translation candidates: A bridge between neural query translation and cross-lingual information retrieval. arXiv preprint arXiv:2010.13658
work page Pith review arXiv 2020
-
[3]
Elizabeth Boschee, Joel Barry, Jayadev Billa, Marjorie Freedman, Thamme Gowda, Constantine Lignos, Chester Palen-Michel, Michael Pust, Banriskhem Kayang Khonglah, Srikanth Madikeri, et al. 2019. Saral: A low-resource cross-lingual domain-focused information retrieval system for effective rapid document triage. In Proceedings of the 57th Annual Meeting of ...
work page 2019
-
[4]
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions
work page 2017
-
[5]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116
arXiv 2019
-
[6]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar \'e , Maria Lomeli, Lucas Hosseini, and Herv \'e J \'e gou. 2024. The faiss library. arXiv preprint arXiv:2401.08281
arXiv 2024
-
[7]
Zhiqi Huang, Hamed Bonab, Sheikh Muhammad Sarwar, Razieh Rahimi, and James Allan. 2021. Mixed attention transformer for leveraging word-level knowledge to neural cross-lingual information retrieval. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 760--770
work page 2021
-
[8]
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118
arXiv 2021
Show all 30 references
-
[9]
Zhuolin Jiang, Amro El-Jaroudi, William Hartmann, Damianos Karakos, and Lingjun Zhao. 2020. Cross-lingual information retrieval with bert. arXiv preprint arXiv:2004.13005
2020 arXiv
-
[10]
Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , pages 39--48
2020
-
[11]
Amrith Krishna, Vishnu Dutt Sharma, Bishal Santra, Aishik Chakraborty, Pavankumar Satuluri, and Pawan Goyal. 2019. Poetry to prose conversion in sanskrit as a linearisation task: A case for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association fo...
2019
-
[12]
Amrith Krishna, Bishal Santra, Ashim Gupta, Pavankumar Satuluri, and Pawan Goyal. 2021. A graph-based framework for structured prediction tasks in sanskrit. Computational Linguistics , 46(4):785--845
2021
-
[13]
Micha \"e l Meyer. 2019. On sanskrit and information retrieval. In 6th International Sanskrit Computational Linguistics Symposium , pages 83--96. Association for Computational Linguistics
2019
-
[14]
Odunayo Ogundepo, Xinyu Zhang, Shuo Sun, Kevin Duh, and Jimmy Lin. 2022. Africlirmatrix: Enabling cross-lingual information retrieval for african languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 8721--8728
2022
-
[15]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9
2019
-
[16]
Jerome Revaud, Jon Almaz \'a n, Rafael S Rezende, and Cesar Roberto de Souza. 2019. Learning with average precision: Training image retrieval with a listwise loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5107--5116
2019
-
[17]
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389
2009
-
[18]
Siba Sankar Sahu and Sukomal Pal. 2023. Building a text retrieval system for the sanskrit language: Exploring indexing, stemming, and searching issues. Computer Speech & Language , 81:101518
2023
-
[19]
Shadi Saleh and Pavel Pecina. 2020. Document translation vs. query translation for cross-lingual information retrieval in the medical domain. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6849--6860
2020
-
[20]
Jivnesh Sandhan, Ashish Gupta, Hrishikesh Terdalkar, Tushar Sandhan, Suvendu Samanta, Laxmidhar Behera, and Pawan Goyal. 2022a. A novel multi-task learning approach for context-sensitive compound type identification in sanskrit. arXiv preprint arXiv:2208.10310
-
[21]
Jivnesh Sandhan, Rathin Singha, Narein Rao, Suvendu Samanta, Laxmidhar Behera, and Pawan Goyal. 2022b. Translist: A transformer-based linguistically informed sanskrit tokenizer. arXiv preprint arXiv:2210.11753
-
[22]
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652
2023 arXiv
-
[23]
Sarkar Sujoy, Amrith Krishna, and Pawan Goyal. 2023. Pre-annotation based approach for development of a sanskrit named entity recognition dataset. In Proceedings of the Computational Sanskrit & Digital Humanities: Selected papers presented at the 18th World Sanskrit Conference...
2023
-
[24]
Shuo Sun and Kevin Duh. 2020. Clirmatrix: A massively large collection of bilingual and multilingual datasets for cross-lingual information retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4160--4170
2020
-
[25]
Shuo Sun, Suzanna Sia, and Kevin Duh. 2020. Clireval: Evaluating machine translation as a cross-lingual information retrieval task. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations , pages 134--141
2020
-
[26]
Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013. A theoretical analysis of ndcg type ranking measures. In Conference on learning theory , pages 25--54. PMLR
2013
-
[27]
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672
2024 arXiv
-
[28]
Mahsa Yarmohammadi, Xutai Ma, Sorami Hisamoto, Muhammad Rahman, Yiming Wang, Hainan Xu, Daniel Povey, Philipp Koehn, and Kevin Duh. 2019. Robust document representations for cross-lingual information retrieval in low-resource settings. In Proceedings of Machine Translation Sum...
2019
-
[29]
Fuwei Zhang, Zhao Zhang, Xiang Ao, Dehong Gao, Fuzhen Zhuang, Yi Wei, and Qing He. 2022. Mind the gap: Cross-lingual information retrieval with hierarchical knowledge enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 4345--4353
2022
-
[30]
Lingjun Zhao, Rabih Zbib, Zhuolin Jiang, Damianos Karakos, and Zhongqiang Huang. 2019. Weakly supervised attentional model for low resource ad-hoc cross-lingual information retrieval. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2...
2019
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.