REVIEW 5 major objections 5 minor 2 cited by
Building Russian Benchmark for Evaluation of Information Retrieval Models
T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper introduces RusBEIR, a zero-shot information-retrieval benchmark for Russian built from 17 datasets, and shows that dense neural models lead on most tasks while BM25 still wins on full-document retrieval.
desk verdict Useful first BEIR-style Russian retrieval benchmark with new datasets, but the abstract's long-document claim about BGE-M3 is undercut by the paper's own 2048-token cap and needs revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the RusBEIR dataset collection itself, released in the format of the standard English zero-shot benchmark BEIR, plus the new wikifacts triplet: article, paragraph, and sentence versions of the same 540 Wikipedia-based facts. These variants let the same query be tested at three document lengths, isolating the input-size limitation of neural encoders. For the lexical baseline, the preprocessing pipeline of lowercasing, punctuation removal, tokenization, PyMorphy3 lemmatization, and stop-word removal is the mechanism that makes BM25 competitive in Russian.
What would settle it
Ask native Russian speakers to re-judge a sample of translated query-document pairs from rus-NFCorpus, rus-ArguAna, rus-SciFact, and rus-SCIDOCS against the original English labels; if many originally relevant pairs are no longer relevant after translation, the reported scores on those datasets need to be recomputed with validated translations.
Extended reading notes
Core claim
The central discovery is that a unified zero-shot benchmark for Russian is feasible from existing and newly built resources, and that its first large-scale run produces a clear ranking: mE5-large and BGE-M3 as dense encoders, combined with a BGE reranker, achieve the best NDCG@10 on most datasets, while BM25 remains the best single model on datasets dominated by long documents. The new wikifacts datasets, built from the same 540 facts at sentence, paragraph, and article granularity, isolate document-length effects without changing the queries. The paper argues that preprocessing, especially lemmatization, materially affects lexical-model performance in Russian, which has rich morphology.
Load-bearing premise
The four machine-translated datasets are assumed to preserve the original English relevance judgments; if translation changes which documents actually answer a query, roughly a quarter of the benchmark's scores would not measure true retrieval quality.
Editorial extensions
If this is right
- Russian-language IR research gains a common open benchmark spanning fact-checking, question answering, citation prediction, and news retrieval.
- A standard recipe for Russian retrieval will be dense first-stage retrieval with a cross-encoder reranker, since the reranked BGE-M3 and mE5-large pipelines lead the benchmark.
- For full-document retrieval in Russian, lemmatized BM25 is a strong and hard-to-beat baseline, especially against encoders whose input windows are too small.
- The wikifacts triplet gives other researchers a controlled testbed for studying how document length affects retrieval performance.
- The benchmark's results align with translated-BEIR findings for other languages, supporting the practice of building multilingual zero-shot resources by translating English datasets.
Reading between the lines
- A direct validation step—back-translating the four machine-translated datasets or having native speakers re-judge relevance—would test whether translation noise changes the reported rankings; the paper does not include such a check.
- The same three-granularity annotation design could be applied to other languages, creating comparable long-document test sets for cross-lingual IR studies.
- RusBEIR's data could be used for fine-tuning rather than only zero-shot evaluation, potentially boosting performance on Russian IR downstream tasks.
- The explicit link between document length and neural failure suggests that adaptive truncation or long-context encoders might close the gap BM25 currently enjoys.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RusBEIR, a BEIR-inspired benchmark for zero-shot information retrieval in Russian, consisting of 17 datasets. The datasets come from four sources: four BEIR datasets machine-translated into Russian, Russian portions of existing multilingual datasets (mMARCO, MIRACL, XQuAD, TyDi QA), existing Russian datasets (RuBQ, Ria-News, SberQuAD, ruSciBench, ru-facts), and three newly created Wikipedia-based datasets (wikifacts-sents, wikifacts-para, wikifacts-articles) built from the same 540 annotated facts at different document granularities. The authors evaluate BM25 with a custom lemmatization-based preprocessing pipeline, several dense retrievers (LaBSE, mE5 variants, BGE-M3, USER-BGE-M3, RoSBERTa), and reranking combinations, reporting NDCG@10, MAP@10, and Recall@10. The central claims are that BM25 remains a strong baseline for full-document retrieval, that neural models outperform BM25 on most datasets, and that neural models such as mE5-large and BGE-M3 face difficulties with long-document retrieval due to input size constraints.
Significance. If the benchmark is valid, RusBEIR would be a valuable community resource: it addresses a real gap in Russian IR evaluation, is BEIR-format compatible, is publicly released, includes a controlled long-document setting through the three wikifacts variants, and compares a reasonable range of lexical, dense, and reranking models. The emphasis on morphological preprocessing for BM25 is also a useful contribution. However, the significance is conditional on several validation steps that the manuscript does not report: translation quality for the four translated datasets, inter-annotator agreement for the new wikifacts annotations, statistical reliability of the model rankings, and disclosure of whether the Russian-trained USER-BGE-M3 model was trained on any RusBEIR test datasets. The paper's headline claim about long-document limitations is weakened by the fact that BGE models were run at a 2048-token cap despite having an 8192-token capacity.
major comments (5)
- [§4.2, §5, Table 3] The abstract and conclusion claim that neural models such as mE5-large and BGE-M3 'face challenges with long-document retrieval due to input size constraints,' but this claim is not supported for BGE-M3. Section 4.2 states that BGE models were run with max-length 2048, while Table 2 lists BGE-M3's maximum input as 8192 tokens. On wikifacts-articles, whose average document length is 2535.9 tokens, this evaluation choice truncates documents and depresses BGE-M3's NDCG@10 (74.50 vs. BM25's 84.28). Section 5 itself concedes this: the BGE results were obtained with maximum input length 2048, although the models can process up to 8192 tokens. The finding that BM25 is the best single model on long-document datasets may change if BGE-M3 is evaluated with its full context window. Additionally, the sentence in Section 5 that BM25 outperforms BGE-M3 by 13 percentage points and mE5-large by 27 percentage points does not match Table 3, where the differences are 9.78 and 18.19 percentage points, respectively. The paper should either run BGE-M3 at 8192 tokens or restrict the input-size-constraint claim to mE5-large.
- [§3.1] The four translated datasets (rus-NFCorpus, rus-ArguAna, rus-SciFact, rus-SCIDOCS) are translated with Google Translate without any reported quality check. Prior analogues take validation seriously: Dutch BEIR samples and checks translations with a native speaker, and Hindi BEIR uses back-translation with Chrf++. The paper cites prior work to justify choosing Google Translate, but that does not establish that the Russian translations preserve query-document relevance. If translations contain errors that change entities, negations, or argumentative structure, the NDCG@10 scores on these four datasets do not measure Russian retrieval quality. The authors should add a human or automatic translation-quality assessment, or at least report error rates per dataset.
- [§4.2] The zero-shot evaluation claim is at risk for USER-BGE-M3, described as 'trained on the Russian datasets.' The paper does not list these training datasets or check them for overlap with RusBEIR test sets. RusBEIR includes RuBQ and Ria-News from ruMTEB, as well as SberQuAD, ruSciBench, and ru-facts, all of which are plausible training sources for a Russian embedding model. If any of these overlap with USER-BGE-M3's training data, its high scores on those datasets reflect in-distribution performance rather than zero-shot generalization. The authors should disclose the full training-data list for USER-BGE-M3 and either exclude overlapping datasets from the zero-shot interpretation or report results separately.
- [§3.4] The new wikifacts datasets are based on student annotations with relevance scores 0, 1, and 2, but the paper reports no inter-annotator agreement, no number of annotators per fact, and no adjudication procedure. Because these datasets are a primary novel contribution and are used to support the long-document retrieval conclusions, the reliability of the relevance labels needs to be established. Report an agreement measure such as Cohen's or quadratic weighted kappa on a held-out annotation subset, or otherwise justify why the labels can be treated as ground truth.
- [§5, Tables 3-5] The paper reports single-point NDCG@10, MAP@10, and Recall@10 values without any error bars or significance tests. Several comparative claims rest on small differences: for example, mE5-large+BGE (65.71) vs. BGE-M3+BGE (65.85) in Table 3, and many near-saturated scores on rus-XQuAD and ru-facts. Without paired bootstrap or per-query variance estimates, the stated ranking of models on individual datasets is not statistically supported. Add significance testing or report per-query confidence intervals for at least the headline comparisons.
minor comments (5)
- [§3.1, footnote rendering] The text contains numerous inline artifacts such as '/one.sup', '/two.sup', and similar footnote markers, which should be rendered as proper footnotes or superscripts.
- [Tables 4 and 5] The reranker column header 'BGE+BGE' in the additional metrics tables is inconsistent with 'BGE-M3+BGE' used in Table 3; use identical model names throughout.
- [§1 and §6] The introduction motivates an investigation of the efficiency trade-off between lexical and dense models, but no efficiency measurements are reported, and the conclusion defers this to future work. Either add a small runtime or memory comparison or remove this promise from the introduction.
- [Abstract and Introduction] The GitHub and HuggingFace links appear only in footnotes; they should be prominently stated in the abstract or introduction so readers can immediately access the released resources.
- [§2] There is a typo, 'signifcantly', in the discussion of Hindi-BEIR results. The manuscript would also benefit from a general proofreading pass for similar minor OCR-style errors.
Circularity Check
No significant circularity: RusBEIR's evaluation results are measured, not derived from their inputs.
full rationale
RusBEIR is an empirical benchmark paper rather than a formal derivation, so the usual circularity patterns do not apply. The central results—NDCG@10/MAP@10/Recall@10 across 17 Russian datasets and the BM25-versus-neural comparison—are direct measurements from public pretrained checkpoints on fixed query-corpus-qrels triples; no parameter is fitted to RusBEIR data and then renamed as a prediction. The translated and newly created datasets are collected from external sources or human annotation, not defined in terms of the models' outputs. Self-citations such as ruMTEB (for RuBQ and Ria-News) and ruRoBERTa (for RoSBERTa) supply datasets and model backbones, but the evaluation is externally checkable and does not rely on an unverified uniqueness claim. The paper's own Section 5 acknowledges that BGE models were evaluated at 2048 tokens although their stated maximum is 8192, so the abstract's input-size-constraint explanation for BGE-M3 is under-supported; that is a correctness/consistency limitation, not circularity. Similarly, any potential training-data overlap for USER-BGE-M3 would be a contamination risk, but the paper provides no evidence of overlap, so it cannot be scored as a demonstrated circular step. Overall, the benchmark construction and evaluation are self-contained empirical contributions.
Assumptions & free parameters
assumptions (5)
- domain assumption Google Translate preserves the relevance relation between queries and documents in the four translated BEIR datasets.
- domain assumption Relevance labels in the original English BEIR datasets remain valid after translation.
- domain assumption The student annotations for the wikifacts datasets are a reliable gold standard.
- domain assumption BM25 with the stated preprocessing (lowercasing, punctuation removal, lemmatization via PyMorphy3, stop-word removal) is a representative lexical baseline.
- domain assumption The selected maximum input length of 2048 tokens for BGE models is a fair setting for comparing long-document retrieval.
Cite this review
Pith. "Pith review of Building Russian Benchmark for Evaluation of Information Retrieval Models." pith.science (2026). https://pith.science/paper/USZW23UF
@misc{pith2026250412879,
author = {Pith},
title = {Pith review of: Building Russian Benchmark for Evaluation of Information Retrieval Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/USZW23UF}},
note = {Machine review of arXiv:2504.12879}
}
read the original abstract
We introduce RusBEIR, a comprehensive benchmark designed for zero-shot evaluation of information retrieval (IR) models in the Russian language. Comprising 17 datasets from various domains, it integrates adapted, translated, and newly created datasets, enabling systematic comparison of lexical and neural models. Our study highlights the importance of preprocessing for lexical models in morphologically rich languages and confirms BM25 as a strong baseline for full-document retrieval. Neural models, such as mE5-large and BGE-M3, demonstrate superior performance on most datasets, but face challenges with long-document retrieval due to input size constraints. RusBEIR offers a unified, open-source framework that promotes research in Russian-language information retrieval.
Forward citations
Cited by 2 Pith papers
-
Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning
State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.
-
LLMs are Also Effective Embedding Models: An In-depth Overview
A structured survey of using decoder-only LLMs as text embedding models, covering prompting, fine-tuning, data construction, benchmarks, and open problems.
Reference graph
Works this paper leans on
-
[1]
Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi
Arkadeep Acharya et al. “Hindi-BEIR: A Large Scale Retri eval Benchmark in Hindi”. In: arXiv preprint arXiv:2408.09437 (2024)
work page Pith review arXiv 2024
-
[2]
On th e Cross-lingual Transferability of Monolingual Representations
Mikel Artetxe, Sebastian Ruder and Dani Yogatama. “On th e Cross-lingual Transferability of Monolingual Representations”. In: Proceedings of the 58th Annual Meeting of the Association fo r Computational Linguistics. Association for Computational Linguistics. 2020
work page 2020
-
[3]
Ms marco: A human generated machine re ading comprehension dataset
Payal Bajaj et al. “Ms marco: A human generated machine re ading comprehension dataset”. In: arXiv preprint arXiv:1611.09268 (2016)
arXiv 2016
-
[4]
BEIR-N L: Zero-shot Information Retrieval Benchmark for the Dutch Language
Nikolay Banar, Ehsan Lotfi and Walter Daelemans. “BEIR-N L: Zero-shot Information Retrieval Benchmark for the Dutch Language”. In: arXiv preprint arXiv:2412.08329 (2024)
arXiv 2024
-
[5]
mmarco: A multilingual version of the ms marco passage ranking dataset
Luiz Bonifacio et al. “mmarco: A multilingual version of the ms marco passage ranking dataset”. In: arXiv preprint arXiv:2108.13897 (2021)
arXiv 2021
-
[6]
Jianlv Chen et al. “Bge m3-embedding: Multi-lingual, mu lti-functionality, multi-granularity text embeddings through self-knowledge distillation”. In: arXiv preprint arXiv:2402.03216 (2024)
arXiv 2024
-
[7]
Jonathan H Clark et al. “Tydi qa: A benchmark for informat ion-seeking question answering in ty pologically di verse languages”. In: Transactions of the Association for Computational Linguis tics 8 (2020), pp. 454–470
work page 2020
-
[8]
PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods
Slawomir Dadas, Micha l Pere lkiewicz and Rafa l Po ´swiata. “PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods”. In:Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resour ces and Evaluation (LREC-COLING 2024). 2024, pp. 12761–12774
work page 2024
Show all 29 references
-
[9]
BERT: Pre-training of Deep Bidirect ional Transformers for Language Under- standing
Jacob Devlin et al. “BERT: Pre-training of Deep Bidirect ional Transformers for Language Under- standing”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sho...
2019 doi
-
[10]
Russian Information Retrieval Ev aluation Seminar
Boris V Dobrov et al. “Russian Information Retrieval Ev aluation Seminar.” In: LREC. 2004
2004
-
[11]
Sberquad–russian reading comprehe nsion dataset: Description and analysis
Pavel Efimov et al. “Sberquad–russian reading comprehe nsion dataset: Description and analysis”. In: Experimental IR Meets Multilinguality, Multimodality, an d Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessalonik i, Greece, September 22–25,...
2020
-
[12]
Language-agnostic BERT Senten ce Embedding
Fangxiaoyu Feng et al. “Language-agnostic BERT Senten ce Embedding”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Li nguistics (Volume 1: Long Papers) . 2022, pp. 878–891
2022
-
[13]
Proceedings of the 15th Annual Meeting of the Forum for Infor mation Retrieval Evaluation
Debasis Ganguly et al. Proceedings of the 15th Annual Meeting of the Forum for Infor mation Retrieval Evaluation. ACM, 2023
2023
-
[14]
Self-attentive model for headline genera- tion
Daniil Gavrilov, Pavel Kalaidin and Valentin Malykh. “ Self-attentive model for headline genera- tion”. In: Advances in Information Retrieval: 41st European Conferen ce on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Pa rt II 41. Springer. 2019, pp. 87–93
2019
-
[15]
Natural language processing: python and NLTK
Nitin Hardeniya et al. Natural language processing: python and NLTK. Packt Publishing Ltd, 2016
2016
-
[16]
PolEval 2022/23 challenge tasks and results
Lukasz Kobyli ´nski et al. “PolEval 2022/23 challenge tasks and results”. I n: 2023 18th Conference on Computer Science and Intelligence Systems (FedCSIS). IEEE. 2023, pp. 1243–1250
2022
-
[17]
Morphological analyzer and generat or for Russian and Ukrainian languages
Mikhail Korobov. “Morphological analyzer and generat or for Russian and Ukrainian languages”. In: Analysis of Images, Social Networks and Texts: 4th Internat ional Conference, AIST 2015, Yekaterinburg, Russia, April 9–11, 2015, Revised Selected Papers 4. Springer. 2015, pp. 320–332
2015
-
[18]
Fact-checking benchmark for the Russian Large Language Models
Anastasia Kozlova, Denis Shevelev and Alena Fenogenov a. “Fact-checking benchmark for the Russian Large Language Models”. In: Proceedings of the International Conference “Dialogue . Vol. 2023. 2023
2023
-
[19]
MTEB: Massive Text Embedding Benchmark
Niklas Muennighoff et al. “MTEB: Massive Text Embedding Benchmark”. In: Proceedings of the 17th Conference of the European Chapter of the Association f or Computational Linguistics. 2023, pp. 2014–2037
2023
-
[20]
chrF: character n-gram F-score for automatic MT evaluat ion
Maja Popovi ´c. “chrF: character n-gram F-score for automatic MT evaluat ion”. In: Proceedings of the tenth workshop on statistical machine translation. 2015, pp. 392–395
2015
-
[21]
Squad: 100,000+ questions for machine co mprehension of text
P Rajpurkar. “Squad: 100,000+ questions for machine co mprehension of text”. In: arXiv preprint arXiv:1606.05250 (2016)
2016 arXiv
-
[22]
RuBQ 2.0: an innovated Russian questi on answering dataset
Ivan Rybin et al. “RuBQ 2.0: an innovated Russian questi on answering dataset”. In: The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings 18 . Springer. 2021, pp. 532–547
2021
-
[23]
The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design
Artem Snegirev et al. “The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design”. In: CoRR (2024)
2024
-
[24]
BEIR: A Heterogeneous Benchmark f or Zero-shot Evalu- ation of Information Retrieval Models
Nandan Thakur et al. “BEIR: A Heterogeneous Benchmark f or Zero-shot Evalu- ation of Information Retrieval Models”. In: Thirty-fifth Conference on Neural Inform- ation Processing Systems Datasets and Benchmarks Track (Ro und 2) . 2021. /u.pc/r.pc/l.pc: https://openreview.net/fo...
2021
-
[25]
Multilingual e5 text embeddings: A te chnical report
Liang Wang et al. “Multilingual e5 text embeddings: A te chnical report”. In: arXiv preprint arXiv:2402.05672 (2024)
2024 arXiv
-
[26]
Text embeddings by weakly-supervise d contrastive pre-training
Liang Wang et al. “Text embeddings by weakly-supervise d contrastive pre-training”. In: arXiv preprint arXiv:2212.03533 (2022)
2022 arXiv
-
[27]
BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Lan- guage
Konrad Wojtasik et al. “BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Lan- guage”. In: Proceedings of the 2024 Joint International Conference on C omputational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024, pp. 2149–2160
2024
-
[28]
Miracl: A multilingual retrieval da taset covering 18 diverse languages
Xinyu Zhang et al. “Miracl: A multilingual retrieval da taset covering 18 diverse languages”. In: Transactions of the Association for Computational Linguis tics 11 (2023), pp. 1114–1131
2023
-
[29]
A Family of Pretrained Transf ormer Language Models for Russian
Dmitry Zmitrovich et al. “A Family of Pretrained Transf ormer Language Models for Russian”. In: Proceedings of the 2024 Joint International Conference on C omputational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024, pp. 507–524. 7 Additional metrics 7...
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.