Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

Building Russian Benchmark for Evaluation of Information Retrieval Models

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read This paper introduces RusBEIR, a zero-shot information-retrieval benchmark for Russian built from 17 datasets, and shows that dense neural models lead on most tasks while BM25 still wins on full-document retrieval.

desk verdict Useful first BEIR-style Russian retrieval benchmark with new datasets, but the abstract's long-document claim about BGE-M3 is undercut by the paper's own 2048-token cap and needs revision. read the letter →

arxiv 2504.12879 v1 pith:USZW23UF submitted 2025-04-17 cs.IR cs.CL

classification cs.IRcs.CL
keywords informationretrievalbenchmarkRussianlanguagezero-shotevaluationBM25denseBEIRdatasets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to give Russian-language information retrieval a standard evaluation resource like the English BEIR benchmark, and to find out which retrieval models actually work in Russian. It assembles 17 datasets from translated English corpora, multilingual collections, existing Russian resources, and newly annotated Wikipedia facts. Across these datasets, multilingual dense encoders such as mE5-large and BGE-M3 outperform BM25 on average, but BM25 remains the strongest single model on four long-document datasets. The paper also demonstrates that lemmatization preprocessing substantially improves lexical models in Russian. If the benchmark is sound, it becomes the common yardstick for comparing retrieval models on Russian.

What carries the argument

The load-bearing object is the RusBEIR dataset collection itself, released in the format of the standard English zero-shot benchmark BEIR, plus the new wikifacts triplet: article, paragraph, and sentence versions of the same 540 Wikipedia-based facts. These variants let the same query be tested at three document lengths, isolating the input-size limitation of neural encoders. For the lexical baseline, the preprocessing pipeline of lowercasing, punctuation removal, tokenization, PyMorphy3 lemmatization, and stop-word removal is the mechanism that makes BM25 competitive in Russian.

What would settle it

Ask native Russian speakers to re-judge a sample of translated query-document pairs from rus-NFCorpus, rus-ArguAna, rus-SciFact, and rus-SCIDOCS against the original English labels; if many originally relevant pairs are no longer relevant after translation, the reported scores on those datasets need to be recomputed with validated translations.

Watch

Extended reading notes

Core claim

The central discovery is that a unified zero-shot benchmark for Russian is feasible from existing and newly built resources, and that its first large-scale run produces a clear ranking: mE5-large and BGE-M3 as dense encoders, combined with a BGE reranker, achieve the best NDCG@10 on most datasets, while BM25 remains the best single model on datasets dominated by long documents. The new wikifacts datasets, built from the same 540 facts at sentence, paragraph, and article granularity, isolate document-length effects without changing the queries. The paper argues that preprocessing, especially lemmatization, materially affects lexical-model performance in Russian, which has rich morphology.

Load-bearing premise

The four machine-translated datasets are assumed to preserve the original English relevance judgments; if translation changes which documents actually answer a query, roughly a quarter of the benchmark's scores would not measure true retrieval quality.

Editorial extensions

If this is right

  • Russian-language IR research gains a common open benchmark spanning fact-checking, question answering, citation prediction, and news retrieval.
  • A standard recipe for Russian retrieval will be dense first-stage retrieval with a cross-encoder reranker, since the reranked BGE-M3 and mE5-large pipelines lead the benchmark.
  • For full-document retrieval in Russian, lemmatized BM25 is a strong and hard-to-beat baseline, especially against encoders whose input windows are too small.
  • The wikifacts triplet gives other researchers a controlled testbed for studying how document length affects retrieval performance.
  • The benchmark's results align with translated-BEIR findings for other languages, supporting the practice of building multilingual zero-shot resources by translating English datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct validation step—back-translating the four machine-translated datasets or having native speakers re-judge relevance—would test whether translation noise changes the reported rankings; the paper does not include such a check.
  • The same three-granularity annotation design could be applied to other languages, creating comparable long-document test sets for cross-lingual IR studies.
  • RusBEIR's data could be used for fine-tuning rather than only zero-shot evaluation, potentially boosting performance on Russian IR downstream tasks.
  • The explicit link between document length and neural failure suggests that adaptive truncation or long-context encoders might close the gap BM25 currently enjoys.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces RusBEIR, a BEIR-inspired benchmark for zero-shot information retrieval in Russian, consisting of 17 datasets. The datasets come from four sources: four BEIR datasets machine-translated into Russian, Russian portions of existing multilingual datasets (mMARCO, MIRACL, XQuAD, TyDi QA), existing Russian datasets (RuBQ, Ria-News, SberQuAD, ruSciBench, ru-facts), and three newly created Wikipedia-based datasets (wikifacts-sents, wikifacts-para, wikifacts-articles) built from the same 540 annotated facts at different document granularities. The authors evaluate BM25 with a custom lemmatization-based preprocessing pipeline, several dense retrievers (LaBSE, mE5 variants, BGE-M3, USER-BGE-M3, RoSBERTa), and reranking combinations, reporting NDCG@10, MAP@10, and Recall@10. The central claims are that BM25 remains a strong baseline for full-document retrieval, that neural models outperform BM25 on most datasets, and that neural models such as mE5-large and BGE-M3 face difficulties with long-document retrieval due to input size constraints.

Significance. If the benchmark is valid, RusBEIR would be a valuable community resource: it addresses a real gap in Russian IR evaluation, is BEIR-format compatible, is publicly released, includes a controlled long-document setting through the three wikifacts variants, and compares a reasonable range of lexical, dense, and reranking models. The emphasis on morphological preprocessing for BM25 is also a useful contribution. However, the significance is conditional on several validation steps that the manuscript does not report: translation quality for the four translated datasets, inter-annotator agreement for the new wikifacts annotations, statistical reliability of the model rankings, and disclosure of whether the Russian-trained USER-BGE-M3 model was trained on any RusBEIR test datasets. The paper's headline claim about long-document limitations is weakened by the fact that BGE models were run at a 2048-token cap despite having an 8192-token capacity.

major comments (5)
  1. [§4.2, §5, Table 3] The abstract and conclusion claim that neural models such as mE5-large and BGE-M3 'face challenges with long-document retrieval due to input size constraints,' but this claim is not supported for BGE-M3. Section 4.2 states that BGE models were run with max-length 2048, while Table 2 lists BGE-M3's maximum input as 8192 tokens. On wikifacts-articles, whose average document length is 2535.9 tokens, this evaluation choice truncates documents and depresses BGE-M3's NDCG@10 (74.50 vs. BM25's 84.28). Section 5 itself concedes this: the BGE results were obtained with maximum input length 2048, although the models can process up to 8192 tokens. The finding that BM25 is the best single model on long-document datasets may change if BGE-M3 is evaluated with its full context window. Additionally, the sentence in Section 5 that BM25 outperforms BGE-M3 by 13 percentage points and mE5-large by 27 percentage points does not match Table 3, where the differences are 9.78 and 18.19 percentage points, respectively. The paper should either run BGE-M3 at 8192 tokens or restrict the input-size-constraint claim to mE5-large.
  2. [§3.1] The four translated datasets (rus-NFCorpus, rus-ArguAna, rus-SciFact, rus-SCIDOCS) are translated with Google Translate without any reported quality check. Prior analogues take validation seriously: Dutch BEIR samples and checks translations with a native speaker, and Hindi BEIR uses back-translation with Chrf++. The paper cites prior work to justify choosing Google Translate, but that does not establish that the Russian translations preserve query-document relevance. If translations contain errors that change entities, negations, or argumentative structure, the NDCG@10 scores on these four datasets do not measure Russian retrieval quality. The authors should add a human or automatic translation-quality assessment, or at least report error rates per dataset.
  3. [§4.2] The zero-shot evaluation claim is at risk for USER-BGE-M3, described as 'trained on the Russian datasets.' The paper does not list these training datasets or check them for overlap with RusBEIR test sets. RusBEIR includes RuBQ and Ria-News from ruMTEB, as well as SberQuAD, ruSciBench, and ru-facts, all of which are plausible training sources for a Russian embedding model. If any of these overlap with USER-BGE-M3's training data, its high scores on those datasets reflect in-distribution performance rather than zero-shot generalization. The authors should disclose the full training-data list for USER-BGE-M3 and either exclude overlapping datasets from the zero-shot interpretation or report results separately.
  4. [§3.4] The new wikifacts datasets are based on student annotations with relevance scores 0, 1, and 2, but the paper reports no inter-annotator agreement, no number of annotators per fact, and no adjudication procedure. Because these datasets are a primary novel contribution and are used to support the long-document retrieval conclusions, the reliability of the relevance labels needs to be established. Report an agreement measure such as Cohen's or quadratic weighted kappa on a held-out annotation subset, or otherwise justify why the labels can be treated as ground truth.
  5. [§5, Tables 3-5] The paper reports single-point NDCG@10, MAP@10, and Recall@10 values without any error bars or significance tests. Several comparative claims rest on small differences: for example, mE5-large+BGE (65.71) vs. BGE-M3+BGE (65.85) in Table 3, and many near-saturated scores on rus-XQuAD and ru-facts. Without paired bootstrap or per-query variance estimates, the stated ranking of models on individual datasets is not statistically supported. Add significance testing or report per-query confidence intervals for at least the headline comparisons.
minor comments (5)
  1. [§3.1, footnote rendering] The text contains numerous inline artifacts such as '/one.sup', '/two.sup', and similar footnote markers, which should be rendered as proper footnotes or superscripts.
  2. [Tables 4 and 5] The reranker column header 'BGE+BGE' in the additional metrics tables is inconsistent with 'BGE-M3+BGE' used in Table 3; use identical model names throughout.
  3. [§1 and §6] The introduction motivates an investigation of the efficiency trade-off between lexical and dense models, but no efficiency measurements are reported, and the conclusion defers this to future work. Either add a small runtime or memory comparison or remove this promise from the introduction.
  4. [Abstract and Introduction] The GitHub and HuggingFace links appear only in footnotes; they should be prominently stated in the abstract or introduction so readers can immediately access the released resources.
  5. [§2] There is a typo, 'signifcantly', in the discussion of Hindi-BEIR results. The manuscript would also benefit from a general proofreading pass for similar minor OCR-style errors.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RusBEIR's evaluation results are measured, not derived from their inputs.

full rationale

RusBEIR is an empirical benchmark paper rather than a formal derivation, so the usual circularity patterns do not apply. The central results—NDCG@10/MAP@10/Recall@10 across 17 Russian datasets and the BM25-versus-neural comparison—are direct measurements from public pretrained checkpoints on fixed query-corpus-qrels triples; no parameter is fitted to RusBEIR data and then renamed as a prediction. The translated and newly created datasets are collected from external sources or human annotation, not defined in terms of the models' outputs. Self-citations such as ruMTEB (for RuBQ and Ria-News) and ruRoBERTa (for RoSBERTa) supply datasets and model backbones, but the evaluation is externally checkable and does not rely on an unverified uniqueness claim. The paper's own Section 5 acknowledges that BGE models were evaluated at 2048 tokens although their stated maximum is 8192, so the abstract's input-size-constraint explanation for BGE-M3 is under-supported; that is a correctness/consistency limitation, not circularity. Similarly, any potential training-data overlap for USER-BGE-M3 would be a contamination risk, but the paper provides no evidence of overlap, so it cannot be scored as a demonstrated circular step. Overall, the benchmark construction and evaluation are self-contained empirical contributions.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The benchmark's validity rests on several unverified assumptions about translation quality and annotation reliability, but no free parameters are fit to data. The central claims are empirical, not derivational.

assumptions (5)
  • domain assumption Google Translate preserves the relevance relation between queries and documents in the four translated BEIR datasets.
    Section 3.1 selects Google Translate based on earlier studies but provides no quality check of Russian translations; if translations corrupt queries or documents, the evaluation on these datasets is unreliable.
  • domain assumption Relevance labels in the original English BEIR datasets remain valid after translation.
    Binary relevance judgments are transferred directly from English to Russian without validation that the translation does not alter the evidence relation (Section 3.1).
  • domain assumption The student annotations for the wikifacts datasets are a reliable gold standard.
    540 facts were annotated by university students with a 0/1/2 scale, but no inter-annotator agreement or adjudication is reported (Section 3.4).
  • domain assumption BM25 with the stated preprocessing (lowercasing, punctuation removal, lemmatization via PyMorphy3, stop-word removal) is a representative lexical baseline.
    The claim that BM25 is a strong baseline depends on these preprocessing choices; no ablation shows their individual contribution, and the language analyzer is disabled (Section 4.1).
  • domain assumption The selected maximum input length of 2048 tokens for BGE models is a fair setting for comparing long-document retrieval.
    Section 5 notes BGE models support 8192 tokens but are only evaluated at 2048; the resulting long-document difficulty is therefore a chosen constraint, not necessarily an inherent model limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Building Russian Benchmark for Evaluation of Information Retrieval Models." pith.science (2026). https://pith.science/paper/USZW23UF

@misc{pith2026250412879,
  author       = {Pith},
  title        = {Pith review of: Building Russian Benchmark for Evaluation of Information Retrieval Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USZW23UF}},
  note         = {Machine review of arXiv:2504.12879}
}
read the original abstract

We introduce RusBEIR, a comprehensive benchmark designed for zero-shot evaluation of information retrieval (IR) models in the Russian language. Comprising 17 datasets from various domains, it integrates adapted, translated, and newly created datasets, enabling systematic comparison of lexical and neural models. Our study highlights the importance of preprocessing for lexical models in morphologically rich languages and confirms BM25 as a strong baseline for full-document retrieval. Neural models, such as mE5-large and BGE-M3, demonstrate superior performance on most datasets, but face challenges with long-document retrieval due to input size constraints. RusBEIR offers a unified, open-source framework that promotes research in Russian-language information retrieval.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.

  2. LLMs are Also Effective Embedding Models: An In-depth Overview

    cs.CL 2024-12 conditional novelty 2.0 of 10

    A structured survey of using decoder-only LLMs as text embedding models, covering prompting, fine-tuning, data construction, benchmarks, and open problems.

Reference graph

Works this paper leans on

29 extracted references · 21 canonical work pages · cited by 2 Pith papers

  1. [1]

    Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi

    Arkadeep Acharya et al. “Hindi-BEIR: A Large Scale Retri eval Benchmark in Hindi”. In: arXiv preprint arXiv:2408.09437 (2024)

  2. [2]

    On th e Cross-lingual Transferability of Monolingual Representations

    Mikel Artetxe, Sebastian Ruder and Dani Yogatama. “On th e Cross-lingual Transferability of Monolingual Representations”. In: Proceedings of the 58th Annual Meeting of the Association fo r Computational Linguistics. Association for Computational Linguistics. 2020

  3. [3]

    Ms marco: A human generated machine re ading comprehension dataset

    Payal Bajaj et al. “Ms marco: A human generated machine re ading comprehension dataset”. In: arXiv preprint arXiv:1611.09268 (2016)

  4. [4]

    BEIR-N L: Zero-shot Information Retrieval Benchmark for the Dutch Language

    Nikolay Banar, Ehsan Lotfi and Walter Daelemans. “BEIR-N L: Zero-shot Information Retrieval Benchmark for the Dutch Language”. In: arXiv preprint arXiv:2412.08329 (2024)

  5. [5]

    mmarco: A multilingual version of the ms marco passage ranking dataset

    Luiz Bonifacio et al. “mmarco: A multilingual version of the ms marco passage ranking dataset”. In: arXiv preprint arXiv:2108.13897 (2021)

  6. [6]

    Bge m3-embedding: Multi-lingual, mu lti-functionality, multi-granularity text embeddings through self-knowledge distillation

    Jianlv Chen et al. “Bge m3-embedding: Multi-lingual, mu lti-functionality, multi-granularity text embeddings through self-knowledge distillation”. In: arXiv preprint arXiv:2402.03216 (2024)

  7. [7]

    Tydi qa: A benchmark for informat ion-seeking question answering in ty pologically di verse languages

    Jonathan H Clark et al. “Tydi qa: A benchmark for informat ion-seeking question answering in ty pologically di verse languages”. In: Transactions of the Association for Computational Linguis tics 8 (2020), pp. 454–470

  8. [8]

    PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods

    Slawomir Dadas, Micha l Pere lkiewicz and Rafa l Po ´swiata. “PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods”. In:Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resour ces and Evaluation (LREC-COLING 2024). 2024, pp. 12761–12774

Show all 29 references
  1. [9]

    BERT: Pre-training of Deep Bidirect ional Transformers for Language Under- standing

    Jacob Devlin et al. “BERT: Pre-training of Deep Bidirect ional Transformers for Language Under- standing”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sho...

  2. [10]

    Russian Information Retrieval Ev aluation Seminar

    Boris V Dobrov et al. “Russian Information Retrieval Ev aluation Seminar.” In: LREC. 2004

  3. [11]

    Sberquad–russian reading comprehe nsion dataset: Description and analysis

    Pavel Efimov et al. “Sberquad–russian reading comprehe nsion dataset: Description and analysis”. In: Experimental IR Meets Multilinguality, Multimodality, an d Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessalonik i, Greece, September 22–25,...

  4. [12]

    Language-agnostic BERT Senten ce Embedding

    Fangxiaoyu Feng et al. “Language-agnostic BERT Senten ce Embedding”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Li nguistics (Volume 1: Long Papers) . 2022, pp. 878–891

  5. [13]

    Proceedings of the 15th Annual Meeting of the Forum for Infor mation Retrieval Evaluation

    Debasis Ganguly et al. Proceedings of the 15th Annual Meeting of the Forum for Infor mation Retrieval Evaluation. ACM, 2023

  6. [14]

    Self-attentive model for headline genera- tion

    Daniil Gavrilov, Pavel Kalaidin and Valentin Malykh. “ Self-attentive model for headline genera- tion”. In: Advances in Information Retrieval: 41st European Conferen ce on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Pa rt II 41. Springer. 2019, pp. 87–93

  7. [15]

    Natural language processing: python and NLTK

    Nitin Hardeniya et al. Natural language processing: python and NLTK. Packt Publishing Ltd, 2016

  8. [16]

    PolEval 2022/23 challenge tasks and results

    Lukasz Kobyli ´nski et al. “PolEval 2022/23 challenge tasks and results”. I n: 2023 18th Conference on Computer Science and Intelligence Systems (FedCSIS). IEEE. 2023, pp. 1243–1250

  9. [17]

    Morphological analyzer and generat or for Russian and Ukrainian languages

    Mikhail Korobov. “Morphological analyzer and generat or for Russian and Ukrainian languages”. In: Analysis of Images, Social Networks and Texts: 4th Internat ional Conference, AIST 2015, Yekaterinburg, Russia, April 9–11, 2015, Revised Selected Papers 4. Springer. 2015, pp. 320–332

  10. [18]

    Fact-checking benchmark for the Russian Large Language Models

    Anastasia Kozlova, Denis Shevelev and Alena Fenogenov a. “Fact-checking benchmark for the Russian Large Language Models”. In: Proceedings of the International Conference “Dialogue . Vol. 2023. 2023

  11. [19]

    MTEB: Massive Text Embedding Benchmark

    Niklas Muennighoff et al. “MTEB: Massive Text Embedding Benchmark”. In: Proceedings of the 17th Conference of the European Chapter of the Association f or Computational Linguistics. 2023, pp. 2014–2037

  12. [20]

    chrF: character n-gram F-score for automatic MT evaluat ion

    Maja Popovi ´c. “chrF: character n-gram F-score for automatic MT evaluat ion”. In: Proceedings of the tenth workshop on statistical machine translation. 2015, pp. 392–395

  13. [21]

    Squad: 100,000+ questions for machine co mprehension of text

    P Rajpurkar. “Squad: 100,000+ questions for machine co mprehension of text”. In: arXiv preprint arXiv:1606.05250 (2016)

  14. [22]

    RuBQ 2.0: an innovated Russian questi on answering dataset

    Ivan Rybin et al. “RuBQ 2.0: an innovated Russian questi on answering dataset”. In: The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings 18 . Springer. 2021, pp. 532–547

  15. [23]

    The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design

    Artem Snegirev et al. “The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design”. In: CoRR (2024)

  16. [24]

    BEIR: A Heterogeneous Benchmark f or Zero-shot Evalu- ation of Information Retrieval Models

    Nandan Thakur et al. “BEIR: A Heterogeneous Benchmark f or Zero-shot Evalu- ation of Information Retrieval Models”. In: Thirty-fifth Conference on Neural Inform- ation Processing Systems Datasets and Benchmarks Track (Ro und 2) . 2021. /u.pc/r.pc/l.pc: https://openreview.net/fo...

  17. [25]

    Multilingual e5 text embeddings: A te chnical report

    Liang Wang et al. “Multilingual e5 text embeddings: A te chnical report”. In: arXiv preprint arXiv:2402.05672 (2024)

  18. [26]

    Text embeddings by weakly-supervise d contrastive pre-training

    Liang Wang et al. “Text embeddings by weakly-supervise d contrastive pre-training”. In: arXiv preprint arXiv:2212.03533 (2022)

  19. [27]

    BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Lan- guage

    Konrad Wojtasik et al. “BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Lan- guage”. In: Proceedings of the 2024 Joint International Conference on C omputational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024, pp. 2149–2160

  20. [28]

    Miracl: A multilingual retrieval da taset covering 18 diverse languages

    Xinyu Zhang et al. “Miracl: A multilingual retrieval da taset covering 18 diverse languages”. In: Transactions of the Association for Computational Linguis tics 11 (2023), pp. 1114–1131

  21. [29]

    A Family of Pretrained Transf ormer Language Models for Russian

    Dmitry Zmitrovich et al. “A Family of Pretrained Transf ormer Language Models for Russian”. In: Proceedings of the 2024 Joint International Conference on C omputational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024, pp. 507–524. 7 Additional metrics 7...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.