{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2021:SPRZFPCVGD7CGFD2YDZTOSG6VP","short_pith_number":"pith:SPRZFPCV","schema_version":"1.0","canonical_sha256":"93e392bc5530fe23147ac0f33748deabf975c816f03c8931b8bfa0fdd46c18cd","source":{"kind":"arxiv","id":"2104.08663","version":4},"attestation_state":"computed","paper":{"title":"BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models","license":"http://creativecommons.org/licenses/by-sa/4.0/","headline":"A benchmark with 18 diverse datasets shows BM25 as a robust zero-shot baseline while re-ranking models lead in performance at higher cost.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.IR","authors_text":"Abhishek Srivastava, Andreas R\\\"uckl\\'e, Iryna Gurevych, Nandan Thakur, Nils Reimers","submitted_at":"2021-04-17T23:29:55Z","abstract_excerpt":"Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems includi"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2104.08663","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by-sa/4.0/","primary_cat":"cs.IR","submitted_at":"2021-04-17T23:29:55Z","cross_cats_sorted":["cs.AI","cs.CL"],"title_canon_sha256":"eff90d01865de86a1111fe3b12bc151e9232bce96af744c034df9e46e1ae06cd","abstract_canon_sha256":"1fd562a991a8991536de793d8cf58f04b93e2e9f72c665030905a21657661fdb"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T03:24:29.177544Z","signature_b64":"rtD5h8DQrcYcTmLMMmjMw9I5HrCUPw6oOH6+XeZA5eBmgdX/DTA5rf9MH39ZbKv52wyZitQTUkH3vkGy703JAw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"93e392bc5530fe23147ac0f33748deabf975c816f03c8931b8bfa0fdd46c18cd","last_reissued_at":"2026-07-05T03:24:29.177026Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T03:24:29.177026Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models","license":"http://creativecommons.org/licenses/by-sa/4.0/","headline":"A benchmark with 18 diverse datasets shows BM25 as a robust zero-shot baseline while re-ranking models lead in performance at higher cost.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.IR","authors_text":"Abhishek Srivastava, Andreas R\\\"uckl\\'e, Iryna Gurevych, Nandan Thakur, Nils Reimers","submitted_at":"2021-04-17T23:29:55Z","abstract_excerpt":"Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems includi"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The 18 selected datasets provide a sufficiently heterogeneous and representative sample of out-of-distribution scenarios to draw general conclusions about model generalization in information retrieval.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"BEIR is a heterogeneous zero-shot benchmark showing BM25 as a robust baseline while re-ranking and late-interaction models perform best on average at higher cost, with dense and sparse models lagging in generalization.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A benchmark with 18 diverse datasets shows BM25 as a robust zero-shot baseline while re-ranking models lead in performance at higher cost.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"c8c5486d65b59f6abd205fc193c2546ab52f251a8c9211c93e46e83dfb64620a"},"source":{"id":"2104.08663","kind":"arxiv","version":4},"verdict":{"id":"96c48c26-5450-4830-9dd1-725c1bcf2020","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-12T14:35:26.616158Z","strongest_claim":"Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches.","one_line_summary":"BEIR is a heterogeneous zero-shot benchmark showing BM25 as a robust baseline while re-ranking and late-interaction models perform best on average at higher cost, with dense and sparse models lagging in generalization.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The 18 selected datasets provide a sufficiently heterogeneous and representative sample of out-of-distribution scenarios to draw general conclusions about model generalization in information retrieval.","pith_extraction_headline":"A benchmark with 18 diverse datasets shows BM25 as a robust zero-shot baseline while re-ranking models lead in performance at higher cost."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2104.08663/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":94,"sample":[{"doi":"","year":2019,"title":"Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019. ReQA: An Evaluation for End-to-End Answer Retrieval Models. InProceedings of the 2nd Workshop on Machine Reading for Question Answering, p","work_id":"5d0113a3-6e54-41bc-b4c0-6c8692485ea0","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi","work_id":"6da63d05-546b-4e55-b023-a4f551c20f18","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2021,"title":"In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pages 547–564, Online","work_id":"9df27e51-690e-449e-a906-b69d6e0b3dce","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2015,"title":"Petr Baudiš and Jan Šediv `y. 2015. Modeling of the question answering task in the yodaqa system. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 222–2","work_id":"6d514303-9ae8-4f7f-81e7-1f9ba1b352d2","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2013,"title":"Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Langu","work_id":"a75402fc-378a-4306-95ce-99b8fe6087b9","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":94,"snapshot_sha256":"b1341d9084d08791920925d03692a5cb3bcde1535bbacca8d41c47bdb9d36331","internal_anchors":2},"formal_canon":{"evidence_count":1,"snapshot_sha256":"9ebd7936b407744ce6e0bc8bbb449774aee4d74c233f913afa29e29a412518b8"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2104.08663","created_at":"2026-07-05T03:24:29.177085+00:00"},{"alias_kind":"arxiv_version","alias_value":"2104.08663v4","created_at":"2026-07-05T03:24:29.177085+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2104.08663","created_at":"2026-07-05T03:24:29.177085+00:00"},{"alias_kind":"pith_short_12","alias_value":"SPRZFPCVGD7C","created_at":"2026-07-05T03:24:29.177085+00:00"},{"alias_kind":"pith_short_16","alias_value":"SPRZFPCVGD7CGFD2","created_at":"2026-07-05T03:24:29.177085+00:00"},{"alias_kind":"pith_short_8","alias_value":"SPRZFPCV","created_at":"2026-07-05T03:24:29.177085+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":76,"internal_anchor_count":76,"sample":[{"citing_arxiv_id":"2607.00004","citing_title":"Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"2606.25249","citing_title":"Adaptive Re-Ranking","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2606.24346","citing_title":"PETRA: Transforming Web Text for Petroleum-Engineering Domain Adaptation","ref_index":110,"is_internal_anchor":true},{"citing_arxiv_id":"2606.22778","citing_title":"HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions","ref_index":129,"is_internal_anchor":true},{"citing_arxiv_id":"2606.22807","citing_title":"KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking","ref_index":34,"is_internal_anchor":true},{"citing_arxiv_id":"2606.22692","citing_title":"VISTA Architect: A graph database-oriented health AI system demonstrated in multidisciplinary tumor boards","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2606.18811","citing_title":"Rescaling MLM-Head for Neural Sparse Retrieval","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2607.01538","citing_title":"Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2606.15903","citing_title":"Control-Plane Placement Shapes Forgetting: An Architectural Study of Agent Memory Across Thirteen System Configurations","ref_index":73,"is_internal_anchor":true},{"citing_arxiv_id":"2606.13204","citing_title":"CoDeR: Local Constraint-Compatible Retrieval Beyond Semantic Similarity","ref_index":20,"is_internal_anchor":true},{"citing_arxiv_id":"2606.08577","citing_title":"When Should Queries Be Decomposed? A Stage-Aware Study of Query Decomposition for Multi-Condition Retrieval","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2607.00012","citing_title":"PRA-RAG: Provably Robust Aggregation in Retrieval-Augmented Generation against Retrieval Corruption","ref_index":123,"is_internal_anchor":true},{"citing_arxiv_id":"2606.05568","citing_title":"ColBERTSaR: Sparsified ColBERT Index via Product Quantization","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2606.01070","citing_title":"Test-Time Training for Zero-Resource Dense Retrieval Reranking","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2606.00402","citing_title":"A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2605.02950","citing_title":"Kernel Affine Hull Machines as Compute-Efficient Encoders for Frozen Semantic Spaces","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2605.07210","citing_title":"DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06647","citing_title":"Superintelligent Retrieval Agent: The Next Frontier of Agentic Retrieval","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2605.24764","citing_title":"Spectral Retrieval: Multi-Scale Sinc Convolution over Token Embeddings for Localized Retrieval in LLM Multi-Agent Systems","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2606.29652","citing_title":"As We May Search","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"2606.30625","citing_title":"Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2605.26352","citing_title":"RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2605.31100","citing_title":"Vector Linking via Cross-Model Local Isometric Consistency","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2605.31377","citing_title":"DynaTree: Dynamic Agentic Retrieval Tree for Time-Sensitive News Retrieval","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2605.31575","citing_title":"SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics","ref_index":12,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":1,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP","json":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP.json","graph_json":"https://pith.science/api/pith-number/SPRZFPCVGD7CGFD2YDZTOSG6VP/graph.json","events_json":"https://pith.science/api/pith-number/SPRZFPCVGD7CGFD2YDZTOSG6VP/events.json","paper":"https://pith.science/paper/SPRZFPCV"},"agent_actions":{"view_html":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP","download_json":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP.json","view_paper":"https://pith.science/paper/SPRZFPCV","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2104.08663&json=true","fetch_graph":"https://pith.science/api/pith-number/SPRZFPCVGD7CGFD2YDZTOSG6VP/graph.json","fetch_events":"https://pith.science/api/pith-number/SPRZFPCVGD7CGFD2YDZTOSG6VP/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP/action/timestamp_anchor","attest_storage":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP/action/storage_attestation","attest_author":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP/action/author_attestation","sign_citation":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP/action/citation_signature","submit_replication":"https://pith.science/pith/SPRZFPCVGD7CGFD2YDZTOSG6VP/action/replication_record"}},"created_at":"2026-07-05T03:24:29.177085+00:00","updated_at":"2026-07-05T03:24:29.177085+00:00"}