REVIEW 12 cited by
CoIR: A Comprehensive Benchmark for Code Information Retrieval Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite the substantial success of Information Retrieval (IR) in various NLP tasks, most IR systems predominantly handle queries and corpora in natural language, neglecting the domain of code retrieval. Code retrieval is critically important yet remains under-explored, with existing methods and benchmarks inadequately representing the diversity of code in various domains and tasks. Addressing this gap, we present COIR (Code Information Retrieval Benchmark), a robust and comprehensive benchmark specifically designed to assess code retrieval capabilities. COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains. We first discuss the construction of COIR and its diverse dataset composition. Further, we evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems. To facilitate easy adoption and integration within existing research workflows, COIR has been developed as a user-friendly Python framework, readily installable via pip. It shares same data schema as other popular benchmarks like MTEB and BEIR, enabling seamless cross-benchmark evaluations. Through COIR, we aim to invigorate research in the code retrieval domain, providing a versatile benchmarking tool that encourages further development and exploration of code retrieval systems. https://github.com/CoIR-team/coir.
Forward citations
Cited by 12 Pith papers
-
Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders
Bekko a8m, with 7.7M active parameters, scores 56.2 on MMTEB Multilingual v2 Retrieval, beating mE5 models and BGE-M3, while a25m reaches 57.5, on par with gte-multilingual-base.
-
Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark
A multi-domain corpus of 5,264 scientific GitHub repos plus two IR benchmarks (219 expert queries; 117,950 snippets / 119,720 queries) shows large domain- and documentation-driven gaps in scientific code search.
-
PICopilot: An LLM-based Agentic Framework for Assisting Photonic Integrated Circuit Design via Script Generation
A training-free multi-agent LLM framework with summary-based hybrid retrieval generates correct PIC design tool scripts from natural language, passing all 48 self-built benchmark tasks where the best baseline passes 27.
-
CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval
A code retrieval benchmark with quality labels for correctness, efficiency, security, and maintainability shows most retrievers fail to prefer high-quality code, and quality-aware fine-tuning improves this without mea...
-
O1 Embedder: Let Retrievers Think Before Action
A jointly trained retriever that first generates query thoughts and then encodes them improves accuracy on 12 retrieval benchmarks.
-
ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle
Fine-tuning a 7B code model on real student submissions produces code that better matches student error patterns, style, and incremental revision trajectories than prompting-only models across two temporal resolutions.
-
A Comparative Study of Specialized LLMs as Dense Retrievers
Specialized Qwen2.5 7B models differ in dense retrieval quality: math and long-reasoning variants degrade performance, while coder and vision-language variants improve zero-shot text and code retrieval.
-
CoRet: Improved Retriever for Code Editing
CoRet fine-tunes a code encoder with a repository-level likelihood loss and call-graph context, improving retrieval recall on SWE-bench and Long Code Arena.
-
MIRB: Mathematical Information Retrieval Benchmark
MIRB, a unified benchmark of four math retrieval tasks across 12 datasets, shows current retrieval models score far lower on premise retrieval than on semantic retrieval, and cross-encoder rerankers often hurt.
-
Granite Embedding R2 Models
Granite Embedding R2 is an Apache-2.0 family of ModernBERT-based retrieval and reranking models that posts high average scores on several benchmarks but falls short of top code-retrieval and reranking baselines.
-
Artificial Intelligence and Misinformation in Art: Can Vision Language Models Judge the Hand or the Machine Behind the Canvas?
The manuscript is internally inconsistent: the abstract claims VLM art-attribution experiments, while the full text is an unrelated hybrid-search benchmark paper.
-
Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks
The MTEB maintainers document their infrastructure for versioning and validating benchmark components, plus a zero-shot score that flags models trained on benchmark tasks.
Discussion (0). Continue with ORCID to comment.