REVIEW 18 cited by
BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.
Forward citations
Cited by 18 Pith papers
-
When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration
Model benchmark performance only weakly predicts how well people learn from AI explanations, with notable outliers across code and math.
-
Contrastive Parameter Disentanglement for Multi-modal Remote Sensing Image Generation
A LoRA-based contrastive parameter disentanglement method plus query-key structure transfer generates semantically consistent, structurally aligned OPT/IR/SAR images from a single text prompt.
-
RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents
RICE-PO trains retrieval agents using retrieval scores of intermediate summaries as local rewards, gated by influence and residual-stability estimates, outperforming group-based RL baselines on BRIGHT and BEIR.
-
TongSearch-QR: Reinforced Query Reasoning for Retrieval
TongSearch-QR trains 1.5B and 7B models with GRPO and a frozen-embedding reward to rewrite queries, reaching 27.9 nDCG@10 on BRIGHT with BM25, above GPT-4o's 26.5.
-
SUCEA: Reasoning-Intensive Retrieval for Adversarial Fact-checking through Claim Decomposition and Editing
SUCEA improves adversarial fact-checking by decomposing claims into atomic sub-claims, editing each sub-claim toward retrieved evidence, and re-retrieving before predicting the final label.
-
MIR: Methodology Inspiration Retrieval for Scientific Research Problems
A new dataset and MAG-guided triplet-loss fine-tuning, plus LLM reranking, improves retrieval of methodologically inspirational papers for research proposals by several points over strong baselines.
-
From Token to Action: State Machine Reasoning to Mitigate Overthinking in Information Retrieval
A state-machine framework that replaces token-level chain-of-thought with discrete query-refinement and reranking actions reduces token use by 74% while improving nDCG@10 on retrieval benchmarks.
-
RaDeR: Reasoning-aware Dense Retrieval Models
A math-trained dense retriever and reranker, built from MCTS reasoning trajectories and self-reflection, outperforms strong baselines on reasoning-intensive retrieval benchmarks and beats BM25 on chain-of-thought queries.
-
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective
Diffusion-language-model embeddings beat LLM embeddings on long-document and reasoning retrieval benchmarks, but the paper overstates some gains and releases no code or data.
-
Rank-K: Test-Time Reasoning for Listwise Reranking
Rank-K, a reasoning-model-based listwise reranker distilled from DeepSeek R1 traces, beats RankZephyr on several benchmarks but only marginally on TREC DL 2019/2020.
-
Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval
Reranker-Guided-Search, a greedy graph search steered by reranker scores, outperforms sequential top-k reranking under a fixed budget on three reasoning-intensive retrieval benchmarks.
-
Can Multi-turn Self-refined Single Agent LMs with Retrieval Solve Hard Coding Problems?
A new 254-problem ICPC benchmark with a multi-turn self-judge plus episodic retrieval method lifts o1's pass@1 from 19.1% to 42.2%, and a small human-in-the-loop study finds o1 can solve 17 of 18 previously unsolvable...
-
Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning
State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.
-
Towards Better Instruction Following Retrieval Models
A new training corpus and embedding model improve instruction-following p-MRR by up to 9 points on FollowIR, MAIR, and Bright benchmarks.
-
REARANK: Reasoning Re-ranking Agent via Reinforcement Learning
Training a listwise reranker with reinforcement learning and explicit reasoning on only 179 annotated queries yields reranking quality comparable to GPT-4.
-
Refine Thought: A Test-Time Inference Method for Embedding Model Reasoning
RT improves embedding-model retrieval on some reasoning-heavy tasks by iteratively refining the query embedding at test time, but reported gains are mixed and the key implementation detail is unspecified.
-
Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval
Reasoning-infused text embedding, which prepends LLM-generated reasoning to queries before embedding, improves zero-shot dense retrieval on BRIGHT.
-
Towards a Large Physics Benchmark
The paper outlines a multi-format, expert-scored living benchmark for evaluating physics understanding and creativity in LLMs, supported so far only by a small pilot.
Discussion (0). Continue with ORCID to comment.