REVIEW 38 cited by
Overview of the TREC 2019 deep learning track
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets. The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries. The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test set of 43 queries. This year 15 groups submitted a total of 75 runs, using various combinations of deep learning, transfer learning and traditional IR ranking methods. Deep learning runs significantly outperformed traditional IR runs. Possible explanations for this result are that we introduced large training data and we included deep models trained on such data in our judging pools, whereas some past studies did not have such training data or pooling.
Forward citations
Cited by 38 Pith papers
-
PLAID-PRF: Pseudo-Relevance Feedback with Centroid-like Tokens in PLAID
PLAID-PRF performs pseudo-relevance feedback by treating PLAID's indexing-time centroid codes as semantic terms, selecting diverse reconstructed token vectors to append to the query and rerunning PLAID.
-
Mutual Linearity in and out of Stationarity for Markov Jump Processes: A Trajectory-Based Approach
Trajectory-level linear response yields mutual linearity of observables under single-edge rate perturbation for Markov jump processes, including non-stationary state and counting observables.
-
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
On a new benchmark of post-April 2025 queries, LLM rerankers show a 5-15% performance drop compared with familiar benchmarks, and lightweight models match them on efficiency and sometimes accuracy.
-
The Matryoshka Hypencoder
A Matryoshka-style loss lets one Hypencoder generate Q-Nets of different widths, so a single model can be deployed at multiple efficiency-effectiveness points.
-
Tool-Adaptive LLM Reranker
Pointwise LLM reranking cast as an agentic MDP with language-preserving warm-up and asymmetric cost-aware RL yields adaptive tool use, SOTA NDCG, and high throughput.
-
Trie-based Experiment Plans for Efficient IR Pipeline Experiments
Trie-based experiment plans reduce the duration of comparative evaluations of IR pipelines by 26% versus linear plans in a BM25-MonoT5-DuoT5 demonstration on MSMARCO v2.
-
BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphs
A tournament-graph framework converts each k-wise comparison into many pairwise edges, infers additional orderings transitively, and cuts LLM reranking token use by 25-40% at equal accuracy.
-
Benchmarking Information Retrieval Models on Complex Retrieval Tasks
CRUMB is a new benchmark for complex, multi-aspect retrieval tasks on which state-of-the-art retrieval models score poorly, and query rewriting does not rescue the best models.
-
Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain
Misleading health documents in RAG context sharply lower LLM accuracy, and heavily helpful-biased retrieval pools restore it.
-
Upcycling Candidate Tokens of Large Language Models for Query Expansion
Using unselected top-k candidate tokens from a single LLM decoding pass as extra query terms improves retrieval over standard keyword expansion while using far fewer tokens than document-level methods.
-
ERank: Fusing Supervised Fine-Tuning and Reinforcement Learning for Effective and Efficient Text Reranking
ERank trains a pointwise reasoning-LLM reranker with fine-grained integer scoring and a listwise reinforcement-learning reward, reporting nDCG@10 of 40.2 on BRIGHT for its 32B variant with BM25 hybrid scoring.
-
Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items
The paper contributes alternative-item relevance judgments for fashion CRS targets and two meta-simulators that let users switch targets, reporting that alternative-aware evaluation raises measured CRS effectiveness.
-
Shifting from Ranking to Set Selection for Retrieval Augmented Generation
SETR identifies a query's information requirements with chain-of-thought reasoning and selects a compact passage set, improving multi-hop RAG accuracy over fixed-top-k reranking baselines.
-
Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation
Pairwise Ranking Distillation trains a pointwise ranker from pairwise LLM teacher labels and matches full-pair distillation using just 2% of pairs on TREC-DL.
-
JointRank: Rank Large Set with Single Pass
JointRank partitions candidates into overlapping blocks, ranks each block in parallel with an LLM, and reconstructs a global ranking by aggregating the resulting pairwise comparisons.
-
Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Information
A summary-based anchor document enables contrastive pointwise scoring that, when averaged with ordinary pointwise scores, improves zero-shot LLM reranking.
-
Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval
Dense retrieval embeddings can be decomposed into interpretable latent concepts that serve both as explanations and as efficient sparse indexing units for retrieval.
-
Disentangling Locality and Entropy in Ranking Distillation
Under ranking distillation, complex hard-negative sampling pipelines yield little or no benefit over BM25 sampling, while intermediate teacher score entropy improves in-domain effectiveness and the paper's generalizat...
-
Modeling Ranking Properties with In-Context Learning
In-context examples that encode a target distribution over document attributes can steer LLM rerankers toward fairness and diversity while roughly preserving relevance on four IR benchmarks.
-
Don't "Overthink" Passage Reranking: Is Reasoning Truly Necessary?
Standard pointwise rerankers beat reasoning-based pointwise rerankers under identical training, and removing the reasoning process at inference partially recovers the gap.
-
Reranking with Compressed Document Representation
A reranker trained on 8-token PISCO document embeddings plus a short query achieves near-identical nDCG@10 to full-text rerankers on BeIR and TREC-DL while running up to 16x faster.
-
An Alternative to FLOPS Regularization to Effectively Productionize SPLADE-Doc
DF-FLOPS, a document-frequency-weighted variant of FLOPS regularization, cuts SPLADE-Doc latency in production Solr from 922 ms to 88-161 ms per query with only a 2.2-point MRR@10 loss in-domain.
-
Rank-K: Test-Time Reasoning for Listwise Reranking
Rank-K, a reasoning-model-based listwise reranker distilled from DeepSeek R1 traces, beats RankZephyr on several benchmarks but only marginally on TREC DL 2019/2020.
-
Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval
Reranker-Guided-Search, a greedy graph search steered by reranker scores, outperforms sequential top-k reranking under a fixed budget on three reasoning-intensive retrieval benchmarks.
-
SEAL: Structure and Element Aware Learning to Improve Long Structured Document Retrieval
A contrastive fine-tuning recipe that adds HTML structure and element-masking signals improves long structured document retrieval, with reported MRR@10 gains of about four points on BGE-M3.
-
Does UMBRELA Work on Other LLMs?
UMBRELA relevance judgments made with DeepSeek V3 are close to GPT-4o, and even small models preserve leaderboard rankings although per-document agreement with humans drops.
-
A Comparative Study of Specialized LLMs as Dense Retrievers
Specialized Qwen2.5 7B models differ in dense retrieval quality: math and long-reasoning variants degrade performance, while coder and vision-language variants improve zero-shot text and code retrieval.
-
On the Merits of LLM-Based Corpus Enrichment
LLM-generated, query-biased documents added to a search corpus improve retrieval effectiveness, RAG answer accuracy, and answer attribution in proof-of-concept experiments that use oracle-selected source documents.
-
Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching
Under an LLM-inference cost model, Quicksort with batching uses roughly 44% fewer inference calls than Heapsort for pairwise document ranking, reversing the classical comparison-count ordering.
-
REARANK: Reasoning Re-ranking Agent via Reinforcement Learning
Training a listwise reranker with reinforcement learning and explicit reasoning on only 179 annotated queries yields reranking quality comparable to GPT-4.
-
RankLLM: A Python Package for Reranking with LLMs
RankLLM is an open-source Python package that modularly supports pointwise, pairwise, and listwise LLM rerankers, with integrated retrieval, evaluation, training, and response analysis, and reproduces results from Ran...
-
Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems
The paper adds Type II error metrics to the evaluation of relevance judgment sets and shows that balanced accuracy and Matthews correlation can summarize qrels' discriminative power in one number.
-
The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers
The paper proposes a full-paper, evidence-aware retrieval agenda for scientific fact-checking and shows that combining verification feedback with semantic reranking improves evidence recall on SciFact-Open and Check-COVID.
-
Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT
Lion and AdamW are compared for reranker fine-tuning, but the reported Lion gains are confounded by a 10x learning-rate difference and an inverted GPU-utilization metric.
-
GOLFer: Smaller LM-Generated Documents Hallucination Filter & Combiner for Query Expansion in Information Retrieval
GOLFer filters hallucinated sentences from small-LM-generated hypothetical documents and reweights the rest into the query, improving retrieval at lower cost than large LLM expansion.
-
Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion
Exp4Fuse improves sparse retrieval by fusing the ranked lists from the original query and an LLM-expanded query using a modified reciprocal rank fusion.
-
A New Query Expansion Approach via Agent-Mediated Dialogic Inquiry
AMD uses three LLM agents (Socratic questioning, dialogic answering, reflective feedback) to generate and refine pseudo-answers for query expansion, reporting gains over prior methods on BEIR and TREC benchmarks.
-
The Societal Impact of Foundation Models: Advancing Evidence-based AI Policy
A dissertation that synthesizes prior work on foundation models into a three-part framework: conceptual framing, empirical measurement (HELM, FMTI), and evidence-based AI policy.
Discussion (0). Continue with ORCID to comment.