REVIEW 12 cited by
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Retrieval and ranking models are the backbone of many applications such as web search, open domain QA, or text-based recommender systems. The latency of neural ranking models at query time is largely dependent on the architecture and deliberate choices by their designers to trade-off effectiveness for higher efficiency. This focus on low query latency of a rising number of efficient ranking architectures make them feasible for production deployment. In machine learning an increasingly common approach to close the effectiveness gap of more efficient models is to apply knowledge distillation from a large teacher model to a smaller student model. We find that different ranking architectures tend to produce output scores in different magnitudes. Based on this finding, we propose a cross-architecture training procedure with a margin focused loss (Margin-MSE), that adapts knowledge distillation to the varying score output distributions of different BERT and non-BERT passage ranking architectures. We apply the teachable information as additional fine-grained labels to existing training triples of the MSMARCO-Passage collection. We evaluate our procedure of distilling knowledge from state-of-the-art concatenated BERT models to four different efficient architectures (TK, ColBERT, PreTT, and a BERT CLS dot product model). We show that across our evaluated architectures our Margin-MSE knowledge distillation significantly improves re-ranking effectiveness without compromising their efficiency. Additionally, we show our general distillation method to improve nearest neighbor based index retrieval with the BERT dot product model, offering competitive results with specialized and much more costly training methods. To benefit the community, we publish the teacher-score training files in a ready-to-use package.
Forward citations
Cited by 12 Pith papers
-
Hypencoder: Hypernetworks for Information Retrieval
Hypencoder replaces the query vector with a query-generated neural network that scores document vectors, and reports consistent gains over dense retrieval baselines across standard, out-of-domain, and hard retrieval b...
-
KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval
A 2B single-vector retriever, KoVRE, trained with hard-negative mining and reranker distillation, achieves top aggregate Korean visual document retrieval scores.
-
Relevance-Based Embeddings: Lightweight Candidate Retrieval via Heavy-Ranker Calls
Relevance vectors to support items/queries, mapped by light networks, uniformly approximate any continuous ranker and improve lightweight candidate retrieval over dual encoders.
-
Disentangling Locality and Entropy in Ranking Distillation
Under ranking distillation, complex hard-negative sampling pipelines yield little or no benefit over BM25 sampling, while intermediate teacher score entropy improves in-domain effectiveness and the paper's generalizat...
-
Towards a Relevance Posterior in Neural Information Access
Fusing a cached query-independent document-quality prior with BM25 or re-rankers improves TREC DL nDCG, with largest gains for RankZephyr, under a prior–likelihood view of retrieval.
-
Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection
Distilling LLM-generated reasoning rationales into mBERT via dual-path contrastive distillation improves cross-lingual stance detection by 1–3% accuracy on three benchmarks.
-
RAMP: Robust Ad Recommendation Under Limited Personalized-Feature Availability via Masking and Alignment Pathways
RAMP splits ad prediction into personalized and non-personalized towers, then uses a distillation-style loss to transfer knowledge so non-personalized predictions stay accurate when user features are absent.
-
Score-Only Distillation for Compact Dense Retrieval
Score-only distillation with a row-centered all-pairs PairMSE objective lets 0.6B bi-encoders recover up to 50% of the base-to-teacher retrieval gap under matched protocols.
-
Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data
Contrastive fine-tuning often degrades strong dense retrievers, while combining cross-encoder listwise distillation with diverse synthetic queries consistently improves them.
-
Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT
Lion and AdamW are compared for reranker fine-tuning, but the reported Lion gains are confounded by a 10x learning-rate difference and an inverted GPU-utilization metric.
-
Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion
Exp4Fuse improves sparse retrieval by fusing the ranked lists from the original query and an LLM-expanded query using a modified reciprocal rank fusion.
-
Semantic Certainty Assessment in Vector Retrieval Systems: A Novel Framework for Embedding Quality Evaluation
A query-level score combining quantization stability and neighborhood density predicts retrieval performance and is claimed to improve Recall@10 by only 2 to 3 percent per dataset.
Discussion (0). Continue with ORCID to comment.