Pith. sign in

REVIEW 3 cited by

Drowning in Documents: Consequences of Scaling Reranker Inference

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.11767 v2 pith:XB7G7DWP submitted 2024-11-18 cs.IR cs.CLcs.LG

Mathew Jacob , Erik Lindgren , Matei Zaharia , Michael Carbin , Omar Khattab , Andrew Drozdov This is my paper · ORCID
classification cs.IRcs.CLcs.LG
keywords rerankersretrievaldocumentsfirst-stageinitialrerankerassumedassumption
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Rerankers, typically cross-encoders, are computationally intensive but are frequently used because they are widely assumed to outperform cheaper initial IR systems. We challenge this assumption by measuring reranker performance for full retrieval, not just re-scoring first-stage retrieval. To provide a more robust evaluation, we prioritize strong first-stage retrieval using modern dense embeddings and test rerankers on a variety of carefully chosen, challenging tasks, including internally curated datasets to avoid contamination, and out-of-domain ones. Our empirical results reveal a surprising trend: the best existing rerankers provide initial improvements when scoring progressively more documents, but their effectiveness gradually declines and can even degrade quality beyond a certain limit. We hope that our findings will spur future research to improve reranking.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. JointRank: Rank Large Set with Single Pass

    cs.IR 2025-06 conditional novelty 6.0 of 10

    JointRank partitions candidates into overlapping blocks, ranks each block in parallel with an LLM, and reconstructs a global ranking by aggregating the resulting pairwise comparisons.

  2. AI5GTest: AI-Driven Specification-Aware Automated Testing and Validation of 5G O-RAN Components

    cs.NI 2025-06 conditional novelty 6.0 of 10

    An LLM-based framework that generates expected O-RAN and 3GPP procedural flows from standards and validates captured signaling logs against them, reporting 100% accuracy on 15 testbed instances and under an hour per t...

  3. RankLLM: A Python Package for Reranking with LLMs

    cs.IR 2025-05 accept novelty 5.0 of 10

    RankLLM is an open-source Python package that modularly supports pointwise, pairwise, and listwise LLM rerankers, with integrated retrieval, evaluation, training, and response analysis, and reproduces results from Ran...

Pith tools