Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Rank-K: Test-Time Reasoning for Listwise Reranking

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a listwise reranker trained to reason over passages before ordering them—by distilling DeepSeek R1's reasoning traces into a 32B model—beats the prior listwise reranker RankZephyr by roughly 23% on BM25 top-20…

desk verdict Solid distillation paper with a real new artifact; the reasoning-trace mechanism holds up on TREC DL, but the headline macro gains are inflated by RankZephyr's collapse on NeuCLIR and the lack of significance tests makes effect sizes uncertain. read the letter →

arxiv 2505.14432 v1 pith:SR3B3JYA submitted 2025-05-20 cs.IR cs.CL

Eugene Yang , Andrew Yates , Kathryn Ricci , Orion Weller , Vivek Chari , Benjamin Van Durme , Dawn Lawrie This is my paper
classification cs.IRcs.CL
keywords listwisererankingtest-timereasoningdistillationretrievaleffectivenesscross-languagelargelanguagemodelschain-of-thoughtnDCG@10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rank-K is a listwise passage reranker built by fine-tuning QwQ-32B on rankings and reasoning traces generated by DeepSeek R1: at test time the model thinks through why passages are relevant and emits an ordering. The paper's central claim is that test-time reasoning is what makes the model strong, not merely the teacher's final ranking. On TREC Deep Learning 2019/2020 and TREC NeuCLIR, Rank-K improves nDCG@10 by about 23% over the prior listwise reranker RankZephyr when reranking BM25 top-20 lists, and by about 19% when reranking SPLADE-v3 results. A version trained only on the rankings, without the reasoning trace, is substantially worse, and Rank-K also beats a pointwise reasoning reranker of comparable size. Because its base model is multilingual, Rank-K reranks non-English passages from English queries about as well as translated passages with no additional training.

What carries the argument

The load-bearing object is the reasoning trace: a chain-of-thought produced by DeepSeek R1 under a prompt that asks the model to judge passages by how well they answer the query or satisfy its intent, to allow ties, not to fact-check, and to output only an ordering such as '[3] > [2] > [4] = [1] > [5]'. The generated traces typically summarize each passage, group passages into buckets, produce partial rankings within buckets, assemble a full ranking, and verify it. Rank-K is trained to reproduce these traces, and at inference it emits multiple partial rankings for about 62% of queries, which the paper interprets as dynamic test-time scaling of reasoning effort.

What would settle it

Train two students on the same R1 examples, one with and one without the reasoning trace, and evaluate both on a human-judged benchmark whose queries were never used while developing the prompt; the paper's claim predicts a large gap on that held-out set, and the gap closing would show the trace effect is partly prompt tuning. A second check is to compare Rank-K with a listwise reranker distilled from a teacher of lower judgment quality but identical traces: if gains track teacher identity rather than the reasoning behavior, the supervision source explains the result.

Watch

Extended reading notes

Core claim

The central discovery is that distilling a reasoning model's thinking trace into a smaller reranker improves listwise ranking beyond what the final ranking alone provides. Rank-K is QwQ-32B fine-tuned with LoRA on 100,000 query-passage-set examples generated by DeepSeek R1 with an explicit ranking prompt that allows ties and forbids overthinking and parametric fact-checking. On the reported collections, Rank-K reaches a macro-average nDCG@10 of 0.525 versus 0.424 for RankZephyr when reranking BM25 top-20 results, and 0.642 versus 0.539 on SPLADE-v3 results. The ablation Naive Rank-K, which is trained on the same data with the reasoning trace removed, drops to 0.492 and 0.621, which the authors take as evidence that the reasoning trace drives the improvement. Rank-K also outperforms the pointwise reasoning reranker Rank1-32B, and it reranks native Persian, Russian, and Chinese passages from English queries as effectively as it reranks their English translations.

Load-bearing premise

Rank-K's quality rests on DeepSeek R1's rankings and reasoning traces being accurate enough to serve as exclusive supervision, and on the prompt that elicits them not being over-tuned to the benchmarks used for evaluation.

Editorial extensions

If this is right

  • A reasoning behavior can be distilled from a much larger teacher into a practical reranker: Rank-K matches DeepSeek R1 on average when reranking SPLADE-v3 results while being far cheaper to run.
  • The reasoning trace, not only the teacher's ranking, is what should be supervised: removing the trace from training costs several points of nDCG@10 across the board.
  • Listwise reasoning across passages earns a clear advantage over pointwise reasoning at similar model scale, since Rank-K beats Rank1-32B on both TREC DL and NeuCLIR.
  • Strong first-stage retrievers are a testbed where rerankers often fail: RankZephyr degrades SPLADE-v3 results by 6% on the macro average, while Rank-K improves them by 12%.
  • Multilingual reranking can be obtained for free from a multilingual base: Rank-K reranks Persian, Russian, and Chinese passages from English queries with no cross-language fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit is that the teacher's relevance judgments set the ceiling on Rank-K's quality; comparing Rank-K against rerankers distilled from teachers of known judgment quality would separate retrieval gains from teacher-identity effects.
  • Because the ranking prompt was iterated on TREC DL 2022, a sibling of the DL 2019/2020 test sets, part of the reported advantage could reflect prompt tuning to the evaluation family; a disjoint held-out benchmark would be needed to rule this out.
  • The observation that a single Rank-K call can rerank 50 passages, beyond its 10-20 training depth, suggests listwise reasoning may generalize along list length; a controlled depth-scaling study could turn reasoning length into a tunable quality-versus-latency knob.
  • The reasoning traces contain summarized and repeatedly re-summarized passages; testing whether compressed traces (for example, summaries only) retain most of the gain would indicate where the computation is actually spent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Rank-K, a listwise passage reranker obtained by fine-tuning QwQ-32B with LoRA on DeepSeek R1's final rankings and reasoning traces for sampled MS MARCO queries. The training pipeline constructs candidate lists by reciprocal rank fusion of three retrieval systems, generates teacher rankings and traces with DeepSeek R1, and trains both Rank-K (with traces) and Naive Rank-K (rankings only). The paper evaluates on TREC DL 2019/2020, TREC NeuCLIR, and BRIGHT, reporting nDCG@10 improvements over RankZephyr of 23% for BM25 and 19% for SPLADE-v3, and demonstrates zero-shot cross-lingual reranking. The central mechanism claim is that the reasoning trace, not just the final ranking, drives the improvement, supported by the Rank-K versus Naive Rank-K comparison.

Significance. If the mechanism is robust, this is a meaningful contribution: it is among the first listwise rerankers to distill a reasoning trace from a large teacher into a smaller student, and it publicly releases the training collection, model, and scripts, which will likely be useful to the community. The work includes a sensible control (Naive Rank-K), evaluation on several benchmarks and languages, and a comparison against the teacher model. However, the evidence for the trace-as-cause claim is weakened by the absence of significance tests, by the small effect sizes on the standard TREC DL collections, and by a partially in-distribution SPLADE-v3 evaluation; the abstract's headline gains therefore overstate what the current evidence cleanly supports.

major comments (4)
  1. [Section 5, Table 3] The paper does not report significance tests, confidence intervals, or error bars anywhere. The central trace-versus-no-trace comparison is Rank-K versus Naive Rank-K; the macro-average deltas are 0.033 nDCG@10 for BM25 (0.525 vs. 0.492) and 0.021 for SPLADE-v3 (0.642 vs. 0.621), while on TREC DL 2019/2020 the per-collection deltas range from 0.005 to 0.065. Without query-level variance estimates, the small gains over RankZephyr on TREC DL (0.010 to 0.017 nDCG@10, with one negative cell) and even the trace-versus-no-trace deltas could be within noise. Please add paired bootstrap or permutation tests over query-level nDCG@10 for the key comparisons (Rank-K vs. RankZephyr and Rank-K vs. Naive Rank-K) and report the resulting confidence intervals or p-values.
  2. [Section 3.1 and Table 3] The training candidate lists are built with reciprocal rank fusion of PLAID-X followed by Mono-mT5, CoCondenser, and SPLADE-v3. Since SPLADE-v3 is one of the fusion components, the SPLADE-v3 evaluation column is partially in-distribution for Rank-K and Naive Rank-K, whereas RankZephyr was trained without such candidates. This confounds the 19% SPLADE-v3 headline gain over RankZephyr. Because BM25 is not used in training-candidate construction, the BM25 column is a cleaner comparison; I recommend either reporting the BM25 result as the primary claim or adding an evaluation on a first-stage retriever not used in the training fusion to support the generality of the SPLADE-v3 result.
  3. [Abstract and Section 5] The headline 23% and 19% gains are macro-averages that combine TREC DL and NeuCLIR. On TREC DL 2019/2020 alone, Rank-K's gains over RankZephyr are 0.010 and 0.017 nDCG@10 for BM25 and +0.010 and -0.005 for SPLADE-v3, while the large macro-average gains come from NeuCLIR, where RankZephyr scores below the first-stage baseline. The abstract should either present per-collection numbers or temper the claim; as written, the headline overstates the evidence on the standard TREC DL benchmarks and hides the fact that those gains are small and not shown to be statistically significant.
  4. [Section 5, Table 1] The Rank-K versus Naive Rank-K ablation does not control for the extra test-time computation used by Rank-K: according to Table 1, the teacher traces make Rank-K generate roughly 2.4 times more output tokens than a ranking-only model. To support the mechanistic claim that the reasoning content, rather than merely the additional serial computation or sampling, is responsible, please include a compute-matched baseline, for example Naive Rank-K with multiple sampling passes (self-consistency) or with a fixed-length, non-reasoning continuation that retains the same final-ranking supervision. This would clarify whether the trace's content is the load-bearing ingredient.
minor comments (6)
  1. [Section 1] The sentence 'We show the Rank-K is more effective' should read 'We show that Rank-K is more effective'.
  2. [Table 2] The table row for Deep Learning 2020 is garbled: '334.792020 54' appears to combine the document length and query count into one cell; please reformat the table into separate columns.
  3. [Section 5.2, Table 6] The claim that differences between translated and native documents are 'negligible' is not supported by significance tests; with 167 to 174 queries per language, a paired test or confidence interval would be appropriate.
  4. [Section 5.3, Figure 3] The histogram is reproduced without axis labels; please add labels for the x-axis ('number of rankings in generation output') and y-axis ('number of queries').
  5. [Section 4] The assertion that document batching is 'orthogonal to the modeling and effectiveness of the reranker' is too strong: Table 4 shows that Rank-K with a sliding window outperforms Rank-K with a single call (e.g., top-50: 32.4 vs. 31.1 nDCG@10). Please soften this claim, since batching clearly affects effectiveness in the reported results.
  6. [Section 6.2] Since Rank-R1 (Zhuang et al., 2025) is a concurrent listwise reasoning reranker, the claim that Rank-K is the 'first listwise reranking model that utilizes test-time reasoning' should be qualified with 'to our knowledge' and the differences from Rank-R1 should be discussed explicitly.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Rank-K is an empirical distillation study evaluated on held-out external benchmarks, with the reasoning-trace ablation providing a genuine control.

full rationale

The paper's central claims are empirical rather than derivational. Rank-K is distilled from an external teacher, DeepSeek R1, and evaluated on TREC DL 2019/2020, NeuCLIR, and BRIGHT against external labels. The teacher's rankings and traces are not derived from Rank-K, and the student is not evaluated on its own training labels. The Naive Rank-K ablation, which trains on the same final rankings without the reasoning trace, is a proper control for isolating the trace's contribution, even if the measured gap is modest on some benchmarks. Self-citations to the authors' prior Rank1 work are used as a baseline comparison, not as load-bearing justification. The prompt was iterated on TREC DL 2022 as a development set, which is a mild benchmark-family overlap, but it is explicitly disclosed and held out from the reported experiments; this is an experimental design concern, not a circularity that makes the stated results equivalent to their inputs. Overall, the result is not forced by definition or by a self-citation chain.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central claim depends on the quality of the R1 teacher signal, on the representativeness of MS MARCO queries, and on the evaluation protocol that excludes batching effects. There are no invented physical entities; the introduced artifacts are the Rank-K and Naive Rank-K models, both publicly testable.

free parameters (4)
  • DeepSeek R1 sampling temperature = 0.7
    Set for training trace generation (Section 3.2); no sensitivity analysis is reported.
  • Document truncation length = 450 tokens
    Applied to NeuCLIR and BRIGHT documents before reranking (Section 4); no ablation on this value is reported.
  • LoRA training hyperparameters = 2 epochs, batch size 128, learning rate 1e-4
    Reported in Section 3.3; chosen without ablations in this paper.
  • Reranking depth and sliding window = top 20 for TREC DL/NeuCLIR; top 100 with window 20 and stride 10 for BRIGHT
    Evaluation protocol choices; the paper argues in Section 4 that batching is orthogonal to reranking effectiveness.
assumptions (5)
  • domain assumption DeepSeek R1's rankings and reasoning traces are an accurate supervision signal.
    The entire training set is generated by R1 (Section 3.2) with only informal inspection of a handful of examples; no human verification of ranking quality is described.
  • domain assumption MS MARCO v1 training queries are representative of the test query distributions.
    Training lists are sampled from MS MARCO v1 (Section 3.1); transfer to TREC DL, NeuCLIR, and BRIGHT assumes this.
  • ad hoc to paper Passage batching is orthogonal to reranker effectiveness.
    Stated in Section 4 ('we argue that the document batching is orthogonal'); used to justify single-call top-20 evaluation. The BRIGHT sliding-window results provide partial empirical support.
  • domain assumption The prompt in Figure 2 is understandable and produces valid rankings from QwQ and R1.
    The prompt was iterated on TREC DL 2022 dev data (Section 3.2) but no instruction-following metric is reported.
  • domain assumption TREC and BRIGHT relevance judgments are reliable ground truth.
    All nDCG@10 scores rely on these labels; no judgment-level analysis is included.
invented entities (2)
  • Rank-K model independent evidence
    purpose: Test-time reasoning listwise reranker distilled from DeepSeek R1 via QwQ-32B
    The model is released on GitHub and HuggingFace and can be run on reranking benchmarks, making its outputs externally testable.
  • Naive Rank-K model independent evidence
    purpose: Ablation model trained on final rankings only, with no reasoning traces
    Used to isolate the contribution of the reasoning trace in Section 5; it is also a testable artifact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rank-K: Test-Time Reasoning for Listwise Reranking." pith.science (2026). https://pith.science/paper/SR3B3JYA

@misc{pith2026250514432,
  author       = {Pith},
  title        = {Pith review of: Rank-K: Test-Time Reasoning for Listwise Reranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SR3B3JYA}},
  note         = {Machine review of arXiv:2505.14432}
}
read the original abstract

Retrieve-and-rerank is a popular retrieval pipeline because of its ability to make slow but effective rerankers efficient enough at query time by reducing the number of comparisons. Recent works in neural rerankers take advantage of large language models for their capability in reasoning between queries and passages and have achieved state-of-the-art retrieval effectiveness. However, such rerankers are resource-intensive, even after heavy optimization. In this work, we introduce Rank-K, a listwise passage reranking model that leverages the reasoning capability of the reasoning language model at query time that provides test time scalability to serve hard queries. We show that Rank-K improves retrieval effectiveness by 23\% over the RankZephyr, the state-of-the-art listwise reranker, when reranking a BM25 initial ranked list and 19\% when reranking strong retrieval results by SPLADE-v3. Since Rank-K is inherently a multilingual model, we found that it ranks passages based on queries in different languages as effectively as it does in monolingual retrieval.

Figures

Figures reproduced from arXiv: 2505.14432 by the authors.

Figure 1
Figure 1. Overview of the training pipeline for Rank-K. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prompt for Reasoning and Ranking Passages [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Histogram of the number of ranking Rank-K generates in reranking for each query [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A partial example thinking process produced by Rank-K. Passage summaries and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Revisiting Text Ranking in Deep Research

    cs.IR 2026-02 conditional novelty 6.0 of 10

    In deep-research agents, passage-level BM25 retrieval plus re-ranking beats larger neural rankers, and converting agent queries into natural questions (Q2Q) improves neural ranking.

Reference graph

Works this paper leans on

39 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [6]

    antization and training of neural networks for e cient integer-arithmetic-only inference.arXiv preprint arXiv:1712.05877,

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Ma hew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. antization and training of neural networks for e cient integer-arithmetic-only inference.arXiv preprint arXiv:1712.05877,

    arXiv

  2. [10]

    Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering

    William Jurayj, Jeffrey Cheng, and Benjamin Van Durme. Is that your final answer? test-time scaling improves selective question answering.arXiv preprint arXiv:2502.13962,

    work page Pith review arXiv

  3. [11]

    Dense passage retrieval for open-domain question answer- ing.arXiv preprint arXiv:2004.04906,

    Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing.arXiv preprint arXiv:2004.04906,

    arXiv 2004

  4. [12]

    Splade-v3: New baselines for splade.arXiv preprint arXiv:2403.06789, 2024a

    Carlos Lassance, Herv´e D´ejean, Thibault Formal, and St´ephane Clinchant. Splade-v3: New baselines for splade.arXiv preprint arXiv:2403.06789, 2024a. Carlos Lassance, Ronak Pradeep, and Jimmy Lin. Naverloo @ trec deep learning and neuclir 2023: As easy as zero, one, two, three — cascading dual eencoders, mono, duo, and listo for ad-hoc retrieval. InThe T...

    arXiv 2023

  5. [13]

    Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W

    Proceedings, 2024b. Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldanini, and Eugene Yang. Overview of the trec 2022 NeuCLIR track. InThe Thirty-first Text REtrieval Conference (TREC

    work page 2022

  6. [14]

    Oard, Luca Soldanini, and Eugene Yang

    Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldanini, and Eugene Yang. Overview of the trec 2023 NeuCLIR track. InThe Thirty- second Text REtrieval Conference (TREC

    work page 2023

  7. [15]

    Zero-shot listwise document reranking with a large language model.arXiv preprint arXiv:2305.02156,

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-shot listwise document reranking with a large language model.arXiv preprint arXiv:2305.02156,

    arXiv

  8. [16]

    Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316,

    Niklas Muennighoff, Nouamane Tazi, Lo¨ıc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316,

    arXiv

Show all 39 references
  1. [17]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand`es, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    arXiv

  2. [19]

    doi: 10.18653/v1/2020.findings-emnlp.63

    Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.63. Andrew Parry, Maik Fr¨obe, Harrisen Scells, Ferdinand Schlatt, Guglielmo Faggioli, Saber Zerhoudi, Sean MacAvaney, and Eugene Yang. Variations in relevance judgments and the shelf life of test ...

    2020 arXiv

  3. [20]

    The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.arXiv preprint arXiv:2101.05667, 2021a

    Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.arXiv preprint arXiv:2101.05667, 2021a. Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The expando-mono-duo design pattern for text...

    2024 arXiv

  4. [21]

    doi: 10.18653/v1/2024.findings-naacl.97

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.97. URL https: //aclanthology.org/2024.findings-naacl.97/. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized tra...

    2024 doi

  5. [22]

    13 Preprint

    URL https://qwenlm.github.io/blog/qwq-32b/. 13 Preprint. Under review. Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. First: Faster improved listwise reranking with single token decoding. In Proceedings of the 2024 Confere...

    2024

  6. [23]

    Improving passage retrieval with zero-shot question generation

    Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3781–3797,

    2022

  7. [24]

    Distillation and refinement of reasoning in small language models for document re-ranking.arXiv preprint arXiv:2504.03947,

    Chris Samarinas and Hamed Zamani. Distillation and refinement of reasoning in small language models for document re-ranking.arXiv preprint arXiv:2504.03947,

    arXiv

  8. [25]

    PLAID: an efficient engine for late interaction retrieval

    Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. PLAID: an efficient engine for late interaction retrieval. InProceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 1747–1756, 2022a. Keshav Santhanam, Omar Khattab, ...

    2022 arXiv

  9. [26]

    Distilling reasoning capabilities into smaller language models

    Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 7059–7073,

    2023

  10. [27]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    arXiv

  11. [28]

    Bright: A realistic and challenging benchmark for reasoning-intensive retrieval.arXiv preprint arXiv:2407.12883,

    Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, et al. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval.arXiv preprint arXiv:2407.12883,

    arXiv

  12. [29]

    Is chatgpt good at search? investigating large language models as re-ranking agents

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Proces...

    2023

  13. [30]

    Scaling down, litting up: Effi- cient zero-shot listwise reranking with seq2seq encoder-decoder models.arXiv preprint arXiv:2312.16098,

    Manveer Singh Tamber, Ronak Pradeep, and Jimmy Lin. Scaling down, litting up: Effi- cient zero-shot listwise reranking with seq2seq encoder-decoder models.arXiv preprint arXiv:2312.16098,

    arXiv

  14. [32]

    Under review

    14 Preprint. Under review. Nathaniel Weir, Peter Clark, and Benjamin Van Durme. Nellie: A neuro-symbolic infer- ence engine for grounded, compositional, and explainable reasoning.arXiv preprint arXiv:2209.07662,

    arXiv

  15. [33]

    Enhancing systematic decompositional natural language inference using informal logic.arXiv preprint arXiv:2402.14798,

    Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, et al. Enhancing systematic decompositional natural language inference using informal logic.arXiv preprint arXiv:2402.14798,

    arXiv

  16. [34]

    Followir: Evaluating and teaching infor- mation retrieval models to follow instructions.arXiv preprint arXiv:2403.15246, 2024a

    Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching infor- mation retrieval models to follow instructions.arXiv preprint arXiv:2403.15246, 2024a. Orion Weller, Benjamin Van Du...

    arXiv

  17. [35]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024a. Eugene Yang, Dawn Lawrie, and James Mayfield. Distillation for multilingual inform...

    2024 arXiv

  18. [36]

    doi: 10.18653/v1/2024.findings-acl.706

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.706. URL https: //aclanthology.org/2024.findings-acl.706/. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. I...

    2024 doi

  19. [37]

    doi: 10.18653/v1/2024.acl-demos.38

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-demos.38. URLhttps://aclanthology.org/2024.acl-demos.38/. Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. Open-source large language models are strong zero-shot query likelihood models for documen...

    2024 doi

  20. [38]

    doi: 10.18653/v1/2023.findings-emnlp.590

    Associ- ation for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.590. URL https://aclanthology.org/2023.findings-emnlp.590/. Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. A setwise approach for effective and highly efficient zero-shot ranki...

    2023 doi

  21. [39]

    Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning.arXiv preprint arXiv:2503.06034,

    Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning.arXiv preprint arXiv:2503.06034,

    arXiv

  22. [2009]

    Overview of the trec 2019 deep learning track.arXiv preprint arXiv:2003.07820,

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track.arXiv preprint arXiv:2003.07820,

    2019 arXiv

  23. [2014]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533,

    arXiv

  24. [2016]

    Multi-stage document ranking with BERT.arXiv preprint arXiv:1910.14424, 2019a

    Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with BERT.arXiv preprint arXiv:1910.14424, 2019a. Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert.arXiv preprint arXiv:1910.14424, 2019b. Rod...

    1910 arXiv

  25. [2017]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

    arXiv

  26. [2020]

    Overview of the trec 2020 deep learning track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track. corr abs/2102.07662 (2021).arXiv preprint arXiv:2102.07662,

    2021 arXiv

  27. [2021]

    Overview of the trec 2022 deep learning track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen Voorhees, and Ian Soboroff. Overview of the trec 2022 deep learning track. InThe Thirty-first Text REtrieval Conference (TREC

    2022

  28. [2022]

    Sparse pairwise re- ranking with pre-trained transformers

    Lukas Gienapp, Maik Fr ¨obe, Matthias Hagen, and Martin Potthast. Sparse pairwise re- ranking with pre-trained transformers. InProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval, pp. 72–80,

    2022

  29. [2023]

    Mistral 7b.arXiv preprint arXiv:2310.06825, 10,

    Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bres- sand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 10,

    arXiv

  30. [2024]

    Neuralmind-unicamp at 2022 trec neuclir: Large boring rerankers for cross-lingual retrieval.arXiv preprint arXiv:2303.16145,

    Vitor Jeronymo, Roberto Lotufo, and Rodrigo Nogueira. Neuralmind-unicamp at 2022 trec neuclir: Large boring rerankers for cross-lingual retrieval.arXiv preprint arXiv:2303.16145,

    2022 arXiv

  31. [2025]

    Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, M´arton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemi´nski, Genta Indra Winata, et al

    URLhttps://arxiv.org/abs/2501.12948. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, M´arton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemi´nski, Genta Indra Winata, et al. Mmteb: Massive multilingual text embedding benchmark.arXiv preprint arXiv:2...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.