Pith. sign in

REVIEW 4 major objections 5 minor 55 references

QBD-RankedDataGen: Generating Custom Ranked Datasets for Improving Query-By-Document Search Using LLM-Reranking with Reduced Human Effort

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes QBD-RankedDataGen, an LLM-reranking pipeline for building custom Query-By-Document datasets with little expert labor, and finds that the generated rankings alone are not enough to improve BM25 tuning—human review of the…

desk verdict Useful process paper with a clean reranking comparison, but the headline conclusion about human review being necessary is contradicted by the paper's own Table 5 and unsupported by the small-sample tuning experiments. read the letter →

arxiv 2505.04732 v1 pith:BHGHSF3R submitted 2025-05-07 cs.IR cs.AI

Sriram Gopalakrishnan , Sunandita Patra This is my paper · ORCID
classification cs.IRcs.AI
keywords Query-By-DocumentdatasetgenerationLLMrerankingBM25tuninghuman-in-the-loopinformationretrievalrankeddatasetsTREC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces QBD-RankedDataGen, a process for building custom ranked datasets for Query-By-Document search—where the query is itself a document, as in patent or legal case matching—using an LLM to rerank retrieval candidates and optionally absorbing expert instructions and explanations. The central claim is that this process can sharply reduce the cost of domain-specific dataset creation while still yielding enough signal to select or tune a retrieval model. On two TREC collections, the LLM rerankers approximate ground-truth rankings reasonably well, and embedding similarity is a strong competitor. However, when the reranked data is used to tune BM25's parameters, the authors find that LLM-derived rankings do not reliably beat default BM25, while tuning on human-reviewed ground truth does. Their conclusion is that the human-review step in the pipeline is probably necessary, and that reranking methods should therefore be chosen with human review cost in mind.

What carries the argument

The load-bearing component is the reranking step in the QBD-RankedDataGen pipeline. Given a query document and a set of candidate documents, an LLM scores each candidate either pointwise (a relevance score in $[-1,1]$ plus a natural-language explanation) or pairwise (each ordered pair receives $+1$, $-1$, or $0$, aggregated across all pairs to produce a total score). The resulting ranking becomes the training signal, which is then used to tune BM25's parameters $k_1$ and $b$ by maximizing mean average precision. The explanations are what keep human review cheap: an expert can read why the LLM ranked as it did and correct only the mistakes, rather than re-annotating from scratch.

What would settle it

Run the same BM25-tuning experiment with a larger training set—several hundred query-candidate pairs per dataset, or all candidates for a random subset of queries—keeping the same LLM rerankers; if any LLM-derived ranking tunes BM25 to beat default $k_1=1.5$, $b=0.75$ on held-out queries, the paper's central negative conclusion fails.

Watch

Extended reading notes

Core claim

The paper proposes that a controllable dataset-generation loop for QBD—retrieve candidates, rerank them with an LLM using pointwise or pairwise scoring, optionally inject expert instructions, and route the results through a human reviewer—can produce custom training signals with far less expert effort than manual annotation. Its empirical result is more pointed: on TREC clinicaltrials 2021 and TREC CORD-19, LLM reranking alone produced rankings that were close to ground truth on several ranking measures, yet BM25 tuned on those LLM-generated rankings generally failed to beat default BM25, whereas BM25 tuned on the ground-truth rankings did. The authors therefore assert that LLM reranking alone may not be sufficient for model selection or fine-tuning, and that adding the human-review step may be essential.

Load-bearing premise

The conclusion that LLM-generated rankings cannot improve BM25 tuning rests on a training set of only 65 (clinicaltrials) and 83 (CORD-19) query-candidate pairs, filtered from 100 randomly selected pairs by dropping queries with a single candidate; if that tiny filtered set is unrepresentative of the test distribution, the negative result would not generalize.

Editorial extensions

If this is right

  • Practitioners can build domain-specific QBD datasets with only a few dozen expert-reviewed pairs instead of thousands of manual labels.
  • If LLM-only rankings are used as training labels, BM25 tuning may match but rarely exceed default parameters; human correction of the rankings recovers most of the tuning benefit.
  • Pairwise LLM comparisons are the safer design for long documents because they fit within context limits and are less sensitive to candidate order than listwise prompting.
  • The same retrieval–rerank–review loop can generate ranked datasets for other long-document retrieval tasks, not only QBD.
  • Embedding-similarity reranking is a cheap and strong baseline that should be included in any evaluation of LLM rerankers for such datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: have a human expert correct only a small random sample of LLM rankings, tune BM25 on the corrected ranks, and measure whether most of the ground-truth tuning gain is recovered; this would isolate how much human review is actually necessary.
  • Because training and test queries overlap in the reported split, the negative result may be specific to transferring across candidate sets rather than across queries; an evaluation that withholds entire queries would test query-level generalization.
  • The strong performance of embedding similarity on the clinicaltrials dataset suggests that for some QBD domains, surface similarity is already close to the target ranking, and LLM reranking should be reserved for domains whose matching criteria are not captured by lexical or dense similarity.
  • If the negative result holds across larger datasets, the design goal of dataset-generation pipelines would shift from maximizing LLM ranking quality to minimizing the number of human corrections needed to reach a target ranking quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces QBD-RankedDataGen, a framework for generating custom ranked datasets for query-by-document (QBD) search. The framework uses LLM-based reranking with optional instruction/explanation components, and the generated rankings are then used to tune BM25 parameters. The authors evaluate pointwise and pairwise LLM reranking methods on the TREC clinicaltrials 2021 and TREC CORD-19 datasets, reporting reranking-quality measures and downstream BM25 performance after tuning on the reranked training signal. The paper's central conclusion is that LLM reranking alone may be insufficient and can hurt retrieval performance, and that human review is necessary. The manuscript also claims reduced human effort and a comparative analysis of cost, speed, and human interface, although no such measurements are reported.

Significance. The QBD problem is genuine and important, and the scarcity of graded QBD datasets is a real obstacle; the use of external TREC ground truth makes the reranking evaluation non-circular and cleanly benchmarked. The paper's framework is a reasonable starting point for combining LLM reranking with expert review, and the published tables provide a useful comparison of pointwise and pairwise prompting strategies. However, the central negative claim about the necessity of human review is not established by the reported experiments, and the effort/cost/speed claims are not measured at all. If the empirical claims are revised to match the evidence, the paper would be a modest but useful contribution; as written, the main conclusion overgeneralizes from a small, ambiguously constructed training set and is partly contradicted by the paper's own Table 5.

major comments (4)
  1. [Evaluating BM25 Fine-Tuned Performance, Table 5, and Conclusion] The claim that "only when BM25 is tuned with the ground truth ... does the performance exceed the default parameter settings" is contradicted by the paper's own Table 5. For CORD-19, BM25 tuned on SCS-instr(gpt4omini) achieves MAP 0.803, Kendall's tau 0.224, and P@3 0.807, all above the default BM25 values of 0.796, 0.199, and 0.800. This is not a small reversal; the tuned model also nearly matches BM25IdealTrain's MAP of 0.807. The conclusion in the text should be restricted to the clinicaltrials dataset or revised to state that LLM-generated rankings helped on one dataset and hurt on the other.
  2. [Abstract, Introduction, and Contribution list] The abstract promises a "comparative analysis of our proposed methods in terms of cost, speed, and the human interface with the domain experts," and the introduction claims that the process "can significantly reduce human effort." No experiment or quantitative analysis in the manuscript measures human effort, expert correction counts, annotation time, monetary cost, or wall-clock speed. The only human-factors support is a qualitative discussion that pairwise comparisons are easier for humans to review. These claims are load-bearing for the paper's framing and should either be supported by measurements or removed/softened to claims about the design's potential.
  3. [Experiments, Dataset split and Tables 4-5] The training signal is very small and the split procedure is described ambiguously: the manuscript says "randomly select 100 pairs, from which we drop queries that have only one candidate," yielding 65 query-candidate pairs for clinicaltrials and 83 for CORD-19. Moreover, all training queries also appear in the test set with different candidates, so the evaluation is not query-disjoint. With 50 Optuna trials and no confidence intervals, bootstrap estimates, or repeated splits, the reported MAP differences (e.g., 0.814-0.830 vs. 0.837 in Table 4) are within plausible sampling noise. The authors should report the exact split procedure, the number of queries and candidates per query in the training set, uncertainty estimates, and ideally multiple random splits.
  4. [Reranking Step, SCS-instr and PCS-instr descriptions] The "instructions" condition is realized by pasting the TREC task description into the prompt, not by actual domain-expert instructions. The authors acknowledge this may have been insufficient, but the conclusion still frames the results as evidence about "expert knowledge" insertion and "human-review" necessity. As an experiment, this condition tests only whether the official task description helps, not whether the framework's expert-instruction mechanism works. The limitation should be stated prominently, and the framework-level claims should not be read as tested for real expert input.
minor comments (5)
  1. [Measures, Eq. (3)] The Spearman rank-correlation formula appears malformed; the numerator and denominator are not properly grouped. Please revise the equation.
  2. [Throughout] There are numerous typos and formatting inconsistencies, including "retrival," "datatsets," "V oorhees," "Bm25" vs. "BM25," and "maybe necessary" in the conclusion. A careful proofread is needed.
  3. [Experiments, Reranking Step] The manuscript does not report the exact prompts, LLM version identifiers, temperature settings, random seeds, or number of runs per experiment, which limits reproducibility of the LLM-generated rankings.
  4. [Datasets, TREC clinicaltrials] The text refers to "TREC clinicaltrials 2021" but cite Roberts et al. 2022, which describes TREC 2022; please verify the track year and align the citation.
  5. [Conclusion and Discussion] The concluding sentence says "adding the human-review step maybe necessary," but the paper does not directly test human review; it tests tuning on ground truth as a proxy. Please state this proxy explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; BM25 tuning is evaluated on held-out TREC test data with no self-citation chain.

full rationale

The paper's central empirical loop is: generate ranked signal by LLM reranking on a training set, tune BM25 {k1,b} with Optuna on that signal, and evaluate on a separate test set against external TREC ground truth. The test candidates are disjoint from the training candidates, and the oracle baselines (BM25IdealTrain and BM25IdealTest) are fitted to ground-truth labels rather than to the LLM output, so the LLM-tuned BM25 results are not equivalent by construction to the reranking inputs. The use of the TREC task description as 'instructions' is a limitation, not a circular step, because the instruction text is an external input and is not fitted to the method's predictions. The paper contains no load-bearing self-citations, no imported uniqueness theorem, and no renamed fit. The skeptical concerns about small 65/83-pair training sets, missing uncertainty estimates, and the Table 5 result that SCS-instr-tuned BM25 exceeds the default BM25 on CORD-19 are correctness and statistical-inference issues, not circularity, and do not raise the circularity score under the review rules.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several domain assumptions about the validity of TREC ground truth, the representativeness of the candidate subset, and the comparability of LLM scores. No new physical or conceptual entities are introduced; QBD-RankedDataGen is a name for a process, not an entity. The only fitted values are the BM25 parameters k1 and b, whose tuned values are not reported.

free parameters (2)
  • BM25 k1 = not reported (Optuna searched [1.2, 2.0])
    Tuned on LLM-generated rankings and on ground truth; the central comparison of tuning methods depends on these fitted values, but the tuned values are not reported.
  • BM25 b = not reported (Optuna searched [0.1, 1.0])
    Same as k1; the fitted value is not reported, making it impossible to reproduce the tuned BM25 configurations.
assumptions (6)
  • domain assumption TREC graded relevance scores (2, 1, 0) constitute a valid ground-truth ranking for QBD.
    The paper uses these grades to evaluate reranking and to define the 'ideal' BM25 tuning; see the Datasets section.
  • domain assumption The subset of candidates given to the reranker (relevant documents plus a small number of irrelevant ones) is representative of the true candidate distribution for QBD.
    The section 'Removing Retrieval Impact on Reranking' states the reranker receives a subset of relevant documents; this assumption underlies all reranking quality measurements.
  • domain assumption An LLM's scalar relevance score for a single candidate is comparable across candidates for the same query, so ranking by these scores is meaningful.
    This is the basis of the Single-Document Scoring Methods; the paper does not test score calibration across candidates.
  • domain assumption Aggregating all pairwise comparison scores by summation yields a valid global ranking.
    Section 'Pairwise-document Scoring Methods' adopts the AllPairs approach from (Qin et al. 2023), but the aggregation's validity for long documents is not independently verified.
  • domain assumption The TREC task descriptions are a reasonable proxy for domain-expert instructions.
    Section 'Reranking Step' says instructions come from the task description accompanying the dataset; this is not validated against actual domain experts.
  • domain assumption Default BM25 parameters (k1=1.5, b=0.75) are a meaningful baseline for long-document QBD retrieval.
    This is a standard default in the IR literature and is used as the baseline throughout the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QBD-RankedDataGen: Generating Custom Ranked Datasets for Improving Query-By-Document Search Using LLM-Reranking with Reduced Human Effort." pith.science (2026). https://pith.science/paper/BHGHSF3R

@misc{pith2026250504732,
  author       = {Pith},
  title        = {Pith review of: QBD-RankedDataGen: Generating Custom Ranked Datasets for Improving Query-By-Document Search Using LLM-Reranking with Reduced Human Effort},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BHGHSF3R}},
  note         = {Machine review of arXiv:2505.04732}
}
read the original abstract

The Query-By-Document (QBD) problem is an information retrieval problem where the query is a document, and the retrieved candidates are documents that match the query document, often in a domain or query specific manner. This can be crucial for tasks such as patent matching, legal or compliance case retrieval, and academic literature review. Existing retrieval methods, including keyword search and document embeddings, can be optimized with domain-specific datasets to improve QBD search performance. However, creating these domain-specific datasets is often costly and time-consuming. Our work introduces a process to generate custom QBD-search datasets and compares a set of methods to use in this problem, which we refer to as QBD-RankedDatagen. We provide a comparative analysis of our proposed methods in terms of cost, speed, and the human interface with the domain experts. The methods we compare leverage Large Language Models (LLMs) which can incorporate domain expert input to produce document scores and rankings, as well as explanations for human review. The process and methods for it that we present can significantly reduce human effort in dataset creation for custom domains while still obtaining sufficient expert knowledge for tuning retrieval models. We evaluate our methods on QBD datasets from the Text Retrieval Conference (TREC) and finetune the parameters of the BM25 model -- which is used in many industrial-strength search engines like OpenSearch -- using the generated data.

Figures

Figures reproduced from arXiv: 2505.04732 by the authors.

Figure 1
Figure 1. Flow of Information for QBD Dataset Generation with LLM and Subject Matter Expert [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 26 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abolghasemi, A.; Verberne, S.; and Azzopardi, L. 2022. Improving BERT-based query-by-document retrieval with multi-task optimization. In European Conference on Information Retrieval, 3--12. Springer

    work page 2022

  4. [4]

    Akiba, T.; Sano, S.; Yanase, T.; Ohta, T.; and Koyama, M. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2623--2631

    2019

  5. [5]

    G.; Friedman, P

    Anisuzzaman, D.; Malins, J. G.; Friedman, P. A.; and Attia, Z. I. 2025. Fine-Tuning Large Language Models for Specialized Use Cases. Mayo Clinic Proceedings: Digital Health, 3(1)

    work page 2025

  6. [6]

    Askari, A.; Aliannejadi, M.; Abolghasemi, A.; Kanoulas, E.; and Verberne, S. 2023 a . Closer: conversational legal longformer with expertise-aware passage response ranker for long contexts. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 25--35

    work page 2023

  7. [7]

    Askari, A.; Aliannejadi, M.; Kanoulas, E.; and Verberne, S. 2023 b . Generating synthetic documents for cross-encoder re-rankers: A comparative study of chatgpt and human experts. arXiv preprint arXiv:2305.02320

    arXiv 2023

  8. [8]

    Askari, A.; Verberne, S.; Abolghasemi, A.; Kraaij, W.; and Pasi, G. 2024. Retrieval for extremely long queries and documents with RPRS: a highly efficient and effective transformer-based re-ranker. ACM Transactions on Information Systems, 42(5): 1--32

    work page 2024

Show all 55 references
  1. [9]

    N.; and Beg, M

    Awan, M. N.; and Beg, M. O. 2021. Top-rank: a topicalpostionrank for extraction and classification of keyphrases in text. Computer Speech & Language, 65: 101116

    2021

  2. [10]

    Baldelli, D.; Jiang, J.; Aizawa, A.; and Torroni, P. 2024. TWOLAR: A TWO-Step LLM-Augmented Distillation Method for Passage Reranking. In European Conference on Information Retrieval, 470--485

    2024

  3. [11]

    Bonifacio, L.; Abonizio, H.; Fadaee, M.; and Nogueira, R. 2022. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202.05144

    2022 arXiv

  4. [12]

    Cho, S.; Jeong, S.; yeon Seo, J.; and Park, J. C. 2023. Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker. In Findings of the Association for Computational Linguistics: ACL 2023, 960--971

    2023

  5. [13]

    Choi, E.; Hewlett, D.; Uszkoreit, J.; Polosukhin, I.; Lacoste, A.; and Berant, J. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 209--220

    2017

  6. [14]

    Cohan, A.; Feldman, S.; Beltagy, I.; Downey, D.; and Weld, D. S. 2020. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180

    2020 arXiv

  7. [15]

    Craswell, N.; Mitra, B.; Yilmaz, E.; Campos, D.; and Voorhees, E. M. 2020. Overview of the TREC 2019 deep learning track. arXiv preprint arXiv:2003.07820

    2020 arXiv

  8. [16]

    Y.; Ma, J.; Luan, Y.; Ni, J.; Lu, J.; Bakalov, A.; Guu, K.; Hall, K

    Dai, Z.; Zhao, V. Y.; Ma, J.; Luan, Y.; Ni, J.; Lu, J.; Bakalov, A.; Guu, K.; Hall, K. B.; and Chang, M.-W. 2022. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint arXiv:2209.11755

    2022 arXiv

  9. [17]

    Danilevsky, M.; Wang, C.; Desai, N.; Ren, X.; Guo, J.; and Han, J. 2014. Automatic construction and ranking of topical keyphrases on collections of short documents. In Proceedings of the 2014 SIAM International Conference on Data Mining, 398--406. SIAM

    2014

  10. [18]

    Frej, J.; Schwab, D.; and Chevallet, J.-P. 2019. WIKIR: A Python toolkit for building a large-scale Wikipedia-based English Information Retrieval Dataset. arXiv preprint arXiv:1912.01901

    2019 arXiv

  11. [19]

    O'Reilly Media, Inc

    Gormley, C.; and Tong, Z. 2015. Elasticsearch: the definitive guide: a distributed real-time search and analytics engine. " O'Reilly Media, Inc."

    2015

  12. [20]

    Guan, Z.; Wu, Z.; Liu, Z.; Wu, D.; Ren, H.; Li, Q.; Li, X.; and Liu, N. 2023. Cohortgpt: An enhanced gpt for participant recruitment in clinical study. arXiv preprint arXiv:2307.11346

    2023 arXiv

  13. [21]

    P.; Chiang, M

    Guo, Y.; Tian, P.; Kalpathy-Cramer, J.; Ostmo, S.; Campbell, J. P.; Chiang, M. F.; Erdogmus, D.; Dy, J. G.; and Ioannidis, S. 2018. Experimental Design under the Bradley-Terry Model. In IJCAI, 2198--2204

    2018

  14. [22]

    He, Z.; Chen, K.; Ren, S.; He, X.; Liu, X.; Sun, J.; and Peng, C. 2024. Match-Unity: Long-Form Text Matching With Knowledge Complementarity. IEEE Access

    2024

  15. [23]

    T.; Kumar, A.; Glass, J.; Ratner, A.; Lee, C.-Y.; Krishna, R.; et al

    Hsieh, C.-Y.; Chuang, Y.-S.; Li, C.-L.; Wang, Z.; Le, L. T.; Kumar, A.; Glass, J.; Ratner, A.; Lee, C.-Y.; Krishna, R.; et al. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:2406.16008

    2024 arXiv

  16. [24]

    Jeronymo, V.; Bonifacio, L.; Abonizio, H.; Fadaee, M.; Lotufo, R.; Zavrel, J.; and Nogueira, R. 2023. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01820

    2023 arXiv

  17. [25]

    Liu, W.; Zhu, Y.; and Dou, Z. 2024. Demorank: Selecting effective demonstrations for large language models in ranking task. arXiv preprint arXiv:2406.16332

    2024 arXiv

  18. [26]

    Liu, Y.; Zhou, H.; Guo, Z.; Shareghi, E.; Vuli \'c , I.; Korhonen, A.; and Collier, N. 2024. Aligning with human judgement: The role of pairwise preference in large language model evaluators. arXiv preprint arXiv:2403.16950

    2024 arXiv

  19. [27]

    Long, L.; Wang, R.; Xiao, R.; Zhao, J.; Ding, X.; Chen, G.; and Wang, H. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey, 2024. URL https://arxiv. org/abs/2406.15126

    2024 arXiv

  20. [28]

    Lu, T.; Gao, M.; Yu, K.; Byerly, A.; and Khashabi, D. 2024. Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell. arXiv preprint arXiv:2406.14673

    2024 arXiv

  21. [29]

    Luo, J.; Chen, X.; He, B.; and Sun, L. 2024. Prp-graph: Pairwise ranking prompting to llms with graph aggregation for effective text re-ranking. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5766--5776

    2024

  22. [30]

    Ma, X.; Wang, L.; Yang, N.; Wei, F.; and Lin, J. 2024. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2421--2425

    2024

  23. [31]

    MacAvaney, S.; Yates, A.; Feldman, S.; Downey, D.; Cohan, A.; and Goharian, N. 2021 a . Simplified data wrangling with ir\_datasets. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2429--2436

    2021

  24. [32]

    MacAvaney, S.; Yates, A.; Feldman, S.; Downey, D.; Cohan, A.; and Goharian, N. 2021 b . Simplified Data Wrangling with irdatasets. In SIGIR

    2021

  25. [33]

    Mansour, W.; Zhuang, S.; Zuccon, G.; and Mackenzie, J. 2024. Revisiting Document Expansion and Filtering for Effective First-Stage Retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 186--196

    2024

  26. [34]

    Nguyen, T.; Rosenberg, M.; Song, X.; Gao, J.; Tiwary, S.; Majumder, R.; and Deng, L. 2016. Ms marco: A human-generated machine reading comprehension dataset

    2016

  27. [35]

    Nogueira, R.; Jiang, Z.; Pradeep, R.; and Lin, J. 2020. Document Ranking with a Pretrained Sequence-to-Sequence Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, 708--718

    2020

  28. [36]

    Nouriinanloo, B.; and Lamothe, M. 2024. Re-Ranking Step by Step: Investigating Pre-Filtering for Re-Ranking with Large Language Models. arXiv preprint arXiv:2406.18740

    2024 arXiv

  29. [37]

    Pradeep, R.; Nogueira, R.; and Lin, J. 2021. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. arXiv preprint arXiv:2101.05667

    2021 arXiv

  30. [38]

    Pradeep, R.; Sharifymoghaddam, S.; and Lin, J. 2023. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088

    2023 arXiv

  31. [39]

    Qin, Z.; Jagerman, R.; Hui, K.; Zhuang, H.; Wu, J.; Yan, L.; Shen, J.; Liu, T.; Liu, J.; Metzler, D.; et al. 2023. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306.17563

    2023 arXiv

  32. [40]

    S.; Meem, J

    Rashid, M. S.; Meem, J. A.; Dong, Y.; and Hristidis, V. 2024. EcoRank: Budget-Constrained Text Re-ranking Using Large Language Models. arXiv preprint arXiv:2402.10866

    2024 arXiv

  33. [41]

    Risch, J.; Alder, N.; Hewel, C.; and Krestel, R. 2020. Patentmatch: a dataset for matching patent claims & prior art. arXiv preprint arXiv:2012.13919

    2020 arXiv

  34. [42]

    M.; Bedrick, S.; and Hersh, W

    Roberts, K.; Demner-Fushman, D.; Voorhees, E. M.; Bedrick, S.; and Hersh, W. R. 2022. Overview of the TREC 2022 Clinical Trials Track. In TREC

    2022

  35. [43]

    Y.; Arora, S.; Guha, N.; and R \'e , C

    Saad-Falcon, J.; Fu, D. Y.; Arora, S.; Guha, N.; and R \'e , C. 2024. Benchmarking and building long-context retrieval models with loco and m2-bert. arXiv preprint arXiv:2402.07440

    2024 arXiv

  36. [44]

    Thakur, N.; Reimers, N.; R \"u ckl \'e , A.; Srivastava, A.; and Gurevych, I. B. 2021. A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663

    2021 arXiv

  37. [45]

    Varadarajan, R.; and Hristidis, V. 2006. A system for query-specific document summarization. In Proceedings of the 15th ACM international conference on Information and knowledge management, 622--631

    2006

  38. [46]

    R.; Lo, K.; Roberts, K.; Soboroff, I.; and Wang, L

    Voorhees, E.; Alam, T.; Bedrick, S.; Demner-Fushman, D.; Hersh, W. R.; Lo, K.; Roberts, K.; Soboroff, I.; and Wang, L. L. 2021. TREC-COVID: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, 1--12. ACM New York, NY, USA

    2021

  39. [47]

    Wang, H.; Hu, K.; Dong, H.; and Gao, L. 2024. DocTabQA: Answering Questions from Long Documents Using Tables. In International Conference on Document Analysis and Recognition, 470--487. Springer

    2024

  40. [48]

    u tze, H.; and Str \

    Wang, M.; Stoll, A.; Lange, L.; Adel, H.; Sch \"u tze, H.; and Str \"o tgen, J. 2025. Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansion. arXiv preprint arXiv:2502.12598

    2025 arXiv

  41. [49]

    Wang, S.; Chen, Z.; Liu, B.; and Emery, S. 2016. Identifying search keywords for finding relevant social media posts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30

    2016

  42. [50]

    Xu, R.; Cui, H.; Yu, Y.; Kan, X.; Shi, W.; Zhuang, Y.; Jin, W.; Ho, J.; and Yang, C. 2023. Knowledge-infused prompting: Assessing and advancing clinical text data generation with large language models. arXiv preprint arXiv:2311.00287

    2023 arXiv

  43. [51]

    Zhang, L.; Zhang, Y.; Long, D.; Xie, P.; Zhang, M.; and Zhang, M. 2024. A Two-Stage Adaptation of Large Language Models for Text Ranking. In Findings of the Association for Computational Linguistics ACL 2024, 11880--11891

    2024

  44. [52]

    Zhang, Z.; Gao, J.; and Ciravegna, F. 2018. Semre-rank: Improving automatic term extraction by incorporating semantic relatedness with personalised pagerank. ACM Transactions on Knowledge Discovery from Data (TKDD), 12(5): 1--41

    2018

  45. [53]

    Zhou, Y.; Huang, H.; Wu, Z.; Liu, Y.; and Wang, X. 2024. SST framework for Document Matching. arXiv preprint arXiv:2412.07573

    2024 arXiv

  46. [54]

    Zhu, Y.; Yuan, H.; Wang, S.; Liu, J.; Liu, W.; Deng, C.; Chen, H.; Liu, Z.; Dou, Z.; and Wen, J.-R. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107

    2023

  47. [55]

    Zhuang, H.; Qin, Z.; Jagerman, R.; Hui, K.; Ma, J.; Lu, J.; Ni, J.; Wang, X.; and Bendersky, M. 2023. Rankt5: Fine-tuning t5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.