{"id":"147f1e1e-b942-4de1-9ad9-dc3d30446bd4","arxiv_id":"2505.04732","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-generated rankings for query-by-document search do not improve BM25 tuning over default parameters unless validated by human ground truth.","lead":"This paper tests whether large language models can cheaply generate ranked training data for query-by-document search, then tune the BM25 search engine on that data. The LLM-generated rankings match human rankings well, but tuning BM25 on them does not beat default settings unless a human corrects the rankings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central negative conclusion is not established: Table 5 contradicts it (SCS-instr-tuned BM25 beats default on CORD-19), and no uncertainty estimates are given for the small 65/83-pair training sets, so 'human review necessary' is an overgeneralization.","rationale":"The paper makes a useful process contribution: it describes a concrete pipeline for generating QBD ranked datasets with LLM reranking and a possible human-review step, and its reranking comparisons in Tables 2-3 are informative. The decisive empirical payoff, however, is the conclusion that LLM-generated rankings are insufficient for tuning BM25 and that human review is needed. That conclusion must survive two checks: it must not be contradicted by the paper's own reported numbers, and the observed differences must be distinguishable from sampling noise. It fails the first check on CORD-19, where Table 5 shows BM25 SCS-instr(gpt4omini) at MAP 0.803 vs the default 0.796, with Kendall's tau 0.224 vs 0.199 and P@3 0.807 vs 0.800—so the text's statement that only ground-truth-tuned BM25 exceeds the default is false for one of the two datasets. It fails the second check on both datasets: training sets of 65 and 83 query-candidate pairs, with 2-6 candidates per query and no variance estimates, mean that differences of 0.007-0.023 MAP are within the range of what random sampling could produce. The ambiguously described split, plus the statement that all training queries reappear in the test set with different candidates, further weakens the inference from tuned parameters to a general claim about LLM reranking. A bootstrap or repeated-split analysis would settle this: if the confidence intervals overlap zero, the appropriate conclusion is that the effect is unmeasured, not that LLM-alone tuning hurts. The framework itself may still be valuable, but the headline negative result needs statistical support or revision. This keeps the reader's conditional verdict rather than escalating to reject, because the flaw is correctable by reanalysis and the process contribution remains intact.","tokens_in":16962,"tokens_out":6518,"duration_ms":66493,"concrete_test":"Bootstrap the BM25 tuning experiment: resample the 65 (clinicaltrials) and 83 (CORD-19) training query-candidate pairs with replacement 1000 times, re-run the Optuna tuning for each method and for the default BM25, evaluate on the existing test split, and report 95% confidence intervals for the difference in MAP (and Kendall's tau) between each LLM-tuned BM25 and the default BM25. If any interval includes zero—or is positive for SCS-instr on CORD-19—then the conclusion that LLM-alone tuning 'can in fact hurt performance' and that human review is necessary is not supported. If feasible, also repeat with a query-disjoint split so no training query appears in the test set, removing the stated overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing empirical claim is the Conclusion's assertion that 'using LLM re-ranking alone to select or fine-tune models may not be sufficient, and can in fact hurt performance; adding the human-review step maybe necessary.' This claim justifies the human-in-the-loop design and the paper's main negative result, but it is not supported by the reported experiments. First, the paper's own Table 5 contradicts it: on CORD-19, BM25 tuned on SCS-instr(gpt4omini) achieves MAP 0.803, above the default BM25's 0.796 and close to BM25IdealTrain's 0.807; Kendall's tau also improves (0.224 vs 0.199) and P@3 improves (0.807 vs 0.800). The text's statement that 'only when BM25 is tuned with the ground truth ... does the performance exceed the default parameter settings' is therefore false for one of the two datasets. Second, even where LLM-tuned BM25 is numerically worse (clinicaltrials, Table 4: MAP 0.814-0.830 vs baseline 0.837), the differences are within what sampling noise from 65 query-candidate pairs and 50 Optuna trials could plausibly produce. No confidence intervals, bootstrap estimates, or repeated-split results are reported. The split procedure is also ambiguous ('randomly select 100 pairs, from which we drop queries that have only one candidate'), and the paper states that all training queries appear in the test set as well (with different candidates), so the evaluation is not a clean query-disjoint held-out test. The necessity of human review is thus an overgeneralization from noisy, partially overlapping, small-sample comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QBD-RankedDataGen, a framework for generating custom ranked datasets for query-by-document (QBD) search. The framework uses LLM-based reranking with optional instruction/explanation components, and the generated rankings are then used to tune BM25 parameters. The authors evaluate pointwise and pairwise LLM reranking methods on the TREC clinicaltrials 2021 and TREC CORD-19 datasets, reporting reranking-quality measures and downstream BM25 performance after tuning on the reranked training signal. The paper's central conclusion is that LLM reranking alone may be insufficient and can hurt retrieval performance, and that human review is necessary. The manuscript also claims reduced human effort and a comparative analysis of cost, speed, and human interface, although no such measurements are reported.","tokens_in":17274,"tokens_out":3437,"duration_ms":35944,"significance":"The QBD problem is genuine and important, and the scarcity of graded QBD datasets is a real obstacle; the use of external TREC ground truth makes the reranking evaluation non-circular and cleanly benchmarked. The paper's framework is a reasonable starting point for combining LLM reranking with expert review, and the published tables provide a useful comparison of pointwise and pairwise prompting strategies. However, the central negative claim about the necessity of human review is not established by the reported experiments, and the effort/cost/speed claims are not measured at all. If the empirical claims are revised to match the evidence, the paper would be a modest but useful contribution; as written, the main conclusion overgeneralizes from a small, ambiguously constructed training set and is partly contradicted by the paper's own Table 5.","major_comments":[{"comment":"The claim that \"only when BM25 is tuned with the ground truth ... does the performance exceed the default parameter settings\" is contradicted by the paper's own Table 5. For CORD-19, BM25 tuned on SCS-instr(gpt4omini) achieves MAP 0.803, Kendall's tau 0.224, and P@3 0.807, all above the default BM25 values of 0.796, 0.199, and 0.800. This is not a small reversal; the tuned model also nearly matches BM25IdealTrain's MAP of 0.807. The conclusion in the text should be restricted to the clinicaltrials dataset or revised to state that LLM-generated rankings helped on one dataset and hurt on the other.","section":"Evaluating BM25 Fine-Tuned Performance, Table 5, and Conclusion"},{"comment":"The abstract promises a \"comparative analysis of our proposed methods in terms of cost, speed, and the human interface with the domain experts,\" and the introduction claims that the process \"can significantly reduce human effort.\" No experiment or quantitative analysis in the manuscript measures human effort, expert correction counts, annotation time, monetary cost, or wall-clock speed. The only human-factors support is a qualitative discussion that pairwise comparisons are easier for humans to review. These claims are load-bearing for the paper's framing and should either be supported by measurements or removed/softened to claims about the design's potential.","section":"Abstract, Introduction, and Contribution list"},{"comment":"The training signal is very small and the split procedure is described ambiguously: the manuscript says \"randomly select 100 pairs, from which we drop queries that have only one candidate,\" yielding 65 query-candidate pairs for clinicaltrials and 83 for CORD-19. Moreover, all training queries also appear in the test set with different candidates, so the evaluation is not query-disjoint. With 50 Optuna trials and no confidence intervals, bootstrap estimates, or repeated splits, the reported MAP differences (e.g., 0.814-0.830 vs. 0.837 in Table 4) are within plausible sampling noise. The authors should report the exact split procedure, the number of queries and candidates per query in the training set, uncertainty estimates, and ideally multiple random splits.","section":"Experiments, Dataset split and Tables 4-5"},{"comment":"The \"instructions\" condition is realized by pasting the TREC task description into the prompt, not by actual domain-expert instructions. The authors acknowledge this may have been insufficient, but the conclusion still frames the results as evidence about \"expert knowledge\" insertion and \"human-review\" necessity. As an experiment, this condition tests only whether the official task description helps, not whether the framework's expert-instruction mechanism works. The limitation should be stated prominently, and the framework-level claims should not be read as tested for real expert input.","section":"Reranking Step, SCS-instr and PCS-instr descriptions"}],"minor_comments":[{"comment":"The Spearman rank-correlation formula appears malformed; the numerator and denominator are not properly grouped. Please revise the equation.","section":"Measures, Eq. (3)"},{"comment":"There are numerous typos and formatting inconsistencies, including \"retrival,\" \"datatsets,\" \"V oorhees,\" \"Bm25\" vs. \"BM25,\" and \"maybe necessary\" in the conclusion. A careful proofread is needed.","section":"Throughout"},{"comment":"The manuscript does not report the exact prompts, LLM version identifiers, temperature settings, random seeds, or number of runs per experiment, which limits reproducibility of the LLM-generated rankings.","section":"Experiments, Reranking Step"},{"comment":"The text refers to \"TREC clinicaltrials 2021\" but cite Roberts et al. 2022, which describes TREC 2022; please verify the track year and align the citation.","section":"Datasets, TREC clinicaltrials"},{"comment":"The concluding sentence says \"adding the human-review step maybe necessary,\" but the paper does not directly test human review; it tests tuning on ground truth as a proxy. Please state this proxy explicitly.","section":"Conclusion and Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper's main empirical claim is overstated relative to the evidence, and the effort/cost/speed promises are unsupported. The framework and reranking comparison are reasonable, and the use of external TREC labels is a strength. With a revised conclusion, an added uncertainty analysis, and a clear statement of what was and was not measured, the paper could become acceptable for publication. The fit with cs.IR venues is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a practical process paper with a clean reranking evaluation, but the central negative conclusion overshoots the data. On CORD-19, their own Table 5 shows SCS-instr-tuned BM25 beats default BM25 on MAP (0.803 vs 0.796), Kendall's tau, and P@3. So the summary statement that only ground-truth tuning exceeds the default is just false for one of the two datasets. The paper's advice that human review is 'maybe necessary' is an overgeneralization from 65 and 83 query-candidate pairs with no confidence intervals.\n\nWhat is genuinely new: applying pointwise/pairwise LLM reranking to the task of generating QBD ranked datasets, and systematically comparing the resulting training signals for BM25 tuning. The reranking tables against TREC ground truth are informative — e.g., embedding similarity beating LLM methods on clinicaltrials is a useful datapoint. The process diagram and the discussion of why pairwise comparisons are easier for human review are sensible. Credit also for noting that the TREC task description as instructions did not help.\n\nSoft spots, in order: (1) The headline claim about LLM reranking 'hurting' is contradicted by Table 5 and not statistically supported. Differences like 0.814 vs 0.837 MAP on clinicaltrials could easily be noise with 50 Optuna trials and tiny training sets. (2) The dataset split is described ambiguously ('randomly select 100 pairs, drop queries that have only one candidate') and training queries reappear in the test set with different candidates, so this is not a clean held-out evaluation. (3) The abstract promises a comparative analysis of cost, speed, and human interface; the paper contains no cost or speed measurements, only a qualitative human-factors discussion. (4) Claimed 'reduced human effort' is not measured — the human-review step is never actually run.\n\nVerdict: the process itself is defensible and the reranking comparison is worth having. The paper deserves a serious referee, but the conclusion needs to be rewritten to match the data, and the split and uncertainty reporting need work. I'd accept it for review with the expectation of major revision.","headline":"Useful process paper with a clean reranking comparison, but the headline conclusion about human review being necessary is contradicted by the paper's own Table 5 and unsupported by the small-sample tuning experiments.","tokens_in":17833,"tokens_out":2201,"would_cite":true,"duration_ms":19921,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes QBD-RankedDataGen, an LLM-reranking pipeline for building custom Query-By-Document datasets with little expert labor, and finds that the generated rankings alone are not enough to improve BM25 tuning—human review of the…","keywords":["Query-By-Document","dataset generation","LLM reranking","BM25 tuning","human-in-the-loop","information retrieval","ranked datasets","TREC"],"falsifier":"Run the same BM25-tuning experiment with a larger training set—several hundred query-candidate pairs per dataset, or all candidates for a random subset of queries—keeping the same LLM rerankers; if any LLM-derived ranking tunes BM25 to beat default $k_1=1.5$, $b=0.75$ on held-out queries, the paper's central negative conclusion fails.","tokens_in":16738,"feed_emoji":"🔎","tokens_out":9881,"duration_ms":93022,"temperature":0.7,"pith_summary":"The paper introduces QBD-RankedDataGen, a process for building custom ranked datasets for Query-By-Document search—where the query is itself a document, as in patent or legal case matching—using an LLM to rerank retrieval candidates and optionally absorbing expert instructions and explanations. The central claim is that this process can sharply reduce the cost of domain-specific dataset creation while still yielding enough signal to select or tune a retrieval model. On two TREC collections, the LLM rerankers approximate ground-truth rankings reasonably well, and embedding similarity is a strong competitor. However, when the reranked data is used to tune BM25's parameters, the authors find that LLM-derived rankings do not reliably beat default BM25, while tuning on human-reviewed ground truth does. Their conclusion is that the human-review step in the pipeline is probably necessary, and that reranking methods should therefore be chosen with human review cost in mind.","feed_headline":"LLM reranking alone can hurt document-match search tuning","feed_subtitle":"New pipeline for query-by-document search shows human-validated rankings are needed to tune BM25.","key_machinery":"The load-bearing component is the reranking step in the QBD-RankedDataGen pipeline. Given a query document and a set of candidate documents, an LLM scores each candidate either pointwise (a relevance score in $[-1,1]$ plus a natural-language explanation) or pairwise (each ordered pair receives $+1$, $-1$, or $0$, aggregated across all pairs to produce a total score). The resulting ranking becomes the training signal, which is then used to tune BM25's parameters $k_1$ and $b$ by maximizing mean average precision. The explanations are what keep human review cheap: an expert can read why the LLM ranked as it did and correct only the mistakes, rather than re-annotating from scratch.","core_discovery":"The paper proposes that a controllable dataset-generation loop for QBD—retrieve candidates, rerank them with an LLM using pointwise or pairwise scoring, optionally inject expert instructions, and route the results through a human reviewer—can produce custom training signals with far less expert effort than manual annotation. Its empirical result is more pointed: on TREC clinicaltrials 2021 and TREC CORD-19, LLM reranking alone produced rankings that were close to ground truth on several ranking measures, yet BM25 tuned on those LLM-generated rankings generally failed to beat default BM25, whereas BM25 tuned on the ground-truth rankings did. The authors therefore assert that LLM reranking alone may not be sufficient for model selection or fine-tuning, and that adding the human-review step may be essential.","pith_inferences":["A testable extension: have a human expert correct only a small random sample of LLM rankings, tune BM25 on the corrected ranks, and measure whether most of the ground-truth tuning gain is recovered; this would isolate how much human review is actually necessary.","Because training and test queries overlap in the reported split, the negative result may be specific to transferring across candidate sets rather than across queries; an evaluation that withholds entire queries would test query-level generalization.","The strong performance of embedding similarity on the clinicaltrials dataset suggests that for some QBD domains, surface similarity is already close to the target ranking, and LLM reranking should be reserved for domains whose matching criteria are not captured by lexical or dense similarity.","If the negative result holds across larger datasets, the design goal of dataset-generation pipelines would shift from maximizing LLM ranking quality to minimizing the number of human corrections needed to reach a target ranking quality."],"forward_implications":["Practitioners can build domain-specific QBD datasets with only a few dozen expert-reviewed pairs instead of thousands of manual labels.","If LLM-only rankings are used as training labels, BM25 tuning may match but rarely exceed default parameters; human correction of the rankings recovers most of the tuning benefit.","Pairwise LLM comparisons are the safer design for long documents because they fit within context limits and are less sensitive to candidate order than listwise prompting.","The same retrieval–rerank–review loop can generate ranked datasets for other long-document retrieval tasks, not only QBD.","Embedding-similarity reranking is a cheap and strong baseline that should be included in any evaluation of LLM rerankers for such datasets."],"supporting_citations":[{"why":"Supplies the TREC clinicaltrials 2021 collection with graded relevance judgments for patient-to-trial matching.","marker":"(Roberts et al. 2022)"},{"why":"Supplies the TREC CORD-19 collection with graded relevance judgments for covid-19 query-to-paper matching.","marker":"(V oorhees et al. 2021)"},{"why":"Provides the irdatasets loader through which both test collections were downloaded and preprocessed.","marker":"(MacAvaney et al. 2021b)"},{"why":"Provides the survey of LLM reranking strategies the paper adapts into pointwise and pairwise scoring methods.","marker":"(Zhu et al. 2023)"},{"why":"Introduces pairwise ranking prompting and AllPairs aggregation, the pattern used for the PCS rerankers, including order swapping.","marker":"(Qin et al. 2023)"},{"why":"Grounds the choice of BM25 as the industrial retrieval model whose parameters are tuned.","marker":"(Gormley and Tong 2015)"},{"why":"Supplies Optuna, the hyperparameter optimizer used to tune BM25 on the generated rankings.","marker":"(Akiba et al. 2019)"},{"why":"Provides the duo-style pairwise comparison design that motivates the paper's pairwise candidate scoring.","marker":"(Pradeep, Nogueira, and Lin 2021)"}],"fun_headline_variants":["LLM reranking alone can hurt BM25 tuning","Human review key for QBD dataset generation","LLM rankings need human check for search tuning","QBD data pipeline: LLM plus human beats LLM alone","For QBD tuning, LLM reranking isn't enough"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that LLM-generated rankings cannot improve BM25 tuning rests on a training set of only 65 (clinicaltrials) and 83 (CORD-19) query-candidate pairs, filtered from 100 randomly selected pairs by dropping queries with a single candidate; if that tiny filtered set is unrepresentative of the test distribution, the negative result would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["LLM reranking alone can hurt BM25 tuning","Human review key for QBD dataset generation","LLM rankings need human check for search tuning","QBD data pipeline: LLM plus human beats LLM alone","For QBD tuning, LLM reranking isn't enough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1393,"prompt_tokens":970,"completion_tokens":423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":586,"tokens_out":423,"duration_ms":4012,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:21:49.127448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same BM25-tuning experiment with a larger training set—several hundred query-candidate pairs per dataset, or all candidates for a random subset of queries—keeping the same LLM rerankers; if any LLM-derived ranking tunes BM25 to beat default $k_1=1.5$, $b=0.75$ on held-out queries, the paper's central negative conclusion fails.","supporting_citations":[{"cited_title":"O'Reilly Media, Inc","cited_arxiv_id":null,"evidence_quote":"Grounds the choice of BM25 as the industrial retrieval model whose parameters are tuned."}],"review_version":1}