Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that one-shot GPT-3.5 prompting generates query variants whose judged relevant document pools overlap human-generated pools by up to 71.1% at depth 100.

desk verdict A clean, reproducible exploratory study showing GPT-3.5 query variants overlap human variants on relevant documents up to 71.1% at depth 100, with the main caveat that the headline number is computed over incomplete judgments. read the letter →

arxiv 2501.17981 v1 pith:6Q4SR2HI submitted 2025-01-29 cs.IR

Marwah Alaofi , Luke Gallagher , Mark Sanderson , Falk Scholer , Paul Thomas This is my paper · ORCID
classification cs.IR
keywords informationretrievaltestcollectionsqueryvariantslargelanguagemodelsdocumentpoolingrelevancejudgmentsone-shotlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Information retrieval test collections are normally built by having crowdsourced workers write many query variants for each topic, which is expensive and hard to scale. This paper asks whether a large language model prompted once with a topic backstory can generate query variants that do the same job, using UQV100, a test collection of one hundred backstories with crowdsourced query variants, as the reference. It compares GPT-3.5 variants against the UQV100 human variants over the collection's corpus, first on similarity of the query strings and then on similarity of the document pools they retrieve when passed through a standard BM25 ranker. The central result is that lexical similarity is modest (Jaccard 7.1–13.5%, coverage up to 18.7%), but judged relevant-document overlap reaches 43.7% at depth 10 and 71.1% at depth 100. The authors conclude that LLM-generated variants could serve as a low-cost complement to human query pooling when building test collections.

What carries the argument

The key object is the query variant set: alternative phrasings of a single information need, generated here by in-context learning, where GPT-3.5 receives a task description, one worked example backstory with human queries, and a new backstory. The comparison machinery is the document pool: for each query set, the union of BM25 results across a topic's variants is measured at increasing depths, with overlap computed as a Jaccard index on all documents and again on judged-relevant documents only. Rank-biased overlap quantifies within-set consistency, and pool growth curves measure how quickly additional variants add new documents.

What would settle it

Judge all (or a random sample of) the documents retrieved only by the GPT-3.5 query sets, then recompute the relevant-document overlap; if a substantial share of those unjudged documents turn out to be relevant, the claimed 71.1% overlap with human pools is not the full picture.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the gap between human and machine query variants is much smaller at the level of retrieved relevant documents than at the level of query wording. A one-shot prompt to GPT-3.5, given the same one hundred backstories used in UQV100, produced about 27–37 variants per topic compared with 57 for the human set, and the generated query strings were only 7.1–13.5% Jaccard-similar to human queries depending on how strictly matches were required. Yet when the variants were run through BM25 and the judged relevant documents were compared, the pools overlapped up to 71.1% at depth 100. The paper interprets this as evidence that generative LLMs can contribute to document pooling for test collection construction, with the caveat that GPT pools contain a larger share of unjudged documents, so the true relevant overlap may not be fully captured by existing judgments.

Load-bearing premise

The comparison treats the UQV100 relevance judgments as the ground truth for relevant documents, so any relevant document retrieved only by a GPT query and never judged is invisible in the 71.1% overlap figure.

Editorial extensions

If this is right

  • If the 71.1% relevant-document overlap holds, LLM-generated query variants could replace a substantial part of the crowdsourcing effort in building new test collection pools.
  • TREC-style collections that already have backstory or topic statements could be expanded cheaply by prompting an LLM, since no human query generation would be needed for the new variants.
  • The lower pool growth and higher within-set consistency of GPT variants imply they sample a narrower diversity of queries, so they are likely a complement rather than a full replacement for human variants.
  • Because GPT pools contain 31–37% unjudged documents, judging those extras is the direct next step; if they contain relevant documents, the current overlap figure underestimates the GPT set's utility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Human evaluation of whether GPT-generated variants read as natural phrasings of a backstory would separate 'looks human' from 'retrieves like human'; the paper leaves that evaluation to future work.
  • Temperature 1.0 gave the highest relevant-document overlap and shorter queries, suggesting that more exploratory sampling might approximate human query diversity better; sampling multiple completions per backstory is a cheap way to test this.
  • Testing the same one-shot template on other LLMs and other topic sets would show whether the 71.1% overlap reflects one-shot prompting in general or something specific to this model and collection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript reports an exploratory study of whether GPT-3.5 (text-davinci-003) prompted with an information-need backstory can generate query variants comparable to the human query variants in the UQV100 test collection. For RQ1, the authors measure lexical overlap between GPT- and human-generated query sets under progressively relaxed matching conditions, reporting Jaccard values up to 13.5% and coverage of up to 18.7%. For RQ2, they run BM25 with both query sets on ClueWeb12-B and compare the resulting document pools, reporting that when only judged relevant documents are considered, GPT pools overlap with human pools up to 71.1% at depth 100. They conclude that LLM-generated variants show potential for replacing human variants during pool construction.

Significance. The potential contribution is real: if LLM-generated query variants can produce document pools close to human-produced pools, test-collection construction and expansion could become substantially cheaper. The paper has concrete strengths: it uses a public corpus and test collection, gives the prompt template and generated queries, reports multiple complementary measures, and transparently notes limitations of the human reference set. The main technical risks are the interpretation of the 71.1% relevant-document overlap in the presence of incomplete relevance judgments, and the use of a symmetric Jaccard metric when the compared pools have very different sizes. These issues are addressable with additional analysis rather than an outright rejection.

major comments (3)
  1. [§3.2, Figure 4, Table 2] The headline 71.1% relevant-document overlap is computed over UQV100 relevance judgments, but Table 2 shows that at depth 10 the GPT-generated pools contain 31–37% unjudged documents compared with 13% for the human pool; unjudged documents are absent from the qrels and are effectively treated as non-relevant in the Jaccard calculation. The manuscript itself labels the unjudged portion as future work ('It would be interesting to further investigate the unjudged portion'), yet Section 4 concludes that GPT 'shows potential for replacing human query variants' on the basis of this overlap. Because the true overlap could differ materially if GPT-only unjudged documents are relevant, the authors should either augment the qrels by judging a sample of GPT-only pool documents (at least at depth 100), or provide a sensitivity analysis that bounds the result under extreme assumptions about the unjudged documents, and adjust the conclusion accordingly.
  2. [§2.3, §3.2, Table 2] The relevant-document Jaccard is symmetric, but the human and GPT pools are very different in size: at depth 10 the human pool averages 190.69 documents per topic, while the GPT pools average 93.55–105.21 documents (Table 2). A smaller GPT pool that is mostly a subset of the larger human pool can therefore achieve a high Jaccard while missing a substantial share of the human-discovered relevant documents. To support the claim that GPT variants 'generate similar sets of relevant documents,' the paper should also report asymmetric coverage measures, for example the fraction of human-pool relevant documents retrieved by the GPT pool and the fraction of GPT-pool relevant documents retrieved by the human pool, at the depths used in Figure 4.
  3. [§3.2, §4] The study does not provide a non-human baseline for the document-overlap results. Without a baseline such as the original TREC topic titles, a random sample of topically related query variants, or the system-pool-only condition, it is difficult to know how much of the 71.1% overlap is attributable to the LLM's query-generation quality and how much would be achieved by any reasonable set of related queries. Adding such a baseline would considerably strengthen the 'potential for replacing human query variants' conclusion.
minor comments (5)
  1. [Figure 3] The axis label '/Q_uery Transformations' appears to contain a typo; it should read 'Query Transformations'.
  2. [Table 2] The notation 'RBP 0.406 +0.111' is unexplained; if the second value is the residual or the standard deviation, please define it, and also define the 'Relevant' and 'Unjudged' columns (proportions of pool documents that are judged relevant and unjudged, respectively).
  3. [§2.1] The statement that few-shot learning was not used because 'we were limited by the number of available backstories' is unclear, since few-shot prompting does not require additional backstories; please clarify the actual constraint.
  4. [§1, References] The citation [12] is used to support the claim that similar findings are reported for 'queries generated to find additional information in response to document summaries,' but the reference points to the CC-News-En corpus paper, which does not obviously contain such an analysis; please verify the citation target.
  5. [§2.3] The sentence beginning 'While we are interested in measuring the overall overlap...' lacks its contrast clause; rephrase for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical comparison with an excluded one-shot example, and the unjudged-pool caveat is a coverage limitation rather than a fitted input.

full rationale

The paper contains no derivational chain that reduces to its own inputs. Human query variants come from the UQV100 test collection; GPT-3.5 variants are generated independently from the same backstories. Query similarity is measured directly by Jaccard index and coverage ratio, and retrieval similarity by BM25 pooling overlap. No parameter is fitted to the headline outcome. The single in-prompt example (topic 275) is explicitly excluded from all analyses, so the one-shot demonstration cannot by construction force the reported overlaps. The only shared resource is UQV100 itself, used both as the source of human queries and as the relevance-judgment ground truth. Two of the present authors co-created UQV100, but it is a publicly available, externally built crowd-sourced collection, not an unverified self-citation or a fitted parameter. The high unjudged fraction of GPT pools (31–37% versus 13% for human pools, Table 2) means the 71.1% relevant-document overlap is computed only over documents already judged relevant in UQV100; the authors explicitly flag this as an open question in Section 3.2. That is an acknowledged data-completeness limitation, not a circular step. No equation, metric, or conclusion is equivalent by construction to an input, and no load-bearing claim is justified solely by self-citation. Therefore the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical comparison, so there are no derived free parameters. The hand-set BM25 settings affect results but are standard defaults. The principal unstated premises are the adequacy of UQV100 judgments and the use of the human set as ground truth; both are partially acknowledged in the text.

free parameters (1)
  • BM25 parameters b and k1 = b=0.4, k1=0.9
    Standard Anserini defaults used for all runs; not tuned to this data, but pool overlaps could shift if retrieval parameters changed.
assumptions (3)
  • domain assumption The UQV100 human query set is treated as the ideal reference set of query variants.
    Section 2.2 states this assumption explicitly and later acknowledges the human set is not exhaustive; all similarity and coverage numbers are relative to it.
  • domain assumption UQV100 relevance judgments are complete enough to support claims about relevant-document overlap.
    Section 2.3 defines relevant overlap using these judgments; Section 3.2 reports 31-37% unjudged documents in GPT pools, so this assumption is load-bearing and only partially acknowledged.
  • domain assumption Excluding the one-shot example topic 275 removes any influence of the example on the comparison.
    Section 2.1 excludes topic 275 to avoid influence of observed data; whether this fully removes contamination is not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study." pith.science (2026). https://pith.science/paper/6Q4SR2HI

@misc{pith2026250117981,
  author       = {Pith},
  title        = {Pith review of: Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6Q4SR2HI}},
  note         = {Machine review of arXiv:2501.17981}
}
read the original abstract

This paper explores the utility of a Large Language Model (LLM) to automatically generate queries and query variants from a description of an information need. Given a set of information needs described as backstories, we explore how similar the queries generated by the LLM are to those generated by humans. We quantify the similarity using different metrics and examine how the use of each set would contribute to document pooling when building test collections. Our results show potential in using LLMs to generate query variants. While they may not fully capture the wide variety of human-generated variants, they generate similar sets of relevant documents, reaching up to 71.1% overlap at a pool depth of 100.

Figures

Figures reproduced from arXiv: 2501.17981 by the authors.

Figure 1
Figure 1. The prompt used to feed the GPT-3.5 model. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Randomly selected example variants generated by [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Jaccard index (left) and coverage ratio (right) be [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The average Jaccard index between the documents [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The average pool size at depth 10 as more variants [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 9 canonical work pages

  1. [1]

    Saling, Mark Sanderson, Falk Scholer, Damiano Spina, and Ryen W

    Marwah Alaofi, Luke Gallagher, Dana Mckay, Lauren L. Saling, Mark Sanderson, Falk Scholer, Damiano Spina, and Ryen W. White. 2022. Where Do Queries Come From?. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval . Association for Computing Machinery, 2850–2862. https://doi.org/10.1145/3477495.3531711

    arXiv 2022

  2. [2]

    Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2016. UQV100: A Test Collection with Query Variability. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval . Association for Computing Machinery, 725–728. https://doi.org/10.1145/2911451. 2914671

    doi:10.1145/2911451 2016

  3. [3]

    Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Unsupervised Dataset Generation for Information Retrieval. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, 2387–2392. https: //doi.org/10.1145/3477495.3531863

    arXiv 2022

  4. [4]

    Timo Breuer, Norbert Fuhr, and Philipp Schaer. 2022. Validating Simulations of User Query Variants. In Advances in Information Retrieval - 44th European Conference on IR Research . Springer, 80–94. https://doi.org/10.1007/978-3-030- 99736-6_6

    doi:10.1007/978-3-030- 2022

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 2020

  6. [6]

    Chris Buckley and Janet A Walz. 1999. The TREC-8 Query Track. In Proceeding of Text Retrieval conference. NIST Special Publication, 500–246

    work page 1999

  7. [7]

    Shane Culpepper, Guglielmo Faggioli, Nicola Ferro, and Oren Kurland

    J. Shane Culpepper, Guglielmo Faggioli, Nicola Ferro, and Oren Kurland. 2021. Topic Difficulty: Collection and Query Formulation Effects. ACM Transactions on Information Systems 40, 1, Article 19 (2021), 36 pages. https://doi.org/10.1145/ 3470563

    work page 2021

  8. [8]

    Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples. (2022). arXiv:2209.11755

    arXiv 2022

Show all 22 references
  1. [9]

    Mitko Gospodinov, Sean MacAvaney, and Craig Macdonald. 2023. Doc2Query–: When Less is More. In Advances in Information Retrieval - 45th European Confer- ence on IR Research. Springer, 414–422. https://doi.org/10.1007/978-3-031-28238- 6_31

    2023 doi

  2. [10]

    Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023. InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval. (2023). arXiv:2301.01820

    2023 arXiv

  3. [11]

    Chris Jordan, Carolyn Watters, and Qigang Gao. 2006. Using Controlled Query Generation to Evaluate Blind Relevance Feedback Algorithms. In Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries . Association for Computing Machinery, 286–295. https://doi.org/1...

    2006

  4. [12]

    Trippas, J

    Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, and Alistair Moffat. 2020. CC-News-En: A Large English News Corpus. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management. Association for Computing...

    2020

  5. [13]

    Alistair Moffat, Peter Bailey, Falk Scholer, and Paul Thomas. 2017. Incorporating User Expectations and Behavior into the Measurement of Search Effectiveness. ACM Transactions on Information Systems 35, 3, Article 24 (2017), 38 pages. https: //doi.org/10.1145/3052768

    2017 doi

  6. [14]

    Alistair Moffat, Falk Scholer, Paul Thomas, and Peter Bailey. 2015. Pooled Evaluation Over Query Variations: Users Are as Diverse as Systems. In Pro- ceedings of the 24th ACM International Conference on Information and Knowl- edge Management. Association for Computing Machiner...

    2015

  7. [15]

    Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. (2019). arXiv:1904.08375

    2019 arXiv

  8. [16]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike,...

    2022

  9. [17]

    Gustavo Penha, Arthur Câmara, and Claudia Hauff. 2022. Evaluating the Ro- bustness of Retrieval Pipelines with Query Variation Generators. In Advances in Information Retrieval - 44th European Conference on IR Research. Springer, 397–412. https://doi.org/10.1007/978-3-030-99736-6_27

    2022 doi

  10. [18]

    Graham Bates

    Karen Spärck Jones and R. Graham Bates. 1977. Report on a Design Study for the “Ideal” Information Retrieval Test Collection . British Library Research and Development Report No. 5428. University of Cambridge

    1977

  11. [19]

    Voorhees

    Ellen M. Voorhees. 2002. The Philosophy of Information Retrieval Evaluation. In Workshop CLEF for European Languages . 355–370

    2002

  12. [20]

    William Webber, Alistair Moffat, and Justin Zobel. 2010. A Similarity Measure for Indefinite Rankings. ACM Transactions on Information Systems 28, 4, Article 20 (2010), 38 pages. https://doi.org/10.1145/1852102.1852106

    2010

  13. [21]

    Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the Use of Lucene for Information Retrieval Research. InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval . Association for Computing Machinery, 1253–12...

    2017

  14. [22]

    Bennett, Nick Craswell, and Saurabh Tiwary

    Hongfei Zhang, Xia Song, Chenyan Xiong, Corby Rosset, Paul N. Bennett, Nick Craswell, and Saurabh Tiwary. 2019. Generic Intent Representation in Web Search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval . Ass...

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.