Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Leveraging LLMs to Evaluate Usefulness of Document

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM usefulness labels, built from click behavior and context, agree with real users better than third-party relevance annotations and improve satisfaction prediction.

desk verdict Solid and useful, but the 'surpass third-party labels' claim is a single-dataset, no-significance result that needs scaling back before publication. read the letter →

arxiv 2506.08626 v2 pith:NGSKDVWG submitted 2025-06-10 cs.IR

Xingzhu Wang , Erhan Zhang , Yiqun Chen , Jinghan Xuan , Yucheng Hou , Yitong Xu , Ying Nie , Shuaiqiang Wang
show 2 more authors
Dawei Yin Jiaxin Mao
This is my paper · ORCID
classification cs.IR
keywords usefulnessevaluationusersatisfactionLLMrelevancejudgmentordinalregressioncascadingbinaryclassifierssearchclickbehaviortestcollectionconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large language models, when shown the searcher's query, task background, and click behavior, can judge how useful a document was to that user more accurately than third-party relevance annotators. The authors build CLUE, a cascading set of LLM binary judges that assigns multilevel usefulness scores by asking, at each level, which remaining clicked documents deserve that tier. They test CLUE against third-party usefulness and relevance labels, a GBDT baseline, and pointwise, pairwise, and listwise LLM prompting, reporting that CLUE outperforms the label baselines and stays competitive with the machine-learning baseline without training data. They then use CLUE's usefulness labels as features in satisfaction-prediction models, reporting gains over behavior-only and relevance-label features on both a lab dataset and industrial search logs. If the claim holds, LLM-produced usefulness labels offer a cheaper, more user-centered substitute for relevance labels in building and evaluating test collections.

What carries the argument

The load-bearing mechanism is the cascade of binary LLM classifiers built on the OneVsPrevious ordinal decomposition. Rather than asking the LLM to produce a multilevel label directly, CLUE presents all clicked documents for a query together and asks, at each stage from top level down, which documents deserve that level; documents that pass majority voting among five permuted voters are assigned that level and removed, and the rest fall to lower levels until leftovers are assigned the bottom level. The prompt packs in the query string, task description and background, click order, dwell time, query-level click aggregates, and six usefulness guidelines (helpful, detailed, related, encyclopedic, specific, comprehensive) harvested from think-aloud comments. This design converts usefulness judgment into a sequence of simpler selection tasks where the LLM's ability to hold context and reason matters more than calibrated probability estimates.

What would settle it

Collect a new dataset of real-user usefulness judgments on queries and tasks not drawn from the UUST design, run CLUE with the published six guidelines, and check whether its agreement with those users (for example, Pearson correlation and F1) still exceeds third-party annotation; if the gap disappears on this held-out population, the reported CLUE advantage is an artifact of the original rubric and task distribution.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is that usefulness, not relevance, is the label that tracks user satisfaction, and that LLMs can produce such labels when the prompt reconstructs the user's situation. Concretely, the paper reports that CLUE with GPT-4o reaches a Pearson correlation of 0.383 with real user usefulness judgments on SIGIR16, beating third-party usefulness annotations (0.354) and relevance annotations (0.320); on UUST it reaches F1 of 0.445 and Pearson correlation of 0.428. On KDD19, a lab dataset with complex tasks, CLUE reaches 0.346 Pearson without any training, which the paper characterizes as comparable to the out-of-distribution GBDT model. The same CLUE labels, when added as features to a satisfaction classifier, outperform relevance-based label features on both KDD19 and SearchLog24Q3. The paper interprets this as evidence that language models can bridge the situational disconnect between third-party annotators and real users.

Load-bearing premise

The six guidelines derived from think-aloud comments of 31 students on 10 tasks are assumed to describe what any searcher means by useful, so LLMs guided by them will agree with real users beyond this study's setup.

Editorial extensions

If this is right

  • CLUE's usefulness labels can be generated without collecting training data for a new domain, unlike GBDT-based usefulness predictors that the paper finds degrade under distribution shift.
  • Search evaluation metrics built on CLUE usefulness labels predict user satisfaction better than metrics built on relevance labels, on both a lab dataset and an industrial search log.
  • The six guidelines derived from the UUST user study transfer to the KDD19 dataset, suggesting a single rubric can guide LLM judges across datasets.
  • Fine-tuning an open-source LLM as separate binary classifiers improves its usefulness judgments, though the paper reports it does not close the gap with the commercial model.
  • Because CLUE labels only clicked documents, the framework reduces annotation effort relative to full relevance judgments while still supporting query-level evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct transfer test would be to run CLUE, with the published six guidelines, on a dataset whose usefulness ground truth comes from a different user population than the one that supplied the guidelines; this would measure rubric generality rather than in-sample fit.
  • The cascade structure could generalize to other graded judgment tasks with short candidate lists, such as rating the helpfulness of answers in community question-answering, where ordinal labels and few candidates are natural.
  • Vote counts from the five voters could be read as soft confidence scores, not just binary decisions, enabling active learning or threshold calibration for usefulness labeling at scale.
  • Feature-level ablations, removing behavioral features, context features, and content features one at a time, could reveal which part of the situational information carries the predictive gain; the paper's ablations cover whole components but not individual feature groups.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CLUE (Cascade LLM-based Usefulness Evaluation), a framework that uses LLMs to generate multilevel usefulness labels for web-search evaluation. CLUE integrates query, document content, user context, and behavioral signals into prompts, decomposes the ordinal usefulness judgment into a cascade of binary selection stages (inspired by ordinal binary decomposition), uses multiple voters to reduce position bias, and optionally fine-tunes an open-source Llama model. The authors introduce a new think-aloud dataset (UUST) from which they extract six usefulness guidelines (helpful, detailed, related, encyclopedic, specific, comprehensive), and they evaluate CLUE on SIGIR16, KDD19, UUST, and an industrial search log (SearchLog24Q3). They compare against third-party annotations (U_a, R_a), a GBDT predictor, and LLM pointwise/pairwise/listwise scoring, and they further show that usefulness labels from CLUE improve query-level satisfaction prediction.

Significance. If the central claims are supported, the paper offers a practical, cheaper, user-centric alternative to relevance annotation, a new think-aloud usefulness dataset, and an open-source implementation. The cascading ordinal decomposition is a sensible adaptation of ordinal-regression ideas to LLM judgment, and the RQ3 result on industrial data (Table 10) is a concrete demonstration that LLM-generated usefulness labels can improve satisfaction prediction. However, the headline claim that the method 'surpasses third-party labeling methods' rests on a single dataset with very small margins and no significance testing, and the usefulness guidelines are extracted from the same UUST data on which they are later ablated, creating an in-sample-leakage risk. The paper's contributions are valuable but require stronger empirical support and more careful framing.

major comments (3)
  1. [Section 5.1, Table 5] The abstract's claim that the approach 'surpass[es] third-party labeling methods' is supported only by the SIGIR16 comparison between CLUE-4o and U_a. The margins are small (Pearson r 0.383 vs 0.354; F1 0.381 vs 0.376; Cohen's κ 0.185 vs 0.168), MAE is worse for CLUE (0.984 vs 0.921), no significance test or confidence interval is reported, and this is the only dataset with a U_a column. The GBDT_iid baseline (Pearson r 0.430) also exceeds CLUE, so the 'surpass' claim is not robust. Please add significance tests (e.g., bootstrap or paired tests over queries) and report effect sizes, or qualify the claim to describe CLUE as comparable to third-party annotation on this single dataset.
  2. [Section 3.1.4 and Section 5.2.4, Figure 4] The six-guideline rubric (helpful, detailed, related, encyclopedic, specific, comprehensive) is extracted from think-aloud comments of the 31 UUST participants, and the guideline ablation is then evaluated on the UUST test set, which is drawn from the same participants and sessions. This is in-sample leakage: the rubric is fitted to the same distribution it is evaluated on. The KDD19 transfer result provides only a second lab-study check without a significance test. To support the claimed generalizability, the guidelines should be derived from a training split of participants/tasks and tested on a held-out split of participants/tasks, or on a third dataset from a different population.
  3. [Section 5.3, Tables 9-10 and Figure 5] The RQ3 comparison confounds label type with input features: U_llm is generated by prompts that include behavior and context features (Table 4), whereas R_llm is explicitly generated without behavior or context features (Table 1), and R'_a is a text-based relevance model. The observed improvement of +U_llm over +R_llm/+R'_a may therefore be due to the additional behavioral/contextual features rather than to the usefulness construct. The authors should add a condition with relevance labels generated using the same behavior/context prompt template, or an ablation in which usefulness is judged without behavior/context, to isolate the effect of the label type.
minor comments (6)
  1. [Section 5.3.2] The text says 'the five-voter version of CULE'; 'CULE' should be 'CLUE'.
  2. [Figure 2 caption] The caption contains the duplicated phrase 'where curly braces curly braces indicates the features'.
  3. [Figure 3] The example text says 'The contnet in [2] and [3]'; 'contnet' should be 'content'.
  4. [Table 8] The row labels for SIGIR16 appear to omit 'FT' on the second row; as printed, both SIGIR16 rows are labeled 'Llama-3', making the table ambiguous.
  5. [Figure 4] The x-axis labels 'w/o w' and 'w/ w' are unclear; they presumably mean 'w/o guideline' and 'w/ guideline'.
  6. [Section 4.4.3 and Appendix A.1] The UUST dataset is said to be released after publication; for reproducibility, consider also releasing the anonymized think-aloud transcripts and the exact prompt templates used in the study.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity; the only partial circularity is the UUST-derived guidelines being evaluated on UUST.

  1. other [Section 3.1.4 and Section 5.2.4, Figure 4]
    "we get the six most frequently occurring descriptive adjectives: helpful, detailed, related, encyclopedic, specific, and comprehensive. We seen them as guidelines and incorporate them into prompt ... (Section 3.1.4) ... incorporating the guidelines obtained from the UUST dataset into prompts leads to notable improvements in performance on the UUST dataset. (Section 5.2.4)"

    The six usefulness guidelines are extracted from the think-aloud comments of the UUST participants, and the UUST dataset is then used as the evaluation set for the guideline ablation. Because the think-aloud comments were collected while the same participants assigned the U_u usefulness labels that serve as ground truth, the UUST improvement of CLUE over w/o G reflects information obtained from the test population, making that specific comparison partly in-sample rather than a true out-of-sample validation. The KDD19 results provide independent transfer evidence, and the central 'surpass third-party labeling' claim on SIGIR16 does not depend on this ablation, so the circularity is localized and partial.

full rationale

The paper's core derivations are not circular: CLUE is validated against externally collected user usefulness labels (U_u) and third-party annotations (U_a), and the RQ3 satisfaction experiments use separate satisfaction ground truth. The claims that relevance labels are weakly correlated with satisfaction and that usefulness is more user-centric are supported by prior work by Mao et al. that is independent of the present paper's fitted values, even though one of the authors is shared; that is ordinary self-citation, not load-bearing circularity. The cascade ordinal structure is explicitly presented as an LLM generalization of the Cascade Linear Utility Model, so no uniqueness theorem is imported to force the choice. The one genuine circularity concern is the guideline ablation: the six adjectives were derived from UUST think-aloud data and then evaluated on UUST, so the reported UUST gain is partly a fit to the evaluation population rather than a prediction. This does not undermine the main external-benchmark claims, but it does inflate one supporting result. Overall circularity score is therefore 2, not higher.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard ordinal regression machinery plus several domain assumptions about user self-reports, behavioral features, guideline generalizability, and silver annotator validity. The most fragile item is the ad hoc guideline lexicon fitted to UUST and then evaluated on UUST.

free parameters (4)
  • voter_count_M = 5
    Number of LLM voters per cascade stage, chosen in Section 5.1.1 without tuning; only ablated as single-voter vs five-voter.
  • guideline_lexicon = helpful, detailed, related, encyclopedic, specific, comprehensive
    Six most frequent usefulness adjectives from UUST think-aloud comments (Section 3.1.4); this is a data-fitted component applied to the same dataset in the UUST evaluation.
  • llm_temperature = 0
    Temperature set to 0 for all judgments (Section 5.1.1) to reduce stochasticity; affects reproducibility.
  • fine_tuning_epochs = 2
    LoRA fine-tuning epochs for the Llama classifier set to 2 (Section 5.2.5); no sweep reported.
assumptions (5)
  • standard math Ordinal binary decomposition (OneVsPrevious) with threshold classifiers is a valid way to reconstruct multilevel ordinal labels.
    Section 3.1.2 invokes Wu et al. [55] and the ordinal regression literature; this is a standard, cited technique not derived in the paper.
  • domain assumption Users' self-reported usefulness ratings U_u are a valid ground truth for document usefulness.
    Table 1 defines U_u as ground truth; the paper does not validate these self-reports against objective task outcomes.
  • ad hoc to paper The six high-frequency adjectives extracted from 31 students' think-aloud comments form a generalizable usefulness rubric.
    Section 3.1.4 derives the guideline from UUST and applies it to UUST test data; this is an assumption specific to this paper.
  • domain assumption Behavioral features (dwell time, click order, session_end, average dwell time) are reliable indicators of user perception when converted to text for an LLM.
    Figure 3 prompt feeds these behavior features; if they are noisy or misleading, the method's edge over third-party annotators is unsupported.
  • domain assumption Expert silver annotators and ERNIE relevance labels are valid proxies for true user satisfaction and relevance on SearchLog24Q3.
    Section 5.3.2 lacks gold user satisfaction labels and uses silver expert labels and R'_a from ERNIE; this assumes those proxies are unbiased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging LLMs to Evaluate Usefulness of Document." pith.science (2026). https://pith.science/paper/NGSKDVWG

@misc{pith2026250608626,
  author       = {Pith},
  title        = {Pith review of: Leveraging LLMs to Evaluate Usefulness of Document},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGSKDVWG}},
  note         = {Machine review of arXiv:2506.08626}
}
read the original abstract

The conventional Cranfield paradigm struggles to effectively capture user satisfaction due to its weak correlation between relevance and satisfaction, alongside the high costs of relevance annotation in building test collections. To tackle these issues, our research explores the potential of leveraging large language models (LLMs) to generate multilevel usefulness labels for evaluation. We introduce a new user-centric evaluation framework that integrates users' search context and behavioral data into LLMs. This framework uses a cascading judgment structure designed for multilevel usefulness assessments, drawing inspiration from ordinal regression techniques. Our study demonstrates that when well-guided with context and behavioral information, LLMs can accurately evaluate usefulness, allowing our approach to surpass third-party labeling methods. Furthermore, we conduct ablation studies to investigate the influence of key components within the framework. We also apply the labels produced by our method to predict user satisfaction, with real-world experiments indicating that these labels substantially improve the performance of satisfaction prediction models.

Figures

Figures reproduced from arXiv: 2506.08626 by the authors.

Figure 1
Figure 1. Comparison between traditional Cranfield evaluation and the usefulness-based framework proposed by Mao et al. [ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Detail prompt of "[1]document content and other [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The prompt of CLUE. We use the DNA prompt [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of the performance of the full model CLUE versus w/o G. The MAE references the right y-axis scale; [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Satisfaction prediction performance (KDD19) Metric 𝑅𝑎 𝑅𝑙𝑙𝑚 𝑈𝑙𝑙𝑚 P - 0.3579 0.3820 R - 0.3619 0.3751 F1 - 0.3591 0.3769 P-𝑟 0.2579 0.2730 0.3087 S-𝜌 0.2692 0.2739 0.3127 C-𝜅 - 0.1166 0.1440 MAE 1.0459 1.0093 Note: 𝑅𝑎 cannot compute some metrics because it ranges from 0 …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLMs for estimating positional bias in logged interaction data

    cs.IR 2025-09 conditional novelty 6.0 of 10

    An LLM-as-a-judge relevance score lets the authors estimate examination propensities from logged clicks, exposing row-column effects in a grid layout and giving an IPS-trained reranker a roughly 2% wNDCG@10 gain.

Reference graph

Works this paper leans on

69 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi

  2. [2]

    Abreu, Arminda Lucia

    Mery Natali Silva. Abreu, Arminda Lucia. Siqueira, Clareci Silva. Cardoso, and Waleska Teixeira. Caiaffa. 2008. Ordinal logistic regression models: application in quality of life studies.Cadernos de saude publica24 Suppl 4 (2008), s581–91

    work page 2008

  3. [3]

    2003.Categorical Data Analysis, Second Edition

    Alan Agresti. 2003.Categorical Data Analysis, Second Edition. Categorical Data Analysis, Second Edition

    work page 2003

  4. [4]

    Azzah Al-Maskari, Mark Sanderson, and Paul Clough. 2007. The relationship between IR effectiveness measures and user satisfaction. InProceedings of the 30th SIGIR Conference. ACM, 773–774

    work page 2007

  5. [5]

    Obead Alhadreti and Pam Mayhew. 2018. Rethinking Thinking Aloud: A Compar- ison of Three Think-Aloud Protocols. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/ 3173574.3173618

    arXiv 2018

  6. [6]

    Allwein, Robert E

    Erin L. Allwein, Robert E. Schapire, and Yoram Singer. 2000. Reducing Mul- ticlass to Binary: A Unifying Approach for Margin Classifiers. InSeventeenth International Conference on Machine Learning

    work page 2000

  7. [7]

    Nicholas J Belkin. 2015. Salton award lecture: People, interacting with information. InProceedings of the 38th International ACM SIGIR Conference on research and development in information retrieval. 1–2

    work page 2015

  8. [8]

    Nicholas J Belkin, Michael Cole, and Jingjing Liu. 2009. A model for evaluation of interactive information retrieval. InProceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation. 7–8

    work page 2009

Show all 69 references
  1. [9]

    2006.Pattern Recognition and Machine Learning

    Christopher Bishop. 2006.Pattern Recognition and Machine Learning. Stat Sci

    2006

  2. [10]

    Yiqun Chen, Kuo Zhou, Yiqun Liu, Min Zhang, and Shaoping Ma. 2017. Meta- evaluation of online and offline web search evaluation metrics. InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM

    2017

  3. [11]

    Jianlin Cheng. 2008. A neural network approach to ordinal regression. In2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence)

    2008

  4. [12]

    Gabriel de Jesus and Sérgio Nunes. 2024. Exploring Large Language Models for Relevance Judgments in Tetun. InLLM4Eval: The First Workshop on Large Language Models for Evaluation in Information Retrieval(Washington DC, United States). https://arxiv.org/abs/2406.07299 arXiv prep...

    2024 arXiv

  5. [13]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. https://arxiv.org/abs/2305.14314. arXiv:2305.14314 [cs.CL]

    2023 arXiv

  6. [14]

    Mouly Dewan, Jiqun Liu, Aditya Gautam, and Chirag Shah. 2025. LLM-Driven Usefulness Judgment for Web Search Evaluation.arXiv preprint arXiv:2504.14401 (2025)

    2025

  7. [15]

    Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. Perspectives on large language models for relevance judgment. InProceedings of the 2023 ACM SIGIR...

    2023

  8. [16]

    Eibe Frank and Mark Hall. 2001. A Simple Approach to Ordinal Classification. Springer-Verlag(2001)

    2001

  9. [17]

    Smucker, and Charles L.A

    Amira Ghenai, Mark D. Smucker, and Charles L.A. Clarke. 2020. A Think- Aloud Study to Understand Factors Affecting Online Health Search. In2020 Conference on Human Information Interaction and Retrieval (CHIIR ’20). ACM, 10. https://doi.org/10.1145/3343413.3377961

    2020

  10. [18]

    Fernández-Navarro, Javier Sánchez-Monedero, and Cesar Martínez

    Pedro Antonio Gutiérrez, María Pérez-Ortiz, F. Fernández-Navarro, Javier Sánchez-Monedero, and Cesar Martínez. 2012. An Experimental Study of Dif- ferent Ordinal Regression Methods and Measures. 296–307. https://doi.org/10. 1007/978-3-642-28931-6_29

    2012

  11. [19]

    Pedro Antonio Gutiérrez, María Pérez-Ortiz, Javier Sánchez-Monedero, Francisco Fernandez-Navarro, and Cesar Martínez. 2015. Ordinal Regression Methods: Survey and Experimental Study.IEEE Transactions on Knowledge and Data Engineering28 (07 2015). https://doi.org/10.1109/TKDE.2...

    2015

  12. [20]

    Ahmed Hassan. 2012. A semi-supervised approach to modeling web search satisfaction. InProceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12). ACM, 275–284

    2012

  13. [21]

    Ahmed Hassan et al . 2010. Beyond DCG: User behavior as a predictor of a successful search. InProceedings of the 3rd ACM International Conference on Web Search and Data Mining (WSDM’10). ACM, 221–230

    2010

  14. [22]

    Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zero-shot rankers for recommender systems. InEuropean Conference on Information Retrieval. Springer, 364–381

    2024

  15. [23]

    Scott B Huffman and Michael Hochster. 2007. How well does result relevance predict session satisfaction?. InProceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 567–574

    2007

  16. [24]

    Scott B Huffman and Michael Hochster. 2007. How well does result relevance predict session satisfaction?. InProceedings of the 30th SIGIR Conference. ACM, 567–574

    2007

  17. [25]

    Huffman and Michael Hochster

    Scott B. Huffman and Michael Hochster. 2007. How well does result relevance predict session satisfaction?. InProceedings of the 30th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 567–574

    2007

  18. [26]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models.arXiv preprint arXiv:2208.032991, 2 (2022), 4. Levera...

    2022 arXiv

  19. [27]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561(2023)

    2023 arXiv

  20. [28]

    Jiepu Jiang, Ahmed Hassan Awadallah, Xiaolin Shi, and Ryen W White. 2015. Understanding and predicting graded search satisfaction. InProceedings of the Eighth ACM International Conference on Web Search and Data Mining. 57–66

    2015

  21. [29]

    2012.Search Engine Effectiveness Evaluation: Methods and Techniques Based on User Click Log Analysis

    He Jing and Li Xiaoming. 2012.Search Engine Effectiveness Evaluation: Methods and Techniques Based on User Click Log Analysis. Search Engine Effectiveness Evaluation: Methods and Techniques Based on User Click Log Analysis

    2012

  22. [30]

    Diane Kelly, Xin Fu, and Chirag Shah. 2007. Effects of rank and precision of search results on users’ evaluations of system performance.University of North Carolina(2007)

    2007

  23. [31]

    Ekaterina Khramtsova, Shengyao Zhuang, Mahsa Baktashmotlagh, and Guido Zuccon. 2024. Leveraging llms for unsupervised dense retriever ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1307–1317

    2024

  24. [32]

    KimKyoung-jae and AhnHyunchul. 2012. A corporate credit rating model us- ing multi-class support vector machines with an ordinal pairwise partitioning approach.Computers and Operations Research(2012)

    2012

  25. [33]

    Mengyang Liu, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. In- vestigating Cognitive Effects in Session-level Search User Satisfaction. InPro- ceedings of the 25th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining (KDD). ACM, Anchorage, AK,...

    2019

  26. [34]

    Sean MacAvaney and Luca Soldaini. 2023. One-shot labeling for automatic rele- vance estimation. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2230–2235

    2023

  27. [35]

    Jiaxin Mao, Yiqun Liu, Huanbo Luan, Min Zhang, and Shaoping Ma. 2017. Un- derstanding and Predicting Usefulness Judgment in Web Search. InProceed- ings of the 40th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval (SIGIR ’17). ACM, New Y...

    2017

  28. [36]

    Jiaxin Mao, Yiqun Liu, Ke Zhou, Jian-Yun Nie, Jingtao Song, Min Zhang, Shaoping Ma, Jiashen Sun, and Hengliang Luo. 2016. When does relevance mean usefulness and user satisfaction in web search?. InProceedings of the 39th International ACM SIGIR conference on Research and Deve...

    2016

  29. [37]

    Peter McCullagh. 2018. Regression Models for Ordinal Data.Journal of the Royal Statistical Society: Series B (Methodological)42, 2 (2018), 109–127

    2018

  30. [38]

    Alistair Moffat, Paul Thomas, and Falk Scholer. 2013. Users versus models: what observation tells us about effectiveness metrics. InProceedings of the 22nd ACM International Conference on Information & Knowledge Management(San Francisco, California, USA)(CIKM ’13). Association...

    2013

  31. [39]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088(2023)

    2023 arXiv

  32. [40]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, et al. 2023. Large language models are effective text rankers with pairwise ranking prompting.arXiv preprint arXiv:2306.17563(2023)

    2023 arXiv

  33. [41]

    Mark Sanderson, Monica Lestari Paramita, Paul Clough, and Evangelos Kanoulas

  34. [42]

    2022.The Notion of Relevance in Information Science: Everybody knows what relevance is

    Tefko Saracevic. 2022.The Notion of Relevance in Information Science: Everybody knows what relevance is. But, what is it really?Springer Nature

    2022

  35. [43]

    Tefko Saracevic, Paul Kantor, Alice Y Chamis, and Donna Trivison. 1988. A study of information seeking and retrieving. I. Background and methodology.Journal of the American Society for Information science39, 3 (1988), 161–176

    1988

  36. [44]

    Schlkopf

    Alex Smola and B. Schlkopf. 2004. A tutorial on support vector regression. Statistics and Computing14 (2004), 199–222

    2004

  37. [45]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? investigat- ing large language models as re-ranking agents.arXiv preprint arXiv:2304.09542 (2023)

    2023 arXiv

  38. [46]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models can Accurately Predict Searcher Preferences. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, USA, 123...

    2024

  39. [47]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    2023 arXiv

  40. [48]

    Maarten Van Someren, Yvonne F Barnard, and J Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive.London: AcademicPress11, 6 (1994)

    1994

  41. [49]

    Willem Waegeman and Luc Boullart. 2009. An ensemble of weighted support vector machines for ordinal regression.Enformatika(2009)

    2009

  42. [50]

    Hao Wang et al. 2014. Modeling action-level satisfaction for search task satisfac- tion prediction. InProceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’14). ACM, 123–132

    2014

  43. [51]

    Jiayin Wang, Fengran Mo, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie

  44. [52]

    Metzler, et al

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dale Schuurmans, Denny Zhou, D. Metzler, et al . 2022. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682(2022)

    2022 arXiv

  45. [53]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, Vol. 35. 24824–24837. https://arxiv.org/abs/2201.11903

    2022 arXiv

  46. [54]

    A User-Centric Benchmark for Evaluating Large Language Models.arXiv preprint arXiv:2404.13940(2024)

    2024 arXiv

  47. [55]

    Hong Wu, Hanqing Lu, and Songde Ma. 2003. A practical SVM-based algorithm for ordinal regression in image retrieval.ACM(2003)

    2003

  48. [56]

    Ya Xu and David Mease. 2009. Evaluating web search using task completion time. InProceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. ACM, 676–677

    2009

  49. [57]

    Witten and Eibe Frank

    Ian H. Witten and Eibe Frank. 2005.Data Mining: Practical Machine Learning Tools and Techniques (Third Edition). Data Mining: Practical Machine Learning Tools and Techniques (Third Edition)

    2005

  50. [58]

    Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, and Jiaxin Mao

  51. [59]

    Fan Zhang, Jiaxin Mao, Yiqun Liu, Xiaohui Xie, Weizhi Ma, Min Zhang, and Shaoping Ma. 2020. Investigating Reference Dependence Effects on User Search Interaction and Satisfaction: A Behavioral Economics Perspective. InProceedings of the 43rd International ACM SIGIR Conference ...

    2020

  52. [60]

    Emine Yilmaz, Manisha Verma, Nick Craswell, Filip Radlinski, and Peter Bailey

  53. [61]

    Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Are Large Language Models Good at Utility Judgments?. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1941–1951

    2024

  54. [62]

    Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu

  55. [63]

    arXiv:2403.09142 [cs.IR] https://arxiv.org/abs/2403.09142

    USimAgent: Large Language Models for Simulating Search Users. arXiv:2403.09142 [cs.IR] https://arxiv.org/abs/2403.09142

    arXiv

  56. [65]

    Fan Zhang, Jiaxin Mao, Yiqun Liu, Xiaohui Xie, Weizhi Ma, Min Zhang, and Shaop- ing Ma. 2020. Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics. InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Informati...

    2020

  57. [69]

    Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2023. Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels.arXiv preprint arXiv:2310.14122(2023)

    2023 arXiv

  58. [2010]

    InProceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

    Do user preferences and evaluation measures line up?. InProceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 555–562

  59. [2014]

    InProceedings of the 23rd ACM international conference on conference on information and knowledge management

    Relevance and effort: An analysis of document utility. InProceedings of the 23rd ACM international conference on conference on information and knowledge management. 91–100

  60. [2019]

    InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    ERNIE: Enhanced Language Representation with Informative Entities. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 1441–1451. https://doi.org/10.18653/v1/P19-1139

    doi

  61. [2024]

    Can We Use Large Language Models to Fill Relevance Judgment Holes? arXiv preprint arXiv:2405.05600(2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.