Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a multimodal LLM, prompted with a four-part structured evaluation, can judge medical article relevance accurately enough to agree with human assessors at Cohen's Kappa 0.6, and uses that judge to expand the…

desk verdict The released resource and prompt recipe are useful, but the 0.6 Kappa was measured on a balanced sample and does not by itself validate the sparse 37x expanded qrels. read the letter →

arxiv 2506.17782 v1 pith:7KJOFKUB submitted 2025-06-21 cs.IR cs.AI

Catarina Pires , Sérgio Nunes , Luís Filipe Teixeira This is my paper · ORCID
classification cs.IRcs.AI
keywords InformationRetrievalRelevanceJudgmentsMultimodalLargeLanguageModelsMLLM-as-a-JudgeMedicalCase-basedImageCLEFmedPromptEngineeringAutomaticTestCollections
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that a multimodal large language model can produce relevance judgments for medical case-based retrieval that are usable in place of costly expert annotation. It targets a specific bottleneck: the ImageCLEFmed 2013 collection has only 15,028 human judgments (4.72% relevant) from pooling, which is too sparse to fairly evaluate dense retrieval systems that surface different documents. The authors build an iterative prompt strategy—system instructions, case presentation, a few-shot relevant example, and the article to judge—and report that Gemini 1.5 Pro agrees with human assessors at Cohen's Kappa 0.6 on a 202-judgment test set. They then apply the judge to expand qrels to 558,653 judgments, covering about 21% of the collection per topic, with 5,950 relevant. If this holds, automated judges become a practical way to densify test collections in multimodal medical IR.

What carries the argument

The central mechanism is an MLLM-as-a-Judge pipeline built from four separated prompts: a system prompt that defines the judge role and relevance criteria (differential diagnosis, clinical evidence, specificity and accuracy), a case prompt providing the patient case text and images, a few-shot prompt with a known relevant article, and an evaluation prompt for the candidate article, with a binary 0/1 output and an instruction to err toward 0 under uncertainty. The separation of prompts and the inclusion of a relevant example are the load-bearing design choices; the paper reports that moving from zero-shot to few-shot and from a single combined prompt to separate prompts raises Kappa from about 0.51 to 0.6. This mechanism converts a small set of human judgments plus a multimodal corpus into a dramatically larger judgment set.

What would settle it

Take a random sample of the previously unjudged articles among the 558,653 expanded judgments, excluding the 202 used for tuning, have expert clinicians label them, and compute Cohen's Kappa between the model and those labels; if agreement on that sample falls well below 0.6—especially for topics whose few-shot example overlapped the test set—the claim that the expansion is as reliable as human judgment is not supported.

Watch

Extended reading notes

Core claim

Using a multimodal LLM as a judge, with binary relevance scoring guided by structured prompts, the paper expands relevance judgments for the ImageCLEFmed 2013 case-based retrieval task by more than 37x while preserving substantial agreement with human assessors. The best prompt setup—separate system, case, example, and evaluation prompts with a few-shot relevant article—reaches Cohen's Kappa 0.6 and about 80% matching judgments on the 202-judgment subset, which the paper interprets as comparable to inter-annotator agreement in this task. Starting from 15,028 manual judgments, the expansion produces 558,653 judgments, raising judged coverage per topic from about 0.57% to 21% and increasing relevant annotations from 709 to 5,950, though the proportion of relevant items falls to 1.07% because most newly judged articles are non-relevant.

Load-bearing premise

The 202 human judgments used to measure agreement are representative of the much larger set of articles the model was then asked to judge.

Editorial extensions

If this is right

  • The expanded qrels allow evaluation of dense and semantic retrieval systems on ImageCLEFmed 2013, which the original sparse pooling cannot fairly support.
  • Automated MLLM judgments can be generated for the long tail of top-ranked candidates, catching relevant documents outside the original pool.
  • The released expanded qrels can serve as a larger test collection and as noisy training data for medical retrieval models.
  • Iterating on prompt structure could push agreement beyond 0.6 and toward expert-level alignment.
  • Per-topic coverage rises from about 0.57% to 21%, giving each query a substantially larger set of judged candidates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 202-judgment test set is balanced while the expanded set is roughly 99% non-relevant, the measured Kappa may not transfer to the full expansion; the paper's own note that 10 of 35 topics had few-shot examples overlapping the test set reinforces this caution.
  • If the expansion contains systematic bias from safety-filtered prompts being marked non-relevant (1,195 instances), evaluations on the expanded qrels could under-count relevant documents.
  • The method could be stress-tested by sampling newly judged articles for expert review and computing per-topic agreement, revealing which case types the MLLM judges well.
  • The same four-prompt judge structure could be applied to other multimodal case-based collections, but the few-shot example selection should be stratified to avoid overlap with the test set.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes using Gemini 1.5 Pro as a multimodal LLM judge to expand relevance judgments for the ImageCLEFmed 2013 case-based retrieval test collection. The authors construct a 202-judgment test subset from existing pool judgments, iterate over prompt configurations, and report Cohen's Kappa 0.6 with roughly 80% matching assessments for their best four-prompt setup. They then apply this setup to judge 558,653 dense-retrieval candidate articles, expanding the qrels from 15,028 to 558,653 judgments, of which 5,950 are marked relevant (1.07%). The paper argues that this demonstrates the potential of MLLMs to scale relevance judgment collection in medical multimodal IR and releases the expanded qrels and source code (Section 4.3).

Significance. If the agreement measured on the test subset transfers to the expanded population, the released resource would be valuable: it raises judged coverage per topic from roughly 0.57% to about 21% and could support evaluation of dense retrieval systems not represented in the original pool. The paper's concrete strengths are the fully specified best-prompt configuration in Appendix A, the public release of data and code, and the use of external human ground truth for the agreement measurement rather than self-consistency alone. The central unresolved question is whether a Kappa measured on a balanced, pool-derived subset supports conclusions about a highly imbalanced, dense-retrieval-derived expansion population.

major comments (3)
  1. [Section 3.2, Tables 1 and 5] The headline Kappa of 0.6 is computed on a test subset deliberately balanced at 97 relevant versus 105 non-relevant judgments, whereas the expanded qrels contain only 1.07% relevant judgments. Cohen's Kappa is not prevalence-invariant, and the same per-class error rates that produce Kappa 0.6 on a balanced sample can yield substantially lower Kappa at 1% prevalence; more importantly, the reported aggregate agreement does not reveal precision on the rare relevant class in the long tail of dense-retrieval candidates. To support the claim that the 37x expansion is usable for evaluation, the authors should report the confusion matrix, sensitivity, and specificity for the relevant class on the test subset, compute expected Kappa under the expansion prevalence, and validate on a random sample of expanded judgments (for example, a few hundred judgments spanning multiple topics) against human assessors. Without this, the transfer of the agreement result to the expansion population is unsubstantiated.
  2. [Section 3.1 and Appendix A.3] For 10 of the 35 topics, the relevant article used as the few-shot example is itself part of the 202-judgment test set. Because the test subset contains at most three relevant articles per topic, this overlap means the prompt provides the human label for a test item, which can inflate the measured agreement. The paper asserts that the effect was small, but it does not quantify the effect. The authors should report Kappa and matching rate separately for the 25 topics without overlap and for the subset of judgments that exclude the overlapping example articles.
  3. [Section 4.2] The paper states that 1,195 judgments were classified as not relevant because the model's safety filter blocked assessment. Treating refusals as non-relevant is a systematic bias, and 1,195 is a substantial fraction of the 5,950 relevant labels in the expansion. The authors should mark these forced labels distinctly in the released qrels and report the main results with and without these 1,195 judgments to show that the expansion's properties are not driven by this decision.
minor comments (5)
  1. [Table 2] Reporting Kappa to four decimal places for an agreement computed on 202 judgments is misleading; please report confidence intervals or at least a standard error.
  2. [Figure 2] The original and expanded relevant counts are plotted on the same axis despite very different scales, which makes the per-topic comparison hard to read; a log scale or two panels would improve clarity.
  3. [Section 3.1] The description says the few-shot example was randomly selected from the ground truth, but no random seed or per-topic selection procedure is described; please specify how the example was chosen and whether the same example was used for all runs.
  4. [Section 3.2] The phrase 'approximately 80% matching assessments' should be made exact by reporting the numerator and denominator (for example, 162 of 202 judgments) alongside the Kappa value.
  5. [Section 5] The self-acknowledged limitation that LLM-generated judgments are constrained by the generating model's capabilities is welcome, but it is a statement of principle, not a substitute for the quantitative validation on the expansion population requested above.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity: few-shot relevant examples are drawn from the same human qrels used to build the test set, with 10/35 topics overlapping; the central expansion is otherwise externally benchmarked.

  1. other [Section 3.1 (Prompt Engineering) and Section 3.2 (Judgment Variance by Prompt Changes); test subset described in Section 3 and Table 1.]
    "Additionally, we experimented with including an example of a medical article relevant to the query topic—sourced from the task’s ground truth relevance judgments (qrels)—by leveraging the few-shot learning technique [2]. ... Consequently, for 10 out of the 35 topics, the pre-selected example overlapped with the test set, potentially introducing some bias in evaluating the MLLM-as-a-Judge."

    The judge's few-shot example is a human-relevant article drawn from the same qrels used to construct the 202-judgment test subset. For 10 of 35 topics the exact example article appears in the test set, so the model's agreement with humans on those items partly measures whether it repeats the label it was shown. Additionally, the prompt configuration was selected by maximizing Kappa on this same test subset, so the final 0.6 is a selected, not a held-out, estimate. This does not make the expansion construction itself circular, but it partially contaminates the headline agreement measure. The paper discloses the overlap but does not quantify its effect, and its assurance that differences were not concentrated in overlapping articles is not a substitute for an uncontaminated test set.

full rationale

The paper's main derivation is not circular: the expansion judgments are produced by Gemini 1.5 Pro and the quality signal (Cohen's Kappa = 0.6) is computed against external ImageCLEFmed 2013 human assessors, not against the model's own outputs. No load-bearing uniqueness theorem or self-citation chain is used; the only self-reference (the released dataset DOI) is a resource pointer. The principal caveats—balanced test subset vs. 1.07% prevalence in the expanded set, force-labeling of safety-filter rejections, and the model-as-judge limitation, including the paper's own citation of Soboroff's critique that LLM labels are constrained by the generating model—are external-validity or bias concerns, not circular reductions. The one concrete circular element is the few-shot relevant example sourced from the same human qrels that define the test set, with explicit overlap on 10/35 topics. That is a disclosed evaluation contamination, warranting a score of 2 rather than 0, but it does not reduce the central claim to its inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The two design choices in free_parameters (test subset balance and the few-shot example) affect the Kappa estimate. The assumptions are domain-level and disclosed.

free parameters (2)
  • Few-shot example article per topic = one relevant article per topic from ground truth qrels
    The choice of a relevant example from the human qrels is a design parameter tuned during prompt iteration (Section 3.1, Section 3.2). For 10 topics the example overlaps the test set, which may inflate measured agreement.
  • Up to 3 relevant and 3 non-relevant articles per topic for the test subset = 200 articles, 202 judgments
    The calibration set balance is chosen by hand (Section 3) and differs drastically from the 1.07% relevant rate in the expanded set.
assumptions (3)
  • domain assumption Relevance in ImageCLEFmed 2013 is appropriately operationalized by the binary criteria in the system prompt (differential diagnosis, clinical evidence, specificity/accuracy).
    The prompt criteria are the authors' paraphrase of the task's relevance definition (Section 3.1, Appendix A.1). If these criteria do not match the original judge instructions, the Kappa estimate is biased. The authors note in Section 5 that they lacked the original task guidelines.
  • domain assumption Cohen's Kappa measured on the 202-judgment subset generalizes to the long tail of articles scored by the MLLM.
    The subset is balanced and pool-derived (Section 3, Table 1); the expansion targets dense-retrieval candidates and is highly imbalanced (Section 4.2). This assumption is not tested and is questioned by the paper's own limitation statement.
  • domain assumption Gemini 1.5 Pro processes images and text well enough that the ground truth relevance agreement is meaningful.
    The entire method rests on the proprietary model's multimodal understanding (Section 3). The paper provides no image-only or text-only ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs." pith.science (2026). https://pith.science/paper/7KJOFKUB

@misc{pith2026250617782,
  author       = {Pith},
  title        = {Pith review of: Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KJOFKUB}},
  note         = {Machine review of arXiv:2506.17782}
}
read the original abstract

Evaluating Information Retrieval (IR) systems relies on high-quality manual relevance judgments (qrels), which are costly and time-consuming to obtain. While pooling reduces the annotation effort, it results in only partially labeled datasets. Large Language Models (LLMs) offer a promising alternative to reducing reliance on manual judgments, particularly in complex domains like medical case-based retrieval, where relevance assessment requires analyzing both textual and visual information. In this work, we explore using a Multimodal Large Language Model (MLLM) to expand relevance judgments, creating a new dataset of automated judgments. Specifically, we employ Gemini 1.5 Pro on the ImageCLEFmed 2013 case-based retrieval task, simulating human assessment through an iteratively refined, structured prompting strategy that integrates binary scoring, instruction-based evaluation, and few-shot learning. We systematically experimented with various prompt configurations to maximize agreement with human judgments. To evaluate agreement between the MLLM and human judgments, we use Cohen's Kappa, achieving a substantial agreement score of 0.6, comparable to inter-annotator agreement typically observed in multimodal retrieval tasks. Starting from the original 15,028 manual judgments (4.72% relevant) across 35 topics, our MLLM-based approach expanded the dataset by over 37x to 558,653 judgments, increasing relevant annotations to 5,950. On average, each medical case query received 15,398 new annotations, with approximately 99% being non-relevant, reflecting the high sparsity typical in this domain. Our results demonstrate the potential of MLLMs to scale relevance judgment collection, offering a promising direction for supporting retrieval evaluation in medical and multimodal IR tasks.

Figures

Figures reproduced from arXiv: 2506.17782 by the authors.

Figure 1
Figure 1. Total number of judgments per topic of after the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Difference between the number of relevant judg [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search

    cs.IR 2026-08 conditional novelty 5.0 of 10

    A production VLM-based relevance-labeling pipeline at Pinterest search produces human-aligned sDCG@K metrics and about a 6× smaller minimum detectable effect in A/B tests.

Reference graph

Works this paper leans on

43 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  3. [3]

    Voorhees

    Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen M. Voorhees. 2007. Bias and the limits of pooling for large collections.Inf. Retr.10, 6 (2007), 491–508. https://doi.org/10.1007/S10791-007-9032-X

    doi:10.1007/s10791-007-9032-x 2007

  4. [4]

    Voorhees

    Chris Buckley and Ellen M. Voorhees. 2004. Retrieval evaluation with incomplete information. InSIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 25-29, 2004, Mark Sanderson, Kalervo Järvelin, James Allan, and Peter Bruza (Eds.). ACM, UK, 25–32. https://do...

    arXiv 2004

  5. [5]

    Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. MLLM-as-a- Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. JMLR.org, Austria, Article 254, ...

    arXiv 2024

  6. [6]

    Qiaosheng Chen, Weiqing Luo, Zixian Huang, Tengteng Lin, Xiaxia Wang, Ahmet Soylu, Basil Ell, Baifan Zhou, Evgeny Kharlamov, and Gong Cheng. 2024. ACOR- DAR 2.0: A Test Collection for Ad Hoc Dataset Retrieval with Densely Pooled Datasets and Question-Style Queries. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in...

    arXiv 2024

  7. [7]

    Cyril Cleverdon. 1967. The Cranfield Tests on Index Language Devices.Aslib Proceedings19, 6 (1967), 173–194. https://doi.org/10.1108/eb050097

    doi:10.1108/eb050097 1967

  8. [8]

    Clough and Mark Sanderson

    Paul D. Clough and Mark Sanderson. 2013. Evaluating the performance of information retrieval systems using test collections.Inf. Res.18, 2 (2013), 1–13. http://www.informationr.net/ir/18-2/paper582.html

    work page 2013

Show all 43 references
  1. [9]

    Clough, Mark Sanderson, and Henning Müller

    Paul D. Clough, Mark Sanderson, and Henning Müller. 2004. The CLEF Cross Language Image Retrieval Track (ImageCLEF) 2004. InImage and Video Retrieval: Medical Case-based Judgments with Multimodal LLMs LLM4Eval ’25, July 17, 2025, Padua, Italy Third International Conference, CI...

    2004 doi

  2. [10]

    Antani, and Henning Müller

    Alba Garcia Seco de Herrera, Jayashree Kalpathy-Cramer, Dina Demner-Fushman, Sameer K. Antani, and Henning Müller. 2013. Overview of the ImageCLEF 2013 Medical Tasks. InWorking Notes for CLEF 2013 Conference , Valencia, Spain, September 23-26, 2013 (CEUR Workshop Proceedings, ...

    2013

  3. [11]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan

  4. [12]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. A Survey on LLM-as-a-Judge.CoRRabs/2411.15594 (2024). https: //doi.org/10.48550/ARXIV.2411.15594 arXiv:2411.15594

    2024 doi arXiv

  5. [13]

    Storås, Asma Ben Abacha, Christoph M

    Bogdan Ionescu, Henning Müller, Ana-Maria Claudia Dragulinescu, Adrian Popescu, Ahmad Idrissi-Yaghir, Alba García Seco de Herrera, Alexandra Andrei, Alexandru Stan, Andrea M. Storås, Asma Ben Abacha, Christoph M. Friedrich, George Ioannidis, Griffin Adams, Henning Schäfer, Hug...

    2023

  6. [14]

    Jeong, Zachary C

    Daniel P. Jeong, Zachary C. Lipton, and Pradeep Ravikumar. 2024. LLM-Select: Feature Selection with Large Language Models.CoRRabs/2407.02694 (2024). https://doi.org/10.48550/ARXIV.2407.02694 arXiv:2407.02694

    2024 doi arXiv

  7. [15]

    Jones, C.J

    K.S. Jones, C.J. Van Rijsbergen, British Library. Research, and Development Department. 1975.Report on the Need for and Provision of an ’ideal’ Information Retrieval Test Collection. University Computer Laboratory, UK

    1975

  8. [16]

    Antani, Steven Bedrick, and Henning Müller

    Jayashree Kalpathy-Cramer, Alba Garcia Seco de Herrera, Dina Demner-Fushman, Sameer K. Antani, Steven Bedrick, and Henning Müller. 2015. Evaluating per- formance of biomedical image retrieval systems - An overview of the medical image retrieval task at ImageCLEF 2004-2013.Comp...

    2015 doi

  9. [17]

    Jayashree Kalpathy-Cramer, Henning Müller, Steven Bedrick, Ivan Eggel, Alba Garcia Seco de Herrera, and Theodora Tsikrika. 2011. Overview of the CLEF 2011 Medical Image Classification and Retrieval Tasks. InCLEF 2011 Labs and Workshop, Notebook Papers, 19-22 September 2011, Am...

    2011

  10. [18]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2024. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge.CoRRabs/2411.1659...

    2024 doi

  11. [19]

    Ruosen Li, Teerth Patel, and Xinya Du. 2024. PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations.Trans. Mach. Learn. Res.2024 (2024). https://openreview.net/forum?id=YVD1QqWRaj

    2024

  12. [20]

    Clough, Thomas Deselaers, and Barbara Caputo (Eds.)

    Henning Müller, Paul D. Clough, Thomas Deselaers, and Barbara Caputo (Eds.). 2010.ImageCLEF, Experimental Evaluation in Visual Information Retrieval. Springer, Berlin. https://doi.org/10.1007/978-3-642-15181-1

    2010 doi

  13. [21]

    Clough, William R

    Henning Müller, Paul D. Clough, William R. Hersh, and Antoine Geissbühler. 2006. Variation of Relevance Assessments for Medical Image Retrieval. InAdaptive Multimedia Retrieval: User, Context, and Feedback, 4th International Workshop, AMR 2006, Geneva, Switzerland, July 27-28,...

    2006

  14. [22]

    Henning Müller, Thomas Deselaers, Thomas Martin Deserno, Jayashree Kalpathy- Cramer, Eugene Kim, and William R. Hersh. 2007. Overview of the Image- CLEFmed 2007 Medical Retrieval and Medical Annotation Tasks. InAdvances in Multilingual and Multimodal Information Retrieval, 8th...

    2007

  15. [23]

    Kahn Jr., and William R

    Henning Müller, Jayashree Kalpathy-Cramer, Ivan Eggel, Steven Bedrick, Saïd Radhouani, Brian Bakke, Charles E. Kahn Jr., and William R. Hersh. 2009. Overview of the CLEF 2009 Medical Image Retrieval Track. InMultilingual Infor- mation Access Evaluation II. Multimedia Experimen...

    2009

  16. [24]

    Kahn Jr., and William R

    Henning Müller, Jayashree Kalpathy-Cramer, Ivan Eggel, Steven Bedrick, Joe Reisetter, Charles E. Kahn Jr., and William R. Hersh. 2010. Overview of the CLEF 2010 Medical Image Retrieval Track. InCLEF 2010 LABs and Workshops, Notebook Papers, 22-23 September 2010, Padua, Italy (...

    2010

  17. [25]

    Catarina Pires, Sérgio Nunes, and Luís Filipe Teixeira. 2025. ImageCLEFmed 2013 Case-based Retrieval Task Relevance Judgments Expansion via an MLLM- as-a-Judge Approach. https://doi.org/10.5281/zenodo.14866103 Dataset, INESC TEC

    2025 doi

  18. [26]

    Filip Radlinski and Nick Craswell. 2010. Comparing the sensitivity of information retrieval metrics. InProceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, Geneva, Switzer- land, July 19-23, 2010, Fabio Cre...

    2010

  19. [27]

    Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, An- drew M

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, An- drew M. Dai, Katie Millican, Ethan Dyer, M...

    2024 doi arXiv

  20. [28]

    Antoine Rosset, Henning Muller, Martina Martins, Natalia Dfouni, Jean-Paul Vallée, and Osman Ratib. 2004. Casimage Project: A Digital Teaching Files Authoring Environment.Journal of Thoracic Imaging19, 2 (2004), 103–108. https://doi.org/10.1097/00005382-200404000-00008

    2004 doi

  21. [29]

    Shreya Shankar, J. D. Zamfirescu-Pereira, Bjoern Hartmann, Aditya G. Parameswaran, and Ian Arawjo. 2024. Who Validates the Validators? Align- ing LLM-Assisted Evaluation of LLM Outputs with Human Preferences. InPro- ceedings of the 37th Annual ACM Symposium on User Interface S...

    2024

  22. [30]

    Yanxin Shen, Lun Wang, Chuanqi Shi, Shaoshuai Du, Yiyi Tao, Yixian Shen, and Hang Zhang. 2024. Comparative Analysis of Listwise Reranking with Large Language Models in Limited-Resource Language Contexts.CoRRabs/2412.20061 (2024). https://doi.org/10.48550/ARXIV.2412.20061 arXiv...

    2024 doi arXiv

  23. [31]

    Ian Soboroff. 2025. Don’t Use LLMs to Make Relevance Judgments.Information Retrieval Research1, 1 (Mar. 2025), 29–46. https://doi.org/10.54195/irrj.19625

    2025 doi

  24. [32]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models can Accurately Predict Searcher Preferences. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, ...

    2024

  25. [33]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro...

    2023

  26. [34]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Proces...

    2023

  27. [39]

    Carefully read the textual description of the patient case and examine the associated images (if available)

  28. [40]

    Review the provided example of a relevant article to understand the standard for relevance

  29. [41]

    Review the content of the retrieved article

  30. [42]

    Assign a relevance score (1 or 0) based on the criteria above

  31. [43]

    Focus on providing objective, evidence-based judgments that would genuinely assist a clinician in diagnosing and managing this patient

    If you are uncertain about the article’s relevance, err on the side of caution and assign a score of 0. Focus on providing objective, evidence-based judgments that would genuinely assist a clinician in diagnosing and managing this patient. Wait for the presentation of the pati...

    2025

  32. [246]

    https://doi.org/10.1007/978-3-540-71545-0_18

    doi

  33. [491]

    https://doi.org/10.1007/978-3-540-85760-0_59

    doi

  34. [2023]

    https://doi.org/10.48550/ARXIV.2304.02554 arXiv:2304.02554

    Human-like Summarization Evaluation with ChatGPT.CoRRabs/2304.02554 (2023). https://doi.org/10.48550/ARXIV.2304.02554 arXiv:2304.02554

    2023 doi arXiv

  35. [2024]

    Can We Use Large Language Models to Fill Relevance Judgment Holes?. In Joint Proceedings of the 1st Workshop on Evaluation Methodologies,Testbeds and Community for Information Access Research (EMTCIR 2024) and the 1st Workshop on User Modelling in Conversational Information Re...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.