{"id":"f99cad9e-27cc-485a-9b55-ac3cff7b1ad9","arxiv_id":"2506.17782","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Gemini 1.5 Pro, prompted with instructions and a relevant example, generated 558,653 relevance judgments for ImageCLEFmed 2013, expanding the manual qrels by 37x with Cohen's Kappa 0.6 against human labels.","lead":"This paper uses Google's Gemini 1.5 Pro, a multimodal AI model, to automatically judge whether medical articles are relevant to patient cases, expanding a small manual dataset of 15,028 judgments to 558,653. It reports 60% agreement with human judges and releases the expanded judgments and code, so the work is mainly useful as a demonstration that AI-generated relevance labels can scale up evaluation data for medical image retrieval.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Kappa 0.6 is measured on a balanced, pool-derived 202-judgment subset; the expanded qrels are 1% relevant and dense-retrieval-derived, so the agreement may not transfer.","rationale":"The paper is a genuinely useful empirical contribution: it releases code and data with a DOI, documents prompt iterations transparently, and the claimed Kappa of 0.6 is modest and plausible. The reader's conditional verdict is appropriate. My stress-test identifies the same load-bearing assumption: the 202-judgment balanced subset is the only evidence of agreement, and it is drawn from the original pool, while the expanded qrels are generated on dense-retrieval candidates with 1% prevalence. Because Kappa is prevalence-sensitive and no per-class error rates or second-generation human checks are reported, the central generalization from test subset to expansion population is unverified. This is not a reason to reject the paper, but it is a reason to keep the verdict conditional and require the concrete validation described above before the expanded qrels are used in downstream evaluation.","tokens_in":14112,"tokens_out":3559,"duration_ms":35518,"concrete_test":"Sample approximately 300 documents from the expanded qrels (200 stratified random relevant labels plus 100 random non-relevant labels, per-topic), have one or two clinicians re-judge them using the ImageCLEFmed 2013 assessment guidelines, and compare to the MLLM labels using per-class precision/recall and Cohen's Kappa, with a sensitivity analysis excluding the 10 topics whose few-shot example overlapped the test set. If Kappa on this expansion sample falls below 0.4 or recall of the relevant class falls below 0.6, the 37x expansion cannot be considered validated at the same quality as the test subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence is Cohen's Kappa = 0.6 on a 202-judgment test subset (Section 3.2, Table 1). That subset was deliberately constructed from existing pool judgments with up to three relevant and three non-relevant per topic (97 relevant / 105 non-relevant), a balanced distribution. The expansion population (Table 5) is 558,653 judgments with 1.07% relevant, produced by the same Gemini model on dense-retrieval candidates that were not in the original pool. Cohen's Kappa is not prevalence-invariant: the same per-class error rates yield a much lower Kappa on a 1% baseline prevalence than on a balanced sample, and a high Kappa on balanced data does not imply high precision on the rare relevant class in the long tail. The paper reports only overall agreement (~80%) and Kappa, not sensitivity and specificity for the relevant class, so the transfer claim (that the expanded qrels are usable at 37x scale) rests on an unmeasured distributional assumption. Two secondary issues compound this: (i) for 10 of 35 topics the few-shot example overlaps the test set (Section 3.2), which can inflate measured agreement, and the authors' assurance that the overlap had little effect is not quantified; (ii) 1,195 judgments were force-labeled non-relevant after safety-filter rejection (Section 4.2), a systematic bias. The self-acknowledged limitation in Section 5 that LLM-generated judgments are constrained by the model's own capabilities reinforces rather than resolves the need for external validation of the expanded set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using Gemini 1.5 Pro as a multimodal LLM judge to expand relevance judgments for the ImageCLEFmed 2013 case-based retrieval test collection. The authors construct a 202-judgment test subset from existing pool judgments, iterate over prompt configurations, and report Cohen's Kappa 0.6 with roughly 80% matching assessments for their best four-prompt setup. They then apply this setup to judge 558,653 dense-retrieval candidate articles, expanding the qrels from 15,028 to 558,653 judgments, of which 5,950 are marked relevant (1.07%). The paper argues that this demonstrates the potential of MLLMs to scale relevance judgment collection in medical multimodal IR and releases the expanded qrels and source code (Section 4.3).","tokens_in":14497,"tokens_out":4071,"duration_ms":41231,"significance":"If the agreement measured on the test subset transfers to the expanded population, the released resource would be valuable: it raises judged coverage per topic from roughly 0.57% to about 21% and could support evaluation of dense retrieval systems not represented in the original pool. The paper's concrete strengths are the fully specified best-prompt configuration in Appendix A, the public release of data and code, and the use of external human ground truth for the agreement measurement rather than self-consistency alone. The central unresolved question is whether a Kappa measured on a balanced, pool-derived subset supports conclusions about a highly imbalanced, dense-retrieval-derived expansion population.","major_comments":[{"comment":"The headline Kappa of 0.6 is computed on a test subset deliberately balanced at 97 relevant versus 105 non-relevant judgments, whereas the expanded qrels contain only 1.07% relevant judgments. Cohen's Kappa is not prevalence-invariant, and the same per-class error rates that produce Kappa 0.6 on a balanced sample can yield substantially lower Kappa at 1% prevalence; more importantly, the reported aggregate agreement does not reveal precision on the rare relevant class in the long tail of dense-retrieval candidates. To support the claim that the 37x expansion is usable for evaluation, the authors should report the confusion matrix, sensitivity, and specificity for the relevant class on the test subset, compute expected Kappa under the expansion prevalence, and validate on a random sample of expanded judgments (for example, a few hundred judgments spanning multiple topics) against human assessors. Without this, the transfer of the agreement result to the expansion population is unsubstantiated.","section":"Section 3.2, Tables 1 and 5"},{"comment":"For 10 of the 35 topics, the relevant article used as the few-shot example is itself part of the 202-judgment test set. Because the test subset contains at most three relevant articles per topic, this overlap means the prompt provides the human label for a test item, which can inflate the measured agreement. The paper asserts that the effect was small, but it does not quantify the effect. The authors should report Kappa and matching rate separately for the 25 topics without overlap and for the subset of judgments that exclude the overlapping example articles.","section":"Section 3.1 and Appendix A.3"},{"comment":"The paper states that 1,195 judgments were classified as not relevant because the model's safety filter blocked assessment. Treating refusals as non-relevant is a systematic bias, and 1,195 is a substantial fraction of the 5,950 relevant labels in the expansion. The authors should mark these forced labels distinctly in the released qrels and report the main results with and without these 1,195 judgments to show that the expansion's properties are not driven by this decision.","section":"Section 4.2"}],"minor_comments":[{"comment":"Reporting Kappa to four decimal places for an agreement computed on 202 judgments is misleading; please report confidence intervals or at least a standard error.","section":"Table 2"},{"comment":"The original and expanded relevant counts are plotted on the same axis despite very different scales, which makes the per-topic comparison hard to read; a log scale or two panels would improve clarity.","section":"Figure 2"},{"comment":"The description says the few-shot example was randomly selected from the ground truth, but no random seed or per-topic selection procedure is described; please specify how the example was chosen and whether the same example was used for all runs.","section":"Section 3.1"},{"comment":"The phrase 'approximately 80% matching assessments' should be made exact by reporting the numerator and denominator (for example, 162 of 202 judgments) alongside the Kappa value.","section":"Section 3.2"},{"comment":"The self-acknowledged limitation that LLM-generated judgments are constrained by the generating model's capabilities is welcome, but it is a statement of principle, not a substitute for the quantitative validation on the expansion population requested above.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, honestly reported workshop paper that ships a genuinely useful artifact — 558,653 expanded relevance judgments for ImageCLEFmed 2013, with code and prompts — but the headline agreement number is measured on a sample that does not look like the expanded set. The 0.6 Cohen's Kappa is real, but it was computed on a deliberately balanced 202-judgment subset (97 relevant, 105 not), while the expanded qrels are 98.9% non-relevant. Kappa is prevalence-sensitive; a model that is good at separating classes on a balanced sample can still have poor precision on the rare relevant class in the long tail. The paper never reports sensitivity or specificity for the relevant class, so the transfer claim is unmeasured. That is the load-bearing soft spot, and I agree with the stress-test note on this point.\n\nWhat the paper does well: the prompt engineering is systematic and clearly described; the authors compare four prompt variants and show a sensible progression to Kappa 0.6 with ~80% agreement. They release the data and code, which is exactly what makes this reproducible. They also disclose the two issues that most authors would hide: for 10 of 35 topics the few-shot example overlaps the test set, and 1,195 judgments were force-labeled non-relevant after safety-filter rejection. The Section 5 acknowledgment that LLM-generated labels are constrained by the generating model is honest, though it also undercuts the claim that the expanded set is a reliable evaluation resource without further checks.\n\nThe few-shot overlap is a real but minor inflation risk: the authors say most differences were not in overlapping articles, but they do not quantify it. The safety-filter labeling is more concerning because it introduces a systematic bias — those articles are not actually judged, and calling them non-relevant is an assumption that could hide relevant material. Both issues are disclosed, but neither is corrected for in the released resource.\n\nWho is this for? People working on LLM-as-a-judge or on evaluating dense retrieval over ImageCLEFmed 2013. The expanded qrels are not gold-standard labels, but they are a plausible noisy resource for system ranking comparisons, and the prompting recipe is a useful template for multimodal relevance assessment. A serious referee should engage with this: the artifact is citable, the experiments are clear, and the main weakness is a missing validation study, not a fatal flaw. I would like to see a follow-up that samples judgments from the expanded population, has clinicians judge a few hundred of them, and reports per-class metrics and per-topic Kappa excluding the overlapping few-shot topics.","headline":"The released resource and prompt recipe are useful, but the 0.6 Kappa was measured on a balanced sample and does not by itself validate the sparse 37x expanded qrels.","tokens_in":15032,"tokens_out":1635,"would_cite":true,"duration_ms":19825,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a multimodal LLM, prompted with a four-part structured evaluation, can judge medical article relevance accurately enough to agree with human assessors at Cohen's Kappa 0.6, and uses that judge to expand the…","keywords":["Information Retrieval","Relevance Judgments","Multimodal Large Language Models","MLLM-as-a-Judge","Medical Case-based Retrieval","ImageCLEFmed","Prompt Engineering","Automatic Test Collections"],"falsifier":"Take a random sample of the previously unjudged articles among the 558,653 expanded judgments, excluding the 202 used for tuning, have expert clinicians label them, and compute Cohen's Kappa between the model and those labels; if agreement on that sample falls well below 0.6—especially for topics whose few-shot example overlapped the test set—the claim that the expansion is as reliable as human judgment is not supported.","tokens_in":13834,"feed_emoji":"🩺","tokens_out":5689,"duration_ms":52209,"temperature":0.7,"pith_summary":"The paper is trying to show that a multimodal large language model can produce relevance judgments for medical case-based retrieval that are usable in place of costly expert annotation. It targets a specific bottleneck: the ImageCLEFmed 2013 collection has only 15,028 human judgments (4.72% relevant) from pooling, which is too sparse to fairly evaluate dense retrieval systems that surface different documents. The authors build an iterative prompt strategy—system instructions, case presentation, a few-shot relevant example, and the article to judge—and report that Gemini 1.5 Pro agrees with human assessors at Cohen's Kappa 0.6 on a 202-judgment test set. They then apply the judge to expand qrels to 558,653 judgments, covering about 21% of the collection per topic, with 5,950 relevant. If this holds, automated judges become a practical way to densify test collections in multimodal medical IR.","feed_headline":"AI judge matches human agreement and expands medical qrels 37-fold","feed_subtitle":"A four-prompt setup reaches Kappa 0.6 and grows 15,028 judgments to 558,653.","key_machinery":"The central mechanism is an MLLM-as-a-Judge pipeline built from four separated prompts: a system prompt that defines the judge role and relevance criteria (differential diagnosis, clinical evidence, specificity and accuracy), a case prompt providing the patient case text and images, a few-shot prompt with a known relevant article, and an evaluation prompt for the candidate article, with a binary 0/1 output and an instruction to err toward 0 under uncertainty. The separation of prompts and the inclusion of a relevant example are the load-bearing design choices; the paper reports that moving from zero-shot to few-shot and from a single combined prompt to separate prompts raises Kappa from about 0.51 to 0.6. This mechanism converts a small set of human judgments plus a multimodal corpus into a dramatically larger judgment set.","core_discovery":"Using a multimodal LLM as a judge, with binary relevance scoring guided by structured prompts, the paper expands relevance judgments for the ImageCLEFmed 2013 case-based retrieval task by more than 37x while preserving substantial agreement with human assessors. The best prompt setup—separate system, case, example, and evaluation prompts with a few-shot relevant article—reaches Cohen's Kappa 0.6 and about 80% matching judgments on the 202-judgment subset, which the paper interprets as comparable to inter-annotator agreement in this task. Starting from 15,028 manual judgments, the expansion produces 558,653 judgments, raising judged coverage per topic from about 0.57% to 21% and increasing relevant annotations from 709 to 5,950, though the proportion of relevant items falls to 1.07% because most newly judged articles are non-relevant.","pith_inferences":["Because the 202-judgment test set is balanced while the expanded set is roughly 99% non-relevant, the measured Kappa may not transfer to the full expansion; the paper's own note that 10 of 35 topics had few-shot examples overlapping the test set reinforces this caution.","If the expansion contains systematic bias from safety-filtered prompts being marked non-relevant (1,195 instances), evaluations on the expanded qrels could under-count relevant documents.","The method could be stress-tested by sampling newly judged articles for expert review and computing per-topic agreement, revealing which case types the MLLM judges well.","The same four-prompt judge structure could be applied to other multimodal case-based collections, but the few-shot example selection should be stratified to avoid overlap with the test set."],"forward_implications":["The expanded qrels allow evaluation of dense and semantic retrieval systems on ImageCLEFmed 2013, which the original sparse pooling cannot fairly support.","Automated MLLM judgments can be generated for the long tail of top-ranked candidates, catching relevant documents outside the original pool.","The released expanded qrels can serve as a larger test collection and as noisy training data for medical retrieval models.","Iterating on prompt structure could push agreement beyond 0.6 and toward expert-level alignment.","Per-topic coverage rises from about 0.57% to 21%, giving each query a substantially larger set of judged candidates."],"supporting_citations":[{"why":"Supplies the ImageCLEFmed 2013 case-based retrieval task, the human qrels, and the evaluation setup that the paper expands.","marker":"[10]"},{"why":"Provides the Gemini 1.5 Pro model used as the multimodal judge.","marker":"[27]"},{"why":"Introduces few-shot learning, the technique used to give the judge a relevant example.","marker":"[2]"},{"why":"Frames MLLM-as-a-Judge evaluation for multimodal outputs, which the paper applies to medical retrieval.","marker":"[5]"},{"why":"Documents the limits and biases of pooling that motivate expanding the qrels.","marker":"[3]"},{"why":"Shows how incomplete judgments bias retrieval evaluation, the core problem the expansion addresses.","marker":"[4]"},{"why":"Prior work on using LLMs to fill relevance judgment holes, which the paper extends to multimodal data.","marker":"[1]"},{"why":"Cautions that LLM-generated labels are bounded by the generating model's capabilities, a limitation the paper acknowledges.","marker":"[31]"}],"fun_headline_variants":["Multimodal LLM judges medical cases, expands qrels 37x","AI judge agrees with humans, scales medical qrels 37-fold","LLM expands medical relevance judgments 37x with Kappa 0.6","Gemini 1.5 Pro: AI judge for medical retrieval, 37x more qrels","MLLM-based qrels expansion: 37x more judgments, Kappa 0.6"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 202 human judgments used to measure agreement are representative of the much larger set of articles the model was then asked to judge.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal LLM judges medical cases, expands qrels 37x","AI judge agrees with humans, scales medical qrels 37-fold","LLM expands medical relevance judgments 37x with Kappa 0.6","Gemini 1.5 Pro: AI judge for medical retrieval, 37x more qrels","MLLM-based qrels expansion: 37x more judgments, Kappa 0.6"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001077,"raw_usage":{"total_tokens":4559,"prompt_tokens":1047,"completion_tokens":3512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":3402}},"tokens_in":663,"tokens_out":3512,"duration_ms":23159,"temperature":1.0,"reasoning_tokens":3402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:59:54.286650+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the previously unjudged articles among the 558,653 expanded judgments, excluding the 202 used for tuning, have expert clinicians label them, and compute Cohen's Kappa between the model and those labels; if agreement on that sample falls well below 0.6—especially for topics whose few-shot example overlapped the test set—the claim that the expansion is as reliable as human judgment is not supported.","supporting_citations":[{"cited_title":"Antani, and Henning Müller","cited_arxiv_id":null,"evidence_quote":"Supplies the ImageCLEFmed 2013 case-based retrieval task, the human qrels, and the evaluation setup that the paper expands."}],"review_version":1}