{"id":"626e555e-f319-4b3b-b43b-b019df9cbc19","arxiv_id":"2501.16481","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CuKPL converts medical images into text descriptions of expert-defined features and asks GPT-4o to classify them as seizure onset zone or not, claiming zero-shot performance that is actually worse than supervised deep learning on cross-hospital data.","lead":"Researchers describe a method (CuKPL) that uses an LLM to classify rare seizure-onset-zone brain images from text descriptions, combining expert knowledge rules with a large language model instead of training a neural network. The method claims zero-shot privacy-preserving classification but requires tuning on one patient's data and does not beat a supervised deep learning baseline on an unseen hospital's data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Table 1 shows CuKPL underperforms DeepXSOZ on cross-center Center B (70.0% vs 90.3% acc; 82.3 vs 94.9 F1), directly contradicting the abstract's unqualified 'outperforming SOTA' claim.","rationale":"Reading in good faith, the paper proposes a coherent pipeline: encode domain knowledge as textual rules, extract image features as text via a knowledge model, and let an LLM classify. The strongest claim in the abstract, however, is empirical: the method 'outperforming state-of-the-art techniques' in a zero-shot, training-free setting. That claim has two conditions: (1) no target-domain data are used for adaptation, and (2) performance exceeds SOTA. Both conditions fail under the paper's own evidence. Condition (2) fails on Center B: DeepXSOZ, the SOTA knowledge+DL baseline, scores 90.3% accuracy and 94.9% F1 versus CuKPL's 70.0% and 82.3%, and the paper concedes that knowledge+DL was 'most effective across evaluations.' Condition (1) fails because the method requires one patient's data to tune thresholds in the knowledge model, and the paper admits it is better suited for domain adaptation than generalization. The reader's verdict of REJECT is therefore well-founded, so I see no need to change it. The comparison on Center B is arguably unfair if CuKPL had access to a Center B patient while the baselines did not, making the central claim even less secure. No machine-checked proofs or exact reproducibility artifacts are provided to offset these issues; the GitHub link lacks a commit hash and full configuration details. In short, the paper's own table contradicts its headline claim, so rejection remains appropriate.","tokens_in":6522,"tokens_out":4794,"duration_ms":42184,"concrete_test":"Run the evaluation protocol from the released repository with explicit logging of what data is used for HKETP threshold tuning. Specifically: (1) determine whether the Center B evaluation used a Center B patient to set thresholds and whether that patient was excluded from the 31-patient test set; (2) verify that DeepXSOZ and other baselines received no Center B data; (3) re-run the cross-center experiment with zero Center B tuning data. If CuKPL's accuracy/F1 fall below the reported 70.0%/82.3%, or remain below DeepXSOZ's 90.3%/94.9%, the abstract's 'outperforming SOTA' claim fails. Also check whether the leave-one-out protocol on Center A used the same single tuning patient in every fold or a different one.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CuKPL is a zero-shot, training-free method that outperforms state-of-the-art techniques. For that claim to hold, CuKPL must beat SOTA not only on the development center but also in the cross-center evaluation that motivates the method. Table 1 (bottom half) shows the opposite: on unseen Center B data CuKPL achieves 70.0% accuracy and 82.3% F1, while DeepXSOZ—the stated SOTA—achieves 90.3% accuracy and 94.9% F1. The paper itself concludes that 'integration of knowledge and DL proved most effective across evaluations,' conceding that the SOTA combination beats CuKPL in the generalization setting. The unqualified abstract statement 'outperforming state-of-the-art techniques' is therefore contradicted by the paper's own numbers. The zero-shot framing is additionally undercut by the method's explicit need for 'one patient's data' to tune knowledge-model thresholds (Sections 3 and 4); the authors admit it is 'more effective for domain adaptation than for domain generalization.' Together these failures mean the strongest claim, as stated, is not supported by the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CuKPL, a method that uses an LLM to convert domain knowledge from literature into structured prompts (HKETP) for zero-shot medical image classification, with a knowledge model encoding image features into text. It is evaluated on seizure onset zone (SOZ) detection from fMRI independent components, comparing with deep learning, knowledge-based, hybrid, and standard zero-shot baselines on Center A and cross-center Center B. The authors report that CuKPL improves accuracy/F1 over DeepXSOZ on Center A but underperforms DeepXSOZ on Center B, and they position the method as zero-shot and privacy-preserving.","tokens_in":6799,"tokens_out":5267,"duration_ms":45508,"significance":"The underlying idea—packaging expert knowledge into textual prompts so that an LLM can classify rare medical images without exposing the image—is attractive and, if rigorously validated, would be a useful privacy-preserving alternative. The paper makes a falsifiable claim ('outperforming state-of-the-art'), and the reported cross-center numbers directly test it. However, in its current form the evidence fails to support the central claim: the method is not zero-shot because thresholds are tuned on one patient's data, and it does not outperform the SOTA on the truly challenging cross-center setting. The code release and detailed dataset description are positives.","major_comments":[{"comment":"The method is not zero-shot as claimed. The paper states 'a human verifier provides feedback and updates, based on one patient's data, to the LLM' and 'It just needs one patient's data during HKETP generation process to tune knowledge model.' This is a calibration/few-shot step, not zero-shot. Moreover, the manuscript does not identify which patient is used or whether that patient belongs to Center A or Center B, so test-set leakage cannot be ruled out. Please specify the tuning procedure, the patient used, and its relationship to the evaluation sets.","section":"Section 3 (HKETP Generation) and Section 4 (Evaluation)"},{"comment":"The abstract claims 'outperforming state-of-the-art techniques' but Table 1 (Center B rows) shows CuKPL achieves 70.0% accuracy and 82.3% F1 versus DeepXSOZ's 90.3% accuracy and 94.9% F1. This contradicts the unqualified claim. Even the paper's own text says 'The integration of knowledge and DL proved most effective across evaluations.' The claim must be qualified to the development-center (Center A) setting or removed.","section":"Abstract and Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any performance metric. With only 316 SOZ images in Center A and 202 in Center B, the reported differences (e.g., 88.4% vs 84.6% accuracy) may be within sampling noise. Please report per-patient variability, standard deviations, or paired statistical tests.","section":"Table 1 and Section 4 (Evaluation)"},{"comment":"The baselines for DL-2D CNN, ViT, EPIK, and DeepXSOZ are not described as re-implemented or re-run on the same data splits. The reader cannot tell whether the comparison is fair (same preprocessing, same independent components, same LOOCV folds, same Center B split). Please state which numbers are reproduced and which are taken from prior publications, and provide the experimental setup for all baselines.","section":"Table 1 and Section 4 (Evaluation)"},{"comment":"No ablations are reported. The contributions of the human verifier, LLM refinement, knowledge model thresholds, and instruction format are never separately evaluated. As a result, the reported CuKPL performance cannot be attributed to the proposed prompt-generation mechanism rather than to the threshold rules inherited from prior work (e.g., Reference [15]).","section":"Section 3 and Table 1"}],"minor_comments":[{"comment":"The method name is written inconsistently: 'CuPKL' appears in Table 1 while 'CuKPL' is used elsewhere; please standardize.","section":"Table 1 and throughout"},{"comment":"The sentence 'CuKPL, on the other hand, which achieved a 3.8% accuracy and 2.2% F1 score improvement on Center A data over SOTA DeepXSOZ' is a sentence fragment and should be revised.","section":"Section 4 (Results Analysis)"},{"comment":"The typo 'Assymetry' should be 'Asymmetry'.","section":"Section 3.1"},{"comment":"The clinical workload-reduction claim based on 'MM SOZs of 28' is not tied to the precision/recall trade-off; clarify how this number is computed and what threshold it corresponds to.","section":"Section 4 (Evaluation)"},{"comment":"The term 'single-domain generalization' is unusual; consider using 'cross-center evaluation' or 'cross-center generalization' for clarity.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"I recommend rejection rather than major revision because the central claims as stated are not supported by the paper's own table, and the one-patient tuning step undermines the zero-shot framing. The issues are not purely presentational; correcting them would require either new experiments or a substantial reframing of the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: if you cite this paper, cite the method, not the abstract. The abstract's unqualified \"outperforming state-of-the-art techniques\" is not supported by the paper's own Table 1. On cross-center Center B, CuKPL gets 70.0% accuracy and 82.3% F1, while DeepXSOZ gets 90.3% and 94.9%. The paper itself concedes in Section 4 that \"integration of knowledge and DL proved most effective across evaluations\" and that CuKPL is \"more effective for domain adaptation than for domain generalization.\" So the headline claim is false as stated.\n\nWhat is actually new: the combination of LLM-generated prompts that embed explicit knowledge rules (e.g., \"1 big red activation,\" asymmetry, gray-matter overlap) and using the LLM itself as the classifier, rather than CLIP-style similarity matching. That is a genuine variation on CuPL, and converting literature-derived rules into structured text for an LLM to execute is a reasonable idea. The evaluation on two fMRI centers with real patient data is valuable, and the method is privacy-preserving in the sense that the image is never sent to the LLM—only numeric features. On the development center, CuKPL improves on DeepXSOZ by a few points, which is credible.\n\nSoft spots, in order of seriousness:\n\n1. The zero-shot claim is undercut by the one-patient tuning step. Section 3 describes human feedback \"based on one patient's data\" to update thresholds in the knowledge components. That is not zero-shot in the standard sense; it is one-shot adaptation. The paper discloses it, but the abstract and the repeated \"zero-shot\" framing are misleading.\n\n2. The cross-center gap is large, and the paper's own numbers show the supervised SOTA combination beats CuKPL. The authors acknowledge this, so the body is honest, but the abstract must be rewritten.\n\n3. No error bars, no ablations, and the baseline numbers come from other papers. The code release lacks a commit hash and configuration details. Minor for a short paper, but they matter for a fuller venue.\n\n4. The knowledge rules come from the authors' prior work, so the novelty is in the prompt engineering, not the knowledge itself.\n\nOverall, this is a modest but legitimate extension with a real clinical target and honest self-assessment in the body. The main failure is the abstract. A serious referee could help the authors fix the framing and add missing details. I would send it to review, expecting heavy revision. I would not cite it in my own work as is.\n\nBest.","headline":"The abstract overclaims: the method is a modest but real extension of CuPL with domain knowledge, and the paper's own cross-center numbers contradict the SOTA claim, yet the body is honest and the work deserves a serious referee.","tokens_in":7313,"tokens_out":2023,"would_cite":false,"duration_ms":18364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot medical image classification of rare events can be done with text-only LLM prompts built from expert knowledge, the paper argues.","keywords":["zero-shot classification","rare event detection","medical image classification","large language models","prompt engineering","seizure onset zone","domain knowledge","fMRI"],"falsifier":"Recompute the Center A results with the tuning patient removed from all reported metrics, or run CuKPL with thresholds fixed from the literature without any patient tuning; if the F1 score falls to the level of the untuned zero-shot baselines, the claimed advantage is not zero-shot.","tokens_in":6314,"feed_emoji":"🧠","tokens_out":7558,"duration_ms":62066,"temperature":0.7,"pith_summary":"This paper tries to establish that rare medical image classification does not require large labeled datasets or a vision encoder: an LLM, given a textual prompt that encodes expert domain knowledge about a rare event, can classify images it has never seen. The authors test this on seizure onset zone (SOZ) detection from resting-state fMRI, where SOZ components make up fewer than 10 percent of independent components. Their method, CuKPL, turns literature-derived knowledge into rule-based textual descriptions of each image and asks an LLM to decide SOZ versus not. On their center-A dataset it reports 88.4 percent accuracy and a 93.8 percent F1 score, above the supervised knowledge-plus-deep-learning baseline, and it keeps an 82.3 percent F1 score when transferred to an unseen center without retraining. If correct, this offers a training-free, privacy-preserving route to rare-event medical classification.","feed_headline":"Training-free LLM prompts outperform deep nets on rare seizure zones","feed_subtitle":"A text-only pipeline converts expert rules into prompts, beating trained models on seizure-onset classification without any training.","key_machinery":"The machinery is the HKETP generation pipeline. Raw expert knowledge from the literature is refined by an LLM into non-overlapping, discriminative knowledge components; each component is encoded as a software function that inspects an independent-component image and emits a natural-language statement with a satisfaction value, such as “Big red activation has asymmetry in hemispheres”. These statements form the Input prompt, which is combined with the refined knowledge as Context and a fixed instruction, and a generative LLM produces a YES/NO classification. A human verifier iteratively adjusts the thresholds and knowledge statements based on one patient’s data before the system is used on other patients.","core_discovery":"The central claim is that the bottleneck in zero-shot medical classification is not the vision model but the prompt: generic templates like “a photo of a {}” fail because they omit the discriminative rules experts use. CuKPL’s contribution is to operationalize those rules as human knowledge-embodied textual prompts (HKETP) and feed them to an instruction-tuned LLM, which classifies by reasoning over image-derived textual facts rather than pixels. The authors show this text-only pipeline surpasses a knowledge-plus-deep-learning system (DeepXSOZ) on the same data, while standard zero-shot methods such as CLIP and CuPL classify virtually everything as noise or resting-state network. They also claim the method is privacy-preserving because the LLM never receives the image itself, only the textual descriptions produced by the knowledge model.","pith_inferences":["The one-patient tuning step blurs the zero-shot claim: the paper never identifies the tuning patient, and the tuned thresholds effectively encode the decision rule, so a strict test would set thresholds from literature alone and report that as a true zero-shot variant.","The pipeline should transfer to other rare events with rule-based radiological signs, such as the proliferative diabetic retinopathy grading named in the conclusion; a multi-disease benchmark would reveal whether the knowledge-encoding step generalizes or is specific to seizure onset zones.","The privacy claim is partial: the LLM never sees raw images, but the textual HKETP descriptions derived from a specific patient’s image are still patient-derived data and may carry identifiable information, a residual risk the paper does not discuss."],"forward_implications":["If CuKPL’s results hold, rare-event medical classification can be built without collecting large training sets, which matters for conditions with very few documented cases.","The method can adapt to a new hospital with only one patient’s data used to tune thresholds, turning a domain-generalization problem into a cheaper domain-adaptation problem.","Because the LLM sees only text, the same prompt-generation framework could be reused across imaging modalities such as MRI, CT, and fundus photography, provided the knowledge components can be computed from the image.","The reported reduction in machine-marked SOZs (28 versus 43 for the knowledge-based EPIK system) suggests the output could prioritize which independent components a neurosurgeon reviews."],"supporting_citations":[{"why":"Introduces the customized-prompt approach (CuPL) that CuKPL extends by adding expert knowledge and a knowledge model.","marker":"[5]"},{"why":"Defines the standard zero-shot classification paradigm with template prompts, which the paper shows fails on rare medical events.","marker":"[6]"},{"why":"Provides the EPIK knowledge-based system and the domain-knowledge rules about seizure onset zones that CuKPL refines into prompts.","marker":"[13]"},{"why":"Describes DeepXSOZ, the knowledge-plus-deep-learning system that CuKPL claims to outperform on the same data.","marker":"[15]"},{"why":"Supplies the supervised knowledge-based baseline (Hunyadi et al.) used for comparison in the evaluation.","marker":"[22]"}],"fun_headline_variants":["Zero-shot LLM prompts outperform deep nets on rare medical images","Training-free prompts: LLM beats deep nets on rare event classification","Expert knowledge guides LLM to classify rare medical events without training","LLM with expert prompts: zero-shot success on rare disease images","Prompt-only LLM classification surpasses trained models for rare events"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method’s claimed zero-shot advantage depends on tuning the knowledge-model thresholds on a single patient’s data, and the paper does not show that this tuning patient is excluded from the evaluation set, so the reported gains could partly come from test-set leakage.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM prompts outperform deep nets on rare medical images","Training-free prompts: LLM beats deep nets on rare event classification","Expert knowledge guides LLM to classify rare medical events without training","LLM with expert prompts: zero-shot success on rare disease images","Prompt-only LLM classification surpasses trained models for rare events"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1531,"prompt_tokens":884,"completion_tokens":647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":559}},"tokens_in":500,"tokens_out":647,"duration_ms":6833,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:58:30.568568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Center A results with the tuning patient removed from all reported metrics, or run CuKPL with thresholds fixed from the literature without any patient tuning; if the F1 score falls to the level of the untuned zero-shot baselines, the claimed advantage is not zero-shot.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the customized-prompt approach (CuPL) that CuKPL extends by adding expert knowledge and a knowledge model."},{"cited_title":"Rules of being not SOZ:","cited_arxiv_id":null,"evidence_quote":"Defines the standard zero-shot classification paradigm with template prompts, which the paper shows fails on rare medical events."},{"cited_title":"A systematic review of rare events detection across modalities using machine learning and deep learning,","cited_arxiv_id":null,"evidence_quote":"Provides the EPIK knowledge-based system and the domain-knowledge rules about seizure onset zones that CuKPL refines into prompts."},{"cited_title":"Inductive or deductive? rethinking the fundamental reasoning abilities of llms,","cited_arxiv_id":null,"evidence_quote":"Describes DeepXSOZ, the knowledge-plus-deep-learning system that CuKPL claims to outperform on the same data."},{"cited_title":"Scaling up visual and vision- language representation learning with noisy text super- vision,","cited_arxiv_id":null,"evidence_quote":"Supplies the supervised knowledge-based baseline (Hunyadi et al.) used for comparison in the evaluation."}],"review_version":1}