Pith. sign in

REVIEW 3 major objections 4 minor 36 references

In-Context Learning for Label-Efficient Cancer Image Classification in Oncology

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Frozen AI reaches 0.81 F1 on cancer images via few-shot prompts

desk verdict A believable but statistically under-supported few-shot benchmark; worth serious peer review only if the authors fix the support-set variance problem. read the letter →

arxiv 2505.08798 v1 pith:YLE37W3N submitted 2025-05-08 eess.IV cs.AIcs.CV

Mobina Shrestha , Bishwas Mandal , Vishal Mandal , Asis Shrestha This is my paper · ORCID
classification eess.IVcs.AIcs.CV
keywords in-contextlearningvision-languagemodelscancerimageclassificationfew-shotpromptinghistopathologydermoscopylabel-efficientGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that vision-language models can be adapted to cancer image classification at inference time, with no parameter updates, by embedding a few labeled examples in the prompt. Across three oncology datasets, all four models improved with 3, 5, and 10-shot prompting over their zero-shot baselines. The headline result is GPT-4o reaching a weighted F1 of 0.81 on binary tasks and 0.60 on seven-class skin lesion classification. The authors argue this makes generalist foundation models a practical route for label-scarce settings, such as rare cancers or resource-constrained clinics, while noting results stay below fine-tuned systems.

What carries the argument

The central mechanism is in-context learning by few-shot prompting: a prompt is built from n labeled image-text support examples followed by an unlabeled query image, and the model produces a diagnosis from randomized multiple-choice options. Generative models like GPT-4o and Paligemma consume the support examples as direct image-text sequences; contrastive models like CLIP and ALIGN are adapted to a retrieval formulation that aggregates similarity scores between the query and class-conditioned prompts informed by the support set. Prompt engineering—stepwise chain-of-thought instructions and option order randomization—carries part of the effect, but the claimed gain is the few-shot conditioning itself.

What would settle it

Repeatedly rerun the 3-, 5-, and 10-shot evaluations with many different randomly drawn support sets and several decoding temperatures, then compare the spread of F1 scores. If the gains shrink to zero or vary by more than the reported confidence intervals across support draws, the central claim of consistent few-shot improvement is not robust.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that in-context learning can serve as a label-efficient alternative to fine-tuning for oncology imaging. Four frozen vision-language models—GPT-4o, Paligemma, CLIP, and ALIGN—were prompted with multiple-choice diagnostics on colorectal polyp, breast metastasis, and skin lesion datasets. Every model improved consistently as support examples increased from 0 to 3, 5, and 10 per class, with GPT-4o reaching F1 0.81 on MHIST, 0.79 on PatchCamelyon, and 0.60 on HAM10000 at 10-shot. The authors interpret the mechanism as localized task conditioning or amortized inference: the support examples re-weight the model's attention toward class-relevant visual features without gradient updates, mirroring how clinicians reason from prior cases.

Load-bearing premise

The results rest on the assumption that one randomly chosen set of a few support examples represents how these models behave, since the confidence intervals only resample the test images and ignore the variance from which support examples were picked and from stochastic text generation.

Editorial extensions

If this is right

  • Label-hungry oncology AI can be deployed without retraining: a handful of labeled images per class is enough to move generalist models from near-chance to substantially better diagnostic performance.
  • Open-source models such as Paligemma and CLIP improve nearly in parallel with GPT-4o, so compute-constrained settings can benefit without relying on proprietary APIs.
  • The gap between binary (F1 around 0.8) and seven-class (F1 around 0.6) results shows in-context learning is strongest when the label space is simple, so multi-class deployment would still need additional support.
  • Because the method changes only the prompt, the same frozen model can be repurposed across different diagnostic tasks in rapid succession, which matters where turnaround time is short.
  • The consistent improvement across three tissue types and four model families suggests the effect is not specific to one dataset or one model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The reported confidence intervals resample the test set only; drawing new support examples or changing decoding temperature would likely widen them, so multi-seed evaluation should be the next test.
  • Editorial inference: Part of the few-shot gain may come from the model exploiting label frequencies or option-position priors rather than learning new visual features; permuting support labels or comparing with scrambled examples would separate these.
  • Editorial inference: If the mechanism is general, similar few-shot gains should appear on other histopathology and dermoscopy benchmarks; that is a direct and cheap extension of this protocol.
  • Editorial inference: The retrieval-based adaptation used for CLIP and ALIGN suggests that even contrastive models without generative output heads can be pulled into the in-context learning paradigm, which broadens the set of deployable models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript reports an empirical comparison of four vision-language models (Paligemma, CLIP, ALIGN, and GPT-4o) under zero-shot and few-shot conditions on three public oncology imaging datasets: MHIST, PatchCamelyon, and HAM10000. The authors report weighted F1 scores with bootstrap 95% CIs and claim that, without any parameter updates, all models show significant gains from few-shot prompting, with GPT-4o achieving F1 0.81 on binary tasks and 0.60 on the seven-class task. The paper positions in-context learning as a label-efficient alternative to fine-tuning for resource-constrained oncology settings.

Significance. If the reported gains are robust, the study would provide a useful practical benchmark for frozen generalist VLMs in oncology classification and would support the feasibility of in-context learning in low-data clinical settings. The manuscript has concrete strengths: it uses standard public datasets, evaluates multiple models under a common protocol, includes class-wise breakdowns, and explicitly checks option-order bias by randomizing label positions. It also acknowledges the performance ceiling relative to fine-tuned systems and the sensitivity of outputs to small prompt variations. However, the central 'significant gains' claim is not backed by appropriate statistical inference, and the CLIP/ALIGN setups are not true in-context learning; both issues need to be fixed before the conclusions can be relied upon.

major comments (3)
  1. [Methods, Performance Evaluation and Statistical Analysis; Abstract; Results] The claim that 'all models showed significant gains with few-shot prompting' is not supported by the reported statistics. The 95% CIs in Table 2 are bootstrap resamples of the fixed 158/161-image test set only, as stated in Methods ('the test dataset was resampled with replacement'), so they do not capture variance from the random selection of 3, 5, or 10 support examples or from stochastic decoding. With only one randomly drawn support set per condition, the point estimates are single realizations. In fact, several zero-shot and 3-shot CIs overlap substantially, e.g., MHIST ALIGN 0.59 (0.52–0.65) vs. 0.62 (0.57–0.69) and PatchCamelyon CLIP 0.52 (0.46–0.55) vs. 0.59 (0.48–0.65). This concern is reinforced by the Summary's own admission that 'even small variations can lead to inconsistent outputs which were observed in our experiments.' The authors should either provide a formal significance test (e.g., paired bootstrap or permutation test across repeated support-set draws) or soften the 'significant' language and restrict the claim to the observed point estimates.
  2. [Methods, In-Context Learning for Sample-Efficient Adaptation; Abstract] The CLIP and ALIGN evaluations are retrieval-based approximations rather than in-context learning in the generative sense. The Methods state that for 'image-text matching models (i.e. CLIP and ALIGN), we adapted the ICL structure to a retrieval-based formulation, aggregating similarity scores between the query image and class-conditioned prompts informed by the support set.' This is essentially a prototype/nearest-neighbor classifier over image-text similarities, not the same mechanism as autoregressive in-context learning in GPT-4o or Paligemma. Because the title and abstract generalize across all four models, the paper should explicitly separate the generative ICL results from the retrieval-based few-shot baselines, or rename the latter accordingly.
  3. [Summary; Methods, Model Implementation and Inference Setup] The paper states that temperature and top_p values in the range 0.5–0.7 were 'explored' during experimentation but the final values used for each model are never reported. If these hyperparameters were chosen after inspecting test-set performance, the reported results would be optimistically biased even though no model parameters were updated. The authors should report the exact decoding settings per model, and ideally show that the main conclusions are stable across a small grid of these settings.
minor comments (4)
  1. [Table 2] Several confidence intervals in Table 2 contain typographical errors, e.g., PatchCamelyon CLIP 3-shot '(0.48-65)' and HAM10000 ALIGN 3-shot '(28-0.42)' are missing leading zeros; these should be corrected to '(0.48–0.65)' and '(0.28–0.42)'.
  2. [Figure 1] The sample model response panel in Figure 1 appears to be empty or garbled in the manuscript text, making it impossible to verify the claimed prompting format; the figure should be regenerated with a visible example response.
  3. [Model Implementation and Inference Setup] The model referred to as 'ALIGN' is explicitly an open-source approximation (kakaobrain/align-base), not the original Google ALIGN model; this is acknowledged in Methods but should also be stated in the Abstract or at least in the first Results paragraph so readers do not infer results for the commercial model.
  4. [References] There are minor formatting inconsistencies in the reference list (e.g., 'ICCAI' for MICCAI, incomplete conference names) and several citations are given as URLs without access dates or version identifiers; the reference list should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports direct benchmark measurements on frozen models, with no derivation reducing to its inputs.

full rationale

This is an empirical benchmarking study. The central claims are measurement statements: few-shot prompting improves weighted F1 over zero-shot for four frozen vision-language models across three public oncology datasets. There is no derivation chain in which one quantity is defined in terms of another, no fitted parameter that is later renamed as a prediction, and no uniqueness theorem or ansatz imported from the authors' prior work. The only self-reference is the sentence in Model Implementation and Inference Setup stating that the same ALIGN approximation 'was previously applied by Mandal et al [36]'; a co-author of the present paper is an author of that prior work. That citation is purely implementation provenance for an open-source checkpoint and does not support any load-bearing inference about the reported F1 gains. The paper's own Summary acknowledges that 'even small variations can lead to inconsistent outputs which were observed in our experiments,' and the confidence intervals in Performance Evaluation and Statistical Analysis bootstrap only the test set rather than support-set selection. These are legitimate statistical-validity concerns about stability and significance, but they are not circularity: the reported scores are still measurements of external test images with frozen models, not quantities forced by construction to equal the inputs. No self-definitional reduction, fitted-input-as-prediction, or self-citation chain is present. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new entities are introduced. The analysis rests on frozen pretrained models and public data; the main uncharged inputs are the undisclosed decoding hyperparameters and the assumption that randomly selected support examples are representative.

free parameters (2)
  • temperature = not reported
    Authors explored values from 0.5 to 0.7 but did not state which value produced the reported results; if chosen by test performance, it is a fitted parameter.
  • top_p = not reported
    Same exploration as temperature; final value not disclosed.
assumptions (3)
  • domain assumption Pretrained VLMs encode sufficient medical visual knowledge to classify cancer images without medical fine-tuning
    The entire evaluation depends on frozen models recognizing histopathology and dermoscopy features; no independent evidence is provided beyond the reported scores.
  • domain assumption Public dataset labels are ground truth
    MHIST, PatchCamelyon, and HAM10000 labels are taken as correct; standard for benchmarks but still an assumption.
  • domain assumption Bootstrap resampling of the test set captures evaluation uncertainty
    The CIs ignore variance from random support example selection and stochastic decoding, so they understate total uncertainty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Context Learning for Label-Efficient Cancer Image Classification in Oncology." pith.science (2026). https://pith.science/paper/YLE37W3N

@misc{pith2026250508798,
  author       = {Pith},
  title        = {Pith review of: In-Context Learning for Label-Efficient Cancer Image Classification in Oncology},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YLE37W3N}},
  note         = {Machine review of arXiv:2505.08798}
}
read the original abstract

The application of AI in oncology has been limited by its reliance on large, annotated datasets and the need for retraining models for domain-specific diagnostic tasks. Taking heed of these limitations, we investigated in-context learning as a pragmatic alternative to model retraining by allowing models to adapt to new diagnostic tasks using only a few labeled examples at inference, without the need for retraining. Using four vision-language models (VLMs)-Paligemma, CLIP, ALIGN and GPT-4o, we evaluated the performance across three oncology datasets: MHIST, PatchCamelyon and HAM10000. To the best of our knowledge, this is the first study to compare the performance of multiple VLMs on different oncology classification tasks. Without any parameter updates, all models showed significant gains with few-shot prompting, with GPT-4o reaching an F1 score of 0.81 in binary classification and 0.60 in multi-class classification settings. While these results remain below the ceiling of fully fine-tuned systems, they highlight the potential of ICL to approximate task-specific behavior using only a handful of examples, reflecting how clinicians often reason from prior cases. Notably, open-source models like Paligemma and CLIP demonstrated competitive gains despite their smaller size, suggesting feasibility for deployment in computing constrained clinical environments. Overall, these findings highlight the potential of ICL as a practical solution in oncology, particularly for rare cancers and resource-limited contexts where fine-tuning is infeasible and annotated data is difficult to obtain.

Figures

Figures reproduced from arXiv: 2505.08798 by the authors.

Figure 2
Figure 2. Zero and few-shot learning performance of four vision-language models (Paligemma, ALIGN, CLIP, GPT-4o) across three cancer image datasets: (A) MHIST, (B) PatchCamelyon, and (C) HAM10000. GPT-4o consistently achieved the highest F1 scores as the number of samples increased. Weighted F1 scores at zero-shot were 0.34 for GPT-4o (CI: 0.29–0.37), 0.35 for Paligemma (CI: 0.28–0.39), 0.32 for CLIP (CI: 0.27–0.35), and 0.32… view at source ↗
Figure 3
Figure 3. Class-wise F1 scores for (A) Paligemma, (B) CLIP, (C) ALIGN, and (D) GPT-4o across MHIST dataset: Hyperplastic Polyp, Sessile Adenoma and PatchCamelyon: Tumor, and No Tumor. Each radar plot displays performance across zero-shot, 3-shot, 5-shot, and 10-shot learning settings. Model performance in multi-class classification tasks (HAM10000) Multi-class classification of skin lesions in the HAM10000 dataset remains a c… view at source ↗
Figure 5
Figure 5. Comparison of VLMs: (A) Paligemma, (B) CLIP, (C) ALIGN, and (D) GPT-4o on the HAM10000 skin lesion classification task. Radar plots show how performance improves with more support examples across lesion categories [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 32 canonical work pages

  1. [1]

    Topol, E. J. (2019). High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine, 25(1), 44–56

    work page 2019

  2. [2]

    Obermeyer, Z., & Emanuel, E. J. (2016). Predicting the future —big data, machine learning, and clinical medicine. The New England Journal of Medicine , 375(13), 1216–1219

    work page 2016

  3. [3]

    M., et al

    McKinney, S. M., et al. (2020). International evaluation of an AI system for breast cancer screening. Nature, 577(7788), 89–94

    work page 2020

  4. [4]

    Esteva, A., et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639), 115–118

    work page 2017

  5. [5]

    Coudray, N., et al. (2018). Classification and mutation prediction from non –small cell lung cancer histopathology images using deep learning. Nature Medicine, 24(10), 1559–1567

    work page 2018

  6. [6]

    Campanella, G., et al. (2019). Clinical -grade computational pathology using weakly supervised deep learning on whole -slide images. Nature Medicine, 25(8), 1301–1309

    work page 2019

  7. [7]

    He, Y ., et al. (2023). Vision foundation models in medical imaging: progress, challenges, and future directions. Nature Biomedical Engineering, 7(1), 6–22

    work page 2023

  8. [8]

    Wang, F., et al. (2019). Clinical risk prediction with machine learning: a systematic review. PLoS Medicine, 16(11), e1002791

    work page 2019

Show all 36 references
  1. [9]

    Kaushal, A., et al. (2020). Addressing bias in AI for health care: A framework for data collection and model development. The Lancet Digital Health, 2(9), e435–e440

    2020

  2. [10]

    Brown, T., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901

    2020

  3. [11]

    Singhal, K., et al. (2023). MedPrompt: Retrieval-augmented in-context learning for clinical question answering. Nature, 624(7980), 56–65

    2023

  4. [12]

    Johnson, A. E. W., et al. (2021). MIMIC-IV: a freely accessible electronic health record dataset. Scientific Data, 8(1), 1–14

    2021

  5. [13]

    Rajpurkar, P., et al. (2022). Multimodal machine learning in healthcare: A review. NPJ Digital Medicine, 5(1), 14

    2022

  6. [14]

    Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018). Deep learning for healthcare: review, opportunities and ch allenges. Briefings in Bioinformatics, 19(6), 1236–1246

    2018

  7. [15]

    Ghassemi, M., Oakden -Rayner, L., & Beam, A. L. (2021). The false hope of c urrent approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 3(11), e745–e750

    2021

  8. [16]

    OpenAI. (2023). GPT-4 Technical Report. https://openai.com/research/gpt-4

    2023

  9. [17]

    DeepMind. (2023). Gemini: Multimodal AI models. https://deepmind.google/technologies/gemini/

    2023

  10. [18]

    Alayrac, J.-B., et al. (2022). Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:2204.14198

    2022 arXiv

  11. [19]

    Liu, H., et al. (2023). Visual instruction tuning. arXiv preprint arXiv:2304.08485

    2023 arXiv

  12. [20]

    Chen, S., et al. (2023). Multimodal few-shot learning with frozen large language models. arXiv preprint arXiv:2306.17143

    2023 arXiv

  13. [21]

    Zhou, C., et al. (2023). InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500

    2023 arXiv

  14. [22]

    Sidhom, J.-W., et al. (2023). Machine learning in oncology: Methods, applications, and challenges. CA: A Cancer Journal for Clinicians, 73(2), 127–151

    2023

  15. [23]

    Ardila, D., et al. (2019). End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nature Medicine, 25(6), 954–961

    2019

  16. [24]

    R., et al

    Zech, J. R., et al. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiogr aphs: a cross-sectional study. PLOS Medicine, 15(11), e1002683

    2018

  17. [25]

    C., Ligero, M., Sainath, S., Ghaffari Laleh, N.,

    Ferber, D., Wölflein, G., Wiest, I. C., Ligero, M., Sainath, S., Ghaffari Laleh, N., ... & Kather, J. N. (2024). In -context learning enables multimodal large language models to classify cancer pathology images. Nature Communications, 15(1), 10104

    2024

  18. [26]

    & Hassanpour, S

    Wei, J., Suriawinata, A., Ren, B., Liu, X., Lisovsky, M., Vaickus, L., ... & Hassanpour, S. (2021). A petri dish for histopathology image analysis. In Artificial Intelligence in Medicine: 19th International Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual ...

    2021

  19. [27]

    S., Linmans, J., Winkens, J., Cohen, T., & Welling, M

    Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., & Welling, M. (2018). Rotation equivariant CNNs for digital pathology. In Medical image computing and computer assisted intervention–mICCAI 2018: 21st international conference, granada, Spain, September 16-20, 2018, proceedi...

    2018

  20. [28]

    E., Veta, M., Van Diest, P

    Bejnordi, B. E., Veta, M., Van Diest, P. J., Van Ginneken, B., Karssemeijer, N., Litjens, G., ... & CAMELYON16 Consortium. (2017). Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama, 318(22), 2199-2210

    2017

  21. [29]

    Tschandl, P., Rosendahl, C., & Kittler, H. (2018). The HAM10000 dataset, a large collection of multi -source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1), 1-9

    2018

  22. [30]

    Last accessed on 4/15/2025

    https://huggingface.co/google/paligemma-3b-mix-224. Last accessed on 4/15/2025

    2025

  23. [31]

    Last accessed on 4/11/2025

    https://huggingface.co/openai/clip-vit-base-patch32. Last accessed on 4/11/2025

    2025

  24. [32]

    Last accessed on 4/27/2025

    https://platform.openai.com/docs/models/gpt-4o. Last accessed on 4/27/2025

    2025

  25. [33]

    Last accessed on 4/08/2025

    https://huggingface.co/kakaobrain/align-base. Last accessed on 4/08/2025

    2025

  26. [34]

    Old Experience Helps: Leveraging Survey Methodology to Improve AI Text Annotation Reliability in Social Sciences, arXiv Feb.2025

    Li, L. Old Experience Helps: Leveraging Survey Methodology to Improve AI Text Annotation Reliability in Social Sciences, arXiv Feb.2025

    2025

  27. [35]

    AI Prompt Engineering in Health Professions Education, 2024

    Le, T. AI Prompt Engineering in Health Professions Education, 2024

    2024

  28. [36]

    (2024, May)

    Mandal, B., Khanal, S., & Caragea, D. (2024, May). Contrastive learning for multimodal classification of crisis related tweets. In Proceedings of the ACM Web Conference 2024 (pp. 4555-4564)

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.