{"id":"0e4c13f4-de9e-41f9-8581-1feb36a9a20e","arxiv_id":"2501.15798","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"KeepFIT V2 pretrains a fundus vision-language model with a small 'elite' image-text dataset plus public categorical labels, reaching performance competitive with models trained on much larger private data.","lead":"The paper releases MM-Retinal V2, a public fundus image-text dataset covering three imaging modalities, and KeepFIT V2, a pretraining method that transfers knowledge from a small set of expert captions into large categorical datasets. A reader might use this to build retinal foundation models without collecting hundreds of thousands of private clinical image-text pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining/evaluation overlap not ruled out: FLAIR's 190K and an unlisted 80K public images may include the evaluation datasets, which would inflate the claimed competitiveness of KeepFIT V2.","rationale":"The reader's weakest assumption focused on category coverage and visual similarity between MM-Retinal V2 and public datasets. That concern is reasonable, but the paper's own report of 'two unseen categories' in ODIR200×3 indicates that category coverage is incomplete even in the current design, yet the method still shows gains in the ablation. The more decisive risk to the central claim is potential dataset overlap between pretraining and evaluation. The pretraining data are public categorical collections, the evaluation data are also public fundus datasets, and the manuscript provides no overlap analysis or explicit exclusion list. If overlap exists, the comparison against private-data models is confounded: an image seen during contrastive pretraining can be recognized via its class prompt in zero-shot, and its features are already tuned for that image's label. This would directly undermine the 'elite knowledge spark' explanation for the reported competitiveness. The concern is not an allegation of misconduct; it is a concrete, checkable gap in experimental reporting that must be closed before the central claim is credible. Given the uncertainty, keeping the reader's CONDITIONAL verdict is appropriate, with the added condition that pretraining/evaluation overlap be ruled out. If the overlap check later reveals contamination, the verdict should move toward REJECT or at least major revision.","tokens_in":23762,"tokens_out":6115,"duration_ms":60166,"concrete_test":"Compute perceptual or exact image hashes for all images in the released pretraining corpora (the FLAIR-derived 190K set, the additional 80K set, and MM-Retinal V2) and for all images in REFUGE, ODIR200×3, iChallenge-AMD, Retina, FIVES, and APTOS. If any exact or near-duplicate overlap is found, exclude those images from pretraining, retrain KeepFIT V2, and re-evaluate Tables I, II, and Fig. 4; if the performance gap to RET-CLIP, ViLRef, and RetiZero widens materially, the headline claim of competitiveness from a small elite spark is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that KeepFIT V2, trained on ~5K elite image-text pairs per modality plus public categorical data, is competitive with models trained on hundreds of thousands of private image-text pairs. Section V-A1 states that the CFP pretraining set is composed of FLAIR's public datasets (over 190K images) plus 'over 80K data samples from public categorical retinal datasets' that are not itemized. The same section lists six downstream evaluation datasets: REFUGE, ODIR200×3, iChallenge-AMD, Retina, FIVES, and APTOS. The manuscript contains no statement that these evaluation datasets (or their source collections) were excluded from pretraining, and FLAIR's 37-dataset collection is known to include widely used public fundus datasets such as APTOS. If evaluation images appeared in pretraining, zero-shot classification—which in this framework measures image-to-text similarity to class-name prompts—can be inflated directly by image memorization, and few-shot/linear probing can be inflated by feature overfitting to test-domain images. The 'five cross-validation folds' noted in Section V-C concerns downstream evaluation splitting, not pretraining-data overlap. Without documented exclusion or a deduplication analysis, the observed parity with private-data models cannot be attributed to the proposed knowledge-spark mechanism.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MM-Retinal V2, a public fundus image-text dataset covering CFP, FFA, and OCT modalities with roughly 5K pairs per modality plus a 452K-utterance ophthalmic text corpus, and proposes KeepFIT V2, a CLIP-style vision-language pretraining model. KeepFIT V2 is pretrained on categorical public datasets together with the small elite image-text dataset. Knowledge transfer is implemented by a semantically oriented cross-attention module and an appearance-oriented VQ-based cross-attention module, whose outputs refine the public-dataset text features through MSE losses (Eqs. 20-21). The authors evaluate zero-shot, few-shot, and linear probing performance across CFP, FFA, and OCT, reporting that KeepFIT V2 is competitive with fundus VLP models trained on hundreds of thousands of private image-text pairs.","tokens_in":24033,"tokens_out":4613,"duration_ms":43355,"significance":"If the claims hold, the work is practically significant: it suggests that a few thousand carefully curated public image-text pairs plus large categorical public datasets can substitute for large private clinical image-text corpora in fundus VLP, and it releases both data and model. The paper's strengths include the breadth of the evaluation (three modalities and three settings), the release of the MM-Retinal V2 dataset, the ablation study in Table VII, and the explicit goal of reducing dependence on private data. The central claim, however, rests on two assumptions that are not yet substantiated: that the pretraining and evaluation data do not overlap, and that MM-Retinal V2 is visually and categorically close enough to the public data for the knowledge-injection retrieval to be relevant.","major_comments":[{"comment":"The manuscript does not rule out overlap between the CFP pretraining data and the evaluation datasets. Section V-A1 states that pretraining uses FLAIR's public datasets (over 190K images) plus an additional 'over 80K data samples from public categorical retinal datasets' that are not itemized, while Section V-D1 evaluates on REFUGE, ODIR200×3, iChallenge-AMD, Retina, FIVES, and APTOS—public collections that are plausibly contained in FLAIR's 37-dataset aggregation or in the additional 80K samples. Because zero-shot classification here is computed from image-to-text similarity (Eqs. 2-3) and few-shot/linear probing use frozen features, memorized or near-duplicate pretraining images would directly inflate the reported parity with private-data models. Please provide an explicit exclusion statement, enumerate the additional 80K public categorical datasets, and include a near-duplicate/hash-based deduplication analysis between all pretraining images and each evaluation set.","section":"Section V-A1 and Section V-D1"},{"comment":"The eleven OCT public pretraining datasets are not enumerated. Section V-A3 says only that 'we also collect eleven OCT public datasets with only category labels for KeepFIT V2 pretraining, totaling over 181K images,' and the evaluation uses OCTID and OCTDL. Since OCTID and OCTDL are themselves public OCT datasets, it is unclear whether they were excluded from pretraining. Please list all eleven pretraining datasets and explicitly state that the two evaluation datasets (and their source collections) were excluded, or the OCT results cannot be attributed to generalization from the proposed method.","section":"Section V-A3, Tables V-VI"},{"comment":"The load-bearing premise that 'MM-Retinal V2 exhibits a high degree of similarity with the public ones and almost covers all the common retinal disease categories' is asserted but not quantitatively validated. The cross-attention retrieval in Eqs. (7) and (18) and the refinement losses in Eqs. (20)-(21) assume that the retrieved elite captions are semantically and visually relevant to the public images; if the similarity or category coverage is poor, the MSE losses will push public text features toward unhelpful targets. Please provide quantitative evidence for this assumption, for example nearest-neighbor retrieval precision against category labels, a per-disease coverage table comparing MM-Retinal V2 with each public pretraining dataset, or distributions of image-similarity scores. Without such validation, the mechanism's contribution to the reported gains is not established.","section":"Section IV-C1, Eqs. (7), (18), (20)-(21)"},{"comment":"The manuscript states that all metrics are averaged across five cross-validation folds, but no standard deviations, confidence intervals, or significance tests are reported anywhere in the tables. Several headline comparisons involve small margins (for example, Table II REFUGE Tip-adapter 10-shot ACC of 91.4 for KeepFIT V2 versus 94.4 for KeepFIT V2L, and Table VI OCTDL few-shot differences around 1-2 points). Because the central claim is about competitiveness with private-data models, please report variance across folds and, where possible, significance tests or paired comparisons for the key zero-shot and few-shot results.","section":"Section V-C, Tables II, IV, VI"}],"minor_comments":[{"comment":"There is a typo: 'KeppFIT V L' should read 'KeepFIT V2L'.","section":"Section V-B"},{"comment":"The word 'pertaining' appears to be a typo for 'pretraining' in 'restricting their application to foundational pertaining' and 'general vision-language pertaining'.","section":"Section II-A and Section IV"},{"comment":"The model name is written inconsistently as 'KeepFITV1', 'KeepFITV2', and 'KeepFIT V2L'; please standardize the spacing for readability.","section":"Tables I-VI"},{"comment":"For reproducibility, please provide the exact names and licenses of the eleven OCT public pretraining datasets and the additional CFP/FFA public categorical datasets, not only the aggregate image counts.","section":"Section V-A3"}],"recommendation":"major_revision","confidential_remarks":"The overlap concern is the most serious issue: if the evaluation datasets are present in the pretraining pool, the central 'competitive with private-data models' claim would be substantially weakened. This is fixable with an explicit exclusion statement and a deduplication audit, which is why I recommend major revision rather than rejection. The self-citation to MM-Retinal V1 is legitimate given that this paper is an extension, but the technical novelty would be clearer if the knowledge-injection module were isolated with a dedicated retrieval-quality analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee, but the central claim has a data-leakage asterisk the authors need to close.\n\nWhat's actually new: MM-Retinal V2 is a multi-modal public dataset (CFP, FFA, OCT, about 5K pairs each, plus a 452K-utterance ophthalmic text corpus). KeepFIT V2's hybrid knowledge injection—semantic cross-attention on contrastive features plus appearance cross-attention on IBQ discrete tokens—is a genuine combination I haven't seen in fundus VLP. The idea of using a small elite set to enrich categorical public data is sensible, and the ablation shows each module earns its place. The authors also release models for all three modalities, which is useful.\n\nSoft spots, in order of severity. First: the stress-test concern holds up. The paper never states that the evaluation sets (REFUGE, ODIR, Retina, iChallenge-AMD, FIVES, APTOS) were excluded from the pretraining corpora. FLAIR's 190K images come from 37 public datasets, and APTOS is a well-known member of that family; the additional 80K public samples aren't itemized. Without an exclusion list or a deduplication analysis, the zero-shot numbers could be inflated by memorization, and the headline claim—parity with private-data models from a small spark—can't be attributed to the mechanism. This is the load-bearing issue.\n\nSecond: no error bars on the five-fold averages. Several 'surpass' claims rest on 1-2% gaps that could easily be noise. Third: the FFA experiments use FFA-IR, which has over a million images and 10K reports. That isn't the small-data regime the abstract emphasizes; the 'spark' story is really supported by CFP and OCT. Fourth: the eleven OCT pretraining datasets are never enumerated, making overlap checks hard. Fifth: the abstract's 'competitive' overstates the ODIR and Retina zero-shot results, where private-data models lead by 5-15 points.\n\nCredit where it's due: the dataset and model releases are real contributions, the method is novel, and the paper is clearly written. The issues are addressable with documentation and variance estimates.\n\nRecommendation: send to peer review, but only after the authors demonstrate no pretraining/evaluation overlap and soften the claims accordingly. A good referee should demand those changes.","headline":"Worth refereeing, but the authors must rule out pretraining/evaluation overlap before the headline claim can be taken at face value.","tokens_in":24582,"tokens_out":4715,"would_cite":true,"duration_ms":41423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MM-Retinal V2 claims that pretraining on a few thousand elite fundus image-text pairs, injected into public categorical datasets, matches models trained on hundreds of thousands of private clinical pairs.","keywords":["fundus image analysis","vision-language pretraining","knowledge injection","multi-modality dataset","CFP FFA OCT","zero-shot classification","contrastive learning","vector quantization"],"falsifier":"Measure per-category image-retrieval precision between MM-Retinal V2 and each public pretraining dataset using the model's own cross-attention scores, then on low-precision categories keep the injection but swap in random expert captions; if performance does not drop, the refinement loss is not transferring the claimed ophthalmic knowledge.","tokens_in":23562,"feed_emoji":"👁️","tokens_out":6903,"duration_ms":55896,"temperature":0.7,"pith_summary":"This paper tackles a data bottleneck in fundus vision-language pretraining: expert image-text pairs are scarce, while large public datasets only carry category labels. The authors build MM-Retinal V2, a public dataset of about 17,000 high-quality image-text pairs across CFP, FFA, and OCT, spanning over 96 diseases. They then train KeepFIT V2, which transfers these elite captions into the public categorical data through a hybrid knowledge-injection scheme, so the model learns from far fewer image-text pairs than prior private-data models. The claim, backed by zero-shot, few-shot, and linear-probing comparisons, is that this small elite spark yields performance competitive with models pretrained on 190K to 450K private image-text pairs. If true, it would mean public categorical data plus a modest amount of expert captions can substitute for large private clinical text collections.","feed_headline":"Elite fundus captions match 450K private pairs","feed_subtitle":"With 17,000 expert pairs and public labels, a fundus vision-language model rivals ones trained on hundreds of thousands of private reports.","key_machinery":"The load-bearing mechanism is the hybrid image-text knowledge injection module. A semantics-oriented branch uses multi-head cross-attention, with public image features as query and MM-Retinal V2 image features as key, to reweight MM-Retinal V2 text features as expert knowledge; an appearance-oriented branch repeats this over discrete tokens produced by index backpropagation quantization (IBQ), a vector-quantization scheme that turns visual feature maps into codebook tokens preserving low-level lesion appearance. Two expert-knowledge refinement losses (MSE between retrieved expert text and public text features) then pull the public dataset's template text toward the expert captions, so the categorical data effectively learns from the elite captions during contrastive pretraining. A preliminary masked-language-modeling pass on the MM-Retinal-Text corpus adapts the text encoder to ophthalmic terminology before any vision-language training.","core_discovery":"The central claim is that knowledge from a small, high-quality image-text dataset can be distilled into a vision-language model through hybrid injection of global semantic and local appearance information, to the point that downstream performance matches models trained on over an order of magnitude more private data. KeepFIT V2 first pretrains its text encoder on ophthalmic text via masked language modeling, then for each public categorical image retrieves expert captions from MM-Retinal V2 based on both contrastive semantic features and vector-quantized appearance tokens, and refines the public text features toward the retrieved expert text with MSE losses. Trained jointly on MM-Retinal V2 and public categorical datasets, the model achieves top results on several CFP benchmarks and strong gains on FFA and OCT, despite using roughly 1% of the image-text pairs used by RET-CLIP, ViLRef, or RetiZero.","pith_inferences":["A testable corollary, not stated in the paper: the gain from knowledge injection should scale with how well MM-Retinal V2's category coverage and visual appearance overlap the public pretraining set; on categories absent from the elite set, injection could even hurt.","The appearance-oriented branch may be doing more than adding detail: quantized token matching could act as a regularizer that prevents the contrastive branch from collapsing onto template text, an effect that could be isolated by ablating the codebook with random assignment.","One could extend the recipe to other scarce-text medical modalities, such as chest X-ray or dermoscopy, whenever a small corpus of diagram-book style expert captions exists; the paper argues this generality but does not demonstrate it.","Another implicit implication is that if a few thousand expert captions suffice, the marginal value of ever-larger private image-text collections may be lower than recent scaling efforts assume, at least for disease categories well covered by public data."],"forward_implications":["KeepFIT V2 attains the best zero-shot average on REFUGE among all compared models and competitive or better few-shot and linear-probing results, despite using about 1% of the image-text pairs used by RET-CLIP, ViLRef, or RetiZero.","The hybrid semantic-plus-appearance injection outperforms either branch alone in the CFP ablation, so both global and local matching contribute to the knowledge transfer.","The same pretraining recipe transfers to FFA and OCT, with the largest relative gains appearing on FFA benchmarks such as MPOS.","Because the scheme relies on public categorical data rather than private clinical text, it offers a reproducible route for other medical imaging domains that face the same scarcity of image-text pairs.","The public release of MM-Retinal V2 and the pretrained weights gives the community a reproducible alternative to the closed, large-scale private corpora used by competing models."],"supporting_citations":[{"why":"Supplies the MM-Retinal V1 baseline and the initial dataset that V2 expands, and every V2 gain is measured against it.","marker":"[40]"},{"why":"FLAIR provides the template-expanded public categorical pretraining setup and is the main small-data baseline.","marker":"[33]"},{"why":"RET-CLIP is the large-scale private-data CFP baseline whose 193,865 image-text pairs KeepFIT V2 aims to match with about 1% of the data.","marker":"[7]"},{"why":"ViLRef is another large-scale private-data baseline, with 451,956 image-text pairs, used in every CFP evaluation table.","marker":"[41]"},{"why":"RetiZero is the third large-scale private-data baseline, with 341,896 pairs, and defines the comparison group for the central claim.","marker":"[37]"},{"why":"IBQ supplies the vector-quantization tokenizer that makes the appearance-oriented knowledge extraction possible.","marker":"[32]"},{"why":"BioClinicalBert is the text encoder architecture that preliminary textual pretraining adapts to ophthalmic knowledge.","marker":"[3]"},{"why":"CLIP is the contrastive vision-language backbone whose image-text alignment loss anchors the whole KeepFIT V2 framework.","marker":"[30]"},{"why":"The DEN dataset, after cleaning and modality filtering, contributes CFP and FFA pairs to MM-Retinal V2.","marker":"[14]"}],"fun_headline_variants":["17K expert fundus pairs match 450K private captions","Small elite fundus data rivals big private pretraining","Public labels plus 17K expert captions match large private sets","Fundus VLP: 1% data achieves private-scale performance","Elite knowledge spark: 1% data matches 450K private"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that MM-Retinal V2 images are visually similar to the public categorical images and cover nearly all the common retinal disease categories; if that overlap is weak, the retrieval will pull in irrelevant expert captions and the refinement loss will push the public text features toward unhelpful targets.","fun_headline_variants_meta":{"raw":{"variants":["17K expert fundus pairs match 450K private captions","Small elite fundus data rivals big private pretraining","Public labels plus 17K expert captions match large private sets","Fundus VLP: 1% data achieves private-scale performance","Elite knowledge spark: 1% data matches 450K private"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":3025,"prompt_tokens":964,"completion_tokens":2061,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1972}},"tokens_in":580,"tokens_out":2061,"duration_ms":15721,"temperature":1.0,"reasoning_tokens":1972,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:56:11.076391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure per-category image-retrieval precision between MM-Retinal V2 and each public pretraining dataset using the model's own cross-attention scores, then on low-precision categories keep the injection but swap in random expert captions; if performance does not drop, the refinement loss is not transferring the claimed ophthalmic knowledge.","supporting_citations":[{"cited_title":"In: International Conference on Medical Image Computing and Computer-Assisted Intervention","cited_arxiv_id":null,"evidence_quote":"Supplies the MM-Retinal V1 baseline and the initial dataset that V2 expands, and every V2 gain is measured against it."},{"cited_title":"In: International Conference on Medical Image Computing and Computer-Assisted Intervention","cited_arxiv_id":null,"evidence_quote":"RET-CLIP is the large-scale private-data CFP baseline whose 193,865 image-text pairs KeepFIT V2 aims to match with about 1% of the data."},{"cited_title":"arXiv e-prints pp","cited_arxiv_id":null,"evidence_quote":"ViLRef is another large-scale private-data baseline, with 451,956 image-text pairs, used in every CFP evaluation table."},{"cited_title":"In: International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"CLIP is the contrastive vision-language backbone whose image-text alignment loss anchors the whole KeepFIT V2 framework."},{"cited_title":"In: Proceedings of the IEEE/CVF winter conference on applications of computer vision","cited_arxiv_id":null,"evidence_quote":"The DEN dataset, after cleaning and modality filtering, contributes CFP and FFA pairs to MM-Retinal V2."}],"review_version":1}