{"id":"281402b7-81d7-4348-819e-94f5c7a40be0","arxiv_id":"2508.11721","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FusionFM systematically compares four ophthalmic foundation models and two fusion strategies, finding DINORET and RetiZero strongest and fusion modestly improving some diagnostics.","lead":"This paper benchmarks four eye-image AI models and tests two ways of combining them for detecting eye and systemic diseases. It finds DINORET and RetiZero perform best, and gating-based fusion gives small gains on glaucoma, AMD, and hypertension.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison fairness unverified: model ranking may reflect preprocessing/fine-tuning mismatches rather than true FM quality.","rationale":"The reader's verdict is UNVERDICTED because the full text is corrupted and unreadable. My stress test does not change that verdict; it reinforces it by identifying a specific, checkable condition that the central claim depends on: the fairness of the cross-model evaluation. The reader's weakest assumption concerned label equivalence and distribution match for external cohorts. I agree that those are important, but the more proximate and load-bearing risk is that the four FMs were not evaluated under comparable protocol conditions. Preprocessing and fine-tuning are the most common sources of unfairness in FM benchmarks, and the abstract's mention of 'standardized datasets' does not guarantee that each model was used as intended. This is not an ad hominem or a manufactured concern; it is a standard correctness risk in benchmarking papers. The concrete test I propose (verify or re-run with per-model native configurations) would directly settle whether the ranking is meaningful. Since neither the full text nor reproducible artifacts are available, the honest verdict remains UNVERDICTED, matching the reader's recommendation.","tokens_in":9960,"tokens_out":2459,"duration_ms":33279,"concrete_test":"Reconstruct the full evaluation protocol from the arXiv source or authors' released code. For each of the four FMs, verify that the input resolution, image normalization, and fine-tuning schedule match the model's officially recommended configuration (e.g., RETFound's 224×224 and its pretraining normalization). If any model was evaluated with a non-default preprocessing, re-run that model on the same data splits with its correct configuration and recompute AUC with 95% bootstrap confidence intervals. If the top-two ranking changes or the reported gains (e.g., RetiZero's external AUC advantage) shrink below statistical significance, the central ranking claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an evidence-based ranking of four ophthalmic foundation models. For this ranking to be valid, each FM must be evaluated under a protocol that is fair to its design. The abstract says datasets were 'standardized,' but each FM has its own native input resolution, normalization, and training configuration. If a single preprocessing pipeline was applied to all models, models trained on images that match that pipeline (e.g., resolution, color normalization) will be favored, and models that expect different preprocessing will be penalized. Similarly, fine-tuning choices — number of epochs, learning rate, whether backbone layers are frozen, pooling head architecture — can shift AUC by several points. Because the full text is unreadable (corrupted beyond section headers), the paper provides no way to verify that the observed AUC gaps (e.g., DINORET/RetiZero superiority, RetiZero's external generalization) are not artifacts of protocol mismatch. The external generalization claim is especially vulnerable: if the external cohort happens to match RetiZero's pretraining distribution or preprocessing, while other models are evaluated with mismatched preprocessing, the 'stronger generalization' could be an artifact. Without tables and protocol details, the ranking is not independently checkable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FusionFM, an evaluation suite and two fusion strategies (including a gating-based approach) for four ophthalmic foundation models (RETFound, VisionFM, RetiZero, DINORET). The authors state that they benchmark these models on standardized datasets from multiple countries for glaucoma, diabetic retinopathy, age-related macular degeneration, diabetes, and hypertension, reporting AUC and F1. The main claims are that DINORET and RetiZero achieve superior performance in both ophthalmic and systemic disease tasks, that RetiZero generalizes better on external datasets, and that gating fusion yields modest improvements in glaucoma, AMD, and hypertension, while systemic disease prediction—especially hypertension—remains challenging. The submitted full text is corrupted beyond section headers, so no methods, results tables, equations, or protocol details are readable. As a result, the evidence supporting the central claims cannot be verified from the manuscript as received.","tokens_in":10238,"tokens_out":3525,"duration_ms":40838,"significance":"If the claims hold, this would be the first systematic comparison and fusion study of ophthalmic foundation models and would provide useful, clinically relevant evidence for model selection. The authors are honest about the modest size of fusion gains and about persistent difficulty in systemic disease prediction, which is a strength. The study also covers multiple countries and external datasets, which is valuable if the datasets and splits are properly described. However, the current submission does not provide the evidence needed to assess the ranking: there are no readable tables, confidence intervals, per-model protocol details, or external cohort definitions. The potential contribution is real, but the manuscript in its present form is not evaluable.","major_comments":[{"comment":"The supplied manuscript text is corrupted beyond section headers; all tables, figures, equations, and result details are unreadable. The central claims—DINORET/RetiZero superiority, RetiZero's external generalization, and gating fusion gains—are empirical and require the full protocol and results. Please resubmit a readable manuscript with all tables, figure captions, equations, and appendices. Without this, no claim can be independently checked.","section":"Full text"},{"comment":"The abstract reports that datasets were 'standardized,' but the full text does not describe the preprocessing, input resolution, normalization, augmentation, fine-tuning schedule, learning rates, number of epochs, frozen layers, or classifier heads used for each foundation model. These four models have different native input specifications and training pipelines; if a single preprocessing pipeline was applied to all, models whose pretraining matches that pipeline will be favored, and models expecting different preprocessing will be penalized. The paper must report per-model protocol details or justify that the chosen pipeline is equally fair to all models. Otherwise the observed AUC gaps may be artifacts of protocol mismatch.","section":"Abstract / Methods"},{"comment":"The abstract reports AUC and F1 values without confidence intervals or significance tests. The claims of 'superior performance' and 'stronger generalization' are comparative; the paper should provide bootstrap confidence intervals or paired significance tests (e.g., DeLong tests) for each task and dataset, with appropriate multiple-comparison control. Since the fusion gains are described as modest, statistical reliability is essential to distinguish genuine improvements from noise.","section":"Abstract / Results"},{"comment":"The claim that RetiZero exhibits stronger generalization on external datasets requires a clear definition of the external cohorts: acquisition device, image resolution, population ethnicity, disease severity, label noise, and potential overlap with pretraining data. The reader cannot assess whether the external cohorts match RetiZero's training distribution more closely than the other models' distributions. Please report per-cohort AUC/F1 for every model and describe the external datasets in sufficient detail.","section":"External generalization claim"},{"comment":"Several co-authors are also developers of some of the benchmarked foundation models. This does not by itself invalidate the study, but it is a load-bearing neutrality concern. The manuscript should disclose these relationships and describe safeguards: a fixed evaluation protocol, blinded or third-party execution, publicly released code, or a pre-registered analysis. As written, the comparison is not demonstrably independent, and the reader cannot exclude protocol-selection bias.","section":"Competing interests / Evaluation neutrality"}],"minor_comments":[{"comment":"The phrase 'first study to systematically evaluate...' should be supported by a related-work section that explicitly discusses prior comparative evaluations of ophthalmic FMs; if none exist, say so with citations to the closest works.","section":"Abstract"},{"comment":"FusionFM is described as a 'comprehensive evaluation suite,' but no code or data availability link is given. If the suite is intended for community use, include a repository URL.","section":"Abstract / Code availability"},{"comment":"Some equations and fragments appear in the corrupted text but cannot be read. Ensure all equations are rendered correctly in the resubmission and that notation for the gating fusion is defined precisely.","section":"Corrupted text"},{"comment":"Where fusion improvements are reported, specify the baseline: improvement over the best single model or over the average of all models, and report the absolute AUC/F1 differences rather than only a qualitative 'modest' characterization.","section":"Results / Fusion gains"}],"recommendation":"major_revision","confidential_remarks":"The full text as submitted is garbled; I could not review the methods or results in any detail. The authors should be asked to resubmit a clean, readable manuscript with all tables and protocol details. The author-overlap with benchmarked FM developers deserves explicit editorial attention; a statement of roles and safeguards should be required. The topic is timely and the honest reporting of modest fusion gains is a positive sign, but the evidence is currently unverifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper asks the right questions—which ophthalmic FM is best, are they equally good across tasks, and does fusion help—and it claims an evidence-based answer. The abstract is clear and the framing is honest: DINORET and RetiZero come out ahead, fusion gives only modest gains, and systemic disease prediction stays hard. That is genuinely useful for model selection, and it is not over-hyped.\n\nWhat is new: a simultaneous head-to-head of four FMs with standardized multi-country datasets and two fusion baselines. The fusion methods themselves are standard, but the systematic comparison is a real service to the field. The authors also deserve credit for reporting that fusion helps only in some tasks and that hypertension prediction remains poor; that kind of negative result is easy to bury.\n\nThe soft spots are serious but mostly unanswerable from what I can read. The provided full text is corrupted beyond the abstract, so I cannot check the tables, the protocol, or the confidence intervals. That is not the authors' fault in my copy, but it means the central ranking is currently an assertion, not a demonstration. The stress-test concern is on point: each FM has its own native resolution, normalization, and optimal fine-tuning recipe. If the authors applied one pipeline to all four, the ranking could reflect preprocessing matching rather than model quality. I cannot tell from the abstract whether they tuned each model fairly. Author overlap with some of the benchmarked FMs is another neutrality question, but that alone is not disqualifying—most benchmark papers come from groups with vested interests. The cure is transparency: protocol details, per-model hyperparameters, and ideally released evaluation code.\n\nBottom line: if the full paper shows comparable training setups and reports uncertainty, this is a solid empirical contribution that belongs in the literature. As it stands, I cannot verify the headline ranking, and I would not cite it yet. But it deserves a serious referee—someone should read the actual methods and tables, not desk-reject on the abstract. Send it to review; ask for the protocol and the code.","headline":"A useful benchmark question buried in an unreadable file: the abstract promises the first systematic comparison of four ophthalmic foundation models, but with the full text corrupted, the ranking is currently unverifiable.","tokens_in":10682,"tokens_out":1472,"would_cite":false,"duration_ms":21021,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FusionFM, a systematic benchmark of four ophthalmic foundation models, reports that DINORET and RetiZero lead on eye and systemic disease tasks, while gated fusion yields modest, task-specific gains.","keywords":["ophthalmic foundation models","retinal imaging","model fusion","gating-based fusion","external validation","systemic disease prediction","glaucoma","diabetic retinopathy"],"falsifier":"Train or fine-tune each of the four foundation models on one shared public retinal dataset with a single fixed preprocessing pipeline and several random splits; if RETFound or VisionFM then match or beat DINORET and RetiZero, the reported ranking is an artifact of dataset handling rather than a property of the models. For the external-generalization claim, the test is whether RetiZero's advantage over DINORET reproduces on an unseen dataset from a different camera and ethnicity mix.","tokens_in":9898,"feed_emoji":"👁️","tokens_out":5088,"duration_ms":55219,"temperature":0.7,"pith_summary":"The paper tries to answer a practical question the field has not settled: among current ophthalmic foundation models, which should a researcher or clinician actually use, and does combining them help? FusionFM benchmarks four pretrained retinal-image models on the same multi-country datasets for glaucoma, diabetic retinopathy, age-related macular degeneration, diabetes, and hypertension. It finds that DINORET and RetiZero outperform RETFound and VisionFM on most tasks, with RetiZero generalizing best to external cohorts. A gating-based fusion head adds modest improvements for glaucoma, AMD, and hypertension, but systemic disease prediction remains difficult, especially hypertension on external data. If correct, the study gives an evidence-based ranking and a reusable evaluation protocol for future ophthalmic foundation models.","feed_headline":"DINORET and RetiZero top four eye AI models in head-to-head test","feed_subtitle":"FusionFM benchmark shows gated fusion lifts glaucoma, AMD, and hypertension predictions only modestly; systemic disease remains hard.","key_machinery":"FusionFM is the central object: an evaluation suite that standardizes datasets and splits across four foundation models, plus two fusion approaches. The load-bearing fusion mechanism is the gating-based head, which learns per-sample weights over the feature vectors of RETFound, VisionFM, RetiZero, and DINORET; this is the component responsible for the reported gains on glaucoma, AMD, and hypertension. The standardized multi-country evaluation protocol is what makes the cross-model ranking meaningful.","core_discovery":"The paper's central claim is that not all ophthalmic foundation models are interchangeable. When evaluated on standardized multi-country datasets with identical protocols, DINORET and RetiZero achieve superior AUC and F1 across both ophthalmic diseases and systemic diseases predicted from retinal images, while RETFound and VisionFM trail. RetiZero shows the strongest performance on external datasets. The paper also claims that a gating-based fusion of the four models' features yields small but consistent gains for glaucoma, age-related macular degeneration, and hypertension, indicating that the models carry complementary information even when their single-model rankings differ. The authors p","pith_inferences":["A reader should not read the modest fusion gains as proof that the individual models are redundant; a natural follow-up would inspect the learned gating weights per disease to see whether gains come from one dominant model or from genuine complementarity.","The ranking likely depends on how labels, preprocessing, and data splits are standardized across models; a testable extension would rerun the four models on one public dataset with a fixed preprocessing pipeline and multiple splits to see whether the DINORET/RetiZero advantage persists.","Because hypertension remains hardest on external data, the retinal signal for hypertension appears weak or site-specific; a follow-up could test whether domain adaptation or federated training on external cohorts closes the gap.","The paper's contribution may end up mattering more as a benchmarking protocol than as a deployable fused model, since the fusion improvements are small and task-specific."],"forward_implications":["A clinic or research group choosing a single ophthalmic foundation model would be better served by DINORET or RetiZero than by RETFound or VisionFM on the evaluated tasks.","RetiZero's external-cohort advantage makes it the safer default for deployment on images from unseen sites, devices, or populations.","Fusing models can recover a few additional points of AUC for glaucoma, AMD, and hypertension, but not enough to make systemic disease prediction clinically reliable on its own.","Systemic disease prediction from retinal images, especially hypertension on external data, remains an open problem even with fusion.","Future ophthalmic foundation model papers should report both within-dataset and external-cohort performance and compare against the models ranked here."],"supporting_citations":[],"fun_headline_variants":["DINORET, RetiZero beat RETFound, VisionFM across eye tasks","Fusing eye models gives small boost for glaucoma, AMD, hypertension","RetiZero generalizes best on external eye datasets, study finds","Eye AI fusion gating helps some diseases, lags on hypertension","Top eye foundation models named: DINORET and RetiZero lead"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The standardized multi-country datasets are labeled, preprocessed, and split equivalently for all four foundation models, so that AUC differences reflect model quality rather than data artifacts, and the external cohorts reflect the training-domain distribution closely enough for generalization claims to hold.","fun_headline_variants_meta":{"raw":{"variants":["DINORET, RetiZero beat RETFound, VisionFM across eye tasks","Fusing eye models gives small boost for glaucoma, AMD, hypertension","RetiZero generalizes best on external eye datasets, study finds","Eye AI fusion gating helps some diseases, lags on hypertension","Top eye foundation models named: DINORET and RetiZero lead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1096,"prompt_tokens":798,"completion_tokens":298,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":202}},"tokens_in":542,"tokens_out":298,"duration_ms":4408,"temperature":1.0,"reasoning_tokens":202,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:05:32.557816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or fine-tune each of the four foundation models on one shared public retinal dataset with a single fixed preprocessing pipeline and several random splits; if RETFound or VisionFM then match or beat DINORET and RetiZero, the reported ranking is an artifact of dataset handling rather than a property of the models. For the external-generalization claim, the test is whether RetiZero's advantage over DINORET reproduces on an unseen dataset from a different camera and ethnicity mix.","supporting_citations":[],"review_version":1}