{"id":"13807fb1-9111-462a-bf2e-9ad97f77c659","arxiv_id":"2607.02569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Last-layer Bayesian and SNGP heads on frozen RETFound improve APTOS selective-referral safety, but false-negative gains are not unique to Bayesian modeling and SNGP fails under APTOS-to-DDR transfer.","lead":"Uncertainty-aware last-layer heads on frozen RETFound features can cut missed referable diabetic-retinopathy cases on one public dataset via selective referral, but the gains weaken and can fail under shift to a second dataset. The work shows why screening claims need safety-coverage tradeoffs and external validation, not accuracy alone.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Selective-referral gains rest on validation-chosen uncertainty rankings and coverage without held-out confirmation of the ranking quality.","rationale":"The paper’s mixed claim is carefully hedged and supported by threshold controls, native DDR weakening, and a clear negative transfer result; those pieces hold. The reader correctly flags frozen last-layer + single external dataset as a limitation of the testbed. The more immediate load-bearing soft spot for the *strongest* safety claim (APTOS selective referral to zero accepted FN) is validation-driven choice of operating point and uncertainty signal without a held-out check that the ranking quality is stable. That does not overturn the methodological point that safety–coverage and second-dataset checks are needed, but it keeps the verdict CONDITIONAL and slightly sharpens why: the internal selective-referral headline needs selection-robust confirmation, not only multi-seed bars or more external cohorts. Agreement with the reader is partial because representation/shift fairness is real, yet the ranking-selection issue more directly underwrites the zero-FN selective-referral result.","tokens_in":9022,"tokens_out":583,"duration_ms":6624,"concrete_test":"Freeze the APTOS SNGP sensitivity checkpoint and its entropy ranking; recompute accepted-case FN/specificity at the same ~80% coverage on a held-out APTOS partition (or nested test fold never used for checkpoint/signal selection). If accepted FN rises above 0 or specificity falls materially relative to Table/Results 5.3, the internal selective-referral claim weakens and should be reported with selection-adjusted uncertainty.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim depends on selective referral (not just thresholding) as the clinically meaningful safety gain: on APTOS, SNGP entropy at ~80% coverage deferred ~20% of cases and drove accepted-case FN to 0 with high accepted specificity (Results 5.3; abstract). That claim requires that the uncertainty ranking itself is reliable and not an artifact of choosing the signal and coverage on the same validation set used for model selection (Bayes/SNGP sens vs val-loss checkpoints in Tables 3–4; Methods 3.4). The paper already shows ranking can collapse under shift (APTOS\to DDR SNGP leaves hundreds of accepted FN at the same coverage; Table 6, Fig. 5), so the internal zero-FN result is load-bearing only if the APTOS ranking generalizes beyond the selection set. Without a fixed protocol (pre-specified signal + coverage, or nested holdout), the headline selective-referral improvement can be partly selection-driven rather than a property of last-layer uncertainty.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"This paper evaluates uncertainty-aware last-layer adaptation of frozen RETFound features for binary referable diabetic retinopathy screening on APTOS 2019 and DDR. It compares a cached-feature softmax head, temperature scaling, variational Bayesian last-layer heads, a diagonal Laplace approximation, and an SNGP-style cached-feature head. The central claim is carefully mixed: on APTOS, uncertainty-aware heads improve high-sensitivity and selective-referral operating points (notably ~20% deferral with zero accepted-case false negatives and high accepted specificity for SNGP entropy), but false-negative reduction is not unique to Bayesian modeling because threshold lowering can achieve the same at high false-positive cost; on DDR, native Bayesian heads qualitatively reproduce the direction with weaker tradeoffs, while an APTOS-trained SNGP checkpoint fails under direct transfer. The authors conclude that trustworthy screening claims require explicit safety–coverage evaluation and second-dataset validation under shift.","tokens_in":9311,"tokens_out":1389,"duration_ms":18372,"significance":"If the results hold, the paper is a useful empirical contribution to safety-centered evaluation of foundation-model adaptation in medical imaging. Its main value is not a new SOTA method but the disciplined framing: threshold-sweep controls, selective-referral metrics, and an explicit negative transfer result that prevent overclaiming uniqueness of Bayesian last-layer uncertainty. The public repository, locked summary tables, and reproducible asset scripts are genuine strengths. The work is incremental methodologically (frozen features + standard last-layer uncertainty families) but addresses a practical gap—how screening systems should be judged when false negatives and human-review burden matter more than aggregate accuracy. That framing is relevant to clinical ML evaluation even if the specific last-layer recipes are not novel.","major_comments":[{"comment":"Results §5.3 and Methods §3.4: the strongest APTOS claim (SNGP predictive entropy at ~80% coverage → zero accepted-case FN with specificity 0.9146) depends on an uncertainty ranking and coverage chosen on the same validation set used for model/checkpoint selection (Tables 3–4: sens vs val-loss checkpoints). Without a pre-specified signal+coverage protocol or nested holdout confirming ranking quality, the headline selective-referral gain can be partly selection-driven. Given that the paper itself shows ranking collapse under shift (Table 6, Fig. 5: APTOS→DDR SNGP still leaves hundreds of accepted FN at the same coverage), the internal zero-FN result needs a fixed evaluation protocol or held-out confirmation to support the safety claim as stated.","section":null},{"comment":"Methods §3.2–3.3 and transfer protocol §4.3 / Table 7: attributing the APTOS→DDR SNGP failure primarily to last-layer uncertainty under shift is under-supported. The experiment freezes RETFound_mae natureCFP 1024-d features and restores an APTOS-fitted diagonal precision state without refit. Representation mismatch, domain-specific random-feature geometry, or the non-refit precision proxy could dominate the collapse (sens 0.2701, FN 616). The paper should either (i) re-fit the last-layer precision/head on DDR as a control, or (ii) clearly limit the claim to “this cached-feature SNGP checkpoint does not transfer without retraining,” rather than treating the result as general evidence about uncertainty robustness under shift.","section":null},{"comment":"Results Tables 3–6: key safety metrics (FN, sensitivity, selective-referral accepted-case FN) are reported as point estimates on single fixed splits (APTOS val/test n=366; DDR val/test ~1878–1879) with no multi-seed variance, bootstrap intervals, or McNemar-style paired tests. On APTOS, moving from 17 FN (softmax) to 3 FN (SNGP sens) or to 0 accepted FN under deferral is load-bearing for the internal-safety narrative; without uncertainty on those counts, it is hard to judge whether the gains are stable or split-specific. At minimum, report variability over seeds or resampling for the primary safety operating points.","section":null}],"minor_comments":[{"comment":"Abstract and §5.1: “deferred approximately 20 percent of cases” is clear, but the corresponding APTOS selective-referral table is not numbered alongside the DDR Table 6; adding an APTOS selective-referral table (coverage, FN, Spec, Acc) would make the strongest result easier to verify.","section":null},{"comment":"Methods §3.3: variational Bayesian and Laplace heads are described only at a high level (diagonal Gaussian posteriors, KL term, diagonal Laplace). Specify prior/posterior parameterization, number of MC samples at inference, and KL weight so that the comparison is reproducible from the text alone, not only the repository.","section":null},{"comment":"Figure 1–6 captions are informative, but several figures are referenced without axis units or exact operating-point labels in the text; ensure each figure states the dataset split (val vs test) used.","section":null},{"comment":"Related Work: FusionFM [Zou et al., 2025] is cited as arXiv:2508.11721; if that preprint post-dates or is concurrent with this work, clarify the relationship so readers do not infer dependence.","section":null},{"comment":"Table 1 / §3.1: binary mapping (grades 0–1 non-referable; 2–4 referable) is clinically standard but should briefly note any disagreement with alternative referable definitions used in prior DR screening papers for comparability.","section":null},{"comment":"§6.5 Limitations correctly notes single external dataset and frozen backbone; consider also stating that temperature scaling cannot change ranking-based confusion matrices at fixed threshold (already in §3.3) earlier when interpreting Softmax+Temp rows in Tables 3–4.","section":null}],"recommendation":"major_revision","confidential_remarks":"Fit is reasonable for a methods/empirical medical-imaging or trustworthy-ML venue that values careful negative results; less so for a pure CV venue seeking architectural novelty. The manuscript is unusually honest about mixed outcomes, which is a plus. The main risk is over-reading the APTOS selective-referral headline without a fixed protocol. I would not reject on novelty grounds alone if the revision tightens evaluation protocol and transfer interpretation."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that this is not a new method paper. It is a careful, safety-centered bake-off of standard last-layer uncertainty heads on frozen RETFound features for referable DR, with a clear negative transfer result. That is still worth reading if you care about how people overclaim uncertainty in medical imaging.\n\nWhat is new is the measurement package, not the architecture. Cached RETFound features, softmax, temperature scaling, variational last-layer Bayes, diagonal Laplace, and an SNGP-style RFF head are all known pieces. The contribution is the comparison under explicit safety–coverage metrics plus the APTOS→DDR SNGP failure (sensitivity collapses to ~0.27, hundreds of accepted false negatives at 80% coverage). The author does not hide the control that matters: ordinary threshold lowering also drives false negatives to zero, at huge false-positive cost. Selective referral is the stronger internal story—on APTOS, ~20% deferral with SNGP entropy gets accepted-case FN to zero while keeping high accepted specificity. That distinction is clinically real and cleanly reported.\n\nSoft spots are real but proportionate. Operating points are validation-selected (sens vs val-loss checkpoints), and the strongest selective-referral claim is not locked by a pre-specified signal/coverage protocol or nested holdout. The stress-test note is fair on that point. The testbed is also narrow: frozen 1024-d features, limited uncertainty families, one external dataset, no multi-seed error bars, and tables reconstructed from locked summaries rather than full raw logs. Those limit how far you can push “screening-ready uncertainty,” but they do not erase the internal gains or the negative transfer evidence. Citations are standard and appropriate; math is light and not load-bearing; code and public data are pointed to.\n\nThis is for people who evaluate defer-to-human pipelines or who keep seeing in-distribution uncertainty papers without shift checks. I would send it to peer review. It is honest, reproducible enough for the claim it actually makes, and the mixed result is more useful than another accuracy-only RETFound fine-tune. Engage if that is your lane; skip if you only want new inference machinery.","headline":"Useful empirical safety hygiene paper: last-layer uncertainty helps on APTOS selective referral, fails under APTOS→DDR transfer, and the author is honest about both.","tokens_in":9929,"tokens_out":537,"would_cite":true,"duration_ms":5419,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Uncertainty-aware last-layer heads can improve safety operating points for referable diabetic retinopathy screening, but only internal gains that fail under dataset shift are not enough for trustworthy claims.","keywords":["diabetic retinopathy screening","RETFound","uncertainty estimation","last-layer adaptation","selective referral","dataset shift","Bayesian neural networks","SNGP"],"falsifier":"A controlled re-run that fine-tunes or re-fits the same uncertainty families on multiple independent external retinal cohorts and still finds that selective-referral ranking and zero-accepted-false-negative operating points collapse under shift would falsify the claim that last-layer uncertainty plus safety-coverage evaluation is enough to support trustworthy screening claims in this setting.","tokens_in":9864,"feed_emoji":"👁️","tokens_out":692,"duration_ms":6356,"temperature":0.7,"pith_summary":"This paper asks a practical safety question about frozen RETFound retinal features plus small last-layer heads for referable diabetic retinopathy screening: can uncertainty-aware heads produce safer operating points than a deterministic cached-feature baseline, and do those gains survive a second dataset? On APTOS, uncertainty-aware heads improved high-sensitivity and selective-referral behavior; the strongest selective-referral result deferred about 20% of cases and drove accepted-case false negatives to zero while keeping high accepted-case specificity. The paper also shows that ordinary threshold tuning can cut false negatives at large false-positive cost, so that reduction is not unique to Bayesian modeling. On DDR, native Bayesian heads moved in the same qualitative direction but with weaker tradeoffs, and an APTOS-trained SNGP-style checkpoint transferred poorly and failed as an external selective-referral signal. The central message is that safety-coverage tradeoffs and second-dataset validation under shift are required before claiming trustworthy retinal screening behavior.","feed_headline":"Uncertainty heads cut missed DR cases, then fail under shift","feed_subtitle":"Internal selective referral hit zero accepted false negatives; second-dataset checks expose the limit.","key_machinery":"Uncertainty-aware last-layer adaptation on frozen 1024-d RETFound features: cached softmax, temperature scaling, variational Bayesian and diagonal Laplace last-layer heads, and an SNGP-style random-feature head, evaluated by full-coverage metrics, threshold sweeps, and selective-referral coverage/referral tradeoffs on APTOS with native and transfer checks on DDR.","core_discovery":"Uncertainty-aware last-layer heads on frozen RETFound features can improve internal safety-oriented operating points for referable diabetic retinopathy screening, especially selective referral, but false-negative reduction is not unique to Bayesian modeling and uncertainty signals that look useful in-distribution can fail under external dataset shift; therefore trustworthy claims require explicit safety-coverage evaluation and second-dataset validation.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Uncertainty heads zero accepted DR misses on APTOS, fail under shift","Last-layer uncertainty cuts FN in-distribution; transfer breaks on DDR","Selective referral hits zero accepted false negatives internally only","Bayesian last layers aid DR safety points yet collapse under dataset shift","Uncertainty signals help APTOS referral but not trustworthy across sites"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The load-bearing premise is that frozen RETFound features plus small last-layer heads are a fair and sufficient testbed for screening-oriented uncertainty claims under shift, so observed internal gains and transfer failures can be attributed mainly to last-layer uncertainty rather than to representation mismatch or a single external dataset.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty heads zero accepted DR misses on APTOS, fail under shift","Last-layer uncertainty cuts FN in-distribution; transfer breaks on DDR","Selective referral hits zero accepted false negatives internally only","Bayesian last layers aid DR safety points yet collapse under dataset shift","Uncertainty signals help APTOS referral but not trustworthy across sites"]},"model":"grok-4.5","effort":"low","cost_usd":0.003394,"raw_usage":{"total_tokens":1159,"prompt_tokens":794,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":33940000,"prompt_tokens_details":{"text_tokens":794,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":295,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":794,"tokens_out":70,"duration_ms":3354,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T10:18:46.018395+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A controlled re-run that fine-tunes or re-fits the same uncertainty families on multiple independent external retinal cohorts and still finds that selective-referral ranking and zero-accepted-false-negative operating points collapse under shift would falsify the claim that last-layer uncertainty plus safety-coverage evaluation is enough to support trustworthy screening claims in this setting.","supporting_citations":[],"review_version":1}