Benchmark Evaluation of Federated Learning on Multi-organ Images

Jianchun Zhu; Jin Liu; Junbin Mao; Ludi Li; Xu Tian

Benchmark Evaluation of Federated Learning on Multi-organ Images

T0 review · 2 major / 6 minor · reviewed 2026-07-10 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MobenFL is the broadest federated-learning benchmark for medical images, pairing 20 algorithms with 22 multi-organ datasets and scoring accuracy, speed, privacy and clinical heterogeneity together.

desk verdict Solid multi-organ FL benchmark that actually ships code and multi-axis tables; the fixed five-client skeleton is a real but not fatal limit on the clinical-realism claim. read the letter →

arxiv 2607.08219 v2 pith:NRXWLDD3 submitted 2026-07-09 cs.CV cs.LG

Junbin Mao , Xu Tian , Jianchun Zhu , Ludi Li , Jin Liu This is my paper · ORCID

classification cs.CVcs.LG

keywords FederatedLearningMedicalImageClassificationMulti-organdataPrivacyrobustnessAlgorithmicefficiencyCross-deviceevaluationNon-IID

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Medical AI is blocked by privacy rules and by huge differences in organs, scanners and diseases across hospitals. Federated learning lets centers train together without sharing raw images, yet no shared yardstick exists to compare the flood of new algorithms under realistic clinical messiness. MobenFL fills that gap by running twenty recent federated methods on twenty-two public imaging collections that span twelve body organs and four modalities. Beyond plain accuracy it also times convergence, measures privacy-noise tolerance and tests three real-world mismatches: different diseases of the same organ, different devices imaging the same disease, and different modalities. The resulting scorecards let clinicians pick an algorithm that matches their data diversity, compute budget and privacy needs instead of relying on single-number leaderboards.

What carries the argument

MobenFL itself: a unified code framework that standardizes twenty algorithms (grouped as extra-network, regularization, aggregation and split-learning) and twenty-two datasets under fixed ResNet-18 training, Dirichlet non-IID splits, early stopping, and three specialized clinical-heterogeneity test suites.

What would settle it

Re-run the exact same twenty algorithms on a new multi-center collection whose client count, scanner mix and class imbalance differ sharply from the five-client Dirichlet-0.5 setup and check whether the recommended algorithm rankings reverse.

Watch

Extended reading notes

Core claim

The paper claims that a single open benchmark covering twenty federated algorithms, twenty-two multi-organ datasets and four evaluation axes (accuracy, efficiency, privacy robustness and clinical-scenario transfer) is both feasible and necessary; without it, claims about which algorithm works in the clinic remain incomparable and untrustworthy.

Load-bearing premise

The claim that five clients, ResNet-18, a fixed learning rate and Dirichlet alpha 0.5 already capture the resource limits and data skews of real multi-hospital deployments.

Editorial extensions

If this is right

Clinicians can now select algorithms by primary need: SplitFed or HarmoFL when accuracy under heterogeneity matters most, FedGH when wall-clock time is scarce.
Privacy-preserving deployments gain a concrete ranking: SplitFed keeps almost full accuracy even under strong differential-privacy noise while most others collapse.
Future algorithm papers must report efficiency and privacy numbers on the same multi-organ suite or risk being incomparable.
Benchmark users receive ready hyper-parameter defaults (batch 16, three local epochs) that generalize across the twelve organs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same multi-axis design could be ported to federated segmentation or detection, exposing whether the current accuracy–efficiency trade-offs reverse when pixel-level labels are required.
Because the privacy test injects noise only on the uplink, a follow-up that also privatizes the downlink or uses secure aggregation would test whether SplitFed’s advantage survives stronger threat models.
The observed organ-specific sensitivity (brain and knee drop hardest under non-IID) suggests that organ-aware client clustering or modality-specific personalization layers may be more fruitful than further generic regularizers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is the broadest medical FL classification benchmark we have so far: 20 recent algorithms (including personalization, split, and aggregation variants) run on 22 public datasets spanning 12 organs, with accuracy, wall-clock efficiency, and a simple DP-noise privacy axis, plus three clinical-scenario cuts (cross-disease, cross-device, cross-modality). Code is released. That combination is new relative to FedCBD and FLamby and is useful.

What they did well is the engineering and the reporting. Everything sits in one framework with standardized ResNet-18 + Adam inputs, early stopping, and Dirichlet non-IID. They give p-values on the IID vs non-IID drops, ternary efficiency plots, and a practical recommendation map (Fig. 16). The multi-organ tables make it easy to see that some methods (HarmoFL, SplitFed, FedNP) hold up better than others once you leave the single-organ comfort zone. Self-citation is light and the baselines are the usual ones.

The soft spot the stress-test flags is real but proportional. Section 4.1 freezes five clients, batch 128, lr 1e-4, α=0.5 for every number that appears later. Real multi-center networks are larger, more unbalanced, and have device-specific BN quirks; rankings could shift. Privacy is just Gaussian noise on parameters, not a full DP-SGD or secure-aggregation stack. Hyper-parameters are not tuned per method, so some algorithms are probably under-sold. These are the usual benchmark caveats, not circular math or invented data. The paper still shows clear thinking and honest engagement with the literature.

Who it is for: anyone building or reviewing medical FL methods who needs a ready multi-organ yardstick and open code. It will not rewrite clinical practice, but it will change how new algorithms get compared. I would bring it to reading group, cite the tables when I need a multi-organ reference, and send it to peer review. The fixed skeleton should be called out and ideally stress-tested in revision, but the artifact itself is solid enough to deserve referee time.

Referee Report

2 major / 6 minor

Summary. The manuscript introduces MobenFL, a federated medical-image classification benchmark that unifies 20 recent FL algorithms (grouped into additional-network, regularization, aggregation-strategy and split-learning families) with 22 public 2-D imaging datasets spanning 12 organs. Under a fixed experimental skeleton (ResNet-18 + Adam, lr = 0.0001, batch 128, five clients, Dirichlet α = 0.5, early-stopping patience 10) the authors report accuracy under IID and non-IID partitions, theoretical and wall-clock efficiency, robustness to simple Gaussian differential-privacy noise, and three clinical-heterogeneity scenarios (cross-organ, cross-disease, cross-device). From these results they derive a recommendation map (Fig. 16) that ranks algorithms for different clinical priorities. Code is released.

Significance. If the rankings prove stable, MobenFL would become the most extensive public reference point for FL medical-image classification, filling a clear gap left by FedCBD and Flamby. The multi-organ coverage, inclusion of split-learning and personalization methods, joint reporting of accuracy/efficiency/privacy, and open-source unified framework are genuine strengths that lower the barrier for subsequent comparative work. The contribution is therefore of clear practical value to the medical-AI community, provided the fixed experimental skeleton is shown not to reverse the reported orderings.

major comments (2)

Section 4.1 freezes every experiment to exactly five clients, Dirichlet α = 0.5, ResNet-18 + Adam (lr = 0.0001), batch size 128 and early-stopping patience 10. All accuracy, efficiency and privacy numbers in Sections 4.2–4.5, and therefore the recommendation map of Fig. 16, are generated under this single skeleton. Real multi-center deployments routinely involve 10–50+ sites with highly unbalanced sample sizes and participation rates far from a uniform five-client Dirichlet draw. Because the central claim of a “comprehensive … evaluation framework for the clinical application of FL” rests on the premise that the skeleton fairly represents clinical conditions, at least a limited sensitivity study (varying client count and α on a representative subset of datasets) is required to establish that the reported rankings do not reverse.
Section 4.5.1 implements differential privacy by adding isotropic Gaussian noise scaled by a free factor ε to the transmitted parameters (Eq. 5). This is not a standard (ε,δ)-DP mechanism with clipping and calibrated noise; consequently the privacy-robustness claims and the special status accorded to SplitFed under “strict privacy” cannot be interpreted as formal DP guarantees. Either a proper DP-SGD implementation with reported privacy budgets should be used, or the language should be revised to “noise robustness” rather than “privacy protection capabilities.”

minor comments (6)

Title and abstract contain the typographical error “Feredated”; correct throughout.
Table 2 lists TurboSVM under both Additional Network Structures and Regularization-Driven; clarify the intended category.
Section 3.2.2 states “FedProx, MOON, PGFed and TurboSVM all fall into this category” while TurboSVM is already placed in the additional-network group; remove the inconsistency.
Figure 4 caption and surrounding text refer to “p-value” without stating the statistical test used; specify the paired test and whether multiple-comparison correction was applied.
Several dataset URLs and accession numbers appear only as footnotes; a consolidated data-availability table would improve reproducibility.
The hyper-parameter study in Section 4.6 is performed only for FedAvg; a short note on whether the recommended (batch=16, local_iter=3) setting transfers to the other 19 algorithms would be useful.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmark paper with no derivation chain; rankings arise from direct runs on held-out public data, not from self-referential definitions or fitted-then-predicted quantities.

full rationale

MobenFL is a pure empirical evaluation framework: 20 published FL algorithms are re-implemented under a fixed skeleton (ResNet-18, Adam, five clients, Dirichlet α=0.5) and scored on 22 public multi-organ datasets for accuracy, wall-clock time, and differential-privacy robustness. No equation defines a quantity in terms of a parameter later presented as a prediction; no uniqueness theorem is imported from the authors’ prior work; self-citations are limited to standard baselines (FedAvg, FedProx, etc.) that are independently reproducible. The experimental skeleton itself is an assumption about clinical realism, not a circular step. Consequently the paper contains zero instances of the six circularity patterns.

Assumptions & free parameters 5 free parameters · 3 assumptions · 1 invented entities

A benchmark paper rests on a handful of experimental design choices that are free parameters and on standard domain assumptions about federated averaging and medical data heterogeneity; no new physical entities are postulated.

free parameters (5)

learning_rate = 0.0001
Fixed at 0.0001 for every algorithm and dataset (Section 4.1); never tuned per method.
batch_size = 128
Fixed at 128 (Section 4.1); later sensitivity study shows it strongly affects accuracy.
num_clients = 5
Fixed at 5 for all experiments (Section 4.1); real hospital networks vary widely.
dirichlet_alpha = 0.5
Non-IID partitions generated with α = 0.5 (Section 4.1.1); controls degree of label skew.
dp_noise_scale_epsilon = 0.01 / 0.1
Privacy experiments inject Gaussian noise scaled by ε ∈ {0.01, 0.1} (Section 4.5); not a full DP-SGD accountant.

assumptions (3)

standard math The global objective is the sample-weighted average of local empirical risks (Eq. 1).
Standard FedAvg derivation; used throughout.
domain assumption Medical imaging data across hospitals are non-IID with respect to organ, disease, device and population.
Stated in Introduction and used to justify Dirichlet partitions and clinical-scenario tests.
ad hoc to paper ResNet-18 + Adam is a sufficiently representative backbone for ranking FL algorithms on 2-D medical images.
Chosen once in Section 4.1 and never varied; may favor methods that interact well with residual networks.

invented entities (1)

MobenFL benchmark suite independent evidence
purpose: Unified code base and evaluation protocol that packages the 20 algorithms and 22 datasets.
The paper’s central contribution; independent evidence is the public GitHub repository.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Benchmark Evaluation of Federated Learning on Multi-organ Images." pith.science (2026). https://pith.science/paper/NRXWLDD3

@misc{pith2026260708219,
  author       = {Pith},
  title        = {Pith review of: Benchmark Evaluation of Federated Learning on Multi-organ Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRXWLDD3}},
  note         = {Machine review of arXiv:2607.08219}
}

read the original abstract

The privacy requirements of medical data and its substantial variations across organs and modalities hinder the clinical implementation of medical AI. Federated learning (FL) is a feasible approach to overcome these challenges. Due to the continuous emergence of FL algorithms and the highly heterogeneous nature of medical data, objectively evaluating their performance in real-world clinical settings remains difficult. Therefore, a comprehensive federated medical imaging benchmark, serving as a unified evaluation standard, is crucial for advancing the technology toward reliable clinical application. Existing federated medical imaging benchmarks have not yet adequately incorporated state-of-the-art algorithms, are limited to data from single organs or modalities, and overly emphasize model accuracy, making it difficult to comprehensively assess the overall efficacy of FL in real-world medical environments. To address these challenges, we developed the MobenFL benchmark. This benchmark integrates 20 cutting-edge FL algorithms and 22 medical imaging datasets, covering 12 critical organs across the human body, surpassing existing benchmark in breadth. In terms of evaluation dimensions, MobenFL not only assesses performance but also systematically incorporates key metrics such as algorithmic efficiency and privacy protection capabilities. Additionally, it conducts specialized evaluations for complex real-world clinical scenarios involving different diseases, devices, and imaging modalities, thereby providing a comprehensive and in-depth evaluation framework for the clinical application of FL in the medical field.