{"id":"2148c2d8-44b2-4240-b34e-6a4db3681f98","arxiv_id":"2412.20250","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A federated learning method for brain tumor segmentation using NNMF-based client selection and HSimAgg aggregation reports Dice of 0.73/0.74/0.82 on external data, but lacks baseline comparisons.","lead":"This paper proposes a recommender engine based on non-negative matrix factorization to select which hospitals join each round of federated brain tumor segmentation training, plus a harmonic-similarity weight aggregation rule. The authors report external validation Dice scores but provide no comparison against random selection or standard federated averaging, leaving the claimed improvement unproven.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No baseline or ablation is reported for the proposed NNMF selection + HSimAgg, so the central 'significantly improves' claim is untestable as written.","rationale":"The paper's empirical contribution is a single run on FeTS2022 with 33 collaborators, reporting Dice and Hausdorff metrics on external validation. The abstract's claim is explicitly comparative: the proposed recommender engine 'significantly improves' precision and efficiency. I looked through Section 3 and Tables 2-3 and found no baseline, no ablation, and no standard deviation across runs. The 'batch-wise collaborator selection policy' described in Section 2 references prior work [8] but does not report its quantitative results to compare against. Figure 2 plots simulation time, convergence score, and Dice over rounds for the proposed method only. Thus the central claim that the method improves over previous selection and aggregation schemes cannot be assessed from the evidence presented. This is not a disagreement with the field's consensus; it is an evidentiary gap in the manuscript itself. The reader's weakest-assumption point about Eq. 6 is also important: the formula as printed divides by model parameter vectors element-wise, which is undefined or unstable when parameters are zero or negative, and the expression does not match the harmonic-mean definition described in the text. However, that issue could potentially be fixed or clarified, whereas the missing comparative evaluation cannot be repaired by clarification. If the authors supplied the code and corrected Eq. 6, the aggregation issue could be settled; no amount of code fixes the lack of baselines. For that reason, I regard the missing comparison as the single most load-bearing concern. I credit the paper for using the public FeTS2022 dataset and the OpenFL framework; the concern is not about data provenance or the authors' integrity but about the support for the comparative claim. The reader's verdict of REJECT remains appropriate, so I do not propose changing it, but my emphasis is on the absent baselines rather than Eq. 6 alone.","tokens_in":7785,"tokens_out":5234,"duration_ms":54587,"concrete_test":"Repeat the Section 3 experiment under identical FeTS2022 partitions and hyperparameters for at least four conditions: (1) random 20% client selection + FedAvg; (2) random 20% + HSimAgg; (3) NNMF-based selection + FedAvg; (4) NNMF-based selection + HSimAgg (proposed). Run each condition with at least 5 random seeds and the same 20 communication rounds and the same 219 external validation scans. If the proposed method's mean Dice (ET, TC, WT) does not exceed the best baseline by more than the seed noise (e.g., non-overlapping 95% confidence intervals or a paired significance test), the headline improvement claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is comparative: NNMF-based recommender selection with HSimAgg 'significantly improves the precision and efficiency' of federated learning over prior selection and aggregation schemes (Abstract, Section 1). Nowhere in Section 3 is any baseline or ablation reported. The external-validation Dice scores (ET 0.7298, TC 0.7424, WT 0.8218) and the training curves in Figure 2 describe only the proposed configuration. There is no comparison to random 20% client selection, to plain FedAvg, to the earlier SimAgg cited in [8], or to any other client-selection policy. Because client selection, initialization, and data partitioning are stochastic, a single run without error bars cannot support a significance claim. This absence is load-bearing: if the proposed method were compared and found not significantly better, the abstract's 'significantly improves' claim would be false; as written, the reported results do not distinguish the method from a well-tuned baseline. The reader's Eq. 6 concern is real, but it is secondary: even a corrected aggregation formula would leave the absence of comparative evaluation unresolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a client-selection protocol for federated learning (FL) that uses non-negative matrix factorization (NNMF) on historical client metrics to rank collaborators, alternating between top-20% and bottom-20% selection in successive rounds, together with a new aggregation rule called HSimAgg (harmonic similarity weighted aggregation). The method is evaluated on the FeTS 2022 brain tumor segmentation data (1,251 training mpMRI scans, 219 external validation scans) across 33 simulated collaborators. The reported external validation Dice scores are 0.7298 for enhancing tumor, 0.7424 for tumor core, and 0.8218 for whole tumor after 20 communication rounds. The central claim, stated in the abstract and Section 1, is that the proposed approach significantly improves the precision and efficiency of federated learning for this task.","tokens_in":8038,"tokens_out":3456,"duration_ms":37278,"significance":"If the comparative claims were substantiated, the paper would offer a useful contribution to client selection in federated medical FL, a topic of active interest. The paper has some strengths: it uses a public benchmark (FeTS/BraTS), reports wall-clock time, memory, and energy consumption, provides pseudocode for the aggregation algorithm, and evaluates on an external validation cohort. However, the significance is not yet established. The experimental section reports only the proposed configuration; there is no comparison to FedAvg, random client selection, the authors' prior SimAgg method [8], or any other selection policy. In addition, the aggregation equation labeled as a harmonic mean is not a harmonic mean and involves element-wise division by model parameters, which are not guaranteed to be nonzero or positive. Because both the selection and aggregation components are load-bearing for the claimed improvement, the current evidence is insufficient to support the abstract's 'significantly improves' statement.","major_comments":[{"comment":"The central comparative claim is unsupported because no baseline or ablation is reported. Table 2 and Table 3 contain only the HSimAgg column, and Figure 2 shows training curves for one configuration. There is no comparison to random 20% client selection, to plain FedAvg, to the authors' earlier SimAgg algorithm [8], or to a variant without NNMF selection. Since client selection and data partitioning are stochastic, a single run without error bars or statistical testing cannot support the word 'significantly' used in the abstract and Section 1. The authors should report results for at least FedAvg with random selection, random selection with HSimAgg, and the previous SimAgg selection policy, with multiple seeds and variance estimates.","section":"§3, Tables 2 and 3"},{"comment":"Equation (6) does not define the harmonic mean and is not a well-defined operation on neural-network parameter vectors. A weighted harmonic mean of values p_c with weights w_c is (Σ w_c) / (Σ w_c / p_c), while Eq. (6) computes (1 / Σ (w_c / p_c)) · Σ (w_c · p_c), which is the ratio of a reciprocal sum and a weighted sum. More seriously, the expression divides element-wise by model parameters p_c, which can be zero or negative in a trained neural network, making the aggregated model undefined or numerically unstable. Since HSimAgg is a named contribution and its correctness depends on this equation, the authors must either correct the formula to an actual harmonic-style aggregation over parameter vectors with a well-defined positive transformation, or remove the harmonic-mean claim and describe a mathematically consistent aggregation.","section":"§2.2, Eq. (6)"},{"comment":"The selection algorithm is underspecified at load-bearing points. The text states that NNMF with two components is applied to a 33×4 matrix, that the first latent variable is used, and that odd rounds select the bottom 20% while even rounds select the top 20%, but it never defines how the NNMF scores are converted into a ranking, how the 'exploration and exploitation' weighting is computed, or how the fallback random selection is triggered beyond 'does not yield a result.' The alternating top/bottom schedule is also not justified; selecting the bottom 20% in half of all rounds appears to deliberately choose poor performers and may harm convergence. A precise algorithmic specification of the selection rule and an ablation showing its effect on the final Dice scores are needed.","section":"§2.2, Recommender Engine Collaborator Selection"},{"comment":"Several free hyperparameters of the proposed method are not reported or analyzed. Table 1 lists learning rate, epochs per round, and communication rounds, but not the NNMF latent dimension (set to 2 in the text), the selection fraction (set to 20%), the epsilon in Eq. (2), or the optimizer and loss function. Since the paper claims that NNMF-based selection improves results, a sensitivity analysis over the latent dimension and selection fraction is necessary to show that the reported performance is not an artifact of a single favorable setting.","section":"§3.1, Table 1"}],"minor_comments":[{"comment":"The mathematical notation in the similarity-weight equations is inconsistent: Eq. (2) introduces a quantity written as 's im' with subscripts, while Eq. (3) and Algorithm 1 use different symbols for the same quantity. Please unify the notation and ensure every symbol is defined before use.","section":"§2.2, Eqs. (2)–(5)"},{"comment":"The phrase 'collaborator selection via reinforcement approaches' is inaccurate, because no reinforcement learning is used in the paper. This should be reworded to describe the NNMF-based recommender selection.","section":"§3.2"},{"comment":"The preprocessing description lists both 'brain extraction' and 'skull stripping,' which appear to be redundant; please clarify whether these are distinct steps or the same step described twice.","section":"§2.1"},{"comment":"The text calls HSimAgg 'an advancement of the SimAgg algorithm' but does not state what HSimAgg changes relative to SimAgg beyond the selection mechanism. A brief comparison to the SimAgg formulation in [8] would help the reader assess the novelty.","section":"§2.2, HSimAgg"},{"comment":"The paper would benefit from a reproducibility statement indicating whether the code, model weights, and experimental configuration are publicly available.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"I do not see a direct circularity problem in the evaluation: the selection uses historical validation metrics from previous rounds, while the final evaluation is on an external validation cohort. The main issues are experimental incompleteness and a mathematically ill-defined aggregation equation. Both are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The editor may also wish to consider whether the paper's contribution is sufficiently differentiated from the authors' prior SimAgg work [8] when the authors add the missing comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does not support its central claim because it never compares its method to anything. The abstract says \"significantly improves\" but the results section reports Dice scores for only one configuration. There are no baselines, no error bars, and no ablations. As written, a reviewer cannot tell if the NNMF selection and HSimAgg aggregation do anything beyond a simple FedAvg or random selection.\n\nWhat is new: applying NNMF-based collaborative filtering to client selection in federated learning for brain tumor segmentation. The authors build on their own SimAgg work and propose HSimAgg as a variant. The experimental setup uses standard FeTS 2022 data and OpenFL, and the external validation numbers are reported. That part is fine.\n\nWhere it falls down. The absence of any comparator is the main problem. The paper claims \"significantly boosts\" but there is no statistical support. A single run without variability measures cannot support that language. Also, equation 6 defines the aggregated parameters with a division by model parameters element-wise, which is undefined for zero or negative parameters and is not the harmonic mean as written. This is a real mathematical issue, not a typo. The alternating top/bottom selection rule is arbitrary and not justified. The cold-start claim is also under-explored: they rely on a fallback to random selection but give no analysis of when that matters.\n\nOn the positive side, the paper is readable and the idea is worth pursuing. The authors show some awareness of prior work. But the current manuscript is not ready. The central claim is untestable, and the aggregation equation needs a fix.\n\nWho would benefit: someone working on client selection in FL for medical imaging might look at this for the general idea, but not for evidence. I would not cite it. For peer review, I would send it to a referee only if the editor thinks the authors can supply the missing baselines and correct the math. As is, I'd lean toward desk reject, but the flaws are fixable, so a borderline accept-to-review is defensible.\n\nRecommendation: if this crosses your desk, ask for a major revision with honest comparisons, multiple runs, and a corrected aggregation formula.","headline":"A plausible idea undone by missing baselines and a questionable aggregation formula.","tokens_in":8513,"tokens_out":3282,"would_cite":false,"duration_ms":31567,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A recommender engine based on non-negative matrix factorization selects federated-learning clients by past performance and expertise; a new harmonic-mean aggregation (HSimAgg) blends their model updates, and the method reports…","keywords":["federated learning","client selection","non-negative matrix factorization","recommender system","harmonic mean aggregation","brain tumor segmentation","glioblastoma","medical image segmentation"],"falsifier":"Run HSimAgg on a trained U-Net whose parameter tensors contain an exact zero (for example after pruning or ReLU dead neurons) and check whether the master model in Eq. 6 becomes NaN; if it does, the aggregation rule is not computable as stated without extra safeguards.","tokens_in":7630,"feed_emoji":"🧠","tokens_out":7852,"duration_ms":73768,"temperature":0.7,"pith_summary":"This paper proposes to treat the selection of collaborators in federated learning as a recommendation problem: a non-negative matrix factorization engine ranks hospitals (clients) by their historical Dice scores, loss, participation frequency, and contribution time, and the server picks the top or bottom fifth of the ranking in alternating rounds. To merge the chosen models, the paper introduces HSimAgg, which weights each collaborator by its harmonic similarity to the unweighted average model and by its sample size, then combines updates with a harmonic mean. The claim is that this pairing selects task-aligned experts and reduces the influence of outlier updates, improving both accuracy and efficiency of federated brain tumor segmentation. On a 33-site split of a glioblastoma dataset, the method reports external-validation Dice scores of 0.7298 (enhancing tumor), 0.7424 (tumor core), and 0.8218 (whole tumor). If the claim holds, federated medical networks can converge with fewer, better-chosen participants and fewer wasted communication rounds.","feed_headline":"Recommender-driven client picks lift federated tumor Dice to 0.82","feed_subtitle":"NNMF ranks hospitals by past performance; harmonic aggregation blends their models on a 33-site glioma benchmark.","key_machinery":"The two load-bearing pieces are the NNMF-based recommender and the HSimAgg aggregator. The recommender decomposes a normalized client-by-metric matrix (4 metrics, 33 clients here) into two low-rank latent factors and uses the first latent component to score collaborators; the schedule alternates between the lowest and highest scorers across rounds, with a random-selection fallback for the cold start. HSimAgg computes, for each selected client, the inverse distance of its parameter vector to the unweighted average model (Eqs. 2–3), normalizes these similarity weights, blends them with sample-size weights (Eq. 4), and finally combines the updates through the harmonic-mean operation in Eq. 6, which is meant to reduce the pull of outlier models.","core_discovery":"The paper's central claim is that a recommender engine built on non-negative matrix factorization can rank federated collaborators from a compact matrix of performance and participation metrics, and that alternating between the lowest-ranked and highest-ranked fifth of collaborators across rounds produces faster, more stable convergence than batch-wise random selection. Aggregating the selected models with HSimAgg, which weights each collaborator by its harmonic similarity to the unweighted average model and by its local sample size, then suppresses divergent parameter updates. The reported external-validation Dice scores—0.7298 for enhancing tumor, 0.7424 for tumor core, and 0.8218 for whole tumor—are presented as evidence that task-aligned collaborator selection improves federated brain tumor segmentation.","pith_inferences":["The paper does not include a head-to-head comparison against random selection or standard FedAvg on the same data split, so the claimed improvement in precision is supported by convergence curves and final scores rather than an ablation; a controlled comparison would separate the contribution of selection from that of aggregation.","Equation 6 divides element-wise by model parameters, and if any parameter is zero the reciprocal is undefined; negative parameters would also make a harmonic mean behave unexpectedly, so a reliable deployment would need clipping or an absolute-value-based similarity.","The odd-round policy of selecting the bottom 20% could slow convergence in highly heterogeneous federations, since consistently poor clients are forced in every other round; annealing the exploration rate over time would be a natural extension."],"forward_implications":["Federated segmentation can operate with only 20% of sites contributing in each round while still achieving competitive Dice scores.","New or seldom-seen clients can join through a random-selection fallback, addressing the cold-start problem without requiring prior performance history.","Because HSimAgg weights down parameter vectors that lie far from the average, training becomes less sensitive to outlier or corrupted client updates.","The NNMF-based selection scheme generalizes to other federated tasks wherever per-client validation metrics such as Dice and loss are logged."],"supporting_citations":[{"why":"Supplies the glioblastoma mpMRI dataset and the 33-client data partition used for training and external validation.","marker":"[5]"},{"why":"The earlier adaptive aggregation work whose batch-wise selection and SimAgg algorithm HSimAgg extends and seeks to improve.","marker":"[8]"},{"why":"Defines the Dice and Hausdorff evaluation protocol used to score the federated segmentation results.","marker":"[13]"},{"why":"The open-source federated learning platform on which the described training and aggregation runs were executed.","marker":"[14]"},{"why":"The federated averaging reference that motivates the need for adaptive, similarity-weighted aggregation in non-IID settings.","marker":"[10]"}],"fun_headline_variants":["NNMF recommender boosts federated tumor segmentation to 0.82 Dice","Smart client selection raises federated brain tumor Dice to 0.82","Task-aligned picks improve federated glioblastoma segmentation","Recommender-driven FL lifts tumor Dice scores to 0.82","Harmonic aggregation plus smart picks hit 0.82 tumor Dice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the aggregation in Eq. 6, $p_m = \\sum_c w_c p_c \\,/\\, \\sum_c (w_c/p_c)$, is defined for real neural-network parameters; if any parameter is zero or negative, the division is undefined or the harmonic mean becomes erratic.","fun_headline_variants_meta":{"raw":{"variants":["NNMF recommender boosts federated tumor segmentation to 0.82 Dice","Smart client selection raises federated brain tumor Dice to 0.82","Task-aligned picks improve federated glioblastoma segmentation","Recommender-driven FL lifts tumor Dice scores to 0.82","Harmonic aggregation plus smart picks hit 0.82 tumor Dice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1610,"prompt_tokens":966,"completion_tokens":644,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":567}},"tokens_in":582,"tokens_out":644,"duration_ms":5410,"temperature":1.0,"reasoning_tokens":567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:24:04.865098+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HSimAgg on a trained U-Net whose parameter tensors contain an exact zero (for example after pruning or ReLU dead neurons) and check whether the master model in Eq. 6 becomes NaN; if it does, the aggregation rule is not computable as stated without extra safeguards.","supporting_citations":[{"cited_title":"org/10.5281/zenodo.6362409","cited_arxiv_id":null,"evidence_quote":"Supplies the glioblastoma mpMRI dataset and the 33-client data partition used for training and external validation."},{"cited_title":"In: Crimi, A., Bakas, S","cited_arxiv_id":null,"evidence_quote":"The earlier adaptive aggregation work whose batch-wise selection and SimAgg algorithm HSimAgg extends and seeks to improve."}],"review_version":1}