{"id":"f4733605-ae16-4107-8f00-ed6255f4f07e","arxiv_id":"2412.14424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Permuting adapter neurons before averaging improves federated vision-language fine-tuning under heterogeneous medical clients compared to prior PEFT-FL baselines.","lead":"FedPIA presents a federated fine-tuning method that permutes adapter neurons before averaging them, using Wasserstein barycenter ideas, for vision-language models trained across medical clients. The authors report consistent gains over prior federated parameter-efficient fine-tuning baselines across five medical vision-language tasks and dozens of datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control isolates the Wasserstein matching: random-permutation ablation and loss-barrier measurement are missing, so the central basin-alignment claim in Eq. (1) is unsupported.","rationale":"The paper's central contribution is the permutation-and-integration mechanism: it claims that Wasserstein-based layerwise matching puts heterogeneous client adapters into the same functional basin before averaging. This is what distinguishes FedPIA from naive FedAvg over adapters, and it is the premise behind Eq. (1), the client-level activation matching, and Fig. 2(c). However, the only evidence for this mechanism is the ablation in Table 5, which turns the entire server- or client-side PIA on and off. That ablation cannot tell whether the specific permutation computed from Euclidean distances or activations is responsible for the gains, or whether any permutation (or even the additional integration step alone) would produce a similar effect. If random permutations perform almost as well, then FedPIA may still be a useful empirical heuristic, but the mechanistic claim about functional basins and Wasserstein barycenters would be unsupported. The convergence curves in Figs. 3 and 4 only show aggregate loss behavior; they do not measure basin overlap. The proposed random-permutation control directly tests the necessity of the matching objective, and the loss-barrier measurement tests whether the alignment actually produces linear mode connectivity. This is a concrete, feasible experiment that would settle the concern. I agree with the reader's weakest assumption: the matching heuristic is the least secure link. The verdict remains CONDITIONAL because the method could still be validated by adding this control and the missing statistical details, and no fatal flaw is established by the current text.","tokens_in":18056,"tokens_out":14236,"duration_ms":129680,"concrete_test":"Run FedPIA (ViLT, Houlsby adapter) on Task 1 exactly as described, but replace the optimization in Eq. (1) and the client-side activation cost with independently drawn random permutations per client, keeping all other steps identical; report final accuracy and also the loss barrier along linear interpolation between each aligned client adapter and the FedAvg anchor. If the random-permutation run is within approximately 1 point of FedPIA's reported 46.66 mean, or the barriers are comparable, the Wasserstein matching is not the driver of the claimed gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing component of FedPIA is the claim that the specific layerwise permutation computed from Euclidean distances between incoming weight rows (server) or mean activations on a random batch (clients) aligns separately trained adapters into a common functional basin, so that averaging is coherent. Table 5 only ablates the whole PIA block (w/o server PIA, w/o client PIA); it never tests whether the particular matching matters. If replacing the Wasserstein permutation with an arbitrary independent random permutation per client gives the same accuracy and the same loss-barrier profile, then the improvement attributed to Wasserstein barycenters actually comes from any permutation-induced smoothing or from the client-side global/local integration, and the central mechanistic claim in Eq. (1) and Fig. 2(c) is unsupported. The paper offers convergence curves instead of a basin-overlap measurement, which cannot distinguish these alternatives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedPIA, a federated parameter-efficient fine-tuning framework for vision-language models in medical imaging. On the server, each client adapter is layerwise permuted via optimal transport to match a FedAvg-initialized global adapter before integration, and on each client the global adapter is permuted to match the local adapter (using activation-based costs) before being combined with it. The method is evaluated on five medical vision-language FL tasks -- VQA, image/text disease classification, and a mixed task -- using two VLM backbones and multiple PEFT adapter types, with claims of consistent improvements over PEFT-FL baselines and parity or gains over full fine-tuning.","tokens_in":18084,"tokens_out":5642,"duration_ms":50799,"significance":"If the reported results are robust, the contribution is practically valuable: FedPIA is a simple, communication-friendly add-on to existing PEFT-FL pipelines that requires no extra training overhead and is orthogonal to the aggregation scheme. The experimental scope is unusually broad (48 datasets, two backbones, five task settings), which strengthens the empirical case. However, the paper's central mechanistic claim -- that Wasserstein-based permutation aligns adapters into a common functional basin -- is not isolated by the provided ablations, and the absence of multi-seed results and of reported hyperparameters limits the reproducibility and strength of the headline claims.","major_comments":[{"comment":"Equation (1) defines the aligned weights for layer l in terms of the previous-layer permutation matrix P^(l-1)_k, but the paper never specifies the boundary condition for the first adapter layer. Without an explicit definition of P^(0)_k, the server-side alignment is not fully defined for l=1. Please state the boundary condition (e.g., P^(0)_k = I) and confirm that the normalization diag(1/beta^(l-1)) is applied consistently for that layer.","section":"Methodology, Eq. (1)"},{"comment":"The central mechanistic claim is that the OT-based permutation brings separately trained adapters into the same functional basin so that averaging is coherent. Table 5 only ablates the presence or absence of the entire server-side and client-side PIA blocks; it does not test whether the specific matching matters. A control with independent random permutations per client and a loss-barrier or mode-connectivity measurement between the permuted adapters and the anchor is needed to support the basin-alignment interpretation in Fig. 2(c). Without such a control, the observed gains could also be explained by any permutation-induced smoothing or by the integration step itself.","section":"Table 5 and Fig. 2(c)"},{"comment":"All reported numbers appear to come from a single run per configuration. Several per-dataset differences are small (e.g., Table 1, VQA-Med 2019 overall: FedPIA (Houlsby) 60.26 vs full fine-tuning 60.43), and there are no error bars, multiple seeds, or significance tests. Given the paper's claim of consistent and robust gains over baselines, at least 3-5 seeds with mean/std, or paired significance tests over clients/tasks, are needed to support that claim.","section":"Tables 1-5"},{"comment":"Two hyperparameters are introduced but never reported: gamma in the server integration formula (the weighted average with exp(-gamma ||W~_k - W_G||_2)) and m, the batch size used for the activation supports in client-level PIA. Values or a sensitivity analysis for both are needed; without them the experiments are not fully reproducible.","section":"Methodology, server-level and client-level PIA"}],"minor_comments":[{"comment":"In the training details, the ALBEF backbone is cited as '( ?)'; the reference is missing and should be added.","section":"Training and Implementation Details"},{"comment":"The method is described as exploiting Wasserstein barycenters, but the actual computation is pairwise optimal-transport matching to an anchor followed by averaging; the relationship to true barycenter computation should be clarified.","section":"Methodology title"},{"comment":"The caption states that 'peaks and troughs represent the losses at the start and end of each communicating round,' but the exact plotted quantity is not described in the text; please clarify what is shown.","section":"Figure 4"},{"comment":"The ablation row labeled 'w/ weight-based PIA' is not defined in the main text; the reader must infer that it replaces activation-based costs with weight-based costs in client-level PIA. Please define it explicitly.","section":"Table 5"},{"comment":"The abstract claims 'over 2000 client-level experiments,' but the main text does not provide a count or breakdown of how this number is obtained; please include a calculation in the experimental setup.","section":"Abstract and Experiments"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is reasonable, but the main text currently overstates the evidence: the basin-alignment mechanism is not isolated, all results are single-run, and two method hyperparameters are unreported. I would be willing to reconsider after the authors add the random-permutation/loss-barrier control, multi-seed results, and explicit hyperparameter reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the take: FedPIA is a genuine new application of optimal-transport-based neuron permutation to adapter fusion in multimodal federated learning, and the evaluation is unusually broad. The main gap is that the ablation never isolates the OT matching itself, so the central basin-alignment mechanism is asserted rather than demonstrated.\n\nWhat's new: prior PEFT-FL methods like FedDAT rely on dual adapters and mutual knowledge distillation; FedPIA instead permutes adapter neurons to a common anchor before averaging, on both the server (client adapters to the FedAvg global) and in clients (global adapter to the local adapter). That two-sided alignment is not in the cited literature. The experiments cover five task settings, 48 datasets, two VLM backbones, and over 2000 client-level runs; the reported gains over FedDAT and vanilla PEFT-FL are consistent across tables. The ablation also shows that both server and client PIA matter, and that activation-based client permutation beats weight-based, which supports that design decision.\n\nSoft spots, in proportion: The stress-test note is right. Table 5 removes whole PIA blocks but never tests whether the specific OT-computed permutation is what helps. A random-permutation control (with matching permutations on incoming/outgoing weights to preserve the function) plus a loss-barrier measurement would isolate the effect of the matching itself. Without that, the improvement could come from any permutation-induced smoothing rather than from Wasserstein alignment. That is the load-bearing concern, and it is not resolved by the convergence curves alone. Second, all results are single-run with no error bars or significance tests; some differences are small, e.g., Task 4 full fine-tuning vs FedPIA. Third, Eq. (1) leaves boundary details undefined (the layer-1 permutation, column marginal normalization), and hyperparameters gamma and activation batch size m are unreported, which limits reproduction. Fourth, the conclusion overstates the full-fine-tuning comparison: in Tasks 2 and 3, full fine-tuning is actually ahead. Minor: the ALBEF reference in the training details is missing.\n\nNone of these are fatal. The idea is plausible, the empirical material is valuable, and the gaps are addressable in revision: add the random-permutation control, report repeated seeds with confidence intervals, specify the boundary conditions and hyperparameters, and temper the full-FT claim. I'd send it to peer review. It will be a useful baseline and reference for OT-based PEFT-FL, and a reading group would get good discussion out of the permutation step.","headline":"A real new entry in PEFT-FL with a broad experimental sweep, but the central Wasserstein-permutation mechanism is not isolated by any control and the full-fine-tuning claim is overreached.","tokens_in":18739,"tokens_out":3587,"would_cite":true,"duration_ms":30073,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedPIA permutes each client's adapter neurons into a common coordinate system before federated averaging, and reports that this consistently beats prior PEFT-FL baselines and can match or exceed full fine-tuning.","keywords":["federated learning","parameter-efficient fine-tuning","adapters","Wasserstein barycenter","optimal transport","vision-language models","medical visual question answering","data heterogeneity"],"falsifier":"Measure the loss landscape between a client adapter and the global adapter produced by FedPIA: if the linear interpolation barrier is not lower than the barrier with naive averaging, the permutation is not actually landing in a shared basin. A second check is to replace the client's random batch with a fixed public batch; a sharp performance drop would show the alignment depends on incidental input samples rather than stable neuron semantics.","tokens_in":17748,"feed_emoji":"🩺","tokens_out":8388,"duration_ms":67757,"temperature":0.7,"pith_summary":"FedPIA is a method for fine-tuning large vision-language models across hospitals and clinics without sharing raw data. The paper claims that separately trained lightweight adapters fail to blend under data, modality, and task heterogeneity because their neurons lack a one-to-one correspondence, so averaging them mixes unrelated features. Its fix is to permute each client adapter's neurons to align with a common reference before aggregating, using layerwise Wasserstein-barycenter matching in the server and again in each client. Across five medical federated settings built from 48 datasets, FedPIA is reported to outperform prior PEFT-FL baselines and, in a task-heterogeneous setting, to exceed full fine-tuning. If true, this makes decentralized fine-tuning of medical foundation models practical without centralized data collection.","feed_headline":"Permuting adapter neurons before averaging beats federated PEFT","feed_subtitle":"Alignment by optimal transport lets heterogeneous medical clients share one coherent adapter.","key_machinery":"The load-bearing object is the layerwise permutation matrix $P_k^{(\\ell)}$, obtained by minimizing the Wasserstein distance $W^{(\\ell)}(\\mu_k^{(\\ell)}, \\nu^{(\\ell)}, C_k^{(\\ell)})$ between probability measures whose supports are adapter neurons. In the server, each neuron's support is its incoming weight row; in the client, the support is the mean activation over a random batch. Equation (1) uses this matrix to align the client adapter weights before integration, and the weighted average of aligned adapters acts as a Wasserstein barycenter. Its role is to break the permutation-invariance barrier that makes naive averaging of independently trained adapters incoherent.","core_discovery":"The paper's central claim is that the failure of naive adapter averaging in heterogeneous federated learning is largely a permutation-alignment problem. Because each client trains its adapter on different data, modalities, and label pools, corresponding neurons sit at distant positions in weight space, and averaging without alignment mixes unrelated features. FedPIA applies a two-step alignment: in the server it permutes each client adapter toward a FedAvg-initialized global adapter, and in each client it permutes the global adapter toward the local adapter before combining them. The paper reports that this consistently outperforms the adaptive PEFT-FL baselines and can match or surpass full fine-tuning, suggesting that preserving client-specific knowledge through alignment is more effective than diluting updates across all parameters.","pith_inferences":["Extension: using a small shared public batch for the client-level activation matching might stabilize the permutation across rounds at no extra privacy cost; the paper's own ablation shows activation-based matching beats weight-based matching, so this is a natural next test.","Extension: because the method exploits permutation symmetry of adapter neurons, it should be tested on LoRA-style low-rank factors after expanding them to full rank; permuting unexpanded A/B factors may not share the same symmetry.","Extension: the convergence plots suggest FedPIA damps round-to-round loss spikes; quantifying oscillation amplitude would give a metric for comparing alignment methods beyond final accuracy."],"forward_implications":["Federated PEFT methods that currently average adapters without alignment can add the permutation step as a drop-in module, since the paper shows either server-side or client-side permutation alone beats the no-permutation baseline.","The method is orthogonal to the choice of aggregation scheme, so the reported gains should persist when FedAvg is replaced by proximal or control-variate federated optimizers.","Because alignment requires no retraining or mutual knowledge distillation, FedPIA adds no training overhead relative to naive adapter averaging, which matters for low-resource medical clients.","In the task-heterogeneous setting the reported gains over full fine-tuning imply that aligned adapter aggregation can replace full fine-tuning for federated medical vision-language applications."],"supporting_citations":[{"why":"supplies the optimal-transport model-fusion machinery that FedPIA adapts for layerwise adapter permutation.","marker":"Singh and Jaggi 2020"},{"why":"provides the Wasserstein-barycenter view that averaging in transport space lands models in the same functional basin.","marker":"Akash, Li, and Trillos 2022"},{"why":"FedAvg is the aggregation scheme used to initialize the global adapter and as the chosen integration baseline.","marker":"McMahan et al. 2017"},{"why":"FedDAT is the state-of-the-art multimodal PEFT-FL baseline that FedPIA compares against and reports gains over.","marker":"Chen et al. 2024"},{"why":"defines the bottleneck adapter architecture used in the primary FedPIA variant.","marker":"Houlsby et al. 2019"},{"why":"AdapterFusion is a baseline fusion strategy and one of the adapter configurations in the experiments.","marker":"Pfeiffer et al. 2020"},{"why":"Compacter is one of the adapter configurations FedPIA is tested with.","marker":"Karimi Mahabadi, Henderson, and Ruder 2021"},{"why":"OmniMedVQA is the dataset source for the modality-specific eight-client setting in Task 2.","marker":"Hu et al. 2024"}],"fun_headline_variants":["Align adapter neurons before averaging for federated PEFT","Wasserstein barycenters fix adapter averaging in federated learning","Permuting adapters before blending boosts federated fine-tuning","Neuron permutation solves heterogeneous adapter fusion in FL","FedPIA: adapter alignment via permutation beats naive averaging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on row-wise Euclidean distances between weight vectors (server) or mean activations over one random batch (client) being a faithful way to decide which neurons correspond across separately trained adapters; if that matching does not track functional correspondence, the permuted average is no better than naive averaging.","fun_headline_variants_meta":{"raw":{"variants":["Align adapter neurons before averaging for federated PEFT","Wasserstein barycenters fix adapter averaging in federated learning","Permuting adapters before blending boosts federated fine-tuning","Neuron permutation solves heterogeneous adapter fusion in FL","FedPIA: adapter alignment via permutation beats naive averaging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000525,"raw_usage":{"total_tokens":2569,"prompt_tokens":1009,"completion_tokens":1560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":1480}},"tokens_in":625,"tokens_out":1560,"duration_ms":10440,"temperature":1.0,"reasoning_tokens":1480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:15:06.963727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the loss landscape between a client adapter and the global adapter produced by FedPIA: if the linear interpolation barrier is not lower than the barrier with naive averaging, the permutation is not actually landing in a shared basin. A second check is to replace the client's random batch with a fixed public batch; a sharp performance drop would show the alignment depends on incidental input samples rather than stable neuron semantics.","supporting_citations":[{"cited_title":"P.; and Jaggi, M","cited_arxiv_id":null,"evidence_quote":"supplies the optimal-transport model-fusion machinery that FedPIA adapts for layerwise adapter permutation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FedAvg is the aggregation scheme used to initialize the global adapter and as the chosen integration baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FedDAT is the state-of-the-art multimodal PEFT-FL baseline that FedPIA compares against and reports gains over."}],"review_version":1}