{"id":"8d7e63c1-f106-4a2e-941a-e070a254e122","arxiv_id":"2411.11912","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"F3OCUS combines per-client LNTK layer importance scores with server-side meta-heuristic optimization of layer diversity to improve federated fine-tuning of vision-language models for medical tasks, and releases the 707,962-sample Ultra-MedVQA dataset.","lead":"F3OCUS is a federated learning method that lets each hospital client fine-tune only the most useful layers of a vision-language model, while a server adjusts layer choices so different clients do not rely on the same layers. It is tested on 58 medical image datasets and a new large medical visual question answering dataset, and it reports accuracy gains over prior layer-selection and federated learning baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theoretical basis for LNTK layer importance is internally inconsistent: Eq. 12 uses labels Y where the squared-error gradient is residual Z-Y, so the λ1 ranking is not derived even before the cross-entropy mismatch; the empirical claim currently lacks code and error bars.","rationale":"Reader's weakest_assumption points at the squared-error/cross-entropy mismatch. I found a sharper version: Eq. (12) is internally inconsistent even under the paper's own squared-error setting, because the loss-reduction identity uses Y where the correct factor is Z-Y. This makes the 'theoretical motivation' not merely inapplicable to the experiments but invalid as a derivation. Since the central novelty is explicitly 'theoretically motivate and leverage' the LNTK eigenvalue, this is load-bearing. The empirical comparisons are extensive, and independent support such as code or error bars is absent, so the main support for the practical claim is the table numbers. The proposed test—re-deriving Eq. 12 and correlating λ1 with actual single-step loss reductions under both MSE and CE—would settle whether the ranking criterion has real predictive content. I therefore keep the reader's CONDITIONAL verdict: the concern is addressable but currently unresolved.","tokens_in":20378,"tokens_out":4812,"duration_ms":50004,"concrete_test":"Re-derive Eq. (12) with r = Z - Y under squared loss; if the result is Σ_l Σ_j λ_j^l ((u_j^l)^T r)^2, then test the ranking empirically on Task 1 with LLaVA-1.5: freeze all layers except one, take one local SGD step per layer separately, and record the actual training-loss reduction. Compute Spearman correlation between per-layer principal eigenvalues λ_1^l and per-layer loss reductions. Repeat with the cross-entropy loss used in the paper. If the correlation is weak or absent, the proposed importance score is an unvalidated heuristic and the theoretical motivation does not carry the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing defect is in the theoretical derivation that justifies the LNTK importance score, not merely its transfer to cross-entropy. In Eq. (12), after writing ΔF ≈ ∇_θ F^T ∇_θ F, the authors factor the NTK and declare ΔF ≈ Σ_l Σ_j λ_j^l ((u_j^l)^T Y)^2. For squared-error loss the correct gradient factor is the residual ∇_Z F = Z - Y, not the target Y, so the derived expression should contain ((u_j^l)^T (Z-Y))^2. Eq. (11) has the same issue: the trajectory U^T f(t) converges to U^T y, and the transient is governed by the initial residual f(0)-y, not by y alone. Consequently, ranking layers by λ_1 alone requires an additional, unstated and untested spectral-alignment assumption on the initial residual or task labels. For the cross-entropy losses actually used in Tasks 1-6, no analogous identity is provided at all. The empirical gains may survive this, but the paper's central claim to have 'theoretically motivate[d]' the importance score is not supported by §4.1 as written, and the experiments lack error bars and released code to stand independently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes F3OCUS, a federated fine-tuning strategy for vision-language foundation models that selects a subset of layers or adapters for each client. Client-level layer importance is scored by the principal eigenvalue of a layerwise Neural Tangent Kernel (LNTK), and a server-side multi-objective meta-heuristic optimization refines these selections to maximize cumulative importance while promoting inter-client layer diversity under per-client compute budgets. The authors claim that this 'define-and-refine' procedure is theoretically motivated by LNTK convergence analysis, and they support it with experiments across six federated vision-language task settings, four VLM architectures, five meta-heuristic solvers, and a newly introduced Ultra-MedVQA dataset of 707,962 medical VQA triplets. The central claim is that F3OCUS improves accuracy and convergence over existing selective fine-tuning, pruning, and personalized-FL baselines while respecting heterogeneous client resources.","tokens_in":20666,"tokens_out":4235,"duration_ms":45564,"significance":"If the central claims hold, the paper addresses a practically important problem: how to fine-tune large VLMs in federated settings where clients have heterogeneous data modalities and compute budgets. The empirical scope is unusually broad, including VQA, disease classification, heterogeneous tasks, multiple model families, and many baselines; the proposed dataset is a potentially valuable community resource. The use of several meta-heuristic solvers and the attempt to give a theoretical convergence analysis are also useful contributions. However, the theoretical derivation that motivates the LNTK importance score contains a substantive algebraic error, the convergence theorems are asserted without proofs in the main text, and the empirical evaluation reports only point estimates with no error bars or released code. These issues make the current version insufficiently supported as a rigorous methodological contribution, although the general framework may be salvageable with corrected theory and stronger empirical reporting.","major_comments":[{"comment":"The derivation of the LNTK-based importance score is algebraically incorrect as written. For squared-error loss F = (1/2)||f(X)-Y||^2, the gradient with respect to the network output is ∇_Z F = f(X)-Y, not Y. Therefore step (v) of Eq. (12) should contain ((u_j^l)^T(f(X)-Y))^2, not ((u_j^l)^T Y)^2. The subsequent claim that layers with larger λ_1^l contribute most to loss reduction is not a consequence of the displayed equations; it requires an additional, unstated assumption that the initial residual f(X)-Y, or the labels Y, align with the top eigenvectors of each layerwise NTK. This is a load-bearing issue because the paper's stated contribution is to 'theoretically motivate and leverage' the principal eigenvalue as an importance score.","section":"§4.1, Eq. (12)"},{"comment":"Equation (11) mischaracterizes the NTK training dynamics. In the linearized regime the correct evolution is (U^T f(t))_j = e^{-ηλ_j t}(U^T f(0))_j + (1-e^{-ηλ_j t})(U^T y)_j. The displayed formula (U^T f(X))_j = (I-e^{-ηλ_j t})(U^T y)_j holds only under the special assumption f(0)=0, and even then the transient convergence rate is governed by the coefficient of the initial condition. Thus the text's statement that λ_1 'plays a dominant role in the convergence dynamics' and that the importance score is 'principled' is not justified by Eq. (11). This needs to be corrected or explicitly reframed as a heuristic approximation.","section":"§4.1, Eq. (11)"},{"comment":"Theorems 1 and 2 are stated in the main text without proofs or even proof sketches, with only a pointer to 'Suppl. §B'. Given that the convergence analysis is one of the paper's advertised contributions and is used to motivate the server-side objectives, the main text should at least state the key proof ideas and the precise role of Assumptions 1–5. If the proofs are only in a supplement, the manuscript should make the supplement available and verifiable. As it stands, the convergence claim is unverifiable from the submitted material.","section":"§4.2, Theorems 1 and 2"},{"comment":"All experimental results are reported as single point estimates without error bars, confidence intervals, or significance tests. Federated fine-tuning is stochastic, and the reported gains of F3OCUS over LNTK and over the best baselines are often only a few accuracy points; without multiple seeds, the robustness of the central empirical claim is unclear. Additionally, the paper states that the dataset is 'released' but gives no URL, and no code is provided. Since the theoretical motivation is currently weakened by the issues in §4.1, the empirical evaluation needs to be strong enough to stand independently, and the present reporting is not yet at that standard.","section":"§6.2, Tables 2–6"}],"minor_comments":[{"comment":"The multi-objective problem produces a Pareto front, but the manuscript does not specify how a single final layer-selection solution is chosen from that front for use in training. The choice of solution-selection rule is a free parameter that should be described precisely for reproducibility.","section":"§5.2, Eq. (21)"},{"comment":"Table 4 compares five meta-heuristic algorithms, but no hyperparameters, population sizes, iteration counts, or convergence criteria are given. Without these details, the comparison among NSGA, ABC, ACO, SA, and MOPSO is difficult to interpret or reproduce.","section":"§5.3, Table 4"},{"comment":"The abstract introduces the dataset as 'MedVQA-FL' while the body and contribution list call it 'Ultra-MedVQA.' This inconsistency should be resolved, and the dataset release mechanism should be stated explicitly.","section":"Abstract and §6.1"},{"comment":"Table 6 is difficult to read: after the first row the task labels disappear, and the header contains duplicate and misaligned abbreviations (e.g., FR appears twice). The table should be reformatted with clear row labels and a non-duplicated legend.","section":"Table 6"},{"comment":"There are numerous typographical and formatting issues, including 'LlA V A' in Table 2, 'TAMT Meyarivan' in reference [20], and inconsistent capitalization of author names in [31]. While minor, these should be corrected before publication.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a substantive flaw in the theoretical derivation of the LNTK importance score. This is not a mere presentation issue; it directly undermines the central claim of theoretical motivation. However, the empirical framework and the general idea of client-level importance plus server-level diversity are plausible and could be resubmitted after a corrected derivation or an explicit reframing as a heuristic, plus stronger empirical validation (seeds, uncertainty estimates, and code/data availability). I do not see grounds for outright rejection, but the revision needs to be substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the bottom line: this is a useful empirical paper for federated fine-tuning of VLMs in medical settings, and the new Ultra-MedVQA dataset alone is a real contribution. But the theoretical story that supposedly motivates the LNTK-based importance score does not hold up as written. The stress-test is right about Eq. 12: for squared-error loss the gradient factor is the residual Z-Y, not Y, so the step to Σλ_j ((u_j^T Y))^2 is not derived. You can't fix that by adding an alignment assumption on labels; the identity is simply wrong as stated. Add the cross-entropy mismatch and the claimed 'theoretical motivation' collapses. The convergence theorems (1 and 2) are deferred to a supplement and involve assumptions that are not checked.\n\nThe empirical work is extensive: 10,000+ client-level experiments, 6 FL task settings, 4 VLM architectures, 58 datasets, comparisons with pruning, layer selection, personalized FL, and PEFT baselines. F3OCUS consistently beats the best layer-selection baseline by roughly 5 points and the LNTK-only variant by ~3 points, and the server-side diversity objective seems to drive real gains (Fig. 7). The meta-heuristic comparison across NSGA, ABC, ACO, SA, MOPSO is a nice touch. The dataset is large and modality-diverse; if released it will be useful.\n\nThe absence of error bars, confidence intervals, and significance tests is a problem given the scale of the claims. No code or data link is provided yet despite the 'release' language. The Pareto-front selection rule and solver hyperparameters are underspecified. These are addressable, but they matter: without code, the reader cannot tell whether the 5-point gains are stable or a few lucky seeds.\n\nWho is this for: anyone working on parameter-efficient FL for large models, especially in medical imaging. It deserves a serious referee, but the reviewer should insist on fixing or reframing the theory, and on releasing code and data. My own verdict: the method is plausible and probably works, but the current paper overstates the theoretical grounding. Conditional, leaning positive.","headline":"A large, genuinely useful empirical study of federated layer selection for VLMs with a new medical VQA dataset, but the theoretical motivation for the LNTK importance score contains a real error and the code/data are not yet available.","tokens_in":21193,"tokens_out":2808,"would_cite":true,"duration_ms":27085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that in federated fine-tuning of vision-language models, client-specific layer selection by layerwise NTK principal eigenvalues, followed by server-side diversity-aware refinement, beats both purely local and purely…","keywords":["Federated Learning","Vision-Language Models","Parameter-Efficient Fine-Tuning","Neural Tangent Kernel","Layer Selection","Meta-heuristic Optimization","Medical Visual Question Answering","Ultra-MedVQA"],"falsifier":"Run F3OCUS on a synthetic client dataset constructed so that the task-relevant signal is concentrated in the layer with the smallest $\\lambda_1^l$; if the principal-eigenvalue ranking is truly controlling convergence, F3OCUS must trail uniform or random layer selection, and if it does not, the claimed mechanism is not doing the work.","tokens_in":20161,"feed_emoji":"🧠","tokens_out":7163,"duration_ms":64655,"temperature":0.7,"pith_summary":"The paper argues that when several resource-limited clients jointly fine-tune a large vision-language model, the choice of which layers each client updates is at least as important as how the updates are aggregated. It proposes an importance score based on the largest eigenvalue of each layer's neural tangent kernel, then a server-side, data-free multi-objective search that keeps high-importance layers while pushing different clients toward different layers, subject to each client's compute budget. The claimed payoff is faster convergence and higher accuracy than existing selective fine-tuning and federated personalization baselines across six vision-language federated settings and four model architectures. The paper also releases Ultra-MedVQA, a medical VQA dataset with 707,962 image-question-answer triplets across nine modalities.","feed_headline":"Layer scores plus diversity improve federated vision-language tuning","feed_subtitle":"NTK eigenvalues pick high-value layers; server-side diversity keeps clients from crowding the same layers.","key_machinery":"The load-bearing object is the layerwise Neural Tangent Kernel $\\Theta_l(X,X) = \\nabla_{\\theta_l} f(X) \\nabla_{\\theta_l} f(X)^T$ for layer $l$, whose principal eigenvalue $\\lambda_1^l$ the paper uses as a client-specific importance score. A Taylor-expansion argument represents the directional loss reduction as a sum over LNTK eigenvalues, and the paper takes the largest eigenvalue as the dominant term under spectral bias. The second mechanism is a server-side multi-objective optimization over the clients' layer-selection masks: maximize $\\sum_i \\sum_l S_i^l$ while minimizing $\\frac{1}{L}\\sum_l (n_l - \\bar{n})^2$, subject to per-client limits $L_{i,\\max}$, solved by one of five meta-heuristics (NSGA-II, ABC, ACO, SA, MOPSO) so that no client data ever reaches the server.","core_discovery":"The paper's claim is that neither a purely client-local layer choice nor a purely global one is optimal for parameter-efficient federated fine-tuning of vision-language models. Its discovery is a two-stage solution: each client scores every layer by the normalized principal eigenvalue of that layer's Neural Tangent Kernel, $S_i^l = \\lambda_{i,1}^l / \\sum_k \\lambda_{i,1}^k$, treating the largest eigenvalue as the layer's contribution to client-specific loss reduction; then a data-free server-side multi-objective meta-heuristic search refines the selections by maximizing total importance while minimizing the variance of the layer-selection histogram across clients, under each client's budget. The paper shows that both ingredients matter: LNTK alone beats existing layer-selection and pruning baselines, and F3OCUS adds a further gain by increasing inter-client layer diversity, with higher accuracy and faster convergence across more than 10,000 client-level experiments and six federated vision-language task settings.","pith_inferences":["Inference: If the principal NTK eigenvalue really tracks per-client convergence speed, the same score could be reused as a cheap, data-free scheduler for layer freezing in centralized settings with heterogeneous GPU memory, not just in federated learning.","Inference: The server-side diversity term may be doing work independent of importance, for example as an implicit regularizer that improves gradient mixing; an ablation that swaps LNTK importance for random scores while keeping the diversity objective would isolate that effect.","Inference: Releasing Ultra-MedVQA is a contribution beyond F3OCUS; the dataset's scale and modality coverage make it a reusable benchmark for centralized medical VQA, not only for federated settings.","Inference: Because the server-side search is data-free and treats the importance vector as input, any future client-side importance estimator could be plugged into F3OCUS without changing the server-side refinement."],"forward_implications":["Clients can specify different compute budgets, such as 2, 4, or 6 trainable layers, and still contribute to a shared vision-language model, because the server-side search respects per-client layer limits.","Pushing layer selections toward diversity across clients increases the fraction of the model that gets updated globally, which is what the paper credits for faster convergence and higher accuracy.","Existing layer-selection or pruning methods can be wrapped by the same server-side diversity refinement, since the diversity objective is decoupled from how the client importance score is produced.","LNTK importance can be computed on the client, and only scores rather than data are sent to the server, preserving the privacy property of federated learning.","The claimed gains hold across four vision-language architectures and six federated settings, including medical VQA and disease classification, so the mechanism is not tied to one model family."],"supporting_citations":[{"why":"Defines the neural tangent kernel whose layerwise decomposition is the basis of the client importance score.","marker":"[34]"},{"why":"Provides the layerwise NTK and spectral-bias-style training analysis that the paper adapts into per-layer importance.","marker":"[69]"},{"why":"Supplies the NSGA-II genetic meta-heuristic used as one server-side optimizer.","marker":"[20]"},{"why":"Supplies the ant-colony optimizer evaluated as a server-side layer-selection strategy.","marker":"[21]"},{"why":"Supplies the multi-objective particle-swarm optimizer evaluated as a server-side strategy.","marker":"[18]"},{"why":"Supplies the artificial bee colony optimizer used for server-side refinement.","marker":"[4]"},{"why":"Supplies simulated annealing, the physics-based meta-heuristic used for server-side refinement.","marker":"[76]"},{"why":"Establishes the FedAvg aggregation baseline that F3OCUS builds on and compares against.","marker":"[58]"},{"why":"A parameter-efficient federated fine-tuning baseline that F3OCUS must beat in communication and accuracy comparisons.","marker":"[14]"},{"why":"Introduces adapter layers, the parameter-efficient modules that F3OCUS selects and fine-tunes in part of the experiments.","marker":"[31]"}],"fun_headline_variants":["NTK eigenvalues and diversity pick optimal layers for federated VLM tuning","Layer importance plus client diversity improve federated vision-language tuning","F3OCUS: server-side diversity and NTK scores guide federated VLM layer choice","Optimal layer selection for federated VLMs via NTK scores and diversity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of layers by their largest NTK eigenvalue is justified only if the loss is squared error and the target labels line up with the layer's top eigenvectors; the experiments instead use cross-entropy-style VQA and classification losses, so if that alignment fails, the importance score rests on an unproven heuristic.","fun_headline_variants_meta":{"raw":{"variants":["NTK eigenvalues and diversity pick optimal layers for federated VLM tuning","Layer importance plus client diversity improve federated vision-language tuning","F3OCUS: server-side diversity and NTK scores guide federated VLM layer choice","Optimal layer selection for federated VLMs via NTK scores and diversity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1619,"prompt_tokens":1005,"completion_tokens":614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":621,"tokens_out":614,"duration_ms":5806,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:49:31.577651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run F3OCUS on a synthetic client dataset constructed so that the task-relevant signal is concentrated in the layer with the smallest $\\lambda_1^l$; if the principal-eigenvalue ranking is truly controlling convergence, F3OCUS must trail uniform or random layer selection, and if it does not, the claimed mechanism is not doing the work.","supporting_citations":[{"cited_title":"Neu- ral tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the neural tangent kernel whose layerwise decomposition is the basis of the client importance score."},{"cited_title":"Train faster, perform better: mod- ular adaptive training in over-parameterized models","cited_arxiv_id":null,"evidence_quote":"Provides the layerwise NTK and spectral-bias-style training analysis that the paper adapts into per-layer importance."},{"cited_title":"Ant colony optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the ant-colony optimizer evaluated as a server-side layer-selection strategy."},{"cited_title":"Simulated annealing","cited_arxiv_id":null,"evidence_quote":"Supplies simulated annealing, the physics-based meta-heuristic used for server-side refinement."},{"cited_title":"Communication- efficient learning of deep networks from decentralized data","cited_arxiv_id":null,"evidence_quote":"Establishes the FedAvg aggregation baseline that F3OCUS builds on and compares against."},{"cited_title":"Parameter-efficient transfer learning for nlp","cited_arxiv_id":null,"evidence_quote":"Introduces adapter layers, the parameter-efficient modules that F3OCUS selects and fine-tunes in part of the experiments."}],"review_version":1}