REVIEW 6 major objections 4 minor
HermesHFL: Incentive-Compatible Hierarchical Federated Unlearning for Dynamic LLM Fine-Tuning
T0 review · 6 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read HermesHFL claims federated LLM unlearning can be scheduled, funded, and reversed in one optimization framework with no full retraining.
desk verdict A timely problem and a plausible framework, but the paper's own tables contradict the headline claim: on SST-2, GenSA and GdyRatio beat Neogen on accuracy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing identity is the contribution-proportional budget rule, Eq. (32): manager j's new budget equals the global budget times the share of total training contribution theta produced by j's selected workers. This turns the three-layer president–manager–worker problem P0 into the two-layer manager–worker problem P1, because the president's budget decision becomes a deterministic function of lower-level choices. Neogen then solves P1 by nesting CMA-ES over continuous payments and penalties (upper level) with CHC over binary participation and association variables (lower level), using a neural network to approximate the mapping from upper-level solutions to lower-level optima so that C
What would settle it
Run the exact P0 optimization and Neogen's P1 optimization on a small instance (e.g., 6 workers, 2 managers, 2 budget levels) by exhaustive enumeration, and compare the resulting president utility under the true optimal budget allocation versus the Eq. (32) proportional allocation. If the proportional allocation is not optimal for P0, the paper's central hierarchy-preservation claim fails; likewise, if the optimum differs, the reported accuracy and utility numbers would change.
Extended reading notes
Core claim
On the paper's own terms, the core discovery is that selective unlearning does not need a separate retraining pipeline: unlearning can be executed inside the same global aggregation round as normal training, by having clients with erasure requests run gradient ascent on their LoRA parameters while the rest run stochastic gradient descent, then aggregating both update types at edge and global levels. The KLD between pre- and post-unlearning output distributions over the departing worker's data verifies erasure; once that KLD exceeds a threshold, the worker's profile is refreshed and it may rejoin. Around this loop the paper wraps an economic layer: each manager signs contracts specifying paym
Load-bearing premise
The conclusion rests on the assumption that the contribution-proportional budget allocation in Eq. (32) exactly preserves the optimum of the original three-level problem; if the president's true optimal allocation differs from that rule, Neogen solves a different problem than P0 and the reported gains do not transfer to the hierarchical setting.
Editorial extensions
If this is right
- If the central claim holds, a federated LLM can honor erasure requests and still keep a single continuously updated global model; no full retraining is needed after each request.
- Unlearning penalties recycled into manager budgets turn client departures from a cost into future recruitment capacity, making churn economically tolerable.
- Because worker profiles are refreshed after verified erasure, a rejoining client cannot silently re-inject the data it asked to erase.
- The NN surrogate's approximation of lower-level optima reduces per-round fitness evaluations by about 25% compared to the surrogate-free evolutionary baseline, which matters when each evaluation involves LLM fine-tuning.
- Balanced positive utilities for both workers and managers across dataset scales suggest the contract design can sustain participation in regimes where fixed-pricing baselines collapse into negative manager utility.
Reading between the lines
- An untested consequence of Eq. (32) is that budget allocation is artificially tied to the current proxy theta for contribution; if quality estimates are noisy, managers with temporarily lucky workers receive budget that a directly optimizing president would not give them. One could test this by comparing P1 against an exhaustive small-grid solution of P0.
- The framework assumes a worker's cost and privacy cost are known to the manager when signing contracts; if workers can misreport these costs, incentive compatibility may fail. A natural extension would be to add truthful-reporting constraints or a screening contract menu.
- The KLD threshold delta is a verification heuristic, not a privacy guarantee; a committed adversary could in principle retain information that does not shift output distributions. A stronger erasure test would combine KLD with membership-inference or gradient-residual checks.
- If this design scales to larger models, the same contribution-proportional budget rule could be paired with Shapley-style contribution measures to make budget shares more robust to collusion among workers under one manager.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HermesHFL, a hierarchical federated learning framework with selective unlearning, client rejoining, and incentive mechanisms for LoRA-based LLM fine-tuning. It formulates a three-layer optimization problem (P0) involving president, managers, and workers, then reduces it to a two-layer problem (P1) via a contribution-proportional budget allocation rule. The proposed solver, Neogen, combines CMA-ES for continuous incentive variables and CHC evolutionary search for discrete selection/association decisions, with a neural-network surrogate to accelerate lower-level optimization. Experiments on SST-2 and AGNews with GPT-2 LoRA compare Neogen against generic optimization heuristics and compare three unlearning strategies. The paper claims that HermesHFL "consistently outperforms state-of-the-art baselines in model utility, unlearning effectiveness, convergence stability, and resource efficiency."
Significance. If the claims were fully supported, the paper would make a useful contribution: a unified model of hierarchical federated unlearning with dynamic client leave-unlearn-rejoin behavior, LoRA-based PEFT, and economic incentives is a relevant and timely problem. The formalization of the lifecycle and the integration of a bilevel evolutionary solver with a neural surrogate are interesting. The paper reports reproducible hyperparameters, albeit without releasing code. However, the central claims are not supported by the paper's own numerical results, the problem reduction is assumed rather than proved, and the validation of unlearning effectiveness is weak. These are load-bearing issues for the stated contribution.
major comments (6)
- [Sec. VII-B / Tables II-III] The abstract and conclusion claim that HermesHFL "consistently outperforms state-of-the-art baselines in model utility." The evaluation defines Acc as the model-utility metric. Tables II and III contradict this claim: on SST-2 Set#1, Neogen achieves 0.795 versus GenSA 0.877, GdyRatio 0.834, and RandAssign 0.845; on Set#2, Neogen achieves 0.837 versus GdyRatio 0.903, GenSA 0.890, and RandAssign 0.879. The text itself states that "GenSA achieves the highest accuracy on SST-2 Set#1 (0.877)." Unless "model utility" is redefined to a composite that excludes accuracy or weights it in an unstated way, the central claim is false on the paper's own data. This is a load-bearing numerical contradiction, not a presentation issue.
- [Sec. VII-B] The fixed-pricing baseline is calibrated to Neogen's output: "the multiplier is set to 3, which corresponds to the midpoint of the p/cost ratio range learned by Neogen." This is circular for the incentive-efficiency comparisons. The baselines are not independent "state-of-the-art" mechanisms; their contracts are constructed from the proposed method's learned range. As a result, utility comparisons (MgU/WkU) and any downstream claims of incentive-compatible superiority are partially built into the baseline design.
- [Sec. V-C / Eq. (32)] The transformation of three-layer problem P0 to two-layer problem P1 assumes that the president's budget allocation equals the contribution-proportional rule of Eq. (32). The text calls this a "presumption," and Appendix A supplies only analogies, not a proof. If the true optimal allocation differs from proportional-to-contribution, Neogen solves P1, not P0, and the claimed hierarchical optimum is not preserved. A formal equivalence result, or a counterexample analysis, is necessary for the paper's central optimization claim.
- [Sec. V-A/B / Eqs. (25)-(30)] The title and abstract claim "incentive-compatible," but the paper provides no mechanism-design analysis. Eqs. (25)-(30) define utilities, and the constraints enforce individual rationality and budget feasibility, yet nothing shows that truthful reporting is a best response, that contracts are incentive-compatible, or that the evolutionary solver finds an equilibrium of the induced game. The term appears to be used descriptively rather than as a verified property.
- [Sec. V-A4 / Eq. (24)] Unlearning effectiveness is verified only by KLD between pre- and post-unlearning predictions on the erased worker's own data. This shows that gradient ascent changed predictions on that data; it does not show closeness to the retraining benchmark A(D_rtn) required by Eq. (4), nor does it test retained-data utility or leakage via membership inference. The reported KLD values (e.g., 7.29 on SST-2) therefore do not establish "unlearning effectiveness" against state-of-the-art baselines, none of which are actual federated-unlearning methods.
- [Sec. VI-C] The convergence proof is a sequence of qualitative observations: CHC is elitist and the search space is finite; the NN is a universal approximator; CMA-ES has known convergence properties. These do not imply convergence of the surrogate-assisted bilevel scheme to a stationary solution of P1. CHC solves (41) only approximately, the NN mapping v*=NN(u;theta) has unknown approximation error, and the upper-level fitness is evaluated with approximate lower-level solutions. The claim that Neogen "converges to a stable stationary solution" is not established.
minor comments (4)
- [Sec. VII-A] The per-class sample counts are said to be "explicitly specified per worker" but are not reported. A reproducibility table for the data splits is needed.
- [Sec. VII-B / Tables II-III] The abstract claims wall-clock resource efficiency, but the optimization tables report only Feval/r. Wall-clock times are given only for the unlearning-strategy comparison. Clarify which efficiency claim is being made.
- [Figures 3-8] In the received version, the figure labels and legends appear garbled (e.g., placeholder sequences in Figs. 3-8). The tables in Appendix C are readable; the figures should be regenerated or supplemented with more readable captions.
- [Abstract / Conclusion] If the experiments are retained, the wording "consistently outperforms" should be replaced with a claim matching the data, e.g., "competitive on SST-2 and superior on AGNews," or provide a formal composite utility that justifies the claimed ordering.
Circularity Check
Baseline pricing multiplier is calibrated to the midpoint of the p/cost ratio range learned by Neogen, so the headline 'consistently outperforms baselines' comparison is partly constructed from the method's own output; the core derivation and unlearning ablation are otherwise self-contained.
-
fitted input called prediction
[Sec. VII-B (Evaluation, benchmark setup), fixed-pricing definition before Fig. 3]
"The payment is defined as 𝑝(𝑔)𝑖,𝑗 = 3𝑐work,(𝑔)𝑖,𝑗; the multiplier is set to 3, which corresponds to the midpoint of the 𝑝(𝑔)𝑖,𝑗/𝑐work,(𝑔)𝑖,𝑗 ratio range learned by Neogen. The optimized payments under Neogen naturally range from 1× to around 5× of 𝑐work,(𝑔)𝑖,𝑗."
The comparative baselines (GenSA, GdyRatio, RandAssign) are configured with a fixed pricing multiplier p=3×cost, where the multiplier 3 is chosen as 'the midpoint of the p/cost ratio range learned by Neogen.' The baseline's central behavioral parameter is thus fitted from the proposed method's own output. Since these baselines are then compared with Neogen to support the abstract's claim that HermesHFL 'consistently outperforms state-of-the-art baselines in model utility,' the comparison is not an independent benchmark: the baselines' selection and utility outcomes are partly determined by a parameter calibrated to the method under test. This is a fitted input to the evaluation constructed from the method's output rather than an externally fixed reference.
full rationale
The core derivation is largely self-contained. The problem formulation (Sec. V), the Neogen bilevel CMA-ES/CHC optimizer with NN surrogate (Sec. VI), and the GA-based unlearning with KLD verification are assembled from stated modeling choices and standard external results (FedAvg [1], CHC [34], universal approximation [35], CMA-ES convergence [36]); no load-bearing claim reduces to a self-citation chain. The only overlapping-author citation ([6]) supports a contextual remark on HFL and LLMs and is not load-bearing. The identified circular step is confined to the benchmark construction: the baseline fixed-pricing multiplier is calibrated to the midpoint of the p/cost ratio range learned by Neogen, meaning the headline outperformance comparison is partly constructed from the proposed method's output. Two further concerns are correctness issues rather than circularity and are noted per the reviewing rule. (i) The P0-to-P1 reduction (Sec. V-C, Eq. (32)) presumes, without proof, that contribution-proportional budget allocation preserves the president's optimum; if the president's optimal allocation differs, Neogen solves a different problem than claimed. This is an unproven equivalence assumption, not a definitional reduction. (ii) Tables II and III internally contradict the abstract's 'consistently outperforms' claim on SST-2 accuracy (Neogen 0.795/0.837 vs GenSA 0.877/0.890 and GdyRatio 0.834/0.903), an internal numerical inconsistency rather than a circular step. The unlearning ablation (HermesHFL vs NoRejoin vs Retrain) is independent of the calibrated baseline and supports the framework's rejoin-unlearning claims. Overall: one genuine circular configuration in the evaluation, no circularity in the central derivation.
Assumptions & free parameters
free parameters (8)
- Unlearning threshold δ =
0.05
- Reputation memory parameters α, H =
α=0.6, H=5
- Utility normalization coefficients λ_mana, λ_pres =
8.0, 0.5
- Unlearning learning rate η̃ and weight scale =
1e-3, 0.1
- Incentive budget β_sum =
30 (Set#1) / 60 (Set#2)
- Baseline fixed pricing multiplier =
3×cost
- CHC/CMA-ES/NN hyperparameters =
pop_u=20, σ_u=0.3, pop_l=20, T=20, p_mut=0.35, r_ham=3, NN hidden=128
- Per-worker per-class sample counts =
not disclosed
assumptions (6)
- domain assumption KLD(P_old || P_new) > δ implies the model is retrain-equivalent without the forgotten data (Eq. 24, Sec. V-A4).
- ad hoc to paper The contribution-proportional budget allocation rule (Eq. 32) faithfully preserves the three-layer optimum P0.
- standard math Universal approximation theorem plus accumulating D ensures NN surrogate error → 0 (Sec. VI-C).
- domain assumption CMA-ES converges to a stationary solution of the upper-level problem P1 (Sec. VI-C).
- standard math Separate FedAvg-style aggregation of LoRA factors A and B is a valid hierarchical fine-tuning aggregation (Eqs. 11-15).
- domain assumption Workers are rational utility maximizers whose behavior is captured by the specified utility functions (Eq. 25); incentive compatibility follows from utility balance.
invented entities (2)
-
Reputation score ρ_fgt with exponentially decaying unlearning window
-
Profile refresh on rejoining
Cite this review
Pith. "Pith review of HermesHFL: Incentive-Compatible Hierarchical Federated Unlearning for Dynamic LLM Fine-Tuning." pith.science (2026). https://pith.science/paper/YEZDDTY2
@misc{pith2026260711528,
author = {Pith},
title = {Pith review of: HermesHFL: Incentive-Compatible Hierarchical Federated Unlearning for Dynamic LLM Fine-Tuning},
year = {2026},
howpublished = {\url{https://pith.science/paper/YEZDDTY2}},
note = {Machine review of arXiv:2607.11528}
}
read the original abstract
Hierarchical federated unlearning (HFUL) for large language model (LLM) fine-tuning faces significant challenges due to hierarchical aggregation, dynamic client participation, and strong parameter coupling in LLM adaptation. Selectively removing client contributions is particularly difficult because model updates propagate across multiple aggregation stages while unlearning requests may coincide with client departures and rejoining. To address these issues, we propose HermesHFL, a hierarchical federated learning framework that supports selective unlearning, dynamic client participation, and client reintegration for scalable LLM fine-tuning via parameter-efficient fine-tuning (PEFT) with LoRA. We formulate a unified optimization problem that jointly models client participation, edge association, incentive allocation, and unlearning under heterogeneous client behaviors. To solve this problem efficiently, we develop Neogen, a neural-guided bilevel evolutionary optimization framework that combines CMA-ES for continuous incentive optimization with a CHC-based evolutionary mechanism for discrete participation and association decisions. A neural surrogate further accelerates optimization and improves search efficiency. Extensive experiments on LLM fine-tuning tasks demonstrate that HermesHFL consistently outperforms state-of-the-art baselines in model utility, unlearning effectiveness, convergence stability, and resource efficiency.
Figures
Figures from the paper (9 more)
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.