Pith. sign in

REVIEW 6 major objections 6 minor 42 references

Many-Task Federated Fine-Tuning via Unified Task Vectors

T0 review · 6 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A federated method called MaTU shows that clients working on many different tasks can jointly fine-tune one shared model by exchanging a single unified task vector plus lightweight masks and scalars, reaching accuracy close to per-task…

desk verdict A solid, incremental many-task FL method with a real communication win, but the abstract oversells 'comparable to per-task fine-tuning' and missing variance/hyperparameter details prevent a clean accept. read the letter →

arxiv 2502.06376 v3 pith:NOKRD3ZL submitted 2025-02-10 cs.LG cs.CV

Vasileios Tsouvalas , Tanir Ozcelebi , Nirvana Meratnia This is my paper · ORCID
classification cs.LGcs.CV
keywords federatedlearningtaskheterogeneityvectorsmodelmergingarithmeticparameter-efficientfine-tuningmany-taskweightdisentanglement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that many-task federated learning, where each client may train on several different tasks at once, can be done without grouping clients or storing client-specific models on the server. The proposed method, MaTU, represents what a client learned as a task vector, condenses all of the client's tasks into one unified task vector, and sends that vector plus lightweight binary masks and scalars to the server. The server aggregates updates task by task and transfers knowledge across related tasks using a similarity score built from sign conflicts between task vectors. Across 30 vision datasets, MaTU reports higher average accuracy than existing many-task federated approaches and results close to per-task fine-tuning, with communication cost that grows only mildly as clients take on more tasks. If correct, this makes heterogeneous federated learning scalable and stateless at the server.

What carries the argument

The machinery is the unified task vector $\tau_n = \sigma_n \odot \mu_n$: the sign of the summed task-vector entries sets direction, and the maximum magnitude aligned with that sign sets scale. Three pieces do the work around it. First, task-specific modulators, a binary mask $m_n^i = (\tau_n^i \odot \tau_n > 0)$ and a scalar $\lambda_n^i = \sum |\tau_n^i| / \sum |m_n^i \odot \tau_n|$, allow one stored vector to reproduce each task update. Second, the server's task-specific mask $\hat{m}^t$ weights each coordinate by the agreement score from Eq. 3, suppressing coordinates where clients disagree. Third, cross-task aggregation uses similarity $S(t,t') = \frac{1}{2}(1 + \frac{1}{d} \sum_i \mathrm{sgn}([\hat{\tau}^t]_i)\,\mathrm{sgn}([\hat{\tau}^{t'}]_i))$ to pick the top-$\kappa$ similar tasks and blend their aggregated vectors through the mask of task $t$. The identity carrying the argument is $\tau^{t,r+1} = \hat{\tau}^{t,r} + \tilde{\tau}^{t,r}$, which splits each task's update into a same-task term and a cross-task transfer term.

What would settle it

Train two tasks known to interfere on the same pretrained ViT-B/32, compute their task vectors, and compare a merge that zeros out sign-conflicting coordinates against one that zeros out the same number of random coordinates; if per-task accuracy does not improve for the sign-conflict mask, the sign-conflict proxy is not doing the work the paper claims.

Watch

Extended reading notes

Core claim

The central claim is that task vectors can serve as the currency of federated fine-tuning under task heterogeneity. Each client trains a low-rank adapter on each local task, forms a task vector from the pretrained weights, and condenses all of its tasks into a unified task vector $\tau = \sigma \odot \mu$ using the dominant sign and the maximum magnitude aligned with that sign. Per-task binary masks and scalar rescalers let the client unpack that single vector for any of its tasks. On the server, an agreement-score mask suppresses unreliable coordinates, a weighted average across clients produces a task-specific vector, and a sign-conflict similarity $S(t,t')$ selects the top-$\kappa$ related tasks for cross-task transfer. The resulting update for a task is the sum of a same-task term and a cross-task term, and the server remains stateless because each round it reconstructs only unified vectors and modulators. The evidence is that this procedure reaches 84.32% average accuracy in the single-task-per-client setting, 79.47% when clients hold multiple tasks, and 77.40% normalized accuracy across 30 tasks, against 67.60% and 52.62% for the strongest baseline in the latter two settings.

Load-bearing premise

The method depends on the assumption that for the pretrained model, each weight's sign in a task vector reliably marks the direction that lowers that task's loss, so sign conflicts between two tasks' vectors identify the weights where the tasks interfere; if this property does not hold, the masks, similarity scores, and cross-task aggregation have no basis.

Editorial extensions

If this is right

  • A federated server can orchestrate arbitrarily many tasks with no per-client or per-group state, because each client contributes one unified vector and per-task modulators.
  • Communication per client stays nearly flat as the number of local tasks grows, since only binary masks and scalars are added, so the advantage over per-task adapter transmission widens with task count.
  • Task relatedness in federated settings can be estimated from weight-space sign agreement alone, without access to task data or labels, enabling knowledge transfer to data-scarce tasks.
  • A single unified task vector can be repurposed at inference for any trained task by applying the corresponding mask and scalar, yielding storage savings that grow with model size.
  • The gap to per-task fine-tuning is small, about six points in the single-task setting and eleven points in the multi-task setting, suggesting a shared model can substitute for per-task models in heterogeneous deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper validates sign-conflict similarity only on one pretrained vision model, the transfer to other base models is untested; if weight disentanglement is weaker in smaller or less thoroughly pretrained models, the masks and similarity scores would likely need to be learned rather than read off signs.
  • As the number of tasks per client grows, the per-task binary mask becomes the dominant communication cost, so compressing masks would be a natural next step that the paper does not explore.
  • The same aggregation scheme could be lifted out of federated learning and applied to centralized model merging, where a unified task vector plus modulators is already a merge recipe and sign-conflict similarity offers a cheap way to choose which tasks to merge.
  • The server's top-$\kappa$ selection is a heuristic; replacing sign agreement with a learned transferability estimate could improve cross-task aggregation, though the paper's own comparison suggests sign agreement tracks such metrics on eight datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes MaTU, a many-task federated fine-tuning method that builds on task arithmetic. Each client trains LoRA task vectors for its local tasks, compresses them into a single "unified" task vector, and transmits that vector together with lightweight modulators (binary masks and scalar rescalers) to a stateless server. The server constructs per-task agreement masks, aggregates same-task contributions, estimates task similarity from sign agreement among aggregated task vectors, and adds a similarity-weighted cross-task aggregation term. The updated unified vector and modulators are sent back to the clients. The paper evaluates MaTU on an 8-task and a 30-task vision benchmark with ViT-B/32, comparing against FedAvg, FedProx, FedPer, MaT-FL, and NTK-FedAvg, and reports communication bitrates.

Significance. The proposed design is attractive: it avoids client clustering and server-side per-client model storage, transmits one unified adapter per client, and the 30-task evaluation is more extensive than is typical for MaT-FL papers. The paper also provides a correlation analysis of the sign-conflict similarity proxy against several transferability metrics and an ablation of cross-task aggregation. If the central claims hold, MaTU would be a strong practical contribution to heterogeneous federated fine-tuning. However, the validation of the similarity proxy does not cover the actual vectors used in the algorithm, several hyperparameters and reconstruction steps are under-specified, and the empirical section lacks variance estimates; these gaps currently prevent full confidence in the claimed margins and in the mechanism responsible for the gains.

major comments (6)
  1. [Section 3.2, Eq. (5); Fig. 3] The sign-conflict similarity proxy is validated in Fig. 3 on raw, individually fine-tuned task vectors, but in the algorithm S(t,t') is computed on the aggregated, masked, and re-scaled vectors \hat \tau^t of Eq. (4). The only in-loop evidence, Fig. 6b, compares sign-based with uniform and no cross-task aggregation on 8 tasks with N=10 and no variance estimates. Because the cross-task term Eq. (6) is credited with part of the gains, please validate the proxy on the actual federated vectors (e.g., compare S on \hat \tau^t with measured pairwise transfer, or repeat the Fig. 3 correlation study using aggregated LoRA vectors from the FL loop), or explicitly limit the claim to the raw-vector setting.
  2. [Section 3.1; Section 3.2] The method assumes weight disentanglement, i.e., that the sign of each weight indicates the direction that minimizes loss and that sign conflicts between task vectors predict interference. This property is cited from task-arithmetic results established for full fine-tuning in the tangent space. MaTU applies sign operations to LoRA adapter weights of rank 16. The paper does not show that sign conflicts in the low-rank adapter coordinates carry the same meaning for the pre-trained ViT-B/32. This premise is load-bearing for the masks (Eq. 3), the similarity matrix (Eq. 5), and the cross-task aggregation (Eq. 6). A concrete test would be to measure pairwise task-transfer matrices on LoRA adapters and compare their rank ordering with sign-conflict counts.
  3. [Section 3.2, Eqs. (4) and (7)] The communication protocol states that clients transmit only the unified task vector \tau_n and the modulators (m_n^t, \lambda_n^t). Eq. (4) then aggregates over terms \lambda_n^t \hat m^t \odot \tau_n^t, but \tau_n^t is not defined in the transmitted set. If \tau_n^t is the original local task vector, the server cannot compute Eq. (4); if it denotes the reconstructed vector \lambda_n^t m_n^t \odot \tau_n, the formula appears to apply \lambda_n^t twice (once inside \tau_n^t and once explicitly). Please define the reconstruction step and rewrite Eqs. (4) and (7) so that they operate only on information available to the server.
  4. [Section 4, Tables 1-2; Figs. 4-6] No variance estimates or number of seeds are reported for any experiment. The protocol involves random client participation (\xi=0.2), Dirichlet task/data splits, and stochastic local training; a single run cannot support point claims such as the 2.21-point margin over FedPer in Table 1 or the normalized 77.40% vs 52.62% comparison in Fig. 4 with confidence. Please report means and standard deviations over at least three independent runs for the main tables and figures, or clearly mark which numbers are single runs and avoid precision claims based on them.
  5. [Section 3.2, Eq. (6)] The top-\kappa parameter in the definition of Z^t is never specified. The footnotes report \rho=0.4 and \epsilon=0.5, but no value or sensitivity analysis is given for \kappa, which directly controls how many similar tasks contribute to the cross-task term Eq. (6). Without this value, the experimental configuration is not fully reproducible. Please report \kappa and, ideally, a small sensitivity study.
  6. [Abstract; Section 4.1] The abstract's claim that results are "comparable to per-task fine-tuning" is stronger than the reported numbers: in Table 2 MaTU reaches 79.47% versus 90.21% for individual centralized fine-tuning, and in Fig. 4 the 30-task normalized accuracy is 77.40%, i.e., an average 22.6% degradation. The method is convincingly better than MaT-FL baselines, but "comparable to per-task fine-tuning" should be qualified (e.g., "substantially closes the gap" or "within X points on the 8-task benchmark").
minor comments (6)
  1. [Introduction] "refereed to as" should be "referred to as".
  2. [Section 3.2, Eq. (3)] The text says "\rho1 is a threshold" but the symbol is \rho; please fix the stray subscript. Also, clarify that \hat m^t is therefore a soft mask in [0,1] and justify why values between \rho and 1 are kept as soft weights rather than binarized.
  3. [Fig. 2 caption] "In all metrics except WTE, higher values correspond to higher correlation" is vague; please state what the heatmap values represent in each panel and how the sign-based panel is normalized.
  4. [References] The reference list contains duplicate entries for TIES-Merging ([Yadav et al., 2023a] and [Yadav et al., 2023b]) with identical titles; please consolidate.
  5. [Section 4.1, Fig. 6b] The "uniform cross-task averaging" variant is not precisely defined; specify whether it uses S=1 for all pairs or averages the same top-\kappa set with equal weights.
  6. [Tables 1 and 2] Please define "bpt" consistently (the text sometimes uses "bpr" for bits per round) and state whether the Individual (Centralized) row is subject to the same LoRA configuration as the federated methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MaTU's aggregation is an iterative update, not a self-referential derivation, and all key building blocks are externally sourced or empirically benchmarked.

full rationale

MaTU's derivation chain does not reduce to its own output. The unified task vector and task-specific masks (Eq. 2 and Section 3.2) are adopted from EMR-Merging [Huang et al., 2024], and the sign-conflict premise is imported from external prior work (Ortiz-Jimenez et al.; Yadav et al.; Muhamed et al.) and tested against five external transferability metrics in Fig. 3, so it is not a self-citation trick. The similarity matrix S(t,t') in Eq. 5 is computed from aggregated task vectors, and Eq. 7 uses S to produce the next round's task vectors; this is an iterative feedback loop rather than a definitional identity—each round's S depends on the previous round's vectors, so no equation is equivalent to its own input by construction. The claimed results are benchmarked against external baselines (FedAvg, FedProx, FedPer, MaT-FL, NTK-FedAvg) on fixed benchmarks, so the reported accuracies are measurements, not fitted parameters renamed as predictions. The only self-citation (Tsouvalas et al., 2023, DeltaMask) is background in Related Work and is not load-bearing. The under-validation of the sign-conflict proxy on in-loop masked LoRA vectors is a correctness and evidence concern, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method leans on assumptions imported from task arithmetic and model merging. The unified vector construction (Eq. 2) is not new; it is cited to Huang et al. 2024. The main free choices are the two thresholds and the unreported top-k. No new entities are introduced.

free parameters (3)
  • mask threshold rho = 0.4
    Threshold for binarizing the agreement score in Eq. 3, adopted from Tenison et al. (2023), not tuned in this paper.
  • similarity threshold epsilon = 0.5
    Minimum sign-agreement for cross-task aggregation in Eq. 6, chosen by hand.
  • top-k similar tasks = not reported
    Number of similar tasks used in cross-task aggregation (Eq. 6); no value is given in the paper.
assumptions (4)
  • domain assumption Task vector signs encode task-specific direction; sign conflicts indicate interference between tasks.
    Invoked in Section 3.2 to justify sign-based masks (Eq. 3) and similarity (Eq. 5); based on prior task arithmetic literature.
  • domain assumption The pre-trained ViT-B/32 model exhibits weight disentanglement, making task arithmetic operations valid.
    Stated in Section 3.1, relying on Ortiz-Jimenez et al.; if false, the unified vector construction loses meaning.
  • domain assumption A single unified task vector combined with binary masks and scalars can approximate each task-specific vector well enough for training.
    Core of the method, adapted from EMR-Merging (Huang et al., 2024); the approximation error is not analyzed.
  • domain assumption The agreement score alpha correlates with client heterogeneity and task interference, justifying the soft thresholding in Eq. 3.
    Cited from Tenison et al.; used to reduce updates for low-agreement parameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Many-Task Federated Fine-Tuning via Unified Task Vectors." pith.science (2026). https://pith.science/paper/NOKRD3ZL

@misc{pith2026250206376,
  author       = {Pith},
  title        = {Pith review of: Many-Task Federated Fine-Tuning via Unified Task Vectors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOKRD3ZL}},
  note         = {Machine review of arXiv:2502.06376}
}
read the original abstract

Federated Learning (FL) traditionally assumes homogeneous client tasks; however, in real-world scenarios, clients often specialize in diverse tasks, introducing task heterogeneity. To address this challenge, Many-Task FL (MaT-FL) has emerged, enabling clients to collaborate effectively despite task diversity. Existing MaT-FL approaches rely on client grouping or personalized layers, requiring the server to manage individual models and failing to account for clients handling multiple tasks. We propose MaTU, a MaT-FL approach that enables joint learning of task vectors across clients, eliminating the need for clustering or client-specific weight storage at the server. Our method introduces a novel aggregation mechanism that determines task similarity based on the direction of clients task vectors and constructs a unified task vector encapsulating all tasks. To address task-specific requirements, we augment the unified task vector with lightweight modulators that facilitate knowledge transfer among related tasks while disentangling dissimilar ones. Evaluated across 30 datasets, MaTU achieves superior performance over state-of-the-art MaT-FL approaches, with results comparable to per-task fine-tuning, while delivering significant communication savings.

Figures

Figures reproduced from arXiv: 2502.06376 by the authors.

Figure 1
Figure 1. MaTU’s training process: Clients update their task vectors, and transmit their “unified” task vector and task modulators to server, which constructs task masks, estimates task similarity via sign conflicts, and construct clients’ task vectors using the top-κ similar tasks. By only adjusting λk, the performance across K tasks can be improved. Building on the observations from [Ortiz￾Jimenez et al., 2023], TIES [Yadav… view at source ↗
Figure 2
Figure 2. Comparison of sign vectors vs. state-of-the-art transfer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Performance evaluation of ViT-B/32 models on the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Impact of scaling number of tasks assigned to clients [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance evaluation of ViT-B/32 on the [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 38 canonical work pages

  1. [1]

    Natural gradient works efficiently in learning

    [Amari, 1998] Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10:251–276,

    work page 1998

  2. [7]

    FedBone: Towards Large-Scale Federated Multi-Task Learning

    [Chen et al., 2023] Yiqiang Chen, Teng Zhang, Xinlong Jiang, Qian Chen, Chenlong Gao, and Wuliang Huang. Fedbone: To- wards large-scale federated multi-task learning. arXiv preprint arXiv:2306.17465,

    work page Pith review arXiv 2023

  3. [8]

    Remote sensing image scene classification: Benchmark and state of the art

    [Cheng et al., 2017] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883,

    work page 2017

  4. [10]

    Cimpoi, S

    [Cimpoi et al., 2014] M. Cimpoi, S. Maji, I. Kokkinos, S. Mo- hamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),

    work page 2014

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    [Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale,

    work page 2021

  6. [15]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,

    [Helber et al., 2017] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,

    work page 2017

  7. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    [Hu et al., 2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language mod- els,

    work page 2021

  8. [17]

    Emr-merging: Tuning-free high-performance model merging,

    [Huang et al., 2024] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high-performance model merging,

    work page 2024

Show all 42 references
  1. [19]

    3d object representations for fine-grained cate- gorization

    [Krause et al., 2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained cate- gorization. In 2013 IEEE International Conference on Computer Vision Workshops, pages 554–561,

    2013

  2. [21]

    Mnist handwritten digit database

    [LeCun et al., 2010] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2,

    2010

  3. [23]

    Federated learning on non-iid data silos: An experimental study,

    [Li et al., 2021] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study,

    2021

  4. [24]

    Wasserstein task embedding for measur- ing task similarities

    [Liu et al., 2025] Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltog- gio, and Soheil Kolouri. Wasserstein task embedding for measur- ing task similarities. Neural Networks, 181:106796,

    2025

  5. [26]

    Communication-efficient learning of deep networks from decen- tralized data

    [McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decen- tralized data. In Artificial intelligence and statistics, pages 1273–

    2017

  6. [27]

    Fed up with complexity: Simplifying many-task federated learning with NTKFedavg

    [Muhamed et al., 2024] Aashiq Muhamed, Meher Mankikar, and Virginia Smith. Fed up with complexity: Simplifying many-task federated learning with NTKFedavg. In Privacy Regulation and Protection in Machine Learning,

    2024

  7. [28]

    Reading digits in natural images with unsupervised feature learning

    [Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning

    2011

  8. [30]

    Task arithmetic in the tangent space: Improved editing of pre-trained models,

    [Ortiz-Jimenez et al., 2023] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models,

    2023

  9. [31]

    Fl-tac: Enhanced fine-tuning in feder- ated learning via low-rank, task-specific adapter clustering.arXiv preprint arXiv:2404.15384,

    [Ping et al., 2024] Siqi Ping, Yuzhu Mao, Yang Liu, Xiao-Ping Zhang, and Wenbo Ding. Fl-tac: Enhanced fine-tuning in feder- ated learning via low-rank, task-specific adapter clustering.arXiv preprint arXiv:2404.15384,

    2024 arXiv

  10. [32]

    The german traffic sign recognition benchmark: A multi-class classification competition

    [Stallkamp et al., 2011] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. InThe 2011 International Joint Conference on Neural Networks, pages 1453– 1460,

    2011

  11. [33]

    Improving lora in privacy-preserving federated learning,

    [Sun et al., 2024] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning,

    2024

  12. [34]

    Gradient masked averaging for federated learning,

    [Tenison et al., 2023] Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan, Edouard Oyallon, Irina Rish, and Eugene Belilovsky. Gradient masked averaging for federated learning,

    2023

  13. [35]

    Federated fine-tuning of foundation models via probabilistic masking,

    [Tsouvalas et al., 2023] Vasileios Tsouvalas, Yuki Asano, and Aaqib Saeed. Federated fine-tuning of foundation models via probabilistic masking,

    2023

  14. [36]

    Spot: Better frozen model adaptation through soft prompt transfer,

    [Vu et al., 2022] Tu Vu, Brian Lester, Noah Constant, Rami Al- Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer,

    2022

  15. [37]

    [Xiao et al., 2010] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3485–3492, June

    2010

  16. [38]

    Adamerg- ing: Adaptive model merging for multi-task learning,

    [Yang et al., 2024] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerg- ing: Adaptive model merging for multi-task learning,

    2024

  17. [39]

    Fedlora: Model-heterogeneous personalized federated learn- ing with lora tuning

    [Yi et al., 2023] Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneous personalized federated learn- ing with lora tuning. arXiv preprint arXiv:2310.13283,

    2023 arXiv

  18. [40]

    Language models are super mario: Absorbing abili- ties from homologous models as a free lunch,

    [Yu et al., 2024] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abili- ties from homologous models as a free lunch,

    2024

  19. [41]

    Taskon- omy: Disentangling task transfer learning,

    [Zamir et al., 2018] Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskon- omy: Disentangling task transfer learning,

    2018

  20. [1998]

    Federated learning with personalization layers,

    [Arivazhagan et al., 2019] Manoj Ghuhan Arivazhagan, Vinay Ag- garwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers,

    2019

  21. [2009]

    Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classifi- cation tasks,

    [Ding et al., 2022] Nan Ding, Xi Chen, Tomer Levinboim, Beer Changpinyo, and Radu Soricut. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classifi- cation tasks,

    2022

  22. [2010]

    Federated opti- mization in heterogeneous networks,

    [Li et al., 2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated opti- mization in heterogeneous networks,

    2020

  23. [2011]

    Flora: Enhancing vision-language models with parameter-efficient federated learning

    [Nguyen et al., 2024] Duy Phuong Nguyen, J Pablo Munoz, and Ali Jannesari. Flora: Enhancing vision-language models with parameter-efficient federated learning. arXiv preprint arXiv:2404.15182,

    2024 arXiv

  24. [2013]

    Federatedscope-llm: A com- prehensive package for fine-tuning large language models in fed- erated learning,

    [Kuang et al., 2023] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm: A com- prehensive package for fine-tuning large language models in fed- erated learning,

    2023

  25. [2014]

    Imagenet: A large-scale hierarchical im- age database

    [Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255,

    2009

  26. [2017]

    Heterogeneous loRA for federated fine-tuning of on-device foundation models

    [Cho et al., 2023] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous loRA for federated fine-tuning of on-device foundation models. InInterna- tional Workshop on Federated Learning in the Age of Foundation Models in Conjunction with N...

    2023

  27. [2018]

    Mas: Towards resource-efficient feder- ated multiple-task learning

    [Zhuang et al., 2023] Weiming Zhuang, Yonggang Wen, Lingjuan Lyu, and Shuai Zhang. Mas: Towards resource-efficient feder- ated multiple-task learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23414– 23424, 2023

    2023

  28. [2019]

    Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr

    [Babakniya et al., 2023] Sara Babakniya, Ahmed Roushdy Elko- rdy, Yahya H. Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models,

    2023

  29. [2020]

    Many-task federated learning: A new problem set- ting and a simple baseline

    [Cai et al., 2023] Ruisi Cai, Xiaohan Chen, Shiwei Liu, Jayanth Srinivasa, Myungjin Lee, Ramana Kompella, and Zhangyang Wang. Many-task federated learning: A new problem set- ting and a simple baseline. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Wor...

    2023

  30. [2021]

    Etran: Energy-based transferability estimation,

    [Gholami et al., 2023] Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. Etran: Energy-based transferability estimation,

    2023

  31. [2022]

    Flower: A friendly federated learning research framework

    [Beutel et al., 2020] Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusm ˜ao, and Nicholas D Lane. Flower: A friendly federated learning research framework. arXiv preprint a...

    2020 arXiv

  32. [2023]

    An information-theoretic approach to transferability in task transfer learning,

    [Bao et al., 2022] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-theoretic approach to transferability in task transfer learning,

    2022

  33. [2024]

    Editing models with task arithmetic,

    [Ilharco et al., 2023] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic,

    2023

  34. [2025]

    Towards hetero-client federated multi-task learning

    [Lu et al., 2023] Yuxiang Lu, Suizhi Huang, Yuwen Yang, Sha- layiding Sirejiding, Yue Ding, and Hongtao Lu. Towards hetero-client federated multi-task learning. arXiv preprint arXiv:2311.13250,

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.