Pith. sign in

REVIEW 7 cited by

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.06432 v2 pith:ZUUVVVXM submitted 2024-01-12 cs.LG cs.DC

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Yae Jee Cho , Luyang Liu , Zheng Xu , Aldi Fahrezi , Gauri Joshi This is my paper
classification cs.LG cs.DC
keywords fine-tuningfederatedheterogeneousdeviceshetloraloraacrosson-device
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Foundation models (FMs) adapt well to specific domains or tasks with fine-tuning, and federated learning (FL) enables the potential for privacy-preserving fine-tuning of the FMs with on-device local data. For federated fine-tuning of FMs, we consider the FMs with small to medium parameter sizes of single digit billion at maximum, referred to as on-device FMs (ODFMs) that can be deployed on devices for inference but can only be fine-tuned with parameter efficient methods. In our work, we tackle the data and system heterogeneity problem of federated fine-tuning of ODFMs by proposing a novel method using heterogeneous low-rank approximations (LoRAs), namely HetLoRA. First, we show that the naive approach of using homogeneous LoRA ranks across devices face a trade-off between overfitting and slow convergence, and thus propose HetLoRA, which allows heterogeneous ranks across client devices and efficiently aggregates and distributes these heterogeneous LoRA modules. By applying rank self-pruning locally and sparsity-weighted aggregation at the server, HetLoRA combines the advantages of high and low-rank LoRAs, which achieves improved convergence speed and final performance compared to homogeneous LoRA. Furthermore, HetLoRA offers enhanced computation efficiency compared to full fine-tuning, making it suitable for federated fine-tuning across heterogeneous devices.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FedMark-FM: Auditable, Risk-Adjusted Data Markets for Federated Foundation-Model Adaptation

    cs.GT 2026-07 conditional novelty 7.0

    FedMark-FM is an auditable data-market framework that prices heterogeneous foundation-model artifacts via pipeline-ordered Shapley valuation and risk-adjusted payments, selecting zero strategic clients while improving...

  2. Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs

    cs.LG 2026-05 unverdicted novelty 7.0

    Semantic consensus on model outputs for public prompts enables federated LLM fine-tuning that matches parameter-aggregation baselines with orders-of-magnitude lower communication.

  3. Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    AS-LoRA adaptively chooses which LoRA factor to update per layer and round using a curvature-aware second-order score, eliminating reconstruction error floors and improving performance in DP federated learning.

  4. Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

    cs.LG 2025-09 conditional novelty 6.0

    PrunedLoRA tunes large-rank LoRA adapters and progressively structurally prunes A and B to a target low rank, empirically closing much of the LoRA-to-full-fine-tuning gap.

  5. DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

    cs.LG 2026-05 unverdicted novelty 5.0

    DP-LAC provides a new adaptive clipping technique for DP-SGD in federated LLM fine-tuning that improves accuracy by 6.6% on average without consuming additional privacy budget or requiring new hyperparameters.

  6. FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMs

    cs.CR 2026-04 unverdicted novelty 5.0

    FedSpy-LLM uses gradient decomposition and iterative alignment to reconstruct larger batches and longer sequences of training data from LLM gradients in federated settings, including with PEFT methods.

  7. ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models

    cs.LG 2025-11 conditional novelty 5.0

    ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.