REVIEW 21 cited by
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks. First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composition step. We show that by separating the two stages, i.e., knowledge extraction and knowledge composition, the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning. Our code and adapters are available at AdapterHub.ml.
Forward citations
Cited by 21 Pith papers
-
FlexOlmo: Open Language Models for Flexible Data Use
FlexOlmo merges independently trained language-model experts, trained on private data, into a single mixture-of-experts model without joint training.
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation
A Fisher-information-guided dynamic selection over a toolbox of LoRA, adapter, and frequency-adapter modules improves cross-domain remote sensing segmentation over static PEFT methods.
-
DivMerge: A divergence-based model merging method for multi-tasking
DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.
-
Time Series Foundation Models for Multivariate Financial Time Series Forecasting
Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...
-
CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning
A single LLaVA-based captioning model continuously controls caption length, descriptiveness, and word uniqueness by interpolating between learned endpoint conditioning vectors.
-
Video-Guided Text-to-Music Generation Using Public Domain Movie Collections
OSSL is the first self-hosted, mood-annotated video-music dataset, and a video adapter on MusicGen-Medium improves film music generation over text-only baselines.
-
GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation
GaRA-SAM improves SAM's robustness to image corruption by using input-dependent gating to adjust the effective rank of low-rank adapters, beating prior methods on robust segmentation benchmarks.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
Sensitivity-LoRA allocates LoRA ranks across layers using Hessian-based sensitivity metrics, improving average GLUE score by 0.74 over AdaLoRA on RoBERTa-base.
-
Tensorized Clustered LoRA Merging for Multi-Task Interference
Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.
-
Few-Shot Object Detection via Spatial-Channel State Space Model
A Mamba-based channel sequence model combined with spatial attention improves few-shot object detection on VOC and COCO.
-
Exploring Adapter Design Tradeoffs for Low Resource Music Generation
Adapter placement, architecture, and size strongly change generation quality and cost for MusicGen and Mustango on two non-Western genres, with late-layer, mid-sized (40M) adapters reported as the best tradeoff.
-
PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs
An iterative hybrid pruning method selects which PEFT modules to keep at each transformer layer, matching or improving fixed PEFT baselines on GLUE at 1% trainable parameters.
-
TALL -- A Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages
A trainable pipeline of translation models and a frozen LLM improves Hebrew last-word prediction accuracy to 5.59%, about twice the best baseline.
-
PFMBench: Protein Foundation Model Benchmark
A comprehensive benchmark of 17 protein foundation models across 38 tasks yields task correlations, a streamlined protocol, and identifies ProTrek as the strongest general performer.
-
LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
Low-rank residual-stream adapters match LoRA at near-equal parameter counts on code fine-tuning and DPO, and support inference-time scaling plus per-token routing of seven behaviors on one frozen 1.5B model.
-
GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation
GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.
-
Weight Spectra Induced Efficient Model Adaptation
Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.
-
MAP: Revisiting Weight Decomposition for Low-Rank Adaptation
MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.
-
Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models
A gated-adapter method with a sparsity regularizer is reported to match or slightly beat full fine-tuning on MNLI and BoolQ using 1.4% of parameters, but lacks the experimental detail needed to verify the claim.
Discussion (0). Sign in to comment.