REVIEW 9 cited by
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.
Forward citations
Cited by 9 Pith papers
-
UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning
UnHype generates concept-specific LoRA unlearning weights on the fly from CLIP text embeddings by training a hypernetwork to follow the gradient of an unlearning loss, enabling single- and multi-concept erasure in dif...
-
Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
A residual adapter called DFN plus singular-value sharpness regularization improves cross-domain few-shot segmentation by 2.69% and 4.68% MIoU over prior state-of-the-art in 1-shot and 5-shot settings.
-
Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
A task-conditioned projection operator converts a large transformer's weights into a smaller task-specialized transformer that outperforms same-size universal conditional models.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
(Almost) Free Modality Stitching of Foundation Models
A hypernetwork that generates connector weights for all image-text model pairs can rank pairs like grid search at about 10x lower training cost, but the best connector lags grid search by a few points.
-
Text-to-LoRA: Instant Transformer Adaption
A hypernetwork can generate task-specific LoRA adapters from a text description, and when trained with supervised fine-tuning it zero-shot outperforms a multi-task LoRA baseline on ten benchmarks.
-
Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generation
MuDi-Pro fine-tunes a multi-guided diffusion transformer with DPO using guidance-score preferences to improve controllability of traffic scenario generation on nuScenes.
-
Capacity and Redundancy Trade-offs in Multi-Task Learning
A shared representation's total per-task information is bounded by capacity plus label redundancy; clustered sharing wins exactly when interference reduction exceeds the redundancy it loses.
-
Meta-aware Learning in text-to-SQL Large Language Model
Combining schema, chain-of-thought, metadata knowledge, and tokenized prompt structures during fine-tuning improves text-to-SQL execution accuracy on private business databases compared to schema-only fine-tuning.
Discussion (0). Continue with ORCID to comment.