Pith. sign in

REVIEW 10 cited by

AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.12410 v2 pith:CQ4VVR5U submitted 2022-05-24 cs.CL cs.AIcs.LG

Yaqing Wang , Sahaj Agarwal , Subhabrata Mukherjee , Xiaodong Liu , Jing Gao , Ahmed Hassan Awadallah , Jianfeng Gao This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords adamixfine-tuningpeftmethodmixtureparameter-efficientparameterstasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules -- given the underlying PEFT method of choice -- introduced in each Transformer layer while keeping most of the PLM weights frozen. For instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture of low rank decomposition matrices like LoRA to improve downstream task performance over the corresponding PEFT methods for fully supervised and few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the same computational cost and the number of tunable parameters as the underlying PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Improved Representation Steering for Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RePS, a reference-free bidirectional preference optimization objective, improves representation steering and suppression for Gemma models, outperforming language-modeling objectives and approaching prompting performance.

  2. Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A gated Shift-FFN adapter that adds the previous token's representation to the current token's before the feedforward layer reduces repetitive looping and improves math accuracy in LoRA fine-tuned models trained on lo...

  3. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  4. Unified Semantic Modeling Framework for Large-Scale Job Understanding at LinkedIn

    cs.AI 2026-06 conditional novelty 5.0 of 10

    A GPT-4-distilled small LM plus grouped LoRA adapters improves LinkedIn's job-attribute classification over legacy models.

  5. Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

    cs.SE 2026-01 conditional novelty 5.0 of 10

    Multi-task QLoRA on Qwen2.5-Coder matches or beats single-task QLoRA and full fine-tuning for code generation and Python summarization, but lags in Java-to-C# translation.

  6. MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection

    cs.SD 2025-09 conditional novelty 5.0 of 10

    MoLEx combines LoRA adapters with a top-K expert router inside a frozen WavLM model, achieving 5.56% EER on ASVSpoof 5 without augmentation.

  7. PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An iterative hybrid pruning method selects which PEFT modules to keep at each transformer layer, matching or improving fixed PEFT baselines on GLUE at 1% trainable parameters.

  8. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  9. $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.

  10. Learning Text Styles: A Study on Transfer, Attribution, and Verification

    cs.CL 2025-07 conditional novelty 3.0 of 10

    A thesis compiles published work claiming that lightweight adapters, contrastive disentanglement, and instruction tuning improve text style transfer, authorship attribution, and authorship verification.

Pith tools