REVIEW 8 cited by
LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Low-Rank Adaptation (LoRA) provides an effective yet efficient solution for fine-tuning large language models (LLM). The modular and plug-and-play nature of LoRA enables the integration of diverse domain-specific LoRAs to enhance the capabilities of LLMs. Previous research on exploiting multiple LoRAs either focuses on specific isolated downstream tasks or fixes the selection of LoRAs during training. However, in real-world scenarios, LLMs receive diverse prompts covering different tasks, and the pool of candidate LoRAs is often dynamically updated. To bridge this gap, we propose LoraRetriever, a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts. LoraRetriever contains three main components: firstly, identifying and retrieving LoRAs relevant to the given input; secondly, formulating strategies for effectively integrating the retrieved LoRAs; and thirdly, developing efficient batch inference to accommodate heterogeneous requests. Experimental results indicate that LoraRetriever consistently outperforms the baselines, highlighting its practical effectiveness and versatility.
Forward citations
Cited by 8 Pith papers
-
FlexOlmo: Open Language Models for Flexible Data Use
FlexOlmo merges independently trained language-model experts, trained on private data, into a single mixture-of-experts model without joint training.
-
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
Joint Information Extraction Across Classical and Modern Chinese with Tea-MOELoRA
Tea-MOELORA uses separate task and era gates over LoRA experts to jointly train relation and event extraction across classical and modern Chinese, improving F1 over joint LoRA and existing LoRA-MoE baselines on most datasets.
-
Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communications
ContextLoRA partitions a single LLM's LoRA parameters by task and trains them in dependency-graph order, while ContextGear adds pipeline parallelism to cut training time, together outperforming multi-task LoRA baselin...
-
MEraser: An Effective Fingerprint Erasure Approach for Large Language Models
By fine-tuning on mismatched pairs and then clean pairs, MEraser drops fingerprint success rate to zero on three backdoor-based fingerprinting schemes across multiple LLMs, with a reusable LoRA adapter for transfer.
-
RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Models
RetrieveAll combines per-language LoRA adapters with retrieval of entity and context examples to improve multilingual NER, claiming an average 12.1% F1 gain on PAN-X.
-
Unlocking the Effectiveness of LoRA-FP for Seamless Transfer Implantation of Fingerprints in Downstream Models
Backdoor fingerprints trained into LoRA adapters on a base LLM transfer to derivative models with 100% trigger success and, in several scenarios, greater robustness than directly injected fingerprints.
Discussion (0). Continue with ORCID to comment.