REVIEW 4 cited by
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task. Model MoErging methods aim to recycle expert models to create an aggregate system with improved performance or generalization. A key component of MoErging methods is the creation of a router that decides which expert model(s) to use for a particular input or application. The promise, effectiveness, and large design space of MoErging has spurred the development of many new methods over the past few years. This rapid pace of development has made it challenging to compare different MoErging methods, which are rarely compared to one another and are often validated in different experimental setups. To remedy such gaps, we present a comprehensive survey of MoErging methods that includes a novel taxonomy for cataloging key design choices and clarifying suitable applications for each method. Apart from surveying MoErging research, we inventory software tools and applications that make use of MoErging. We additionally discuss related fields of study such as model merging, multitask learning, and mixture-of-experts models. Taken as a whole, our survey provides a unified overview of existing MoErging methods and creates a solid foundation for future work in this burgeoning field.
Forward citations
Cited by 4 Pith papers
-
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
When One LLM Drools, Multi-LLM Collaboration Rules
A position paper that introduces a four-level taxonomy of multi-LLM collaboration (API, text, logit, weight) and argues it is essential for reliability, pluralism, and democratization.
-
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
Sparse adapters trained with max connection sensitivity outperform LoRA and full fine-tuning both alone and after merging 20 task experts, but still lag multitask training on unseen tasks.
Discussion (0). Continue with ORCID to comment.