REVIEW 6 cited by
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We propose Model Swarms, a collaborative search algorithm to adapt LLMs via swarm intelligence, the collective behavior guiding individual systems. Specifically, Model Swarms starts with a pool of LLM experts and a utility function. Guided by the best-found checkpoints across models, diverse LLM experts collaboratively move in the weight space and optimize a utility function representing model adaptation objectives. Compared to existing model composition approaches, Model Swarms offers tuning-free model adaptation, works in low-data regimes with as few as 200 examples, and does not require assumptions about specific experts in the swarm or how they should be composed. Extensive experiments demonstrate that Model Swarms could flexibly adapt LLM experts to a single task, multi-task domains, reward models, as well as diverse human interests, improving over 12 model composition baselines by up to 21.0% across tasks and contexts. Further analysis reveals that LLM experts discover previously unseen capabilities in initial checkpoints and that Model Swarms enable the weak-to-strong transition of experts through the collaborative search process.
Forward citations
Cited by 6 Pith papers
-
Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition
Across four standard merging methods, refusal behavior from a large task vector overwrites fine-grained harm classification, leaving at most 12.9% accuracy.
-
Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging
A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimoda...
-
When One LLM Drools, Multi-LLM Collaboration Rules
A position paper that introduces a four-level taxonomy of multi-LLM collaboration (API, text, logit, weight) and argues it is essential for reliability, pluralism, and democratization.
-
PSO-Merging: Merging Models Based on Particle Swarm Optimization
PSO-Merging applies particle swarm optimization over model weight space, seeded with original and sparsified experts, to build multitask models that outperform existing merging baselines on several language benchmarks.
-
The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
Clustering-based routing plus self-consistency voting among ten 7B open models reportedly outranks GPT-4.1 and GPT-4.5 on average over 15 diverse benchmarks.
-
Why Do More Experts Fail? A Theoretical Analysis of Model Merging
The paper claims to prove an upper bound and diminishing returns in model merging, but the proofs are not sound and the heavy-tailed claim is contradicted by its own equations.
Discussion (0). Continue with ORCID to comment.