REVIEW 8 cited by
Fusing Models with Complementary Expertise
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Training AI models that generalize across tasks and domains has long been among the open problems driving AI research. The emergence of Foundation Models made it easier to obtain expert models for a given task, but the heterogeneity of data that may be encountered at test time often means that any single expert is insufficient. We consider the Fusion of Experts (FoE) problem of fusing outputs of expert models with complementary knowledge of the data distribution and formulate it as an instance of supervised learning. Our method is applicable to both discriminative and generative tasks and leads to significant performance improvements in image and text classification, text summarization, multiple-choice QA, and automatic evaluation of generated text. We also extend our method to the "frugal" setting where it is desired to reduce the number of expert model evaluations at test time. Our implementation is publicly available at https://github.com/hwang595/FoE-ICLR2024.
Forward citations
Cited by 8 Pith papers
-
One for All: Update Parameterized Knowledge Across Multiple Models
One fine-tuned small model plus an ensemble step can update a fact across multiple large language models with a single edit, outperforming separate per-model editing.
-
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Introduces a multi-role red teaming framework using attacker and jury models that increases attack success rates by up to 7.9% on LLM faithfulness in question-answering tasks.
-
Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation
Adaptive selection and dynamic weighted fusion of source LLMs reduces knowledge interference and improves target model accuracy compared to FuseLLM.
-
LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead
LightRouter uses short preview outputs to filter a pool of LLMs down to two, then aggregates their full responses, beating ensemble baselines and matching costlier models.
-
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
CoSD fuses two LLMs at decoding time: it keeps the draft model's tokens unless a confidence rule or a decision tree over token probabilities says the assistant model's token is better.
-
KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems
A knowledge-aware Bayesian bandit that selects specialist LLM experts per task is reported to beat Mixture-of-Agents on AlpacaEval 2.0 and MT-Bench while using fewer models.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
Discussion (0). Continue with ORCID to comment.