Pith. sign in

REVIEW 9 cited by

Knowledge Fusion of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.10491 v2 pith:XDIPRS7A submitted 2024-01-19 cs.CL

Fanqi Wan , Xinting Huang , Deng Cai , Xiaojun Quan , Wei Bi , Shuming Shi This is my paper · ORCID
classification cs.CL
keywords llmscapabilitiesmodelfusionknowledgemodelsapproachcode
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures--Llama-2, MPT, and OpenLLaMA--across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at \url{https://github.com/fanqiwan/FuseLLM}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text

    cs.AI 2026-05 conditional novelty 6.0 of 10

    A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.

  2. Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units

    cs.AI 2025-08 conditional novelty 6.0 of 10

    MCSU-based vocabulary alignment plus distance-based dynamic selection (DDS) lets several LLMs vote token-by-token, beating single models and prior ensemble baselines on multiple reasoning benchmarks without training.

  3. Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

    cs.AI 2026-07 conditional novelty 5.0 of 10

    After truncating or expanding checkpoints to a shared shape, small-ratio weight averaging slightly improves average benchmark scores over strong Qwen sources, but headline gains are inflated by per-task best-ratio selection.

  4. CoCre-Sam (Kokkuri-san): Modeling Ouija Board as Collective Langevin Dynamics Sampling from Fused Language Models

    cs.MA 2025-07 conditional novelty 5.0 of 10

    Ouija board movement is modeled as collective Langevin dynamics sampling from a product-of-experts fusion of the participants' language models.

  5. Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    An affine map trained on The Pile transfers steering vectors from Gemma-2B to Gemma-9B and reproduces much of the large model's native steering behavior.

  6. LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead

    cs.AI 2025-05 conditional novelty 5.0 of 10

    LightRouter uses short preview outputs to filter a pool of LLMs down to two, then aggregates their full responses, beating ensemble baselines and matching costlier models.

  7. Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

    cs.CL 2025-02 conditional novelty 5.0 of 10

    CoSD fuses two LLMs at decoding time: it keeps the draft model's tokens unless a confidence rule or a decision tree over token probabilities says the assistant model's token is better.

  8. Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration

    cs.NI 2025-07 conditional novelty 4.0 of 10

    A survey of multi-LLM systems in edge computing, covering architectures, enabling technologies, trust mechanisms, applications, and open datasets for edge general intelligence.

  9. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

Pith tools