Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TTMM pre-trains many local LoRA experts on data clusters and merges the most relevant few at test time, approximating test-time training with a 100x speedup and near-TTT perplexity.

desk verdict Solid method paper; main claim needs a cleaner evaluation on a uniform holdout. read the letter →

arxiv 2505.14136 v2 pith:Q6O45TFZ submitted 2025-05-20 cs.LG cs.AI

Ryo Bertolissi , Jonas Hübotter , Ido Hakimi , Andreas Krause This is my paper
classification cs.LGcs.AI
keywords modeltest-timettmmexpertsmodelstrainingcostmerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models usually do the same amount of work for every question: one fixed model produces every answer. An alternative, called test-time training (TTT), briefly fine-tunes the model on data related to the current prompt, which improves accuracy but is costly because every prompt needs its own gradient updates. This paper proposes Test-Time Model Merging (TTMM), which moves that work to training time. The training data is clustered into many groups, and a small low-rank adapter is trained for each group. At test time, the model computes a quick similarity between the prompt and the cluster centers, picks a few relevant adapters, and merges their parameters into a single model before generating tokens.

The experiments compare TTMM to TTT, to ensembling the adapters, and to a single fine-tuned model on Wikipedia and GitHub Python data, using Llama and Qwen base models. With ten active experts, TTMM reaches perplexity close to TTT, and on Wikipedia it is slightly better, while adding only a fixed overhead equivalent to generating about twenty tokens. Ensembling gives slightly better accuracy but costs a full forward pass per expert. The paper also gives a bound intended to show why a locally trained expert can approximate a TTT model, though that proof has a gap in the multi-step case.

The main limitations are that the perplexity results in the main table have no error bars, the test set is deliberately balanced across clusters, and the gains on MMLU are small. The method is still an attractive way to make prompt-specific adaptation cheap.

Extended reading notes

Core claim

The central claim is that TTMM 'approaches the performance of TTT without almost any test-time overhead' (Section 1, Figure 1) and 'with a 1B parameter base model, TTMM is more than 100x faster than TTT at test-time' (Abstract). If true, prompt-specific transductive adaptation can be amortized into pre-trained local adapters and executed at near-base inference cost, with accuracy close to per-prompt fine-tuning.

Load-bearing premise

The approximation argument assumes that clusters selected by centroid inner product contain data close to the prompt's nearest neighbors and have small diameter, which is required by Proposition C.1 (D' ∩ D_x* ≠ ∅) but not guaranteed by bisecting k-means. The proof's T-step recursion also needs an unstated Lipschitz condition on gradients in parameter space, so the formal claim that TTMM approximates TTT is not fully established.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Test-Time Model Merging (TTMM), a transductive adaptation method that pre-trains a large number of cluster-specific LoRA adapters on a clustering of the training corpus and, at test time, merges a small number of adapters whose centroids are closest to the prompt embedding, using sparse-softmax cross-attention weights. The authors argue that TTMM approximates test-time training (TTT) while amortizing its cost, and report perplexity improvements on Wikipedia and GitHub Python with Llama-3.2-1B and Qwen2.5-1.5B, a more than 100x test-time speedup over TTT, and small MMLU accuracy gains. The paper includes latency benchmarks, ablations of cross-attention versus uniform weighting, qualitative examples, and a theoretical proposition (Informal Proposition 3.1, formalized as Proposition C.1).

Significance. If the headline comparisons are correct, TTMM is practically valuable: it offers a way to spend training compute on many local experts and then perform prompt-specific adaptation at near-base inference cost. The manuscript has genuine strengths: it releases code and models; it carefully splits documents into prefix and suffix to avoid leaking the response into the prompt; it evaluates two base models and two corpora plus MMLU; it benchmarks test-time latency concretely; and it isolates the effect of cross-attention weighting from uniform weighting. However, the central accuracy claim rests primarily on an evaluation protocol that is stratified by cluster, and the theoretical appendix does not establish the claimed approximation for the multi-expert sparse-softmax merging actually used. These issues are fixable but currently make the strength of the headline claim uncertain.

major comments (3)
  1. [Appendix E, Table 1, Figure 1] The test-set construction in Appendix E is cluster-stratified: 'the test sets were constructed by selecting a single random example from each cluster's holdout set.' With K=100 in the main experiments, this yields only roughly 100 test documents, and since bisecting k-means produces highly imbalanced cluster sizes, this design oversamples small or rare clusters relative to their corpus frequency. That favors TTMM, because each rare-cluster test prompt has a dedicated expert trained on that cluster, whereas TTT must retrieve the 100 nearest training neighbors and may have less relevant data for rare topics. The Wikipedia Llama result in Table 1 (7.510 for TTMM vs 7.559 for TTT) may therefore not reflect performance on a uniform random holdout. Please rerun the main comparisons on a uniform random holdout, report the number of test documents, and provide standard errors or variance across seeds.
  2. [Appendix C, Informal Proposition 3.1] The proof of the multi-step version is not established. The recursion claimed at the end of Appendix C, ||θ^(t+1)_x* - θ'^(t+1)|| <= ||θ^(t)_x* - θ'^(t)|| + ηG(diam(D_x*) + diam(D')), requires a bound on how the gradient difference evolves as the two parameter vectors move apart, but Assumption (A1) only bounds gradient differences as a function of input distance x - x', not as a function of parameter distance. Without an additional Lipschitz condition on gradients in parameter space, the factor T in Informal Proposition 3.1 is unjustified. Moreover, Proposition C.1 treats a single selected expert, while Algorithm 2 uses a sparse-softmax weighted merge of several experts; none of the theoretical statements covers the actual merging operation. Please either prove a version for the merged estimator under explicit assumptions or clearly label the proposition as informal intuition only.
  3. [Section 3.2, Approximation 2] The approximation argument relies on the condition D' ∩ D_x* ≠ ∅ in Proposition C.1, but the TTMM selection rule does not guarantee that the selected cluster contains any of the prompt's true nearest neighbors. The paper's own one-dimensional example {−1, 0, 1} with prompt 0 shows that centroid-based selection can pick the cluster {−1, 1}, which has no nearest neighbor of the prompt. Since bisecting k-means does not by itself ensure small cluster diameter or that the selected centroid's cluster intersects the nearest-neighbor set, the statement that Proposition C.1 'motivates the approximation of TTT by TTMM' is stronger than what is proved. Please add a concrete analysis or an empirical check of how often the active experts actually contain true nearest neighbors of the prompt, or soften the claim accordingly.
minor comments (4)
  1. [Appendix D.3] There is a typo in the sentence 'SIFT yields only a negligible improvmenet over the cross-attention mechanism'; it should read 'improvement'.
  2. [Section 3.1] The phrase 'TTMM costs ≈20 tokens' appears as an incomplete sentence immediately after the paragraph on latency. Please rewrite it as a full sentence and state the exact conditions (number of active experts, generation length, hardware) used for this estimate.
  3. [Algorithm 2, Figure 5] The sparsity parameter is defined as τ ∈ [0, 1/K), but Figure 5 reports results with τ = 0.1 for K = 1000 experts, which violates the stated range. Please reconcile the definition with the reported hyperparameter values, or clarify that τ is treated as a fraction of total probability mass rather than a threshold on individual softmax weights.
  4. [Table 5] The comparison between 'summarized by centroid' and 'not summarized by centroid' should state explicitly that both variants use the same number of active experts, the same temperature, and the same sparsity, so that the 7.669 vs 7.633 difference is attributable solely to the centroid approximation.
Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The main results depend on choosing the number of clusters, temperature, sparsity, and active expert count; these are selected using the same corpora and holdout data. The approximation argument further relies on the locality of the embedding space, small cluster diameters, and smoothness assumptions that are stated only partially. No new entities are introduced.

free parameters (4)
  • Number of clusters K = 100 (with 1000 in ablations)
    Determines granularity of local experts and CPU memory requirement; selected via elbow method on k-means loss (Figure 6).
  • Temperature beta = Not stated as a single value; tuned per dataset on holdout
    Controls softmax sharpness of merging coefficients; tuned on holdout data in Algorithm 2.
  • Sparsity tau = 0.01
    Prunes low-weight experts; fixed globally after being selected on holdout as the best accuracy/efficiency trade-off.
  • Number of active experts N (upper bound) = 10
    Used as upper bound for dynamic selection; chosen based on coverage saturation in Figure 6.
assumptions (3)
  • domain assumption The loss L is G-smooth in input x and the network f is L-Lipschitz in parameters theta (Assumptions A1, A2, Appendix C).
    Used to prove Proposition 3.1; the recursion to T steps additionally requires smoothness of gradients in theta, which is not stated.
  • domain assumption A normalized sequence embedding model phi exists and captures semantic locality.
    Clustering and expert selection rely on all-mpnet-base-v2 embeddings being a meaningful similarity space for prompts.
  • ad hoc to paper Clusters produced by bisecting k-means have small diameter and cover the data.
    Approximation 2 and Proposition 3.1 require small-diameter clusters that contain a prompt's nearest neighbors; this is not guaranteed by the algorithm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging." pith.science (2026). https://pith.science/paper/Q6O45TFZ

@misc{pith2026250514136,
  author       = {Pith},
  title        = {Pith review of: Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6O45TFZ}},
  note         = {Machine review of arXiv:2505.14136}
}
read the original abstract

Mixture of expert (MoE) models are a promising approach to increasing model capacity without increasing inference cost, and are core components of many state-of-the-art language models. However, current MoE models typically use only few experts due to prohibitive training and inference cost. We propose Test-Time Model Merging (TTMM) which scales the MoE paradigm to an order of magnitude more experts and uses model merging to avoid almost any test-time overhead. We show that TTMM is an approximation of test-time training (TTT), which fine-tunes an expert model for each prediction task, i.e., prompt. TTT has recently been shown to significantly improve language models, but is computationally expensive. We find that performance of TTMM improves with more experts and approaches the performance of TTT. Moreover, we find that with a 1B parameter base model, TTMM is more than 100x faster than TTT at test-time by amortizing the cost of TTT at train-time. Thus, TTMM offers a promising cost-effective approach to scale test-time training.

Figures

Figures reproduced from arXiv: 2505.14136 by the authors.

Figure 1
Figure 1. Accuracy gains of TTMM when scaling test-time compute (left) and pre-training [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Standard inductive language modeling pre-trains a single model on the dataset (shown in gray) which is then used for prediction. Test-Time Training (TTT) improves performance by fine-tuning the model at test-time on data related to the prompt, but is computationally expensive. Test-Time Model Merging (TTMM) fine-tunes the pre-trained model to many local neighborhoods at train-time, and merges the local models relate… view at source ↗
Figure 3
Figure 3. Illustration of TTMM: At train-time (Algorithm 1), TTMM clusters the training data (shown in gray) into many local neighborhoods and trains a separate expert model for each cluster. At test-time (Algorithm 2), TTMM dynamically selects a subset of expert models related to the prompt and merges their parameters to form a single task-specific model. The test-time stage is compute and memory efficient, using the previou… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: TTMM has negligible compute overhead at test-time. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: On each dataset, we evaluate the performance improvement of TTMM’s [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Left: Perplexity (brighter is better) of 1000 experts on clusters of the Wikipedia dataset. The lowest perplexity is achieved on the diagonal, i.e., when the expert model is evaluated on holdout data from the cluster it was trained on. The block-diagonal structure is a…
Figure 7
Figure 7. Figure 7: We merge LoRA adapters using PyTorch’s einsum, before applying the low-rank updates to the model parameters. A Additional Related Work Retrieval-Augmented Generation. An alternative to specializing model parameters at test￾time (i.e., TTT) is to instead specialize pred…
Figure 8
Figure 8. Figure 8: We evaluate weighting experts according to SIFT’s uncertainty estimates, which [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: We evaluate ensembling models with inner product weighting (like in TTMM) [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: On Wikipedia and with Llama-3.2-1B as base model, we evaluate different [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Selecting the most confident 32-token prefix and completing only it gives better accuracy per compute than majority voting on five math reasoning datasets, using only the model's own confidence as a selector.

Reference graph

Works this paper leans on

83 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    Git re-basin: Merging models modulo permutation symmetries

    Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023

    2023

  2. [2]

    The surprising effectiveness of test-time training for few-shot-learning

    Ekin Aky \"u rek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot-learning. In ICML, 2025

    2025

  3. [3]

    Locally weighted learning

    Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, 1997

    1997

  4. [4]

    Active fine-tuning of multi-task policies

    Marco Bagatella, Jonas Hübotter, Georg Martius, and Andreas Krause. Active fine-tuning of multi-task policies. In ICML, 2025

    work page 2025

  5. [5]

    Improving language models by retrieving from trillions of tokens

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML, 2022

    work page 2022

  6. [6]

    Local learning algorithms

    L \'e on Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 4 0 (6), 1992

    work page 1992

  7. [7]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165, 2020

    arXiv 2005

  8. [8]

    Dam: Dynamic adapter merging for continual video qa learning

    Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan-Bo Lin, Mohit Bansal, and Gedas Bertasius. Dam: Dynamic adapter merging for continual video qa learning. arXiv preprint arXiv:2403.08755, 2024

    arXiv 2024

Show all 83 references
  1. [9]

    Unified scaling laws for routed language models

    Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In ICML, 2022

    2022

  2. [10]

    Robust locally weighted regression and smoothing scatterplots

    William S Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American statistical association, 74 0 (368), 1979

    1979

  3. [11]

    Locally weighted regression: an approach to regression analysis by local fitting

    William S Cleveland and Susan J Devlin. Locally weighted regression: an approach to regression analysis by local fitting. Journal of the American statistical association, 83 0 (403), 1988

    1988

  4. [12]

    Model merging by uncertainty-based gradient matching

    Nico Daheim, Thomas M \"o llenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In ICLR, 2024

    2024

  5. [13]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  6. [14]

    One-minute video generation with test-time training

    Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. arXiv preprint arXiv:2504.05298, 2025

    2025 arXiv

  7. [15]

    Ensemble methods in machine learning

    Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, 2000

    2000

  8. [16]

    Test-time training with masked autoencoders

    Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. In NeurIPS, 2021

    2021

  9. [17]

    Loss surfaces, mode connectivity, and fast ensembling of dnns

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In NeurIPS, 2018

    2018

  10. [18]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  12. [20]

    Scaling expert language models with unsupervised domain discovery

    Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery. arXiv preprint arXiv:2303.14177, 2023

    2023 arXiv

  13. [21]

    Retrieval augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In ICML, 2020

    2020

  14. [22]

    Test-time training on nearest neighbors for large language models

    Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In ICLR, 2024

    2024

  15. [23]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021

    2021

  16. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022

    2022

  17. [25]

    Transductive active learning: Theory and applications

    Jonas Hübotter, Lenart Treven, Yarden As, and Andreas Krause. Transductive active learning: Theory and applications. In NeurIPS, 2024

    2024

  18. [26]

    Efficiently learning at test-time: Active fine-tuning of llms

    Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms. In ICLR, 2025

    2025

  19. [27]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023

    2023

  20. [28]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In UAI, 2018

    2018

  21. [29]

    Data clustering: 50 years beyond k-means

    Anil K Jain. Data clustering: 50 years beyond k-means. Pattern recognition letters, 31 0 (8), 2010

    2010

  22. [30]

    Online domain adaptation of a pre-trained cascade of classifiers

    Vidit Jain and Erik Learned-Miller. Online domain adaptation of a pre-trained cascade of classifiers. In CVPR, 2011

    2011

  23. [31]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In ACL, 2023

    2023

  24. [32]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023

    2023

  25. [33]

    Why train everything? tint a single layer for multi-task model merging

    Aecheon Jung, Seunghwan Lee, Dongyoon Han, and Sungeun Hong. Why train everything? tint a single layer for multi-task model merging. arXiv preprint arXiv:2412.19098, 2024

    2024 arXiv

  26. [34]

    Generalization through memorization: Nearest neighbor language models

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In ICLR, 2020

    2020

  27. [35]

    Dynamic evaluation of neural sequence models

    Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In ICML, 2018

    2018

  28. [36]

    Dynamic evaluation of transformer language models

    Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of transformer language models. arXiv preprint arXiv:1904.08378, 2019

    1904 arXiv

  29. [37]

    Mediator: Memory-efficient llm merging with less parameter conflicts and uncertainty based routing

    Kunfeng Lai, Zhenheng Tang, Xinglin Pan, Peijie Dong, Xiang Liu, Haolan Chen, Li Shen, Bo Li, and Xiaowen Chu. Mediator: Memory-efficient llm merging with less parameter conflicts and uncertainty based routing. arXiv preprint arXiv:2502.04411, 2025

    2025 arXiv

  30. [38]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS, 2019

    2019

  31. [39]

    Branch-train-merge: Embarrassingly parallel training of expert language models

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306, 2022

    2022 arXiv

  32. [40]

    Dexperts: Decoding-time controlled text generation with experts and anti-experts

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In ACL, 2021

    2021

  33. [41]

    Multivariable functional interpolation and adaptive networks

    David Lowe and D Broomhead. Multivariable functional interpolation and adaptive networks. Complex systems, 2 0 (3), 1988

    1988

  34. [42]

    Twin-merging: Dynamic integration of modular expertise in model merging

    Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. In NeurIPS, 2024

    2024

  35. [43]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. In NeurIPS, 2022

    2022

  36. [44]

    Pack of llms: Model fusion at test-time via perplexity optimization

    Costas Mavromatis, Petros Karypis, and George Karypis. Pack of llms: Model fusion at test-time via perplexity optimization. In COLM, 2024

    2024

  37. [45]

    Fast learning in networks of locally-tuned processing units

    John Moody and Christian J Darken. Fast learning in networks of locally-tuned processing units. Neural computation, 1 0 (2), 1989

    1989

  38. [46]

    On estimating regression

    Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9 0 (1), 1964

    1964

  39. [47]

    Dawin: Training-free dynamic weight interpolation for robust adaptation

    Changdae Oh, Yixuan Li, Kyungwoo Song, Sangdoo Yun, and Dongyoon Han. Dawin: Training-free dynamic weight interpolation for robust adaptation. In ICLR, 2025

    2025

  40. [48]

    On estimation of a probability density function and mode

    Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathematical statistics, 33 0 (3), 1962

    1962

  41. [49]

    Pytorch: An imperative style, high-performance deep learning library

    A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019

    1912 arXiv

  42. [50]

    Mingle: Mixtures of null-space gated low-rank experts for test-time continual model merging

    Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, and Hongliang Li. Mingle: Mixtures of null-space gated low-rank experts for test-time continual model merging. arXiv preprint arXiv:2505.11883, 2025

    2025

  43. [51]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In IJCNLP, 2019

    2019

  44. [52]

    Remarks on Some Nonparametric Estimates of a Density Function

    Murray Rosenblatt. Remarks on Some Nonparametric Estimates of a Density Function . The Annals of Mathematical Statistics, 27 0 (3), 1956

    1956

  45. [53]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023

    2023

  46. [54]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR, 2017

    2017

  47. [55]

    zero-shot

    Assaf Shocher, Nadav Cohen, and Michal Irani. “zero-shot” super-resolution using deep internal learning. In CVPR, 2018

    2018

  48. [56]

    Ladder: Self-improving llms through recursive problem decomposition

    Toby Simonds and Akira Yoshiyama. Ladder: Self-improving llms through recursive problem decomposition. arXiv preprint arXiv:2503.00735, 2025

    2025 arXiv

  49. [57]

    Mpnet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurIPS, 2020

    2020

  50. [58]

    Branch-train-mix: Mixing expert llms into a mixture-of-experts llm

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozi \`e re, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. Branch-train-mix: Mixing expert llms into a mixture-of-experts llm. arXiv preprint arXiv:2403.07816, 2024

    2024 arXiv

  51. [59]

    Test-time training with self-supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020

    2020

  52. [60]

    Learning to (learn at test time): Rnns with expressive hidden states

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024

    2024 arXiv

  53. [61]

    Merging multi-task models via weight-ensembling mixture of experts

    Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. In ICML, 2024 a

    2024

  54. [62]

    Parameter efficient multi-task model fusion with partial linearization

    Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial linearization. In ICLR, 2024 b

    2024

  55. [63]

    Merging models on the fly without retraining: A sequential approach to scalable continual model merging

    Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging. arXiv preprint arXiv:2501.09522, 2025

    2025 arXiv

  56. [64]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023

    2023

  57. [65]

    Smooth regression analysis

    Geoffrey S Watson. Smooth regression analysis. Sankhy \=a : The Indian Journal of Statistics, Series A , 1964

    1964

  58. [66]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022

    2022

  59. [67]

    Wikimedia downloads, 2025

    Wikimedia Foundation . Wikimedia downloads, 2025. URL https://dumps.wikimedia.org

    2025

  60. [68]

    Deep learning is not so mysterious or different

    Andrew Gordon Wilson. Deep learning is not so mysterious or different. arXiv preprint arXiv:2503.02113, 2025

    2025 arXiv

  61. [69]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  62. [70]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In NeurIPS, 2023

    2023

  63. [71]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    2024 arXiv

  64. [72]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666, 2024 b

    2024 arXiv

  65. [73]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024 c

    2024

  66. [74]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024

    2024

  67. [75]

    Jasper and stella: distillation of sota embedding models

    Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models. arXiv preprint arXiv:2412.19048, 2024 a

    2024 arXiv

  68. [76]

    Bam! just like that: Simple and efficient parameter upcycling for mixture of experts

    Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar Talupuru, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom, Sebastian Ruder, Ahmet \"U st \"u n, et al. Bam! just like that: Simple and efficient parameter upcycling for mixture of experts. In NeurIPS, 2024 b

    2024

  69. [77]

    Bts: Harmonizing specialized experts into a generalist llm

    Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, et al. Bts: Harmonizing specialized experts into a generalist llm. arXiv preprint arXiv:2502.00075, 2025

    2025 arXiv

  70. [78]

    Mixture-of-experts with expert choice routing

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. In NeurIPS, 2022

    2022

  71. [79]

    Ttrl: Test-time reinforcement learning

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025

    2025 arXiv

  72. [80]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  73. [81]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  74. [82]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  75. [83]

    G5ɞ8E(wZ N8,q

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.