Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Merge to Mix: Mixing Datasets via Model Merging

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Merged model accuracy predicts fine-tuned-on-the-mixture accuracy, so dataset mixtures can be selected by parameter averaging instead of retraining.

desk verdict A genuinely new surrogate idea for dataset mixture selection, with real but incomplete empirical support; the theory overclaims what Pearson correlation buys you. read the letter →

arxiv 2505.16066 v1 pith:RKJ4SD6Y submitted 2025-05-21 cs.LG cs.AIcs.CL

Zhixu Silvia Tao , Kasper Vinken , Hao-Wei Yeh , Avi Cooper , Xavier Boix This is my paper
classification cs.LGcs.AIcs.CL
keywords modelmergingdatasetmixingdataselectionfine-tuningsurrogateparameteraveragingtaskarithmeticlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that when several models are fine-tuned on separate datasets and then merged by averaging their parameters, the merged model's accuracy on a target task tracks the accuracy of a model fine-tuned on the union of those datasets. If this correlation holds, the best dataset mixture can be found by evaluating cheap merged models instead of running a full fine-tuning run for every candidate mixture. The authors report average Pearson correlations of 0.78 on eight vision benchmarks and 0.57 on nine language benchmarks, and show that selecting by the merged model outperforms similarity-based and random baselines, often approaching the oracle's accuracy.

What carries the argument

The surrogate model $\tilde{\theta}(S_\alpha) = \frac{1}{|S_\alpha|}\sum_{i:\alpha_i=1}\theta_i^*$, the parameter-space average of individually fine-tuned models. It carries the argument by replacing the expensive map $\alpha\mapsto\theta^*(S_\alpha)$ with a cheap map $\alpha\mapsto\tilde{\theta}(S_\alpha)$, under the assumption that the two are positively correlated on the target task.

What would settle it

On a fresh suite of datasets, compute for every mixture both the merged-model accuracy and the full-fine-tuned accuracy on a target task. The claim fails if a task shows high overall Pearson correlation between the two, yet the mixture scoring best under the merged model is far below the oracle in fine-tuned accuracy—for instance, if the surrogate's top mixture ranks near the bottom among fine-tuned mixtures. Reporting the top-k overlap of the two rankings would settle whether the arg-min transfer holds.

Watch

Extended reading notes

Core claim

The central claim is formalized as Eq. (5): the performance of the merged model $\tilde{\theta}(S_\alpha)$ (the plain average of models fine-tuned on each selected dataset) is positively correlated with the performance of the mixture-fine-tuned model $\theta^*(S_\alpha)$ on the target task. Because a positive correlation preserves the location of the arg min under monotone scaling, the surrogate's best mixture approximately equals the true best mixture, which is stated as Eq. (6). The paper treats this correlation as an empirical regularity and uses it to turn dataset-mixture selection into a parameter-averaging problem: fine-tune once per dataset, average subsets of those models, evaluate each average on the target, and fine-tune only the winning mixture.

Load-bearing premise

The load-bearing premise is that the merged model's accuracy tracks the mixture-fine-tuned model's accuracy closely enough near the top of the ranking that the surrogate's best mixture is essentially the true best mixture, a condition the paper validates on only eight vision and nine language benchmarks.

Editorial extensions

If this is right

  • Mixture selection can be done with $N$ fine-tuning runs instead of up to $2^N$, making exhaustive search over binary mixtures practical for moderate $N$.
  • The selected mixture, when fine-tuned, yields test accuracy close to the oracle chosen by exhaustive fine-tuning; in several vision cases the merged model's own accuracy even exceeds the oracle.
  • Similarity-based selection, a common cheap alternative, shows weak or negative correlation with true mixture-fine-tuned accuracy (0.19 and 0.17 for cosine; -0.29 and -0.14 for L2), so embedding similarity is not a reliable proxy.
  • The method applies across modalities: the correlation appears for both CLIP-ViT-B-32 image classification and Llama-3-8B-Instruct with IA3 fine-tuning on Japanese language skills.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the correlation is a general property of fine-tuned model families, the same trick could rank weighted mixtures (with weights entering the average) or larger dataset pools by combining the surrogate with a search algorithm rather than exhaustive enumeration.
  • The reported correlation is averaged over tasks; the paper does not check whether the surrogate's arg min lies near the true arg min specifically at the top of the ranking. A testable next step is to measure the performance gap between the surrogate-selected mixture and the oracle on new task suites, not just the global Pearson coefficient.
  • The observation that merged models sometimes beat the oracle and resist fine-tuning degradation suggests the merging step itself acts as a regularizer; probing why averaging protects against hurtful fine-tuning could extend the method's applicability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Merge to Mix, a method for selecting a subset of datasets to fine-tune a pretrained LM on a target task. The method first fine-tunes one model per candidate dataset from the same pretrained checkpoint, then for every binary mixture vector it averages the corresponding model weights (Eq. 4) and evaluates the resulting merged model on the target validation set; the mixture with the best surrogate score is selected and used for a final fine-tuning run. The central assumption, stated in Section 3.2, is that the loss of the merged model is positively correlated with the loss of a model fine-tuned directly on the same mixture (Eq. 5), so that the arg min transfers (Eq. 6). The paper reports average Pearson correlations of 0.78 on 8 image classification tasks (CLIP-ViT-B-32) and 0.57 on 9 language tasks (Llama-3-8B-Instruct with IA3), and compares the selected mixtures against all-data fine-tuning, six similarity-based baselines, random selection, and an oracle in Tables 1 and 2.

Significance. The core idea is timely and practically valuable: replacing one full fine-tuning run per candidate mixture with a parameter-averaging surrogate turns exponential mixture search into something tractable for moderate N, and the initial evidence that merged-model accuracy tracks mixture-fine-tuned accuracy is nontrivial. The paper is also honest about the scope of its experiments (N in {6,7}) and proposes concrete extensions to larger N. However, the load-bearing claim is not just that the two quantities are correlated over the whole distribution, but that the surrogate optima are close to the true optima. The reported evidence, especially the moderate language correlation and the visible gaps in Table 2, does not yet establish that claim. If the revision adds rank-based agreement metrics and error bars, the contribution would be substantially strengthened.

major comments (4)
  1. [Section 3.2, Eq. (5)-(6)] The notation in Eq. (5) is incorrect: proportional-to (∝) is a much stronger statement than 'positively correlated', and the paper does not establish proportionality. Eq. (6) then invokes invariance of the arg min under positive scalar scaling, but Pearson correlation does not imply scalar proportionality or even monotonicity. What is needed for Eq. (6) is that the surrogate loss is a monotonically increasing function of the true loss, at least in a neighborhood of the optimum. As written, the central choice of the surrogate-selected mixture is not justified by the correlation evidence. Please state the assumption as a monotone/rank-preserving relationship, or prove a weaker sufficient condition, and report the corresponding empirical quantities (e.g., Spearman's rho, Kendall's tau, top-k overlap).
  2. [Section 4.2] The correlation analysis averages Pearson coefficients over all 2^N - 1 mixtures per task. A high global Pearson correlation can coexist with poor ordering near the top of the surrogate score, which is exactly the regime used for selection. The paper should report, per task, the rank correlation, the fraction of true top-k mixtures that appear in the surrogate top-k, and the true-performance percentile of the mixture selected by the surrogate. The gaps in Table 2 (Mathematical Reasoning: 0.386 selected vs 0.490 oracle; Question Answering: 0.395 vs 0.462) suggest that this is not a purely hypothetical concern.
  3. [Tables 1 and 2] No error bars, confidence intervals, or significance tests are reported for any of the selection methods. Since the differences between Merge to Mix and the best similarity baseline are often small (e.g., Translation 0.211 vs 0.208), it is not possible to assess whether the reported improvements are reliable. Please report standard deviations over multiple seeds or, if that is infeasible, bootstrap confidence intervals and paired tests over the task-level results.
  4. [Section 4.3 and Abstract] The abstract and Section 4.3 claim that Merge to Mix 'surpasses state-of-the-art methods in dataset selection', but the experimental comparison includes only all-data fine-tuning, similarity-based heuristics, and random selection. Recent dataset selection methods such as proxy-model or predictive-model approaches (e.g., DoGE, DSDM, RegMix, mentioned in Related Work) are not evaluated. Either add such comparisons or soften the claim to 'surpasses the tested baselines'.
minor comments (6)
  1. [Section 3.2, Eq. (5)] The phrase 'positively correlated' is used to justify ∝; please replace the proportional symbol with a monotone-relationship or correlation notation, and state explicitly that Eq. (6) is an approximation that requires rank preservation.
  2. [Algorithm 1] The loop at line 3 uses index n but the assignment at line 4 writes θ*_i; the index should be consistent (either i throughout or n throughout).
  3. [Section 3.3] The text introduces 'the algorithm of Merge and Mix' while the paper and Algorithm 1 are titled 'Merge to Mix'; please unify the naming.
  4. [Appendix A.3] For the language experiments, please report the optimizer, learning-rate schedule, batch size, and IA3 hyperparameters (rank, scaling) so that the fine-tuning procedure is fully reproducible.
  5. [Tables 1 and 2] The column headers contain missing spaces ('SimilaritySelection', 'RandomSelection') and the formatting of bold/underline is inconsistent; please clean up the tables.
  6. [Section 1] The phrase 'large models (LMs)' is redundant; consider 'large models' or 'language models' since the paper also covers vision models.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; central surrogate claim is empirically validated on held-out mixtures, with only one minor non-load-bearing self-citation.

full rationale

The derivation chain is self-contained. Eq (4) defines the merged-model surrogate, Eq (5) posits positive correlation with the mixture-fine-tuned model, Eq (6) transfers the arg min, and Section 4.3 evaluates the selected mixture by actually fine-tuning on it and reporting held-out test accuracy. The surrogate is chosen using the target validation set, while the reported final performance comes from an independently fine-tuned model on a test split, so no fitted quantity is renamed as a prediction and no result is forced by construction. The single-dataset points, where the merged and fine-tuned models coincide by definition, are explicitly excluded from the reported correlations, which is an honest treatment rather than a circular move. The only self-citation is [55], used to justify uniform fine-tuning hyperparameters; it does not establish the correlation in Eq (5) or the transfer in Eq (6), so it is not load-bearing. The gap between the written proportionality in Eq (5) and the actually measured Pearson correlation is a correctness and robustness concern, not a circularity, and does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method's central assumption is the empirical correlation in Eq (5), which is a domain assumption tested on a limited set of tasks. No parameters are fitted; the method relies on standard model merging and fine-tuning practices.

assumptions (4)
  • domain assumption Fine-tuned model parameters can be meaningfully averaged to produce a usable model
    The method relies on model merging, established in prior work (e.g., model soups, task arithmetic). Invoked in Section 3.2 when defining the surrogate.
  • domain assumption Uniform fine-tuning hyperparameters across datasets improve merging
    The paper cites [55] by the same first author to justify using the same learning rate, epochs, and batch size for all datasets. Invoked in Section 3.3 Step 1.
  • domain assumption The performance of the merged model is positively correlated with the mixture-fine-tuned model (Eq 5)
    This is the central empirical assumption, validated only on 8 vision and 9 language tasks in Section 4.2. Invoked in Section 3.2 Equation (5).
  • standard math Arg min is invariant to positive scalar scaling
    Used in Section 3.2 Equation (6) to claim that minimizing the surrogate loss approximates minimizing the true loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Merge to Mix: Mixing Datasets via Model Merging." pith.science (2026). https://pith.science/paper/RKJ4SD6Y

@misc{pith2026250516066,
  author       = {Pith},
  title        = {Pith review of: Merge to Mix: Mixing Datasets via Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKJ4SD6Y}},
  note         = {Machine review of arXiv:2505.16066}
}
abstract

Mixing datasets for fine-tuning large models (LMs) has become critical for maximizing performance on downstream tasks. However, composing effective dataset mixtures typically relies on heuristics and trial-and-error, often requiring multiple fine-tuning runs to achieve the desired outcome. We propose a novel method, $\textit{Merge to Mix}$, that accelerates composing dataset mixtures through model merging. Model merging is a recent technique that combines the abilities of multiple individually fine-tuned LMs into a single LM by using a few simple arithmetic operations. Our key insight is that merging models individually fine-tuned on each dataset in a mixture can effectively serve as a surrogate for a model fine-tuned on the entire mixture. Merge to Mix leverages this insight to accelerate selecting dataset mixtures without requiring full fine-tuning on each candidate mixture. Our experiments demonstrate that Merge to Mix surpasses state-of-the-art methods in dataset selection for fine-tuning LMs.

Figures

Figures reproduced from arXiv: 2505.16066 by the authors.

Figure 1
Figure 1. Correlation plots between the performance of mixture-fine-tuned models and different [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Individual correlation plot between merged model and mixture-fine-tuned model for each [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Individual correlation plot between the performance of merged models and mixture-fine [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Correlation plots between the performance of the mixture-fine-tuned models and average of [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Correlation plots between the performance of the mixture-fine-tuned models and average of [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Correlation plots between the performance of the mixture-fine-tuned models and maximum [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Correlation plots between the performance of the mixture-fine-tuned models and minimum [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Decentralized Multi-task Dataset Valuation via Model Merging

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Task-arithmetic model merging approximates multi-task coalition utilities well enough to recover Dataset Shapley rankings privately and without retraining.

  2. Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Weight interpolation between two DeepSeek models, restricted to routed expert tensors, yields a 671B Chimera model with near-R1 reasoning and roughly 40% lower output-token count.

Reference graph

Works this paper leans on

69 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Strong Copyright Protection for Language Models via Adaptive Model Fusion

    Javier Abad, Konstantin Donhauser, Francesco Pinto, and Fanny Yang. Strong copyright protection for language models via adaptive model fusion.arXiv preprint arXiv:2407.20105, 2024

    work page Pith review arXiv 2024

  2. [2]

    Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

    arXiv 2023

  3. [3]

    Unsupervised Domain Clusters in Pretrained Language Models

    Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. arXiv preprint arXiv:2004.02105, 2020

    work page Pith review arXiv 2004

  4. [4]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  5. [5]

    Git re-basin: Merging models modulo permutation symmetries.arXiv preprint arXiv:2209.04836, 2022

    Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries.arXiv preprint arXiv:2209.04836, 2022

    arXiv 2022

  6. [6]

    A survey on data selection for language models.arXiv preprint arXiv:2402.16827, 2024

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models.arXiv preprint arXiv:2402.16827, 2024

    arXiv 2024

  7. [7]

    Efficient online data mixing for language model pre-training.arXiv preprint arXiv:2312.02406, 2023

    Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training.arXiv preprint arXiv:2312.02406, 2023

    arXiv 2023

  8. [8]

    Instruction mining: Instruction data selection for tuning large language models.arXiv preprint arXiv:2307.06290, 2023

    Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models.arXiv preprint arXiv:2307.06290, 2023

    arXiv 2023

Show all 69 references
  1. [9]

    Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701, 2023

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701, 2023

    2023 arXiv

  2. [10]

    Skill-it! a data-driven skills framework for understanding and training language models.Advances in Neural Information Processing Systems, 36:36000–36040, 2023

    Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christo- pher Ré. Skill-it! a data-driven skills framework for understanding and training language models.Advances in Neural Information Processing Systems, 36:36000–36040, 2023

    2023

  3. [11]

    Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    2017

  4. [12]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014

    2014

  5. [13]

    Selection via proxy: Efficient data selection for deep learning.arXiv preprint arXiv:1906.11829, 2019

    Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning.arXiv preprint arXiv:1906.11829, 2019

    1906 arXiv

  6. [14]

    Cold fusion: Collaborative descent for distributed multitask finetuning.arXiv preprint arXiv:2212.01378, 2022

    Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning.arXiv preprint arXiv:2212.01378, 2022. 10

    2022 arXiv

  7. [15]

    Sketchy moment matching: Toward fast and provable data selection for finetuning.arXiv preprint arXiv:2407.06120, 2024

    Yijun Dong, Hoang Phan, Xiang Pan, and Qi Lei. Sketchy moment matching: Toward fast and provable data selection for finetuning.arXiv preprint arXiv:2407.06120, 2024

    2024 arXiv

  8. [16]

    Dsdm: Model-aware dataset selection with datamodels.arXiv preprint arXiv:2401.12926, 2024

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels.arXiv preprint arXiv:2401.12926, 2024

    2024 arXiv

  9. [17]

    Doge: Domain reweighting with generalization estimation.arXiv preprint arXiv:2310.15393, 2023

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation.arXiv preprint arXiv:2310.15393, 2023

    2023 arXiv

  10. [18]

    Automatic document selection for efficient encoder pretraining.arXiv preprint arXiv:2210.10951, 2022

    Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc. Automatic document selection for efficient encoder pretraining.arXiv preprint arXiv:2210.10951, 2022

    2022 arXiv

  11. [19]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    2023 arXiv

  12. [20]

    Don’t stop pretraining: Adapt language models to domains and tasks.arXiv preprint arXiv:2004.10964, 2020

    Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks.arXiv preprint arXiv:2004.10964, 2020

    2004 arXiv

  13. [21]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019

    2019

  14. [22]

    Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  15. [23]

    Datamodels: Predicting predictions from training data.arXiv preprint arXiv:2202.00622, 2022

    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data.arXiv preprint arXiv:2202.00622, 2022

    2022 arXiv

  16. [24]

    On the complementarity of data selection and fine tuning for domain adaptation.arXiv preprint arXiv:2109.07591, 2021

    Dan Iter and David Grangier. On the complementarity of data selection and fine tuning for domain adaptation.arXiv preprint arXiv:2109.07591, 2021

    2021 arXiv

  17. [25]

    Data-efficient finetuning using cross-task nearest neighbors.arXiv preprint arXiv:2212.00196, 2022

    Hamish Ivison, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors.arXiv preprint arXiv:2212.00196, 2022

    2022 arXiv

  18. [26]

    Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2018

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  19. [27]

    The principles of data-centric ai (dcai).arXiv preprint arXiv:2211.14611, 2022

    Mohammad Hossein Jarrahi, Ali Memariani, and Shion Guha. The principles of data-centric ai (dcai).arXiv preprint arXiv:2211.14611, 2022

    2022 arXiv

  20. [28]

    Dataless knowledge fusion by merging weights of language models.arXiv preprint arXiv:2212.09849, 2022

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models.arXiv preprint arXiv:2212.09849, 2022

    2022 arXiv

  21. [29]

    3d object representations for fine- grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. InProceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013

    2013

  22. [30]

    mcot: Multilingual instruction tuning for reasoning consis- tency in language models.arXiv preprint arXiv:2406.02301, 2024

    Huiyuan Lai and Malvina Nissim. mcot: Multilingual instruction tuning for reasoning consis- tency in language models.arXiv preprint arXiv:2406.02301, 2024

    2024 arXiv

  23. [31]

    Training subset selection for weak supervision.Advances in Neural Information Processing Systems, 35:16023–16036, 2022

    Hunter Lang, Aravindan Vijayaraghavan, and David Sontag. Training subset selection for weak supervision.Advances in Neural Information Processing Systems, 35:16023–16036, 2022

    2022

  24. [32]

    The bigscience roots corpus: A 1.6 tb composite multilingual dataset

    Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro V on Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neura...

    2022

  25. [33]

    The mnist database of handwritten digits.http://yann

    Yann LeCun. The mnist database of handwritten digits.http://yann. lecun. com/exdb/mnist/, 1998. 11

    1998

  26. [34]

    When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers

    Hongkang Li, Yihua Zhang, Shuai Zhang, Pin-Yu Chen, Sijia Liu, and Meng Wang. When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [35]

    From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning.arXiv preprint arXiv:2308.12032, 2023

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning.arXiv preprint arXiv:2308.12032, 2023

    2023 arXiv

  28. [36]

    Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023

    Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023

    2023

  29. [37]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    1950

  30. [38]

    Regmix: Data mixture as regression for language model pre-training

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024

    2024 arXiv

  31. [39]

    Towards safer large language models through machine unlearning.arXiv preprint arXiv:2402.10058, 2024

    Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning.arXiv preprint arXiv:2402.10058, 2024

    2024 arXiv

  32. [40]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. InInternational Conference on Machine Learning, pages 22631–22648. PMLR, 2023

    2023

  33. [41]

    D2 pruning: Message passing for balancing diversity and difficulty in data pruning.arXiv preprint arXiv:2310.07931, 2023

    Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning.arXiv preprint arXiv:2310.07931, 2023

    2023 arXiv

  34. [42]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    2022

  35. [43]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. InNIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011

    2011

  36. [44]

    Confident learning: Estimating uncertainty in dataset labels.Journal of Artificial Intelligence Research, 70:1373–1411, 2021

    Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels.Journal of Artificial Intelligence Research, 70:1373–1411, 2021

    2021

  37. [45]

    Task arithmetic in the tangent space: Improved editing of pre-trained models.Advances in Neural Information Processing Systems, 36:66727–66754, 2023

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models.Advances in Neural Information Processing Systems, 36:66727–66754, 2023

    2023

  38. [46]

    Sentence encoders on stilts: Supplemen- tary training on intermediate labeled-data tasks.arXiv preprint arXiv:1811.01088, 2018

    Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplemen- tary training on intermediate labeled-data tasks.arXiv preprint arXiv:1811.01088, 2018

    2018 arXiv

  39. [47]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  40. [48]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  41. [49]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  42. [50]

    Diverse weight averaging for out-of-distribution generalization

    Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gal- linari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821–10836, 2022. 12

    2022

  43. [51]

    Slimpajama-dc: Under- standing data combinations for llm training.arXiv preprint arXiv:2309.10818, 2023

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. Slimpajama-dc: Under- standing data combinations for llm training.arXiv preprint arXiv:2309.10818, 2023

    2023 arXiv

  44. [52]

    Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemi´nski, Hakimeh ...

    2024

  45. [53]

    The german traffic sign recognition benchmark: a multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. InThe 2011 international joint conference on neural networks, pages 1453–1460. IEEE, 2011

    2011

  46. [54]

    Zipit! merging models from different tasks without training.arXiv preprint arXiv:2305.03053, 2023

    George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training.arXiv preprint arXiv:2305.03053, 2023

    2023 arXiv

  47. [55]

    Task arithmetic through the lens of one-shot federated learning.arXiv preprint arXiv:2411.18607, 2024

    Zhixu Tao, Ian Mason, Sanjeev Kulkarni, and Xavier Boix. Task arithmetic through the lens of one-shot federated learning.arXiv preprint arXiv:2411.18607, 2024

    2024 arXiv

  48. [56]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  49. [57]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. InProceedings of the IEEE/CVF conference on computer vision ...

    2022

  50. [58]

    Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    2024 arXiv

  51. [59]

    Moderate coreset: A universal method of data selection for real-world data-efficient deep learning

    Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  52. [60]

    Sun database: Exploring a large collection of scene categories.International Journal of Computer Vision, 119:3–22, 2016

    Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories.International Journal of Computer Vision, 119:3–22, 2016

    2016

  53. [61]

    Lm-cocktail: Resilient tuning of language models via model merging.arXiv preprint arXiv:2311.13534, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. Lm-cocktail: Resilient tuning of language models via model merging.arXiv preprint arXiv:2311.13534, 2023

    2023 arXiv

  54. [62]

    Doremi: Optimizing data mixtures speeds up language model pretraining.Advances in Neural Information Processing Systems, 36:69798–69818, 2023

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining.Advances in Neural Information Processing Systems, 36:69798–69818, 2023

    2023

  55. [63]

    Data selection for language models via importance resampling.Advances in Neural Information Processing Systems, 36:34201–34227, 2023

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling.Advances in Neural Information Processing Systems, 36:34201–34227, 2023

    2023

  56. [64]

    Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023. 13

    2023

  57. [65]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance.arXiv preprint arXiv:2403.16952, 2024

    Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance.arXiv preprint arXiv:2403.16952, 2024

    2024 arXiv

  58. [66]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InForty-first International Conference on Machine Learning, 2024

    2024

  59. [67]

    Data-centric artificial intelligence: A survey.ACM Computing Surveys, 57(5):1–42, 2025

    Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey.ACM Computing Surveys, 57(5):1–42, 2025

    2025

  60. [68]

    Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    2023

  61. [69]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023. 14 A Additional Experimental Setup Details In ...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.