Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that rotation symmetries of transformer attention layers are continuous, and that aligning models by rotation improves fusion.

desk verdict Correct symmetry identity with a clean closed-form matching algorithm, but the empirical case for fusion gains is overstated, especially in the ViT results. read the letter →

arxiv 2502.00264 v2 pith:AP5HL6I5 submitted 2025-02-01 cs.LG cs.CV

Binchi Zhang , Zaiyi Zheng , Zhengzhang Chen , Jundong Li This is my paper · ORCID
classification cs.LGcs.CV
keywords rotationsymmetryparameterspacemodelfusionself-attentionOrthogonalProcrustesweightmatchingtransformeralignmentmerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces rotation symmetry for transformer self-attention layers: for each attention head, applying an orthogonal rotation to the query and key matrices, and the same kind of rotation to the value and output matrices, leaves the layer's computed function unchanged for every input. This makes the equivalence class of an attention layer continuous, beyond the discrete permutation symmetries used for MLPs. The paper then turns this symmetry into a model-fusion tool: a closed-form, Kabsch-style algorithm rotates one model's attention parameters to lie as close as possible to another model's parameters before the two are averaged or merged. Across language and vision transformers, this alignment consistently improves fusion accuracy, reduces the distance between merged models, and lowers the loss barrier along the interpolation path.

What carries the argument

The carrying object is the rotation-symmetry equivalence class of a self-attention layer, together with the reduction of parameter matching to an Orthogonal Procrustes problem: the optimal rotation satisfies $R = UV^⊤$, where $U Σ V^⊤$ is the singular value decomposition of a cross-covariance matrix built from the query, key, and bias matrices, and separately from the value, output, and bias matrices. Algorithm 1 applies this per head, treating one model as the anchor and rotating the other, and a rescaling step, a scalar $a$ multiplying the query/key blocks and dividing the value/output blocks, is layered on afterward.

What would settle it

Run the proposed rotation on one of the two attention layers and compare its output to the original on a batch of inputs; any difference beyond numerical precision would refute the claimed invariance. Separately, construct or find a pair of models for which the distance-minimizing rotation gives a higher loss barrier than a deliberately worse rotation, which would break the presumed link between weight distance and fusion quality.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a self-attention layer is invariant under per-head orthogonal transformations: with independent rotation matrices $R_{qk}^h$ and $R_{vo}^h$, the replacements $W_Q^h → (R_{qk}^h)^⊤ W_Q^h$, $b_Q^h → b_Q^h R_{qk}^h$, $W_K^h → (R_{qk}^h)^⊤ W_K^h$, $b_K^h → b_K^h R_{qk}^h$, $W_V^h → (R_{vo}^h)^⊤ W_V^h$, $b_V^h → b_V^h R_{vo}^h$, $W_O^h → W_O^h R_{vo}^h$ preserve the attention output for any input. The paper further claims that, given two attention layers, the rotation that minimizes the Frobenius (sum-of-squared-entries) distance between them is obtained in closed form from a singular value decomposition, and that matching one layer to the other by this rotation reduces the distance between end models, improves the fused model, and flattens the loss barrier between them.

Load-bearing premise

The load-bearing premise is that minimizing the ordinary Euclidean distance between the matched attention weights is the right objective for fusion, so that a rotation making the weights numerically closer reliably makes the merged model better; the paper verifies this empirically but does not prove a monotone relationship.

Editorial extensions

If this is right

  • Any transformer attention layer has infinitely many functionally identical parameter settings parameterized by continuous rotations, not just finitely many permutations.
  • The proposed matching raises the accuracy of simple, Fisher, RegMean, and optimal-transport fusion methods across RoBERTa, DeBERTa, and ViT models in the reported experiments.
  • Matched models are closer in parameter space and exhibit a lower loss barrier along linear interpolation than unmatched or permutation-matched models.
  • Matching only a few early attention layers already yields most of the fusion gain, so the method can be applied cheaply.
  • The matching overhead is small relative to fine-tuning or merging, about 1.6 to 3.5 seconds per model pair in the reported settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not test: the same rotation alignment should improve federated averaging or any weight-space interpolation method, because the mechanism claimed is the reduced weight distance rather than anything specific to the fusion baselines used.
  • Because heads rotate independently, the equivalence class grows with head count and hidden dimension; a testable consequence is that rotation alignment matters more for larger, wider transformers, which would explain the paper's observation that the larger model benefits more from matching.
  • The sequential rotation-then-rescaling is admitted to be a practical approximation; a joint optimization over $(R, a)$ is the natural next experiment and would isolate whether the remaining fusion gap comes from that approximation.
  • The early-layers-matter result is demonstrated on two GLUE tasks; whether it transfers to other benchmarks or model families is open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a continuous rotational symmetry for transformer self-attention layers, showing that applying independent orthogonal transformations to each head's query/key and value/output weight matrices (Equation 5) preserves the layer function exactly. It then proposes a plug-and-play parameter matching algorithm for model fusion: feedforward blocks are matched by permutation (Equation 7), attention blocks by a Kabsch-style orthogonal Procrustes solution (Algorithm 1), optionally followed by a rescaling step (Equations 11-12). The authors evaluate the method by fusing fine-tuned RoBERTa and DeBERTa models on NLP benchmarks and pretrained ViTs on CIFAR-10, reporting accuracy and loss-barrier results. The central algebraic derivation is straightforward and correct, and the matching algorithm is computationally efficient, but the empirical support for the strong claim of 'substantially improves model fusion' is thin and partly inconsistent with the reported tables and ablations.

Significance. If the results are confirmed, the paper makes a useful contribution: it identifies an exact, continuous symmetry of attention layers that goes beyond discrete permutation symmetries, and it contributes a closed-form, efficient matching procedure that can be attached to existing fusion methods. The vector-space derivation in Section 3.2 is clean and the complexity analysis is order-correct, and the authors are careful to include code, an appendix proof, and a limitations section. The strength of the paper is theoretical and algorithmic; the claimed empirical gains, however, are currently not established at the level needed to support the headline conclusion, and several load-bearing claims need to be either strengthened or carefully scoped.

major comments (4)
  1. [§5.2, Tables 1-2, Figure 3] The empirical evidence for the headline 'substantially improves model fusion' is not robust. In the ViT experiments (Table 2), several baselines are at or near chance: Simple averaging gives 7.60% without matching and 10.19% with matching, where random guessing on CIFAR-10 is 10%; the best OT baselines improve by only 0.02-0.10 accuracy points after matching (e.g., OT-ACTS 61.15 to 61.25). No error bars, variance estimates, or multiple seeds are reported, so differences of this size may be run-to-run noise. The ablation in Figure 3 compounds this concern: for Fisher, attention-only matching (w/o FFN) achieves 20.21% while the complete method achieves 18.58%, meaning the full plug-and-play module is not consistently better than its attention-only component. The statement that 'Our parameter matching algorithm consistently improves the performance of different model fusion methods' is also contradicted by Table 1 entries with zero or negative change, e.g., DeBERTa Simple on Emotion stays at 2.99 and Fisher on NER-CoNLL for RoBERTa decreases from 23.06 to 23.05.
  2. [§4.2, Theorem 4.1, Algorithm 1] The closed-form solution R1 = U V^T is optimal over the orthogonal group, but not necessarily over the proper rotation group. If det(U V^T) = -1, the returned matrix is a reflection, not a rotation, and therefore lies outside the set R over which Equation (9) is minimized. Since Equation (5) and Algorithm 1 explicitly call R a rotation matrix, this distinction matters. The standard fix is the Umeyama correction (U diag(1,...,1,-1) V^T when the determinant is negative), which is cited in the paper but not implemented or discussed. If the authors intend R to mean the full orthogonal group, they should say so explicitly; otherwise Algorithm 1 should include the determinant-correction step to actually return a rotation.
  3. [§4.2, Equations (11)-(12); Appendix D] The 'theoretically optimal parameter matching' claim is broader than what is proven. The optimality result in Theorem 4.1 is for the binary, rotation-only, Frobenius-norm objective; the rescaling step is applied sequentially after rotation, and the authors themselves note in Appendix D that the sequential approach 'does not guarantee global optimality for the joint optimization over (R, alpha)'. Likewise, the multi-model extension is acknowledged to lack the global optimality of the pairwise case. The paper should state these scoping conditions prominently in the main text and in the abstract-adjacent claims, so that 'optimal' is understood as optimal for the specific weight-space objective under the stated constraints, not as a guarantee about fusion utility.
  4. [§4.2, Equation (8)] The matching objective minimizes Frobenius distance in weight space, which is only a proxy for the fusion objective. The paper motivates this by citing the convexity/closeness heuristic of Wortsman et al., and the loss-barrier experiment in Section 5.4 provides some supporting evidence, but no experiment establishes a monotone relationship between the specific rotation-aligned weight distance and the final fusion metric. Since the choice of the exact objective is what makes the closed-form solution possible, this limitation should be discussed explicitly; as written, the text moves from 'minimize parameter distance' to 'improves fusion' without acknowledging that the two are not guaranteed to be monotonically related across heterogeneous layers, heads, and merging methods.
minor comments (5)
  1. [§4.2, Theorem 4.1 and Algorithm 1] The theorem statement and Algorithm 1 say the solution is obtained by 'eigendecomposition', but the proof uses a singular value decomposition; the wording should be corrected to SVD everywhere.
  2. [Appendix A] Equation (13) in the appendix duplicates the theorem statement as Equation (9) in the main text, and Equation (14) duplicates Equation (10); the numbering should be adjusted to avoid confusion.
  3. [Figure 3] The label 'OT-ACTS-EMB' in Figure 3 appears to be a typo for 'OT-ACTS-EMD' used in Table 2.
  4. [§5.6] The phrase 'matching head layers yields greater improvements' is ambiguous: Figure 6(a) shows that early layers (low layer indices) matter most, but 'head layers' could be read as attention heads. Please use 'early layers' or 'initial layers' for clarity.
  5. [§5.2] The paper does not report the number of random seeds or any variance measure for any of the fusion results in Tables 1-2; such information is needed to assess the significance of the small differences reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: rotation symmetry follows from the algebraic identity RR^T = I, and the matching algorithm solves an explicit Procrustes objective with fusion gains tested against external baselines.

full rationale

Rotation symmetry (Eq. 5) is derived in Section 3.2 from RR^T = I applied to the query-key and value-output products; it is an algebraic identity, not a fitted assumption. The matching objective (Eq. 8) is stated explicitly, and Theorem 4.1 (proved in Appendix A) gives the closed-form Orthogonal Procrustes solution, a standard external result (Kabsch, 1976; Schonemann, 1966). The paper's fusion claims are evaluated on held-out tasks against external baselines (Tables 1-2), so the outcome is not encoded in the optimization objective. The Figure 4 distance reduction is a direct consequence of minimizing Eq. 8, but the paper presents it as a sanity check and does not use it as the central predictive result; the load-bearing fusion evidence is independent. No load-bearing argument reduces to a self-citation chain: the cited prior results (Wortsman et al., 2022; Ainsworth et al., 2023; Jin et al., 2023) are external and falsifiable, and self-references in the bibliography are incidental. No enumerated circularity pattern applies.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or architectural entities are postulated. The rotation symmetry is a known algebraic identity applied to a new setting, so the ledger is dominated by domain assumptions about weight-space distance and LayerNorm behavior rather than free parameters or invented constructs. The main free parameter is the choice of which layers to match, which is selected by validation performance.

free parameters (1)
  • per-layer selection of matched layers = not fixed (single-layer matching, tail-layers matching)
    The paper matches individual layers independently and selects the merged model with the highest validation performance (Section 5.2, ViT Settings; Section 5.6). The choice of which layers to match (head vs. tail) is data-driven.
assumptions (3)
  • domain assumption The losses and merging methods used (Simple, Fisher, RegMean) benefit from reducing Euclidean distance between weight vectors.
    This is cited from Wortsman et al. (2022) and used in Section 4.1 as the motivation for parameter matching. It is not proven in this paper.
  • domain assumption The weight space distance used in the Procrustes objective is an appropriate proxy for fusion utility.
    This is the core modeling assumption behind Algorithm 1: minimizing Frobenius distance on attention weights is claimed to help fusion, and it is only verified empirically, not derived.
  • domain assumption Rotation matrices can be applied independently per head and per layer without interacting with LayerNorm or residual connections in a way that breaks functional equivalence.
    The paper's derivation in Section 3.2 places LayerNorm and residuals around the attention block, and the symmetry derivation ignores the effect of LayerNorm on the rotated representation. If LayerNorm statistics are input-dependent, rotating the pre-norm features may not preserve exact functional equivalence unless the effect is absorbed, which is not shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion." pith.science (2026). https://pith.science/paper/AP5HL6I5

@misc{pith2026250200264,
  author       = {Pith},
  title        = {Pith review of: Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AP5HL6I5}},
  note         = {Machine review of arXiv:2502.00264}
}
read the original abstract

Symmetry in the parameter space of deep neural networks (DNNs) has proven beneficial for various deep learning applications. A well-known example is the permutation symmetry in Multi-Layer Perceptrons (MLPs), where permuting the rows of weight matrices in one layer and applying the inverse permutation to adjacent layers yields a functionally equivalent model. While permutation symmetry fully characterizes the equivalence set for MLPs, its discrete nature limits its utility for transformers. In this paper, we introduce rotation symmetry, a novel form of parameter space symmetry for transformers that generalizes permutation symmetry by rotating parameter matrices in self-attention layers. Unlike permutation symmetry, rotation symmetry operates in a continuous domain, thereby significantly expanding the equivalence set for transformers. Based on this property, we propose a theoretically optimal parameter matching algorithm as a plug-and-play module to enhance model fusion. We evaluate our approach using pre-trained transformers across diverse natural language and vision tasks. Experimental results demonstrate that our rotation symmetry-based matching algorithm substantially improves model fusion, highlighting the potential of parameter space symmetry to facilitate model fusion. Our code is available on https://github.com/zhengzaiyi/RotationSymmetry.

Figures

Figures reproduced from arXiv: 2502.00264 by the authors.

Figure 1
Figure 1. The rotation symmetry of self-attention layers. & Raffel, 2022; Wortsman et al., 2022; Yadav et al., 2023; Jin et al., 2023; Daheim et al., 2024; Yang et al., 2024). Model fusion is proposed to merge multiple given end mod￾els trained in different settings (e.g., upon different datasets and hyperparameter settings) in the parameter space to im￾prove model utility and robustness. Compared with en￾semble learning (Die… view at source ↗
Figure 3
Figure 3. Ablation Study of ViT merging over the image classification task. “ATTN” is short for “attention”. We compare our matching algorithm with its three variants (w/o ATTN/FFN/rescaling) and the original performance (w/o match) on all six merging baselines. 600 610 620 630 640 650 660 670 Distance w/o match Git Re-Basin OT-Fusion w/o ATTN, rescaling w/o ATTN w/o rescaling match (Ours) 651.57 625.73 626.16 640.01 639.96 6… view at source ↗
Figure 4
Figure 4. The Euclidean Distance of end ViT models after different parameter matching algorithms. fine-tuned models) and out-of-domain (grouped models) experiments. Notably, we match only the parameters within the attention layers, while the classifier module is directly copied from the model fine-tuned on the corresponding downstream task. For additional details on datasets and baseline methods, please refer to Appendix C. 5… view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: Loss landscapes and barriers between the two pretrained ViT models under four distinct matching settings. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Toward Manifest Relationality in Transformers via Symmetry Reduction

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Transformer attention and parameter optimization can be rewritten on symmetry-reduced relational variables (Gram matrices and invariant parameter composites), removing coordinate redundancies by construction.

  2. Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

    cs.LG 2025-09 conditional novelty 5.0 of 10

    MergeLock applies random invertible matrix transformations to Transformer attention weights, preserving the model's own output while forcing any merged model's accuracy down to near random.

Reference graph

Works this paper leans on

101 extracted references · 52 canonical work pages · cited by 2 Pith papers

  1. [1]

    Git re-basin: Merging models modulo permutation symmetries

    Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023

    2023

  2. [2]

    and Jodoin, P.-M

    Armenta, M. and Jodoin, P.-M. The representation theory of neural networks. Mathematics, 9 0 (24): 0 3216, 2021

    2021

  3. [3]

    Ensemble of averages: Improving model selection and boosting performance in domain generalization

    Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35: 0 8265--8277, 2022

    2022

  4. [4]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  5. [5]

    Symmetry-invariant optimization in deep networks

    Badrinarayanan, V., Mishra, B., and Cipolla, R. Symmetry-invariant optimization in deep networks. arXiv preprint arXiv:1511.01754, 2015

    work page Pith review arXiv 2015

  6. [6]

    Benton, G., Maddox, W., Lotfi, S., and Wilson, A. G. G. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pp.\ 769--779, 2021

    2021

  7. [7]

    and Klinger, R

    Bostan, L.-A.-M. and Klinger, R. An analysis of annotated corpora for emotion classification in text. In Bender, E. M., Derczynski, L., and Isabelle, P. (eds.), Proceedings of the 27th International Conference on Computational Linguistics, pp.\ 2104--2119, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https://aclan...

    2018

  8. [8]

    Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape

    Brea, J., Simsek, B., Illing, B., and Gerstner, W. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiv:1907.02911, 2019

    arXiv 1907

Show all 101 references
  1. [9]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  2. [10]

    Burkard, R. E. and Cela, E. Linear assignment problems and extensions. In Handbook of combinatorial optimization: Supplement volume A, pp.\ 75--149. 1999

    1999

  3. [11]

    Swad: Domain generalization by seeking flat minima

    Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 0 22405--22418, 2021

    2021

  4. [12]

    Fusing finetuned models for better pretraining

    Choshen, L., Venezian, E., Slonim, N., and Katz, Y. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044, 2022

    2022 arXiv

  5. [13]

    V., and Manning, C

    Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020

    2020

  6. [14]

    Daheim, N., M \"o llenhoff, T., Ponti, E., Gurevych, I., and Khan, M. E. Model merging by uncertainty-based gradient matching. In International Conference on Learning Representations, 2024

    2024

  7. [15]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2019

  8. [16]

    Dietterich, T. G. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp.\ 1--15, 2000

    2000

  9. [17]

    A survey on ensemble learning

    Dong, X., Yu, Z., Cao, W., Shi, Y., and Ma, Q. A survey on ensemble learning. Frontiers of Computer Science, 14: 0 241--258, 2020

    2020

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/ab...

    2021 arXiv

  11. [19]

    S., Hu, W., and Lee, J

    Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018

    2018

  12. [20]

    The role of permutation invariance in linear mode connectivity of neural networks

    Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022

    2022

  13. [21]

    Proving linear mode connectivity of neural networks via optimal transport

    Ferbach, D., Goujaud, B., Gidel, G., and Dieuleveut, A. Proving linear mode connectivity of neural networks via optimal transport. In International Conference on Artificial Intelligence and Statistics, pp.\ 3853--3861, 2024

    2024

  14. [22]

    and Amari, S.-i

    Fukumizu, K. and Amari, S.-i. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural networks, 13 0 (3): 0 317--327, 2000

    2000

  15. [23]

    Deep sparse rectifier neural networks

    Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp.\ 315--323, 2011

    2011

  16. [24]

    On the symmetries of deep learning models and their internal representations

    Godfrey, C., Brown, D., Emerson, T., and Kvinge, H. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35: 0 11893--11905, 2022

    2022

  17. [25]

    Gower, J. C. and Dijksterhuis, G. B. Procrustes problems, volume 30. OUP Oxford, 2004

    2004

  18. [26]

    Hidden symmetries of relu networks

    Grigsby, E., Lindsey, K., and Rolnick, D. Hidden symmetries of relu networks. In International Conference on Machine Learning, pp.\ 11734--11760, 2023

    2023

  19. [27]

    A., and DeCoste, D

    Gupta, V., Serrano, S. A., and DeCoste, D. Stochastic weight averaging in parallel: Large-batch training that generalizes well. In International Conference on Learning Representations, 2020

    2020

  20. [28]

    Model merging and safety alignment: One bad model spoils the bunch

    Hammoud, H., Michieli, U., Pizzati, F., Torr, P., Bibi, A., Ghanem, B., and Ozay, M. Model merging and safety alignment: One bad model spoils the bunch. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 13033--13046, 2024

    2024

  21. [29]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  22. [30]

    Deberta: Decoding-enhanced bert with disentangled attention, 2021

    He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding-enhanced bert with disentangled attention, 2021

    2021

  23. [31]

    Explaining graph neural networks with large language models: A counterfactual perspective on molecule graphs

    He, Y., Zheng, Z., Soga, P., Zhu, Y., Dong, Y., and Li, J. Explaining graph neural networks with large language models: A counterfactual perspective on molecule graphs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 7079--7096, 2024

    2024

  24. [32]

    On the algebraic structure of feedforward network weight spaces

    Hecht-Nielsen, R. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp.\ 129--135. 1990

    1990

  25. [33]

    O nto N otes: The 90 \ In Moore, R

    Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., and Weischedel, R. O nto N otes: The 90 \ In Moore, R. C., Bilmes, J., Chu-Carroll, J., and Sanderson, M. (eds.), Proceedings of the Human Language Technology Conference of the NAACL , Companion Volume: Short Papers , pp.\ 57--60,...

    2006

  26. [34]

    Emr-merging: Tuning-free high-performance model merging

    Huang, C., Ye, P., Chen, T., He, T., Yue, X., and Ouyang, W. Emr-merging: Tuning-free high-performance model merging. Advances in Neural Information Processing Systems, 37: 0 122741--122769, 2024

    2024

  27. [35]

    Imfeld, M., Graldi, J., Giordano, M., Hofmann, T., Anagnostidis, S., and Singh, S. P. Transformer fusion with optimal transport. In International Conference on Learning Representations, 2024

    2024

  28. [36]

    Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence (UAI), pp.\ 876--885, 2018

    2018

  29. [37]

    Dataless knowledge fusion by merging weights of language models

    Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations, 2023

    2023

  30. [38]

    A solution for the best rotation to relate two sets of vectors

    Kabsch, W. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32 0 (5): 0 922--923, 1976

    1976

  31. [39]

    Scale equivariant graph metanetworks

    Kalogeropoulos, I., Bouritsas, G., and Panagakis, Y. Scale equivariant graph metanetworks. Advances in neural information processing systems, 37: 0 106800--106840, 2024

    2024

  32. [40]

    B., Yu, F

    Kone c n \`y , J., McMahan, H. B., Yu, F. X., Richt \'a rik, P., Suresh, A. T., and Bacon, D. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016

    2016 arXiv

  33. [41]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  34. [42]

    Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 0 (1-2): 0 83--97, 1955

    1955

  35. [43]

    L., and Tanaka, H

    Kunin, D., Sagastuy-Brena, J., Ganguli, S., Yamins, D. L., and Tanaka, H. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021

    2021

  36. [44]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017

    2017

  37. [45]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Associ...

    2020

  38. [46]

    A., and Zettlemoyer, L

    Li, M., Gururangan, S., Dettmers, T., Lewis, M., Althoff, T., Smith, N. A., and Zettlemoyer, L. Branch-train-merge: Embarrassingly parallel training of expert language models. In First Workshop on Interpolation Regularizers and Beyond at NeurIPS 2022, 2022

    2022

  39. [47]

    Deep model fusion: A survey

    Li, W., Peng, Y., Zhang, M., Ding, L., Hu, H., and Shen, L. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023

    2023 arXiv

  40. [48]

    T., Lorraine, J., and Lucas, J

    Lim, D., Maron, H., Law, M. T., Lorraine, J., and Lucas, J. Graph metanetworks for processing diverse neural architectures. In International Conference on Learning Representations, 2024 a

    2024

  41. [49]

    The empirical impact of neural parameter symmetries, or lack thereof

    Lim, D., Putterman, T., Walters, R., Maron, H., and Jegelka, S. The empirical impact of neural parameter symmetries, or lack thereof. Advances in Neural Information Processing Systems, 37: 0 28322--28358, 2024 b

    2024

  42. [50]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  43. [51]

    Symmetry induces structure and constraint of learning

    Liu, Z. Symmetry induces structure and constraint of learning. In International Conference on Machine Learning, 2024

    2024

  44. [52]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 10012--10022, 2021

    2021

  45. [53]

    S., Bigelow, E

    Lubana, E. S., Bigelow, E. J., Dick, R. P., Krueger, D., and Tanaka, H. Mechanistic mode connectivity. In International Conference on Machine Learning, pp.\ 22965--23004, 2023

    2023

  46. [54]

    and Toth, P

    Martello, S. and Toth, P. Linear assignment problems. In North-Holland Mathematics Studies, volume 132, pp.\ 259--282. 1987

    1987

  47. [55]

    Matena, M. S. and Raffel, C. A. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35: 0 17703--17716, 2022

    2022

  48. [56]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.\ 1273--1282, 2017

    2017

  49. [57]

    G-sgd: Optimizing relu neural networks in its positively scale-invariant space

    Meng, Q., Zheng, S., Zhang, H., Chen, W., Ye, Q., Ma, Z.-M., Yu, N., and Liu, T.-Y. G-sgd: Optimizing relu neural networks in its positively scale-invariant space. In International Conference on Learning Representations, 2019

    2019

  50. [58]

    and Hinton, G

    Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, pp.\ 807--814, 2010

    2010

  51. [59]

    Equivariant architectures for learning in deep weight spaces

    Navon, A., Shamsian, A., Achituve, I., Fetaya, E., Chechik, G., and Maron, H. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pp.\ 25790--25816, 2023

    2023

  52. [60]

    Equivariant deep weight space alignment

    Navon, A., Shamsian, A., Fetaya, E., Chechik, G., Dym, N., and Maron, H. Equivariant deep weight space alignment. In International Conference on Machine Learning, pp.\ 37376--37395, 2024

    2024

  53. [61]

    R., and Srebro, N

    Neyshabur, B., Salakhutdinov, R. R., and Srebro, N. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015

    2015

  54. [62]

    and Kim, S

    Park, N. and Kim, S. How do vision transformers work?, 2022. URL https://arxiv.org/abs/2202.06709

    2022 arXiv

  55. [63]

    Pe \ n a, F. A. G., Medeiros, H. R., Dubail, T., Aminbeidokhti, M., Granger, E., and Pedersoli, M. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 20237--20246, 2023

    2023

  56. [64]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763, 2021

    2021

  57. [65]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  58. [66]

    Diverse weight averaging for out-of-distribution generalization

    Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35: 0 10821--10836, 2022

    2022

  59. [67]

    On permutation symmetries in bayesian neural network posteriors: a variational perspective

    Rossi, S., Singh, A., and Hannagan, T. On permutation symmetries in bayesian neural network posteriors: a variational perspective. Advances in Neural Information Processing Systems, 36, 2023

    2023

  60. [68]

    and Rokach, L

    Sagi, O. and Rokach, L. Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8 0 (4): 0 e1249, 2018

    2018

  61. [69]

    Sch \"o nemann, P. H. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31 0 (1): 0 1--10, 1966

    1966

  62. [70]

    Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances

    Simsek, B., Ged, F., Jacot, A., Spadaro, F., Hongler, C., Gerstner, W., and Brea, J. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp.\ 9722--9732, 2021

    2021

  63. [71]

    Singh, S. P. and Jaggi, M. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33: 0 22045--22055, 2020

    2020

  64. [72]

    Linear algebra and its applications

    Strang, G. Linear algebra and its applications. Academic Press, 1976

    1976

  65. [73]

    Optimizing mode connectivity via neuron alignment

    Tatro, N., Chen, P.-Y., Das, P., Melnyk, I., Sattigeri, P., and Lai, R. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33: 0 15300--15311, 2020

    2020

  66. [74]

    Tjong Kim Sang, E. F. and De Meulder, F. Introduction to the C o NLL -2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT - NAACL 2003 , pp.\ 142--147, 2003. URL https://aclanthology.org/W03-0419

    2003

  67. [75]

    N., The, A

    Tran, V.-H., Vo, T. N., The, A. N., Huu, T. T., Nguyen-Nhat, M.-K., Tran, T., Pham, D.-T., and Nguyen, T. M. Equivariant neural functional networks for transformers. arXiv preprint arXiv:2410.04209, 2024

    2024 arXiv

  68. [76]

    Least-squares estimation of transformation parameters between two point patterns

    Umeyama, S. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13 0 (04): 0 376--380, 1991

    1991

  69. [77]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  70. [78]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019

    2019

  71. [79]

    Federated learning with matched averaging

    Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. In International Conference on Learning Representations, 2020

    2020

  72. [80]

    and Liu, L

    Wei, X. and Liu, L. Are large language models good in-context learners for financial sentiment analysis? arXiv preprint arXiv:2503.04873, 2025

    2025 arXiv

  73. [81]

    Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A

    Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Internation...

    2022

  74. [82]

    A., and Bansal, M

    Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2023

    2023

  75. [83]

    What matters for model merging at scale? arXiv preprint arXiv:2410.03617, 2024

    Yadav, P., Vu, T., Lai, J., Chronopoulou, A., Faruqui, M., Bansal, M., and Munkhdalai, T. What matters for model merging at scale? arXiv preprint arXiv:2410.03617, 2024

    2024 arXiv

  76. [84]

    Adamerging: Adaptive model merging for multi-task learning

    Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024

    2024

  77. [85]

    Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. J. Graph transformer networks. Advances in neural information processing systems, 32, 2019

    2019

  78. [86]

    Bayesian nonparametric federated learning of neural networks

    Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pp.\ 7252--7261, 2019

    2019

  79. [87]

    Improving learning to optimize using parameter symmetries

    Zamir, G., Dokania, A., Zhao, B., and Yu, R. Improving learning to optimize using parameter symmetries. arXiv preprint arXiv:2504.15399, 2025

    2025 arXiv

  80. [88]

    Resolving editing-unlearning conflicts: A knowledge codebook framework for large language model updating

    Zhang, B., Chen, Z., Zheng, Z., Li, J., and Chen, H. Resolving editing-unlearning conflicts: A knowledge codebook framework for large language model updating. arXiv preprint arXiv:2502.00158, 2025

    2025 arXiv

  81. [89]

    Symmetry teleportation for accelerated optimization

    Zhao, B., Dehmamy, N., Walters, R., and Yu, R. Symmetry teleportation for accelerated optimization. Advances in neural information processing systems, 35: 0 16679--16690, 2022

    2022

  82. [90]

    Symmetries, flat minima, and the conserved quantities of gradient flow

    Zhao, B., Ganev, I., Walters, R., Yu, R., and Dehmamy, N. Symmetries, flat minima, and the conserved quantities of gradient flow. In The Eleventh International Conference on Learning Representations, 2023

    2023

  83. [91]

    M., Walters, R., and Yu, R

    Zhao, B., Gower, R. M., Walters, R., and Yu, R. Improving convergence and generalization using parameter symmetries. In International Conference on Learning Representations, 2024

    2024

  84. [92]

    Kg-cf: Knowledge graph completion with context filtering under the guidance of large language models

    Zheng, Z., Dong, Y., Wang, S., Liu, H., Wang, Q., and Li, J. Kg-cf: Knowledge graph completion with context filtering under the guidance of large language models. In 2024 IEEE International Conference on Big Data (BigData), pp.\ 805--810, 2024

    2024

  85. [93]

    Z., and Finn, C

    Zhou, A., Yang, K., Burns, K., Cardace, A., Jiang, Y., Sokota, S., Kolter, J. Z., and Finn, C. Permutation equivariant neural functionals. Advances in neural information processing systems, 36: 0 24966--24992, 2023 a

    2023

  86. [94]

    Z., and Finn, C

    Zhou, A., Yang, K., Jiang, Y., Burns, K., Xu, W., Sokota, S., Kolter, J. Z., and Finn, C. Neural functional transformers. Advances in neural information processing systems, 36: 0 77485--77502, 2023 b

    2023

  87. [95]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.\ 11106--11115, 2021

    2021

  88. [96]

    Metagpt: Merging large language models using model exclusive task arithmetic

    Zhou, Y., Song, L., Wang, B., and Chen, W. Metagpt: Merging large language models using model exclusive task arithmetic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 1711--1724, 2024

    2024

  89. [97]

    Going beyond linear mode connectivity: The layerwise linear feature connectivity

    Zhou, Z., Yang, Y., Yang, X., Yan, J., and Hu, W. Going beyond linear mode connectivity: The layerwise linear feature connectivity. Advances in Neural Information Processing Systems, 36, 2023 c

    2023

  90. [98]

    Understanding and modeling job marketplace with pretrained language models

    Zhu, Y., Wu, L., Zhang, B., Wang, S., Guo, Q., Hong, L., Simon, L., and Li, J. Understanding and modeling job marketplace with pretrained language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp.\ 5143--5150, 2024

    2024

  91. [99]

    Parameter symmetry and noise equilibrium of stochastic gradient descent

    Ziyin, L., Wang, M., Li, H., and Wu, L. Parameter symmetry and noise equilibrium of stochastic gradient descent. arXiv preprint arXiv:2402.07193, 2024

    2024 arXiv

  92. [100]

    Ziyin, L., Xu, Y., and Chuang, I. L. Remove symmetries to control model expressivity and improve optimization. In International Conference on Learning Representations, 2025

    2025

  93. [101]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.