Pith. sign in

REVIEW 4 major objections 4 minor 67 references

DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that multi-task fine-tuning of pre-trained vision transformers should preserve the weight matrices' singular vectors and reshape only their singular values through learned diffeomorphisms, outperforming low-rank…

desk verdict A genuinely new and efficient way to adapt ViTs via diffeomorphic singular-value modulation, but the SOTA claim rests on a single run and needs a multi-seed paired rerun before I'd trust it. read the letter →

arxiv 2502.06029 v3 pith:HMIBALIA submitted 2025-02-09 cs.CV

Krishna Sri Ipsit Mantri , Carola-Bibiane Schönlieb , Bruno Ribeiro , Chaim Baskin , Moshe Eliasof This is my paper · ORCID
classification cs.CV
keywords multi-tasklearningparameter-efficientfine-tuningsingularvaluedecompositiondiffeomorphictransformationsCPABvisiontransformersdensepredictionlow-rankadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-task fine-tuning of a pre-trained vision transformer usually forces all tasks to share a low-rank subspace, and the paper argues this is why tasks interfere. The proposed alternative, DiTASK, freezes each weight matrix and learns a small continuous, invertible, order-preserving map that reshapes its singular values, keeping the pre-trained singular vectors intact. A shared map catches task synergies and per-task maps capture task-specific needs, so the encoder stays full-rank while using only a few dozen parameters per layer. On PASCAL-MTL and NYUD, the paper reports that this beats single-task fine-tuning and existing multi-task PEFT methods, including MTLoRA, with about 75% fewer trainable parameters. If true, this means parameter-efficient multi-task adaptation does not have to sacrifice performance to save parameters.

What carries the argument

The load-bearing mechanism is the CPAB (continuous piecewise-affine based) diffeomorphism: a one-dimensional, invertible, order-preserving map $f_{\theta}$ defined by integrating a continuous piecewise-affine velocity field over a closed interval $[a,b]$, parameterized by only $N_P$ values per layer. Applied to the singular values of a frozen weight matrix through $W_A = U\operatorname{diag}(f_{\theta}(\sigma_i))V^{\top}$, it produces a full-rank adaptation that keeps the pre-trained singular vectors and the ordering of singular values while costing $O(N_P)$ parameters instead of $O(r(c_1+c_2))$. A joint transformation $f_{\theta_j}$ captures shared structure across tasks, and per-task transformations $f_{\theta_k}$ capture task-specific adjustments, so the same frozen matrix serves all tasks without a shared low-rank bottleneck.

What would settle it

Inspect the implementation of Eq. (6) and the singular values of the frozen encoder weights during training: if any $\sigma_i$ lies outside the CPAB interval $[a,b]$, the transformation $f_{\theta}$ is undefined on it, which would show that the reported results rely on an unstated clipping, rescaling, or domain-extension rule.

Watch

Extended reading notes

Core claim

The central discovery is that the interference seen in LoRA-style multi-task adaptation can be avoided by preserving the singular vectors of pre-trained weight matrices and modulating only their singular values with diffeomorphic maps. For a frozen weight matrix $W = U\Sigma V^{\top}$, DiTASK produces $W_A = U\operatorname{diag}(f_{\theta}(\sigma_1),\dots,f_{\theta}(\sigma_p))V^{\top}$, using one shared transformation $f_{\theta_j}$ for task synergy and separate transformations $f_{\theta_k}$ per task. Because each $f_{\theta}$ is a CPAB diffeomorphism, the update is full-rank and preserves the relative ordering of singular values, which the paper identifies with the pre-trained feature hierarchy. The reported numbers on PASCAL-MTL are a +3.22% relative improvement over single-task fine-tuning, compared with +2.55% for MTLoRA at rank 64, using 1.61M trainable encoder parameters versus 6.40M, and the advantage grows when the backbone is scaled up.

Load-bearing premise

The whole method assumes that every singular value of every adapted weight matrix can be fed into the CPAB transformation, but the transformation is only defined on a fixed finite interval and the paper does not state how singular values are brought into that interval or what happens to any that fall outside it.

Editorial extensions

If this is right

  • If the main result holds, multi-task fine-tuning no longer forces a trade-off between parameter efficiency and task performance: a few dozen parameters per layer can improve over single-task baselines.
  • The results imply that the failure of fixed-rank PEFT methods in multi-task settings is due to shared low-rank subspaces, not to insufficient capacity, so those methods should be re-examined with this mechanism in mind.
  • The scaling experiments imply that larger pre-trained backbones offer richer singular-value spectra, making the approach more beneficial as model size grows.
  • The VTAB results reported in the appendix suggest the same singular-value-preserving recipe transfers to single-task adaptation with roughly an order of magnitude fewer parameters than LoRA.
  • The shared-plus-task-specific design permits a single forward pass for all tasks, which the paper notes is not available with adapter- or prompt-based baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper never specifies how singular values, which are not bounded a priori, are mapped into the CPAB interval $[a,b]$; this hidden normalization choice could affect both expressivity and stability, so the comparison may partly depend on it.
  • Inference: if preserving singular vectors is the active ingredient, the same recipe could be tested in continual learning, where forgetting is often attributed to overwriting pre-trained directions rather than to re-scaling them.
  • Inference: because CPAB diffeomorphisms are monotone, DiTASK cannot reorder the relative importance of features; testing non-monotone maps would separate the benefit of preserving order from the benefit of preserving the vectors themselves.
  • Inference: the reported gap over MTLoRA widens with backbone capacity, suggesting that the method's advantage may be even larger in larger-scale multi-task systems with task-specific decoders and loss-balancing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces DiTASK, a parameter-efficient multi-task fine-tuning method that keeps the singular vectors of pre-trained weight matrices frozen and adapts only the singular values using learnable CPAB diffeomorphisms. A joint transformation module and per-task transformation modules are applied to the SVD of each attention projection in a Swin Transformer, and the adapted weights are used with a shared HRNet decoder. The authors claim that this preserves the geometric structure of pre-trained features, yields full-rank updates, and outperforms MTLoRA and other PEFT baselines on PASCAL-MTL and NYUD while using far fewer trainable parameters. The paper includes ablation studies on joint versus task-wise modulation, tessellation size, backbone scale, and pre-training data scale, and the code is publicly released.

Significance. The underlying idea is timely and potentially useful: modulating singular values while freezing singular vectors is a clean way to adapt a shared backbone to multiple tasks without forcing tasks into a shared low-rank subspace, and the reported parameter counts are genuinely small. The manuscript also contains a broad comparison against many PEFT families and releases code, both of which are positive for reproducibility. However, the central SOTA claim rests on a small single-run margin over MTLoRA, and the claimed theoretical analysis is not actually a proof. If the empirical headline survives multi-seed verification and the domain-mapping issue is resolved, the method would be a solid contribution to parameter-efficient multi-task dense prediction.

major comments (4)
  1. [Section 4.3 and abstract] The abstract and contribution list state that the paper 'theoretically analyzes' DITASK and shows that it 'achieves full-rank updates during optimization,' but Section 4.3 contains no theorem, lemma, or proof. The full-rank property and the preservation of singular-value ordering are immediate consequences of the parametrization in Eq. (6) together with the monotonicity of diffeomorphisms, so they are definitional rather than derived results. This overstates the theoretical contribution and should be rewritten as a statement of properties of the construction, with explicit proofs or references if any nontrivial claim is intended.
  2. [Definition 3.1 and Eq. (6)] The manuscript never specifies how the singular values of an arbitrary weight matrix are mapped into the domain [a,b] on which the CPAB transformation f_theta is defined. Eq. (6) applies f_theta to every sigma_i, but if some singular values fall outside [a,b], the transformation is undefined as written. The paper needs to state the normalization or domain-extension rule used in the implementation, and discuss how that choice affects expressivity and numerical stability. This is not a cosmetic gap: without it, the method is not fully specified and the reported results cannot be reproduced from the text alone.
  3. [Section 5.2 and Table 1] The central SOTA claim rests on a single run: Table 1 reports no seeds, no error bars, and no variance for any of the four task metrics. The composite margin over MTLoRA(r=64) is only 0.67 percentage points in Delta_m, and MTLoRA is numerically better on SALIENCY (65.40 vs 65.00) and NORMALS (16.60 vs 17.10). The manuscript also does not state whether MTLoRA and the other baselines were rerun under the same environment or taken from prior papers. Because run-to-run variation of this magnitude is common in dense-prediction fine-tuning, the headline '26.27% improvement' and the 'state-of-the-art' claim require a multi-seed paired comparison before they can be accepted.
  4. [Table 1 vs Table 3] There is an unresolved inconsistency in the parameter accounting: Table 1 reports 1.61M 'trainable Swin parameters' for MTL-DITASK, while Table 3 reports only 8.5K 'adaptation parameters' for DITASK. The text also uses '75% fewer parameters' and '4x fewer parameters' without clarifying which count is being compared. Since the parameter-efficiency claim is one of the two main contributions, the paper must define the counting convention (backbone-only, decoder-excluded, LayerNorm included or not) and report the same convention for all rows and tables.
minor comments (4)
  1. [Throughout] The method name is spelled inconsistently as both DiTASK and DITASK; please standardize.
  2. [Section 2] Internal cross-references such as 'in the Section 1' appear to point to figures rather than sections; these should be corrected or removed.
  3. [Eq. (7)] The definition of Delta_m is clear, but the text should state explicitly which single-task baseline (per-backbone or per-dataset) is used, since Table 2 and Table 4 change the backbone and dataset.
  4. [Supplementary Table 5] The claim that larger pre-training scale helps is not supported by the SEMSEG column, where ImageNet-1k (70.09) outperforms ImageNet-21k (69.06); this deserves a brief comment or correction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DITASK's core properties follow from its definition, and its empirical claims are tested against external benchmarks.

full rationale

The derivation chain is self-contained. The adapted weight W_A in Eq. (6) is defined as U diag(f_theta(sigma_i)) V^T, so the claims that singular vectors are preserved and that the update is full-rank are analytical consequences of the parametrization, not fitted quantities renamed as predictions. The ordering-preservation claim in Section 4.3 follows from the monotonicity of diffeomorphisms. These are design guarantees rather than circular reductions: the method is not fitted to the quantities it then reports as results, and the benchmark numbers in Tables 1 and 4 are measured on PASCAL-MTL and NYUD against external baselines (MTLoRA, LoRA, adapters, and full fine-tuning). The self-citations [32,33] are contextual related-work references (DiGRAF and a geometric fine-tuning workshop paper) and are not load-bearing; the CPAB diffeomorphism machinery is attributed to the external works of Freifeld et al. [14,15] and the Martinez et al. package [34]. Two non-circularity caveats should be recorded separately. First, the full-rank update statement is asserted without a formal proof, but it is a direct corollary of Eq. (6) and monotonicity, so it is an omitted proof rather than a circular step. Second, Definition 3.1 (Eq. 4) defines f_theta on a closed interval [a,b], while Section 4.2 (Eq. 6) applies f_theta to arbitrary singular values without specifying the domain mapping; this is a specification gap that should be checked against the released code, but it does not make any reported result equivalent to its inputs. Reproducibility concerns about single-run baselines and the absence of error bars affect confidence in the SOTA claim, but they are verification issues, not circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. The free parameters are the CPAB hyperparameters and the undisclosed domain mapping. The axioms are mostly standard linear algebra plus two domain assumptions about the semantic meaning of singular vectors and the role of rank in task interference.

free parameters (5)
  • CPAB tessellation size NP = 32
    Selected by grid search over {16, 32, 64, 128} on PASCAL MTL, as shown in Supplementary Figure 8.
  • CPAB interval domain [a,b] = unspecified
    The domain of the velocity field is never given; the paper does not describe how singular values are mapped into this interval, so the method is not fully specified.
  • dropout = grid over {0.0, 0.05, 0.5}
    Tuned per dataset using the Weights & Biases framework, as described in Supplementary C.
  • learning rate = grid over {0.005, 0.0005, 0.00005}
    Tuned per dataset via grid search.
  • weight decay = grid over {0.05, 0.005, 0.0005, 0.00005}
    Tuned per dataset via grid search.
assumptions (5)
  • standard math Every real matrix has an SVD, and U and V are orthonormal bases.
    Used in Eq. (1) to decompose W and in Eq. (6) to reconstruct the adapted weight matrix.
  • domain assumption Pre-trained weight singular vectors encode semantically meaningful visual features that should be preserved during adaptation.
    Motivating example in Section 1 and Figure 2; this is the central premise of the method and is not proven.
  • domain assumption Low-rank updates cause task interference in MTL, and full-rank updates reduce it.
    Inferred from the incremental learning phenomenon in [5]; stated as a conjecture in Sections 1 and 2.
  • standard math CPAB diffeomorphisms are monotone and therefore preserve the ordering of singular values.
    Used in Section 4.3 to claim feature relativity preservation.
  • ad hoc to paper Applying the same scalar diffeomorphism to all singular values is expressive enough for task-specific adaptation.
    Underlies Eq. (6); the paper provides no capacity analysis or lower bound on expressivity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations." pith.science (2026). https://pith.science/paper/HMIBALIA

@misc{pith2026250206029,
  author       = {Pith},
  title        = {Pith review of: DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HMIBALIA}},
  note         = {Machine review of arXiv:2502.06029}
}
read the original abstract

Pre-trained Vision Transformers now serve as powerful tools for computer vision. Yet, efficiently adapting them for multiple tasks remains a challenge that arises from the need to modify the rich hidden representations encoded by the learned weight matrices, without inducing interference between tasks. Current parameter-efficient methods like LoRA, which apply low-rank updates, force tasks to compete within constrained subspaces, ultimately degrading performance. We introduce DiTASK a novel Diffeomorphic Multi-Task Fine-Tuning approach that maintains pre-trained representations by preserving weight matrix singular vectors, while enabling task-specific adaptations through neural diffeomorphic transformations of the singular values. By following this approach, DiTASK enables both shared and task-specific feature modulations with minimal added parameters. Our theoretical analysis shows that DITASK achieves full-rank updates during optimization, preserving the geometric structure of pre-trained features, and establishing a new paradigm for efficient multi-task learning (MTL). Our experiments on PASCAL MTL and NYUD show that DiTASK achieves state-of-the-art performance across four dense prediction tasks, using 75% fewer parameters than existing methods. Our code is available [here](https://github.com/ipsitmantri/DiTASK).

Figures

Figures reproduced from arXiv: 2502.06029 by the authors.

Figure 1
Figure 1. Performance–efficiency trade-off in multi-task [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Image recovery comparison of LoRA and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of our DITASK within Multi-Task Learning (MTL). (Left) Input features x are transformed via a modulated weight matrix WA, constructed by applying a neural diffeomorphism f θ to the singular values of a pre-trained weight matrix W, resulting in updated features WAx. (Right) We use two sets of DITASK modules: a joint transformation module for learning task synergies, and task-specific modules for individual a… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: DITASK in the i-th Swin Transformer stage for multi-task learning. Task-agnostic modules are applied in all blocks except the last, which uses task-specific modules to capture task-dependent features. 3. Preliminaries and Background We combine two key components: the w…
Figure 5
Figure 5. Figure 5: Comparison of adaptation parameter budget of the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Semantic segmentation predictions on PASCAL MTL [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Effect of task-specific and task-agnostic components [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Effect of tessellation size using DITASK’s perfor￾mance on PASCAL MTL • Task-specific metrics: MIOU for segmentation tasks and RMSE for surface normals and depth estimation. • Average relative improvement across tasks: ∆m = 1 K X K k=1 (−1)lk (Mk − Mst,k) Mst,k , (7) w…
Figure 11
Figure 11. Figure 11: Qualitative comparison of depth estimation on repre [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 51 canonical work pages

  1. [1]

    Mtlora: Low-rank adaptation approach for effi- cient multi-task learning

    Ahmed Agiza, Marina Neseem, and Sherief Reda. Mtlora: Low-rank adaptation approach for effi- cient multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16196–16205, 2024. 1, 2, 3, 4, 5, 6, 7

    work page 2024

  2. [2]

    Construction of bayesian deformable mod- els via a stochastic approximation algorithm: a con- vergence study

    Stéphanie Allassonnière, Estelle Kuhn, and Alain Trouvé. Construction of bayesian deformable mod- els via a stochastic approximation algorithm: a con- vergence study. Bernoulli, 2010. 4

    work page 2010

  3. [3]

    Bayesian mixed effect atlas esti- mation with a diffeomorphic deformation model

    Stéphanie Allassonnière, Stanley Durrleman, and Estelle Kuhn. Bayesian mixed effect atlas esti- mation with a diffeomorphic deformation model. SIAM Journal on Imaging Sciences , 8(3):1367– 1395, 2015. 4

    work page 2015

  4. [4]

    Experiment tracking with weights and biases, 2020

    Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb. com/. Software available from wandb.com. 1

    work page 2020

  5. [5]

    Transformers learn through gradual rank increase

    Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, and Joshua Susskind. Transformers learn through gradual rank increase. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 24519–24551. Curran Associates, Inc., 2023. URL https : / / proceedings . ne...

    work page 2023

  6. [6]

    What makes pre-trained visual representations suc- cessful for robust manipulation?, 2023

    Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations suc- cessful for robust manipulation?, 2023. URL https://arxiv.org/abs/2312.12444. 2

    arXiv 2023

  7. [7]

    Multitask learning

    Rich Caruana. Multitask learning. Machine learn- ing, 28:41–75, 1997. 4

    work page 1997

  8. [8]

    Trainable highly-expressive activa- tion functions

    Irit Chelly, Shahaf E Finder, Shira Ifergane, and Oren Freifeld. Trainable highly-expressive activa- tion functions. In European Conference on Com- puter Vision, 2024. 4

    work page 2024

Show all 67 references
  1. [9]

    Class-balanced loss based on effective number of samples, 2019

    Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples, 2019. URL https: //arxiv.org/abs/1901.05555. 6

    2019 arXiv

  2. [10]

    Imagenet: A large-scale hierar- chical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierar- chical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 1

    2009

  3. [11]

    Deep diffeomorphic transformer net- works

    Nicki Skafte Detlefsen, Oren Freifeld, and Søren Hauberg. Deep diffeomorphic transformer net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4403–4412, 2018. 4

    2018

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  5. [13]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Inter- national Journal of Computer Vision, 88:303–338, 06 2010. doi: 10.1007/s11263-009-0275-4. 2, 6

    2010 doi

  6. [14]

    Highly-expressive spaces of well-behaved transformations: Keeping it simple

    Oren Freifeld, Soren Hauberg, Kayhan Batmanghe- lich, and John W Fisher. Highly-expressive spaces of well-behaved transformations: Keeping it simple. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 2911–2919, 2015. 2, 4, 5

    2015

  7. [15]

    Transformations based on continuous piecewise-affine velocity fields

    Oren Freifeld, Soren Hauberg, Kayhan Batmanghe- lich, and Jonn W Fisher. Transformations based on continuous piecewise-affine velocity fields. IEEE transactions on pattern analysis and machine intel- ligence, 39(12):2496–2509, 2017. 2, 4, 5

    2017

  8. [16]

    Preserv- ing pre-trained features helps calibrate fine-tuned language models, 2023

    Guande He, Jianfei Chen, and Jun Zhu. Preserv- ing pre-trained features helps calibrate fine-tuned language models, 2023. URL https://arxiv. org/abs/2305.19249. 2

    2023 arXiv

  9. [17]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021. 7, 4

    2021 arXiv

  10. [18]

    Parameter-efficient transfer learning for nlp, 2019

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzeb- ski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019. URL https://arxiv.org/abs/ 1902.00751. 1, 2, 3, 6

    2019 arXiv

  11. [19]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https://openreview.net/forum?id= nZeVKeeFYf9. ...

    2022

  12. [20]

    Hospedales

    Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CV...

    2022

  13. [21]

    Going beyond multi-task dense prediction with synergy embedding models

    Huimin Huang, Yawen Huang, Lanfen Lin, Ruofeng Tong, Yen-Wei Chen, Hao Zheng, Yuexi- ang Li, and Yefeng Zheng. Going beyond multi-task dense prediction with synergy embedding models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p...

    2024

  14. [22]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , pages 709–727. Springer, 2022. 3, 6, 7

    2022

  15. [23]

    Compacter: Efficient low-rank hypercomplex adapter layers

    Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neu- ral Information Processing Systems, 34:1022–1035,

  16. [24]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 7482–7491, 2018. 3

    2018

  17. [25]

    Svft: Parameter- efficient fine-tuning with singular vectors, 2024

    Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bo- jchevski, and Sujay Sanghavi. Svft: Parameter- efficient fine-tuning with singular vectors, 2024. 3, 7

    2024

  18. [26]

    Dora: Weight- decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight- decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024. 3, 7

    2024 arXiv

  19. [27]

    Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks

    Yen-Cheng Liu, Chih-Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks. Ad- vances in Neural Information Processing Systems, 35:36889–36901, 2022. 3, 6, 7

    2022

  20. [28]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 5, 6

    2021

  21. [29]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https: //arxiv.org/abs/1711.05101. 1

    2019 arXiv

  22. [30]

    Investigating forgetting in pre-trained representations through continual learn- ing, 2023

    Yun Luo, Zhen Yang, Xuefeng Bai, Fandong Meng, Jie Zhou, and Yue Zhang. Investigating forgetting in pre-trained representations through continual learn- ing, 2023. URL https://arxiv.org/abs/ 2305.05968. 2

    2023 arXiv

  23. [31]

    Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks

    Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.04489, 2021. 2, 3, 6, 7

    2021 arXiv

  24. [32]

    Rethink- ing fine-tuning through geometric perspective

    Krishna Sri Ipsit Mantri, Moshe Eliasof, Carola- Bibiane Schönlieb, and Bruno Ribeiro. Rethink- ing fine-tuning through geometric perspective. In UniReps: 2nd Edition of the Workshop on Unify- ing Representations in Neural Models, 2024. URL https://openreview.net/forum?id= FFQ...

    2024

  25. [33]

    DiGRAF: Dif- feomorphic graph-adaptive activation function

    Krishna Sri Ipsit Mantri, Xinzhi Wang, Carola- Bibiane Schönlieb, Bruno Ribeiro, Beatrice Bevilacqua, and Moshe Eliasof. DiGRAF: Dif- feomorphic graph-adaptive activation function. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL http...

    2024

  26. [34]

    Closed-form diffeomorphic transforma- tions for time series alignment

    Iñigo Martinez, Elisabeth Viles, and Igor G Olaizola. Closed-form diffeomorphic transforma- tions for time series alignment. In International Conference on Machine Learning , pages 15122– 15158. PMLR, 2022. 4, 5

    2022

  27. [35]

    Cross-stitch networks for multi- task learning

    Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi- task learning. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition , pages 3994–4003, 2016. 3

    2016

  28. [36]

    Insights on representational similarity in neural networks with canonical correlation

    Ari Morcos, Maithra Raghu, and Samy Ben- gio. Insights on representational similarity in neural networks with canonical correlation. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Syste...

    2018

  29. [37]

    Pre-trained vision and language trans- formers are few-shot incremental learners

    Keon-Hee Park, Kyungwoo Song, and Gyeong- Moon Park. Pre-trained vision and language trans- formers are few-shot incremental learners. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 23881–23890, June 2024. 2

    2024

  30. [38]

    Language mod- els are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language mod- els are unsupervised multitask learners. 2019. URL https : / / api . semanticscholar . org / CorpusID:160025533. 3

    2019

  31. [39]

    Imagenet large scale visual recog- nition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recog- nition challenge. International journal of computer vision, 115:211–252, 2015. 1, 6

    2015

  32. [40]

    Saxe, James L

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks,

  33. [41]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision– ECCV 2012: 12th European Conference on Com- puter Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746–760. ...

    2012

  34. [42]

    How to train your vit? data, augmentation, and regularization in vision transformers

    Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270, 2021. 1

    2021 arXiv

  35. [43]

    Varia- tional pdes for acceleration on manifolds and appli- cation to diffeomorphisms

    Ganesh Sundaramoorthi and Anthony Yezzi. Varia- tional pdes for acceleration on manifolds and appli- cation to diffeomorphisms. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Infor- mation Processing Systems,...

    2018

  36. [44]

    Vl- adapter: Parameter-efficient transfer learning for vision-and-language tasks

    Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl- adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5227–5237, 2022. 3, 6, 7

    2022

  37. [45]

    Is learning the n-th thing any easier than learning the first? In D

    Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In D. Touret- zky, M.C. Mozer, and M. Hasselmo, editors, Advances in Neural Information Process- ing Systems , volume 8. MIT Press, 1995. URL https : / / proceedings . neurips . cc / paper _ files ...

    1995

  38. [46]

    Mti-net: Multi-scale task interac- tion networks for multi-task learning

    Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Mti-net: Multi-scale task interac- tion networks for multi-task learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 527–543. Springer, 2020. 6

    2020

  39. [47]

    Multi-task learning for dense pre- diction tasks: A survey

    Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense pre- diction tasks: A survey. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 44(7):3614– 3633, 2022. doi: 10.1109/TPAMI...

    2022

  40. [48]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information pro- cessing systems, 30, 2017. 1

    2017

  41. [49]

    Continuous piecewise- affine based motion model for image animation

    Hexiang Wang, Fengqi Liu, Qianyu Zhou, Ran Yi, Xin Tan, and Lizhuang Ma. Continuous piecewise- affine based motion model for image animation. arXiv preprint arXiv:2401.09146, 2024. 4

    2024 arXiv

  42. [50]

    Deep high-resolution representation learning for vi- sual recognition

    Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for vi- sual recognition. IEEE transactions on pattern anal- ysis and machine intelligence, 43(10):...

  43. [51]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, ...

    2021

  44. [52]

    Regularization-free diffeomorphic temporal alignment nets

    Ron Shapira Weber and Oren Freifeld. Regularization-free diffeomorphic temporal alignment nets. In International Conference on Machine Learning, pages 30794–30826. PMLR,

  45. [53]

    Man- ning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, At- ticus Geiger, Dan Jurafsky, Christopher D. Man- ning, and Christopher Potts. ReFT: Representa- tion finetuning for language models. 2024. URL arxiv.org/abs/2404.03592. 3, 7

    2024 arXiv

  46. [54]

    Pad-net: Multi-tasks guided prediction-and- distillation network for simultaneous depth estima- tion and scene parsing

    Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction-and- distillation network for simultaneous depth estima- tion and scene parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 675–684, 2018. 6

    2018

  47. [55]

    Inverted pyramid multi- task transformer for dense scene understanding

    Hanrong Ye and Dan Xu. Inverted pyramid multi- task transformer for dense scene understanding. In Computer Vision–ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23–27, 2022, Pro- ceedings, Part XXVII , pages 514–530. Springer,

    2022

  48. [56]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages ...

    2020

  49. [57]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824– 5836, 2020. 4

    2020

  50. [58]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Gold- berg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021. 3, 6, 7

    2021

  51. [59]

    A large-scale study of repre- sentation learning with the visual task adaptation benchmark, 2020

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lu- cas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Ne...

    2020 arXiv

  52. [60]

    Bayesian statistical shape analysis on the manifold of diffeo- morphisms

    Miaomiao Zhang and P Thomas Fletcher. Bayesian statistical shape analysis on the manifold of diffeo- morphisms. Algorithmic Advances in Riemannian Geometry and Applications: For Machine Learn- ing, Computer Vision, Statistics, and Optimization, pages 1–23, 2016. 4 DITASK: Mult...

    2016

  53. [62]

    W = UΣV⊤, where: U ∈ Rc2×c2 , Σ = diag(σ1,

    Compute the Singular Value Decomposition (SVD) of W: A. W = UΣV⊤, where: U ∈ Rc2×c2 , Σ = diag(σ1, . . . , σp), V ∈ Rc1×c1 . // Joint Adaptation

  54. [63]

    ΣJ = diag(f θj (σ1), · · ·, fθj (σp))

  55. [64]

    Construct WJ = UΣJ V⊤

  56. [65]

    h = WJ x // Task-Specific Adaptation

  57. [66]

    xk = x if not last block, else xk B

    For k = 1, · · ·, Kdo A. xk = x if not last block, else xk B. Σk = diag(f θk (σ1), · · ·, fθk (σp)). C. Construct Wk = UΣkV⊤. D. hk = Wk xk

  58. [67]

    Gradient Analysis We analyze the memory requirements for low-rank adap- tation methods, such as LoRA, and compare them with DITASK in terms of gradient storage

    Return h, {hk}K k=1 B. Gradient Analysis We analyze the memory requirements for low-rank adap- tation methods, such as LoRA, and compare them with DITASK in terms of gradient storage. LoRA adapts a pre-trained weight matrix W ∈ Rc2×c1 using two learnable low rank matricesB ∈ R...

  59. [2014]

    org / abs / 1312.6120

    URL https : / / arxiv . org / abs / 1312.6120. 1

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.