Pith. sign in

REVIEW 2 major objections 5 minor 74 references

SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbation

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAMO is a sharpness-aware multi-task optimizer that blends the averaged-loss gradient with low-cost forward-pass estimates of per-task gradients, reporting improved task balance and lower cost than per-task SAM approaches.

desk verdict A genuinely new MTL-SAM combination with solid benchmark gains, but the paper's headline efficiency claim contradicts its own Algorithm 1 and needs code-level verification before it can be taken at face value. read the letter →

arxiv 2507.07883 v3 pith:K2V7VBKO submitted 2025-07-10 cs.LG

Hao Ban , Gokul Ram Subramani , Kaiyi Ji This is my paper
classification cs.LG
keywords multi-tasklearningsharpness-awareminimizationtaskconflictsgradientmanipulationzeroth-orderoptimizationlosslandscapeflatnessefficienttrainingforward-passestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the task conflicts that limit multi-task learning can be reduced by adding a sharpness-seeking perturbation that mixes two kinds of information: the gradient of the averaged loss, shared across tasks, and per-task gradients. The authors report that global information alone and local information alone each help, but neither dominates, so they build SAMO, a perturbation that blends both. The local part is computed with a two-forward-pass zeroth-order estimator and layerwise normalization, avoiding the extra backward passes that earlier per-task SAM methods require. On five benchmarks, SAMO improves three existing multi-task optimizers and runs substantially faster than the per-task SAM baseline F-MTL.

What carries the argument

The load-bearing object is the joint global–local perturbation defined in Eq. (1): each task gets its own perturbation $\hat{\epsilon}_i$ formed by the normalized weighted sum of the global gradient $\nabla_\theta l_0$ and the task gradient $\nabla_\theta l_i$, weighted by $\alpha \in [0,1]$. The local gradient is not backpropagated; it comes from the SPSA estimator $(\ell_i(\theta+\mu z_i)-\ell_i(\theta-\mu z_i))z_i/(2\mu)$ with a random Gaussian direction $z_i$, and Eq. (4) rescales it layerwise so each layer's norm matches the corresponding layer of the global gradient. Together these pieces convert per-task flatness information into a forward-pass-only perturbation, which is what makes SAMO lightweight.

What would settle it

Profile one SAMO training step on CelebA (40 tasks) and count the number of backward passes while tracking peak memory. If the backward-pass count grows with the number of tasks rather than staying constant, the paper's efficiency claim is false and SAMO is no cheaper than per-task SAM.

Watch

Extended reading notes

Core claim

The paper's central claim is that a flatness-seeking perturbation for multi-task optimization should combine the average-loss gradient with per-task gradients, and that the combination can be made cheap. For each task $i$, SAMO perturbs the parameters by $\hat{\epsilon}_i = \rho (\alpha \nabla_\theta l_0 + (1-\alpha)\nabla_\theta l_i)/\|\alpha \nabla_\theta l_0 + (1-\alpha)\nabla_\theta l_i\|$, where $l_0$ is the averaged loss and $\alpha$ balances global against local information. The gradient that the optimizer receives is then evaluated at $\theta + \hat{\epsilon}_i$. To avoid $K$ extra backward passes, the local gradients $\nabla_\theta l_i$ are replaced by a simultaneous-perturbation stochastic approximation that needs only two forward passes per task, after which each layer is rescaled to the norm of the global gradient. Wrapped around any gradient-manipulation method $M$, this yields the SAMO update, which the paper reports improves LS, MGDA, and FairGrad across Cityscapes, NYU-v2, CelebA, QM9, and Office-Home while staying much faster than F-MTL.

Load-bearing premise

The claim that SAMO needs only one backward pass per step—rather than one per task—is stated in the cost table but not derived; if that accounting is wrong, the method is not lighter than existing per-task SAM approaches.

Editorial extensions

If this is right

  • SAMO can be dropped onto any gradient-manipulation method that consumes per-task gradients; the paper demonstrates it with LS, MGDA, and FairGrad.
  • Because the added per-task work is forward passes rather than backpropagation, the overhead no longer grows with the cost of a backward pass, making the method more attractive as the number of tasks grows.
  • The reported link between flatter loss landscapes and higher cosine similarity among task gradients suggests that other flatness-seeking optimizers could also reduce task conflicts.
  • The sensitivity analysis shows the mixing weight $\alpha$ is stable in $[0.1,0.6]$, while the perturbation radius $\rho$ must be tuned per dataset, especially when task loss scales differ.
  • The positive results on Office-Home, a multi-input benchmark, suggest SAMO's benefits extend to settings where each task has its own data and pretrained features are shared.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that if the single-backward-pass cost model is correct, SAMO's marginal cost of adding a task is only two forward passes, so profiling studies at high task counts (e.g., 40 tasks) would be a sharp test of the method's scalability.
  • A natural extension would be to apply the joint global–local perturbation outside multi-task learning, wherever a shared network faces conflicting objectives—continual learning and multi-user personalization are two such settings—and to measure whether task-gradient cosine similarity rises there as it does in this paper's CelebA experiments.
  • The paper's Office-Home results show smaller gains and attribute them to weak gradient conflicts in a pretrained model; this suggests the method's benefit should scale with measured gradient disagreement, a hypothesis testable by sorting benchmarks by their task-gradient cosine similarity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies sharpness-aware minimization for multi-task learning. It first presents empirical observations that SAM flattens the loss landscape, increases cosine similarity among task gradients, and mitigates task conflicts. Motivated by this, it proposes SAMO, which constructs a joint global-local perturbation: a global component from the average-loss gradient plus task-specific local components. To avoid the K-fold backward-pass cost of per-task SAM, the local gradient directions are approximated with a simultaneous-perturbation stochastic approximation estimator computed with forward passes, followed by a layerwise normalization. Experiments on Cityscapes, NYU-v2, CelebA, QM9, and Office-Home show that wrapping LS, MGDA, and FairGrad with SAMO improves their reported Δm% or accuracy, and the paper claims an efficiency advantage over F-MTL.

Significance. If the efficiency claim is correct, SAMO would be a practical way to bring sharpness-aware training to multi-task learning without the per-task backward-pass overhead of F-MTL, and the breadth of the empirical study is a strength. The code is released, which allows direct verification. However, the paper's central efficiency claim is internally inconsistent with its own algorithm and equations, and the reported performance gains are not accompanied by error bars or multiple-seed results. Both issues are load-bearing for the paper's main contributions, so the current version does not yet support its headline conclusions.

major comments (2)
  1. [§5.1–5.2, Eq. (2), Algorithm 1 line 8, Table 3] The computational cost model contradicts the algorithm. Eq. (2) defines g_i^SAMO = ∇θ li(θ + ε_i), which is a per-task gradient at a task-specific perturbed parameter. Computing this gradient requires one backward pass through the shared network for each of the K tasks. The SPSA estimator in Eq. (3) is used only to approximate the local gradients in Eq. (1) that form the perturbation direction; it is not applied to the second-stage gradient. A faithful implementation of Algorithm 1 therefore costs at least (K+1)Cb + 2KCf, not Cb + 2KCf as listed in Table 3. The paper's own sentence 'the total computation cost of our SAMO involves K + 1 gradient computations' (Section 5.2) supports this reading. Consequently, the abstract's claim that SAMO introduces 'only forward passes', the cost model in Table 3, and the efficiency comparison in Figure 4 are not supported by the described method. The released code should be checked: if it indeed performs K+1 backward passes per step, the cost model and efficiency claims must be corrected and the 'lightweight' characterization downgraded; if it uses a forward-only approximation for Eq. (2), that approximation must be specified and analyzed.
  2. [§6, Tables 4–8 and Appendix Tables 11–14] No error bars or multiple-seed results are reported anywhere in the paper. Many of the claimed improvements are small in absolute terms (e.g., SAMO-FairGrad on Cityscapes improves Δm% from 3.90 to -0.62; on NYU-v2 from -4.96 to -6.55; on CelebA from 0.37 to -0.74). Without quantifying run-to-run variability, the conclusions that SAMO 'consistently enhances' MTL methods and achieves 'state-of-the-art' performance cannot be assessed. The main comparisons should report the mean and standard deviation over at least three independent runs, and the number of seeds used should be stated.
minor comments (5)
  1. [§1, Contributions] There is a typo: 'stat-of-the-art' should be 'state-of-the-art'.
  2. [Table 7] In the SAMO-FairGrad row, the entry '77.0390.25' appears to be two numbers ('77.03' and '90.25') merged without a separator.
  3. [§6.5] The phrase 'perturbate model parameters' should be 'perturb model parameters', and the efficiency comparison should report absolute wall-clock times and memory usage in addition to relative ratios.
  4. [Appendix B] Sensitivity analyses are reported for α and ρ but not for the SPSA step-size µ in Eq. (3), even though µ is a user-set parameter that affects the bias-variance tradeoff of the forward-only gradient approximation.
  5. [§6.2, Tables 4–6] The comparison protocol for F-MTL is unclear: F-MTL results are taken from the original paper for Cityscapes and NYU-v2, but F-MTL is not reported for CelebA and QM9; please clarify how the baselines were obtained in each table and whether the same backbone and training schedule were used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SAMO's performance claims are empirical against external benchmarks; the flagged efficiency inconsistency is a correctness issue, not a circular derivation.

full rationale

The paper's central claims are empirical: SAMO is evaluated on external MTL benchmarks (Cityscapes, NYU-v2, CelebA, QM9, Office Home) against standard baselines, and its improvements are reported as measured outcomes rather than derived from its own assumptions. The hyperparameters alpha, rho, and mu are tuned via sensitivity analysis and standard validation practices, not fitted to the test metrics and then presented as predictions, so the fitted-input-called-prediction pattern does not apply. The paper's motivating observation that SAM mitigates task conflicts is an empirical finding, not a definitional consequence of the method. Self-citations, such as FairGrad [6] by the same group, are used as baseline methods and as prior work, but they are not invoked to justify SAMO's core perturbation mechanism or to forbid alternatives; there is no load-bearing self-citation chain and no imported uniqueness theorem. The only substantive concern in the manuscript is an internal consistency issue in the efficiency accounting: Algorithm 1 and Eq. (2) appear to require per-task backward passes, while Table 3 lists only Cb + 2KCf overhead and the text claims 'only forward passes.' That is a potential correctness or reproducibility problem with the runtime claim, but it is not circularity: the efficiency claim is not derived from itself, nor is any predicted quantity equivalent to a fitted input. Therefore, no circular step meets the evidentiary bar, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central performance claim depends on three tuned scalars and on the validity of the forward-only gradient approximation. The cost model assumption is especially fragile because it contradicts the formal equations.

free parameters (3)
  • alpha (global-local weight) = 0.5 (0.1 for QM9)
    Balances global and local perturbation. Tuned per dataset; sensitivity analysis shows performance varies from 14.30 to 24.81 on Cityscapes and from -6.55 to -4.56 on NYU-v2 across alpha values.
  • rho (perturbation radius) = 0.001 (0.00001 for QM9)
    SAM perturbation size. Tuned per dataset; Table 10 shows strong sensitivity, e.g., SAMO-FairGrad on Cityscapes ranges from -0.62 to 14.03 across rho values.
  • mu (SPSA step) = 0.01
    Step size for the SPSA estimator in Eq. (3). Fixed without sensitivity analysis.
assumptions (3)
  • domain assumption The layerwise-normalized SPSA estimator (Eq. 4) faithfully approximates the true task gradient direction for the purpose of computing SAM perturbations.
    No theoretical guarantee or error bound is provided; the ablation only compares global normalization and no normalization, not against exact gradients.
  • domain assumption Flatter loss landscapes reduce task conflict and improve MTL generalization.
    Supported by the toy example and Hessian measurements in Section 4, but not formally proved.
  • ad hoc to paper The computational cost model in Table 3 is correct, i.e., Eq. (2) can be evaluated without per-task backward passes.
    Table 3 lists SAMO cost as Cb + 2KCf, but Eq. (2) computes per-task gradients at perturbed points, which normally require K backward passes. The paper never clarifies how these are avoided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbation." pith.science (2026). https://pith.science/paper/K2V7VBKO

@misc{pith2026250707883,
  author       = {Pith},
  title        = {Pith review of: SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K2V7VBKO}},
  note         = {Machine review of arXiv:2507.07883}
}
read the original abstract

Multi-task learning (MTL) enables a joint model to capture commonalities across multiple tasks, reducing computation costs and improving data efficiency. However, a major challenge in MTL optimization is task conflicts, where the task gradients differ in direction or magnitude, limiting model performance compared to single-task counterparts. Sharpness-aware minimization (SAM) minimizes task loss while simultaneously reducing the sharpness of the loss landscape. Our empirical observations show that SAM effectively mitigates task conflicts in MTL. Motivated by these findings, we explore integrating SAM into MTL but face two key challenges. While both the average loss gradient and individual task gradients-referred to as global and local information-contribute to SAM, how to combine them remains unclear. Moreover, directly computing each task gradient introduces significant computational and memory overheads. To address these challenges, we propose SAMO, a lightweight \textbf{S}harpness-\textbf{A}ware \textbf{M}ulti-task \textbf{O}ptimization approach, that leverages a joint global-local perturbation. The local perturbations are approximated using only forward passes and are layerwise normalized to improve efficiency. Extensive experiments on a suite of multi-task benchmarks demonstrate both the effectiveness and efficiency of our method. Code is available at https://github.com/OptMN-Lab/SAMO.

Figures

Figures reproduced from arXiv: 2507.07883 by the authors.

Figure 1
Figure 1. Optimization trajectories for a two-objective synthetic [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Cosine similarities of task gradients for LS (left) and LS [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Our SAMO computes joint global and local perturbations for all tasks. Left: The joint perturbation is obtained by a weighted [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Efficiency comparison on Cityscapes (2-task) and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Cosine similarities of task gradients after 5th (left) and 10th (right) epoch. In each figure, LS is on the left and LS with SAM is [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Feature visualization. Left: FairGrad . Right: SAMO-FairGrad. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 65 canonical work pages

  1. [1]

    Sharp-maml: Sharpness-aware model-agnostic meta learning

    Momin Abbas, Quan Xiao, Lisha Chen, Pin-Yu Chen, and Tianyi Chen. Sharp-maml: Sharpness-aware model-agnostic meta learning. In International Conference on Machine Learning, pages 10–32. PMLR, 2022. 1

    work page 2022

  2. [2]

    Towards understanding sharpness-aware minimization

    Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In Interna- tional Conference on Machine Learning , pages 639–668. PMLR, 2022. 2

    work page 2022

  3. [3]

    Sharpness-aware minimization leads to low-rank features

    Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. Advances in Neural Information Process- ing Systems, 36:47032–47051, 2023. 3

    work page 2023

  4. [4]

    Segnet: A deep convolutional encoder-decoder architecture for image segmentation

    Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 39(12):2481–2495, 2017. 6

    work page 2017

  5. [5]

    Sharpness-aware minimization improves language model generalization

    Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 7360–7371, 2022. 1

    work page 2022

  6. [6]

    Fair resource allocation in multi-task learning

    Hao Ban and Kaiyi Ji. Fair resource allocation in multi-task learning. In International Conference on Machine Learning, pages 2715–2731. PMLR, 2024. 1, 2, 6, 7

    work page 2024

  7. [7]

    Automated Search for Resource-Efficient Branched Multi-Task Networks

    David Bruggemann, Menelaos Kanakis, Stamatios Geor- goulis, and Luc Van Gool. Automated search for resource- efficient branched multi-task networks. arXiv preprint arXiv:2008.10292, 2020. 2

    work page Pith review arXiv 2008

  8. [8]

    Swad: Domain generalization by seeking flat minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems , 34: 22405–22418, 2021. 1

    work page 2021

Show all 74 references
  1. [9]

    Multi-task learning in natural language processing: An overview

    Shijie Chen, Yu Zhang, and Qiang Yang. Multi-task learning in natural language processing: An overview. ACM Comput- ing Surveys, 56(12):1–32, 2024. 1

    2024

  2. [10]

    Gradient-based multi-objective deep learning: Algorithms, theories, applica- tions, and beyond

    Weiyu Chen, Xiaoyuan Zhang, Baijiong Lin, Xi Lin, Han Zhao, Qingfu Zhang, and James T Kwok. Gradient-based multi-objective deep learning: Algorithms, theories, applica- tions, and beyond. arXiv preprint arXiv:2501.10945, 2025. 1

    2025 arXiv

  3. [11]

    Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and An- drew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Inter- national Conference on Machine Learning , pages 794–803. PMLR, 2018. 2

    2018

  4. [12]

    Just pick a sign: Optimizing deep multitask models with gra- dient sign dropout

    Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gra- dient sign dropout. Advances in Neural Information Process- ing Systems, 33:2039–2050, 2020. 7

    2020

  5. [13]

    Mod-squad: Designing mixtures of experts as modular multi-task learners

    Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G Learned-Miller, and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages ...

    2023

  6. [14]

    Why does sharpness- aware minimization generalize better than sgd? Advances in Neural Information Processing Systems, 36:72325–72376,

    Zixiang Chen, Junkai Zhang, Yiwen Kou, Xiangning Chen, Cho-Jui Hsieh, and Quanquan Gu. Why does sharpness- aware minimization generalize better than sgd? Advances in Neural Information Processing Systems, 36:72325–72376,

  7. [15]

    Deep learning in video multi-object tracking: A survey.Neu- rocomputing, 381:61–88, 2020

    Gioele Ciaparrone, Francisco Luque S ´anchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, and Francisco Herrera. Deep learning in video multi-object tracking: A survey.Neu- rocomputing, 381:61–88, 2020. 1

    2020

  8. [16]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Re...

    2016

  9. [17]

    Multi-task learning with deep neural networks: A survey

    Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020. 1

    2009 arXiv

  10. [18]

    Instance-aware se- mantic segmentation via multi-task network cascades

    Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware se- mantic segmentation via multi-task network cascades. In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, pages 3150–3158, 2016. 1

    2016

  11. [19]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. Ieee, 2009. 8

    2009

  12. [20]

    Multiple-gradient descent algorithm (mgda) for multiobjective optimization

    Jean-Antoine D ´esid´eri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313–318, 2012. 2

    2012

  13. [21]

    Sharp minima can generalize for deep nets

    Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In In- ternational Conference on Machine Learning , pages 1019–

  14. [22]

    Efficient sharpness-aware minimization for improved training of neu- ral networks

    Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Lian- gli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-aware minimization for improved training of neu- ral networks. In International Conference on Learning Rep- resentations. 2

  15. [23]

    M 3vit: Mixture-of-experts vision transformer for efficient multi- task learning with model-accelerator co-design

    Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, et al. M 3vit: Mixture-of-experts vision transformer for efficient multi- task learning with model-accelerator co-design. Advances in Neural Information Processing Systems, 35:28...

  16. [24]

    Miti- gating gradient bias in multi-objective learning: A provably convergent approach

    Heshan Devaka Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Miti- gating gradient bias in multi-objective learning: A provably convergent approach. In The Eleventh International Confer- ence on Learning Representations. 2, 7

  17. [25]

    Fast graph repre- sentation learning with pytorch geometric

    Matthias Fey and Jan Eric Lenssen. Fast graph repre- sentation learning with pytorch geometric. arXiv preprint arXiv:1903.02428, 2019. 6

    1903 arXiv

  18. [26]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations. 1, 2, 3

  19. [27]

    Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction

    Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 3205–3214, 2019. 2

    2019

  20. [28]

    Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning

    Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11543–11552, 2020. 2

    2020

  21. [29]

    Variance-reduced zeroth-order methods for fine-tuning language models

    Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. In Forty-first In- ternational Conference on Machine Learning. 5

  22. [30]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 8

    2016

  23. [31]

    Simplifying neu- ral nets by discovering flat minima

    Sepp Hochreiter and J ¨urgen Schmidhuber. Simplifying neu- ral nets by discovering flat minima. Advances in Neural In- formation Processing Systems, 7, 1994. 2

    1994

  24. [32]

    Flat minima.Neu- ral Computation, 9(1):1–42, 1997

    Sepp Hochreiter and J ¨urgen Schmidhuber. Flat minima.Neu- ral Computation, 9(1):1–42, 1997. 2

    1997

  25. [33]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations. 4

  26. [34]

    The break-even point on optimization trajectories of deep neural networks

    Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, De- vansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. InInternational Conference on Learn- ing Representations. 3

  27. [35]

    Rotograd: Gradient ho- mogenization in multitask learning

    Adri ´an Javaloy and Isabel Valera. Rotograd: Gradient ho- mogenization in multitask learning. In International Confer- ence on Learning Representations. 2

  28. [36]

    Fantastic generalization mea- sures and where to find them

    Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization mea- sures and where to find them. In International Conference on Learning Representations. 2

  29. [37]

    Multi-task learning using uncertainty to weigh losses for scene geome- try and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geome- try and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7482– 7491, 2018. 1, 2, 6

    2018

  30. [38]

    On large- batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large- batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Rep- resentations, 2017. 2

    2017

  31. [39]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Ad- vances in Neural Information Processing Systems, 31, 2018. 2

    2018

  32. [40]

    LibMTL: A Python library for multi-task learning

    Baijiong Lin and Yu Zhang. LibMTL: A Python library for multi-task learning. Journal of Machine Learning Research, 24(209):1–7, 2023. 8

    2023

  33. [41]

    Rea- sonable effectiveness of random weighting: A litmus test for multi-task learning

    Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor Tsang. Rea- sonable effectiveness of random weighting: A litmus test for multi-task learning. Transactions on Machine Learning Re- search. 2, 6

  34. [42]

    Conflict-averse gradient descent for multi-task learn- ing

    Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learn- ing. Advances in Neural Information Processing Systems , 34:18878–18890, 2021. 1, 2, 6, 7

    2021

  35. [43]

    Famo: Fast adaptive multitask optimization

    Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization. Advances in Neural Information Processing Systems, 36:57226–57243, 2023. 1, 6, 7

    2023

  36. [44]

    Towards impartial multi-task learning

    Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. InInternational Con- ference on Learning Representations. 7

  37. [45]

    End- to-end multi-task learning with attention

    Shikun Liu, Edward Johns, and Andrew J Davison. End- to-end multi-task learning with attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1871–1880, 2019. 6

    2019

  38. [46]

    Towards efficient and scalable sharpness-aware minimization

    Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12360– 12370, 2022. 2

    2022

  39. [47]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision , pages 3730–3738, 2015. 6

    2015

  40. [48]

    Multi- task learning as a bargaining game

    Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi- task learning as a bargaining game. In International Con- ference on Machine Learning, pages 16428–16446. PMLR,

  41. [49]

    Improving multi-task learning via seeking task-based flat regions

    Hoang Phan, Lam Tran, Ngoc N Tran, Nhat Ho, Dinh Phung, and Trung Le. Improving multi-task learning via seeking task-based flat regions. arXiv preprint arXiv:2211.13723 ,

  42. [50]

    Quantum chemistry structures and properties of 134 kilo molecules

    Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole V on Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1): 1–7, 2014. 6

    2014

  43. [51]

    An overview of multi-task learning in deep neural networks

    Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017. 1

    2017 arXiv

  44. [52]

    Latent multi-task architecture learning

    Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task architecture learning. In Proceedings of the AAAI Conference on Artificial Intel- ligence, pages 4822–4829, 2019. 2

    2019

  45. [53]

    Multi-task learning as multi-objective optimization

    Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in Neural Informa- tion Processing Systems, 31, 2018. 1, 2, 6

    2018

  46. [54]

    Independent component alignment for multi-task learning

    Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component alignment for multi-task learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 20083–20093, 2023. 1, 2

    2023

  47. [55]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision–ECCV 2012: 12th Eu- ropean Conference on Computer Vision, Florence, Italy, Oc- tober 7-13, 2012, Proceedings, Part V 12 , pages 746–760...

    2012

  48. [56]

    Multivariate stochastic approximation using a simultaneous perturbation gradient approximation

    James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3):332–341, 1992. 5

    1992

  49. [57]

    Sharpness-aware minimization enhances fea- ture quality via balanced learning

    Jacob Mitchell Springer, Vaishnavh Nagarajan, and Aditi Raghunathan. Sharpness-aware minimization enhances fea- ture quality via balanced learning. In The Twelfth Interna- tional Conference on Learning Representations. 3

  50. [58]

    Av-superb: A multi-task eval- uation benchmark for audio-visual representation models

    Yuan Tseng, Layne Berry, Yi-Ting Chen, I-Hsiang Chiu, Hsuan-Hao Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung- Yu Wang, Haibin Wu, et al. Av-superb: A multi-task eval- uation benchmark for audio-visual representation models. In ICASSP 2024-2024 IEEE International Conference on ...

    2024

  51. [59]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9 (11), 2008. 3

    2008

  52. [60]

    Multi-task learning for dense prediction tasks: A survey

    Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3614–3633, 2021. 1

    2021

  53. [61]

    Deep hashing network for unsupervised domain adaptation

    Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5018–5027, 2017. 8

    2017

  54. [62]

    Sharpness-aware gradient matching for domain generaliza- tion

    Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generaliza- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3769–3778,

  55. [63]

    Vi- ola: Conditional language models for speech recognition, synthesis, and translation

    Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Vi- ola: Conditional language models for speech recognition, synthesis, and translation. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing, 2024. 1

    2024

  56. [64]

    Theoretical study of conflict-avoidant multi-objective reinforcement learning

    Yudan Wang, Peiyao Xiao, Hao Ban, Kaiyi Ji, and Shaofeng Zou. Theoretical study of conflict-avoidant multi-objective reinforcement learning. IEEE Transactions on Information Theory, 2025. 2

    2025

  57. [65]

    How sharpness- aware minimization minimizes sharpness? In The Eleventh International Conference on Learning Representa- tions, 2023

    Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How sharpness- aware minimization minimizes sharpness? In The Eleventh International Conference on Learning Representa- tions, 2023. 3

    2023

  58. [66]

    Direction-oriented multi-objective learning: Simple and provable stochastic al- gorithms

    Peiyao Xiao, Hao Ban, and Kaiyi Ji. Direction-oriented multi-objective learning: Simple and provable stochastic al- gorithms. Advances in Neural Information Processing Sys- tems, 36:4509–4533, 2023. 1, 2, 6

    2023

  59. [67]

    Scalable bilevel loss balancing for multi-task learning.arXiv preprint arXiv:2502.08585, 2025

    Peiyao Xiao, Chaosheng Dong, Shaofeng Zou, and Kaiyi Ji. Scalable bilevel loss balancing for multi-task learning.arXiv preprint arXiv:2502.08585, 2025. 2

    2025

  60. [68]

    Hessian-based analysis of large batch training and robustness to adversaries

    Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. Advances in Neural Information Processing Systems, 31, 2018. 3

    2018

  61. [69]

    Unleashing the power of multi- task learning: A comprehensive survey spanning traditional, deep, and pretrained foundation model eras

    Jun Yu, Yutong Dai, Xiaokang Liu, Jin Huang, Yishan Shen, Ke Zhang, Rong Zhou, Eashan Adhikarla, Wenx- uan Ye, Yixin Liu, et al. Unleashing the power of multi- task learning: A comprehensive survey spanning traditional, deep, and pretrained foundation model eras. arXiv preprin...

    2024 arXiv

  62. [70]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Pro- cessing Systems, 33:5824–5836, 2020. 2, 6

    2020

  63. [71]

    On the convergence of multi-objective optimization under general- ized smoothness

    Qi Zhang, Peiyao Xiao, Kaiyi Ji, and Shaofeng Zou. On the convergence of multi-objective optimization under general- ized smoothness. arXiv e-prints, pages arXiv–2405, 2024. 2

    2024

  64. [72]

    A survey on multi-task learning

    Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34 (12):5586–5609, 2021. 1

    2021

  65. [73]

    A survey of multi-task learning in natural lan- guage processing: Regarding task relatedness and training methods

    Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. A survey of multi-task learning in natural lan- guage processing: Regarding task relatedness and training methods. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computati...

    2023

  66. [74]

    Surrogate gap minimization improves sharpness- aware training

    Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, James s Duncan, Ting Liu, et al. Surrogate gap minimization improves sharpness- aware training. In International Conference on Learning Representations. 2 SAMO: A Lightweight Sharpness-Aware Ap...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.