pith. sign in

arxiv: 2606.22068 · v2 · pith:Z42VY2DAnew · submitted 2026-06-20 · 💻 cs.LG · cs.AI

Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate

Mathew Mithra Noel , Arindam Banerjee , Yug D. Oswal , Geraldine Bessie Amali D , Venkataraman Muthiah-Nakarajan This is my paper

Pith reviewed 2026-06-26 12:29 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords robust regressionoutlier robustnessloss functionslearning rate modulationsquare root losssmooth mean absolute errorgradient descent
0
0 comments X

The pith

Alternate loss functions achieve robustness to outliers by modulating the learning rate in regression.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to establish that robustness to outliers in supervised models can be obtained by choosing loss functions that automatically reduce the learning rate on points producing large errors. These functions must be infinitely differentiable and strictly convex or quasiconvex while approximating absolute error more closely than Huber or log-cosh losses. The authors introduce Square Root Loss and Smooth Mean Absolute Error, show their superior performance across benchmarks, and supply two new robust linear regression models together with vectorized gradient updates suited to GPUs. A sympathetic reader would care because real datasets routinely contain noise that inflates prediction errors, and this method ties robustness directly to the shape of the loss.

Core claim

The paper claims that a reduction of the learning rate is achieved by using alternate loss functions that are infinitely differentiable, strictly convex or quasiconvex and more closely approximate the absolute error than Huber and log-cosh losses, with the Square Root Loss and Smooth Mean Absolute Error losses demonstrating superior performance in regression models trained on benchmarks and datasets contaminated with outliers; two new robust linear regression models and highly vectorized parameter update formulae for stochastic and batch gradient descent are also presented.

What carries the argument

The Square Root Loss and Smooth Mean Absolute Error functions that reduce the effective learning rate for outlier points while remaining infinitely differentiable and convex or quasiconvex.

If this is right

  • SRL and SMAE losses yield superior performance compared with Huber and log-cosh on a wide variety of benchmarks and datasets.
  • Two new robust linear regression models are obtained that inherit the learning-rate modulation property.
  • Highly vectorized robust parameter update formulae enable efficient stochastic and batch gradient descent on modern GPUs.
  • The loss functions remain infinitely differentiable, strictly convex or quasiconvex while approximating absolute error more closely than prior robust losses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same loss functions could be substituted into non-linear models such as neural networks to test whether the outlier robustness generalizes beyond linear regression.
  • The explicit connection between loss shape and per-example learning rate suggests a route to designing other adaptive optimizers that act directly on the loss surface.
  • Because the updates are already vectorized, scaling experiments on larger contaminated datasets would directly measure whether the robustness persists at scale.
  • If the approximation to absolute error is the key driver, replacing SRL or SMAE with other smooth absolute-error approximations should produce comparable robustness gains.

Load-bearing premise

That modulating the learning rate through these specific loss functions produces genuine robustness without hidden costs such as slower overall convergence or sensitivity to hyperparameter choices.

What would settle it

If regression models trained with SRL or SMAE fail to produce lower prediction errors than models trained with mean squared error or Huber loss on multiple datasets known to contain outliers, the robustness claim would be falsified.

Figures

Figures reproduced from arXiv: 2606.22068 by Arindam Banerjee, Geraldine Bessie Amali D, Mathew Mithra Noel, Venkataraman Muthiah-Nakarajan, Yug D. Oswal.

Figure 1
Figure 1. Figure 1: Plot of different loss functions used for regression. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Effect of using different loss functions for linear regression when the dataset has a signifi [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: California housing 3D plot References [1] Boyd, S. & Vandenberghe, L. (2004). Convex Optimization, Cambridge Univer￾sity Press, Cambridge. [2] Rockafellar, R. T. (1970). Convex Analysis, Princeton University Press, Prince￾ton, NJ. [3] Nesterov, Y. (2003). Introductory Lectures on Convex Optimization: A Basic Course, Springer Science & Business Media. [4] Bertsekas, D. P., Nedić, A. & Ozdaglar, A. E. (2003)… view at source ↗
Figure 4
Figure 4. Figure 4: Concrete strength 3D plot [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Wine quality 3D plot [5] Nocedal, J. & Wright, S. J.(2006). Numerical Optimization, Springer Science & Business Media, 2nd edition. [6] Goodfellow, Y. & Bengio, A. C. (2016). Deep Learning, MIT Press. [7] Janocha, K. & Czarnecki, W. M. (2017). On loss functions for deep neural networks in classification, arXiv preprint. arXiv:1702.05659 [8] Zhao, H. Gallo, O. Frosio, I. & Kautz, J. (2015). Loss functions f… view at source ↗
read the original abstract

Most real-world datasets used for training supervised learning models are contaminated with noisy data and outliers leading to large prediction errors. This paper proposes a new approach for achieving robustness where the learning rate is modulated by a factor that is sensitive to outliers. In this approach a reduction of the learning rate is shown to be achieved by using alternate loss functions that are infinitely differentiable, strictly convex or quasiconvex and more closely approximate the absolute error than Huber and log-cosh losses. A comparison of the performance of regression models trained with different loss functions on a wide variety of benchmarks and datasets is presented to demonstrate the superior performance of the Square Root Loss (SRL) and Smooth Mean Absolute Error (SMAE) losses proposed in this paper. Two new robust linear regression models are presented. Highly vectorized robust parameter update formulae that take advantage of modern GPUs for both stochastic and batch gradient descent are presented.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 0 minor

Summary. The paper claims that robustness to outliers can be achieved by modulating the learning rate via alternate loss functions that are infinitely differentiable, strictly convex or quasiconvex, and approximate absolute error more closely than Huber or log-cosh losses. It introduces Square Root Loss (SRL) and Smooth Mean Absolute Error (SMAE) as superior options, presents two new robust linear regression models, and supplies highly vectorized parameter-update formulae for stochastic and batch gradient descent, with benchmark comparisons asserted to demonstrate superior performance.

Significance. If the central claim holds with supporting derivations and experiments, the approach would provide a lightweight mechanism for outlier robustness that integrates directly into standard gradient-based training without explicit outlier detection or weighting schemes.

major comments (3)
  1. [Abstract] Abstract: the assertion of benchmark superiority for SRL and SMAE is stated without any experimental details, datasets, error bars, statistical tests, or epoch counts, leaving the performance claim unsupported and load-bearing for the paper's main result.
  2. [Abstract] The central mechanism—that the proposed losses modulate the learning rate for large residuals without hidden costs such as slower convergence or increased hyperparameter sensitivity—is asserted but not supported by any derivative expressions, growth-rate analysis for |r| ≫ 1, or stability checks in the provided text.
  3. [Abstract] No evidence is supplied that the vectorized update formulae preserve the claimed robustness properties under both stochastic and batch settings, which is required to substantiate the practical advantage of the two new regression models.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments. We address each major comment below, agreeing where revisions are needed to better support the claims in the abstract while noting that supporting details appear in the body of the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the assertion of benchmark superiority for SRL and SMAE is stated without any experimental details, datasets, error bars, statistical tests, or epoch counts, leaving the performance claim unsupported and load-bearing for the paper's main result.

    Authors: The abstract is intended as a concise summary. The full experimental details—including the datasets used, epoch counts, error bars from multiple independent runs, and statistical tests—are provided in the Experiments section. We will revise the abstract to include a brief reference to the experimental protocol and the statistical support for the performance claims. revision: yes

  2. Referee: [Abstract] The central mechanism—that the proposed losses modulate the learning rate for large residuals without hidden costs such as slower convergence or increased hyperparameter sensitivity—is asserted but not supported by any derivative expressions, growth-rate analysis for |r| ≫ 1, or stability checks in the provided text.

    Authors: The loss functions, their derivatives, and the resulting gradient saturation for large residuals are derived in Section 3. This saturation produces the learning-rate modulation effect without introducing new hyperparameters. We will add an explicit growth-rate analysis for |r| ≫ 1 together with a short stability discussion in the revised manuscript. revision: partial

  3. Referee: [Abstract] No evidence is supplied that the vectorized update formulae preserve the claimed robustness properties under both stochastic and batch settings, which is required to substantiate the practical advantage of the two new regression models.

    Authors: The vectorized update rules are obtained by direct vectorization of the per-example gradients of the proposed losses; the robustness properties are therefore preserved by construction in both the stochastic and batch cases. The manuscript reports results for both settings on the benchmark suite. We will add a clarifying statement in the revision that explicitly links the vectorized formulae to the inherited robustness properties. revision: yes

Circularity Check

0 steps flagged

No circularity: claims rest on new loss definitions and empirical benchmarks

full rationale

The paper defines SRL and SMAE as new loss functions with stated properties (infinitely differentiable, strictly convex/quasiconvex, closer to absolute error than Huber/log-cosh) and presents them as modulating learning rate for robustness. It then reports benchmark comparisons and supplies vectorized update formulae for two new linear regression models. No equations, fitting procedures, or self-citations are shown that would make any performance claim or robustness prediction equivalent to its own inputs by construction. The derivation chain is therefore self-contained against external benchmarks rather than internally forced.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; ledger left empty.

pith-pipeline@v0.9.1-grok · 5705 in / 904 out tokens · 24113 ms · 2026-06-26T12:29:09.561335+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 7 canonical work pages · 2 internal anchors

  1. [1]

    & Vandenberghe, L

    Boyd, S. & Vandenberghe, L. (2004). Convex Optimization, Cambridge Univer- sity Press, Cambridge

    2004

  2. [2]

    Rockafellar, R. T. (1970). Convex Analysis, Princeton University Press, Prince- ton, NJ

    1970

  3. [3]

    Nesterov, Y. (2003). Introductory Lectures on Convex Optimization: A Basic Course, Springer Science & Business Media

    2003

  4. [4]

    P., Nedić, A

    Bertsekas, D. P., Nedić, A. & Ozdaglar, A. E. (2003). Convex Analysis and Optimization, Athena Scientific, Belmont, MA, 2003. 16 Figure 4: Concrete strength 3D plot Figure 5: Wine quality 3D plot

    2003

  5. [5]

    & Wright, S

    Nocedal, J. & Wright, S. J.(2006). Numerical Optimization, Springer Science & Business Media, 2nd edition

    2006

  6. [6]

    & Bengio, A

    Goodfellow, Y. & Bengio, A. C. (2016). Deep Learning, MIT Press

    2016

  7. [7]

    On Loss Functions for Deep Neural Networks in Classification

    Janocha, K. & Czarnecki, W. M. (2017). On loss functions for deep neural networks in classification, arXiv preprint. arXiv:1702.05659

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  8. [8]

    Loss Functions for Neural Networks for Image Processing

    Zhao, H. Gallo, O. Frosio, I. & Kautz, J. (2015). Loss functions for neural networks for image processing, arXiv preprint. arXiv:1511.08861

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  9. [9]

    Amari, S.-i. (1993). Backpropagation and stochastic gradient descent method, Neurocomputing 5 (4-5), 185–196

    1993

  10. [10]

    A survey on intrusion de- tection system: Feature selection, model, performance measures, application perspective, challenges, and fu- ture research directions,

    Terven, J. Cordova-Esparza, D.-M. Romero-Gonz´alez, J.-A. Ram´ırez- Pe- draza, A. & Ch´avez-Urbiola, E. A. (2025). A comprehensive survey of loss functions and metrics in deep learning, Artificial Intelligence Review 58 (7). 17 doi:10.1007/s10462- 025-11198-7. URL http://dx.doi.org/10.1007/s10462-025- 11198-7

    work page doi:10.1007/s10462- 2025

  11. [11]

    Tian, Y. Su, D. Lauria, S. & Liu, X. (2022). Recent advances on loss functions in deep learning for computer vision, Neuro- comput. 497 (C) 129–158. doi:10.1016/j.neucom.2022.04.127. URL https://doi.org/10.1016/j.neucom.2022.04.127

    work page doi:10.1016/j.neucom.2022.04.127 2022

  12. [12]

    Wang, Q. Ma, Y. Zhao, K. & Tian, Y. (2020) A comprehensive survey of loss functions in machine learning, Annals of Data Science, 1–26

    2020

  13. [13]

    & Belkin, M

    Hui, L. & Belkin, M. (2020) Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks, arXiv preprint. arXiv:2006.07322

    work page arXiv 2020

  14. [14]

    Hastie, T. (2009). The elements of statistical learning: data mining, inference, and prediction

    2009

  15. [15]

    Brownlee, J. (2019). Probability for machine learning: Discover how to harness uncertainty with Python, Machine Learning Master

    2019

  16. [16]

    & Degen, J

    Franke, M. & Degen, J. (2023) Softmax Tutorial

    2023

  17. [17]

    Rumelhart, D. E. Durbin, R. Golden, R. & Chauvin, Y. (2013). Backpropaga- tion: The basic theory, in: Backpropagation, Psychology Press, 1–34

    2013

  18. [18]

    Khan, D. M. Yaqoob, A. Zubair, S. Khan, M. A. Ahmad, Z. & Alamri, O. A. (2021). Applications of robust regression techniques: an econometric approach, Mathematical Problems in Engineering 2021 (1) 6525079

    2021

  19. [19]

    Ginsberg, S

    Malek-Ahmadi, M. Ginsberg, S. D. Alldred, M. J. Counts, S. E. Ikonomovic, M. D. Abrahamson, E. E. Perez, S. E. & Mufson, E. J. (2024). Application of robust regression in translational neuroscience studies with non-gaussian outcome data, Frontiers in Aging Neuroscience 15, 1299451

    2024

  20. [20]

    & Panagiotakos, D

    Varin, S. & Panagiotakos, D. B. (2020). A review of robust regression in biomed- ical science research, Archives of Medical Science 16 (5), 1267–1269

    2020

  21. [21]

    Jajo, N. K. (2005). A review of robust regression and diagnostic procedures in linear regression, Acta Mathematicae Applicatae Sinica 21 (2), 209–224. 18

    2005

  22. [22]

    Mintz, D

    Meer, P. Mintz, D. Rosenfeld, A. & Kim, D. Y. (1991). Robust regression meth- ods for computer vision: A review, International journal of computer vision 6 (1), 59–70

    1991

  23. [23]

    Hodson, T. O. (2022). Root mean square error (rmse) or mean absolute error (mae): When to use them or not, Geoscientific Model Development Discussions 2022, 1–10

    2022

  24. [24]

    Willmott, C. J. & Matsuura, K. (2005). Advantages of the mean absolute er- ror (mae) over the root mean square error (rmse) in assessing average model performance, Climate research 30 (1), 79–82

    2005

  25. [25]

    Brassington, G. (2017). Mean absolute error and root mean square error: which is the better metric for assessing model performance?, in: EGU general assembly conference abstracts, 3574

    2017

  26. [26]

    & Draxler, R

    Chai, T. & Draxler, R. R. (2014). Root mean square error (rmse) or mean absolute error (mae)?–arguments against avoiding rmse in the literature, Geo- scientific model development 7 (3), 1247–1250

    2014

  27. [27]

    Qi, J. Du, J. Siniscalchi, S. M. Ma, X. & Lee, C.-H. (2020) On mean absolute error for deep neural network based vector-to-vector regression, IEEE Signal Processing Letters 27, 1485–1489

    2020

  28. [28]

    Huber, P. J. (1964). Robust Estimation of a Location Parameter, The Annals of Mathematical Statistics 35 (1), 73 – 101. doi:10.1214/aoms/1177703732. URL https://doi.org/10.1214/aoms/1177703732

    work page doi:10.1214/aoms/1177703732 1964

  29. [29]

    Huber, P. J. (1992) Robust estimation of a location parameter, in: Break- throughs in statistics: Methodology and distribution, Springer, pp. 492–518

    1992

  30. [30]

    Saleh, R. A. & Saleh, (2022). A. Statistical properties of the log-cosh loss func- tion used in machine learning, arXiv preprint. arXiv:2208.04564

    work page arXiv 2022

  31. [31]

    Witten, D

    James, G. Witten, D. Hastie, T. Tibshirani, R. & Taylor, J. (2023) Linear regression, in: An introduction to statistical learning: With applications in python, Springer, 69–134

    2023

  32. [32]

    Montgomery, D. C. Peck, E. A. & Vining, G. G. (2021) Introduction to linear regression analysis, John Wiley & Sons. 19

    2021

  33. [33]

    Fard, V. B. (2025). Housing price prediction using deep neural networks: A case study on california data, International Journal of Operations Research and Artificial Intelligence 1 (1), 47–53

    2025

  34. [34]

    Slo´nski, M. (2010). A comparison of model selection methods for compressive strength prediction of high-performance concrete using neural networks, Com- puters & Structures, 88 (21-22), 1248–1253

    2010

  35. [35]

    Cerdeira, A

    Cortez, P. Cerdeira, A. Almeida, F. Matos, T. & Reis, J. (2009). Wine quality data set—uci machine learning repository. 20

    2009