Pith. sign in

REVIEW 3 major objections 4 minor 103 references

HEM: a margin-based loss for visual categorisation tasks

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HEM, a margin-based replacement for cross-entropy loss, claims better robustness, continual learning, and segmentation at about one percent clean-accuracy cost.

desk verdict A genuinely new margin-loss variant with a broad, mostly careful empirical case; the fixed margin heuristic is the one soft spot that could bite. read the letter →

arxiv 2501.12191 v2 pith:IHRYIP3C submitted 2025-01-21 cs.LG cs.CV

Michael W. Spratling , Heiko H. Schütt This is my paper · ORCID
classification cs.LGcs.CV
keywords deeplearninglossfunctionscross-entropymargin-basedunknownclassrejectionimbalanceddatacontinualsemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes replacing cross-entropy loss with a margin-based loss called high error margin (HEM) for training deep classifiers. The central claim is that HEM is more effective than cross-entropy across unknown-class rejection, adversarial robustness, learning with imbalanced data, continual learning, and pixel-level semantic segmentation, while giving up only about one percent clean accuracy on balanced image classification. The motivation is that cross-entropy keeps increasing confidence and rewriting weights even after a sample is correctly classified, which produces overconfident predictions and catastrophic forgetting; a margin loss stops updating once the correct class is sufficiently ahead. The paper evaluates HEM against cross-entropy and against specialist losses such as LogitNorm, logit-adjusted, and DICE, and concludes that HEM is a general-purpose replacement for cross-entropy that performs well on all tested tasks.

What carries the argument

The machinery is the HEM loss, a revised multiclass margin (hinge) loss. For a sample with correct-class logit $y_l$, each wrong logit $y_i$ contributes an error $e_i = \max(0, y_i - y_l + \mu_i)$, and HEM then discards every error below the mean error for that sample and averages only the above-mean errors, with the threshold mean detached from the gradient graph. This keeps the loss focused on the largest errors at the start of training while preventing it from collapsing toward zero when only a few hard errors remain late in training. The margin $\mu = \sqrt{2000/\sum_i s_i}$, where $s_i$ is the number of training samples in class $i$, sets the required gap between the correct and wrong logits; HEM+ gives each class its own margin $\mu_i = \sqrt{2000/(n s_i)}$ so minority classes demand a larger gap. Because the loss is zero once the margin is met, training stops pushing confidence up and stops overwriting previously learned weights.

What would settle it

Re-run the HEM-versus-cross-entropy comparisons with the margin chosen separately for each dataset and architecture, for example by cross-validating $\mu$ over a small grid on a held-out slice of each training set, instead of fixing it by $\sqrt{2000/\sum_i s_i}$. If HEM's advantages on unknown-class rejection, continual learning, or segmentation shrink or reverse under per-task margin tuning, the claim that HEM is a general-purpose replacement for cross-entropy would be refuted; if the advantages persist, the fixed heuristic is not the load-bearing part.

Watch

Extended reading notes

Core claim

The central discovery is that the training failures of the older multiclass margin loss can be fixed by how errors are combined, and the fixed loss then matches cross-entropy on accuracy while being much better on robustness. Rather than averaging all per-logit errors, HEM keeps only errors above the mean and averages those, so the loss stays large while only a few hard errors remain and it does not vanish prematurely during training. The paper reports that HEM-trained networks reject unknown classes and AutoAttack perturbations far better than cross-entropy-trained networks, learn longer-tailed data and continual-learning sequences with less forgetting, and, with class-count-adjusted margins (HEM+), segment images more accurately than cross-entropy and DICE. HEM roughly ties the specialist LogitNorm loss on unknown-class rejection, trails only logit-adjusted loss on clean imbalanced accuracy, and beats all tested alternatives on the other tasks. The paper's conclusion is that HEM+ is a general-purpose loss that almost always performs better than cross-entropy and adds no computational or development cost.

Load-bearing premise

The load-bearing assumption is that one fixed rule for setting the margin, the gap between the correct class score and the wrong-class scores, works well across every dataset, architecture, and task tested; if the ideal gap differs by task, HEM's reported advantages could shrink or reverse.

Editorial extensions

If this is right

  • If the central claim holds, switching the default classification loss from cross-entropy to HEM+ should improve unknown-class rejection and adversarial-attack detection with little or no clean-accuracy cost.
  • A single general-purpose loss could replace task-specific losses: HEM roughly matches LogitNorm on unknown-class rejection, beats logit-adjusted loss on robustness and segmentation, and beats DICE on every tested task.
  • Because all training recipes in the study were optimized for cross-entropy, the roughly one-percent clean-accuracy gap is the disadvantage case; tuning hyperparameters for HEM should reduce or eliminate it.
  • For continual learning, HEM's tendency to stop updating weights for already-learned samples should reduce catastrophic forgetting relative to cross-entropy across different anti-forgetting strategies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's image experiments, HEM+ should transfer to any classifier that emits logits, including text and audio models, because the loss depends only on final logits and per-class sample counts; the paper notes the possibility but does not test it.
  • Because HEM produces zero gradient for already-correct samples, a natural and testable extension is to skip or down-weight those samples in later training epochs to cut compute; the paper lists this as future work rather than demonstrating it.
  • The margin heuristic is the main free parameter; learning the margins from data, or annealing them during training, could close the small clean-accuracy gap to cross-entropy while preserving the robustness gains, though the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript proposes high error margin (HEM) loss, a variant of multiclass margin loss in which per-sample errors below the mean are discarded and the mean of the remaining above-threshold errors is used as the loss, together with HEM+, a version with per-class margins scaled by class sample counts. The authors evaluate these losses against cross-entropy and several specialized losses on standard image classification (MNIST, CIFAR-10/100, TinyImageNet, ImageNet-1k; 14 architectures), unknown-class rejection, AutoAttack rejection, long-tailed/imbalanced learning, continual learning, and semantic segmentation. The headline claim is that HEM/HEM+ is a general-purpose replacement for cross-entropy: it loses only a small amount of clean and corrupt accuracy on balanced data while matching or improving on CE across the other tasks, and it often outperforms losses specialized for those tasks.

Significance. The empirical scope is a genuine strength: multiple architectures, five classification datasets including ImageNet-1k, several continual-learning strategies, three segmentation benchmarks, multiple trials with reported standard deviations, and public code. If the results hold, HEM+ is practically attractive because replacing CE with it is a simple change with no extra inference cost. The main caveat is that the method's only hyperparameter, the margin, is set by a heuristic calibrated on two CIFAR-10 conditions and is never validated in the large-scale or pixel-level settings where the headline OOD and adversarial-robustness advantages are claimed. The paper also overstates the segmentation result for plain HEM, which is supported only for the HEM+ variant.

major comments (3)
  1. [§4.2, Appendix B.1, Fig. 10] The margin is the only free parameter of the proposed loss, and it is chosen from a pilot on ResNet18/CIFAR10: M=2000 is fixed so that µ=sqrt(2000/Σ_i s_i) gives µ=0.2 for full CIFAR10 and µ=2 for the 50-samples-per-class version. Fig. 10(b) shows that unknown-class-rejection AUROC under MSP is strongly margin-dependent, yet the same M is used for ImageNet1k (where µ is about 0.04) and for pixel-level semantic segmentation without any reported sweep or validation. Because the paper's main advantages over CE are precisely in OOD rejection and adversarial robustness, the unsupported extrapolation of the margin heuristic is load-bearing. I would need to see margin-sensitivity experiments for at least the large-scale classification and segmentation settings, or some other demonstration that the conclusions are stable across a range of M values; the paper's own admission in §5.2 that 'there may be more optimal settings for the margins' makes this validation necessary.
  2. [Abstract, §5.4, §6] The abstract and conclusion state that HEM is more effective than CE for semantic segmentation, but the results in §5.4 show that plain HEM performs comparably to CE on segmentation and that only HEM+, whose per-class margins exploit class imbalance, is clearly superior. Since HEM and HEM+ coincide only for balanced data and the segmentation datasets are imbalanced, the supported claim is about HEM+, not about HEM. The abstract and summary should be revised to distinguish the two variants; otherwise the central 'general-purpose replacement' claim overstates the evidence.
  3. [§5, Figs. 3, 7, 8] The evaluation reports means and standard deviations over trials but no paired comparisons, confidence intervals, or effect sizes for the loss comparisons. Several central claims rest on average differences over a modest number of conditions—for example, the continual-learning results in Fig. 7 and the segmentation results in Fig. 8—and without a paired analysis the reader cannot tell whether the advantages are consistent across architectures and datasets or driven by a few favorable conditions. Please report paired differences relative to CE on the same architecture/dataset, with confidence intervals or a signed-rank test, for the main pairwise comparisons.
minor comments (4)
  1. [§4.1, Eq. (8)] Equation (8) defines the loss per sample, but the batch-level aggregation and the handling of the case where no per-sample error exceeds the mean are not specified; please state the exact implementation (including any numerical safeguard for an empty numerator/denominator), since this affects the gradients.
  2. [§5.1.1, §A.3] There are several typos that should be corrected: 'using using' and 'serveral' in §5.1.1, and 'PermutedMIST' in the appendix/Figure 7 discussion.
  3. [Table 1] The reported MM and HEM values for the third and fourth rows do not appear to follow directly from Eqs. (6)–(8) with the stated µ=0.5; please verify the numerical examples or clarify how they were computed.
  4. [§5.2, §5.4] For HEM+, the definition of s_i in the margin formula is clear for image classification but is not explicitly defined for pixel-level segmentation; please state whether s_i denotes the number of pixels of class i in the training set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HEM is an empirical benchmark comparison, and the margin heuristic is pilot-tuned rather than fitted to the claimed outcomes.

full rationale

The paper proposes a new loss function and evaluates it empirically; it does not attempt to derive its benchmark results from the loss definition. The margin heuristic µ = sqrt(2000 / Σ s_i) is selected in preliminary experiments on ResNet18/CIFAR10 (Appendix B.1), but the headline comparisons on unknown class rejection, adversarial robustness, imbalanced learning, continual learning, and semantic segmentation are measured on held-out benchmarks with training hyperparameters taken from prior literature. No equation in the paper reduces a predicted result to the fitted margin by construction, and the paper does not claim to predict the margin's transfer; it reports observed performance. The self-citation to Spratling (2023) supplies the DAR evaluation metric and the term 'unknown class rejection', but this citation does not establish HEM's performance and is not a load-bearing premise for the central empirical claims. The paper even reports conditions where CE outperforms HEM, so the conclusion is not forced by definition or by a self-citation chain. The margin extrapolation to ImageNet and segmentation is a legitimate robustness concern but belongs to correctness risk, not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper contributes an empirical loss function, not a derivation. It rests on a tuned constant M=2000 and on heuristic assumptions about training dynamics and hyperparameter transfer. No new physical or formal entities are introduced.

free parameters (1)
  • M (margin constant) = 2000
    Fixed in the HEM margin formula mu = sqrt(M / sum_i s_i), selected from pilot experiments in Appendix B.1; no theoretical derivation, and it controls the size of the decision margin and OOD confidence.
assumptions (4)
  • domain assumption A margin-based loss that stops updating once the margin is satisfied will reduce over-confidence and catastrophic forgetting.
    Used to motivate HEM in Section 2.1 and 2.2; the paper does not prove this behavior, though the empirical results are consistent with it.
  • domain assumption The mean-error threshold with detached mean (Eq. 8) yields gradients that train DNNs effectively despite non-smoothness.
    No convergence proof is given; this is validated only empirically through the experiments.
  • domain assumption Hyperparameters from the literature, optimized for CE loss, are a fair testing ground for comparing losses.
    The entire evaluation protocol rests on the assumption that using CE-optimized recipes is a neutral or CE-favoring setting.
  • ad hoc to paper Pilot results on ResNet18/CIFAR10 generalize to other architectures and datasets.
    The margin constant M and the LogitNorm temperature are chosen on this single condition, then applied everywhere.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HEM: a margin-based loss for visual categorisation tasks." pith.science (2026). https://pith.science/paper/IHRYIP3C

@misc{pith2026250112191,
  author       = {Pith},
  title        = {Pith review of: HEM: a margin-based loss for visual categorisation tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IHRYIP3C}},
  note         = {Machine review of arXiv:2501.12191}
}
read the original abstract

Training deep neural networks (DNNs) on classification tasks can be performed with a number of different losses, but cross-entropy (CE) loss is the de-facto standard. Here, we propose an alternative loss, high error margin (HEM), which is a margin based loss modified to improve the training dynamics of neural networks. HEM loss is evaluated extensively using a wide range of DNN architectures and benchmark datasets with all experimental settings and training hyper-parameters taken from the literature, and hence, optimised for CE loss. HEM is found to be more effective than CE loss across a range of image-based tasks: unknown class rejection, adversarial robustness, learning with imbalanced data, continual learning, and semantic segmentation (a pixel-wise classification task). HEM is inferior to CE only in terms of clean and corrupt image classification with balanced training data, and this difference is small. We also compare HEM to specialised losses that have previously been proposed to improve performance for specific vision tasks. LogitNorm, a loss achieving state-of-the-art performance on unknown class rejection, produces similar performance to HEM for this task, but is much poorer for continual learning and semantic segmentation. Logit-adjusted loss, designed for imbalanced data, has superior results to HEM for that task, but performs worse on unknown class rejection and semantic segmentation. DICE, a popular loss for semantic segmentation, is inferior to HEM loss on all tasks, including semantic segmentation. Overall, HEM is competitive with the best alternative loss for all the tasks we have used and performs better than all other tested losses in terms of rejecting out-of-distribution examples, for continual learning, and by a substantial margin for semantic segmentation.

Figures

Figures reproduced from arXiv: 2501.12191 by the authors.

Figure 1
Figure 1. Summary results for all the different tasks considered, comparing the average performance of cross-entropy (CE) loss to each of the alternative losses that have been evaluated: LogitNorm (LN), Logit-adjusted (LA), DICE, multi-class margin (MM), high error margin (HEM), and high error margin with adjusted margins (HEM+). Results are averaged using the arithmetic mean over all other factors that were varied in the exp… view at source ↗
Figure 2
Figure 2. Results when learning with standard data-sets and testing with clean and corrupt images. (a) and (b) directly com￾pare the performance produced by HEM and cross-entropy (CE) losses when used to train networks with MNIST, CIFAR10, CIFAR100, TinyImageNet (TIN), and ImageNet1k (IN) using three different network architectures for each data-set. For each data-set the size of the marker used corresponds to the size of the… view at source ↗
Figure 3
Figure 3. Results when learning with standard data-sets and testing on unknown and adversarial images. This figure has an identical format to [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Prediction confidence after learning with standard data-sets. Results are for WRN22-10 networks trained on CIFAR10. Each graph shows histograms of the number of samples classified with different levels of prediction confidence (MSP). Separate histograms are shown for t…
Figure 5
Figure 5. Figure 5: Results when learning with imbalanced data-sets and testing on clean and corrupt images. (a) and (b) directly compare the performance produced by HEM and cross-entropy (CE) losses when used to train networks with long-tailed (LT) CIFAR10 and CIFAR100 each with imbalanc…
Figure 6
Figure 6. Figure 6: Results when learning with imbalanced data-sets and testing on unknown and adversarial images. This figure has an identical format to [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Results for continual learning. (a) to (d) directly compare the performance produced by HEM and cross-entropy (CE) losses when applied to the PermutedMNIST, SplitMNIST, SplitCIFAR10 and SplitCIFAR100 tasks. Results above the diagonal are conditions where better perform…
Figure 8
Figure 8. Figure 8: Results for semantic segmentation. (a), (b), and (c) directly compare the performance produced by HEM+ and cross-entropy (CE) losses when applied to the CamVid, Cityscapes, and SBD benchmarks. Results above the diagonal are conditions where better performance was obtai…
Figure 9
Figure 9. Figure 9: The effects of the loss hyper-parameter on LogitNorm (LN) and Multi-class Margin (MM) losses. Results for LN are shown in (a) and (b). Results for MM are shown in (c) and (d). Performance metrics are averaged over five trials performed with each parameter value, and th…
Figure 10
Figure 10. Figure 10: The effects of the HEM loss margin. Performance metrics are averaged over five trials performed with each margin value, and the error bars show the standard deviation recorded across these five trials. Experiments were performed using the ResNet18 architecture. Result…
Figure 11
Figure 11. Figure 11: Learning dynamics for WRN22-10 networks trained on CIFAR10 with (a) cross-entropy (CE) loss, (b) high error margin (HEM) loss. Each graph shows the change during training of the loss, the mean percentage AUROC averaged over seven data-sets containing unknown classes (…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

103 extracted references · 17 canonical work pages

  1. [1]

    and Mian, A

    Akhtar, N. and Mian, A. (2018). Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access , 6:14410--30. doi:10.1109/ACCESS.2018.2807385

    arXiv 2018

  2. [2]

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man \'e , D. (2016). Concrete problems in AI safety. arXiv:1606.06565

    arXiv 2016

  3. [3]

    Awasthi, P., Mao, A., Mohri, M., and Zhong, Y. (2023). Theoretically grounded loss functions and algorithms for adversarial robustness. In Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 206 of Proceedings of Machine Learning Research , pages 10077...

    2023

  4. [4]

    Azad, R., Heidary, M., Yilmaz, K., H \"u ttemann, M., Karimijafarbigloo, S., Wu, Y., Schmeink, A., and Merhof, D. (2023). Loss functions in the era of semantic segmentation: A survey and outlook. arXiv:2312.05391

    arXiv 2023

  5. [5]

    Badrinarayanan, V., Kendall, A., and Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. doi:10.1109/TPAMI.2016.2644615

    arXiv 2017

  6. [6]

    Bitterwolf, J., Meinke, A., Augustin, M., and Hein, M. (2022). Breaking down out-of-distribution detection: Many methods based on OOD training data estimate a combination of the same core quantities. In Proceedings of the International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 2041--74. arXiv:2206.09880

    arXiv 2022

  7. [7]

    S., Malhotra, G., Dujmovi \'c , M., Montero, M

    Bowers, J. S., Malhotra, G., Dujmovi \'c , M., Montero, M. L., Tsvetkov, C., Biscione, V., Puebla, G., Adolfi, F., Hummel, J. E., Heaton, R. F., Evans, B. D., Mitchell, J., , and Blything, R. (2023). Deep problems with neural network models of human vision. Behavioral and Brain Sciences , in press. doi:10.1017/S0140525X22002813

    doi:10.1017/s0140525x22002813 2023

  8. [8]

    J., Fauqueur, J., and Cipolla, R

    Brostow, G. J., Fauqueur, J., and Cipolla, R. (2009). Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters , 30(2):88--97. doi:10.1016/j.patrec.2008.04.005

    doi:10.1016/j.patrec.2008.04.005 2009

Show all 103 references
  1. [9]

    Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. (2019). Learning imbalanced datasets with label-distribution-aware margin loss. In Proceedings of the Conference on Advances in Neural Information Processing Systems , pages 1567--78, Red Hook, NY, USA. Curran Associates I...

    2019 arXiv

  2. [10]

    Carta, A., Pellegrini, L., Cossu, A., Hemati, H., and Lomonaco, V. (2023). Avalanche: A pytorch library for deep continual learning. Journal of Machine Learning Research , 24(363):1--6. http://jmlr.org/papers/v24/23-0130.html

    2023

  3. [11]

    K., Torr, P

    Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H. S., and Ranzato, M. (2019). On tiny episodic memories in continual learning. arXiv:1902.10486

    2019 arXiv

  4. [12]

    Chen, Y., Lin, Y., Xu, R., and Vela, P. A. (2023). WDiscOOD : Out-of-distribution detection via whitened linear discriminant analysis. In Proceedings of the International Conference on Computer Vision . arXiv:2303.07543

    2023 arXiv

  5. [13]

    Cheng, Z., Zhu, F., Zhang, X.-Y., and Liu, C.-L. (2023). Average of pruning: Improving performance and stability of out-of-distribution detection. arXiv:2303.01201

    2023 arXiv

  6. [14]

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , and Vedaldi, A. (2014). Describing textures in the wild. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

    2014

  7. [15]

    Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., and Ha, D. (2018). Deep learning for classical japanese literature. In Proceedings of the Conference on Advances in Neural Information Processing Systems , Workshop on Machine Learning for Creativity and Des...

    2018 arXiv

  8. [16]

    Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016). The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recogn...

    2016 arXiv

  9. [17]

    and Singer, Y

    Crammer, K. and Singer, Y. (2002). On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research , 2:265--92

    2002

  10. [18]

    and Hein, M

    Croce, F. and Hein, M. (2020). Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Proceedings of the International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 2206--16. arXiv:...

    2020 arXiv

  11. [19]

    Cui, J., Tian, Z., Zhong, Z., Qi, X., Yu, B., and Zhang, H. (2024). Decoupled kullback-leibler divergence loss. In Proceedings of the Conference on Advances in Neural Information Processing Systems . arXiv:2305.13948

    2024 arXiv

  12. [20]

    Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. (2019). Class-balanced loss based on effective number of samples. arXiv:1901.05555

    2019 arXiv

  13. [21]

    N., Bellinger, C., Roberts, M., and Chawla, N

    Dablain, D., Jacobson, K. N., Bellinger, C., Roberts, M., and Chawla, N. V. (2024). Understanding CNN fragility when learning with imbalanced data. Machine Learning , 113:4785--810. doi:10.1007/s10994-023-06326-9

    2024 doi

  14. [22]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the ...

    2020 arXiv

  15. [23]

    French, R. M. (2003). Catastrophic forgetting in connectionist networks. In Nadel, L., editor, Encyclopedia of Cognitive Science , volume 1, pages 431--5. Nature Publishing Group, London, UK

    2003

  16. [24]

    Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665--73. doi:10.1038/s42256-020-00257-z. arXiv:2004.07780

    2020 arXiv

  17. [25]

    Geirhos, R., Temme, C. R. M., Rauber, J., Sch \"u tt, H. H., Bethge, M., and Wichmann, F. A. (2018). Generalisation in humans and deep neural networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems . arXiv:1808.08750

    2018 arXiv

  18. [26]

    Hariharan, B., Arbel \'a ez, P., Bourdev, L., Maji, S., and Malik, J. (2011). Semantic contours from inverse detectors. In Proceedings of the International Conference on Computer Vision

    2011

  19. [27]

    He, K., Zhang, X., Ren, S., and Sun, J. (2016a). Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 770--8. arXiv:1512.03385

    2016 arXiv

  20. [28]

    He, K., Zhang, X., Ren, S., and Sun, J. (2016b). Identity mappings in deep residual networks. In Proceedings of the European Conference on Computer Vision . arXiv:1603.05027

    2016 arXiv

  21. [29]

    He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., and Li, M. (2018). Bag of tricks for image classification with convolutional neural networks. arXiv:1812.01187

    2018 arXiv

  22. [30]

    Heaven, D. (2019). Why deep-learning AI s are so easy to fool. Nature , 574:163--6

    2019

  23. [31]

    Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., , and Song, D. (2022a). Scaling out-of-distribution detection for real-world settings. In Proceedings of the International Conference on Machine Learning . arXiv:1911.11132

    2022 arXiv

  24. [32]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. G. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations . arXiv:1903.12261

    2019 arXiv

  25. [33]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Representations . arXiv:1610.02136

    2017 arXiv

  26. [34]

    Hendrycks, D., Mazeika, M., and Dietterich, T. (2019). Deep anomaly detection with outlier exposure. In Proceedings of the International Conference on Learning Representations . arXiv:1812.04606

    2019 arXiv

  27. [35]

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. (2021). Natural adversarial examples. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . arXiv:1907.07174

    2021 arXiv

  28. [36]

    Hendrycks, D., Zou, A., Mazeika, M., Tang, L., Li, B., Song, D., and Steinhardt, J. (2022b). Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . arXiv:2112.05135

    2022 arXiv

  29. [37]

    V., and Adam, H

    Howard, A., Sandler, M., Chu, G., Chen, L., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q. V., and Adam, H. (2019). Searching for mobilenet V3 . In Proceedings of the International Conference on Computer Vision . arXiv:1905.02244

    2019 arXiv

  30. [38]

    G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H

    Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

    2017 arXiv

  31. [39]

    Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . arXiv:1608.06993

    2017 arXiv

  32. [40]

    Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. (2019). Adversarial examples are not bugs, they are features. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R., editors, Proceedings of the Conference on ...

    2019 arXiv

  33. [41]

    Irandoust, S., Durand, T., Rakhmangulova, Y., Zi, W., and Hajimirsadeghi, H. (2022). Training a vision transformer from scratch in less than 24 hours with 1 GPU . In Proceedings of the Conference on Advances in Neural Information Processing Systems , Has it Trained Yet? Worksh...

    2022 arXiv

  34. [42]

    Jung, H., Ju, J., Jung, M., and Kim, J. (2016). Less-forgetting learning in deep neural networks. arXiv:1607.00122

    2016 arXiv

  35. [43]

    Kanai, S., Yamaguchi, S., Yamada, M., Takahashi, H., Ohno, K., and Ida, Y. (2023). One-vs-the-rest loss to focus on important samples in adversarial training. In Proceedings of the International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Res...

    2023 arXiv

  36. [44]

    Kannan, H., Kurakin, A., and Goodfellow, I. (2018). Adversarial logit pairing. arXiv:1803.06373

    2018 arXiv

  37. [45]

    Kim, H. (2021). Torchattacks: a pytorch repository for adversarial attacks. arXiv:2010.01950

    2021 arXiv

  38. [46]

    Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations . arXiv:1412.6980

    2015 arXiv

  39. [47]

    Kirchheim, K., Filax, M., and Ortmeier, F. (2022). Pytorch- OOD : A library for out-of-distribution detection based on pytorch. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , Workshops, pages 4351--60. doi:10.1109/CVPRW56347...

    2022

  40. [48]

    Kirillov, A., Girshick, R., He, K., and Doll \'a r, P. (2019). Panoptic feature pyramid networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . arXiv:1901.02446

    2019 arXiv

  41. [49]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of...

    2017 doi

  42. [50]

    Kumano, S., Kera, H., and Yamasaki, T. (2022). Are DNNs fooled by extremely unrecognizable images? arXiv:2012.03843

    2022 arXiv

  43. [51]

    M., Salakhutdinov, R., and Tenenbaum, J

    Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. (2015). Human-level concept learning through probabilistic program induction. Science , 350(6266):1332--8. doi:10.1126/science.aab3050

    2015 doi

  44. [52]

    LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278--324. doi:10.1109/5.726791

    1998 doi

  45. [53]

    Lee, J., Prabhushankar, M., and AlRegib, G. (2022). Gradient-based adversarial and out-of-distribution detection. In Proceedings of the International Conference on Machine Learning , Workshop on New Frontiers in Adversarial Machine Learning. arXiv:2206.08255

    2022 arXiv

  46. [54]

    and Hoiem, D

    Li, Z. and Hoiem, D. (2016). Learning without forgetting. In Proceedings of the European Conference on Computer Vision , pages 614--29. arXiv:1606.09282

    2016 arXiv

  47. [55]

    Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., Wei, F., and Guo, B. (2022). Swin transformer V2 : Scaling up capacity and resolution. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

    2022

  48. [56]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the International Conference on Computer Vision , pages 10012--22. arXiv:2103.14030

    2021 arXiv

  49. [57]

    L., Lange, M

    Lomonaco, V., Pellegrini, L., Cossu, A., Carta, A., Graffieti, G., Hayes, T. L., Lange, M. D., Masana, M., Pomponi, J., van de Ven, G., Mundt, M., She, Q., Cooper, K., Forest, J., Belouadah, E., Calderara, S., Parisi, G. I., Cuzzolin, F., Tolias, A., Scardapane, S., Antiga, L....

    2021 arXiv

  50. [58]

    Ma, J., Chen, J., Ng, M., Huang, R., Li, Y., Li, C., Yang, X., and Martel, A. L. (2021). Loss odyssey in medical image segmentation. Medical Image Analysis , 71:102035. doi:10.1016/j.media.2021.102035

    2021

  51. [59]

    Mao, C., Zhong, Z., Yang, J., Vondrick, C., and Ray, B. (2019). Metric learning for adversarial robustness. In Proceedings of the Conference on Advances in Neural Information Processing Systems . arXiv:1909.00900

    2019 arXiv

  52. [60]

    Marcus, G. (2020). The next decade in AI : Four steps towards robust artificial intelligence. arXiv:2002.06177

    2020 arXiv

  53. [61]

    K., Jayasumana, S., Rawat, A

    Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. (2021). Long-tail learning via logit adjustment. In Proceedings of the International Conference on Learning Representations . arXiv:2007.07314

    2021 arXiv

  54. [62]

    Milletari, F., Navab, N., and Ahmadi, S.-A. (2016). V -net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV) , pages 565--71. doi:10.1109/3DV.2016.79

    2016 doi

  55. [63]

    Ming, Y., Sun, Y., Dia, O., and Li, Y. (2023). How to exploit hyperspherical embeddings for out-of-distribution detection? In Proceedings of the International Conference on Learning Representations . https://openreview.net/forum?id=aEFaE0W5pAd

    2023

  56. [64]

    Mohseni, S., Pitale, M., Yadawa, J., and Wang, Z. (2020). Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5216--23. doi:10.1609/aaai.v34i04.5966

    2020 doi

  57. [65]

    and Gilmer, J

    Mu, N. and Gilmer, J. (2019). MNIST-C: A robustness benchmark for computer vision. arXiv:1906.02337

    2019 arXiv

  58. [66]

    Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. In Proceedings of the Conference on Advances in Neural Information Processing Systems , Workshop on Deep Learning and Unsupervised...

    2011

  59. [67]

    Nguyen, A., Yosinski, J., and Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . arXiv:1412.1897

    2015 arXiv

  60. [68]

    Pang, T., Xu, K., Dong, Y., Du, C., Chen, N., and Zhu, J. (2020). Rethinking softmax cross-entropy loss for adversarial robustness. In Proceedings of the International Conference on Learning Representations . arXiv:1905.10626

    2020 arXiv

  61. [69]

    Pang, T., Yang, X., Dong, Y., Su, H., and Zhu, J. (2021). Bag of tricks for adversarial training. In Proceedings of the International Conference on Learning Representations . arXiv:2010.00467

    2021 arXiv

  62. [70]

    K., Wang, Z., Kan, P., Fernandes, E., and Jha, S

    Panum, T. K., Wang, Z., Kan, P., Fernandes, E., and Jha, S. (2021). Exploring adversarial robustness of deep metric learning. arXiv:2102.07265

    2021 arXiv

  63. [71]

    B., and Swami, A

    Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A. (2016). The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy . arXiv:1511.07528

    2016 arXiv

  64. [72]

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: A...

    2019 arXiv

  65. [73]

    Ren, J., Yu, C., Sheng, S., Ma, X., Zhao, H., Yi, S., and Li, H. (2020). Balanced meta-softmax for long-tailed visual recognition. In Proceedings of the Conference on Advances in Neural Information Processing Systems . arXiv:2007.10740

    2020 arXiv

  66. [74]

    Rice, L., Wong, E., and Kolter, J. Z. (2020). Overfitting in adversarially robust deep learning. arXiv:2002.11569

    2020 arXiv

  67. [75]

    Robins, A. (1993). Catastrophic forgetting in neural networks: the role of rehearsal mechanisms. In Proceedings of the First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems , pages 65--8. IEEE

    1993

  68. [76]

    D., Jalaian, B., and Jha, S

    Roy, A., Cobb, A., Bastian, N. D., Jalaian, B., and Jha, S. (2022). Runtime monitoring of deep neural networks using top-down context models inspired by predictive processing and dual process theory. In Proceedings of the AAAI Conference on Artificial Intelligence

    2022

  69. [77]

    and Wichert, A

    Sa-Couto, L. and Wichert, A. (2021). Simple Convolutional-Based Models: Are They Learning the Task or the Data? Neural Computation , 33(12):3334--50. doi:10.1162/neco\_a\_01446

    2021 doi

  70. [78]

    Serre, T. (2019). Deep learning: The good, the bad, and the ugly. Annual Review of Vision Science , 5(1):399--426. doi:10.1146/annurev-vision-091718-014951

    2019 doi

  71. [79]

    Spratling, M. W. (2023). A comprehensive assessment benchmark for rigorously evaluating deep learning image classifiers. arXiv:2308.04137

    2023 arXiv

  72. [80]

    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 1--9. arXiv:1409.4842

    2015 arXiv

  73. [81]

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 2818--26. arXiv:1512.00567

    2016 arXiv

  74. [82]

    J., and Shin, J

    Tack, J., Yu, S., Jeong, J., Kim, M., Hwang, S. J., and Shin, J. (2022). Consistency regularization for adversarial robustness. In Proceedings of the AAAI Conference on Artificial Intelligence . arXiv:2103.04623

    2022 arXiv

  75. [83]

    Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., and Yan, J. (2020). Equalization loss for long-tailed object recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . arXiv:2003.05176

    2020 arXiv

  76. [84]

    and Le, Q

    Tan, M. and Le, Q. V. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning . arXiv:1905.11946

    2019 arXiv

  77. [85]

    M., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S

    Van Horn , G., Aodha, O. M., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. (2018). The inaturalist species classification and detection dataset. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition ...

    2018 arXiv

  78. [86]

    Varrette, S., Cartiaux, H., Peter, S., Kieffer, E., Valette, T., and Olloh, A. (2022). Management of an Academic HPC & Research Computing Facility: The ULHPC Experience 2.0 . In Proc. of the 6th ACM High Performance Computing and Cluster Technologies Conf. (HPCCT 2022) , Fuzho...

    2022

  79. [87]

    Vaze, S., Han, K., Vedaldi, A., and Zisserman, A. (2022). Open-set recognition: a good closed-set classifier is all you need? In Proceedings of the International Conference on Learning Representations . arXiv:2110.06207

    2022 arXiv

  80. [88]

    Wei, H., Xie, R., Cheng, H., Feng, L., An, B., and Li, Y. (2022). Mitigating neural network overconfidence with logit normalization. In Proceedings of the International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research . arXiv:2205.09310

    2022 arXiv

  81. [89]

    Wightman, R., Touvron, H., and J \'e gou, H. (2021). Resnet strikes back: An improved training procedure in timm. arXiv:2110.00476

    2021 arXiv

  82. [90]

    Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion- MNIST : a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747

    2017 arXiv

  83. [91]

    Xie, S., Girshick, R., Doll \'a r, P., Tu, Z., and He, K. (2017). Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . arXiv:1611.05431

    2017 arXiv

  84. [92]

    Xu-Darme, R., Girard-Satabin, J., Hond, D., Incorvaia, G., and Chihani, Z. (2023). Interpretable out-of-distribution detection using pattern identification. https://hal-cea.archives-ouvertes.fr/cea-03951966

    2023

  85. [93]

    Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., Du, X., Zhou, K., Zhang, W., Hendrycks, D., Li, Y., and Liu, Z. (2022). OpenOOD : Benchmarking generalized out-of-distribution detection. In Proceedings of the Conference on Advance...

    2022 arXiv

  86. [94]

    Yang, T., Huang, Y., Xie, Y., Liu, J., and Wang, S. (2023). MixOOD : Improving out-of-distribution detection with enhanced data mixup. ACM Transactions on Multimedia Computing, Communications, and Applications . doi:10.1145/3578935

    2023 doi

  87. [95]

    and Xu, C.-Z

    Yu, Y. and Xu, C.-Z. (2023). Efficient loss function by minimizing the detrimental effect of floating-point errors on gradient-based attacks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 4056--66. doi:10.1109/CVPR527...

    2023

  88. [96]

    Yuille, A. L. and Liu, C. (2021). Deep nets: What have they ever done for vision? International Journal of Computer Vision , 129:781--802. doi:10.1007/s11263-020-01405-z

    2021 doi

  89. [97]

    and Komodakis, N

    Zagoruyko, S. and Komodakis, N. (2016). Wide residual networks. In Proceedings of the British Machine Vision Conference . arXiv:1605.07146

    2016 arXiv

  90. [98]

    Zenke, F., Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence. In Proceedings of the International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 3987--95. arXiv:1703.04200

    2017 arXiv

  91. [99]

    P., Ghaoui, L

    Zhang, H., Yu, Y., Jiao, J., Xing, E. P., Ghaoui, L. E., and Jordan, M. I. (2019). Theoretically principled trade-off between robustness and accuracy. In Proceedings of the International Conference on Machine Learning . arXiv:1901.08573

    2019 arXiv

  92. [100]

    Zhang, L. H. and Ranganath, R. (2023). Robustness to spurious correlations improves semantic out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence . arXiv:2302.04132

    2023 arXiv

  93. [101]

    Zhao, L., Teng, Y., and Wang, L. (2024). Logit normalization for long-tail object detection. International Journal of Computer Vision , 132:2114--34. doi:10.1007/s11263-023-01971-y

    2024 doi

  94. [102]

    Zhu, Q., Zheng, G., and Yan, Y. (2023). Effective out-of-distribution detection in classifier based on PEDCC -loss. Neural Processing Letters , 55:1937--49. doi:10.1007/s11063-022-10970-y

    2023 doi

  95. [103]

    Zhu, Y., Chen, Y., Li, X., Zhang, R., Xue, H., Tian, X., Jiang, R., Zheng, B., and Chen, Y. (2024). Rethinking out-of-distribution detection from a human-centric perspective. International Journal of Computer Vision , 132:4633--50. doi:10.1007/s11263-024-02099-3. arXiv:2211.16778

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.