Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Enhancing Cross Entropy with a Linearly Adaptive Loss Function for Optimized Classification Performance

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a one-line modification of cross entropy, $-[1-Q(x_c)]\log Q(x_c)$, improves classification accuracy on CIFAR-100.

desk verdict The loss is focal loss with gamma=1, the derivation is mathematically invalid, and the experiments are too thin to support the novelty claims. read the letter →

arxiv 2507.10574 v1 pith:PCO3U5NH submitted 2025-07-10 cs.LG cs.AIcs.NE

Jae Wan Shim This is my paper · ORCID
classification cs.LGcs.AIcs.NE
keywords linearlyadaptivecrossentropylossJeffreysdivergenceKullback-Leiblerone-hotencodingCIFAR-100ResNetfunctiondesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a drop-in replacement for the cross entropy loss in classification: the Linearly Adaptive Cross Entropy Loss, $Adp(P,Q)=-[1-Q(x_c)]\log Q(x_c)$, where $Q(x_c)$ is the predicted probability of the true class. The paper claims that this loss follows from Jeffreys divergence when labels are one-hot, and that it consistently outperforms standard cross entropy throughout training. The supporting experiment trains an 18-layer ResNet on CIFAR-100 for 200 epochs, five trials per loss, and reports a mean top-5 error of $6.2\%$ for the new loss versus $6.7\%$ for cross entropy. If this holds, a one-line change to the loss function would improve classifier accuracy at essentially no extra computational cost.

What carries the argument

Jeffreys divergence, $J(P,Q)=D(P,Q)+D(Q,P)$, is the symmetric sum of the two Kullback-Leibler divergences, and it is the object from which the proposed loss is supposedly obtained. For one-hot labels the paper keeps only the true-class term in each direction, $D(P,Q)\approx -\log Q(x_c)$ and $D(Q,P)\approx Q(x_c)\log Q(x_c)$, whose sum is $-[1-Q(x_c)]\log Q(x_c)$. The extra factor $1-Q(x_c)$ is the operative novelty: it makes the loss depend on the predicted probability of the true class and vanishes as that probability approaches one.

What would settle it

A direct check is to compute $\sum_{i\neq c} q_i \log(q_i/\varepsilon)$ with a small positive $\varepsilon$ replacing the zero probabilities of the one-hot label during CIFAR-100 training and see whether it is negligible compared with $q_c \log q_c$; alternatively, train the same ResNet using the exact Jeffreys divergence instead of the proposed loss and see whether the accuracy difference persists. If the omitted sum is substantial, or the exact divergence does not reproduce the claimed gains, Eq. (8) is the point where the derivation fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the usual one-hot cross entropy loss, $-\log Q(x_c)$, is improved by multiplying it by $1-Q(x_c)$, giving $Adp(P,Q)=-[1-Q(x_c)]\log Q(x_c)$. The author presents this formula as the simplification of Jeffreys divergence $J(P,Q)=D(P,Q)+D(Q,P)$ under one-hot labels, with $D(P,Q)$ reducing to $-\log Q(x_c)$ and $D(Q,P)$ reducing to $Q(x_c)\log Q(x_c)$. The empirical claim is that on a ResNet-based model trained on CIFAR-100, this loss reaches a mean top-5 error rate of $6.2\%$ compared with $6.7\%$ for cross entropy, averaged over five trials over the final ten epochs.

Load-bearing premise

The load-bearing premise is that, for one-hot labels, the reverse divergence can be approximated by its true-class term alone, with all false-class terms dropped; if that approximation is not valid, the proposed loss is a heuristic rather than a derived information-theoretic measure.

Editorial extensions

If this is right

  • The proposed loss can replace cross entropy in any softmax classifier without changing the network architecture.
  • Training with the new loss costs only one subtraction and one multiplication per sample beyond ordinary cross entropy.
  • If the reported advantage generalizes, models trained with the new loss on CIFAR-100 should show lower top-5 error than cross entropy-trained models across repeated runs.
  • The loss is compatible with standard training pipelines such as SGD with momentum, stepwise learning-rate decay, and data augmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the derivation's key approximation, dropping all false-class terms from $D(Q,P)$, is not checked numerically; a direct evaluation during training would show whether the claimed information-theoretic basis holds or whether the loss works for a different reason.
  • Editorial inference: because the loss is only compared with plain cross entropy on CIFAR-100, a natural next test is to compare it with other simple reweightings such as focal loss and label smoothing on several datasets, including CIFAR-10 and ImageNet.
  • Editorial inference: the factor $1-Q(x_c)$ shrinks the loss contribution of confident correct predictions, so the loss may interact with noisy labels differently than cross entropy; the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a loss function Adp(P,Q) = -(1 - Q(x_c)) log Q(x_c) as a drop-in replacement for cross entropy in classification. It claims this loss is derived from Jeffreys divergence, using the approximations D(P,Q) ≈ -log Q(x_c) and D(Q,P) ≈ Q(x_c) log Q(x_c) for one-hot labels. The authors evaluate the loss on CIFAR-100 with an 18-layer ResNet over 200 epochs, reporting lower top-5 error compared to plain cross entropy (6.2% vs 6.7%) and only two extra arithmetic operations per sample.

Significance. If the derivation were valid and the empirical result held broadly, this would be a simple, almost cost-free modification to cross entropy that improves accuracy. The manuscript has some strengths: the loss is simple and clearly specified; the experimental section reports five independent trials with standard deviations, which is a reasonable first step; and the computational overhead is negligible. However, the central theoretical claim is not sound, and the empirical evidence is limited to a single dataset and a single baseline. The paper provides no code, and the data availability statement is vague. The proposed loss might still be useful as an empirical trick, but the paper does not establish that in its current form.

major comments (3)
  1. [Linearly Adaptive Cross Entropy Loss, Eq. (8)] The approximation D(Q,P) ≈ Q(x_c) log Q(x_c) is mathematically untenable. For any distributions, D(Q,P) = Σ_i Q_i log(Q_i/P_i) ≥ 0. For one-hot P with softened false-class probabilities ε, the dropped terms Σ_{i≠c} Q_i log(Q_i/ε) diverge to +∞ as ε→0 whenever any Q_i (i≠c) is nonzero, while Q_c log Q_c ≤ 0. The approximation thus has the wrong sign and cannot be obtained by dropping terms. For example, with Q_c=0.5 and one false class with probability 0.5 and ε=10^-6, the true D(Q,P) is approximately 7.6, whereas Q_c log Q_c ≈ -0.35. Since Eq. (4) is constructed from Eqs. (7) and (8), the claimed derivation from Jeffreys divergence collapses.
  2. [Simulation results, Table 1 and Figs. 1-2] The empirical claim that the proposed loss 'consistently outperforms' cross entropy is not adequately supported. The evaluation uses only CIFAR-100, a single architecture (ResNet-18), and compares only against plain cross entropy. No significance test is reported; although the means differ (6.2 vs 6.7), the reader cannot assess whether the difference is statistically robust given only mean and standard deviation over five trials. Moreover, no comparison with standard modifications such as label smoothing or focal loss is provided, so the specific benefit of the extra term is not isolated. At minimum, a paired test across multiple datasets and a state-of-the-art baseline are needed before making a general claim.
  3. [Linearly Adaptive Cross Entropy Loss, text after Eq. (7)] The paper justifies Eq. (7) by 'using a very small value instead of zero to avoid singularities' but never specifies the value or demonstrates that the results are insensitive to it. This is not a minor issue because Eq. (8) is infinite in the one-hot limit, and the entire derivation depends on a limit that is not defined. The authors should either provide a rigorous limiting argument or abandon the claim of a derivation.
minor comments (5)
  1. [Abstract and Introduction] The manuscript contains awkward phrasing, e.g., 'One of loss functions using in machine learning' and 'the proposed one has been evaluated'. The paper would benefit from thorough copyediting.
  2. [Fig. 1 caption] The caption refers to 'orange (upper) line' and 'blue (lower) line', but colors are not distinguishable in grayscale print; use distinct line styles or symbols.
  3. [Table 1] The table labels a column 'top-5 error (%)' while the text discusses 'test accuracy'; clarify whether the numbers are error rates or accuracies. The row 'mean and std.' is also confusingly placed.
  4. [Data availability] The section is misspelled as 'Data availiability' and the statement 'available from the corresponding author on reasonable request' is not a standard data availability statement for a public dataset such as CIFAR-100.
  5. [Computation cost] The claim of 'practically the same efficiency' is not supported by any measured runtime; reporting wall-clock time or FLOPs per step would strengthen this claim.

Circularity Check

1 steps flagged · score 6.0 of 10

The claimed Jeffreys-divergence derivation of Adp(P,Q) is post hoc: Eq. (8) drops the false-class terms of D(Q,P), keeping only the term needed to reproduce the already-defined loss.

  1. other [Linearly Adaptive Cross Entropy Loss section, Eq. (8) and the following paragraph]
    "Also, Eq. (8) is obtained by dropping the false classes terms in D(Q, P )."

    Eq. (8) is the sole bridge from Jeffreys divergence to the proposed loss: with Eq. (7), Eq. (8) asserts J(P,Q) ≈ -log Q(x_c) + Q(x_c) log Q(x_c) = Adp(P,Q), the target defined in Eq. (4). But D(Q,P) = Q(x_c) log Q(x_c) + Σ_{i≠c} Q(x_i) log(Q(x_i)/P(x_i)); for one-hot labels with P(x_i)=ε for i≠c, the dropped false-class sum tends to +∞ as ε→0, so it cannot be discarded. The 'simplification' is therefore an arbitrary truncation that exactly forces the pre-specified Adp, making the information-theoretic derivation equivalent to its own conclusion by construction.

full rationale

The central theoretical claim — that Adp(P,Q) is a novel measure 'derived from information theory' — is circular/post hoc: Eq. (8) is not derived from D(Q,P); it is an ad hoc truncation chosen to reproduce the loss defined earlier in Eq. (4), and the dropped terms diverge for one-hot labels. This is a genuine reduction of the derivation to its own target. The empirical comparison against cross entropy (Figs. 1-2, Table 1) is an independent external benchmark and is not circular, so the overall circularity is partial rather than total. No load-bearing self-citation chain is present: the information-theoretic references are standard background, not the source of the problem.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central derivation relies on an unjustified truncation of the Jeffreys divergence, plus a hand-chosen epsilon; the rest is standard probability definitions. No new physical or conceptual entities are introduced beyond the loss function itself.

free parameters (1)
  • epsilon (one-hot smoothing constant) = not specified
    Mentioned in the derivation (after Eq. 7) as a small value replacing zero in P(xi) to avoid singularities; not quantified and not present in the final loss, but it is an ad hoc choice.
assumptions (3)
  • ad hoc to paper Dropping the false-class terms in D(Q,P) yields a valid approximation (Eq. 8).
    The terms dropped are Q(xi) log(Q(xi)/P(xi)) for i != c; as P(xi) approaches 0, these terms approach infinity, so they cannot be neglected. This simplification is the key step that produces the proposed loss, but it is mathematically unjustified.
  • ad hoc to paper Using a very small value instead of zero for one-hot entries avoids singularities without affecting results.
    The paper states this after Eq. (7) as a way to avoid division by zero, but the value is never specified and the subsequent derivation drops the affected terms entirely, so the justification is circular.
  • standard math Standard definitions of KL divergence and Jeffreys divergence apply as given.
    The paper uses D(P,Q) = sum P log(P/Q) and J = D(P,Q) + D(Q,P), which are standard textbook definitions, so no issue there.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Cross Entropy with a Linearly Adaptive Loss Function for Optimized Classification Performance." pith.science (2026). https://pith.science/paper/PCO3U5NH

@misc{pith2026250710574,
  author       = {Pith},
  title        = {Pith review of: Enhancing Cross Entropy with a Linearly Adaptive Loss Function for Optimized Classification Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCO3U5NH}},
  note         = {Machine review of arXiv:2507.10574}
}
read the original abstract

We propose the Linearly Adaptive Cross Entropy Loss function. This is a novel measure derived from the information theory. In comparison to the standard cross entropy loss function, the proposed one has an additional term that depends on the predicted probability of the true class. This feature serves to enhance the optimization process in classification tasks involving one-hot encoded class labels. The proposed one has been evaluated on a ResNet-based model using the CIFAR-100 dataset. Preliminary results show that the proposed one consistently outperforms the standard cross entropy loss function in terms of classification accuracy. Moreover, the proposed one maintains simplicity, achieving practically the same efficiency to the traditional cross entropy loss. These findings suggest that our approach could broaden the scope for future research into loss function design.

Figures

Figures reproduced from arXiv: 2507.10574 by the authors.

Figure 1
Figure 1. Comparison of Test Data Accuracy between Cross Entro [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Detailed View of Fig. 1 Focusing on Epochs 100 through 200. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

  1. [1]

    C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal , 27(3):379–423, 1948

    work page 1948

  2. [2]

    C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal , 27(4):623–656, 1948

    work page 1948

  3. [3]

    Kullback and R

    S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 – 86, 1951

    work page 1951

  4. [4]

    H. Jeffreys. Theory of Probability . International series of monographs on physics. Clarendon Press, 1948

    work page 1948

  5. [5]

    Deep learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016

    2016

  6. [6]

    J. A. Nelder and R. W. M. Wedderburn. Generalized linear models. Journal of the Royal Statistical Society. Series A (General) , 135(3):370–384, 1972

    work page 1972

  7. [7]

    Generalized linear models

    Peter McCullagh. Generalized linear models . Routledge, 2019

    work page 2019

  8. [8]

    Cross-entropy los s functions: Theoretical analysis and applications

    Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy los s functions: Theoretical analysis and applications. In Andreas Krause, Emma Brunskill, Kyunghyun Ch o, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , ...

    work page 2023

Show all 23 references
  1. [9]

    Uniface : Unified cross-entropy loss for deep face recognition

    Jiancan Zhou, Xi Jia, Qiufu Li, Linlin Shen, and Jinming Duan. Uniface : Unified cross-entropy loss for deep face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 20730–20739, October 2023

    2023

  2. [10]

    Li Li, Miloˇ s Doroslovaˇ cki, and Murray H. Loew. Approximating t he gradient of cross-entropy loss function. IEEE Access, 8:111626–111635, 2020. 10

    2020

  3. [11]

    An alternative cross entropy loss for learnin g-to-rank

    Sebastian Bruch. An alternative cross entropy loss for learnin g-to-rank. In Proceedings of the Web Conference 2021 , WWW ’21, page 118–126, New York, NY, USA, 2021. Association for Computing Machinery

    2021

  4. [12]

    Ad- dressing imbalance in multi-label classification using weighted cross en tropy loss function

    Mohammad Reza Rezaei-Dastjerdehei, Amirmohammad Mijani, an d Emad Fatemizadeh. Ad- dressing imbalance in multi-label classification using weighted cross en tropy loss function. In 2020 27th National and 5th International Iranian Conferenc e on Biomedical Engineering (ICBME), pa...

    2020

  5. [13]

    Generalized cross entropy loss f or training deep neural networks with noisy labels

    Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss f or training deep neural networks with noisy labels. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curr...

    2018

  6. [14]

    Taming the cross ent ropy loss

    Manuel Martinez and Rainer Stiefelhagen. Taming the cross ent ropy loss. In Thomas Brox, Andr´ es Bruhn, and Mario Fritz, editors, Pattern Recognition, pages 628–637, Cham, 2019. Springer International Publishing

    2019

  7. [15]

    Dua l cross-entropy loss for small-sample fine-grained vehicle classification

    Xiaoxu Li, Liyun Yu, Dongliang Chang, Zhanyu Ma, and Jie Cao. Dua l cross-entropy loss for small-sample fine-grained vehicle classification. IEEE Transactions on Vehicular Technology , 68(5):4204–4212, 2019

    2019

  8. [16]

    Rethinking softmax cross-entropy loss for adversarial robustness

    Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross-entropy loss for adversarial robustness. arXiv preprint arXiv:1905.10626 , 2019

    1905 arXiv

  9. [17]

    An analysis of the softmax cross entropy loss for learning-to-rank with binary relev ance

    Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Mar c Najork. An analysis of the softmax cross entropy loss for learning-to-rank with binary relev ance. In Proceedings of the 2019 ACM SIGIR international conference on theory of inform ation retrieval , pages 75–78, 2019

    2019

  10. [18]

    Mpce: a maximum probability based cross entropy loss function for n eural network classifica- tion

    Yangfan Zhou, Xin Wang, Mingchuan Zhang, Junlong Zhu, Ruijua n Zheng, and Qingtao Wu. Mpce: a maximum probability based cross entropy loss function for n eural network classifica- tion. IEEE Access, 7:146331–146341, 2019. 11

    2019

  11. [19]

    The real-world-weight cross -entropy loss function: Modeling the costs of mislabeling

    Yaoshiang Ho and Samuel Wookey. The real-world-weight cross -entropy loss function: Modeling the costs of mislabeling. IEEE Access, 8:4806–4813, 2020

    2020

  12. [20]

    Deep res idual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep res idual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pa ttern recognition, pages 770–778, 2016

    2016

  13. [21]

    Intriguing properties of neural netw orks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan B runa, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural netw orks. arXiv preprint arXiv:1312.6199, 2013

    2013 arXiv

  14. [22]

    Exp laining and harnessing adver- sarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Exp laining and harnessing adver- sarial examples. arXiv preprint arXiv:1412.6572 , 2014

    2014 arXiv

  15. [23]

    A review on multi-label learning a lgorithms

    Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning a lgorithms. IEEE trans- actions on knowledge and data engineering , 26(8):1819–1837, 2013. Additional Information – competing interests The author(s) declare no competing interests. Data availiability The datas...

    2013

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.