SoftReMish: A Novel Activation Function for Enhanced Convolutional Neural Networks for Visual Recognition Performance

Mustafa Bayram G\"ucen

REVIEW 4 major objections 5 minor 3 references

SoftReMish: A Novel Activation Function for Enhanced Convolutional Neural Networks for Visual Recognition Performance

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes SoftReMish, a smooth activation built from Mish with a scaling parameter, and claims it beats ReLU, Tanh, and Mish on the MNIST digit benchmark.

desk verdict SoftReMish is just a rescaled Mish, and the reported validation losses are numerically impossible under the stated softmax cross-entropy setup, so the central empirical claim fails. read the letter →

arxiv 2507.06148 v1 pith:MH3E5KLV submitted 2025-07-08 cs.CV cs.AIcs.NE

Mustafa Bayram Gücen This is my paper

classification cs.CVcs.AIcs.NE

keywords activationfunctionSoftReMishconvolutionalneuralnetworkimageclassificationMNISTLUvalidationaccuracy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a small change to the Mish activation—multiplying the input to the softplus inside the hyperbolic tangent by a scaling constant—yields a function, SoftReMish, that converges better and generalizes better than ReLU, Tanh, and Mish. The evidence is a single CNN trained on the MNIST digit dataset with each activation swapped into every trainable layer. SoftReMish is reported to reach 99.41% validation accuracy and a validation loss of $3.14 \times 10^{-8}$, the best of the four functions. If the result is right, a one-line activation change would improve image-classification models without any architectural modification.

What carries the argument

The object that carries the argument is the activation function itself: $f(x) = x \tanh(\operatorname{softplus}(\alpha x))$, where $\operatorname{softplus}(z) = \ln(1 + e^z)$ and $\alpha = 2$ in the experiments. Scaling the softplus input by $\alpha$ makes the transition from negative to positive values steeper than in Mish (the $\alpha = 1$ case), while the outer $x \tanh(\cdot)$ keeps the function smooth and non-monotonic, which the paper claims preserves gradient flow and reduces overfitting.

What would settle it

Compute the minimum possible average categorical cross-entropy on 10,000 validation images when 59 are misclassified: each error must assign probability at most 1/2 to the true class, contributing at least $\ln 2 \approx 0.69$, so the average is at least $59\ln 2/10000 \approx 4.1 \times 10^{-3}$. The reported $3.14 \times 10^{-8}$ is smaller by five orders of magnitude, so checking the actual loss function and data split used in training settles whether the claim is meaningful.

Watch

Extended reading notes

Core claim

The central claim is that SoftReMish, defined by $f(x) = x \tanh(\ln(1 + \exp(\alpha x)))$ with $\alpha = 2$, outperforms ReLU, Tanh, and Mish when used as the activation in every trainable layer of a two-convolutional-layer CNN on MNIST. The paper reports validation accuracy of 0.9941 for SoftReMish versus 0.9923 (ReLU), 0.9918 (Tanh), and 0.9907 (Mish), and validation loss of $3.137582 \times 10^{-8}$ versus $10^{-4}$-scale losses for the other functions. On this basis, the paper concludes that SoftReMish gives better convergence behavior and generalization capability, attributing the gain to a steeper activation response in high-value regions while preserving the smooth, non-monotonic, self-regularizing character of Mish.

Load-bearing premise

The load-bearing premise is that the reported validation loss and validation accuracy were computed for the same model, with the same loss function, on the same evaluation set; if that premise fails, the claimed advantage of SoftReMish is not established.

Editorial extensions

If this is right

Every layer of a CNN can switch to SoftReMish by a one-line change, leaving the architecture and training schedule untouched.
On MNIST, the reported ranking places a smooth non-monotonic activation above the piecewise-linear ReLU and the saturating Tanh.
The $\alpha$ parameter provides a tunable steepness knob that generalizes the Mish formula, so the proposal is effectively a family of activations rather than a single curve.
The paper concludes that the very low validation loss points to reduced overfitting, which would make SoftReMish relevant in accuracy-critical image tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Since SoftReMish reduces to Mish at $\alpha = 1$, a sweep over $\alpha$ (including $1$) would separate the effect of the scaling parameter from the effect of the smooth non-monotonic form itself.
The reported loss and accuracy figures are hard to reconcile under standard categorical cross-entropy, so the exact loss function, label handling, and validation split would need to be specified for the comparison to be independently checkable.
Testing the same architecture on a larger benchmark, such as CIFAR-10 or a natural-image subset, would show whether the small MNIST margin survives when inputs have color and higher resolution.
A natural extension is to make $\alpha$ learnable per layer rather than fixed at 2, so the network can adapt its activation steepness during training.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper should not be published as is. The one genuinely new element is the choice alpha=2 in the formula, and that is not enough to support a new activation function. The central empirical claim is unsupported, and the headline numbers cannot be right.

What the paper does well is modest: the CNN architecture is described clearly, the activation definitions are explicit, and the figures for the function and its derivatives are helpful. The author also cites the relevant prior work on ReLU, Tanh, and Mish, and wisely frames the study as an initial finding rather than a finished solution. That is where the credit ends.

The soft spot is structural. SoftReMish(x) = x*tanh(softplus(2x)) = Mish(2x)/2. Since scaling the input and output of a neuron can be absorbed into the neighboring layer weights, this defines the same function family as Mish. Calling it a novel activation is not accurate. The alpha=2 parameter is a free choice, evaluated only on MNIST, so any observed difference is a rescaling artifact unless shown otherwise.

The reported losses are worse than merely odd. With 10,000 validation samples and 99.41% accuracy, there are about 59 errors. Under softmax cross-entropy, each error contributes at least ln(2) ≈ 0.693 to the average loss, so the validation loss cannot be below about 4.1e-3. The paper reports 3.14e-8. Even the ReLU loss, 3.22e-4 with 99.23% accuracy, is below the corresponding lower bound. These numbers are not achievable with the stated softmax output and a cross-entropy objective. The abstract calls this a minimum training loss while the results and conclusion call it validation loss, which suggests the metric was mislabeled or the number is simply wrong.

There are also no error bars, no seeds, no hyperparameter details, and only a single run per activation. The accuracy differences are tiny: 0.9941 versus 0.9923, 0.9918, and 0.9907. Without the impossibly low loss values, there is no demonstrated advantage, and the claim of statistical or practical meaningfulness is not supported.

This paper is for a reader who wants to see one more activation-function variant added to a benchmark list, but it is not a serious scientific contribution. A desk rejection is the right outcome. If the author returns with corrected loss reporting, multiple seeds, and a proper comparison against Mish with matched weight scaling, the underlying question might be worth a short empirical note, but not in this form.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new activation function called SoftReMish, defined as f(x) = x * tanh(ln(1 + exp(alpha x))) with alpha set to 2, and evaluates it on a standard two-convolutional-layer CNN trained on MNIST. The authors compare SoftReMish against ReLU, Tanh, and Mish, reporting the highest validation accuracy (0.9941) and the lowest validation loss (3.137582e-8) for SoftReMish. From these results they conclude that SoftReMish offers better convergence behavior and generalization capability for visual recognition tasks.

Significance. If the empirical claims were correct, SoftReMish would be a simple drop-in activation function with a small but consistent accuracy improvement over ReLU and Mish on MNIST. The paper clearly describes the function, includes plots of the function and its derivatives, and uses a standard benchmark and architecture. However, the central evidence is undermined by internally inconsistent loss values, the lack of any experimental repetitions or hyperparameter details, and the algebraic fact that SoftReMish with alpha=2 is a rescaling of Mish. As presented, the contribution is not established.

major comments (4)

[Results, Table 2] The reported validation loss of 3.137582e-8 for SoftReMish is mathematically inconsistent with the reported validation accuracy of 0.9941 under the standard softmax cross-entropy objective. With 10,000 validation samples, an accuracy of 0.9941 implies about 59 misclassifications. For any misclassified sample, the true class probability cannot exceed 0.5 because the predicted class has probability at least as large, so each error contributes at least -ln(0.5) = ln(2) to the average loss; correct samples contribute nonnegative loss. The average validation loss must therefore be at least 59 * ln(2) / 10000, approximately 4.09e-3, which is more than five orders of magnitude larger than the reported value. Since the Methods state a softmax output layer, the loss function must be cross-entropy (or the paper must explicitly specify otherwise); the reported loss values cannot be valid validation losses. This invalidates the central claim of better convergence and generalization.
[Results] The entire comparison rests on a single training run with no reported random seeds, no repeated trials, no error bars, and no details of the optimizer, learning rate, batch size, number of epochs, or regularization. The accuracy differences (0.9941 vs 0.9923, 0.9918, 0.9907) are called 'significant' in the Results and Conclusion, but without an estimate of run-to-run variability these differences are within the typical noise for MNIST training with this architecture. The claim of statistical or practical significance is therefore unsupported.
[Methods, Activation Functions and SoftReMish] The proposed SoftReMish with alpha=2 is algebraically a rescaling of Mish: substituting y=2x gives f(x) = 0.5 * y * tanh(ln(1+exp(y))) = 0.5 * Mish(2x). The input scaling by a factor of 2 can be absorbed into the weights of the preceding layer, and the output scaling by 0.5 can be absorbed into the weights of the following layer. Consequently, the function class implemented by the network is the same as with Mish. The paper therefore does not introduce a new activation function class; the reported advantage may reflect a favorable hyperparameter choice for input/output scaling rather than a property of a genuinely new function.
[Abstract and Results] The paper is internally inconsistent in reporting the key metric: the Abstract states that the model performance was assessed in terms of 'minimum training loss', while the Results and Conclusion refer to the same values as 'validation loss'. Additionally, the loss function is never explicitly defined beyond the softmax output layer. The metric must be clarified and the loss values recomputed with a correct validation loss before any comparison can be interpreted.

minor comments (5)

[Methods, Materials] The sentence 'The libraries such as TensorFlow or PyTorch (depending on implementation) were used to construct and evaluate the CNN architecture' is vague and should specify the actual deep learning framework, version, and any relevant library details.
[Methods, Convolutional Neural Networks] There is a grammatical error in 'The softmax activation function have been used in this layer'; it should be 'has been used'.
[Throughout] There are several typographical and formatting issues, including 'convolutiona l' with an extra space, 'Appendîx' with a Turkish character, and inconsistent spacing around mathematical expressions. A careful proofread would improve readability.
[Results] The statement that the accuracy improvements are 'statistically and practically meaningful in high-precision tasks' is not justified by any statistical test or repeated-measure analysis; this wording should be tempered or supported.
[References] Several claims about the advantages and drawbacks of activation functions are not directly supported by citations in the text; for example, the specific claims about Mish's self-regularizing properties cite Misra (2019) but further comparative statements could benefit from additional references to the comparative studies mentioned in the introduction.

Circularity Check

1 steps flagged · score 6.0 of 10

SoftReMish is, by the paper's own formula, a scaled copy of Mish (Mish(2x)/2), so the claimed advantage over Mish reduces to the hand-set scaling α=2 rather than a novel functional form.

renaming known result [Material and Methods, 'Activation Functions and SoftReMish' (SoftReMish definition; alpha=2)]
"SoftReMish is a novel activation function proposed as a smooth and bounded alternative that combines the benefits of Mish and exponential-based transformations. Mathematically defined as 𝑓(𝑥) = 𝑥. 𝑡𝑎𝑛ℎ(𝑙𝑛(1 + 𝑒𝑥𝑝(𝛼𝑥))) ... In this study, alpha is set to 2 in order to perform the numerical calculations."

With Mish(z)=z·tanh(ln(1+e^z)) as given in the same paper, substituting z=αx gives SoftReMish(x)=x·tanh(softplus(αx))=(z/α)·tanh(softplus(z))=Mish(z)/α=Mish(αx)/α. Thus SoftReMish is not a new functional form; it is exactly Mish with input scaled by α and output scaled by 1/α. At α=2, it is Mish(2x)/2. The paper presents this rescaling as a novel activation and attributes its better validation loss and accuracy to the new function, whereas the comparison against Mish reduces by construction to the effect of the hand-set parameter α=2, evaluated on the same MNIST data. The claimed novelty and superiority are therefore a renamed and rescaled version of the very baseline it is compared with.

full rationale

The only load-bearing circular step is the renaming of a known function: SoftReMish(x)=Mish(αx)/α follows immediately from the paper's own equations. Because α is set by hand to 2 and the comparison is made on the same MNIST data, the reported performance advantage over Mish is attributable to that scaling choice, not to a new activation function. There is no self-citation chain: the Mish baseline is cited to the external work of Misra (2019), and no uniqueness theorem is imported. A separate, serious correctness problem is that the reported validation loss of 3.137582e-8 is mathematically inconsistent with softmax cross-entropy at 99.41% accuracy (roughly 59 errors would force average loss at least 59·ln(2)/10000 ≈ 4.09e-3), but this is an inconsistency or reporting error rather than a circular derivation. The circularity score reflects the central claim's reduction to a parameterized rescaling of the baseline function.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central comparison depends on one hand-set constant (alpha=2) and on unstated assumptions about the loss metric and equal hyperparameter tuning. The proposed function is a rescaled Mish, so no new function family is introduced. The MNIST-only evidence is not externally validated.

free parameters (1)

alpha = 2
Scale parameter inside softplus, hand-set in Methods ('alpha is set to 2 in order to perform the numerical calculations'). No tuning procedure or theory is given, and it is equivalent to rescaling Mish input by alpha and output by 1/alpha.

assumptions (3)

domain assumption Categorical cross-entropy is the training and validation loss.
The network ends in softmax and the paper reports 'validation loss', but the loss definition is never stated. The internal consistency of Table 2 depends on this assumption.
domain assumption MNIST results transfer to visual recognition tasks.
The conclusion generalizes from MNIST to broader visual recognition despite testing only one dataset and one small architecture.
domain assumption The compared activation functions received equal and appropriate hyperparameter tuning.
The paper keeps the architecture constant but never reports whether learning rate, epochs, batch size, or weight initialization were tuned per activation, so the comparisons may be confounded.

invented entities (1)

SoftReMish activation function
purpose: Proposed nonlinearity to improve CNN convergence and accuracy.
The only evidence is a single MNIST benchmark with a hand-set alpha. By the paper's own equation it equals a rescaled Mish, so there is no independent falsifiable handle outside this dataset.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SoftReMish: A Novel Activation Function for Enhanced Convolutional Neural Networks for Visual Recognition Performance." pith.science (2026). https://pith.science/paper/MH3E5KLV

@misc{pith2026250706148,
  author       = {Pith},
  title        = {Pith review of: SoftReMish: A Novel Activation Function for Enhanced Convolutional Neural Networks for Visual Recognition Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MH3E5KLV}},
  note         = {Machine review of arXiv:2507.06148}
}

read the original abstract

In this study, SoftReMish, a new activation function designed to improve the performance of convolutional neural networks (CNNs) in image classification tasks, is proposed. Using the MNIST dataset, a standard CNN architecture consisting of two convolutional layers, max pooling, and fully connected layers was implemented. SoftReMish was evaluated against popular activation functions including ReLU, Tanh, and Mish by replacing the activation function in all trainable layers. The model performance was assessed in terms of minimum training loss and maximum validation accuracy. Results showed that SoftReMish achieved a minimum loss (3.14e-8) and a validation accuracy (99.41%), outperforming all other functions tested. These findings demonstrate that SoftReMish offers better convergence behavior and generalization capability, making it a promising candidate for visual recognition tasks.

Figures

Figures reproduced from arXiv: 2507.06148 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

[2012]

In Advances in neural information processing systems, pages 1097–1105

Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. LeCun Y, Bottou L, Bengio Y, Haffner P, 1998 . Gradient -based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. Misra D, 2019 . Mish: A self regularized non -monotonic activation funct...

arXiv 1998
[2020]

Soft-Root-Sign Activation Function

Soft -root-sign activation function. arXiv preprint arXiv:2003.00547. APPENDİX Conference Publication This paper was originally published in the proceedings of the 6th International Applied Statistics Congress (UYIK-2025), held in Ankara, Türkiye, on May 14–16,

work page Pith review arXiv 2003
[2025]

Citation: Mustafa Bayram Gücen, SoftReMish: A Novel Activation Function for Enhanced Convolutional Neural Networks for Visual Recognition Performance, Proceedings of the 6th International Applied Statistics Congress (UYIK-2025), Ankara, Türkiye, May 14–16, 2025, pp. 657–664. Available online: https://www.uyik.org/uploads/uyik-2025-proceeding-books.pdf

work page 2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2012]

In Advances in neural information processing systems, pages 1097–1105

Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. LeCun Y, Bottou L, Bengio Y, Haffner P, 1998 . Gradient -based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. Misra D, 2019 . Mish: A self regularized non -monotonic activation funct...

arXiv 1998

[2] [2020]

Soft-Root-Sign Activation Function

Soft -root-sign activation function. arXiv preprint arXiv:2003.00547. APPENDİX Conference Publication This paper was originally published in the proceedings of the 6th International Applied Statistics Congress (UYIK-2025), held in Ankara, Türkiye, on May 14–16,

work page Pith review arXiv 2003

[3] [2025]

Citation: Mustafa Bayram Gücen, SoftReMish: A Novel Activation Function for Enhanced Convolutional Neural Networks for Visual Recognition Performance, Proceedings of the 6th International Applied Statistics Congress (UYIK-2025), Ankara, Türkiye, May 14–16, 2025, pp. 657–664. Available online: https://www.uyik.org/uploads/uyik-2025-proceeding-books.pdf

work page 2025