REVIEW 3 major objections 6 minor 29 references
Alternative Loss Function in Evaluation of Transformer Models
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Transformer models trained with a directional loss beat LSTM and buy-and-hold on six assets.
desk verdict The paper applies the authors' MADL loss to Transformers on six assets with a long walk-forward out-of-sample setup, but the loss as written has zero gradient, so the described training loop cannot update the network and the headline results are not credible as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Mean Absolute Directional Loss (MADL), defined as $\mathrm{MADL} = \frac{1}{N}\sum_{i=1}^{N} (-1)\,\mathrm{sign}(R_i\,\hat{R}_i)\,|R_i|$, where $R_i$ is the realized return and $\hat{R}_i$ the model's predicted return for interval $i$. The loss rewards a correct directional call with the absolute size of the realized move and punishes a wrong call the same way, so minimizing it is equivalent to maximizing the profit of a strategy that goes long or short according to the predicted sign. The paper uses MADL as the training objective inside a standard gradient-based deep-learning loop with the Adam optimizer, together with a walk-forward expanding-window scheme that retrains on a growing window and predicts one year ahead.
What would settle it
Compute $\partial \mathrm{MADL}/\partial \hat{R}_i$: it is zero for every $\hat{R}_i$ except at $\hat{R}_i = 0$. Then train a small network using exactly the described setup with MADL as the loss and Adam as the optimizer; if the weights remain unchanged after several epochs, the reported trading results cannot be produced by learning under the loss as written.
Extended reading notes
Core claim
The paper's central empirical claim is that Transformer models trained with MADL generate trading signals whose risk-adjusted performance exceeds both LSTM and buy-and-hold on every asset tested. Across six assets, the Transformer attains the highest values on all reported information ratios, which annualize return and penalize volatility, drawdown, and loss duration. For example, on Bitcoin the Transformer attains a drawdown-adjusted information ratio (IR**) of 5.301, versus 1.939 for LSTM and 1.287 for buy-and-hold. The paper attributes the edge to the loss function: because MADL penalizes wrong directional forecasts in proportion to the magnitude of the realized move, the model learns to prioritize large correct bets rather than minimizing squared error.
Load-bearing premise
The training procedure assumes MADL has a usable gradient with respect to the predicted return, but because the sign function has zero derivative almost everywhere, the loss as written cannot update network weights through ordinary backpropagation.
Editorial extensions
If this is right
- If the claim holds, loss-function design becomes a first-order decision in building machine-learning trading strategies: aligning the objective with the trading rule matters as much as model architecture.
- The results would support the view that Transformers, despite being designed for natural language, can be competitive on financial time series when paired with a suitable objective.
- The walk-forward, expanding-window evaluation with multi-year out-of-sample periods offers a template for reducing overfitting claims in algorithmic trading research.
- MADL could in principle be applied to other model classes and asset classes, extending the comparison beyond LSTM versus Transformer.
Reading between the lines
- A critical reading of Eq. (4) suggests that the reported training may have actually used a differentiable surrogate of the sign function, such as a straight-through estimator or a soft sign; if so, the paper's central comparison is really about that surrogate, not about MADL as written.
- The two architectures are compared under different hyperparameters (learning rate 0.5 vs 0.01, 300 vs 50 epochs, different regularization), so the 'Transformer versus LSTM' conclusion conflates architecture choice with tuning; an equal-budget comparison would isolate the loss-function effect.
- A natural follow-up experiment would train both models with a differentiable approximation of MADL and with mean squared error, holding all other settings fixed, to see whether the directional loss itself, rather than the choice of model, drives the reported outperformance.
- The policy implications about regulation discussed in the paper may be premature until the non-differentiability issue and the hyperparameter imbalance are resolved.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies the Mean Absolute Directional Loss (MADL), introduced in the authors' earlier work, as the training objective for Transformer and LSTM models that generate long/short signals on daily returns for three equities and three cryptocurrencies. A walk-forward procedure with a multi-year out-of-sample period is used, and strategy performance is summarized by annualized return, volatility, drawdown, and several information-ratio variants. The central claim, stated in the abstract and conclusions, is that Transformer models trained with MADL produce significantly better risk-adjusted returns than LSTM models and Buy&Hold in almost every asset considered.
Significance. If the central empirical claim were credible, the paper would offer useful evidence on Transformer architectures for algorithmic investment strategies, with strengths including a multi-asset design, long out-of-sample periods, walk-forward validation, and comparison against both a classical benchmark (Buy&Hold) and a strong baseline (LSTM). The paper also makes a methodological point about selecting loss functions for financial forecasting. However, the reported results depend on the described training procedure actually minimizing MADL, and that mechanism is not credible as written; moreover, the 'significantly better' language is unsupported by statistical inference, and the strategy returns appear to be gross of transaction costs despite high turnover. These issues place the main contribution in doubt.
major comments (3)
- [§3.5, Eq. (4); §3.4; Conclusions] The loss in Eq. (4) has zero gradient with respect to the predicted return almost everywhere. Because sign(R_i * R_hat_i) is piecewise constant, its derivative with respect to R_hat_i is zero except at the single point R_hat_i = 0, and abs(R_i) does not depend on R_hat_i, so ∂MADL/∂R_hat_i = 0 almost everywhere. In a standard Keras/TensorFlow backpropagation loop with the Adam optimizer, as described in §3.4, this produces zero weight updates, meaning the models are not trained by minimizing MADL. The paper notes in the Conclusions that MADL has 'the problem of its non-differentiability in certain areas', but it never specifies a surrogate gradient, a straight-through estimator, or a two-stage training procedure. The reported strategy results therefore lack a credible learning mechanism as described; the authors must either define and implement a differentiable version of the loss, provide code or pseudocode showing the actual training objective, and rerun the experiments, or substantially weaken the claims.
- [Abstract; §4, Tables 2-3] The claim that Transformer results are 'significantly better' than LSTM results is not supported by any statistical test. Tables 2 and 3 report point estimates for a single walk-forward path, with no confidence intervals, no bootstrap or Diebold-Mariano tests of return differences, and no correction for multiple comparisons across six assets. Without such procedures, the word 'significantly' is unjustified. The authors should either add appropriate significance tests for the reported performance metrics or replace 'significantly' with descriptive language such as 'consistently higher in the sample'.
- [§4, Tables 2-3; §3.6] The reported strategy performance appears to be gross of transaction costs, yet the strategies execute many trades: nTrades ranges from about 1031 to 1723 over the out-of-sample period, and the daily signal construction implies potentially frequent position changes. For the claim that these strategies yield 'abnormal risk-adjusted returns' in a realistic setting, costs are a first-order concern. The authors should state whether the reported returns are net of transaction costs and, if not, provide a sensitivity analysis under realistic cost assumptions for equities and cryptocurrencies.
minor comments (6)
- [§3.3, Table 3] The data description says Ethereum starts on '2015-08-07', but this is likely a typo for Litecoin; additionally, the table header 'EHT' should be 'ETH', and the table note contains a duplicated 'Note:'.
- [§1] 'ChatGTP' should be 'ChatGPT'.
- [§3.3, §3.4, Table 1] The description of the training window is inconsistent: §3.3 says the test set size is '252/365 days' and Table 1 says 'Train/test size 252/252', while §3.4 describes an expanding window that grows 'up to 4 years'. Please clarify whether the training window expands without bound or is capped, and state the exact units (trading days versus calendar days).
- [§3.6, Eq. (10)] In Eq. (10), the notation 'ARC' should probably be 'aRC' for consistency with Eq. (5). Also, the bullet list in §3.3 mentions 'IR, IR, IR' where three distinct information ratios are presumably meant.
- [§2] References [12] and [2] each appear twice in the sentence listing common flaws in backtesting studies; please remove the duplicates.
- [§3.5, Eq. (4)] The notation in Eq. (4) mixes 'abs(R_i)' with the text; it would be clearer to use |R_i| throughout. The behavior at R_i = 0 or R_hat_i = 0 is also undefined and should be specified.
Circularity Check
No significant circularity: Transformer/LSTM comparison rests on independent out-of-sample evaluation; MADL is an input loss taken from prior work, not a derived claim.
full rationale
The paper's central claim compares Transformer and LSTM strategies trained with MADL against Buy&Hold on six assets over multi-year out-of-sample windows. The loss function in Eq. (4) is explicitly defined and is used as the training objective for both model classes; it is not a quantity derived from the reported results and then reused as evidence. The performance metrics (aRC, aSD, MD, MLD, IR*, IR**, IR***) are standard external statistics computed from equity lines, and the benchmarks (LSTM, B&H) are independent. The only self-citation is attribution of MADL to refs [14,15], which are prior papers by the same authors; however, Eq. (4) supplies the definition and this paper's empirical evaluation is self-contained against external benchmarks, so that citation is not load-bearing in the derivation. A separate issue, the non-differentiability of sign() in Eq. (4) and its implication for gradient-based training in Section 3.4, is flagged in the conclusions as 'the problem of its non-differentiability in certain areas ([14])'; that is a correctness/mechanism concern rather than a circularity of the derivation chain, and therefore does not change the circularity score.
Assumptions & free parameters
free parameters (8)
- Transformer learning rate =
0.01
- LSTM learning rate =
0.5
- Sequence length =
4
- Transformer architecture =
2 attention layers, 4 heads, key dim 64
- LSTM architecture =
3 layers, 512/256/128 units
- Regularization strengths =
dropout 0.3 / L2 0.02 (transformer); dropout 0 / L2 1e-6 (LSTM)
- Train/test window lengths =
252/252 days
- Training epochs =
50 (transformer), 300 (LSTM)
assumptions (5)
- ad hoc to paper MADL can be minimized by gradient descent.
- domain assumption Gross returns without transaction costs are adequate for comparing strategies.
- domain assumption Walk-forward out-of-sample performance generalizes to future trading.
- domain assumption The selected assets do not suffer from survivorship bias.
- domain assumption One hyperparameter configuration is valid for all six assets.
Cite this review
Pith. "Pith review of Alternative Loss Function in Evaluation of Transformer Models." pith.science (2026). https://pith.science/paper/OBWZQ353
@misc{pith2026250716548,
author = {Pith},
title = {Pith review of: Alternative Loss Function in Evaluation of Transformer Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/OBWZQ353}},
note = {Machine review of arXiv:2507.16548}
}
read the original abstract
The proper design and architecture of testing machine learning models, especially in their application to quantitative finance problems, is crucial. The most important aspect of this process is selecting an adequate loss function for training, validation, estimation purposes, and hyperparameter tuning. Therefore, in this research, through empirical experiments on equity and cryptocurrency assets, we apply the Mean Absolute Directional Loss (MADL) function, which is more adequate for optimizing forecast-generating models used in algorithmic investment strategies. The MADL function results are compared between Transformer and LSTM models, and we show that in almost every case, Transformer results are significantly better than those obtained with LSTM.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[14]
J. Micha ´nków, P. Sakowski, and R. ´Slepaczuk. Lstm in algorithmic investment strate- gies on btc and s&p500 index. Sensors, 22(3), 2022. ISSN 1424-8220. doi: 10.3390/s22030917
-
[1]
D. H. Bailey, J. Borwein, M. Lopez de Prado, A. Salehipour, and Q. J. Zhu. Backtest overfitting in financial markets. Automated Trader, 2016
work page 2016
-
[2]
D. H. Bailey, J. Borwein, M. Lopez de Prado, and Q. J. Zhu. The probability of backtest overfitting. Journal of Computational Finance, forthcoming, 2016
work page 2016
-
[3]
J. B. Chakole, M. S. Kolhe, G. D. Mahapurush, A. Yadav, and M. P. Kurhekar. A q-learning agent for automated trading in equity stock markets. Expert Systems with Applications, 163:113761, 2021. ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa. 2020.113761
arXiv 2021
-
[4]
E. Chan. Algorithmic trading: winning strategies and their rationale, volume 625. John Wiley & Sons, 2013
work page 2013
-
[5]
E. P. Chan. Quantitative trading: how to build your own algorithmic trading business . John Wiley & Sons, 2021
work page 2021
-
[6]
F. Chollet. Deep Learning with Python, 2nd ed. Manning Publications Co., 2021
work page 2021
-
[7]
L. Di Persio and O. Honchar. Artificial neural networks architectures for stock price prediction: Comparisons and applications. International Journal of Circuits, Systems And Signal Processing, 10:403–413, Jan. 2016
work page 2016
Show all 29 references
-
[8]
Grobys, S
K. Grobys, S. Ahmed, and N. Sapkota. Technical trading rules in the cryptocurrency market. Finance Research Letters, 32:101396, 2020. ISSN 1544-6123. doi: https: //doi.org/10.1016/j.frl.2019.101396
2020
-
[9]
Hochreiter and J
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9 (8):1735–1780, 11 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735
1997 doi
-
[10]
S. Jansen. Machine Learning for Algorithmic Trading: Predictive models to extract signals from market and alternative data for systematic trading strategies with Python. Packt Publishing Ltd, 2020
2020
-
[11]
Kijewski, R
M. Kijewski, R. ´Slepaczuk, and M. Wysocki. Predicting prices of s&p 500 in- dex using classical methods and recurrent neural networks. ISD2024 Proceedings. Gda´ nsk, Poland: University of Gda´ nsk. ISBN: 978-83-972632-0-8., 2024. doi: https: //doi.org/10.62036/ISD.2024.89
2024 doi
-
[12]
Lopez de Prado
M. Lopez de Prado. What to look for in a backtest. Available at SSRN, 2013
2013
-
[13]
T. Ma, W. Wang, and Y . Chen. Attention is all you need: An interpretable transformer- MICHA ´NKÓW ET AL . A LTERNATIVE LOSS FUNCTION IN EVALUATION OF TRANSFORMER MODELS . . . based asset allocation approach. International Review of Financial Analysis , 90(C),
-
[15]
Micha ´nków, P
J. Micha ´nków, P. Sakowski, and R. ´Slepaczuk. Mean absolute directional loss as a new loss function for machine learning problems in algorithmic investment strategies. Journal of Computational Science , 81:102375, 2024. ISSN 1877-7503. doi: https: //doi.org/10.1016/j.jocs.20...
2024
-
[16]
J. Qiu, B. Wang, and C. Zhou. Forecasting stock prices with long-short term memory neural network based on attention mechanism. PLOS ONE, 15(1):1–15, 01 2020. doi: 10.1371/journal.pone.0227222
2020 doi
-
[17]
A. Raudys. Portfolio of global futures algorithmic trading strategies for best out-of- sample performance. In International Conference on Business Information Systems , pages 424–435. Springer, 2016
2016
-
[18]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023
2023
-
[19]
C. Wang, Y . Chen, S. Zhang, and Q. Zhang. Stock market index prediction using deep transformer model. Expert Systems with Applications, 208:118128, 2022. ISSN 0957-
2022
-
[20]
Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun. Transformers in time series: A survey, 2023
2023
-
[21]
Wiecki, A
T. Wiecki, A. Campbell, J. Lent, and J. Stauth. All that glitters is not gold: Comparing backtest and out-of-sample performance on a large cohort of trading algorithms. The Journal of Investing, 25(3):69–80, 2016
2016
-
[22]
J. Yang, Y . Li, X. Chen, J. Cao, and K. Jiang. Deep Learning for Stock Selection Based on High Frequency Price-V olume Data.arXiv:1911.02502 [cs, q-fin], Nov. 2019. arXiv: 1911.02502
1911 arXiv
-
[23]
A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effective for time series forecasting?, 2022
2022
-
[24]
Z. Zeng, R. Kaur, S. Siddagangappa, S. Rahimi, T. Balch, and M. Veloso. Financial time series forecasting using cnn and transformer, 2023
2023
-
[25]
Zhang, C
R. Zhang, C. Huang, W. Zhang, and S. Chen. Multi Factor Stock Selection Model Based on LSTM. International Journal of Economics and Finance, 10(8):1–36, 2018. Publisher: Canadian Center of Science and Education
2018
-
[26]
K. Zhou, W. Y . Wang, T. Hu, and C. H. Wu. Comparison of time series forecasting based on statistical arima model and lstm with attention mechanism.Journal of Physics: Con- ference Series, 1631(1):012141, sep 2020. doi: 10.1088/1742-6596/1631/1/012141
2020 doi
-
[27]
´Slepaczuk, P
R. ´Slepaczuk, P. Sakowski, and G. Zakrzewski. Investment strategies that beat the market. what can we squeeze from the market? Financial Internet Quarterly, 14(4): 36–55, 2018. doi: doi:10.2478/fiqf-2018-0026
2018 doi
-
[2023]
doi: 10.1016/j.irfa.2023.10287
2023
-
[4174]
doi: https://doi.org/10.1016/j.eswa.2022.118128
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.