Pith. sign in

REVIEW 3 major objections 6 minor 29 references

Alternative Loss Function in Evaluation of Transformer Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Transformer models trained with a directional loss beat LSTM and buy-and-hold on six assets.

desk verdict The paper applies the authors' MADL loss to Transformers on six assets with a long walk-forward out-of-sample setup, but the loss as written has zero gradient, so the described training loop cannot update the network and the headline results are not credible as reported. read the letter →

arxiv 2507.16548 v2 pith:OBWZQ353 submitted 2025-07-22 q-fin.CP cs.LGq-fin.TR

classification q-fin.CPcs.LGq-fin.TR
keywords DeepLearningNeuralNetworksLSTMAlgorithmicInvestmentStrategiesLossFunctionTransformerMeanAbsoluteDirectionalWalk-forwardvalidation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the choice of training loss function is as important as model architecture in machine-learning trading systems. The paper applies a recently proposed Mean Absolute Directional Loss (MADL) to Transformer models, and reports that the resulting strategies outperform both LSTM models and a simple buy-and-hold benchmark on six assets: three equities (JP Morgan, S&P 500, Exxon Mobil) and three cryptocurrencies (Bitcoin, Ethereum, Litecoin). The out-of-sample period spans eight to nine years under a walk-forward expanding-window procedure. If the claim is right, training forecasts with an objective that directly rewards correct direction and scales with realized move size is a concrete way to improve risk-adjusted returns in algorithmic trading.

What carries the argument

The central object is the Mean Absolute Directional Loss (MADL), defined as $\mathrm{MADL} = \frac{1}{N}\sum_{i=1}^{N} (-1)\,\mathrm{sign}(R_i\,\hat{R}_i)\,|R_i|$, where $R_i$ is the realized return and $\hat{R}_i$ the model's predicted return for interval $i$. The loss rewards a correct directional call with the absolute size of the realized move and punishes a wrong call the same way, so minimizing it is equivalent to maximizing the profit of a strategy that goes long or short according to the predicted sign. The paper uses MADL as the training objective inside a standard gradient-based deep-learning loop with the Adam optimizer, together with a walk-forward expanding-window scheme that retrains on a growing window and predicts one year ahead.

What would settle it

Compute $\partial \mathrm{MADL}/\partial \hat{R}_i$: it is zero for every $\hat{R}_i$ except at $\hat{R}_i = 0$. Then train a small network using exactly the described setup with MADL as the loss and Adam as the optimizer; if the weights remain unchanged after several epochs, the reported trading results cannot be produced by learning under the loss as written.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is that Transformer models trained with MADL generate trading signals whose risk-adjusted performance exceeds both LSTM and buy-and-hold on every asset tested. Across six assets, the Transformer attains the highest values on all reported information ratios, which annualize return and penalize volatility, drawdown, and loss duration. For example, on Bitcoin the Transformer attains a drawdown-adjusted information ratio (IR**) of 5.301, versus 1.939 for LSTM and 1.287 for buy-and-hold. The paper attributes the edge to the loss function: because MADL penalizes wrong directional forecasts in proportion to the magnitude of the realized move, the model learns to prioritize large correct bets rather than minimizing squared error.

Load-bearing premise

The training procedure assumes MADL has a usable gradient with respect to the predicted return, but because the sign function has zero derivative almost everywhere, the loss as written cannot update network weights through ordinary backpropagation.

Editorial extensions

If this is right

  • If the claim holds, loss-function design becomes a first-order decision in building machine-learning trading strategies: aligning the objective with the trading rule matters as much as model architecture.
  • The results would support the view that Transformers, despite being designed for natural language, can be competitive on financial time series when paired with a suitable objective.
  • The walk-forward, expanding-window evaluation with multi-year out-of-sample periods offers a template for reducing overfitting claims in algorithmic trading research.
  • MADL could in principle be applied to other model classes and asset classes, extending the comparison beyond LSTM versus Transformer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A critical reading of Eq. (4) suggests that the reported training may have actually used a differentiable surrogate of the sign function, such as a straight-through estimator or a soft sign; if so, the paper's central comparison is really about that surrogate, not about MADL as written.
  • The two architectures are compared under different hyperparameters (learning rate 0.5 vs 0.01, 300 vs 50 epochs, different regularization), so the 'Transformer versus LSTM' conclusion conflates architecture choice with tuning; an equal-budget comparison would isolate the loss-function effect.
  • A natural follow-up experiment would train both models with a differentiable approximation of MADL and with mean squared error, holding all other settings fixed, to see whether the directional loss itself, rather than the choice of model, drives the reported outperformance.
  • The policy implications about regulation discussed in the paper may be premature until the non-differentiability issue and the hyperparameter imbalance are resolved.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper applies the Mean Absolute Directional Loss (MADL), introduced in the authors' earlier work, as the training objective for Transformer and LSTM models that generate long/short signals on daily returns for three equities and three cryptocurrencies. A walk-forward procedure with a multi-year out-of-sample period is used, and strategy performance is summarized by annualized return, volatility, drawdown, and several information-ratio variants. The central claim, stated in the abstract and conclusions, is that Transformer models trained with MADL produce significantly better risk-adjusted returns than LSTM models and Buy&Hold in almost every asset considered.

Significance. If the central empirical claim were credible, the paper would offer useful evidence on Transformer architectures for algorithmic investment strategies, with strengths including a multi-asset design, long out-of-sample periods, walk-forward validation, and comparison against both a classical benchmark (Buy&Hold) and a strong baseline (LSTM). The paper also makes a methodological point about selecting loss functions for financial forecasting. However, the reported results depend on the described training procedure actually minimizing MADL, and that mechanism is not credible as written; moreover, the 'significantly better' language is unsupported by statistical inference, and the strategy returns appear to be gross of transaction costs despite high turnover. These issues place the main contribution in doubt.

major comments (3)
  1. [§3.5, Eq. (4); §3.4; Conclusions] The loss in Eq. (4) has zero gradient with respect to the predicted return almost everywhere. Because sign(R_i * R_hat_i) is piecewise constant, its derivative with respect to R_hat_i is zero except at the single point R_hat_i = 0, and abs(R_i) does not depend on R_hat_i, so ∂MADL/∂R_hat_i = 0 almost everywhere. In a standard Keras/TensorFlow backpropagation loop with the Adam optimizer, as described in §3.4, this produces zero weight updates, meaning the models are not trained by minimizing MADL. The paper notes in the Conclusions that MADL has 'the problem of its non-differentiability in certain areas', but it never specifies a surrogate gradient, a straight-through estimator, or a two-stage training procedure. The reported strategy results therefore lack a credible learning mechanism as described; the authors must either define and implement a differentiable version of the loss, provide code or pseudocode showing the actual training objective, and rerun the experiments, or substantially weaken the claims.
  2. [Abstract; §4, Tables 2-3] The claim that Transformer results are 'significantly better' than LSTM results is not supported by any statistical test. Tables 2 and 3 report point estimates for a single walk-forward path, with no confidence intervals, no bootstrap or Diebold-Mariano tests of return differences, and no correction for multiple comparisons across six assets. Without such procedures, the word 'significantly' is unjustified. The authors should either add appropriate significance tests for the reported performance metrics or replace 'significantly' with descriptive language such as 'consistently higher in the sample'.
  3. [§4, Tables 2-3; §3.6] The reported strategy performance appears to be gross of transaction costs, yet the strategies execute many trades: nTrades ranges from about 1031 to 1723 over the out-of-sample period, and the daily signal construction implies potentially frequent position changes. For the claim that these strategies yield 'abnormal risk-adjusted returns' in a realistic setting, costs are a first-order concern. The authors should state whether the reported returns are net of transaction costs and, if not, provide a sensitivity analysis under realistic cost assumptions for equities and cryptocurrencies.
minor comments (6)
  1. [§3.3, Table 3] The data description says Ethereum starts on '2015-08-07', but this is likely a typo for Litecoin; additionally, the table header 'EHT' should be 'ETH', and the table note contains a duplicated 'Note:'.
  2. [§1] 'ChatGTP' should be 'ChatGPT'.
  3. [§3.3, §3.4, Table 1] The description of the training window is inconsistent: §3.3 says the test set size is '252/365 days' and Table 1 says 'Train/test size 252/252', while §3.4 describes an expanding window that grows 'up to 4 years'. Please clarify whether the training window expands without bound or is capped, and state the exact units (trading days versus calendar days).
  4. [§3.6, Eq. (10)] In Eq. (10), the notation 'ARC' should probably be 'aRC' for consistency with Eq. (5). Also, the bullet list in §3.3 mentions 'IR, IR, IR' where three distinct information ratios are presumably meant.
  5. [§2] References [12] and [2] each appear twice in the sentence listing common flaws in backtesting studies; please remove the duplicates.
  6. [§3.5, Eq. (4)] The notation in Eq. (4) mixes 'abs(R_i)' with the text; it would be clearer to use |R_i| throughout. The behavior at R_i = 0 or R_hat_i = 0 is also undefined and should be specified.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: Transformer/LSTM comparison rests on independent out-of-sample evaluation; MADL is an input loss taken from prior work, not a derived claim.

full rationale

The paper's central claim compares Transformer and LSTM strategies trained with MADL against Buy&Hold on six assets over multi-year out-of-sample windows. The loss function in Eq. (4) is explicitly defined and is used as the training objective for both model classes; it is not a quantity derived from the reported results and then reused as evidence. The performance metrics (aRC, aSD, MD, MLD, IR*, IR**, IR***) are standard external statistics computed from equity lines, and the benchmarks (LSTM, B&H) are independent. The only self-citation is attribution of MADL to refs [14,15], which are prior papers by the same authors; however, Eq. (4) supplies the definition and this paper's empirical evaluation is self-contained against external benchmarks, so that citation is not load-bearing in the derivation. A separate issue, the non-differentiability of sign() in Eq. (4) and its implication for gradient-based training in Section 3.4, is flagged in the conclusions as 'the problem of its non-differentiability in certain areas ([14])'; that is a correctness/mechanism concern rather than a circularity of the derivation chain, and therefore does not change the circularity score.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests on hyperparameters chosen by hand (Table 1), on an unstated assumption that the non-differentiable MADL loss can be minimized by gradient descent, on gross returns without transaction costs, and on walk-forward performance being representative. The MADL loss itself is introduced in the authors' own prior work, so using it is not circular in this paper, but it is also not independently verified here.

free parameters (8)
  • Transformer learning rate = 0.01
    Hand-picked; no tuning grid or sensitivity analysis is reported in Table 1.
  • LSTM learning rate = 0.5
    Hand-picked; no rationale is given for this unusually high Adam learning rate in Table 1.
  • Sequence length = 4
    It is a hand-picked input window used to form the key/query structure; see Section 3.3.
  • Transformer architecture = 2 attention layers, 4 heads, key dim 64
    Hand-picked architecture for the Transformer; no ablation is reported in Table 1.
  • LSTM architecture = 3 layers, 512/256/128 units
    Hand-picked architecture for the LSTM; no ablation is reported in Table 1.
  • Regularization strengths = dropout 0.3 / L2 0.02 (transformer); dropout 0 / L2 1e-6 (LSTM)
    Hand-picked regularization strengths for both models; reported in Table 1.
  • Train/test window lengths = 252/252 days
    Hand-picked train and test window lengths; described in Section 3.3.
  • Training epochs = 50 (transformer), 300 (LSTM)
    Hand-picked number of training epochs; described in Section 3.4.
assumptions (5)
  • ad hoc to paper MADL can be minimized by gradient descent.
    The paper assumes that MADL in Eq. (4) can be minimized by backpropagation, but sign() has zero gradient almost everywhere and no surrogate is described.
  • domain assumption Gross returns without transaction costs are adequate for comparing strategies.
    Strategy returns are evaluated gross of transaction costs and slippage, which is not stated explicitly in Section 4.
  • domain assumption Walk-forward out-of-sample performance generalizes to future trading.
    The walk-forward out-of-sample period is treated as representative of real future performance.
  • domain assumption The selected assets do not suffer from survivorship bias.
    The six assets are selected with hindsight based on index membership and market cap, which can introduce survivorship bias in Section 3.3.
  • domain assumption One hyperparameter configuration is valid for all six assets.
    A single hyperparameter configuration is assumed to work for all six assets, with no sensitivity analysis in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Alternative Loss Function in Evaluation of Transformer Models." pith.science (2026). https://pith.science/paper/OBWZQ353

@misc{pith2026250716548,
  author       = {Pith},
  title        = {Pith review of: Alternative Loss Function in Evaluation of Transformer Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBWZQ353}},
  note         = {Machine review of arXiv:2507.16548}
}
read the original abstract

The proper design and architecture of testing machine learning models, especially in their application to quantitative finance problems, is crucial. The most important aspect of this process is selecting an adequate loss function for training, validation, estimation purposes, and hyperparameter tuning. Therefore, in this research, through empirical experiments on equity and cryptocurrency assets, we apply the Mean Absolute Directional Loss (MADL) function, which is more adequate for optimizing forecast-generating models used in algorithmic investment strategies. The MADL function results are compared between Transformer and LSTM models, and we show that in almost every case, Transformer results are significantly better than those obtained with LSTM.

Figures

Figures reproduced from arXiv: 2507.16548 by the authors.

Figure 1
Figure 1. The structure of the Transformer model with special attention to input and output layers. Source: [18]. A Transformer model consists of two main components: the encoder and the decoder. The encoder extracts features from the input, while the decoder generates output based on this rep￾resentation ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Transformer model with two different attention mechanisms: Scaled Dot-Product Attention and Multi-Head Attention. Source: [18] [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. LSTM cells presented in this Fig. show the information flow between the main LSTM gates: input, output, and forget. Source: [6] [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Equity lines for JPM, SPX, XOM, BTC, ETH and LTC. 5. Conclusions In this study, we evaluate the application of the Mean Absolute Directional Loss function ([14]) in algorithmic trading with two machine learning algorithms: the transformer model ([18]) and the LSTM ([9]…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 23 canonical work pages

  1. [14]

    Micha ´nków, P

    J. Micha ´nków, P. Sakowski, and R. ´Slepaczuk. Lstm in algorithmic investment strate- gies on btc and s&p500 index. Sensors, 22(3), 2022. ISSN 1424-8220. doi: 10.3390/s22030917

  2. [1]

    D. H. Bailey, J. Borwein, M. Lopez de Prado, A. Salehipour, and Q. J. Zhu. Backtest overfitting in financial markets. Automated Trader, 2016

  3. [2]

    D. H. Bailey, J. Borwein, M. Lopez de Prado, and Q. J. Zhu. The probability of backtest overfitting. Journal of Computational Finance, forthcoming, 2016

  4. [3]

    J. B. Chakole, M. S. Kolhe, G. D. Mahapurush, A. Yadav, and M. P. Kurhekar. A q-learning agent for automated trading in equity stock markets. Expert Systems with Applications, 163:113761, 2021. ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa. 2020.113761

  5. [4]

    E. Chan. Algorithmic trading: winning strategies and their rationale, volume 625. John Wiley & Sons, 2013

  6. [5]

    E. P. Chan. Quantitative trading: how to build your own algorithmic trading business . John Wiley & Sons, 2021

  7. [6]

    F. Chollet. Deep Learning with Python, 2nd ed. Manning Publications Co., 2021

  8. [7]

    Di Persio and O

    L. Di Persio and O. Honchar. Artificial neural networks architectures for stock price prediction: Comparisons and applications. International Journal of Circuits, Systems And Signal Processing, 10:403–413, Jan. 2016

Show all 29 references
  1. [8]

    Grobys, S

    K. Grobys, S. Ahmed, and N. Sapkota. Technical trading rules in the cryptocurrency market. Finance Research Letters, 32:101396, 2020. ISSN 1544-6123. doi: https: //doi.org/10.1016/j.frl.2019.101396

  2. [9]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9 (8):1735–1780, 11 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735

  3. [10]

    S. Jansen. Machine Learning for Algorithmic Trading: Predictive models to extract signals from market and alternative data for systematic trading strategies with Python. Packt Publishing Ltd, 2020

  4. [11]

    Kijewski, R

    M. Kijewski, R. ´Slepaczuk, and M. Wysocki. Predicting prices of s&p 500 in- dex using classical methods and recurrent neural networks. ISD2024 Proceedings. Gda´ nsk, Poland: University of Gda´ nsk. ISBN: 978-83-972632-0-8., 2024. doi: https: //doi.org/10.62036/ISD.2024.89

  5. [12]

    Lopez de Prado

    M. Lopez de Prado. What to look for in a backtest. Available at SSRN, 2013

  6. [13]

    T. Ma, W. Wang, and Y . Chen. Attention is all you need: An interpretable transformer- MICHA ´NKÓW ET AL . A LTERNATIVE LOSS FUNCTION IN EVALUATION OF TRANSFORMER MODELS . . . based asset allocation approach. International Review of Financial Analysis , 90(C),

  7. [15]

    Micha ´nków, P

    J. Micha ´nków, P. Sakowski, and R. ´Slepaczuk. Mean absolute directional loss as a new loss function for machine learning problems in algorithmic investment strategies. Journal of Computational Science , 81:102375, 2024. ISSN 1877-7503. doi: https: //doi.org/10.1016/j.jocs.20...

  8. [16]

    J. Qiu, B. Wang, and C. Zhou. Forecasting stock prices with long-short term memory neural network based on attention mechanism. PLOS ONE, 15(1):1–15, 01 2020. doi: 10.1371/journal.pone.0227222

  9. [17]

    A. Raudys. Portfolio of global futures algorithmic trading strategies for best out-of- sample performance. In International Conference on Business Information Systems , pages 424–435. Springer, 2016

  10. [18]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023

  11. [19]

    C. Wang, Y . Chen, S. Zhang, and Q. Zhang. Stock market index prediction using deep transformer model. Expert Systems with Applications, 208:118128, 2022. ISSN 0957-

  12. [20]

    Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun. Transformers in time series: A survey, 2023

  13. [21]

    Wiecki, A

    T. Wiecki, A. Campbell, J. Lent, and J. Stauth. All that glitters is not gold: Comparing backtest and out-of-sample performance on a large cohort of trading algorithms. The Journal of Investing, 25(3):69–80, 2016

  14. [22]

    J. Yang, Y . Li, X. Chen, J. Cao, and K. Jiang. Deep Learning for Stock Selection Based on High Frequency Price-V olume Data.arXiv:1911.02502 [cs, q-fin], Nov. 2019. arXiv: 1911.02502

  15. [23]

    A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effective for time series forecasting?, 2022

  16. [24]

    Z. Zeng, R. Kaur, S. Siddagangappa, S. Rahimi, T. Balch, and M. Veloso. Financial time series forecasting using cnn and transformer, 2023

  17. [25]

    Zhang, C

    R. Zhang, C. Huang, W. Zhang, and S. Chen. Multi Factor Stock Selection Model Based on LSTM. International Journal of Economics and Finance, 10(8):1–36, 2018. Publisher: Canadian Center of Science and Education

  18. [26]

    K. Zhou, W. Y . Wang, T. Hu, and C. H. Wu. Comparison of time series forecasting based on statistical arima model and lstm with attention mechanism.Journal of Physics: Con- ference Series, 1631(1):012141, sep 2020. doi: 10.1088/1742-6596/1631/1/012141

  19. [27]

    ´Slepaczuk, P

    R. ´Slepaczuk, P. Sakowski, and G. Zakrzewski. Investment strategies that beat the market. what can we squeeze from the market? Financial Internet Quarterly, 14(4): 36–55, 2018. doi: doi:10.2478/fiqf-2018-0026

  20. [2023]

    doi: 10.1016/j.irfa.2023.10287

  21. [4174]

    doi: https://doi.org/10.1016/j.eswa.2022.118128

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.