Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Synthetic Time Series Forecasting with Transformer Architectures: Extensive Simulation Benchmarks

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In 1,500 synthetic runs, patch-based tokenization with a learned positional embedding (PatchTST) gives the best accuracy-stability balance, while trend-seasonal decomposition wins on noisy signals and sparse attention proves fragile.

desk verdict The paper's headline claim contradicts its own reported results, and the missing train/test protocol makes the rankings unverifiable; the benchmark and Deep Koopformer idea are worth a serious referee, not a desk reject. read the letter →

arxiv 2505.20048 v1 pith:C4GM5EX3 submitted 2025-05-26 cs.LG cs.SYeess.SY

Ali Forootani , Mohammad Khosravi This is my paper · ORCID
classification cs.LGcs.SYeess.SY MSC 68T07
keywords timeseriesforecastingTransformerarchitecturesPatchTSTAutoformerInformerKoopmanoperatorsyntheticbenchmarknoiserobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to standardize how Transformer architectures are compared for time series forecasting by running one controlled benchmark: Autoformer, Informer, and PatchTST, each in a Minimal, Standard, and Full complexity variant, across ten synthetic signals, five patch lengths, five forecast horizons, and clean versus noisy conditions, for 1,500 experiments in total. The central claim is that PatchTST Standard, which cuts the series into patches and learns a global positional embedding, achieves the best accuracy-stability balance overall, while Autoformer's trend-seasonal decomposition makes it the most accurate family on smooth and trend-dominated signals, especially when noise is added, and Informer's ProbSparse attention is the most fragile at long horizons. The point of using synthetic signals is that each one isolates a specific failure mode, such as exponential growth, a localized bump, or a modulated oscillation, so that performance differences can be attributed to architecture rather than to dataset accidents. The paper also introduces Deep Koopformer, a hybrid that passes Transformer-encoded patches through a spectrally constrained Koopman operator so latent states evolve linearly and stably, and it reports stable forecasts on a noisy Van der Pol oscillator and the Lorenz system. If the rankings hold, model selection for forecasting reduces to matching signal structure, whether trend, noise, or chaos, to the right inductive bias: decomposition, patching, or operator-based latent dynamics.

What carries the argument

The load-bearing objects are the three architectural mechanisms under test. PatchTST Standard's mechanism is temporal patching with a learned global positional embedding: the input series is cut into fixed-length patches before attention, which shortens the token sequence and lets the model adapt its temporal representation during training. Autoformer's mechanism is a moving-average decomposition that splits each input into a trend component and a seasonal residual before the attention layers, so attention only ever models the detrended part. Informer's mechanism is ProbSparse attention, which scores each query by the gap between its maximum and average alignment with all keys and then computes full attention only for the top $u = O(\log P)$ queries. For the second half of the paper, the central object is the Koopman-enhanced Transformer (Deep Koopformer): a Transformer encoder maps patches to latent states, a learned linear operator $K = U\operatorname{diag}(S)V^\top$ with singular values capped at 0.99 propagates those states forward, a Lyapunov-inspired loss penalizes growth of the latent norm, and Householder orthogonalization keeps the factor matrices well conditioned. The Koopman operator is what carries the stability argument, converting nonlinear dynamics into constrained linear evolution so that bounded long-horizon rollout is a design guarantee rather than an empirical hope.

What would settle it

Re-run the same 1,500-configuration grid with an explicit temporal holdout, for example training on the first 400 time steps of each normalized 500-point signal and evaluating only on the final 100 steps, using the paper's reported hyperparameters, and check whether PatchTST Standard still beats Autoformer on mean RMSE and MAE. If the ordering flips once a true held-out split is imposed, the benchmark's ranking is an artifact of the missing evaluation protocol.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a map from signal structure and noise regime to architecture. PatchTST Standard, which combines patch-based tokenization with a learnable global positional embedding shared across all positions, is claimed to be the most consistent and robust variant, holding low RMSE and MAE across the full grid of patch lengths and horizons in both clean and noisy conditions. Autoformer Minimal and Standard, built on moving-average trend-seasonal decomposition, are claimed to be the most accurate on smooth, periodic, and trend-dominated signals and to remain the leading family under noise, with aggregate mean RMSE below roughly 0.045 in clean settings and 0.076 in noisy settings. Informer is claimed to be the least stable family: its ProbSparse attention reduces computation but produces the largest errors and the greatest sensitivity to horizon, especially when paired with short patches. The paper further argues that adding a stability-constrained Koopman operator, parameterized as $K = U\operatorname{diag}(S)V^\top$ with the spectral radius capped at 0.99, a Lyapunov term $\operatorname{ReLU}(\|z_{t+1}\|_2 - \|z_t\|_2)$ penalizing latent energy growth, and Householder orthogonalization of $U$ and $V$, turns Transformer forecasters into stable predictors for nonlinear and chaotic systems, demonstrated on a noisy Van der Pol oscillator and the Lorenz attractor.

Load-bearing premise

The rankings assume the reported RMSE and MAE values measure genuine forecasting skill on data the models never trained on, but the paper never describes how each 500-point signal was divided into training and test portions, so the scores could instead reflect in-sample fit and memorization.

Editorial extensions

If this is right

  • Model selection guidance: reach for a decomposition-based forecaster (Autoformer Minimal or Standard) on smooth, trend-heavy, or noisy signals, and for PatchTST Standard when the priority is consistent accuracy across horizons and noise levels.
  • Sparse attention buys efficiency at an accuracy cost: Informer's ProbSparse variants show the highest errors and the most variability, degrading fastest as forecast horizon grows.
  • Capacity beyond the Standard variants does not pay off: across all three families the best configurations concentrate at patch lengths 12 to 16 and horizons 2 to 6, with the Full encoder-decoder variants rarely beating the simpler ones.
  • Bounded latent dynamics stabilize long-horizon rollout: with the spectral radius capped below 1 and a Lyapunov penalty on latent energy growth, Deep Koopformer keeps forecasts bounded on chaotic systems where unconstrained attention can diverge.
  • Optimal configuration is horizon-dependent: short horizons favor Minimal and Standard variants, while the Full designs earn their extra cost mainly on noisy or structurally complex signals with longer horizons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I read the paper's own aggregated heatmaps as showing Autoformer with the lower mean RMSE and MAE in most cells, which suggests the headline claim is really an accuracy-stability trade-off judgment that counts consistency and efficiency as well as raw error; a per-configuration win-rate analysis would be needed to make the ranking unambiguous.
  • Because every result comes from normalized 500-point synthetic signals, a direct transfer experiment, re-running the same nine-variant grid on real energy, finance, or health series with the same patch and horizon settings, would test whether the synthetic ordering survives on real data.
  • The Deep Koopformer experiments report tracked spectral-radius curves and visual trajectory agreement rather than side-by-side RMSE against the plain, non-Koopman variants on the same Van der Pol and Lorenz data, so the marginal benefit of the operator constraint over the bare encoder remains unquantified.
  • Because Deep Koopformer caps the spectral radius at 0.99, its latent dynamics contract and the model cannot in principle represent persistently growing series; the paper's own Exponential Growth signal is the obvious counterexample, making the stability guarantee a representational ceiling that could be tested directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a unified benchmark of Autoformer, Informer, and PatchTST, each implemented in Minimal, Standard, and Full variants, on ten synthetic univariate signals under clean and noisy conditions. It reports RMSE and MAE heatmaps over grids of patch lengths and forecast horizons, along with per-signal best-configuration tables. The paper also sketches a Koopman-operator-enhanced Transformer framework called Deep Koopformer and presents trajectory and spectral-radius plots for noisy Van der Pol and Lorenz systems. The stated headline result is that PatchTST Standard achieves the best overall performance, while the paper's own aggregated cross-family comparison sections report that the Autoformer family has the lowest errors.

Significance. A careful, reproducible comparison of Transformer forecasting variants on controlled synthetic signals would be useful to practitioners, and the authors provide open-source code and a wide grid of configurations. If the reported ranking were robust and properly evaluated out of sample, it would inform model selection. However, the internal contradiction between the headline claim and the paper's own aggregated heatmaps, together with the absence of a stated train/test split, prevents the empirical contribution from being accepted as evidence for any ranking. The Deep Koopformer material is promising as a research direction but is not yet supported by quantitative evaluation.

major comments (4)
  1. [Introduction item (iii), Section VIII, Sections VI.G-VI.H, Figs. 5-6] The central claim that PatchTST Standard achieves the best overall performance under clean and noisy conditions is contradicted by the paper's own aggregated results. Section VI.G states that the Autoformer variants achieve the lowest average RMSE and MAE across nearly all configurations, and Section VI.H states that the Autoformer family remains the most accurate under noisy conditions. Figures 5 and 6 show lower RMSE and MAE values for the Autoformer family than for the PatchTST family in nearly every cell of the grid. The Introduction and Conclusion therefore assert a ranking that the reported data do not support; the authors must either revise the headline claim to match the reported aggregation or provide a different, clearly described aggregation that justifies the PatchTST Standard claim.
  2. [Section VI.C] The evaluation protocol never specifies how the 500-point signals are divided into training and test sets, whether a validation split or roll-forward scheme is used, how many random seeds are averaged, or how the RMSE and MAE values are computed over the forecast horizon. Without this information, the rankings in Figs. 2-6 and Tables IV-IX cannot be interpreted as out-of-sample forecasting skill; they may reflect in-sample fit or memorization. This omission is load-bearing for every cross-family comparison in the paper.
  3. [Section VI.C] The stated experiment count is internally inconsistent. With three model families, each having three variants, evaluated on five patch lengths, five forecast horizons, and ten signals, there are 9 x 5 x 5 x 10 = 2250 evaluations per noise regime, or 4500 total. The text instead says there are 750 configurations per regime and 1500 total, and the abstract says 'over 1500 controlled experiments.' The per-variant heatmaps in Figs. 2-4 require the larger count, so the counting basis must be corrected and clarified.
  4. [Section VII.B-D, Figs. 7-10] The Deep Koopformer results do not support the claimed demonstration of efficacy. No quantitative error metrics, baseline comparisons, train/test split, or seed information are reported for the Van der Pol and Lorenz experiments, and the text states only that variants 'perform similarly well' without numerical evidence. In addition, the spectral radius is capped at 0.99 by construction in Eqs. (43)-(46), so Figs. 8 and 10, which show the spectral radius approaching a value below 1, chiefly illustrate the imposed constraint rather than provide independent evidence of learned stability.
minor comments (5)
  1. [Conclusion and Abstract] There are typos and inconsistent capitalization: 'enahnced' in the Conclusion and 'Patchtst' in the Abstract should be 'enhanced' and 'PatchTST.'
  2. [Section VI.B] The hardware setup section mentions 'wind power time series data,' but the experiments in the paper use synthetic signals; this appears to be leftover text from another project and should be removed or corrected.
  3. [Theorem 1] The proof of Theorem 1 is a sketch: the bound on the approximation error is asserted as C*epsilon with epsilon decaying exponentially in u, but no derivation is given, and the complexity statement assumes LQ ~ LK. The authors should state the assumptions precisely and provide a derivation or cite a complete proof.
  4. [Section VI.A] The random temporal shift in the noise injection uses s_noisy(t) replaced by s(t + Delta t) with Delta t in [-10, 10], but boundary handling for t + Delta t outside the sampled range [0, 499] is not specified.
  5. [References] References [28] and [34] are the same paper, and references [30] and [35] are also the same paper; these duplicates should be consolidated.

Circularity Check

1 steps flagged · score 2.0 of 10

Benchmark itself is self-contained; only the Koopman stability claim is circular by construction (spectral radius capped at 0.99 then cited as validation), giving a low score of 2.

  1. self definitional [Section VII.B (Eqs. 42–46) and Section VII.D (Van der Pol evaluation, Fig. 8 discussion)]
    "The Koopman operator is parametrized as: K = U · diag(S) · V ⊤ ... Si = σ(Sraw i ) · 0.99. ... By construction, the spectral radius of K is strictly controlled: ρ(K) = max |λi| ≤0.99. ... This stability is further ensured by the Koopman operator, which constrains the evolution of the latent states, and is validated through the spectral radius of the operator during training (see Fig. 8)."

    The spectral radius bound is not an empirical finding but a hard constraint imposed in the parametrization: S_i = σ(Sraw_i)·0.99 makes ρ(K) ≤ 0.99 by definition. The paper then presents the bounded spectral radius observed during training as 'validation' that the Koopman operator enforces stable latent dynamics. Reporting the imposed cap as evidence of stability is equivalent to restating the design choice as a result. This does not make the main benchmark circular, since the benchmark rankings are external comparisons, but the stability claim for Deep Koopformer reduces to its construction.

full rationale

The central benchmark is an independent empirical comparison: Autoformer, Informer, and PatchTST variants are trained under a common configuration and evaluated on synthetic signals, so the rankings in Figs. 2–6 are not derived from the models' definitions. No parameter is fitted to the headline outcome, and the paper does not rely on a self-citation chain to justify its main comparisons. The Koopman section contains the one genuinely circular step: the spectral radius is capped at 0.99 by construction and then cited as evidence of stability, which is a design feature reported as validation. Separately, but not as circularity, the Introduction (iii) and Conclusion claim PatchTST Standard is best overall while the paper's own aggregated cross-family results (Figs. 5–6 and Sections VI.G–VI.H) place Autoformer first, and the training setup in Section VI.C omits any stated train/test split, so out-of-sample status is unclear. Those are correctness/validity concerns, not circular reductions. Overall circularity is therefore low.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central benchmark depends on protocol choices (hyperparameters, grids, noise levels, assumed disjoint train/test data) that are not justified by sensitivity analysis. The Deep Koopformer section depends on the assumption that a finite-dimensional Koopman-invariant subspace is learned by a Transformer encoder. These are explicit or implicit assumptions rather than derived results.

free parameters (6)
  • Model capacity hyperparameters = d_model=8, heads=2, FFN=32, encoder layers=2, decoder layers=1
    Fixed for all models; unusually small capacity may favor models with strong inductive bias (Autoformer decomposition, PatchTST patching) and inflate relative rankings.
  • Patch length grid = {4, 8, 12, 16, 20}
    Sweep values chosen by hand; the optimal patch length range 12-16 is a key finding, so the grid bounds the conclusion.
  • Forecast horizon grid = clean {2,4,6,8,10}; noisy {4,8,12,16,20}
    Different horizon sets appear for clean vs noisy heatmaps, while per-signal tables include 12, 16, 20; this inconsistency affects all horizon-sensitivity claims.
  • Spectral radius bound = 0.99
    In Deep Koopformer, stability is enforced by capping S_i at 0.99, so the reported spectral stability is a consequence of this choice.
  • Lyapunov loss weight = not stated
    The loss is 'MSE augmented with a stability penalty', but no coefficient is given, so the regularization strength is unspecified.
  • Noise levels = Gaussian sigma=0.10, multiplicative sigma=0.08, shift 10%; Koopman examples sigma=0.02, 0.5
    Noise amplitudes are hand-selected; conclusions about noise robustness are relative to these levels.
assumptions (5)
  • domain assumption Finite-dimensional Koopman invariant subspace exists and is learnable by a Transformer encoder
    Deep Koopformer assumes the latent representation z_t = Encoder(x_t) evolves linearly under K (Eq. 41); no proof that the Transformer encoder spans a Koopman-invariant subspace.
  • domain assumption Training and evaluation windows are disjoint
    No data split is described in Section VI.C; if windows are shared between training and evaluation, all RMSE values would be in-sample.
  • domain assumption Synthetic signals are representative of real-world forecasting challenges
    The paper generalizes from 10 analytic signals to practical settings without real datasets.
  • domain assumption Identical hyperparameters yield fair comparisons
    All models use d_model=8, etc., but no sensitivity analysis is provided; the ranking may depend on this specific capacity budget.
  • ad hoc to paper Theorem 1 attention complexity reduction holds as stated
    The proof asserts an exponential error bound without defining epsilon or proving decay; complexity claims for Informer Standard and Full rely on it.
invented entities (1)
  • Finite-dimensional Koopman latent operator K
    purpose: Linear propagation of latent states for stability and interpretability in Deep Koopformer
    K is learned on the same data and constrained to be contractive; the paper offers no falsifiable prediction outside its own datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synthetic Time Series Forecasting with Transformer Architectures: Extensive Simulation Benchmarks." pith.science (2026). https://pith.science/paper/C4GM5EX3

@misc{pith2026250520048,
  author       = {Pith},
  title        = {Pith review of: Synthetic Time Series Forecasting with Transformer Architectures: Extensive Simulation Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C4GM5EX3}},
  note         = {Machine review of arXiv:2505.20048}
}
read the original abstract

Time series forecasting plays a critical role in domains such as energy, finance, and healthcare, where accurate predictions inform decision-making under uncertainty. Although Transformer-based models have demonstrated success in sequential modeling, their adoption for time series remains limited by challenges such as noise sensitivity, long-range dependencies, and a lack of inductive bias for temporal structure. In this work, we present a unified and principled framework for benchmarking three prominent Transformer forecasting architectures-Autoformer, Informer, and Patchtst-each evaluated through three architectural variants: Minimal, Standard, and Full, representing increasing levels of complexity and modeling capacity. We conduct over 1500 controlled experiments on a suite of ten synthetic signals, spanning five patch lengths and five forecast horizons under both clean and noisy conditions. Our analysis reveals consistent patterns across model families. To advance this landscape further, we introduce the Koopman-enhanced Transformer framework, Deep Koopformer, which integrates operator-theoretic latent state modeling to improve stability and interpretability. We demonstrate its efficacy on nonlinear and chaotic dynamical systems. Our results highlight Koopman based Transformer as a promising hybrid approach for robust, interpretable, and theoretically grounded time series forecasting in noisy and complex real-world conditions.

Figures

Figures reproduced from arXiv: 2505.20048 by the authors.

Figure 1
Figure 1. 10 synthetic time series data used for benchmarking transformer-based forecasting models. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Performance of PatchTST variants (Minimal, Standard, Full) on various patch lengths and forecast horizons, averaged over all signals. Left column: RMSE heatmaps. Right column: MAE heatmaps. TABLE VI: Best Informer Model per Signal Based on RMSE & MAE (Clean Signals) Signal Model Patch Horizon RMSE MAE 2nd Order Polynomial Standard 12 10 0.0421 0.0309 Cosine + Trend Standard 12 2 0.0127 0.0100 CosEnvelope × Sine Mini… view at source ↗
Figure 3
Figure 3. Performance of Informer variants (Minimal, Standard, Full) on various patch lengths and forecast horizons, averaged over all signals. variety of configurations. The RMSE values for noisy signals range from 0.0278 to 0.0562, and the MAE values range from 0.0220 to 0.0457, indicating that it remains relatively robust in noisy scenarios. The best performance in noisy environments is observed at shorter patch lengths (4… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance of Autoformer variants (Minimal, Standard, Full) on various patch lengths and forecast horizons, averaged over all signals. TABLE VIII: Best Autoformer Model per Signal Based on RMSE & MAE (Clean Signals) Signal Model Patch Horizon RMSE MAE 2nd Order Polyno…
Figure 5
Figure 5. Figure 5: Aggregated RMSE and MAE heatmaps compare [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Aggregated RMSE and MAE heatmaps (averaged over model variants and noisy signals) compare [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Deep Koopformer based Transformers Variants Com [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Comparison of Deep Koopman Operator Stability [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Deep Koopformer PatchTSTprediction on Lorenz Dynamical System 0 50 100 150 200 250 300 Training Step (×10 epochs) 0.88 0.90 0.92 0.94 0.96 Max Eigenvalue (Spectral Radius) Deep Koopman Operator Stability Evolution Deep Koopformer PatchTST [PITH_FULL_IMAGE:figures/full…
Figure 10
Figure 10. Figure 10: Deep Koopman Operator Stability Evolution with [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 29 canonical work pages

  1. [1]

    Climate Aware Deep Neural Networks (CADNN) for Wind Power Simulation

    A. Forootani, D. E. Aliabadi, and D. Thraen, “Climate aware deep neural networks (cadnn) for wind power simulation,” arXiv preprint arXiv:2412.12160, 2024

    work page Pith review arXiv 2024

  2. [2]

    Energy time series forecasting based on pattern sequence similarity,

    F. M. Alvarez, A. Troncoso, J. C. Riquelme, and J. S. A. Ruiz, “Energy time series forecasting based on pattern sequence similarity,” IEEE Transactions on Knowledge and Data Engineering , vol. 23, no. 8, pp. 1230–1243, 2010

    work page 2010

  3. [3]

    Financial time series forecasting-a deep learning approach,

    A. Dingli and K. S. Fournier, “Financial time series forecasting-a deep learning approach,” International Journal of Machine Learning and Computing, vol. 7, no. 5, pp. 118–122, 2017

    work page 2017

  4. [4]

    Unveiling the multi- dimensional spatio-temporal fusion transformer (mdstft): A revolution- ary deep learning framework for enhanced multi-variate time series forecasting,

    S. Wang, Y . Lin, Y . Jia, J. Sun, and Z. Yang, “Unveiling the multi- dimensional spatio-temporal fusion transformer (mdstft): A revolution- ary deep learning framework for enhanced multi-variate time series forecasting,” IEEE Access, 2024

    work page 2024

  5. [5]

    Multi-resolution expansion of analysis in time-frequency domain for time series forecasting,

    K. Yan, C. Long, H. Wu, and Z. Wen, “Multi-resolution expansion of analysis in time-frequency domain for time series forecasting,” IEEE Transactions on Knowledge and Data Engineering , 2024

    work page 2024

  6. [6]

    Compressive spatio-temporal forecasting of meteorological quantities and photo- voltaic power,

    A. Tascikaraoglu, B. M. Sanandaji, G. Chicco, V . Cocina, F. Sper- tino, O. Erdinc, N. G. Paterakis, and J. P. Catal ˜ao, “Compressive spatio-temporal forecasting of meteorological quantities and photo- voltaic power,” IEEE Transactions on Sustainable Energy , vol. 7, no. 3, pp. 1295–1305, 2016

    work page 2016

  7. [7]

    Deep learning for time series forecasting: Tutorial and literature survey,

    K. Benidis, S. S. Rangapuram, V . Flunkert, Y . Wang, D. Maddix, C. Turkmen, J. Gasthaus, M. Bohlke-Schneider, D. Salinas, L. Stella, et al., “Deep learning for time series forecasting: Tutorial and literature survey,” ACM Computing Surveys , vol. 55, no. 6, pp. 1–36, 2022

    work page 2022

  8. [8]

    Temporal convolutional neural (tcn) network for an effective weather forecasting using time-series data from the local weather station,

    P. Hewage, A. Behera, M. Trovati, E. Pereira, M. Ghahremani, F. Palmieri, and Y . Liu, “Temporal convolutional neural (tcn) network for an effective weather forecasting using time-series data from the local weather station,” Soft Computing, vol. 24, pp. 16453–16482, 2020

    work page 2020

Show all 35 references
  1. [9]

    Time-series forecasting with deep learning: a survey,

    L. Bryan and Z. Stefan, “Time-series forecasting with deep learning: a survey,” Phil. Trans. R. Soc. A , 2021

    2021

  2. [10]

    Deep learning for time series forecasting: a survey,

    J. F. Torres, D. Hadjout, A. Sebaa, F. Mart´ınez- ´Alvarez, and A. Troncoso, “Deep learning for time series forecasting: a survey,” Big Data, vol. 9, no. 1, pp. 3–21, 2021. 20

    2021

  3. [11]

    An experi- mental review on deep learning architectures for time series forecasting,

    P. Lara-Ben ´ıtez, M. Carranza-Garc ´ıa, and J. C. Riquelme, “An experi- mental review on deep learning architectures for time series forecasting,” International Journal of Neural Systems , vol. 31, no. 03, p. 2130001, 2021

    2021

  4. [12]

    Ammus: A survey of transformer-based pretrained models in natural language processing,

    K. S. Kalyan, A. Rajasekharan, and S. Sangeetha, “Ammus: A survey of transformer-based pretrained models in natural language processing,” arXiv preprint arXiv:2108.05542 , 2021

    2021 arXiv

  5. [13]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  6. [14]

    Transformers in vision: A survey,

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys (CSUR) , 2021

    2021

  7. [15]

    A comparative study on transformer vs rnn in speech applications,

    S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y . Soplin, R. Yamamoto, X. Wang, et al. , “A comparative study on transformer vs rnn in speech applications,” inIEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pp. 449–456, I...

    2019

  8. [16]

    Trans- formers in time series: A survey,

    Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun, “Trans- formers in time series: A survey,” arXiv preprint arXiv:2202.07125 , 2022

    2022 arXiv

  9. [17]

    A practical survey on faster and lighter transformers,

    Q. Fournier, G. M. Caron, and D. Aloise, “A practical survey on faster and lighter transformers,” ACM Computing Surveys , vol. 55, no. 14s, pp. 1–40, 2023

    2023

  10. [18]

    Long-short transformer: Efficient transformers for language and vision,

    C. Zhu, W. Ping, C. Xiao, M. Shoeybi, T. Goldstein, A. Anandkumar, and B. Catanzaro, “Long-short transformer: Efficient transformers for language and vision,” Advances in neural information processing sys- tems, vol. 34, pp. 17723–17736, 2021

    2021

  11. [19]

    Informer: Beyond efficient transformer for long sequence time-series forecasting,

    H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in The Thirty-Fifth AAAI Conference on Artificial Intelli- gence, vol. 35, pp. 11106–11115, 2021

    2021

  12. [20]

    Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting,

    H. Wu, J. Xu, J. Wang, and M. Long, “Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting,” in Advances in Neural Information Processing Systems , 2021

    2021

  13. [21]

    FEDformer: Frequency enhanced decomposed transformer for long-term series fore- casting,

    T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “FEDformer: Frequency enhanced decomposed transformer for long-term series fore- casting,” in Proc. 39th International Conference on Machine Learning , 2022

    2022

  14. [22]

    Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting,

    S. Liu, H. Yu, C. Liao, J. Li, W. Lin, A. X. Liu, and S. Dustdar, “Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting,” in International Conference on Learn- ing Representations, 2022

    2022

  15. [23]

    Are transformers effective for time series forecasting?,

    A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are transformers effective for time series forecasting?,” arXiv preprint arXiv:2205.13504 , 2022

    2022 arXiv

  16. [24]

    A time series is worth 64 words: Long-term forecasting with transformers,

    Y . Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” arXiv preprint arXiv:2211.14730, 2022

    2022 arXiv

  17. [25]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” in International Conference on Learni...

    2021

  18. [26]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Information Processing Systems, vol. 33, pp. 12449– 12460, 2020

    2020

  19. [27]

    Time series clas- sification using multi-channels deep convolutional neural networks,

    Y . Zheng, Q. Liu, E. Chen, Y . Ge, and J. L. Zhao, “Time series clas- sification using multi-channels deep convolutional neural networks,” in International conference on web-age information management, pp. 298– 310, Springer, 2014

    2014

  20. [28]

    Deep learning for universal linear embeddings of nonlinear dynamics,

    B. Lusch, J. N. Kutz, and S. L. Brunton, “Deep learning for universal linear embeddings of nonlinear dynamics,” Nature Communications , vol. 9, p. 4950, Nov. 2018

    2018

  21. [29]

    Learning deep neural network representations for Koopman operators of nonlinear dynamical systems,

    E. Yeung, S. Kundu, and N. Hodas, “Learning deep neural network representations for Koopman operators of nonlinear dynamical systems,” in 2019 American Control Conference (ACC) , pp. 4832–4839, 2019

    2019

  22. [30]

    Learning Koopman invariant subspaces for dynamic mode decomposition,

    N. Takeishi, Y . Kawahara, and T. Yairi, “Learning Koopman invariant subspaces for dynamic mode decomposition,” in Advances in Neural Information Processing Systems (I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Cu...

    2017

  23. [31]

    Dissipative deep neural dynamical systems,

    J. Drgo ˇna, A. Tuor, S. Vasisht, and D. Vrabie, “Dissipative deep neural dynamical systems,” IEEE Open Journal of Control Systems , vol. 1, pp. 100–112, 2022

    2022

  24. [32]

    Constrained block nonlinear neural dynamical models,

    E. Skomski, S. Vasisht, C. Wight, A. Tuor, J. Drgo ˇna, and D. Vrabie, “Constrained block nonlinear neural dynamical models,” in 2021 Amer- ican Control Conference (ACC) , pp. 3993–4000, 2021

    2021

  25. [33]

    On the stochastic stability of deep markov models,

    J. Drgona, S. Mukherjee, J. Zhang, F. Liu, and M. Halappanavar, “On the stochastic stability of deep markov models,” in Advances in Neural Information Processing Systems (M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, eds.), vol. 34, pp. 24033– 24047, Cu...

    2021

  26. [34]

    Deep learning for universal linear embeddings of nonlinear dynamics,

    B. Lusch, J. N. Kutz, and S. L. Brunton, “Deep learning for universal linear embeddings of nonlinear dynamics,” Nature communications , vol. 9, no. 1, p. 4950, 2018

    2018

  27. [35]

    Learning koopman invariant subspaces for dynamic mode decomposition,

    N. Takeishi, Y . Kawahara, and T. Yairi, “Learning koopman invariant subspaces for dynamic mode decomposition,” Advances in neural infor- mation processing systems , vol. 30, 2017. 21 Algorithm 1 PatchTST Full Forecasting Procedure 1: Input: Patch sequence x ∈ RB×P ×1; forecas...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.