Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Transformers and Their Roles as Time Series Foundation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that transformers can fit autoregressive time series by in-context gradient descent and that pretraining on weakly dependent series generalizes at a square-root-n rate.

desk verdict The paper's approximation theorem is built on a scaling bug in Lemma 3.2; the idea is right, but the proof as written does not go through. read the letter →

arxiv 2502.03383 v1 pith:MC4GGVNK submitted 2025-02-05 cs.LG cs.AI

Dennis Wu , Yihan He , Yuan Cao , Jianqing Fan , Han Liu This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0762M10
keywords timeseriesfoundationmodelstransformersin-contextlearningautoregressivegeneralizationboundsDobrushinconditionMOIRAIany-variateattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to prove that transformer-based time series foundation models are not just empirically successful but theoretically sound. Its central claim is that a transformer can take any univariate or multivariate time series, treat the history as in-context examples, and implement gradient descent to fit an autoregressive (AR) model to it. It further claims that MOIRAI's any-variate encoding lets one transformer handle an arbitrary number of covariates by automatically choosing the dimensionality of the AR model. For learning from many series, it proves a pretraining generalization bound: when pretraining data satisfies Dobrushin's condition, the test error of the empirical risk minimizer decays as $n^{-1/2}$ in the number of series. If these results hold, they explain how one model can forecast many datasets without retraining.

What carries the argument

The load-bearing mechanism is in-context gradient descent: the paper shows that one attention layer can reformat a raw time series, with labels in the first row and lagged values below, into the feature/label/position-block format that earlier in-context-learning results use, and then a shallow transformer simulates gradient descent steps on the least-squares AR loss, so each layer corresponds to one gradient step. For MOIRAI, the additional machinery is the any-variate encoding: flattening $d$ covariates into a long one-dimensional sequence plus time and variate index tokens, which any-variate attention uses, via biases $u_1, u_2$ and a block-diagonal mask $U$, to apply the same operation variate-wise. The generalization machinery is Dobrushin's condition, a measure of weak dependence quantified by influence coefficients $\alpha(P)$ and $\alpha_{\log}(P)$; it supplies concentration and a Dudley-entropy chaining argument for dependent, non-i.i.d. data.

What would settle it

Train a ReLU-attention transformer with any-variate encoding on synthetic $\mathrm{AR}_d(q)$ data with held-out $(d,q)$, then compare its predictions to the least-squares ERM fit on the same lookback: if prediction error does not shrink with input length $T$ and approach the ERM prediction as $\epsilon\to 0$, the approximation claim fails; separately, checking whether pretraining on series with non-identical marginals breaks the $n^{-1/2}$ bound would test the unstated Assumption D.9.

Watch

Extended reading notes

Core claim

On the approximation side, the paper constructs transformers that provably fit AR models via gradient descent: Proposition 3.4 shows an $L$-layer transformer whose prediction $\hat{x}_T$ on any $\mathrm{AR}_1(q)$ series is within $\epsilon$ of the inner product of the empirical risk minimizer $\hat{w}_{\mathrm{ERM}}$ with the lagged features, with $L$ logarithmic in $1/\epsilon$. Theorem 3.8 extends this to MOIRAI: there exists an $(L_1+L_2)$-layer MOIRAI transformer that, on any time series with $d \le d_{\max}$ covariates and lag $q \le q_{\max}$, predicts within $\epsilon$ of the AR fit, and the construction uses the any-variate attention's time and variate indices to select the correct covariate dimension. On the generalization side, Theorem 4.5 states that under Dobrushin's condition $\alpha(P^{(T)})<1$, the pretrained empirical risk minimizer's test loss is bounded by the best-in-class loss plus a term of order $B_x^2/(1-\alpha(P^{(T)})) \sqrt{(L(MD^2+DD')\zeta + \log(1/\epsilon))/n}$, with probability at least $1-\epsilon$. The upshot is a parameter-count-style bound that needs no i.i.d. assumption and gives $n^{-1/2}$ sample complexity when model parameters are treated as constants.

Load-bearing premise

The load-bearing premise is that the cited in-context-learning theorem transfers to this paper's ReLU-attention, rearranged-token, well-conditioned-loss setup, which the paper states as immediate corollaries without verifying the theorem's hypotheses, and that the pretraining data satisfies the equal per-time-step marginals that the generalization proof assumes in Assumption D.9 but omits from the theorem statement; if either fails, the stated bounds do not follow.

Editorial extensions

If this is right

  • A transformer pretrained on AR data should predict as if it had computed the least-squares AR fit on the given lookback; longer input sequences give more in-context examples and should lower prediction error, matching the paper's experiments.
  • MOIRAI's design choices, flattening covariates and adding time and variate indices, are justified: they are what allow a single model to handle arbitrary $d$ and $q$ without architectural changes.
  • Pretraining on weakly dependent series is learnable: under Dobrushin's condition the ERM's test error converges to the best-in-class error at rate $n^{-1/2}$.
  • The approximation error of the constructed transformer shrinks exponentially in the number of layers, $O(e^{-L})$, so modest depth suffices for small $\epsilon$.
  • The bounds cover non-i.i.d. data, so they apply to time series with mild temporal dependence, not only independent samples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not stated by the paper: if the in-context-gradient-descent mechanism is as general as the appendix's non-linear AR and maximum-likelihood results suggest, the same framework should predict which loss function a time-series foundation model implicitly optimizes, and this is testable by probing attention maps.
  • The paper's bound treats the Dobrushin coefficient as a constant; a natural extension would be to ask whether stronger decay of dependence, such as $\alpha(P)\to 0$ or explicit mixing conditions, yields faster than $n^{-1/2}$ pretraining rates.
  • The paper does not analyze patch embeddings, yet MOIRAI in practice patches inputs; the theory's token-by-token construction suggests that patching should act like a fixed linear aggregation of lagged features, which could be tested by comparing patched and unpatched models on AR data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper claims to provide a theoretical foundation for transformer-based time series foundation models. Its approximation results assert that standard transformers can fit an autoregressive model on univariate time series by implementing gradient descent in-context (Proposition 3.4), and that the MOIRAI architecture can automatically adapt to an arbitrary number of covariates (Theorem 3.8). Its generalization results give pretraining bounds when the data satisfies Dobrushin's condition, with a claimed n^{-1/2} rate (Theorem 4.5), plus AR-specific corollaries and a stationary AR(1) example. The main proofs are reductions to external results: [10] for in-context linear regression and [37] for concentration under Dobrushin's condition. Experiments on synthetic AR data compare MOIRAI with least squares and show decreasing prediction error with longer input sequences.

Significance. If the results were established, they would be among the first theoretical explanations of why transformer-based time series foundation models work, and the use of Dobrushin's condition for pretraining bounds is natural and interesting. The paper has clear strengths: explicit token-layout constructions, a substantial Lipschitz analysis for the any-variate attention, extension to non-linear AR and maximum likelihood estimation in the appendix, and experiments that qualitatively support the in-context learning story. However, the central approximation claim depends on a construction whose normalization contradicts the stated norm bound, and the generalization claim rests on an unstated equal-marginals assumption; the reductions to [10] and [37] are not verified. As a result, the main theorems are not currently established.

major comments (4)
  1. [§D.1, Lemma D.1 / Definition 2.1] Under Definition 2.1, attention has an outer normalization 1/N and σ(t)=ReLU(t)/N, so for the single active key in Lemma D.1 the attention matrix has entries 1/N, not 1. After the outer 1/N, a lagged token contributes V h_{i-m}/N^2, whereas Eq. (3.4) shows the lagged value at its original scale. To realize Eq. (3.4) the value matrices V_m would need to scale as N^2, contradicting the norm bound ||θ||_op ≤ |4R+8β^{-1}| in Proposition 3.4 and Theorem 3.8 (R is independent of N). The appendix's statement that σ(...) yields a 'rotation matrix' is therefore not correct for the normalization of Definition 2.1; Lemma 3.2 is not established as stated.
  2. [§3.1, Proposition 3.4] The proposition is claimed to follow immediately from [10, Theorem 4], but the paper never verifies the hypotheses of that theorem. First, [10] assumes the ICL token layout of Eq. (3.3) with feature/label/position blocks in each column, whereas after Lemma 3.2 the columns have the structure of Eq. (3.4) with lagged values and labels interleaved in the first q rows and a different positional block. Second, the attention map in Definition 2.1 uses ReLU with 1/N normalization, and no argument shows that this matches the attention used in [10, Theorem 4]. Third, Assumption 3.3 merely postulates that the ERM loss is α-strongly convex, β-smooth, and bounded-solution; the paper gives no verification of these properties for the finite-sample AR problem, so the abstract's unconditional existence claim is not supported by the stated conditional proposition. The same gaps transfer to Theorem 3.8 through Lemma 3.5.
  3. [§D.4, Theorem D.11 / Theorem 4.5] The proof of Theorem 4.5 invokes Theorem D.11 from [37], whose hypotheses include Assumption D.9, i.e., that for each sampled chain the marginals of (z_t) are identical across t. This assumption appears only in the appendix proof and is not stated in Theorem 4.5, nor is it verified in Corollary 4.6 for AR processes (non-stationary AR processes do not have equal time marginals). Since Theorem D.11 is the only concentration tool used, the stated generalization bound holds only under an additional, unstated hypothesis. The theorem statement also needs to relate the assumed condition α_log(P^{(T)}) < 1/2 to the α(P^{(T)}) appearing in the denominator of the bound.
  4. [§4.2, Corollary 4.6] The probability expression Δ(1−ε) with Δ = O(1 − (σ_ϵ/(B_x B_w e^{-L/(2κ)}))^2) is not a valid probability as written: if the bracketed ratio exceeds 1, Δ is negative. The corollary needs a regime condition (or a lower bound on Δ) that ensures Δ ∈ [0,1], and the parameter dependencies should be stated explicitly.
minor comments (5)
  1. [§3.2] The main text says the proof of Theorem 3.8 is in Appendix C, but the proof appears in Appendix D.2; Appendix C contains background lemmas.
  2. [§3.1, Eq. (3.6)] In Proposition 3.4, which is stated for AR_1(q), Eq. (3.6) uses the multivariate notation [x^1_{t−1:t−q}; ...; x^d_{t−1:t−q}]; either set d=1 explicitly or remove the multivariate bracket.
  3. [§4.2, Theorem 4.5] The definition of ζ contains a missing closing parenthesis: ζ = O(log(2 + max{B, R, Bx, T, d})).
  4. [Remark 3.7] Remark 3.7 asserts that Lemmas 3.2 and 3.5 generalize to softmax and linear attention 'by considering perturbations' without giving a statement or proof; this should be moved to a conjecture or supported with an argument.
  5. [Lemma D.1] The proof of Lemma D.1 is carried out for a simplified positional encoding with only zero and one-hot vectors; the passage back to the full p_i of Eq. (3.1) is asserted rather than shown and should be written out.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the central results are assembled from external theorems and explicit constructions, with only a minor non-load-bearing self-citation in the limitations.

full rationale

Walking the derivation chain, I find no load-bearing circular step. The approximation results (Proposition 3.4 and Theorem 3.8) are obtained by combining the explicit constructions in Lemma 3.2 / Lemma 3.5 with the external in-context learning theorem of Bai et al. [10, Theorem 4]. Lemma D.1 gives an explicit construction of W_K and W_Q and does not assume the target output; the reformatting lemma is a genuine construction rather than a restatement of the conclusion. The generalization bound in Theorem 4.5 is derived from the external Dobrushin-condition results of Dagan et al. [37] and Kulske [38], plus Lipschitz and covering-number estimates; no parameter is fitted to the quantity being predicted. The experiments are behavioral checks of a trained model, not fitted inputs renamed as predictions. The only self-citation is [21] in the limitations paragraph, where it is listed alongside [10, 20] as prior theoretical work using ReLU attention; it is not used in any proof and is not load-bearing. Two non-circular rigor concerns should be weighed separately: first, under Definition 2.1 the attention output carries a 1/N^2 scaling, so the proof of Lemma 3.2 may not deliver the claimed columns with the stated norm bound ||theta||_op <= |4R + 8 beta^{-1}|; second, the proof of Theorem 4.5 invokes Assumption D.9, which is absent from the theorem statement. These are correctness and verification risks, not circularity, because the derivation is not equivalent to its inputs by construction. Accordingly, the circularity score is 1.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claims rest on external black-box theorems [10, Theorem 4] and [37, Theorem 5.3], on standard smooth-function approximation (Lemma C.5), and on domain assumptions about the AR process and weak dependence. No free parameters are fitted to data, and no new entities are invented beyond the MOIRAI components taken from [1].

assumptions (6)
  • domain assumption [10, Theorem 4] (in-context gradient descent for linear regression) applies verbatim to the reformatted time series.
    Proposition 3.4 and Theorem 3.8 are stated as immediate corollaries of this black-box theorem. Its hypotheses on attention mechanism, token layout, and loss are assumed to hold for the output of Lemma 3.2/3.5 (Section 3.1).
  • domain assumption Assumption 3.3: the least-squares ERM is well-conditioned (alpha-strongly convex, beta-smooth) with bounded solution ||bw_ERM|| <= B_w/2.
    Controls the number of layers L = O(kappa log(1/epsilon)) and the norm of the constructed transformer in Proposition 3.4 and Theorem 3.8.
  • domain assumption Assumption D.9: each time step's marginal distribution is the same (equal marginals).
    Required by [37, Theorem 5.3] which Theorem 4.5's proof invokes, but omitted from the statement of Theorem 4.5 (Appendix D.4).
  • domain assumption Dobrushin's condition alpha_log(P^{(T)}) < 1/2.
    Core hypothesis of Theorem 4.5 enabling concentration of sums of weakly dependent random variables.
  • domain assumption Bounded features: ||x_{t-q:t-1}||_2 <= B_x.
    Assumed in the data model (Eq. 2.1) and used throughout Lipschitz, covering, and generalization estimates.
  • domain assumption ReLU attention with scaling 1/N can host the constructions of [10].
    Definition 2.1 sets sigma(t)=ReLU(t)/N; the paper cites [10,20,21] for the same approach but does not verify the invoked theorem's attention type (Section 2.1, Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers and Their Roles as Time Series Foundation Models." pith.science (2026). https://pith.science/paper/MC4GGVNK

@misc{pith2026250203383,
  author       = {Pith},
  title        = {Pith review of: Transformers and Their Roles as Time Series Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MC4GGVNK}},
  note         = {Machine review of arXiv:2502.03383}
}
read the original abstract

We give a comprehensive analysis of transformers as time series foundation models, focusing on their approximation and generalization capabilities. First, we demonstrate that there exist transformers that fit an autoregressive model on input univariate time series via gradient descent. We then analyze MOIRAI, a multivariate time series foundation model capable of handling an arbitrary number of covariates. We prove that it is capable of automatically fitting autoregressive models with an arbitrary number of covariates, offering insights into its design and empirical success. For generalization, we establish bounds for pretraining when the data satisfies Dobrushin's condition. Experiments support our theoretical findings, highlighting the efficacy of transformers as time series foundation models.

Figures

Figures reproduced from arXiv: 2502.03383 by the authors.

Figure 1
Figure 1. Top: Model performance on data with different number of covariates. For both MOIRAI and MOIRAI-relu, we observe their performance behave like least squares. As in our construction, the longer the lookback size is, the more examples available for transformers to fit an AR model. Note that our test data has variance σ 2 = 1, thus the MSE for both models are expected to converge to 1 as the lookback size increases. Bot… view at source ↗
Figure 2
Figure 2. We observe that when least squares regression fails to obtain the optimal error rate for prediction, [PITH_FULL_IMAGE:figures/full_fig_p034_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A hand-constructed O(ln L + T)-layer Transformer is shown to approximate low-rank hidden Markov models in-context, with lower layers extracting local history features and upper layers performing regression-based decoding.

  2. Large Causal Models for Temporal Causal Discovery

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A transformer pretrained on a large mixed corpus of synthetic and simulated realistic time series can discover lagged causal graphs zero-shot on datasets up to 12 variables, outperforming several classical baselines.

Reference graph

Works this paper leans on

42 extracted references · 21 canonical work pages · cited by 2 Pith papers

  1. [10]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024

    work page 2024

  2. [37]

    Learning from weakly dependent data under dobrushin’s condition

    Yuval Dagan, Constantinos Daskalakis, Nishanth Dikkala, and Siddhartha Jayanti. Learning from weakly dependent data under dobrushin’s condition. In Conference on Learning Theory, pages 914–928. PMLR, 2019

    work page 2019

  3. [1]

    Unified training of universal time series forecasting transformers

    Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592, 2024

    arXiv 2024

  4. [2]

    Chronos: Learning the language of time series

    Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815, 2024

    arXiv 2024

  5. [3]

    Foundation models for time series analysis: A tutorial and survey

    Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6555–6565, 2024

    2024

  6. [4]

    A decoder-only foundation model for time-series forecasting

    Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688, 2023

    arXiv 2023

  7. [5]

    Lag-llama: Towards foundation models for time series forecasting

    Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloˇ s, Hena Ghonia, Nadhir Hassen, Anderson Schneider, et al. Lag-llama: Towards foundation models for time series forecasting. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023

    work page 2023

  8. [6]

    Time series analysis

    James D Hamilton. Time series analysis. Princeton university press, 2020

    2020

Show all 42 references
  1. [7]

    Time series techniques for economists

    Terence C Mills. Time series techniques for economists. Cambridge University Press, 1990

    1990

  2. [8]

    P. L. Dobrushin. The description of a random field by means of conditional probabilities and conditions of its regularity. Theory of Probability and Its Applications, 13:197–224, 1968

    1968

  3. [9]

    Completely analytical interactions: constructive descrip- tion

    Roland L Dobrushin and Senya B Shlosman. Completely analytical interactions: constructive descrip- tion. Journal of Statistical Physics, 46:983–1014, 1987

    1987

  4. [11]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo˜ ao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174. PMLR, 2023

    2023

  5. [12]

    Transformers as algorithms: Generalization and stability in in-context learning

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pages 19565–19594. PMLR, 2023

    2023

  6. [13]

    One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention

    Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576, 2023

    2023 arXiv

  7. [14]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [15]

    Trained transformers learn linear models in-context

    Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1–55, 2024

    2024

  9. [16]

    Language models are few-shot learn- ers

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learn- ers. Advances in neural information processing systems, 33:1877–1901, 2020. 13

    1901

  10. [17]

    What can transformers learn in- context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in- context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022

    2022

  11. [18]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  12. [19]

    What learning algo- rithm is in-context learning? investigations with linear models

    Ekin Aky¨ urek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm is in-context learning? investigations with linear models. In The Eleventh International Confer- ence on Learning Representations, 2023

    2023

  13. [20]

    Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining

    Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. arXiv preprint arXiv:2310.08566, 2023

    2023 arXiv

  14. [21]

    Learning spectral methods by transformers

    Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and Han Liu. Learning spectral methods by transformers. arXiv preprint arXiv:2501.01312, 2025

    2025 arXiv

  15. [22]

    Replacing softmax with relu in vision transformers

    Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers. arXiv preprint arXiv:2309.08586, 2023

    2023 arXiv

  16. [23]

    Sparse attention with linear units

    Biao Zhang, Ivan Titov, and Rico Sennrich. Sparse attention with linear units. arXiv preprint arXiv:2104.07012, 2021

    2021 arXiv

  17. [24]

    A study on relu and softmax in transformer

    Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer. arXiv preprint arXiv:2302.06461, 2023

    2023 arXiv

  18. [25]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  19. [26]

    Strategies to leverage foundational model knowledge in object affordance grounding

    Arushi Rai, Kyle Buettner, and Adriana Kovashka. Strategies to leverage foundational model knowledge in object affordance grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1714–1723, 2024

    2024

  20. [27]

    Monash time series forecasting archive

    Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob J Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643, 2021

    2021 arXiv

  21. [28]

    Gluonts: Probabilistic and neural time series modeling in python

    Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. Journal of Machi...

    2020

  22. [29]

    Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419–22430, 2021

    2021

  23. [30]

    Modeling long-and short-term temporal patterns with deep neural networks

    Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95–104, 2018

    2018

  24. [31]

    itrans- former: Inverted transformers are effective for time series forecasting

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itrans- former: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023

    2023 arXiv

  25. [32]

    A time series is worth 64 words: Long-term forecasting with transformers

    Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022

    2022 arXiv

  26. [33]

    Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting

    Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Represen- tations, 2023. 14

    2023

  27. [34]

    How transformers learn causal structure with gradient descent

    Eshaan Nichani, Alex Damian, and Jason D Lee. How transformers learn causal structure with gradient descent. arXiv preprint arXiv:2402.14735, 2024

    2024 arXiv

  28. [35]

    How do transformers perform in-context autoregressive learning? arXiv preprint arXiv:2402.05787, 2024

    Michael E Sander, Raja Giryes, Taiji Suzuki, Mathieu Blondel, and Gabriel Peyr´ e. How do transformers perform in-context autoregressive learning? arXiv preprint arXiv:2402.05787, 2024

    2024 arXiv

  29. [36]

    High-dimensional statistics: A non-asymptotic viewpoint, volume 48

    Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019

    2019

  30. [38]

    Concentration inequalities for functions of gibbs fields with application to diffraction and random gibbs measures

    Christof K¨ ulske. Concentration inequalities for functions of gibbs fields with application to diffraction and random gibbs measures. Communications in mathematical physics, 239:29–51, 2003. 15 SUPPLEMENTARY MATERIAL A T able of Notations 16 B Related W orks 17 C Additional T...

    2003

  31. [39]

    Both the activation function r and the loss function ℓ is C 4-smooth

  32. [40]

    W is a closed domain such that W ⊂ w = [vk; uk]k∈[K] ∈ RK(d+1) : ||vk||2 ≤ Bv, |uK| ≤Bu , and ProjW = MLPθ2 for some MLP layer with hidden dimension Dw and ||θ2||op ≤ Cw. Then there exists a ( L1 + 2L2)-layer MOIRAI transformer with max ℓ∈[L1+1,2L2] M (ℓ) ≤ eO(ε−2), max ℓ∈[L1+...

  33. [41]

    to get P(T ) j , for j = 1, · · ·, n

    Sample n distributions from π i.i.d. to get P(T ) j , for j = 1, · · ·, n

  34. [42]

    We assume that for each j ∈ [n], (zj,t) has marginals equal to some distribution D for t = 1, · · ·, T

    For each distribution P(T ) j , we sample ( zj,1, · · ·, zj,T ) Assumption D.9. We assume that for each j ∈ [n], (zj,t) has marginals equal to some distribution D for t = 1, · · ·, T. We first present several lemma and theorems that will be used later. Lemma D.10 ([36, Example...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.