Pith. sign in

REVIEW 5 major objections 5 minor 67 references

Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CHARM, a 7-million-parameter model, embeds multivariate time series by conditioning on plain-language sensor descriptions, and its frozen embeddings outperform task-specific baselines on classification, forecasting, and anomaly detection.

desk verdict Novel channel-description-conditioned time series encoder, but the headline SOTA is not established because the evaluation selects test-set-best checkpoints and probe settings on the target benchmarks. read the letter →

arxiv 2505.14543 v1 pith:4WEFVKIK submitted 2025-05-20 cs.LG

Utsav Dutta , Sina Khoshfetrat Pakazad , Henrik Ohlsson This is my paper
classification cs.LG
keywords timeseriesfoundationmodelself-supervisedlearningJEPAchanneldescriptionsmultivariaterepresentationanomalydetectionforecasting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CHARM is a 7M-parameter embedding model for multivariate time series whose central idea is that the meaning of each channel, usually a short text label such as "oil temperature," should shape how the model convolves, attends, and gates over the signal, while the learned representation stays invariant to channel order. The paper claims that a single frozen encoder trained with a JEPA-style latent-prediction objective, rather than reconstruction or next-step forecasting, yields transferable embeddings that outperform previous representation-learning methods on UEA classification, SKAB anomaly detection, and ETT forecasting. The reason to care is the implied shift: time series foundation models need not be forecasting-only, and sensor metadata is not a nuisance variable but a first-class input. The model also produces interpretable cross-channel gate heatmaps that recover a known causal asymmetry in the ETT data, where the target variable attends to all inputs but the inputs do not attend back.

What carries the argument

The central object is the contextual attention layer, whose attention score for each channel-time pair adds three terms: vanilla self-attention with rotary position embeddings, a learnable channel-lag tensor $\Delta_{i,j,t} = \Delta_{j,i,-t}$ that encodes the dependency strength between channels at temporal offsets, and a gating penalty $-\lambda_G G_d[i,j]$ derived from channel descriptions. The gate is computed as $G_d = \mathrm{ReLU}(Z - S)$, where $S = E_d E_d^\top$ is the pairwise similarity of frozen description embeddings and $Z[i,j] = \sigma(E_d[i,:] W_b E_d[j,:]^\top)$ is a learned similarity threshold; a large $\lambda_G$ lets the gate act as a mask that blocks selected cross-channel interactions. A second branch generates TCN kernels and per-layer gates directly from the same text embeddings. Together these mechanisms carry the semantic injection, maintain invariance to channel ordering, and produce the interpretable gating heatmaps discussed in the paper.

What would settle it

Re-evaluate CHARM with a single fixed checkpoint (for example, the last epoch), a single SVM hyperparameter setting across all UEA datasets, and a pretraining corpus that excludes the benchmark datasets' training splits; if the claimed 1.5% classification gain, 0.08 F1 gain, and ETT error reductions shrink or reverse, the state-of-the-art claim depends on the protocol rather than on the learned representations.

Watch

Extended reading notes

Core claim

In the paper's own terms, CHARM learns shared, transferable, and domain-aware time series representations by injecting textual channel descriptions directly into the architecture. A description-aware temporal convolutional network gates each convolutional layer and generates the convolutional kernels from frozen text embeddings, and a contextual attention layer extends self-attention with two description-conditioned terms: a symmetric inter-channel time-offset bias that encodes dependencies at temporal lags, and a gating mask that suppresses cross-channel attention between channels whose descriptions are dissimilar. The model is trained with JEPA: the context encoder sees perturbed views of the input, a target encoder updated by exponential moving average sees clean views, and the loss aligns the predictor's outputs with targets at three aggregation levels. On the benchmark evaluations, the paper reports state-of-the-art embedding-model results: 0.788 average accuracy on UEA classification (1.5% above the prior embedding method), an F1 of 0.86 on SKAB anomaly detection, and substantially lower MSE and MAE than baselines on the ETT forecasting datasets.

Load-bearing premise

The reported state-of-the-art rests on the evaluation protocol: the encoder is pretrained on the training splits of the same benchmark datasets it is later evaluated on, the SVM probe is tuned separately for each classification dataset, and the best checkpoint during training is selected for the reported scores.

Editorial extensions

If this is right

  • A single frozen encoder with lightweight probes can compete with task-specific supervised models on classification, forecasting, and anomaly detection, reducing the need for per-dataset architecture and feature engineering.
  • Adding textual channel descriptions improves both downstream accuracy and an embedding-quality proxy (effective rank) compared with the same architecture without text, which suggests that richer sensor metadata directly buys representation quality.
  • The JEPA latent-prediction objective produces embeddings whose similarity structure becomes more class-aligned during training, and whose inter-channel gates recover directional causal dependencies, giving practitioners a built-in interpretability tool.
  • Using only the final time step's embedding is nearly as accurate as stacking all embeddings on some forecasting tasks, so cheap, low-latency probe inference is feasible in deployment.
  • The architecture accepts arbitrary numbers of channels and time windows because descriptions are attached per channel, so the same pretrained encoder can be applied to new sensor configurations without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to shuffle, blank, or paraphrase the channel descriptions at inference time; if the paper's mechanism is right, downstream performance should degrade measurably when descriptions stop matching the actual sensors.
  • The O(C^2 T^2) attention cost limits the approach to modest channel and window sizes, so a natural neighbor-problem test is whether the same gating idea transfers to linear-attention or state-space backbones that scale to thousands of channels.
  • Because the encoder is pretrained on the training splits of the benchmark datasets, the stronger "foundational" claim is left open: pretraining on unrelated industrial datasets and then freezing the encoder for benchmark evaluation would be the decisive test of generalization.
  • The inter-channel time-offset tensor is symmetric by construction, but real cross-channel causality can be lopsided; extending the model to learn asymmetric lag dependencies would test whether the symmetry constraint ever limits forecasting or interpretability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces CHARM, a 7M-parameter self-supervised embedding model for multivariate time series that conditions a temporal convolutional network and a transformer-style attention stack on per-channel textual descriptions. The model is trained with a JEPA-style latent prediction objective using two augmentation tasks (causal prediction and smoothing) and an ℓ1-based multi-resolution loss with regularization on learned inter-channel gating and time-offset tensors. The authors evaluate frozen CHARM embeddings on UEA classification, SKAB anomaly detection, and seven forecasting benchmarks, reporting state-of-the-art results against representation-learning and forecasting baselines.

Significance. If the reported numbers were obtained under a clean protocol, the paper would make a solid contribution: it is among the first to integrate per-channel text descriptions into a time-series representation learner, the JEPA formulation avoids raw-signal reconstruction, and the gating analyses in Figures 7 and 26 suggest interpretable learned structure. The detailed hyperparameter table (Table 4), the PyTorch-style pseudocode for the attention construction (Appendix C.5.1), and the ablation showing gains from text (Table 16) are useful for reproducibility. However, the empirical claims that carry the paper are not reliable under the protocol actually used, so the significance is contingent on a substantially revised evaluation.

major comments (5)
  1. [Appendix I.1, Tables 6-7] The reported UEA classification results are selected on the test set. The text states that the authors 'probe the model's checkpoints as it evolves during training, and report the best performing scores in Table 7' and that they 'tune the hyperparameters for each UEA dataset separately' via a grid search over the SVM's C values (Table 6). No validation split is described for either selection step. The reported average accuracy of 0.788 (vs. 0.773 for T-Rep) and the seven first-place ranks are therefore the best over checkpoints and per-dataset SVM settings, rather than the performance of a fixed configuration. Because the comparison baselines report a single trained model without this selection, the +1.5% margin is not an unbiased estimate of relative performance.
  2. [Appendix I.3, Tables 11-12] The forecasting results are obtained with similar test-set selection. Table 11 reports 'the mean and standard deviation of the best performing scores over 5 runs,' which is a best-of-5 selection; in addition, Table 12 is used to pick the pooling/head configuration separately for each dataset based on test error. The baselines in Table 11 are taken from published results without an equivalent selection procedure. Consequently, the large MSE/MAE gaps on the ETT datasets (e.g., 0.436 vs. 0.763 for T-Rep on ETTh1) are not established as genuine improvements, and the reported standard deviations do not account for selection over runs and probe configurations.
  3. [Appendix G, H] The self-supervised pretraining corpus includes the train splits of the downstream evaluation benchmarks (UEA, ETT, Weather, Exchange Rate, Illness, SKAB), as stated in Appendix G and H. The model therefore has seen the training distribution of every evaluation dataset during pretraining, whereas the comparison representation learners are dataset-specific and are trained only on the target dataset. This setup cannot support the paper's 'foundation model' claim: it does not demonstrate cross-dataset generalization. An evaluation on held-out datasets that are absent from the pretraining corpus is needed, or the claims must be reframed as in-domain pretraining with a restricted scope.
  4. [Table 1a, Appendix I.2] The SKAB anomaly-detection comparison is also confounded by the same in-domain pretraining: SKAB appears in the pretraining corpus (Appendix G), and the linear reconstruction probe is trained on the SKAB train split before evaluation on the test split. The baselines are autoencoder models trained directly on the benchmark. The reported F1 improvement of 0.08 (0.86 vs. 0.78) is therefore not evidence that CHARM's representations transfer to anomaly-detection settings it has not seen during pretraining.
  5. [Abstract, Section 1.1] The abstract and introduction claim that CHARM 'achieves state-of-the-art performance across diverse downstream tasks, setting a new benchmark for time series representation learning.' Given the protocol issues detailed above, this claim is unsupported. Furthermore, on the UEA classification benchmark the comparison set contains only dataset-specific representation learners, while the other foundation-type models considered in the paper (MOMENT, UniTS) are compared only on forecasting. The 'new benchmark' claim is broader than the evidence presented.
minor comments (5)
  1. [Appendix I.1] The text says the evaluation uses 'a subset of 15 UEA datasets,' but Table 7 contains 16 datasets; the count should be corrected.
  2. [Appendix I.3] The sentence 'We report the mean and standard deviation of the best performing scores over 5 runs' is ambiguous; if the mean over runs is intended, the word 'best' should be removed, and if a selection over runs is intended, the selection criterion should be stated.
  3. [Section 2.2.3] The notation µ_j for the channel-mean function is confusing because the subscript j is also the summation index; a neutral symbol such as µ_c or an overbar would be clearer.
  4. [Table 4] The hyperparameter row 'scale factor 2 1.25' appears to contain a formatting error; the intended key and value should be clarified.
  5. [Appendix F] The limitations section mentions computational cost and dependence on description quality, but it does not acknowledge the test-set selection in the evaluation protocol or the use of downstream benchmark train splits during pretraining; these are the most consequential limitations of the present empirical study.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline SOTA numbers are selected over test-set configurations (checkpoint, SVM C, runs); the JEPA pretraining itself is not circular.

  1. fitted input called prediction [Appendix I.1, Tables 6/7; Table 1b]
    "To ensure a fair comparison with other representation learning models, we use a Support Vector Machine ... and perform a grid search over the SVM's hyperparameters Table 6 for each dataset. We tune the hyperparameters for each UEA dataset separately. Additionally, we probe the model's checkpoints as it evolves during training, and report the best performing scores in Table 7."

    The reported UEA average accuracy (0.788, with 7 firsts) is not the accuracy of a fixed encoder/probe: the SVM cost C is grid-searched per dataset and the training checkpoint is selected by test accuracy. The headline number is therefore, by construction, the maximum over test-evaluated configurations, and the claimed +1.5% margin over T-Rep is a selected optimum rather than an unbiased prediction. This is a fitted input (C, checkpoint) renamed as the model's predictive accuracy.

  2. fitted input called prediction [Appendix I.3, Table 11]
    "We report the mean and standard deviation of the best performing scores over 5 runs in Table 11."

    The ETT MSE/MAE values (e.g., average MSE 0.333 vs T-Rep 0.986) are the best over repeated stochastic runs, so the reported error is a minimum-type statistic chosen from test evaluations rather than the expected performance of a single run. This selection mechanically lowers the reported error and inflates the claimed forecasting advantage; the number is constructed as an optimum over runs, not predicted.

full rationale

The core methodological derivation is not circular: the JEPA-style self-supervised loss, the description-conditioned TCN, and the inter-channel attention/gating modules are defined from first principles in the paper, and the pretraining objective does not use downstream labels. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in through citation. The circularity burden is confined to the empirical SOTA claims: the UEA classification and ETT forecasting numbers are obtained by selecting the best test-set configuration (SVM C, checkpoint, or run), so those reported predictions are partially forced by the evaluation protocol rather than by the learned representation alone. The additional disclosure that the model is pretrained on the training splits of the same benchmarks (Appendix G/H) weakens the 'foundation transfer' interpretation but is not by itself a construction-level circularity. Overall, the architectural contribution has independent content, but the headline quantitative claims reduce in part to test-set selection, giving a partial-circularity score of 6.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims rest mainly on modeling assumptions (informative channel descriptions, symmetric inter-channel dependencies) and on an evaluation protocol that selects the best checkpoint and probe settings on the test set. No new physical entities are introduced.

free parameters (4)
  • Loss weights λ1, λ2 = 1e-5 each
    Regularization weights for gating sparsity and temporal-offset consistency, set by hand (Table 4) with no sensitivity analysis.
  • Gating scalar λG = not specified
    Multiplies the inter-channel gate in softmax attention (Section 2.1.2); the paper only says it is 'typically a large positive number'.
  • SVM C per UEA dataset = grid {1e-4 ... 1e4}
    The downstream probe hyperparameter is tuned per dataset (Appendix I.1), contributing to the selection-bias issue.
  • JEPA augmentation ranges (Cmin, Cmax, Tmin, Tmax) = 0.3, 0.4, 0.1, 0.2
    Masking fractions chosen by hand (Table 4); no ablation is reported for these values.
assumptions (3)
  • domain assumption Channel descriptions are available, aligned with channel order, and semantically informative for every input (Section B: 'We assume that the descriptions in D are aligned with the corresponding columns of T').
    The whole text-conditioning machinery has nothing to condition on if descriptions are missing or uninformative; the authors note in Limitations that this is a bottleneck.
  • ad hoc to paper Inter-channel dependencies at equal and opposite lags are symmetric, Δ[i,j,t] = Δ[j,i,-t] (Section 2.1.2).
    Enforced by construction; may not hold for causal relationships where one sensor drives another with a delay (the authors' own gate analysis in Figure 27 shows asymmetric dependence).
  • domain assumption The self-distillation / EMA JEPA objective with multi-scale ℓ1 loss produces transferable representations that linear probes can exploit (Sections 2.2.2 and 2.2.3).
    Standard SSL assumption; not proven for time series beyond the empirical results, whose validity is in question.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions." pith.science (2026). https://pith.science/paper/4WEFVKIK

@misc{pith2026250514543,
  author       = {Pith},
  title        = {Pith review of: Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4WEFVKIK}},
  note         = {Machine review of arXiv:2505.14543}
}
abstract

Traditional time series models are task-specific and often depend on dataset-specific training and extensive feature engineering. While Transformer-based architectures have improved scalability, foundation models, commonplace in text, vision, and audio, remain under-explored for time series and are largely restricted to forecasting. We introduce $\textbf{CHARM}$, a foundation embedding model for multivariate time series that learns shared, transferable, and domain-aware representations. To address the unique difficulties of time series foundation learning, $\textbf{CHARM}$ incorporates architectural innovations that integrate channel-level textual descriptions while remaining invariant to channel order. The model is trained using a Joint Embedding Predictive Architecture (JEPA), with novel augmentation schemes and a loss function designed to improve interpretability and training stability. Our $7$M-parameter model achieves state-of-the-art performance across diverse downstream tasks, setting a new benchmark for time series representation learning.

Figures

Figures reproduced from arXiv: 2505.14543 by the authors.

Figure 1
Figure 1. Overview of the model architecture, featuring a context-aware temporal convolutional network and a series of contextual attention layers, each guided by textual descriptions of the input time series channels. selectively integrate signals in a structured manner, while maintaining invariance to channel ordering. See Section 2.1.2 for details. Self-supervised training with JEPA for time series We adapt the JEPA to the… view at source ↗
Figure 2
Figure 2. Schematic of the context-aware temporal convolutional network, performing initial featurization of multivariate time series inputs guided by granular textual descriptions of each channel. as Gc = sigmoid(EdWg),Wg ∈ R De×N , with N denoting the number of stacked convolutional layers in the TCN. Each element Gc[i, j], which corresponds to the soft gate associated with channel i and layer j of the TCN which is then inc… view at source ↗
Figure 3
Figure 3. Description-aware gating mecha￾nism, selectively suppressing cross-channel attention. 2𝐷! 𝐶 𝐶 ! Ed[i, :], Ed[j, :]" 𝐷! 𝐶 Reshape and Concat Linear Construct Projection 2𝑇 𝐶 𝐶 ∆i,j,t = ∆j,i,−t Inter-channel Time-offset Attention [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (21 more)
Figure 5
Figure 5. Figure 5: Overview of JEPA, showing its three encoders and how they process augmented views of the same data instance. with normalization computed along the embedding dimension only. The complete architecture is denoted as Eθ, such that Y = Eθ(T, D, pos). While this outlines the…
Figure 6
Figure 6. Figure 6: Comparison of our model’s errors (MSE, MAE) vs baselines for 4/7 forecasting datasets Dataset Metric Best Baseline (Value) CHARM Absolute ∆ ETT (Avg.) MSE T-Rep (0.986) 0.333 -0.653 ETT (Avg.) MAE T-Rep (0.702) 0.358 -0.344 Weather MSE MOMENTLP (0.228) 0.244 +0.016 Wea…
Figure 7
Figure 7. Figure 7: Evolution of Channel Gates for the ETT Dataset. A causal structure evolves over training, where the target causal variable Oil Temperature attends to all other independent channels but not vice versa. Extended discussion on evolution of channel gates can be found in Ap…
Figure 9
Figure 9. Figure 9: Fast attention-weight matrix construction 1 def build_attention_weight_matrix_fast ( time_deltas : Tensor , 2 T_proj : Tensor ) -> Tensor : 3 """ 4 Block - wise assembly via tensor indexing and reshape . 5 """ 6 B , C , _ , T1 = T_proj . shape 7 T = time_deltas . size …
Figure 10
Figure 10. Figure 10: JEPA Tasks Visualized : Causal Prediction (left) Smoothing (right) [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Context and Target Network 1 class ContextTgtEncoder : 2 def forward ( self , x , ctx_idx ) : 3 """ 4 x : [... , T, C] 5 """ 6 x = self . featurizer ( x ) 7 for layer in self . encoder_layers : 8 x = layer (x , ctx_idx ) 9 return x [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 12
Figure 12. Figure 12: , and [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 14
Figure 14. Figure 14: Ablation analysis of adding text layers across tasks: (a) classification (UEA), (b) forecasting (ETTh1), (c) representation learning (LIDAR). Metric w/ text w/o text % improvement ETTh1 MSE ↓ 0.48 0.50 +4.00% # correct (UEA) ↑ 4768 4696 +1.53% LIDAR ↑ 4.61 3.34 +38.02…
Figure 15
Figure 15. Figure 15: Illness Forecasts [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Illness Forecasts 29 [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: ETTh1 Forecasts [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: ETTh1 Forecasts 30 [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: ETTh2 Forecasts [PITH_FULL_IMAGE:figures/full_fig_p031_19.png]
Figure 20
Figure 20. Figure 20: ETTh2 Forecasts 31 [PITH_FULL_IMAGE:figures/full_fig_p031_20.png]
Figure 21
Figure 21. Figure 21: ETTm1 Forecasts [PITH_FULL_IMAGE:figures/full_fig_p032_21.png]
Figure 22
Figure 22. Figure 22: ETTm1 Forecasts 32 [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]
Figure 23
Figure 23. Figure 23: Evolution of BasicMotions similarity heatmaps over training epochs (a) Epoch 0 (b) Epoch 3 (c) Epoch 6 (d) Epoch 9 [PITH_FULL_IMAGE:figures/full_fig_p033_23.png]
Figure 24
Figure 24. Figure 24: Evolution of Skoltech Anomaly Benchmark similarity heatmaps over training epochs (a) Epoch 0 (b) Epoch 3 (c) Epoch 6 (d) Epoch 9 [PITH_FULL_IMAGE:figures/full_fig_p033_24.png]
Figure 25
Figure 25. Figure 25: Evolution of Epilepsy similarity heatmaps over training epochs 7The heatmaps have a block structure because the labels are grouped together on each axis before plotting. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_25.png]
Figure 26
Figure 26. Figure 26: Evolution of inter channel gates during training. Checkpoints extracted at epoch=0;step=49, epoch=0;step=499, epoch=0;step=999, epoch=2;step=49, epoch=6;step=49, epoch=8;step=49. Each row represents a particular dataset. Each column represents a sampled checkpoint as …
Figure 27
Figure 27. Figure 27: Evolution of Channel Gates for the ETT Dataset The ETT dataset introduced by (Zhou et al., 2021a) comprises seven variables: High Useful Load, Middle Useful Load, Low Useful Load, High Useless Load, Middle Useless Load, Low Useless Load, and Oil Temperature. Among the…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 31 canonical work pages

  1. [1]

    Chronos: Learning the language of time series

    Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024

    work page 2024

  2. [2]

    ViViT : A video vision transformer, 2021

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. ViViT : A video vision transformer, 2021. URL https://arxiv.org/abs/2103.15691

    arXiv 2021

  3. [3]

    Self-supervised learning from images with a joint-embedding predictive architecture, 2023

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture, 2023. URL https://arxiv.org/abs/2301.08243

    arXiv 2023

  4. [4]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 0 12449--12460, 2020

    2020

  5. [5]

    The UEA multivariate time series classification archive, 2018, 2018

    Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018, 2018. URL https://arxiv.org/abs/1811.00075

    arXiv 2018

  6. [6]

    An empirical evaluation of generic convolutional and recurrent networks for sequence modeling

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018

    arXiv 2018

  7. [7]

    A cookbook of self-supervised learning, 2023

    Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learning, 2023. URL h...

    arXiv 2023

  8. [8]

    Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features

    Adrien Bardes, Jean Ponce, and Yann LeCun. Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features. arXiv preprint arXiv:2307.12698, 2023. URL https://doi.org/10.48550/arXiv.2307.12698

    arXiv doi:10.48550/arxiv.2307.12698 2023

Show all 67 references
  1. [9]

    Revisiting feature prediction for learning visual representations from video, 2024

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video, 2024. URL https://arxiv.org/abs/2404.08471

    2024 arXiv

  2. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  3. [11]

    Learning contrastive embedding in low-dimensional space

    Shuo Chen, Chen Gong, Jun Li, Jian Yang, Gang Niu, and Masashi Sugiyama. Learning contrastive embedding in low-dimensional space. Advances in Neural Information Processing Systems, 35: 0 6345--6357, 2022

    2022

  4. [12]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607, 2020

    2020

  5. [13]

    Debiased contrastive learning

    Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in neural information processing systems, 33: 0 8765--8775, 2020

    2020

  6. [14]

    A decoder-only foundation model for time-series forecasting

    Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp.\ 10148--10167, 2024

    2024

  7. [15]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  8. [16]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang...

    2019

  9. [17]

    Deep learning for event-driven stock prediction

    Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. Deep learning for event-driven stock prediction. In IJCAI, volume 15, pp.\ 2327--2333, 2015

    2015

  10. [18]

    Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition

    Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 5884--5888, 2018. doi:10.1109/ICASSP.2018.8462506

    2018

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021 arXiv

  12. [20]

    Video representation learning with joint-embedding predictive architectures

    Katrina Drozdov, Ravid Shwartz-Ziv, and Yann LeCun. Video representation learning with joint-embedding predictive architectures. arXiv preprint arXiv:2412.10925, 2024. URL https://doi.org/10.48550/arXiv.2412.10925

    2024 doi arXiv

  13. [21]

    PyTorch Lightning , March 2019

    William Falcon and The PyTorch Lightning team . PyTorch Lightning , March 2019. URL https://github.com/Lightning-AI/lightning

    2019

  14. [22]

    T-Rep : Representation learning for time series using time-embeddings

    Archibald Felix Fraikin, Adrien Bennetot, and Stephanie Allassonniere. T-Rep : Representation learning for time series using time-embeddings. In The 12th International Conference on Learning Representations, 2024

    2024

  15. [23]

    Unsupervised scalable representation learning for multivariate time series

    Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in neural information processing systems, volume 32, pp.\ 4650--4661, 2019

    2019

  16. [24]

    UniTS: a unified multi-task time series model, 2024

    Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. UniTS: a unified multi-task time series model, 2024. URL https://arxiv.org/abs/2403.00131

    2024 arXiv

  17. [25]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.\ 249--256. JMLR Workshop and Conference Proceedings, 2010

    2010

  18. [26]

    AST: audio spectrogram transformer, 2021

    Yuan Gong, Yu-An Chung, and James Glass. AST: audio spectrogram transformer, 2021. URL https://arxiv.org/abs/2104.01778

    2021 arXiv

  19. [27]

    MOMENT: a family of open time-series foundation models, 2024

    Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. MOMENT: a family of open time-series foundation models, 2024. URL https://arxiv.org/abs/2402.03885

    2024 arXiv

  20. [28]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch \'e , Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neur...

    2020

  21. [29]

    Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R \'e mi Munos, and Michal Valko

    Jean-Bastien Grill, Florian Strub, Florent Altch \'e , Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R \'e mi Munos, and Michal Valko. Bootstrap your own...

    2006 doi arXiv

  22. [30]

    Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network

    Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature medicine, 25 0 (1): 0 65--69, 2019

    2019

  23. [31]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp.\ 1026--1034, 2015

    2015

  24. [32]

    SAMformer : Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention, 2024

    Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas, and Ievgen Redko. SAMformer : Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention, 2024. URL https://...

    2024 arXiv

  25. [33]

    Neural machine translation in linear time

    Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099, 2016

    2016 arXiv

  26. [34]

    Katser and Vyacheslav O

    Iurii D. Katser and Vyacheslav O. Kozitsin. Skoltech anomaly benchmark (SKAB) . https://www.kaggle.com/dsv/1693952, 2020

    2020

  27. [35]

    A comprehensive survey of deep learning for time series forecasting: Architectural diversity and open challenges

    Jongseon Kim, Hyungjoon Kim, HyunGi Kim, Dongjun Lee, and Sungroh Yoon. A comprehensive survey of deep learning for time series forecasting: Architectural diversity and open challenges. Artificial Intelligence Review, 58, 2025

    2025

  28. [36]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 4015--4026, 2023

    2023

  29. [37]

    A path towards autonomous machine intelligence

    Yann LeCun. A path towards autonomous machine intelligence. Open Review, 62 0 (1): 0 1--62, 2022

    2022

  30. [38]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing...

    2020

  31. [39]

    Temporal convolutional attention neural networks for time series forecasting

    Yang Lin, Irena Koprinska, and Mashud Rana. Temporal convolutional attention neural networks for time series forecasting. In 2021 International joint conference on neural networks (IJCNN), pp.\ 1--8, 2021

    2021

  32. [40]

    iTransformer : Inverted transformers are effective for time series forecasting

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. iTransformer : Inverted transformers are effective for time series forecasting. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024

    2024

  33. [41]

    Yiqin Nie, Zhaohan Wang, Zhiwei Li, Yujing Zhang, Yao Zhang, Yuxuan Wang, Yafeng Wang, and Philip S. Yu. A time series is worth 64 words: Long-term forecasting with transformers. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023

    2023

  34. [42]

    Nomic embed: Training a reproducible long context text embedder

    Zach Nussbaum, John Xavier Morris, Andriy Mulyar, and Brandon Duderstadt. Nomic embed: Training a reproducible long context text embedder. Transactions on Machine Learning Research, 2025

    2025

  35. [43]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

    2025 arXiv

  36. [44]

    Dinov2: learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  37. [45]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp.\ 1310--1318, 2013

    2013

  38. [46]

    Pytorch: An imperative style, high-performance deep learning library, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019 arXiv

  39. [47]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf, 2018. OpenAI

    2018

  40. [48]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

    2021

  41. [49]

    DeepAR : Probabilistic forecasting with autoregressive recurrent networks

    David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR : Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36: 0 1181--1191, 2020

    2020

  42. [50]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  43. [51]

    Machine learning for predictive maintenance: A multiple classifier approach

    Gian Antonio Susto, Andrea Schirru, Simone Pampuri, Se \'a n McLoone, and Alessandro Beghi. Machine learning for predictive maintenance: A multiple classifier approach. IEEE transactions on industrial informatics, 11 0 (3): 0 812--820, 2014

    2014

  44. [52]

    Susskind, and Etai Littwin

    Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint embedding ssl architectures, 2023. URL https://arxiv.org/abs/2312.04000

    2023 arXiv

  45. [53]

    Unsupervised representation learning for time series with temporal neighborhood coding

    Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations (ICLR), 2021

    2021

  46. [54]

    Universal time-series representation learning: A survey, 2024

    Patara Trirat, Yooju Shin, Junhyeok Kang, Youngeun Nam, Jihye Na, Minyoung Bae, Joeun Kim, Byunghyun Kim, and Jae-Gil Lee. Universal time-series representation learning: A survey, 2024. URL https://arxiv.org/abs/2401.03717

    2024 arXiv

  47. [55]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pp.\ 5998--6008, 2017

    2017

  48. [56]

    Khang H. N. Vo, Duc P. T. Nguyen, Thong Nguyen, and Tho T. Quan. Ti-jepa: An innovative energy-based joint embedding strategy for text-image multimodal systems. arXiv preprint arXiv:2503.06380, 2025. URL https://doi.org/10.48550/arXiv.2503.06380

    2025 doi arXiv

  49. [57]

    Unified training of universal time series forecasting transformers

    Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp.\ 53140--53164, 2024

    2024

  50. [58]

    Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting

    Haixu Wu, Yao Xu, Jindong Wang, Guodong Long, Chengqi Wang, and Lijun Yao. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, volume 34, pp.\ 22419--22430, 2021

    2021

  51. [59]

    TS2Vec : Towards universal representation of time series

    Ziheng Yue, Yuxuan Zhang, Yifan Sun, Yifan Wang, and Zenglin Huang. TS2Vec : Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 8075--8083, 2022

    2022

  52. [60]

    Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting

    Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023

    2023

  53. [61]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), pp.\ 11106--11115, 2021 a

    2021

  54. [62]

    Informer: Beyond efficient transformer for long sequence time-series forecasting, 2021 b

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, 2021 b . URL https://arxiv.org/abs/2012.07436

    2021 arXiv

  55. [63]

    FEDformer : Frequency enhanced decomposed transformer for long-term series forecasting, 2022

    Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer : Frequency enhanced decomposed transformer for long-term series forecasting, 2022. URL https://arxiv.org/abs/2201.12740

    2022 arXiv

  56. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  57. [65]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  58. [66]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  59. [67]

    u ttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rockt \

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.