Pith. sign in

REVIEW 3 major objections 5 minor 19 references

Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No amount of training data lets a randomly initialized model match a pretrained-language initialization on time series forecasting.

desk verdict Solid empirical core — pretrained T5 beats random init for time series in the low-data regime — but the non-vanishing transfer gap is an extrapolation from finite training curves, not a demonstrated result. read the letter →

arxiv 2506.21570 v1 pith:4WJ5AIG3 submitted 2025-06-12 cs.CL cs.AIcs.LG

Roland Riachi , Kashif Rasul , Arjun Ashok , Prateek Humane , Alexis Roger , Andrew R. Williams , Yuriy Nevmyvaka , Irina Rish This is my paper
classification cs.CLcs.AIcs.LG
keywords transferlearningtimeseriesforecastinglanguagemodelinitializationT5tokenizationscalinglawseffectivelow-dataregime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning a T5 text-to-text transformer on univariate time series gives better validation loss when the backbone starts from pretrained language weights rather than random weights, and the paper claims this advantage is permanent. Across three tokenizers (raw values, lag vectors, and bins) and three backbone sizes (60M, 220M, 770M), the pretrained-initialized models eventually improve smoothly while the randomly initialized models plateau. The paper defines an 'effective transfer' measure—how many extra training tokens a random model would need to match the pretrained model's loss—and finds it grows without bound. This contradicts the scaling-law expectation, from Hernandez et al. (2021), that the transfer gap should vanish as relevant training data increases. If correct, the result means language pretraining changes the loss landscape itself, not just the starting point.

What carries the argument

The central object is the effective transfer measure $D_T(\ell) = L_R^{-1}(\ell) - L_P^{-1}(\ell)$, the difference in the number of time series training tokens needed for a randomly initialized model and a pretrained-language-initialized model to reach the same validation loss $\ell$. When the random model's loss converges first, the random inverse curve hits a floor and $D_T(\ell)$ blows up, making the non-vanishing transfer gap precise. The argument is carried by controlled comparisons of three backbone initializations (random, pretrained T5, instruction-tuned Flan-T5), three tokenizers (naive, lag, bin), and three T5 sizes, trained on the LOTSA dataset with a shared output head from Chronos-style bin forecasting.

What would settle it

Extend randomly initialized T5 training on the same LOTSA data to several times the current token budget with a wider hyperparameter search; if the validation loss resumes a smooth decrease and eventually reaches or crosses the pretrained-initialized models' loss, the claimed permanent transfer gap is refuted.

Watch

Extended reading notes

Core claim

The central claim is that the benefit of initializing a time series forecaster from pretrained language weights is asymptotic, not just a finite-data head start. The paper shows that on the LOTSA training distribution, T5 models fine-tuned from T5-Efficient weights keep decreasing in validation loss long after identically sized models trained from random weights have converged, so the gap between the two curves does not close as more time series tokens are seen. The paper quantifies this as effective transfer, the extra data a randomly initialized model would need to reach a given loss level, and reports vertical asymptotes in this quantity—the formal symptom of a transfer gap that no amount of additional tokens can close. It further reports that instruction-tuned Flan-T5 initialization transfers worse than pretraining-only T5, despite being better on language tasks, and that larger pretrained backbones help while larger random backbones hurt.

Load-bearing premise

The load-bearing assumption is that the randomly initialized models have truly converged when their validation loss flattens, rather than only slowing down within the training budget used; if longer training or better hyperparameters would make them improve again, the non-vanishing transfer gap could disappear.

Editorial extensions

If this is right

  • Pretrained-language initialization lowers validation loss for time series forecasting compared with random initialization across all tested tokenizers and backbone sizes in the low-data regime.
  • The advantage persists at the end of training, so the benefit is not simply a faster start but a lower asymptotic loss curve.
  • Upstream instruction tuning does not help transfer: Flan-T5 underperforms the pretraining-only T5 despite stronger language performance, indicating that better language scores do not imply better time series transfer.
  • Larger pretrained backbones improve downstream loss, while larger randomly initialized backbones worsen it, so scale and initialization interact.
  • Bin tokenization, which turns forecasting into a classification problem over quantized values, gives the smoothest convergence and lowest loss among the tokenizers tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transfer gap is truly asymptotic for T5, a natural next test is whether it appears for other language backbones and for other continuous-signal modalities such as EEG or sensor data.
  • Because the random models' plateau is the load-bearing observation, the paper's claim would be strengthened by training random initializations for several times longer and with a wider hyperparameter sweep to show the plateau is not a training-budget artifact.
  • The success of bin tokenization suggests the underlying advantage may be alignment with the discrete-input inductive bias of language models; one could test this by comparing against an encoder pretrained on unrelated discrete sequences rather than natural language.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies whether initializing a T5 encoder-decoder with pre-trained language-model weights helps when fine-tuning on univariate time series forecasting. The authors compare random, T5-pretrained, and Flan-T5 initializations across three tokenizers (naive, lag, bin) and three model sizes, and they define an 'effective transfer' metric based on the difference in training tokens needed to reach a given validation loss. Their central claim is that the validation-loss advantage of pre-trained initialization is non-vanishing: no amount of additional training tokens would let a randomly initialized model match a pre-trained initialization, contrary to the transfer-gap scaling law of Hernandez et al. (2021). They also report that instruction-tuned Flan-T5 underperforms the purely pre-trained T5 initialization for this task.

Significance. If the non-vanishing transfer gap were established, it would be an interesting counterexample to the vanishing-gap scaling law of Hernandez et al. (2021) and would suggest that language pretraining imparts an asymptotic inductive bias for time series modeling. The paper's controlled comparison of tokenizers, initializations, and model sizes, along with the proposed effective-transfer metric, is a useful empirical contribution. The authors are appropriately cautious in the conclusion about the limited T5 family and about distinguishing 'more data' from 'structural similarity.' However, the headline asymptotic claim is not supported by the presented finite-budget training curves, so the paper currently overstates its main contribution.

major comments (3)
  1. [Section 3, Figures 1-3; Appendix D] The central claim that 'no amount of additional tokens' would let a randomly initialized model match a pre-trained-language initialization is an asymptotic statement, but the evidence consists entirely of finite-budget training curves. The randomly initialized models are described as having 'converged at a higher loss value and earlier,' yet the paper does not report the total number of training tokens, a convergence criterion, loss derivatives, gradient norms, or any learning-rate restarts. Transformer training from scratch is known to exhibit long plateaus before later improvement, so the observed plateau does not establish true convergence. The vertical asymptotes in Figure 2(b) and the effective-transfer definition in Appendix D occur precisely when L_R^{-1}(ℓ) is undefined because the randomly initialized model never reached that loss level within the tested token budget; this only shows that the random model did not reach that loss in the experiment, not that it cannot. Please either add long-horizon training with schedule restarts and convergence diagnostics, or explicitly reframe the contribution as a finite-budget statement and soften the claimed contradiction with Hernandez et al. (2021).
  2. [Section 2 and Appendix B] The hyperparameter comparison is not a sufficient control for the asymptotic claim. Appendix B sweeps learning rate, batch size, weight decay, and warm-up duration, but it does not vary total training duration or use learning-rate restarts after apparent convergence. Since the paper's headline is about whether random initialization can eventually catch up, fixing a single budget and visually declaring convergence is not a fair test of that question. In addition, the main figures (Figures 1-3) show no error bars or per-seed variability despite the statement that runs used three random seeds; Appendix B reports standard deviations only for the hyperparameter sweep. The paper should report seed variability in the main comparisons, at least for the figures that support the 'holds across design choices' claim.
  3. [Appendix D] The effective-transfer metric D_T(ℓ) is defined through inverse functions L_R^{-1}(ℓ) and L_P^{-1}(ℓ), which require the validation-loss curves to be monotonic in training tokens. The paper does not establish monotonicity, and the Flan-T5 curve is explicitly described as 'much noisier and spikier' in Section 3. Without a stated procedure for inverting non-monotonic curves (e.g., fitting a smooth monotone curve and taking the first hitting time), the metric is not well-defined for all loss levels and the quantitative claims in Figure 2(b) are not reproducible. Please specify the inversion procedure and restrict D_T to loss levels at which the inverse is unambiguous.
minor comments (5)
  1. [Figure 2(a)] The caption says 'lower loses'; this should be 'lower losses.'
  2. [Section 2.2] The lag tokenizer dimension is not stated; given s_t = [x_t, x_{t-ℓ_1}, ..., x_{t-ℓ_p}]^T, d_token = p+1, and the lag set L used in the experiments should be specified.
  3. [Section 2] The paper refers throughout to 'validation loss' but never defines the loss function used for evaluation; please state whether it is the negative log-likelihood of the predictive distribution and describe the validation split.
  4. [Section 3] The sentence 'This behavior contrasts with Hernandez et al. (2021)' should be reconciled with the later sentence 'Consistent with Hernandez et al. (2021)' in the same section; the paper should clarify that the contrast concerns the vanishing of the transfer gap with data, not the model-size dependence.
  5. [Section 2.1] The exact pre-trained checkpoints are not specified (e.g., specific T5-Efficient and Flan-T5 model identifiers); please list them for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the transfer-gap claim is an empirical reading of measured loss curves against a random baseline, not a fitted or self-citation-derived result.

full rationale

The paper's central claim is that a non-vanishing transfer gap exists between language-initialized and randomly initialized models. This claim is operationalized in Appendix D through the definition D_T(l) = L_R^{-1}(l) - L_P^{-1}(l), where L_R and L_P are the measured validation-loss curves. The 'no amount of additional tokens' assertion follows from the observed early flattening of the random-init curves, not from a fitted parameter that bakes in the conclusion. The comparison is external: randomly initialized T5 baselines are trained under the same protocol and compared against pretrained-initialization runs. Self-citations appear (Bayazi et al. for the T5 backbone choice, Rasul et al. for lag tokenization), but they justify design choices and baselines, not the load-bearing inference about the transfer gap; that inference rests directly on the reported loss curves. Whether the random-init plateaus are true convergence or merely finite-budget artifacts is a correctness and evidence-robustness concern, not a circularity concern. No equation reduces to its own input, and no prediction is forced by construction. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or forces. Its central claims rest on an empirical comparison, a chosen metric (validation loss), and a set of design choices (tokenizers, embedding initialization, training budget). The most load-bearing assumption is that a plateau in random-init validation loss is convergence, which underpins the 'no amount of tokens' claim.

free parameters (4)
  • Binning range a for bin tokenizer
    The bin tokenizer uses B=4096 bins in range [-a, a]; the value of a is not reported in the main text or appendices and affects the loss curves.
  • Lag indices L for lag tokenizer
    The lag tokenizer depends on a set of lag indices L inherited from Lag-Llama (Rasul et al., 2024); the specific indices are not listed in this paper.
  • Training duration (number of time series tokens)
    The paper reports loss as a function of training tokens but never states how long training ran; the 'asymptotic' claims depend on this.
  • Final training hyperparameters (learning rate, batch size, weight decay, warmup)
    Appendix B lists the sweep ranges but not the chosen configuration for the main results; the paper says a sweep was done but does not report the selected values.
assumptions (4)
  • domain assumption The randomly initialized baseline trained with the chosen hyperparameters has converged when its validation loss plateaus during the training budget.
    The non-vanishing transfer gap claim requires interpreting a plateau as asymptotic convergence; a slowly decreasing loss that resumes after longer training would invalidate the 'no amount of tokens' statement. This enters in Section 3 and Appendix D.
  • domain assumption Validation loss on the LOTSA forecasting task is a faithful proxy for forecasting ability.
    All claims about transfer quality are stated in terms of validation loss on the training distribution; no downstream forecasting metrics are reported.
  • domain assumption The T5-Efficient encoder-decoder architecture is a representative language backbone for studying transfer.
    The authors state in Section 2 that the choice of backbone matters and that a systematic ablation is out of scope; all conclusions are limited to the T5 family.
  • ad hoc to paper Initializing the embedding matrix for continuous tokens to the mean vocabulary vector is a reasonable way to reuse pretrained weights.
    Appendix C.2 defines this initialization; it is a design choice that could either help or handicap the pretrained models relative to random, and its fairness is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting." pith.science (2026). https://pith.science/paper/4WJ5AIG3

@misc{pith2026250621570,
  author       = {Pith},
  title        = {Pith review of: Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4WJ5AIG3}},
  note         = {Machine review of arXiv:2506.21570}
}
read the original abstract

Recent works have demonstrated the effectiveness of adapting pre-trained language models (LMs) for forecasting time series in the low-data regime. We build upon these findings by analyzing the effective transfer from language models to time series forecasting under various design choices including upstream post-training, time series tokenizer and language backbone size. In the low-data regime, these design choices have a significant impact on the validation loss, with clear-cut choices that outperform others. Contrary to Hernandez et al. (2021), we observe that the validation loss of the LMs continues to smoothly decrease long after the validation loss of the randomly initialized models has converged, leading to a non-vanishing transfer gap that holds across design choices. These findings not only help shed light on the effective use of compute-efficient training for time series, but also open the way for the study of modality-agnostic properties of data distributions leveraged by these models.

Figures

Figures reproduced from arXiv: 2506.21570 by the authors.

Figure 2
Figure 2. (a) Validation losses across tokenizers. Naive and lag tokenizations yield significantly worse zero-shot, i.e. initial, vali￾dation loss with pre-trained weights against random initializations. Nevertheless, after 1M time series training tokens, pre-trained models all have lower loses than their randomly initialized counter￾parts. Notably, each pre-trained model eventually achieves similar or lower validation loss c… view at source ↗
Figure 3
Figure 3. Validation losses across backbones sizes. Validation losses of models initialized with language weights decrease as model size increases, whereas those of randomly initialized do not. Moreover, across all model scales, the randomly initialized models converge early in training while the models with pre-trained weights did not converge within the length of training tested. In the low-data regime, pretrained larger mo… view at source ↗
Figure 5
Figure 5. Difference in validation loss (random minus lan￾guage) across tokenizers for the 220M parameter T5 back￾bone [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Difference in validation loss (random minus lan￾guage) across model backbone scales [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Mean validation loss across hyperparameter configurations [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Validation losses across hyperparameter configurations. C. Embeddings As mentioned above, language data is discretely valued whereas time series data is continuously valued. This distinction affects how we usually embed tokens for LLMs, which explicitly assign learnabl…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 5 canonical work pages

  1. [1]

    F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S

    Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., Zschiegner, J., Maddix, D. C., Wang, H., Mahoney, M. W., Torkkola, K., Wilson, A. G., Bohlke-Schneider, M., and Wang, Y. Chronos: Learning the language of time series, 2024. URL https://arxiv.org/abs/2403.07815

    arXiv 2024

  2. [2]

    Bayazi, M. J. D., Ghonia, H., Riachi, R., Aristimunha, B., Khorasani, A., Arefin, M. R., Darabi, A., Dumas, G., and Rish, I. General-purpose brain foundation models for time-series neuroimaging data. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024

    work page 2024

  3. [3]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., G...

    arXiv 2022

  4. [4]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts,...

    arXiv 2022

  5. [5]

    Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters, 2024. URL https://arxiv.org/abs/2310.07820

    arXiv 2024

  6. [6]

    Scaling laws for transfer, 2021

    Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer, 2021

    work page 2021

  7. [7]

    Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., and Wen, Q

    Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., and Wen, Q. Time-llm: Time series forecasting by reprogramming large language models, 2023

    work page 2023

  8. [8]

    J., Brown, T

    Kaplan, J., McCandlish, S., Henighan, T. J., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. ArXiv, abs/2001.08361, 2020. URL https://www.arxiv.org/abs/2001.08361

    arXiv 2001

Show all 19 references
  1. [9]

    Pqmass: Probabilistic assessment of the quality of generative models using probability mass estimation

    Lemos, P., Sharief, S., Malkin, N., Salhi, S., Stone, C., Perreault-Levasseur, L., and Hezaveh, Y. Pqmass: Probabilistic assessment of the quality of generative models using probability mass estimation. arXiv preprint arXiv:2402.04355, 2024

    2024 arXiv

  2. [10]

    Pretrained transformers as universal computation engines, 2021

    Lu, K., Grover, A., Abbeel, P., and Mordatch, I. Pretrained transformers as universal computation engines, 2021. URL https://arxiv.org/abs/2103.05247

    2021 arXiv

  3. [11]

    On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines, 2021

    Mosbach, M., Andriushchenko, M., and Klakow, D. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines, 2021. URL https://arxiv.org/abs/2006.04884

    2021 arXiv

  4. [12]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019. URL http://arxiv.org/abs/1910.10683

    1910 arXiv

  5. [13]

    R., Ghonia, H., Bhagwatkar, R., Khorasani, A., Bayazi, M

    Rasul, K., Ashok, A., Williams, A. R., Ghonia, H., Bhagwatkar, R., Khorasani, A., Bayazi, M. J. D., Adamopoulos, G., Riachi, R., Hassen, N., Biloš, M., Garg, S., Schneider, A., Chapados, N., Drouin, A., Zantedeschi, V., Nevmyvaka, Y., and Rish, I. Lag-llama: Towards foundation...

    2024 arXiv

  6. [14]

    Requeima, J., Bronskill, J., Choi, D., Turner, R., and Duvenaud, D. K. Llm processes: Numerical predictive distributions conditioned on natural language. Advances in Neural Information Processing Systems, 37: 0 109609--109671, 2024

    2024

  7. [15]

    W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D

    Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022. URL https://arxiv.org/abs/2109.10686

    2022 arXiv

  8. [16]

    Y., Guu, K., Yu, A

    Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners, 2022. URL https://arxiv.org/abs/2109.01652

    2022 arXiv

  9. [17]

    R., Ashok, A., Marcotte, \'E ., Zantedeschi, V., Subramanian, J., Riachi, R., Requeima, J., Lacoste, A., Rish, I., Chapados, N., et al

    Williams, A. R., Ashok, A., Marcotte, \'E ., Zantedeschi, V., Subramanian, J., Riachi, R., Requeima, J., Lacoste, A., Rish, I., Chapados, N., et al. Context is key: A benchmark for forecasting with essential textual information. arXiv preprint arXiv:2410.18959, 2024

    arXiv

  10. [18]

    Unified training of universal time series forecasting transformers, 2024

    Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., and Sahoo, D. Unified training of universal time series forecasting transformers, 2024. URL https://arxiv.org/abs/2402.02592

    2024 arXiv

  11. [19]

    One fits all:power general time series analysis by pretrained lm, 2023

    Zhou, T., Niu, P., Wang, X., Sun, L., and Jin, R. One fits all:power general time series analysis by pretrained lm, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.