{"id":"cb76cc94-50d9-4c06-ab54-d942ef9a6549","arxiv_id":"2412.16643","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Using K-means and DTW to retrieve similar historical sequences and add them to an LLM prompt improves M4 forecast accuracy by 2.97% on average over Time-LLM.","lead":"TimeRAG adds a retrieval step to frozen large language models used for time series forecasting: it stores representative historical sequences in a knowledge base, finds similar ones with dynamic time warping, and pastes them into the prompt. On the M4 benchmark this reportedly improves forecasting accuracy by about 3 percent on average over the Time-LLM baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.97% RAG gain is not causally identified: TimeRAG vs Time-LLM changes the LLM backbone, prompt content, and prompt length simultaneously, so the improvement cannot be attributed to DTW retrieval without a controlled ablation.","rationale":"The reader's weakest assumption is valid and central: the paper never isolates whether DTW-retrieved references, rather than added prompt context, cause the gain. My concern is a deeper version of the same lack of isolation, because the TimeRAG versus Time-LLM comparison also confounds the frozen LLM backbone and possibly the training recipe; Section III-A specifies Llama3 for TimeRAG but does not specify the backbone behind the Time-LLM column. If the baseline uses a different model family, then even a perfect end-to-end replication would not identify the retrieval effect. The paper is not internally inconsistent, and the proposed pipeline is plausible, so a controlled ablation rather than rejection is the right remedy. The concrete test above is the minimal experiment that separates retrieval content from prompt-length and baseline-model effects; it also addresses noise by requiring multiple seeds. Because the reader already assigned CONDITIONAL and this stress test reinforces that condition rather than overturning the paper, the verdict should remain unchanged.","tokens_in":8024,"tokens_out":5576,"duration_ms":48453,"concrete_test":"Re-run the M4 comparison under one frozen LLM (Llama3) with identical training schedule, reprogramming layer, and evaluation protocol, in four arms: (1) Time-LLM's original prompt; (2) TimeRAG with DTW top-5 references; (3) TimeRAG with K random references from the same knowledge base, matched for length and format; (4) TimeRAG with the same prompt length but references replaced by neutral filler text. Use at least 5 random seeds and report mean ± std for SMAPE/MASE/OWA per frequency. If arm 2 does not beat arm 1 beyond seed noise, or does not beat arms 3 and 4, the 2.97% claim is not attributable to DTW retrieval and the paper should be revised to a weaker or conditional claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests entirely on the end-to-end gap in Table II between TimeRAG and Time-LLM. Section III-A states that TimeRAG is 'trained based on Llama3' but never states which frozen LLM and training recipe produced the Time-LLM column. Since RAG is claimed to boost an 'original model' without modifying its parameters, the original model must be identical except for retrieval; if the Time-LLM numbers come from a different backbone (e.g., the original Time-LLM paper), part or all of the 2.97% could be a base-model effect. Independently of the backbone, the TimeRAG prompt differs from Time-LLM's in at least two non-RAG ways: it adds five reference sequences plus text descriptions and roughly triples the input length. Section III-B attributes the gain to 'the augmented knowledge base' without any ablation that holds prompt format and length fixed and varies only whether the added references are DTW-similar. Many Table II differences are tiny (yearly SMAPE 15.317 vs 15.318; daily OWA 1.057 vs 1.059), so the aggregate 2.97% is also sensitive to noise and to the one large weekly MASE movement. In short, the design establishes correlation between the TimeRAG pipeline and better scores, not causation from retrieval.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes TimeRAG, a retrieval-augmented framework for LLM time series forecasting. It constructs a knowledge base from M4 sequences via sliding-window slicing and K-means clustering, retrieves top-K similar reference sequences using Dynamic Time Warping, and concatenates these references with the query as a textual prompt fed to a frozen LLM through a reprogramming layer. The abstract and Section III-B claim that RAG improves the prediction accuracy of the original model by 2.97% on average, specifically reducing SMAPE by 1.13%, MASE by 4.78%, and OWA by 3.00% relative to Time-LLM on the M4 benchmark.","tokens_in":8301,"tokens_out":5169,"duration_ms":46089,"significance":"If the result holds, the contribution is practically valuable: a simple retrieval wrapper that improves a frozen LLM forecaster without modifying the foundation model's parameters, evaluated on the standard public M4 benchmark. The paper reports all three M4 metrics by frequency, and TimeRAG achieves the best average MASE and OWA among the included baselines, with top-three results in 14 of 18 metric-frequency cells. The main strength is the clear high-level idea and the end-to-end evidence on a widely used benchmark. However, the central causal claim that DTW-based retrieval drives the gains is not yet established, because no ablation isolates retrieval from prompt-format and prompt-length changes, no statistical uncertainty is reported, and the comparability of the Time-LLM baseline is under-specified. The paper would also benefit from code and explicit hyperparameter values.","major_comments":[{"comment":"The 2.97% benefit is attributed to retrieval, but the comparison changes several variables at once. Relative to Time-LLM, TimeRAG adds five retrieved reference sequences, their text descriptions, and substantially longer prompt context. Section III-B credits the improvement to 'the augmented knowledge base' with no ablation that holds the prompt format and length fixed and varies only whether the added references are DTW-similar. Since the paper's central claim is that retrieval causes the improvement, please add ablations such as (i) TimeRAG with randomly retrieved references, (ii) TimeRAG with longer context but no retrieved references, and (iii) TimeRAG with references that are similar under a different distance. Without these, the observed gain may be a prompt-engineering effect rather than a retrieval effect.","section":"Section III-B and Section II-C"},{"comment":"The baseline comparison is not fully specified. The paper states that TimeRAG is trained based on Llama3 with a maximum of 50 epochs, Adam, SMAPE loss, and a maximum learning rate of 0.01, but it never states which frozen LLM and training recipe produced the Time-LLM column. If the Time-LLM numbers are taken from the original Time-LLM paper or use a different backbone, part of the reported 2.97% gap could be a base-model or training-effect rather than an effect of RAG. Please state explicitly that Time-LLM was reproduced with the same frozen LLM, reprogramming layer, prompt format (minus retrieval), optimizer, epochs, learning rate schedule, and data split; otherwise the central comparison is confounded.","section":"Section III-A and Table II"},{"comment":"The statistical robustness of the headline gain is not demonstrated. Several frequency-level differences are extremely small (yearly SMAPE 15.317 vs 15.318, daily OWA 1.057 vs 1.059, quarterly MASE 1.230 vs 1.229), while the aggregate improvement is driven substantially by one large weekly MASE movement from 4.330 to 3.762. No error bars, number of independent runs, random seeds, or significance tests are reported. Please report variance across runs/seeds and a paired significance test, either per series or across the 18 metric-frequency cells, so that the reader can assess whether the 2.97% average improvement is systematic rather than noise.","section":"Table II and Section III-B"},{"comment":"The relationship between the knowledge base and the train/test split is ambiguous and potentially dangerous. Section III-A says knowledge bases are built for each M4 frequency and 'the remaining data' is then split into training, validation, and test sets, while the note in Fig. 1 says the historical sequence is independent of model training and test sets. If the knowledge base is constructed from the full M4 dataset, including test-series slices or future portions of test series, retrieval could leak information and inflate the reported improvements. Please clarify that the knowledge base is built from the training split only, and verify that no knowledge-base segment overlaps the test horizons of any test sequence.","section":"Section III-A, Table I, and Fig. 1 note"}],"minor_comments":[{"comment":"Equation (2) defines d(i, j) = (Xinput_i, XL_j)^2, which appears to be a typo for d(i, j) = (Xinput_i - XL_j)^2; please correct the formula.","section":"Eq. (2)"},{"comment":"The abstract says 'datasets from various domains,' but the experiments use only the M4 dataset. Either add additional datasets or soften the claim to 'domains within M4.'","section":"Section III-A"},{"comment":"Hyperparameters are not reported: the number of K-means clusters K, the sliding-window length L, the step size S, and the number of retrieved sequences (stated as five in Section III-A but not listed as a table value). Please provide these values for reproducibility.","section":"Section II-B and Section III-A"},{"comment":"The phrase 'accuracy improvement of 2.97%' is not precisely defined. Please state the formula used to aggregate the SMAPE, MASE, and OWA relative reductions, since averaging percentage reductions of different metrics is not self-explanatory.","section":"Section III-B"},{"comment":"Reference [23] cites 'Autoformer: Searching transformers for visual recognition' by Chen et al., but the Autoformer time-series model is a different paper; please replace it with the correct citation (Wu et al., 'Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting').","section":"References"},{"comment":"Reference [10] ends with '[Online]. Available:' but the URL is missing; please complete the citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TimeRAG is the kind of paper that reads better as a workshop entry than as a settled result. The new piece is real: nobody in the cited references has combined a k-means time-series knowledge base, DTW retrieval, and prompt augmentation for a frozen LLM forecaster, and the M4 numbers in Table II are new. The idea is cheap and easy to try, and it does not require fine-tuning the backbone. Credit where due: the pipeline is described clearly, the math is standard, and the table shows TimeRAG placing top-three in 14 of 18 frequency–metric cells, with a genuinely large gain on weekly MASE (4.330 to 3.762).\n\nThe soft spots are exactly the stress-test note, and they are load-bearing rather than cosmetic. The comparison against Time-LLM is not controlled. Section III-A says TimeRAG uses Llama3 and a reprogramming layer trained with SMAPE loss, but it never says which frozen LLM or training recipe produced the Time-LLM column. If that column came from the original Time-LLM paper, part or all of the 2.97% could be a base-model effect rather than retrieval. The prompt also differs in two non-RAG ways: it adds five reference sequences plus generated text descriptions and roughly triples the input length. Section III-B credits the full gain to 'the augmented knowledge base' without an ablation that holds prompt format and length fixed and varies only whether the added sequences are DTW-similar. Several individual differences are within rounding (yearly SMAPE 15.317 vs 15.318; daily OWA 1.057 vs 1.059), and the aggregate 2.97% is heavily influenced by the one large weekly MASE move. No error bars, significance tests, or code are provided.\n\nSo the paper is honest engineering progress, but the causal claim is not established. I would not want to cite the 2.97% in my own work until there is a reproducibility release with an ablation that contrasts DTW-retrieved references against random or length-matched references. That experiment is straightforward and would likely settle it.\n\nWho this is for: practitioners with frozen LLM forecasting pipelines, and researchers working on RAG for time series, both of whom can try the method in an afternoon. It deserves a serious referee rather than a desk reject, but the referee should demand the code and the ablation. My verdict: conditional accept at best; as is, it is a useful preliminary report.","headline":"A plausible RAG-for-frozen-LLM forecasting trick with a small reported M4 gain, but the 2.97% claim is not causally identified and needs an ablation and code before I'd trust it.","tokens_in":8854,"tokens_out":2533,"would_cite":false,"duration_ms":22044,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TimeRAG claims that retrieval-augmented generation improves LLM time series forecasting by 2.97% on average.","keywords":["time series forecasting","retrieval-augmented generation","large language models","dynamic time warping","knowledge base","M4 benchmark","prompt reprogramming"],"falsifier":"A controlled experiment that replaces DTW selection with random or fixed reference sequences while keeping prompt length constant; if accuracy does not degrade, the central retrieval claim is falsified.","tokens_in":7809,"feed_emoji":"📈","tokens_out":3535,"duration_ms":29166,"temperature":0.7,"pith_summary":"This paper tries to establish that retrieval-augmented generation (RAG) can make frozen large language models better at time series forecasting without retraining their parameters. It proposes TimeRAG, which builds a knowledge base of historically similar sequences, retrieves the most pattern-like ones with dynamic time warping, and pastes them into the prompt alongside the query sequence. On the M4 benchmark, the paper reports a 2.97% average accuracy improvement over the Time-LLM baseline, with the largest gains on weekly data. A sympathetic reader would care because the method points to a cheap, domain-adaptive way to improve LLM forecasters by supplying relevant examples at inference time.","feed_headline":"Retrieved examples make LLM forecasts 2.97% more accurate","feed_subtitle":"DTW-similar historical series added to prompts trim error on M4 across all six frequencies.","key_machinery":"The mechanism is the retrieval-augmented prompt: a Time Series Knowledge Base built by sliding-window slicing plus K-means clustering, a Dynamic Time Warping (DTW) similarity measure that tolerates temporal distortions, and a reprogramming layer that aligns the numeric sequence with text prompts. DTW computes the minimal-cost warping path between the query and each candidate, and the top-K candidates are inserted into the prompt as extra reference data. This added context is what the paper credits for the accuracy gains.","core_discovery":"The central claim is that adding a DTW-based retrieval stage to an existing LLM forecaster (Time-LLM) improves forecast accuracy across all six M4 frequencies without modifying the LLM's weights. TimeRAG first slices historical sequences, uses K-means to store representative segments in a knowledge base, retrieves the top-5 closest segments to each query under DTW, and concatenates them with the query into one natural-language prompt via Time-LLM's reprogramming layer. In experiments the paper reports average reductions of 1.13% in SMAPE, 4.78% in MASE, and 3.00% in OWA relative to Time-LLM, and top-three performance in 14 of 18 metric-frequency comparisons.","pith_inferences":["The paper does not isolate retrieval from other prompt changes, so a plausible extension is to test whether randomly chosen reference sequences or longer prompts alone reproduce the gain; if they do, the DTW similarity is not the active ingredient.","The knowledge base construction can be seen as a form of exemplar selection for in-context learning; comparing DTW to cosine similarity or learned embeddings on the same prompt template would clarify when retrieval helps.","Because the authors report separate knowledge bases per frequency, the approach implicitly assumes within-frequency similarity is the right inductive bias; cross-frequency retrieval is a natural stress test.","The reported 2.97% average improvement is modest, so a useful next experiment would measure statistical significance and variance across multiple seeds and knowledge-base sizes."],"forward_implications":["Frozen LLMs can be adapted to new forecasting domains by retrieving relevant examples, without gradient updates to the model.","The method's success on M4 suggests DTW is a practical similarity signal for finding useful reference sequences across frequencies.","TimeRAG's average MASE of 2.72 and OWA of 1.03 would place it ahead of the reported Transformer-based and linear baselines on M4.","The approach should transfer to any LLM forecaster that uses textual prompts, since the retrieval and prompt-rewriting stages are model-agnostic."],"supporting_citations":[{"why":"Defines retrieval-augmented generation, the core technique being applied to time series.","marker":"[10]"},{"why":"Time-LLM is the baseline LLM forecaster and the source of the reprogramming layer that TimeRAG builds on.","marker":"[13]"},{"why":"Provides the dynamic time warping distance used to retrieve similar sequences from the knowledge base.","marker":"[11]"},{"why":"Introduces the M4 dataset that supplies all evaluation data and benchmark comparisons.","marker":"[12]"},{"why":"Motivates the sliding-window segmentation by showing LLMs lose information in long contexts.","marker":"[16]"}],"fun_headline_variants":["RAG boosts LLM time series forecasting by 2.97%","Retrieval-augmented LLM forecasting: 2.97% average gain","TimeRAG: DTW-retrieved examples improve LLM forecasts","Training-free retrieval improves LLM time series accuracy","LLM forecasters get 2.97% boost from retrieved historical patterns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper credits the accuracy gain to DTW-based retrieval of similar patterns, but the prompt also gets longer and contains more text; without an ablation that isolates retrieval, the improvement might come from any added context.","fun_headline_variants_meta":{"raw":{"variants":["RAG boosts LLM time series forecasting by 2.97%","Retrieval-augmented LLM forecasting: 2.97% average gain","TimeRAG: DTW-retrieved examples improve LLM forecasts","Training-free retrieval improves LLM time series accuracy","LLM forecasters get 2.97% boost from retrieved historical patterns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2533,"prompt_tokens":825,"completion_tokens":1708,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":1614}},"tokens_in":441,"tokens_out":1708,"duration_ms":11936,"temperature":1.0,"reasoning_tokens":1614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:22:52.563642+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that replaces DTW selection with random or fixed reference sequences while keeping prompt length constant; if accuracy does not degrade, the central retrieval claim is falsified.","supporting_citations":[{"cited_title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the core technique being applied to time series."},{"cited_title":"Lost in the middle: How language models use long contexts,","cited_arxiv_id":null,"evidence_quote":"Motivates the sliding-window segmentation by showing LLMs lose information in long contexts."}],"review_version":1}